JP2019160171A - Display device and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device and a display method capable of displaying a maintenance view having high visual comprehension and workability to a person in charge of maintenance and inspection using AR technology. SOLUTION: An image pickup unit for acquiring an image in a real space, a display unit for displaying an image, component information related to a component of a device to be maintained, augmented reality information associated with the component information, and maintenance contents. The control unit includes a storage unit that stores the display mode of the augmented reality information for each component, and a control unit that controls the operation of the imaging unit and the display unit and displays the augmented reality information superimposed on the image. , The image of the target device acquired by the imaging unit is displayed on the display unit, the parts in the image are determined based on the component information acquired from the storage unit, and the augmented reality information acquired from the storage unit is to be maintained. The display mode of the parts and the parts that are not subject to maintenance is different. [Selection diagram] Fig. 2

Description

  The present invention relates to a display device and a display method.

  Augmented reality (AR) technology is known that superimposes and displays information created by a computer on information given from an actual environment.

  As a device using such a technique, a terminal device that can suppress the complication of the screen even when the amount of information to be displayed in a real space image is increased is known. In such a terminal device, the display on the screen of an object at a distance is suppressed (Patent Document 1).

  However, when AR technology is applied to, for example, maintenance and inspection of an image forming apparatus such as a printer, components and the like on which annotations are to be displayed exist closely together in the apparatus. With technology, it is difficult to suppress display on the screen.

  The present invention has been made in view of the above-described problem, and can display a maintenance view with high understanding and workability for a person in charge of maintenance inspection using AR technology. An object is to provide a display device and a display method.

  In order to solve the above-described problems and achieve the object, the present invention provides an imaging unit that acquires an image of real space, a display unit that displays the image, component information regarding components of a maintenance target device, and the component Controlling the operation of the imaging unit and the display unit, the storage unit storing the augmented reality information associated with the information, and the display mode of the augmented reality information for each part according to the content of the maintenance, A control unit that superimposes and displays the augmented reality information on the image, and the control unit causes the display unit to display the image of the target device acquired by the imaging unit, and from the storage unit. The component in the image is determined based on the acquired component information, and the component that is the maintenance target and the component that is not the maintenance target among the augmented reality information acquired from the storage unit Varying the mode of the display.

  According to the present invention, it is possible to display a maintenance view that is visually understandable and workable for a person in charge of performing maintenance and inspection using the AR technology.

FIG. 1 is a diagram illustrating an example of a configuration of a display system according to the first embodiment. FIG. 2 is a diagram for explaining a hardware configuration of a multifunction peripheral that can be connected to the portable terminal according to the first embodiment. FIG. 3 is a diagram illustrating a hardware configuration of the portable terminal according to the first embodiment. FIG. 4 is a diagram illustrating an example of a functional configuration of the portable terminal according to the first embodiment. FIG. 5 is a diagram illustrating a display example on the portable terminal according to the first embodiment. FIG. 6 is a diagram illustrating imaging data captured from the imaging unit in the maintenance operation mode on the portable terminal according to the first embodiment. FIG. 7 shows maintenance mode-specific AR annotation display information held by the storage unit of the portable terminal according to the first embodiment. FIG. 8 is a diagram illustrating a display example of a maintenance operation execution view on the portable terminal according to the first embodiment. FIG. 9 is a flowchart illustrating an example of a display processing procedure in the portable terminal according to the first embodiment. FIG. 10 is a diagram illustrating a display example of the maintenance operation execution view on the portable terminal according to the comparative example. FIG. 11 is a diagram illustrating a display example of the maintenance operation execution view on the portable terminal according to the first modification of the first embodiment. FIG. 12 is a diagram illustrating a display example of the maintenance operation execution view on the portable terminal according to the second modification of the first embodiment. FIG. 13 is a diagram illustrating imaging data of the rear surface of the multifunction peripheral captured from the imaging unit in the maintenance operation mode on the portable terminal according to the second embodiment. FIG. 14 is coordinate definition information for each component held by the storage unit of the portable terminal according to the second embodiment. FIG. 15 is a diagram illustrating an example in which AR annotation information is superimposed on a live view on the portable terminal according to the second embodiment. FIG. 16 is a diagram for explaining a component discrimination method when there is no reference point in the image on the portable terminal according to the second embodiment. FIG. 17 is a flowchart illustrating an example of a display processing procedure in the portable terminal 150 according to the second embodiment.

  Hereinafter, embodiments of a display device and a display method according to the present invention will be described in detail with reference to FIGS. Further, the present invention is not limited by the following embodiments, and components in the following embodiments can be easily conceived by those skilled in the art, are substantially the same, and have a so-called equivalent range. Is included. Furthermore, various omissions, substitutions, changes, and combinations of the components can be made without departing from the scope of the following embodiments.

[Embodiment 1]
First, Embodiment 1 will be described with reference to FIGS.

(Overall configuration of display system)
FIG. 1 is a diagram illustrating an example of a configuration of a display system 1 according to the first embodiment. As shown in FIG. 1, the display system 1 includes a multifunction peripheral (MFP) 200 as an image forming apparatus and a portable terminal 100 as a display device. The multifunction device 200 and the portable terminal 100 are connected via a communication network 90 such as the Internet, for example. The line of the communication network 90 may be a wired connection, a wireless connection, or a connection method in which both are mixed. Here, it is assumed that the multifunction device 200 and the portable terminal 100 have a wireless communication function and exchange information by wireless connection.

  The portable terminal 100 is a tablet PC (Personal Computer), a smartphone, a mobile PC, a wearable terminal, or the like having an AR function. The portable terminal 100 is configured to be able to exchange information with the multifunction device 200 via the communication network 90.

  The multifunction device 200 is a device sold or leased to a customer by a device manufacturer or device vendor who is an operator of the portable terminal 100, and is installed in the customer environment of the customer. The multi-function device 200 has at least two functions among a printing function, a copying function, a scanner function, and a facsimile function. In the example of FIG. 1, only one multifunction device 200 is illustrated, but a plurality of multifunction devices may exist.

(Hardware configuration of MFP)
FIG. 2 is a diagram for explaining a hardware configuration of the multifunction device 200 that can be connected to the portable terminal 100 according to the first embodiment. 2 mainly shows a configuration related to wireless connection with the portable terminal 100, and configurations related to other functions of the multifunction device 200 are simplified or omitted.

  As shown in FIG. 2, the multifunction device 200 includes a controller 210, a printer engine 227 connected to the controller 210, and a panel device 228. The controller 210 includes a CPU (Central Processing Unit) 211, a ROM (Read-Only Memory) 212, a RAM (Random Access Memory) 213, an NVRAM (Non-Volatile Random Access Memory) 214, a communication interface (I / F) network 15 (I / F). An I / F 216, an engine I / F 217, a panel I / F 218, and an HDD (Hard Disk Drive) 219 are provided.

  The controller 210 executes print processing in accordance with the operation mode set at the time of the operation and print control data from a host PC (Personal Computer) 236 connected via the network I / F 216. Further, the controller 210 executes various operations input from the portable terminal 100 connected via the communication I / F 215.

  The CPU 211 processes print data from the host PC 236 in accordance with a control program stored in the ROM 212.

  The ROM 212 stores a control program in which the CPU 211 performs overall management of data executed in the controller 210 and peripheral modules.

  The RAM 213 is used as a page memory created by the CPU 211 according to the control program, a work memory necessary for the control program to operate, and the like. The RAM 213 is used as a buffer for managing and temporarily storing print data from the host PC 236 in units of pages. The RAM 213 is also used as a bit map memory for storing image data converted from print data.

  The NVRAM 214 is a non-volatile RAM in which data that is to be retained even after the power is turned off, for example, information such as predetermined setting values and predetermined printing condition information set via the panel device 228 used for printing control is stored. is there.

  The HDD 219 is a hard disk such as a magnetic disk, and is used for accumulating print data from the host PC 236 or the like and accumulating bitmap data converted from the print data. The HDD 219 is used for various purposes as a buffer, a work area, or the like for storing such data or for temporary storage. The HDD 219 may be used for storing various types of font data used for printing.

  The communication I / F 215 is an interface that performs wireless communication (Wi-Fi communication) with an external device such as the portable terminal 100.

  The network I / F 216 is an interface that transmits and receives print data sent as a print job from the host PC 236 to the multifunction device 200 and a status signal from the host PC 236 to the multifunction device 200.

  The engine I / F 217 is an interface that exchanges control signals from the controller 210 to the printer engine 227 and status signals from the printer engine 227 to the controller 210.

  For example, a printer engine 227 is connected to the engine I / F 217. The printer engine 227 is a mechanism that accepts print data input and prints out an image. The printer engine 227 can form an image by a method such as a laser method, an LED method, or an ink jet method.

  The panel I / F 218 exchanges signals with the panel device 228 to the controller 210 so that the user can switch the status display of the multi-function device 200, the operation mode, the font, and the like via the panel device 228. Interface.

  A panel device 228 is connected to the panel I / F 218. The panel device 228 is a man-machine interface unit that receives input from the user of the multifunction device 200 and presents information to the user. The panel device 228 includes buttons that allow the multifunction device 200 to be operated and indicators (display devices such as LEDs and LCDs, speakers, etc.) that display the state of the multifunction device 200.

(Hardware configuration of portable terminal)
FIG. 3 is a diagram illustrating a hardware configuration of the portable terminal 100 according to the first embodiment. As shown in FIG. 3, the portable terminal 100 includes a controller 110, a display device 121, an input device 122, a camera 123, a GPS (Global Positioning System) module 124, and an orientation sensor 125 connected to the controller 110. The controller 110 includes a CPU 111, a ROM 112, a RAM 113, an EEPROM (Electrically Erasable Programmable Read-Only Memory) 114, a communication I / F 115, and an input / output I / F 116.

  The controller 110 controls the entire portable terminal 100 and implements functions such as searching, acquiring and displaying information, taking and playing pictures and videos, playing and storing them, and sending and receiving mails via the communication network 90, for example. . In addition, necessary information is appropriately displayed to the user via the display device 121 for realizing those functions.

  The CPU 111 executes a process of controlling the operation of the portable terminal 100 according to a control program stored in the ROM 112 and controlling the multifunction device 200 wirelessly connected to the portable terminal 100.

  The ROM 112 stores a control program for the CPU 111 to manage data executed in the controller 110 and to control peripheral modules in an integrated manner.

  The RAM 113 is used as a work memory necessary for the CPU 111 to operate the control program. The RAM 113 is also used as a buffer for temporarily storing information acquired via the communication network 90 and information acquired via the camera 123.

  The EEPROM 114 is a nonvolatile ROM that stores data that is to be retained even when the power is turned off, for example, setting information of the portable terminal 100.

  The communication I / F 115 is an interface that performs wireless communication (Wi-Fi communication) with an external device such as the MFP 200.

  The input / output I / F 116 transmits and receives signals between the controller 110 and various devices provided in the portable terminal 100, such as the display device 121, the input device 122, the camera 123, the GPS module 124, the direction sensor 125, and the like. Interface.

  The display device 121 displays characters, numbers, various screens, operation icons, images acquired by the camera 123, and the like. The display device 121 is a CRT (Cathode Ray Tube) display, a LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or the like. The display device 121 may be a touch panel device. In this case, the display device 121 further includes a contact sensor and a position sensor not shown.

  The input device 122 performs operations such as input of characters and numbers, selection of various instructions, and movement of a cursor. The input device 122 may be, for example, a keypad provided on the casing of the portable terminal 100, or may be a device such as a mouse or a keyboard. In addition to these configurations, a touch panel device integrated with the display device 121 may be used.

  The camera 123 is a part of the portable terminal 100 and is provided on the back surface of the display device 121, for example. The direction in which the portable terminal 100 is directed and the video imaged by the camera 123 and displayed on the display device 121 are set to be substantially the same. Although the resolution of the camera 123 can be selected as appropriate, it is desirable that the photographed photo data and video data have a quality that can be fully utilized for printing and video reproduction. A plurality of cameras 123 may be arranged to function as a stereo camera.

  The GPS module 124 receives radio waves from a plurality of GPS satellites, and acquires the current position information of the portable terminal 100 from their intensities. The GPS module 124 communicates with the line of the communication network 90 via the communication I / F 115 to acquire information on satellites existing near the portable terminal 100 as assist data, and perform AGPS (Assisted Global). (Positioning System) module.

  The direction sensor 125 detects the direction in which the portable terminal 100 is facing, the attitude and angle of the portable terminal 100. The direction sensor 125 is a combination of a plurality of geomagnetic sensors, for example. The direction sensor 125 may be a motion sensor that combines a plurality of acceleration sensors or gyros in addition to the geomagnetic sensor. Examples of the motion sensor include a 6-axis sensor that combines a triaxial geomagnetic sensor and a triaxial acceleration sensor, and a 5-axis sensor that combines a 3-axis geomagnetic sensor and a biaxial acceleration sensor.

(Functional configuration of portable terminal)
FIG. 4 is a diagram illustrating an example of a functional configuration of the portable terminal 100 according to the first embodiment. For convenience of explanation, FIG. 4 mainly illustrates functions related to the first embodiment, but the functions of the portable terminal 100 are not limited to these.

  As illustrated in FIG. 4, the portable terminal 100 as the display device according to the first embodiment is associated with the display unit 20 that displays an image and the component information and component information regarding the components of the multifunction device 200 as the maintenance target device. The operation of the storage unit 30 for storing the augmented reality information and the display mode of the augmented reality information for each part according to the content of the maintenance, the imaging unit 12 and the display unit 20, and the augmented reality information in the image The control unit 10 displays the image of the multifunction device 200 acquired by the imaging unit 12 on the display unit 20 and displays an image based on the component information acquired from the storage unit 30. In the augmented reality information acquired from the storage unit 30, the display mode of the parts to be maintained and the parts not to be maintained is changed. Below, the function of the portable terminal 100 of Embodiment 1 is demonstrated in detail.

  The portable terminal 100 includes a control unit 10, a display unit 20, and a storage unit 30. These are communicably connected to each other. Further, the communication unit 11, the imaging unit 12, the position information acquisition unit 13, the azimuth information acquisition unit 14, and the key input unit 15 are connected to the control unit 10. The display unit 20 includes a position detection unit 21, a display controller 22, a screen providing unit 23, and a contact detection unit 24.

  The control unit 10 controls the operation of each unit and realizes predetermined information processing. The control unit 10 is a functional block that is virtually configured by executing a program stored in the storage unit 30 on the CPU 111 in FIG. 3, and includes the communication unit 11, the imaging unit 12, and the position of the portable terminal 100. Various functions of the portable terminal 100 can be performed by exchanging data and control signals with each functional block such as the information acquisition unit 13, the direction information acquisition unit 14, the key input unit 15, the display unit 20, and the storage unit 30. Realize.

  The communication unit 11 is a module that is connected to a line of the communication network 90 and communicates with other terminal devices and server systems. Examples of the communication here include circuit-switched communication performed by communication connection with other terminal apparatuses, and packet-switched communication performed with a server system connected via a communication network 90 line. Further, when acquiring the position information of the portable terminal 100 by the AGPS method, the communication unit 11 cooperates with the position information acquisition unit 13 to assist the acquisition of the terminal position information. The communication unit 11 is realized by, for example, the communication I / F 115 in FIG.

  The imaging unit 12 acquires a real space image. More specifically, the imaging unit 12 is a module that has a predetermined optical system and an image receiving element and provides a function of acquiring a digital image. When a digital image is captured by the portable terminal 100, the activated imaging unit 12 normally performs a process of acquiring data by the image receiving element at a predetermined exposure time from a subject image acquired by the optical system at a predetermined time interval. The live view function is realized by repeatedly displaying the data on the display unit 20. In this state, when the user further performs a predetermined operation with the key input unit 15, the imaging unit 12 generates image data under imaging conditions set from the subject image acquired by the optical system, and the generated image data Is stored in the storage unit 30. The portable terminal 100 using the AR technology displays information that brings augmented reality to the real space image displayed on the display unit 20 by the above-described live view function. The imaging unit 12 is realized by, for example, the camera 123 in FIG.

  The position information acquisition unit 13 acquires position information of the portable terminal 100. The position information acquisition unit 13 is realized by, for example, the GPS module 124 of FIG. As described above, it is preferable to employ a module compatible with the AGPS method because the position information of the portable terminal 100 can be acquired more quickly and accurately. However, the position information acquisition unit 13 is not limited to the one using GPS or AGPS, and may be configured to acquire terminal position information without depending on GPS. For example, a method may be used in which a plurality of wireless communication base stations such as WiFi and Bluetooth (registered trademark) are arranged in the space, and terminal location information is acquired based on the radio field intensity therefrom.

  The position information acquisition unit 13 detects the direction in which the portable terminal 100 is facing. The control unit 10 can recognize the posture and angle at which the user holds the portable terminal 100 from the detection value of the position information acquisition unit 13. The position information acquisition unit 13 is realized by, for example, the orientation sensor 125 in FIG. When the azimuth sensor 125 is a motion sensor, it is possible to detect not only information on the attitude and angle of the portable terminal 100 but also the direction and speed of the operation in which the user moves or tilts the portable terminal 100. It becomes.

  The key input unit 15 receives an input by a user's key operation. When the user operates, a key event which is a signal corresponding to each key or button is notified to the control unit 10 and is used for various operations and controls. The key input unit 15 is realized by, for example, the input device 122 of FIG.

  The display unit 20 displays various screens. Hereinafter, it is a structure when the display part 20 is a touch panel. The display controller 22 controls the entire display unit 20. The screen providing unit 23 displays a screen under the control of the display controller 22. The contact detection unit 24 detects contact of the user with the screen. The position detection unit 21 detects the position of the screen touched by the user. The display unit 20 is realized by, for example, the display device 121 of FIG. 3, the input device 122, a contact sensor (not shown), a position sensor, and a program that operates on the CPU 111.

  The touch panel type display unit 20 receives data and control signals from the control unit 10 by the display controller 22 and displays a predetermined screen on the screen providing unit 23. In addition, the display unit 20 receives an operation of a user who touches the screen by the contact detection unit 24, detects the contact position by the position detection unit 21, and notifies the control unit 10 of a position signal regarding the contact position as an input. In the portable terminal 100, augmented reality is provided to the user by adding augmented reality information to the real space video acquired by the imaging unit 12 and displaying it according to the operation described later, and the display function and touch panel function are used. Provides a user interface.

  The storage unit 30 is a memory that stores predetermined information under the control of the control unit 10 and provides the stored information to the control unit 10. In addition, the storage unit 30 stores various programs executed by the control unit 10, and the control unit 10 reads and executes them appropriately. A unique terminal ID is assigned to the portable terminal 100. This terminal ID is also stored in the storage unit 30 together with other data. Further, the storage unit 30 stores augmented reality information (AR annotation information) to be described later and display / non-display correspondence information of the augmented reality information for each graphics object for each of a plurality of functions to be maintained. The storage unit 30 also stores component information that associates the augmented reality information with each component of the multifunction device 200. The storage unit 30 is realized by, for example, the ROM 112, the RAM 113, and the EEPROM 114 shown in FIG.

(Operation example of portable terminal)
Next, an operation example of the portable terminal 100 will be described with reference to FIGS. FIG. 5 is a diagram illustrating a display example on the portable terminal 100 according to the first embodiment. By performing an operation for instructing the start of the maintenance operation mode, the person in charge of maintenance can put the MFP 200 that is the operation target into a state in which maintenance operation can be performed, and can start the maintenance operation.

  That is, when maintenance of the multifunction device 200 is performed, an operation of pressing the maintenance button is performed from the screen display of FIG. 5 displayed on the display unit 20 of the portable terminal 100. Further, the maintenance person can further select a maintenance mode. Thereby, it is possible to switch to the display of the maintenance operation execution view.

  FIG. 6 is a diagram illustrating imaging data captured from the imaging unit 12 in the maintenance operation mode on the portable terminal 100 according to the first embodiment. By the operation of pressing the maintenance button, the display on the display unit 20 of the portable terminal 100 is first switched to the live view from the imaging unit 12 as shown in FIG.

  The control unit 10 of the portable terminal 100 performs control for specifying the component type included in the imaging data by performing image analysis of the imaging data. Each circular mark on the live view of FIG. 6 represents the identified part. In addition, the control unit 10 acquires graphics object information of AR annotation information corresponding to each component and held in the storage unit 30. Further, the control unit 10 acquires the maintenance mode-specific AR annotation display information held in the storage unit 30, and selects a component that displays the AR annotation information and a component that does not display it according to the selected maintenance mode.

  FIG. 7 shows maintenance mode-specific AR annotation display information held by the storage unit 30 of the portable terminal 100 according to the first embodiment. The maintenance modes that can be selected by the maintenance staff are divided for each part of the multifunction device 200 such as a scanner unit, a main body, and a paper feed tray unit. Alternatively, the maintenance mode may be divided for each state of the multi-function device 200 such as when a failure occurs, after inspection after recovery, or during periodic inspection. As shown in FIG. 7, in the AR annotation display information for each maintenance mode, ON / OFF of AR annotation display for each part is set according to each maintenance mode 1, 2, 3,. In FIG. 7, “0” represents a non-displayed part, and “1” represents a displayed part. For example, in maintenance mode 2, “1” is set for motors A, F, and G, and “0” is set for other components. For this reason, in the maintenance mode 2, only the motors A, F, and G are displayed as AR annotation information.

  FIG. 8 is a diagram illustrating a display example of the maintenance operation execution view on the portable terminal 100 according to the first embodiment. In FIG. 8, the AR annotation information is superimposed on the live view. As shown in FIG. 8, AR annotation information is displayed only for some of the components with circular marks. In other words, when the maintenance person selects the maintenance mode, only the AR annotation information of the parts related to the selected maintenance mode is displayed without displaying the AR annotation information of the parts not related to the selected maintenance mode. Is displayed. As described above, in the portable terminal 100, AR annotation information corresponding to a part corresponding to the maintenance target part or the state of the multifunction device 200 is selectively displayed.

  Further, as shown in FIG. 8, after acquiring the component state information on the MFP 200 side via the communication unit 11 and the communication network 90, the acquisition result may be added to the AR annotation information and displayed. Good. Thereby, for example, the rotation state of the motor and the ON / OFF detection state of the sensor may be displayed.

  Moreover, you may comprise so that maintenance operation | movement can be implemented from the portable terminal 100 with respect to maintenance object components. Such a configuration can be realized by providing an operation interface for instructing execution of a maintenance operation for a maintenance target component or the like on the portable terminal 100 and the multifunction device 200 side. During the execution operation by the maintenance staff, it is possible to instruct the multifunction device 200 to execute the maintenance operation of the target component via the communication unit 11 and the communication network 90 line.

(Example of display processing on a portable terminal)
Next, an example of display processing in the portable terminal 100 will be described with reference to FIG. FIG. 9 is a flowchart illustrating an example of a procedure of display processing in the portable terminal 100 according to the first embodiment.

  As shown in FIG. 9, in step S101, the control unit 10 of the portable terminal 100 determines whether or not the maintenance person has pressed the maintenance button and instructed the start of the maintenance operation mode. If there is an instruction to start the maintenance operation mode (Yes), in step S102, the control unit 10 determines whether or not the maintenance mode is selected by the maintenance person. If the maintenance mode is selected (Yes), the control unit 10 instructs the imaging unit 12 to activate the camera 123 in step S103. In addition, the control unit 10 instructs the position information acquisition unit 13 and the orientation information acquisition unit 14, and the display controller 22 of the display unit 20 instructs the position detection unit 21 and the contact detection unit 24 to activate various sensors. Thereby, live view display of the image acquired from the camera 123 is started.

  In step S <b> 104, the control unit 10 analyzes the imaging data of the camera 123 to detect components included in the imaging data. In addition, the control unit 10 refers to the storage unit 30 and acquires AR annotation information corresponding to each detected component.

  In step S <b> 105, the control unit 10 refers to the storage unit 30 and acquires maintenance mode-specific AR annotation display information. Based on such information, the control unit 10 selects parts corresponding to the selected maintenance mode.

  In step S <b> 106, the control unit 10 instructs the display controller 22 to display the AR annotation information of the selected part on the screen providing unit 23 of the display unit 20. Thereby, the maintenance operation execution view in which the AR annotation information is superimposed on the live view is displayed.

  In step S <b> 107, the control unit 10 determines whether a maintenance operation instruction has been input by the maintenance staff. If there is no instruction (No), the process proceeds to step S109. If there is an instruction (Yes), in step S <b> 108, the control unit 10 instructs the communication unit 11 to transmit an instruction for the instructed operation to the multifunction device 200. Upon receiving the operation command, the multifunction device 200 operates each unit according to the operation command.

  In step S109, the control unit 10 determines whether or not the maintenance person has instructed the end of the maintenance operation mode. If there is no termination instruction (No), the process returns to step S104, and the display of the maintenance operation execution view is continued. If there is an end instruction (Yes), the display of the maintenance operation execution view is ended.

  Thus, the display process in the portable terminal 100 ends.

  For example, the terminal device of Patent Document 1 has a mechanism for displaying a plurality of graphics objects on the display unit as one collective graphics object. Then, an object with a greater distance from the current position of the terminal device is treated as a collective object, thereby suppressing display on the screen for objects that are far away.

  However, for example, in a multifunctional machine having multiple functions such as a printer and a copying machine, the internal structure of the apparatus is complicated, and a large number of parts are densely packed in a narrow area. For this reason, in the method of determining whether to link a plurality of graphics objects depending on the distance from the current position as in Patent Document 1, for example, the portable terminal 100 ′ of the comparative example of FIG. As described above, it is impossible to avoid displaying a large number of AR annotation information of parts irrelevant to the maintenance work to be performed during maintenance work.

  In the portable terminal 100 according to the first embodiment, based on the maintenance mode-specific AR annotation display information held in the storage unit 30, the maintenance operation is performed without displaying the AR annotation information of parts that are not related to the maintenance operation to be performed. It is possible to selectively display information on parts necessary for the operation. As a result, even in the case where parts are concentrated in a dense location of the multi-function device 200, it is possible to avoid a complicated display in which AR annotation information of all parts is displayed, and visibility of maintenance personnel It is possible to realize a maintenance operation execution view display that is superior to the above. Therefore, the maintenance staff can improve the maintenance work efficiency and reduce maintenance procedure errors based on the maintenance operation execution view that is visually understandable and workable.

(Modification 1)
Next, a portable terminal 100a according to Modification 1 of Embodiment 1 will be described with reference to FIG. The portable terminal 100a according to the first modification of the first embodiment is different from the first embodiment described above in that component information that is not a maintenance work target is displayed inconspicuously.

  As shown in FIG. 11, in the maintenance operation execution view on the portable terminal 100a according to the first modification of the first embodiment, the brightness of the display of the AR annotation information and the parts that are not targeted for the selected maintenance mode are displayed. At least one of the saturations is set to be lower than the display of the AR annotation information of the target part for the selected maintenance mode. Thereby, the AR annotation information of the non-target part is displayed lighter than the AR annotation information display of the target part. Alternatively, for example, the AR annotation information of the non-target part is displayed in monochrome with respect to the AR annotation information of the target part that is in color display. As a result, the display of the AR annotation information of the non-target component can be made inconspicuous. That is, it is possible to make the AR annotation information display of the target part more prominent.

(Modification 2)
Next, the portable terminal 100b according to the second modification of the first embodiment will be described with reference to FIG. The portable terminal 100b according to the second modification of the first embodiment is different from the first embodiment in that predetermined information is also displayed for components outside the maintenance operation execution view.

  As shown in FIG. 12, in the maintenance operation execution view on the portable terminal 100b according to the second modification of the first embodiment, the position where the target component for the selected maintenance mode is out of the field of view of the camera. Exists. In such a case, the portable terminal 100b according to the second modification of the first embodiment displays AR annotation information indicating that the part is outside the display image of the maintenance operation execution view.

  Further, as shown in FIG. 12, an arrow or the like may be used to indicate in which direction the part outside the display image of the maintenance operation execution view exists outside the display image. Such a configuration is realized, for example, by storing position information of each component in advance as component information in the storage unit of the portable terminal 100b. The control unit of the portable terminal 100b refers to the information stored in the storage unit and displays in which direction the corresponding part exists outside the display image of the maintenance operation execution view based on the actual maintenance operation execution view. Let At this time, the control unit may use the position information of the portable terminal 100b acquired by the position information acquisition unit, and the attitude and angle information of the portable terminal 100b acquired by the orientation sensor.

(Other variations)

  In the first embodiment described above, the image forming apparatus is the multifunction device 200, but the present invention is not limited to this. The image forming apparatus may be an apparatus having a single function among a printing apparatus, a copier, a multifunction machine, a scanner apparatus, a facsimile apparatus, and the like. In addition, the device that is the target of the portable terminals 100, 100a, and 100b of the first embodiment and the first and second modifications is not limited to the image forming apparatus. The target device may be a projector, a camera, an air conditioner, a refrigerator, a fluorescent lamp, a vending machine, various electronic devices such as a handheld terminal, and an information processing device such as a PC.

[Embodiment 2]
Next, Embodiment 2 will be described with reference to FIGS. In the second embodiment, a part type specifying method based on image analysis of captured data and a maintenance operation from a portable terminal will be described in more detail. In the following description, the configuration of the bad part of the portable terminal 150 of the second embodiment will be described with reference to FIGS. 3 and 4 of the first embodiment and the same reference numerals as those of the first embodiment will be used for convenience. This does not deny that the portable terminal 150 of the second embodiment has a different function from the portable terminal 100 of the first embodiment.

(Operation example of portable terminal)
First, an operation example of the portable terminal 150 according to the second embodiment will be described with reference to FIGS.

  FIG. 13 is a diagram illustrating imaging data captured from the imaging unit 12 in the maintenance operation mode on the portable terminal 150 according to the second embodiment. FIG. 13 shows, for example, imaging data of the rear surface of the multifunction device 200. Each unit of the multifunction device 200 such as the scanner unit 240, the main body unit 250, and the paper feed tray unit 260, different types of motors 281 to 283, and different types of devices are illustrated. Photo images of different fans 291 and 292 are displayed. Also shown is a paper transport path 270 superimposed on the photographic image. The paper transport path 270 is a line drawing that images the path from when the transfer to which the image is transferred is fed to when it is abolished.

  As shown in FIG. 13, also in the portable terminal 150 of the second embodiment, an object is extracted from an image input from the camera 123, and a component type such as a motor or a fan is determined from the contour or feature point of each object.

  In the portable terminal 150 according to the second embodiment, when a plurality of the same components are detected, the use of each component is determined from the positional relationship and AR annotation information is displayed. This is because, for example, the same type of motor may be used for both the paper feed unit and the paper discharge unit.

  In the portable terminal 150 according to the second embodiment, as the AR annotation information, an ON / OFF button for operating these components is displayed in addition to the usage of the components. The part is operated when the ON button is pressed, and stopped when the OFF button is pressed.

  In order to realize the above function, the portable terminal 150 uses the markerless AR technology.

  As shown in FIG. 13, in the markerless AR technique, for example, a certain position on the multi-function device 200 is set as the reference point SP. In the example of FIG. 13, a point on the boundary line between the scanner unit 240 and the main body unit 250 and on the left side when viewed from the back of the multi-function device 200 is used as the reference point SP. Although only one reference point SP is provided in FIG. 13, a plurality of reference points SP may be provided. Then, the component application is determined from the positional relationship between the reference point SP and the determined center position CP of each component. The storage unit 30 holds center coordinate data of each part based on the reference point SP.

  FIG. 14 is coordinate definition information for each component held by the storage unit 30 of the portable terminal 150 according to the second embodiment. The coordinate definition information for each component is included in the component information, for example. As shown in FIG. 14, in the coordinate definition information for each part, center coordinates based on the reference point SP are defined for each part. In the coordinate definition information for each part, a part name indicating the use for each part is defined. The coordinate definition information for each part defines an operation mode for each part such as a rotation direction and a speed.

  The control unit 10 of the portable terminal 150 calculates the positional relationship with the reference point SP for the component whose outline can be grasped in the input image from the camera 123, and collates with the coordinate definition information in the storage unit 30 to check each component. Determine usage. The control unit 10 displays the determined result of the part name or the like as AR annotation information. Further, an ON / OFF button is also displayed below the part name display.

  FIG. 15 is a diagram illustrating an example in which AR annotation information is superimposed on a live view on the portable terminal 150 according to the second embodiment. As shown in FIG. 15, on the portable terminal 150, parts in the input image from the camera 123 are determined based on the information in FIGS. 13 and 14, and AR annotation information is displayed. The maintenance staff can discriminate each component based on this information.

  In addition, when the maintenance staff presses an ON button displayed on the screen, for example, a command is sent from the portable terminal 150 to the multifunction device 200 via the communication network 90, and the target component is operated in the standard mode or the maintenance mode. The The standard mode means that, for example, in the case of a motor, the rotation direction and speed during normal operation are values according to the use of the component. There are several modes in the maintenance mode, such as the maintenance modes 1, 2, 3,... In the first embodiment described above, and the mode is set so that the mode can be changed according to the content of the maintenance work. .

  Incidentally, there is a case where the reference point SP is not included in the input image from the camera 123.

  FIG. 16 is a diagram for explaining a component discrimination method when the reference point SP is not included in the image on the portable terminal 150 according to the second embodiment. As shown in FIG. 16, when the reference point SP does not exist in the input image from the camera 123, the use of each component is determined based on the positional relationship between components recognized in the vicinity in the input image.

(Example of display processing on a portable terminal)
Next, an example of display processing in the portable terminal 150 will be described with reference to FIG. FIG. 17 is a flowchart illustrating an example of a display processing procedure in the portable terminal 150 according to the second embodiment.

  Steps S201 to S204 in FIG. 17 are the same flow as steps S101 to S104 in FIG. 9 in the first embodiment.

  In step S <b> 205, the control unit 10 of the portable terminal 150 refers to the storage unit 30 and acquires coordinate definition information for each part. In step S <b> 206, the control unit 10 determines whether or not the reference point SP is included in the input image from the camera 123.

  When the reference point SP is included (Yes), the use of each component is determined from the center coordinates of each component based on the reference point SP in step S207a.

  When the reference point SP is not included (No), the use of each component is determined based on the mutual positional relationship between the components in step S207b.

  In step S208, the control unit 10 causes the display controller 22 of the display unit 20 to display AR annotation information for each part.

  In step S209, the control unit 10 determines whether or not the maintenance person has instructed the end of the maintenance operation mode. If there is no end instruction (No), the process returns to step S204. If there is an end instruction (Yes), the display of the maintenance operation execution view is ended.

  Thus, the display process in the portable terminal 150 ends.

  The portable terminal 150 of the second embodiment also has the same effect as the portable terminal 100 of the first embodiment.

  In addition, since the portable terminal 150 according to the second embodiment can discriminate parts with different uses even if the type is the same, a maintenance operation execution view display with even better visibility for maintenance personnel is realized. it can.

  In the portable terminal 150 according to the second embodiment, since each component can be operated from the portable terminal 150, for example, output check and input check of a sensor, a motor, and the like are performed on the portable terminal 150. Can be done easily.

(Other variations)

  In the second embodiment described above, the portable terminal 150 independent from the multifunction device 200 is used. However, the present invention is not limited to this. For example, the panel device 228 (see FIG. 2) included in the multi-function device 200 may be configured to be removable and configured to be usable as a portable terminal.

  Each of the above-described first and second embodiments and modifications can be arbitrarily combined.

DESCRIPTION OF SYMBOLS 1 Display system 10 Control part 11 Communication part 12 Imaging part 13 Position information acquisition part 14 Direction information acquisition part 15 Key input part 20 Display part 21 Position detection part 22 Display controller 23 Screen provision part 24 Contact detection part 30 Storage part 100 Portable type Terminal 110 Controller 111 CPU
112 ROM
113 RAM
114 EEPROM
115 Communication I / F
116 I / O I / F
121 Display Device 122 Input Device 123 Camera 124 GPS Module 125 Direction Sensor 200 Multifunction Device

Japanese Patent No. 5315111

Claims (9)

An imaging unit for acquiring an image of real space;
A display unit for displaying the image;
A storage unit that stores part information related to parts of a maintenance target device, augmented reality information associated with the part information, and a display mode of the augmented reality information for each part according to the content of the maintenance;
A control unit that controls operations of the imaging unit and the display unit, and causes the augmented reality information to be displayed superimposed on the image.
The controller is
The image of the target device acquired by the imaging unit is displayed on the display unit, the component in the image is determined based on the component information acquired from the storage unit, and the extension acquired from the storage unit Among the real information, the display mode of the part that is the maintenance target and the part that is not the maintenance target is changed.
Display device. The controller is
Displaying the augmented reality information of the component that is the subject of maintenance to be more prominent than the augmented reality information of the component that is not the subject of maintenance;
The display device according to claim 1. The controller is
At least one of the brightness and saturation of the display of the augmented reality information of the part not subject to maintenance is lower than the brightness and saturation of the display of the augmented reality information of the part subject to maintenance To
The display device according to claim 1. The controller is
Non-displaying the augmented reality information of the parts not subject to maintenance,
The display device according to claim 1. The component information stored in the storage unit includes position information for each component,
The controller is
When the part to be maintained is not included in the image, the part not included in the image is outside the range of the image based on the position information for each part acquired from the storage unit. Display,
The display device according to any one of claims 1 to 4. The component information stored in the storage unit includes usage information for each component,
The controller is
Based on the usage information for each part acquired from the storage unit, the augmented reality information indicating the usage of the part to be maintained is displayed.
The display device according to any one of claims 1 to 5. The component information stored in the storage unit includes position information indicating a positional relationship between the components,
The controller is
Determining the component in the image from position information indicating the positional relationship between the components acquired from the storage unit, and specifying the use of the determined component;
The display device according to claim 6. The component information stored in the storage unit includes position information with respect to a reference point for each component,
The controller is
Determining the component in the image from the positional relationship between the reference point and the component acquired from the storage unit, and specifying the use of the determined component;
The display device according to claim 6 or 7. A display method executed by a display device including a display unit,
An imaging step of acquiring an image of a real space of the maintenance target device;
An image display step of displaying the image of the target device on the display unit;
A determination step of determining the component in the image based on component information regarding the component of the target device;
Of the augmented reality information associated with the component information, the display mode of the component that is the maintenance target and the component that is not the maintenance target are changed, and the augmented reality information is superimposed on the image, An augmented reality information display step to be displayed on the display unit,
Display method.