JP2022000795A - Information management device - Google Patents

Abstract

To make it possible to view related information pertaining to an object by simply photographing the object.SOLUTION: In a related information database of a management server 20, a prior registered registration image and related information corresponding to the registration image are associated and stored. A photographing section 31 photographs an image of a real space around an HMD. If any of a plurality of registration images stored in the related information database of the management server 20 is included in the image photographed by the photographing section 31 of the HMD 10, a display section 38 displays the related information corresponding to the registration image in a manner to be present around the registration image.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information management device.

Patent Document 1 discloses an information processing apparatus that superimposes an additional image, which is a virtual object, on a real space image and presents it to a user at a position corresponding to a viewpoint in real space.

Japanese Unexamined Patent Publication No. 2015-228050

An object of the present invention is to provide an information management device capable of viewing related information about an object only by imaging the object with an information terminal.

The information management device according to the first aspect of the present invention includes a storage means for storing pre-registered registered images in association with registrant information and related information.
A receiving means for receiving an image transmitted by an information terminal, and
When the image received by the receiving means is stored as a registered image in the storage means, the related information associated with the registered image and registered by a predetermined registrant is referred to as the related information. It has a transmission means for transmitting to an information terminal.

The information management device of the second aspect of the present invention is the information management device of the first aspect, in which the receiving means further receives information regarding the designation of the registrant.
The predetermined registrant is determined by the information regarding the designation of the registrant received by the receiving means.

The information management device of the third aspect of the present invention further includes an extraction means for extracting an image of an object from an image received by the receiving means in the information management device of the first aspect.
When the image extracted by the extraction means is stored as a registered image in the storage means, the transmission means is related information associated with the registered image and is registered by a predetermined registrant. The related information is transmitted to the information terminal.

The information management device of the fourth aspect of the present invention is the information management device of the second aspect, when the receiving means does not receive the information regarding the designation of the registrant.
The transmission means transmits the related information registered by all the registrants to the information terminal.

The information management device according to the fifth aspect of the present invention is an image area to be registered as related information in an image captured by the information terminal in the information management device according to any one of the first to fourth aspects. The reception means that accepts the designation of
Further, it is provided with a registration means for registering an image of an image area received by the reception means in the storage means as the related information.

According to the information management device of the first aspect of the present invention, it is possible to see related information about an object only by imaging the object with an information terminal.
Further, according to the information management device of the first aspect of the present invention, it is possible to see the related information registered by the predetermined registrant on the information terminal.

According to the information management device of the second aspect of the present invention, the related information registered by the designated registrant can be viewed on the information terminal.

According to the information management device of the third aspect of the present invention, when the image of the object extracted from the image received from the information terminal is registered as the registered image, the related information associated with the registered image is informationed. It becomes possible to send to the terminal.

According to the information management device of the fourth aspect of the present invention, it is possible to display related information on the information terminal even if the registrant is not specified.

According to the information management device of the fifth aspect of the present invention, it is possible to register an image of a part of an image region in an image captured by an information terminal as related information.

It is a figure which shows the system structure of the related information display system of one Embodiment of this invention. It is a figure which shows the hardware composition of the HMD10 shown in FIG. It is a block diagram which shows the functional structure of HMD10 in one Embodiment of this invention. It is a block diagram which shows the functional structure of the management server 20 in one Embodiment of this invention. It is a figure which shows the appearance of HMD10 in one Embodiment of this invention. It is a figure for demonstrating the state when the HMD10 is attached. It is a flowchart for demonstrating the operation at the time of saving the related information to be displayed in management server 20. It is a figure which shows an example of the cover image of a book. It is a figure which shows the state of specifying the area which wants to save as related information. It is a figure which shows an example of the related information database 43 in which a registered image and related information are stored. It is a flowchart for demonstrating the operation in the case of displaying the related information with respect to the captured image. It is a figure for demonstrating the state which a book is placed on a table. It is a figure which shows an example of the field of view when a user sees a book placed on a table through a display part 38. It is a figure which shows the state in the case where the related information is displayed by taking an image of a book by the camera of the mobile terminal apparatus 70. It is a figure for demonstrating the state when the related image is displayed around the book of the bookshelf. It is a figure for demonstrating the state when the cautionary note information is displayed around the format of a part drawing. It is a figure for demonstrating the state when the tourist information of the building is displayed around the building in the city.

Next, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing a system configuration of a related information display system according to an embodiment of the present invention.

In recent years, using a head-mounted display (hereinafter abbreviated as HMD (Head Mounted Display)), VR (Virtual Reality) that gives the user the appearance of being in a virtual space has been realized in various devices. ing.

However, with this VR technology, the visual information in the real space is blocked for the person wearing the HMD. Therefore, there is a technology called AR (Augmented Reality), which is a technology to superimpose an artificially generated image on a real space image, and a technology called AR (Augmented Reality), which mixes real space and virtual space to create real objects and virtual ones. A technology called MR (Mixed Reality) that builds a new space where things influence each other in real time has been proposed.

Here, in AR technology, an artificially generated image is superimposed and displayed on an actual image captured by an image pickup device such as a camera, whereas in MR technology, a user wearing an HMD is a transmissive type. The difference is that the state of the real space can be directly viewed in real time through the display of.

Then, in the related information display system of the present embodiment, by using such MR technology, in the real space visually recognized by the user in real time, the object is placed around a specific object registered in advance. It is configured to add and display related information.

As shown in FIG. 1, the related information display system of the present embodiment includes a plurality of head-mounted displays (hereinafter, abbreviated as HMD (Head Mounted Display)) 10 for each user to wear on the head. It is composed of a management server 20 that manages a registered image registered in advance and related information related to the registered image in association with each other, and a wireless LAN terminal 30.

The HMD (head-mounted display device) 10 is used by being mounted on the user's head and has a transmissive display unit capable of visually recognizing the real space, and the user can use the transmissive display unit externally. It is possible to visually recognize the state of. Then, in the HMD10, by displaying the related information on the transparent display unit, the user can obtain the visual sense as if the related information about the object exists around the object in the real space. It is structured so that it can be done.

Each HMD 10 is connected to the management server 20 by transmitting / receiving data to / from the wireless LAN terminal 30 via a wireless communication line such as Bluetooth (registered trademark) or Wi-Fi.

Next, the hardware configuration of the HMD 10 shown in FIG. 1 is shown in FIG.

As shown in FIG. 2, the HMD 10 is a communication interface that transmits and receives data to and from an external device such as a management server 20 via a CPU 11, a memory 12, a storage device 13 such as a flash memory, and a wireless communication line. (IF) 14, a camera 15 for capturing an external image, and a display device 16 for displaying related information. These components are connected to each other via a control bus 17.

The CPU 11 executes a predetermined process based on the control program stored in the memory 12 or the storage device 13 to control the operation of the HMD 10.

FIG. 3 is a block diagram showing a functional configuration of the HMD 10 realized by executing the above control program.

As shown in FIG. 3, the HMD 10 of the present embodiment includes an image pickup unit 31, an object recognition unit 32, a gesture recognition unit 33, an image extraction unit 34, a communication unit 35, and a related information acquisition unit 36. It is composed of a virtual object drawing control unit 37 and a display unit 38.

The image pickup unit 31 captures an image of the real space around the own device.

The gesture recognition unit 33 recognizes the position and operation of the fingertip of the user wearing the own device from the image captured by the image pickup unit 31.

When recognizing the position of the fingertip of the user, the gesture recognition unit 33 identifies the position according to a plurality of instruction operations of the user. For example, when the user specifies a rectangular area, the positions of two corners on the diagonal may be specified, or the rectangular area may be specified by dragging along the diagonal from one corner. You may do it.

The object recognition unit 32 recognizes an image to be registered as a registered image and an image to be displayed with related information from the image captured by the image pickup unit 31. For example, when an image of a book is included in the captured image and an attempt is made to display related information about the book, the object recognition unit 32 uses the image captured by the image pickup unit 31 to obtain a book. Recognize the image of.

When the object recognition unit 32 recognizes the registered image, it is possible to use a method in which model learning is performed in advance using various learning data to create an discriminative model. For example, a discriminative model for identifying an image of a book is generated by performing model learning using images of various books.

Then, the object recognition unit 32 recognizes an image to be extracted as a registered image by identifying the image of the book from the image captured by the image pickup unit 31 using a pre-generated identification model.

The object recognition unit 32 may perform image processing such as keystone correction and distortion correction on the image captured by the image pickup unit 31 and then recognize the registered image.

The image extraction unit 34 extracts an image recognized as a registered image by the object recognition unit 32 or an image to be displayed as related information from the image captured by the image pickup unit 31, or is recognized by the gesture recognition unit 33. Extract the image to be registered as the related information specified by the user's gesture.

The image extraction unit 34 may perform image processing such as keystone correction and distortion correction on the related image extracted from the captured image.

By communicating with the management server 20, the communication unit 34 transmits the image extracted by the image extraction unit 34 to the management server 20 and receives the related information transmitted from the management server 35.

The related information acquisition unit 36 acquires the related information transmitted from the management server 20 via the communication unit 35.

The virtual object drawing control unit 37 performs drawing processing based on the related information acquired by the related information acquisition unit 36, and is displayed on the display unit 38 so that the related image exists around the registered image to be displayed. Performs various drawing processes.

The display unit 38 is a transparent display unit that can visually recognize the real space, and displays the related image drawn by the virtual object drawing control unit 37 by using, for example, a holography technique.

Further, the functional configuration of the management server 20 is shown in FIG. As shown in FIG. 4, the management server 20 includes a communication unit 41, a related information management unit 42, and a related information database 43.

The related information database 43 is a storage means for storing the registered image registered in advance and the related information corresponding to the registered image in association with each other.

Communicates between the communication unit 41 and the HMD 10, receives the registered image and related information about the registered image, receives the registered image extracted from the captured image, and displays it around the registered image. Processing to send related information to be done.

The related information management unit 42 and the related information stored in the related information database 43 are managed, and when the registered image and the related information related to the registered image are transmitted via the communication unit 41, this registered image. And related information are associated with each other and stored in the related information database 43.

That is, it functions as a receiving means for receiving the designation of the image area to be registered as related information in the image captured by the gesture recognition unit 33 and the imaging unit 31. Then, the related information management unit 42 functions as a registration means for registering the image of the image area received by the gesture recognition unit 33 in the related information database 43 as related information.

Further, when the related information management unit 42 transmits an image to be displayed as related information such as an image of a book via the communication unit 41, the related information management unit 42 stores the image in the related information database 43. It is determined whether or not it is similar to any of the registered images of. Then, when a similar registered image exists, the related information management unit 42 reads the related information registered corresponding to the registered image from the related information database 43, and sets the HMD 10 via the communication unit 41. Send.

In the related information display system of the present embodiment, due to such a configuration, a plurality of registered images stored in the related information database 43 of the management server 20 are included in the images captured by the image pickup unit 31 of the HMD 10. When any one of the registered images is included, the display unit 38 displays the related information corresponding to the registered image so as to exist around the registered image.

In the present embodiment, since the display unit 38 is a transparent display unit that can visually recognize the real space, the virtual object drawing control unit 37 draws an image of related information corresponding to the registered image and actually draws the image. Control is performed to display on the display unit 38 so that it exists in the space.

Further, in the case where the display unit 38 displays the image captured by the image pickup unit 31 as it is, the display unit 38 adds related information corresponding to the registered image to the image captured by the image pickup unit 31. And display.

When the related information display system of the present embodiment is used by a plurality of users, the related information database 43 may store the information of the registrant who has registered the related information together with the related information. good.

In such a configuration, the display unit 38 displays only the related information registered by the registrant who satisfies the preset condition so as to exist around the registered image. For example, when the user makes a setting so that only the related information registered by the user is displayed, the control is such that only the related information registered by the user of the HMD 10 is displayed on the display unit 38. Will be done.

Furthermore, if the user makes a setting so that only the related information registered by the user designated by himself / herself is displayed, only the related information registered by the user of HMD10 and the designated user is displayed. The control as displayed on the unit 38 is performed.

Further, when the registrant is not specified, the display unit 38 controls to display all the related information registered for the registered image.

Next, the appearance of the HMD 10 described above is shown in FIG. As shown in FIG. 5, the HMD 10 includes a transparent display unit 38 that allows the wearing user to visually recognize the external appearance. Then, a part of the HMD 10 is provided with an image pickup unit 31 for capturing an image of the outside state, and captures an image of the outside state that the user is visually recognizing through the transmissive display unit 38. It is structured so that it can be done.

FIG. 6 shows a state when such an HMD 10 is attached to the head. Referring to FIG. 6, the user wearing the HMD 10 has his / her eyes covered with the display unit 38, but since the display unit 38 is a transmissive type, he / she can visually recognize the external appearance. I understand that.

Next, the operation of the related information display system of the present embodiment will be described in detail with reference to the drawings.

First, the operation when the related information to be displayed is saved in the management server 20 will be described with reference to the flowchart of FIG. 7. In the following description, the case where the object for which the related information is to be displayed is a book will be described.

First, the region including the book to be registered is imaged by the image pickup unit 31 of the HMD 10. Then, the object recognition unit 32 recognizes the book from the image captured by the image pickup unit 31, and the image extraction unit 34 extracts the cover image of the book as shown in FIG. 8 (step S101).

Next, when the user opens an important page to be registered as related information of this book and specifies an area to be saved as related information, for example, as shown in FIG. 9, the gesture recognition unit 33 recognizes the specified area. (Step S102).

Then, the image of the designated area is extracted by the image extraction unit 34, transmitted to the management server 20 together with the cover image, and stored in association with the related information database 43 (step S103).

FIG. 10 shows an example of the related information database 43 in which the registered image and the related information are stored in this way.

With reference to FIG. 10, it can be seen that the cover image of the book is registered as a registered image, and the related image to be displayed with respect to the registered image is stored in association with the related information. In addition, the information of the registrant who made the registration is also stored in each related information.

In the present embodiment, the case where the image related to the registered image is registered as the related image will be described as the related information, but it is also possible to register information other than the image such as character information as the related information. be.

Next, the operation in the case of displaying the related information with respect to the captured image in the state where the registered image and the related information are stored in this way will be described with reference to the flowchart of FIG.

Here, a case where a user wearing the HMD 10 sees a state in which a book is placed on a table as shown in FIG. 12 will be described.

First, when the object recognition unit 32 recognizes the cover image of the book in the image captured by the image pickup unit 31, the image extraction unit 34 extracts the cover image from the captured image (step S201).

For example, when the image captured by the image pickup unit 31 is an image in which a book is placed on a table as shown in FIG. 12, the image extraction unit 34 extracts the cover image of the book.

Then, the extracted cover image is transmitted to the related information management unit 42 of the management server 20 via the communication unit 35 of the HMD 10 and the communication unit 41 of the management server 20. Then, the related information management unit 42 determines whether or not a registered image similar to the received cover image is registered in the related information database 43 (step S202).

When a registered image similar to the extracted cover image is registered in the related information database 43 (yes in step S202), the related information management unit 42 has a condition set in advance by the registrant of the registered image. It is determined whether or not the condition is satisfied (step S203).

Here, it is assumed that the condition of the registrant to display the related information is satisfied. Therefore, the related information management unit 42 acquires the related information stored corresponding to the registered image similar to the extracted cover image from the related information database 43 (step S204).

Then, the related information acquired by the related information management unit 42 is transmitted to the related information acquisition unit 36 via the communication unit 41 of the management server 20 and the communication unit 35 of the HMD 10. Then, the virtual object drawing control unit 37 draws a related image to be displayed on the display unit 38 based on the related information received by the related information acquisition unit 36, whereby the display unit 38 draws a related image around the cover image. The related image is displayed as if it exists (step S205).

FIG. 13 shows an example of the field of view when the user sees through the display unit 38 as a result of performing such processing.

In FIG. 13, it is shown that the related images 61 and 62 are displayed so as to exist around the book placed on the table.

That is, the user wearing the HMD 10 simply looks at the book placed on the table, and the related images 61 and 62 are displayed in the surroundings in the user's field of view. By registering the important information of this book in advance as related information, the contents of the important information can be viewed without opening the book.

In the present embodiment, the case where the HMD 10 worn by the user on the head is used has been described, but for example, as shown in FIG. 14, related information is added by capturing a book with the camera of the mobile terminal device 70. It is also possible to display the displayed image. In the case of such a configuration, unlike the above-mentioned HMD10, the related image is added to the captured image and displayed on the display.

Further, according to the present embodiment, by registering the spine cover image as the registered image instead of the cover image of the book, it is necessary to take the trouble of taking out the book from the bookshelf and opening the contents of the book displayed on the bookshelf. It is also possible to know the contents of the book without it.

Specifically, by registering the spine covers of a plurality of books as registered images and registering the images of important pages in the registered images as related information, as shown in FIG. 15, the books on the bookshelf It is possible to grasp the important page contents of each book just by looking at.

In this embodiment, the case where the target for displaying the related information is a book has been described, but the target for displaying the related information is not limited to the book, and the building, documents in a specific format, etc. It is also possible to display the related information corresponding to the registered image as the object.

For example, as shown in FIG. 16, by registering the format of the parts drawing as a registered image and making the cautionary note at the time of design correspond as related information, the cautionary note information can be obtained only by looking at the format of the parts drawing. It is also possible to display in the surroundings.

Further, for example, as shown in FIG. 17, by registering an image of a building of a tourist attraction as a registered image and associating the tourist information of the building with related information, this building can be displayed in the city. It is also possible to display tourist information of the building in the surroundings just by looking at it.

Further, in the present embodiment, the description has been made using a configuration in which the related information database 43 for storing the related information in association with the registered image is provided in the management server 20. However, a storage means corresponding to the related information database 43 may be provided in the HMD 10. With such a configuration, the HMD 10 can display related images without having to exchange data with the management server 20.

[Modification example]
In the above embodiment, the case where the present invention is applied to an HMD worn by a user on the head and used has been described, but the present invention is not limited to this, and the present invention is not limited to this, and smartphones, tablet terminals, etc. The present invention can be similarly applied to any information processing device provided with an image pickup device and a display device.

10 HMD (Head Mounted Display: Head-mounted display device)
11 CPU
12 Memory 13 Storage device 14 Communication interface (IF)
15 Camera 16 Display device 17 Control bus 20 Management server 30 Wireless LAN terminal 31 Imaging unit 32 Object recognition unit 33 Gesture recognition unit 34 Image extraction unit 35 Communication unit 36 Related information acquisition unit 37 Virtual object drawing control unit 38 Display unit 41 Communication unit 42 Related Information Management Department 43 Related Information Database 61, 62 Related Images 70 Mobile Terminal Device

Claims (5)

A storage means for storing pre-registered registered images in association with registrant information and related information, and
A receiving means for receiving an image transmitted by an information terminal, and
When the image received by the receiving means is stored as a registered image in the storage means, the related information associated with the registered image and registered by a predetermined registrant is referred to as the related information. The means of transmission to the information terminal and
Information management device with. The receiving means further receives information regarding the designation of the registrant, and the receiving means further receives information.
The information management device according to claim 1, wherein the predetermined registrant is defined by the information regarding the designation of the registrant received by the receiving means. The receiving means further has an extraction means for extracting an image of an object from the received image.
When the image extracted by the extraction means is stored as a registered image in the storage means, the transmission means is related information associated with the registered image and is registered by a predetermined registrant. The information management device according to claim 1, wherein the related information is transmitted to the information terminal. When the receiving means does not receive the information regarding the designation of the registrant,
The information management device according to claim 2, wherein the transmission means transmits related information registered by all registrants to the information terminal. A reception means that accepts designation of an image area to be registered as related information in an image captured by the information terminal.
The information management device according to any one of claims 1 to 4, further comprising a registration means for registering an image of an image area received by the reception means in the storage means as the related information.

Images