JP2012034216A - Image providing system, image display device, and image display program - Google Patents

Abstract

An image providing system, an image display device, and an image display program that enable a user to easily grasp the reliability of image data obtained by photographing each point on a map image are provided.
In an image providing system 1, image data acquired by a fixed point camera 10 or a user terminal 5 is stored in an image management database together with position information and a time stamp at the time of acquisition. In the image management server 2, the map data stored in the map data storage area is returned to the user terminal 5 in response to a request from the user terminal 5. At this time, the time stamp associated with the position information included in the display range of the map data is also returned to the user terminal 5. In the user terminal 5, a heat map is generated based on the map data and time stamp acquired from the image management server 2 and displayed on the HMD.
[Selection] Figure 2

Description

  The present invention relates to an image providing system that provides images acquired in various places, an image display device that displays images acquired in various places, and an image display program.

  2. Description of the Related Art Conventionally, a system that can obtain images from various locations is known (for example, see Patent Document 1). Specifically, in the image display device, when a location is specified and a current video acquisition instruction is input, imaging data of the current video is acquired from an external video acquisition device existing at the specified location. In the image display device, when a location is specified and an instruction to obtain a past video is input, imaging data of the past video is acquired from the imaging data storage unit or the center station. Thereby, the image display apparatus can display the present image or the past image of the designated place.

JP 2003-299156 A

  In the above system, when a current video or a past video at a specified location is displayed, the information reliability is higher as the video is newer, while the information reliability is lower as the video is older. That is, the acquisition date and time of video data greatly affects the identity (hereinafter referred to as video reliability) with the cityscape, landscape, state, etc. at the current designated location. However, it is difficult for the user to grasp whether the video at the designated location is new or old, that is, the reliability of the video.

  The present invention has been made to solve the above-described problems, and provides an image providing system, an image display device, and an image that allow a user to easily grasp the reliability of image data obtained by photographing each point on a map image. An object is to provide a display program.

  In the image providing system according to the first aspect of the present invention, an image acquisition device that acquires image data, a server that manages the image data, and an image display device that displays the image data are connected via a network. In the image providing system, the image acquisition device includes: an image data acquisition unit that acquires the image data from an imaging unit that images the periphery of the image acquisition device; and a position that detects a current position of the image acquisition device. Position information acquisition means for acquiring position information indicating the acquisition position of the image data from detection means, reliability information generation means for generating reliability information indicating temporal reliability of the image data, and the image data The image data acquired by the acquisition means, the position information acquired by the position information acquisition means, and generated by the reliability information generation means. And the information transmission means for transmitting the reliability information to the server, wherein the server associates the image data received from the image acquisition device, the position information, and the reliability information, Corresponding to storage control means to be stored in the first storage device, map data stored in the second storage device in response to a request from the image display device, and the position information included in the display range of the map data Information transmission means for returning the reliability information attached to the image display device, and the image display device transmits the request to the server, whereby the map data is transmitted from the image display device. And information acquisition means for acquiring the reliability information, and the time of the image data based on the map data and the reliability information acquired by the information acquisition means Composite image generation means for generating a composite image in which a special display indicating reliability is added to the map data, and composite image display means for displaying the composite image generated by the composite image generation means on a first display device And.

  According to the first aspect, the image data acquired by the image acquisition device is stored in the first storage device together with the position information and the reliability information. In the server, the map data stored in the second storage device is returned to the image display device in response to a request from the image display device. At this time, the reliability information associated with the position information included in the display range of the map data is also returned to the image display device. In the image display device, a composite image is generated based on the map data and reliability information acquired from the server and displayed on the first display device. The composite image is an image in which a special display indicating temporal reliability of the image data is added to the map data. Therefore, the user of the image display device can easily grasp the reliability of the image data obtained by photographing each point on the map image by referring to the composite image displayed on the first display device.

  In the image providing system according to the second aspect of the present invention, an image acquisition device that acquires image data, a server that manages the image data, and an image display device that displays the image data are connected via a network. In the image providing system, the image acquisition device includes: an image data acquisition unit that acquires the image data from an imaging unit that images the periphery of the image acquisition device; and a position that detects a current position of the image acquisition device. Position information acquisition means for acquiring position information indicating the acquisition position of the image data from detection means, reliability information generation means for generating reliability information indicating temporal reliability of the image data, and the image data The image data acquired by the acquisition means, the position information acquired by the position information acquisition means, and generated by the reliability information generation means. And the information transmission means for transmitting the reliability information to the server, wherein the server associates the image data received from the image acquisition device, the position information, and the reliability information, Corresponding to storage control means to be stored in the first storage device, map data stored in the second storage device in response to a request from the image display device, and the position information included in the display range of the map data A composite image generating means for generating a composite image in which a special display indicating the temporal reliability of the image data is added to the map data based on the reliability information attached; and the composite image generating means Information transmission means for returning the generated composite image to the image display device, and the image display device transmits the request to the server, whereby the image table is displayed. An information acquisition means for acquiring the combined image from the device, the composite image obtained by the information obtaining unit, and a composite image display means for displaying on the first display device.

  According to the second aspect, the image data acquired by the image acquisition device is stored in the first storage device together with the position information and the reliability information. In response to a request from the image display device, the server synthesizes the map data stored in the second storage device and the reliability information associated with the position information included in the display range of the map data. An image is generated. The composite image is an image in which a special display indicating temporal reliability of the image data is added to the map data. The composite image generated by the server is returned to the image display device and displayed on the first display device. Therefore, the user of the image display device can easily grasp the reliability of the image data obtained by photographing each point on the map image by referring to the composite image displayed on the first display device.

  In any one of the first aspect and the second aspect, the reliability information is a time stamp indicating a date and time when the image data is acquired, and the special display indicates the image data whose acquisition date and time indicated by the time stamp is new. And the display mode which can visually distinguish the image data with the old acquisition date and time indicated by the time stamp on the map data. In this case, the user of the image display device can visually discriminate between a point where the acquisition date / time of the image data is new and a point where the acquisition date / time of the image data is old with reference to the composite image.

  In any one of the first aspect and the second aspect, the reliability information is update frequency data indicating a frequency at which the image data is updated, and the special display has a high update frequency indicated by the update frequency data. The display mode may be such that the image data and the image data having a high update frequency indicated by the update frequency data can be visually distinguished on the map data. In this case, the user of the image display device can visually determine a point where the update frequency of the image data is high and a point where the update frequency of the image data is low with reference to the composite image.

  In any one of the first aspect and the second aspect, the image display device includes: a position specifying unit for a user to specify a target position in the composite image displayed on the first display device; and the position specifying unit. Target data acquisition means for acquiring the target data that is the image data corresponding to the target position specified by the first storage device, and the target data acquired by the target data acquisition means You may provide the object data display means displayed on an apparatus. In this case, the user of the image display device can view the image data on the second display device after confirming the temporal reliability of the image data at each point on the composite image.

  In any one of the first aspect and the second aspect, the image display device warns the second display device when the temporal reliability indicated by the reliability information corresponding to the target data is lower than a predetermined value. A warning display means for displaying may be provided. In this case, the user of the image display device can surely grasp that the temporal reliability of the image data at the designated point is low.

  In any one of the first aspect and the second aspect, when the target data acquisition unit acquires a plurality of the target data, the image display device converts the plurality of target data into a list form or a thumbnail form. A target list display unit for displaying the target data selected by a user among the plurality of target data displayed by the target list display unit; It may be displayed on the device. In this case, when there are a plurality of image data at the designated point, the user of the image display device can freely select the image data to be displayed on the second display device.

  An image display device according to a third aspect of the present invention refers to a storage device for storing map data, map data acquisition means for acquiring the map data, image data, and position information indicating the acquisition position of the image data And the storage device that stores the reliability information indicating the temporal reliability of the image data in association with each other, and the reliability associated with the position information included in the display range of the map data Based on the reliability information acquisition means for acquiring information, the map data acquired by the map data acquisition means, and the reliability information acquired by the reliability information acquisition means, the time of the image data A composite image generating means for generating a composite image in which a special display indicating reliability is added to the map data; and the composite image generated by the composite image generating means And a composite image display means for displaying on the display device.

  According to the third aspect, the map data and the reliability information of the image data associated with the position information included in the display range of the map data are acquired from the storage device. A composite image is generated based on the acquired map data and reliability information and displayed on the display device. The composite image is an image in which a special display indicating temporal reliability of the image data is added to the map data. Therefore, the user can easily grasp the reliability of the image data obtained by photographing each point on the map image by referring to the composite image displayed on the display device.

  An image display program according to a fourth aspect of the present invention refers to a storage device that stores map data in a computer, a map data acquisition step for acquiring the map data, image data, and an acquisition position of the image data. The storage device stores the positional information to be displayed and the reliability information indicating the temporal reliability of the image data in association with each other, and is associated with the positional information included in the display range of the map data. The image data based on the reliability information acquisition step for acquiring the reliability information, the map data acquired by the map data acquisition step, and the reliability information acquired by the reliability information acquisition step. A composite image generating step for generating a composite image in which a special display indicating temporal reliability of the map is added to the map data; and the composite image The combined image generated by the generating step, characterized in that to perform the composite image display step of displaying on the display device.

  According to the fourth aspect, the map data and the reliability information of the image data associated with the position information included in the display range of the map data are acquired from the storage device. A composite image is generated based on the acquired map data and reliability information and displayed on the display device. The composite image is an image in which a special display indicating temporal reliability of the image data is added to the map data. Therefore, the user can easily grasp the reliability of the image data obtained by photographing each point on the map image by referring to the composite image displayed on the display device.

It is a figure which shows one specific example of the street corner in which the fixed point camera 10 was provided. 1 is an overall configuration diagram of an image providing system 1. FIG. 3 is a block diagram showing an electrical configuration of the image management server 2. FIG. 3 is a diagram illustrating a data configuration of an image management database 70. FIG. 3 is a diagram showing a data configuration of a position management database 80. FIG. It is the perspective view seen from the slanting upper part of user terminal 5. It is a block diagram which shows the electric constitution of HMD200. 2 is a block diagram showing an electrical configuration of an HMD control terminal 100. FIG. 4 is a flowchart showing upload processing of a fixed point camera 10. It is a flowchart which shows the upload process of the user terminal 5. 4 is a flowchart showing DB update processing of the image management server 2. It is a flowchart which shows DB update processing of the position management server 3. FIG. 4 is a flowchart showing image providing processing of the image management server 2. 5 is a flowchart showing image display processing of the user terminal 5. It is a figure which shows one specific example of the map data. It is a figure which shows one specific example of the heat map. It is a flowchart which shows a data display process.

  Embodiments of the present invention will be described with reference to the drawings. The drawings to be referred to are used to explain technical features that can be adopted by the present invention, and are merely illustrative examples.

  The overall configuration of the image providing system 1 according to the present embodiment will be described with reference to FIGS. 1 and 2. The image providing system 1 is a system that provides image data taken at a point designated on a map image (hereinafter referred to as a designated point) to a user via a network. In the image providing system 1, an image management server 2, a position management server 3, a plurality of user terminals 5, and a plurality of fixed point cameras 10 are connected via a network 4.

  The image management server 2 is a computer that manages image data taken in various places, and is connected to the network 4 by wire. The location management server 3 is a computer that manages the current location of the user terminal 5, and is connected to the network 4 by wire. The network 4 is the Internet that can be connected via a public network.

  The user terminal 5 is a small and lightweight terminal device carried by a user who uses the image providing system 1, and is connected to the network 4 wirelessly. The user terminal 5 acquires the image data of the point designated by the user on the map image from the image management server 2 or the fixed point camera 10 and displays it. In addition, the user terminal 5 captures the current position at a predetermined time interval and transmits the captured image data to the image management server 2. For this reason, the image data captured by the user terminal 5 displays a cityscape, a landscape, and the like in the vicinity of the current position of the user terminal 5.

  The fixed point camera 10 is a camera installed in various places, and is connected to the network 4 by wire. The fixed point camera 10 transmits the captured image data to the image management server 2 and the user terminal 5. The fixed point camera 10 of the present embodiment is provided at main intersections at street corners, and always shoots the four directions of east, west, south, and north of the intersection. Therefore, the image data photographed by the fixed point camera 10 shows the road traffic situation near the intersection where the fixed point camera 10 is set, the flow of pedestrians, and the like.

  The electrical configuration of the image management server 2 will be described with reference to FIG. The image management server 2 is a general-purpose server, and includes a CPU 20, a ROM 21, a RAM 22, an HDD 23, a communication device 24, and an I / O interface 29. The ROM 21, RAM 22, and I / O interface 29 are each connected to the CPU 20. The HDD 23 and the communication device 24 are connected to the I / O interface 29. The communication device 24 is a controller that controls data communication via the network 4.

  The HDD 33 is a large-capacity hard disk drive, and is provided with a program storage area 231, a map data storage area 232, an image data storage area 233, and the like. Various programs for operating the image management server 2 are stored in the program storage area 231, and programs for executing DB update processing (see FIG. 11) and image providing processing (see FIG. 13) described later are also stored. ing.

  The map data storage area 232 stores map data indicating a map image for the user to specify a point. In the present embodiment, a plurality of map data are prepared in advance so that at least the entire map of Japan and various places can be displayed at a plurality of scales. In addition, CPU20 can specify the positional information (latitude and longitude) of an actual point corresponding to the display position on the map image displayed based on map data.

  The image data storage area 233 stores image data obtained by photographing various places. The image data storage area 233 is provided with an image management database 70 for managing image data. As shown in FIG. 4, the image management database 70 stores a record indicating image data, position information indicating an acquisition position (that is, a shooting point) of image data, and a time stamp indicating the acquisition date and time of image data. Store every data. The position information indicates the latitude and longitude of the shooting point and the camera direction at the time of shooting.

  The electrical configuration of the location management server 3 will be described. The position management server 3 is a general-purpose server, and includes a CPU, a ROM, a RAM, an HDD, a communication device, and an I / O interface (not shown), like the image management server 2 (FIG. 3). However, the HDD of the location management server 3 is provided with a current location storage area (not shown) in addition to the program storage area. In this program storage area, various programs for operating the location management server 3 are stored.

  In the current position storage area, position information indicating the current position of the user terminal 5 (hereinafter referred to as current position data) is stored. In the current position data storage area, a position management database 80 for managing current position data is provided. As shown in FIG. 5, in the position management database 80, a record including a terminal ID unique to the user terminal 5 and the current position of the user terminal 5 is provided for each user terminal 5. The current position is the latest position information (latitude, longitude, and direction) acquired from the user terminal 5.

  The electrical configuration of the fixed point camera 10 will be described. Although not shown, the fixed-point camera 10 includes a camera unit that captures an installation point, and a controller that controls the camera unit and the like. The controller includes a CPU, a ROM, a RAM, and the like, and a control program is stored in the ROM. This control program causes the CPU to execute an image upload process (see FIG. 9) described later. Further, the ROM stores a terminal ID unique to the fixed point camera 10, position information of the fixed point camera 10, and the like.

  The physical configuration of the user terminal 5 will be described with reference to FIG. The user terminal 5 of the present embodiment includes a head mounted display (hereinafter referred to as HMD) 200 and an HMD control terminal 100. The HMD 200 is a glasses-type display device that a user can wear on the head and view an image. The HMD control terminal 100 is a small and lightweight electronic device connected to the HMD 200 via a cable 190, and controls display of the HMD 200 and the like.

  The HMD 200 of the present embodiment is a so-called retinal scanning display. That is, the HMD 200 scans a laser beam (hereinafter referred to as video light A) modulated in accordance with a signal of image data (hereinafter referred to as an image signal) that is visually recognized by the user, and the at least one eye of the user. Emits the retina. Thereby, HMD200 can project an image directly on a user's retina, and can visually recognize an image. The HMD 200 includes at least an emission device 210, a half mirror 250, and a head mounting portion 220. The head mounting unit 220 supports the emission device 210 and fixes the HMD 200 to the user's head. The half mirror 250 is disposed at the light emission port of the emission device 210.

  The emission device 210 emits video light A corresponding to the image signal to the half mirror 250. The half mirror 250 is in a fixed position with respect to the emission device 210. The half mirror 250 reflects the image light A emitted from the emission device 210 toward the eyes of the user. The half mirror 250 is formed by evaporating a metal thin film on a transparent resin plate so as to have a predetermined reflectance (for example, 50%). Therefore, the half mirror 250 transmits part of the external light B from the outside and guides it to the user's eyes. In other words, the half mirror 250 causes the image light A incident from the side of the user and the external light B from the outside to enter the user's eyes. As a result, the user can visually recognize the actual field of view and the image based on the video light A.

  The electrical configuration of the HMD 200 will be described with reference to FIG. The HMD 200 includes a display unit 240, a device connection interface 243, a flash memory 249, a control unit 246, a camera 207, and a power supply unit 247. The control unit 246 includes at least a CPU 261, a ROM 262, and a RAM 248, and controls the entire HMD 200. The control unit 246 executes various processes described below when the CPU 261 reads out various programs stored in the ROM 262.

  The display part 240 makes a user visually recognize an image. The display unit 240 includes an image signal processing unit 270, a laser group 272, and a laser driver group 271. The image signal processing unit 270 receives an image signal from the control unit 246. The image signal processing unit 270 converts the received image signal into signals necessary for direct projection onto the user's retina. The laser group 272 includes a blue output laser (B laser) 721, a green output laser (G laser) 722, and a red output laser (R laser) 723. The laser group 272 outputs blue, green, and red laser beams. The laser driver group 271 performs control for outputting laser light from the laser group 272. The image signal processing unit 270 is electrically connected to the laser driver group 271. Hereinafter, the B laser 721, the G laser 722, and the R laser 723 are collectively referred to as lasers.

  The display unit 240 includes a vertical scanning mirror 812, a vertical scanning control circuit 811, a horizontal scanning mirror 792, and a horizontal scanning control circuit 791. The vertical scanning mirror 812 performs scanning by reflecting the laser beam output from the laser in the vertical direction. The vertical scanning control circuit 811 performs drive control of the vertical scanning mirror 812. The horizontal scanning mirror 792 performs scanning by reflecting the laser beam output from the laser in the horizontal direction. The horizontal scanning control circuit 791 performs drive control of the horizontal scanning mirror 792. The image signal processing unit 270 is electrically connected to the vertical scanning control circuit 811 and the horizontal scanning control circuit 791.

  The image signal processing unit 270 is electrically connected to the control unit 246 via a bus. Therefore, the image signal processing unit 270 can cause each laser to output laser light with a color and timing according to the image signal received from the control unit 246. Further, the image signal processing unit 270 can reflect the laser light in a direction corresponding to the image signal received from the control unit 246. As a result, the HMD 200 can scan a light beam corresponding to the image signal in a two-dimensional direction, and guide the scanned light to the user's eyes to form a display image on the retina.

  The camera control unit 299 includes a camera 207 and a camera control circuit 208. The camera 207 captures the same direction as the user's line of sight (see FIG. 6). A camera control circuit 208 controls the camera 207. The camera control circuit 208 is electrically connected to the control unit 246 via a bus. Therefore, the control unit 246 can cause the camera 207 to perform shooting or acquire image data shot by the camera 207.

  The GPS control unit 289 includes a GPS receiver 288 and a GPS control circuit 287. The GPS receiver 288 acquires the latitude and longitude indicating the current position of the HMD 200. The GPS control circuit 287 controls the GPS receiver 288. The GPS control circuit 287 is electrically connected to the control unit 246 through a bus. Therefore, the control unit 246 can acquire the latitude and longitude of the current position from the GPS receiver 288.

  The accelerometer control unit 239 includes an accelerometer 238 and an accelerometer control circuit 237. The accelerometer 238 acquires the direction in which the user wearing the HMD 200 is traveling (that is, the shooting direction of the camera 207). The accelerometer control circuit 237 controls the accelerometer 238. The accelerometer control circuit 237 is electrically connected to the control unit 246 via a bus. Therefore, the control unit 246 can acquire the direction of the HMD 200 (that is, the shooting direction of the camera 207) from the accelerometer 238.

The device connection interface 243 is a controller that controls data communication via the cable 190. The device connection interface 243 is electrically connected to the control unit 246 via a bus. Therefore, the control unit 246 can receive image data from the HMD control terminal 100 and allow the user to visually recognize the image data. In addition, the control unit 246 can transmit image data captured by the camera 207 and position information at the time of capturing to the HMD control terminal 100.

  The power supply unit 247 includes a battery 259 and a charge control circuit 260. The battery 259 is a power source that drives the HMD 200. The charge control circuit 260 supplies the power of the battery 259 to the HMD 200. The flash memory 249, the video RAM 244, and the font ROM 245 are electrically connected to the control unit 246 via a bus. The control unit 246 can appropriately refer to information stored in the flash memory 249, the video RAM 244, and the font ROM 245.

  The electrical configuration of the HMD control terminal 100 will be described with reference to FIG. The HMD control terminal 100 is an external device that supplies image data to the HMD 200. The HMD control terminal 100 includes a CPU 120, ROM 121, RAM 122, HDD 123, communication device 124, operation key 125, device connection interface 126, and I / O interface 129. The ROM 121, RAM 122, and I / O interface 129 are each connected to the CPU 120. The HDD 123, the communication device 124, the operation keys 125, and the device connection interface 126 are connected to the I / O interface 129.

  The communication device 124 is a controller that controls data communication via the network 4. The device connection interface 126 is a controller that controls data communication via the cable 190. The HDD 133 is a large-capacity hard disk drive, and stores various programs for operating the HMD control terminal 100 and image data to be displayed on the HMD 200. The HDD 133 also stores a program for executing an upload process (see FIG. 10) and an image display process (see FIG. 14) described later.

  With reference to FIGS. 9-17, the process performed with the image provision system 1 is demonstrated. More specifically, the processes executed by the image management server 2, the position management server 3, the user terminal 5, and the fixed point camera 10 will be described.

  With reference to FIG. 9, the upload process executed by the fixed point camera 10 will be described. When the fixed point camera 10 is powered on, the CPU of the fixed point camera 10 repeatedly executes the upload process (FIG. 9) based on the control program stored in the ROM of the fixed point camera 10.

  As shown in FIG. 9, in the upload process of the fixed point camera 10, it is determined whether it is a predetermined timing for uploading image data (S1). Specifically, every time 10 minutes have passed since the previous data upload (step S9 described later), it is determined that the predetermined timing is reached (step S1: YES). In this case, image data indicating the video currently being shot is acquired from the camera unit that is always shooting the installation point (S3). The current date and time indicated by a timer (not shown) built in the fixed point camera 10 is acquired as a time stamp (S5). The positional information (that is, the latitude, longitude, and direction of the fixed point camera 10) stored in the ROM of the fixed point camera 10 is acquired (S7).

  After execution of step S7, data upload to the image management server 2 is executed (S9). Specifically, a data file including the image data acquired in step S3, the time stamp acquired in step S5, and the position information acquired in step S7 is sent to the image management server 2 via the network 4. Sent. After the execution of step S9 or when it is not a predetermined timing (S1: NO), the process returns to step S1. With the above processing, the fixed point camera 10 uploads the image data showing the current state of the installation point to the image management server 2 at intervals of 10 minutes.

  With reference to FIG. 10, the upload process performed by the user terminal 5 is demonstrated. When the user terminal 5 (that is, the HMD control terminal 100 and the HMD 200) is powered on, the CPU 120 repeatedly executes this process based on a program stored in the HDD 133.

  As shown in FIG. 10, in the upload process of the user terminal 5, the same processes as steps S1 to S9 of FIG. 9 are executed (S11 to S19). Specifically, every time 10 minutes have passed since the previous data upload (step S19 described later), it is determined that the predetermined timing is reached (step S11: YES). In step S11, even when a condition other than the elapsed time is satisfied (for example, when the user performs imaging using the camera 207 via the operation key 125), it is determined that the predetermined timing is reached. Good. In this case, the camera 207 of the HMD 200 captures the current position, and image data captured by the camera 207 is acquired from the HMD 200 (S13). The current date and time indicated by a timer (not shown) built in the HMD control terminal 100 is acquired as a time stamp (S15).

  Further, the GPS receiver 288 is caused to measure the latitude and longitude of the current position. The accelerometer 238 is caused to measure the direction of the user terminal 5. The latitude, longitude, and direction measured in this way are acquired from the HMD 200 as position information of the current position (S17). A data file including the information acquired in steps S13, S15, and S17 is transmitted to the image management server 2 via the network 4 (S19).

  However, after execution of step S19, the current position notification is transmitted to the position management server 3 via the network 4 (S21). The current position notification includes the terminal ID unique to the user terminal 5 and the position information acquired in step S17. After execution of step S21, or when it is not a predetermined timing (S11: NO), the process returns to step S11. Through the above processing, the user terminal 5 uploads the image data showing the current position of the user terminal 5 to the image management server 2 at intervals of 10 minutes. Further, the current position of the user terminal 5 is notified to the position management server 3 at intervals of 10 minutes.

  With reference to FIG. 11, a DB update process executed by the image management server 2 will be described. When the image management server 2 is powered on, the CPU 20 repeatedly executes this process based on a program stored in the HDD 33.

  As shown in FIG. 11, in the DB update process of the image management server 2, it is determined whether or not there is a data file upload (S31). Specifically, when a data file is transmitted from the user terminal 5 or the fixed point camera 10, it is determined that there is an upload (S31: YES). In this case, the uploaded data file is received (S33), and the image management database 70 is updated (S35). After execution of step S35 or when there is no upload (S31: NO), the process returns to step S31 and waits for the upload of the data file. Through the above processing, the image management server 2 can collect image data of various locations from the fixed point camera 10 and the user terminal 5.

  In the example illustrated in FIG. 4, a record including the image data “camera_a.jpg” is registered in the image management database 70 based on the data file uploaded from the fixed point camera 10 “camera_a”. In this record, position information indicating the installation position of the fixed point camera 10 “camera_a” and a time stamp indicating the acquisition date and time of the image data “camera_a.jpg” are set.

  Further, a record including the image data “hmd001_a.jpg” is registered in the image management database 70 based on the data file uploaded from the user terminal 5 “hmd001” worn by the user A (see FIG. 2). . In this record, for image data “camera_a.jpg”, position information indicating the position of the user terminal 5 “hmd001” at the time of acquisition and a time stamp indicating the acquisition date and time are set.

  With reference to FIG. 12, the DB update process executed by the location management server 3 will be described. When the position management server 3 is powered on, the CPU repeatedly executes this process based on a program stored in the HDD.

  As shown in FIG. 12, in the DB update process of the location management server 3, it is determined whether or not there is a current location notification (S41). Specifically, when the current position notification is received from the user terminal 5, it is determined that there is a current position notification (S41: YES). In this case, the location management database 80 is updated based on the received current location notification (S43). After execution of step S43 or when there is no current position notification (S43: NO), the process returns to step S41 and waits for reception of the current position notification. With the above processing, the position management server 3 can always grasp the installation position of the fixed point camera 10 fixed in each place and the current position of the user terminal 5 that can move to each place.

  In the example shown in FIG. 5, a record including the terminal ID “hmd001” is registered in the location management database 80 based on the current location notification received from the user terminal 5 “hmd001” worn by the user A (see FIG. 2). Has been. In this record, position information indicating the current position of the user terminal 5 “hmd001” is set.

  With reference to FIG. 13, the image providing process executed by the image management server 2 will be described. When the image management server 2 is powered on, the CPU 20 repeatedly executes this process based on a program stored in the HDD 33.

  As shown in FIG. 13, in the image providing process of the image management server 2, it is determined whether a map request has been received (S51). The map request is a signal for requesting map data from the image management server 2 by the user terminal 5, and includes a display range of a map image designated by the user (hereinafter referred to as a map display range). When a map request is received (S51: YES), map data indicating the map display range included in the map request is acquired from the map data storage area 232 (S53).

  A time stamp corresponding to the map display range of the map data acquired in step S53 is acquired from the image management database 70 (S55). Specifically, all image data having latitude and longitude included in the map display range of the map data is searched from the image management database 70. A time stamp associated with the image data hit in the search is acquired from the image management database 70. After execution of step S55, map drawing information is transmitted to the requesting user terminal 5 via the network 4 (S57). The map drawing information is information used to display a map image on the user terminal 5, and includes the map data acquired in step S53 and the time stamp acquired in step S55.

  For example, when the map display range includes latitude “+33 35 42.374” and longitude “+130 24 21.35”, time stamps are acquired from all records including position information matching the latitude and longitude (S55). ). In the example of the image management database 70 illustrated in FIG. 4, the time stamp “100707120230” is acquired. In step S57, the map data corresponding to the map display range is transmitted to the requesting user terminal 5 together with the time stamp indicating the image acquisition date and time at each point in the map display range.

  After execution of step S57, it is determined whether an image request has been received (S59). The image request is a signal for the user terminal 5 to request image data from the image management server 2 and includes position information of a point designated by the user (hereinafter referred to as a designated point). When the image request is received (S59: YES), the image data associated with the position information of the designated point is specified with reference to the image management database 70 (S61). A list of image data, which is a list of image data specified in step S61, is transmitted to the requesting user terminal 5 via the network 4 (S63).

  For example, when the image request includes latitude “+33 35 42.374”, longitude “+130 24 21.35”, and direction “north”, image data from all records including position information matching the latitude, longitude, and direction. Is identified (S61). In the example of the image management database 70 illustrated in FIG. 4, the image data “camera_a.jpg” is specified. In step S63, thumbnail images obtained by reducing and displaying each of these image data and an image data list indicating the acquisition date and time are transmitted to the user terminal 5 that has made the request.

  After execution of step S63, it is determined whether an image selection instruction has been received (S65). The image selection instruction includes identification information of image data selected by the user from the image data list. Based on the identification information included in the image selection instruction, the CPU 20 can identify the image data selected by the user from among the image data identified in step S61. When an image selection instruction is received (S65: YES), it is determined whether the image data selected by the user is the current image (S67). Specifically, the time stamp of the image data selected by the user is specified with reference to the image management database 70. If the elapsed time from the identified time stamp to the current date and time is within a predetermined value (in this embodiment, less than 10 minutes), it is determined that the image data is the current image (S67: YES).

  In this case, an image provision instruction is transmitted to the designated point via the network 4 (S69). The image providing instruction provides image data being captured by the requesting user terminal 5 (in this embodiment, video data that is a moving image that can be played back in real time) to the image acquisition device existing at the designated point. Is a signal for instructing. More specifically, in step S <b> 69, the CPU 20 inquires of the position management server 3 about the image acquisition device existing at the designated point. In response to an inquiry from the image management server 2, the CPU of the position management server 3 refers to the position management database 80 to identify the image acquisition device corresponding to the designated point and notifies the image management server 2 of the image acquisition device. The CPU 20 transmits a request for transmitting video data in real time to the requesting user terminal 5 to the image acquisition apparatus notified from the position management server 3.

  For example, when the image data indicated by the image selection instruction is “camera_a.jpg”, the time difference between the time stamp “100707120230” and the current date is less than 10 minutes. Therefore, when the position information of the designated point is latitude “+33 35 42.374”, longitude “+130 24 21.35”, and direction “north”, the image acquisition device is specified from the record including this position information (S67). : YES, S69). In the example of the position management database 80 illustrated in FIG. 5, the fixed point camera 10 “camera_a” is specified as the image acquisition device. In step S71, a request is transmitted to the fixed point camera 10 “camera_a”, thereby providing the requesting user terminal 5 with the video data of the fixed point camera 10 “camera_a” in real time.

  On the other hand, if the image data is not the current image (S67: NO), the image data selected by the user is read from the image management database 70 and transmitted to the requesting user terminal 5 via the network 4 (S71). ). For example, when the image data indicated by the image selection instruction is “hmd — 001.jpg”, the time difference between the time stamp “100707080513” and the current date and time is 10 minutes or more (S67: NO). Therefore, in step S71, the image data “hmd — 001.jpg” is read from the image management database 70 and transmitted to the requesting user terminal 5.

  In addition, after execution of step S69 or step S71, the process returns to step S51. Similarly, if a map request is not received (S51: NO), an image request is not received (S59: NO), and an image selection instruction is not received (S65: NO), the process returns to step S51. With the above processing, the image management server 2 can provide map data and image data of a designated point in response to a request from the user terminal 5. When the image acquisition device exists at the designated point, the video data can be provided to the user terminal 5 in real time from the image acquisition device at the designated point.

  With reference to FIG. 14, the image display process performed by the user terminal 5 is demonstrated. The HDD 133 stores an application program that displays image data, map data, and the like provided from the image management server 2. The image display process is repeatedly executed by the CPU 120 when the application program is activated.

  As shown in FIG. 14, in the image display process of the user terminal 5, it is determined whether or not there is a map display operation (S101). The map display operation is such that the user wearing the HMD 200 moves, enlarges, reduces, etc. a map image (in detail, a heat map described later) projected on the eyes using the operation keys 125 (see FIG. 8). It is an operation to do. When there is a map display operation (S101: YES), a map request is transmitted to the image management server 2 via the network 4 (S103). After execution of step S103, it is determined whether map drawing information has been received (S105: YES). When the map drawing information is received (S105), a heat map is created (S107). The heat map is an image in which a special display indicating temporal reliability of the image data is added to the map data. Temporal reliability means whether or not the elapsed time calculated from the acquisition of image data is small (that is, whether or not the image data is new).

  For example, when the user B wearing the user terminal 5 (see FIG. 2) sets a map display range including the latitude “+33 35 42.374” and the longitude “+130 24 21.035”, this map display range Is transmitted (S103). As described above, the map drawing information including the map data and the time stamp is returned from the image management server 2. The CPU 120 creates a heat map based on the map drawing information received from the image management server 2 (S107).

  As shown in FIG. 15, the map data 300 included in the map drawing information is a map image showing a map display range. In step S107, the display area for each place included in the map display range is color-coded according to the time stamp of each point included in the map drawing information. In the present embodiment, a heat map is created that indicates the newness of the update date and time of image data at each point included in the map display range in a color-coded manner. In this heat map, a point where the time stamp is new is colored in a warm color, while a point where the time stamp is old is colored in a cold color.

  Specifically, in the heat map 310 illustrated in FIG. 16, when the time stamp indicates less than 10 minutes from the current date and time, the display area of the point corresponding to the time stamp is colored “red”. When the time stamp indicates 10 minutes or more from the current date and time and less than 1 hour, the display area of the point corresponding to the time stamp is colored “pink”. When the time stamp indicates one hour or more from the current date and time and less than one day, the display area of the point corresponding to the time stamp is colored “yellow”. When the time stamp indicates one day or more from the current date and time, the display area of the point corresponding to the time stamp is colored “colorless”. Since the display area of the point where the time stamp does not exist does not include image data obtained by photographing the point, the display area of the point is colored “black”.

  Therefore, the user can easily visually determine whether or not the image data of each place included in the map display range is new with reference to the heat map 310. In other words, the user can visually discriminate between a point where the image data acquisition date is new and a point where the image data acquisition date is old with reference to the heat map 310.

  The heat map created in step S107 is displayed on the HMD 200 (S109). Specifically, the heat map is displayed on the eyes of the user wearing the HMD 200. After execution of step S109, it is determined whether or not there is an input of a designated point / direction (S111). For example, when the user performs a cursor operation with the operation key 125 and designates the coordinate position and the shooting direction on the heat map, it is determined that there is an input of the designated point / direction (S111: YES). In this case, an image request for requesting image data at the designated point is transmitted to the image management server 2 via the network 4 (S113). After execution of step S113, it is determined whether an image data list has been received (S115). When the image data list is received (S115: YES), the image data list is displayed on the HMD 200 (S117).

  For example, when the user B wearing the user terminal 5 (see FIG. 2) specifies the latitude “+33 35 42.374”, the longitude “+130 24 21.35”, and the direction “north” on the heat map, An image request including the position information is transmitted (S113). As described above, the image management server 2 returns a list of image data including the thumbnail image of the image data of the designated point (image data “camera_a.jpg” in the example of FIG. 4) and the acquisition date and time. Based on the image data list received from the image management server 2, the CPU 120 displays the image data of the designated point in the form of thumbnails on the eyes of the user wearing the HMD 200 (S117).

  After execution of step S117, it is determined whether image data has been selected (S119). For example, when the user performs a cursor operation with the operation key 125 and selects a thumbnail image from the image data list, it is determined that image data has been selected (S119: YES). In this case, an image selection instruction including identification information of the selected image data is transmitted to the image management server 2 via the network 4 (S121). After execution of step S121, a data display process described later is executed (S123), and the process returns to step S101.

  Similarly, when there is no map display operation (S101: NO), when map drawing information is not received (S105: NO), when there is no input of a designated point / direction (S111: NO), an image data list is received. If not (S115: NO) and if no target image is selected (S119: NO), the process returns to step S101. With the above processing, the user terminal 5 can display a heat map indicating the temporal reliability of image data obtained by photographing each point on the map image in response to a user request.

  As shown in FIG. 17, in the data display process (S123), it is determined whether image data has been received (S151). When image data is received from the image management server 2 (S151: YES), it is determined whether the image data is data older than a predetermined time (S153). Specifically, when the acquisition date and time indicated by the time stamp of the image data indicates one month or more before the current date and time, it is determined that the image data is data older than a predetermined time (S153: YES). In this case, a predetermined warning is displayed on the HMD 200 (S155). Examples of the predetermined warning include “This image was acquired more than a month ago and may be different from the current landscape”.

  After the execution of step S155, or when the image data is not older than the predetermined time (S153: NO), the received image data is displayed on the HMD 200 (S157). If a warning is displayed in step S155, image data is displayed together with the warning. On the other hand, if no image data has been received (S151: NO), it is determined whether video data has been received (S159). When video data is received from the fixed point camera 10 (S159: YES), the received video data is streamed and played back by the HMD 200 (S161).

  For example, when the user B selects the image data “camera_a.jpg” from the image data list, an image selection instruction including the identification information is transmitted (S121). As described above, the image data “camera_a.jpg” is returned from the image management server 2. The CPU 120 displays the image data “camera_a.jpg” received from the image management server 2 on the eyes of the user wearing the HMD 200 (S157).

  On the other hand, when the user B selects the image data “hmd_001.jpg” from the image data list, an image selection instruction including the identification information is transmitted (S121). As described above, real-time video data is transmitted from the fixed point camera 10 “hmd_001”. The CPU 120 displays the video data received from the fixed point camera 10 “hmd_001” on the eyes of the user wearing the HMD 200 (S161).

  In addition, after execution of step S157 or step S161, or when no video data is received (S159: NO), the data display process (FIG. 15) ends, and the process returns to the image display process (FIG. 14). . Through the above processing, the user terminal 5 can acquire and display image data or video data of a designated point designated from the heat map from the image management server 2 or the fixed point camera 10.

  As described above, according to the image providing system 1 according to the present embodiment, the image data acquired by the fixed point camera 10 or the user terminal 5 is stored in the image management database 70 together with the position information and time stamp at the time of acquisition. Remembered. In the image management server 2, the map data stored in the map data storage area 232 is returned to the user terminal 5 in response to a request from the user terminal 5. At this time, the time stamp associated with the position information included in the display range of the map data is also returned to the user terminal 5.

  In the user terminal 5, a heat map is generated based on the map data and time stamp acquired from the image management server 2 and displayed on the HMD 200. A heat map is an image in which a special display indicating temporal reliability of image data is added to the map data. Therefore, the user of the user terminal 5 can easily grasp the reliability of the image data obtained by photographing each point on the map image by referring to the heat map displayed on the HMD 200.

  Further, the user of the user terminal 5 displays the image data of the designated point selected on the heat map on the HMD 200. Therefore, the user can browse the image data with the HMD 200 after confirming the temporal reliability of the image data at each point on the heat map. Further, when the image data of the designated point selected on the heat map is older than a predetermined time, a warning is displayed on the HMD 200. Therefore, the user can surely grasp that the temporal reliability of the image data at the designated point is low. In addition, when there are a plurality of image data at a designated point, the user can freely select image data to be displayed on the HMD 200 from the image data list.

  In the above embodiment, the fixed point camera 10 and the user terminal 5 correspond to the “image acquisition device” of the present invention. The image management server 2 corresponds to the “server” of the present invention. The user terminal 5 corresponds to the “image display device” of the present invention. The CPU of the fixed point camera 10 that executes step S3 and the CPU 120 that executes step S13 correspond to the “image data acquisition unit” of the present invention. The CPU of the fixed point camera 10 that executes step S7 and the CPU 120 that executes step S17 correspond to the “positional information acquisition unit” of the present invention. The CPU of the fixed point camera 10 that executes step S5 and the CPU 120 that executes step S15 correspond to the “reliability information generation unit” of the present invention. The CPU of the fixed-point camera 10 that executes step S9 and the CPU 120 that executes step S19 correspond to the “information transmitting unit” of the present invention.

  The CPU 20 executing step S35 corresponds to the “storage control unit” of the present invention. The CPU 20 that executes step S57 corresponds to the “information transmitting unit” of the present invention. The CPU 120 executing steps S103 and S105 corresponds to the “information acquisition unit” of the present invention. The CPU 120 that executes step S107 corresponds to the “composite image generation unit” of the present invention. The CPU 120 executing step S109 corresponds to the “composite image display unit” of the present invention. The CPU 120 that executes step S111 corresponds to the “position specifying means” of the present invention. The CPU 120 that executes steps S151 and S159 corresponds to the “target data acquisition unit” of the present invention. The CPU 120 that executes steps S157 and S161 corresponds to the “target data display unit” of the present invention. The CPU 120 executing step S155 corresponds to the “warning display means” of the present invention. The CPU 120 executing step S117 corresponds to the “target list display unit” of the present invention.

  The CPU 120 executing steps S103 and S105 corresponds to the “map data acquisition unit” and “reliability information acquisition unit” of the present invention. Steps S103 and S105 correspond to the “map data acquisition step” and “reliability information acquisition step” of the present invention. Step S107 corresponds to the “composite image generation step” of the present invention. Step S109 corresponds to the “composite image display step” of the present invention.

  In addition, this invention is not limited to the said embodiment, The change in the range which does not change the summary of invention is possible. In the above embodiment, the retinal scanning display is exemplified as the HMD 200, but the display method can be changed. For example, it may be a head mounted display of another display method such as a liquid crystal display or an organic EL (ElectroLuminescence) display. Further, instead of the HMD 200, a display device that can be carried by the user, such as a mobile phone, a notebook computer, or a PDA, may be used.

  Moreover, in the said embodiment, the time stamp which shows the acquisition date and time of image data was illustrated as reliability information which shows the temporal reliability of image data. For example, the update frequency of the image data may be used as the reliability information. In this case, in the image management database 70, the update frequency of the image data (for example, 10 minute interval, 1 hour interval, 1 day interval, etc.) may be managed instead of the time stamp.

  At the time of creating the heat map (S107), the time interval of the update frequency of the image data at each point included in the map display range (that is, the number of updates per unit time) is color-coded and shown stepwise. A heat map is created. In this heat map, warm colors are used for points with a high update frequency, while cool colors are used for points with a low update frequency. As a result, the user can easily visually determine whether or not the image data of each place included in the map display range is new with reference to the heat map. In other words, the user can visually discriminate between a point where the image data update frequency is high and a point where the image data update frequency is low with reference to the heat map.

  Further, a part of the image display process (FIG. 14) of the user terminal 5 may be executed by another computer. For example, in the image management server 2, instead of transmitting the map drawing information (S57), heat map creation (S107) may be executed. Then, the heat map created by the image management server 2 may be transmitted to the user terminal 5. In this case, the heat map can be displayed on the HMD 200 while omitting the heat map creation process in the user terminal 5.

DESCRIPTION OF SYMBOLS 1 Image provision system 2 Image management server 3 Position management server 4 Network 5 User terminal 10 Fixed point camera 70 Image management database 80 Position management database 100 HMD control terminal 120 CPU
121 ROM
122 RAM
123 HDD
200 Head Mount Display 300 Map Data 310 Heat Map

Claims (9)

An image providing system in which an image acquisition device that acquires image data, a server that manages the image data, and an image display device that displays the image data are connected via a network,
The image acquisition device includes:
Image data acquisition means for acquiring the image data from an imaging means for imaging the periphery of the image acquisition device;
Position information acquisition means for acquiring position information indicating an acquisition position of the image data from position detection means for detecting a current position of the image acquisition device;
Reliability information generating means for generating reliability information indicating temporal reliability of the image data;
The image data acquired by the image data acquisition means, the position information acquired by the position information acquisition means, and the reliability information generated by the reliability information generation means are transmitted to the server. An information transmission means,
The server
Storage control means for associating the image data received from the image acquisition device, the position information, and the reliability information, and storing them in a first storage device;
In response to a request from the image display device, the map data stored in the second storage device, and the reliability information associated with the position information included in the display range of the map data, An information transmission means for replying to the image display device,
The image display device includes:
Information acquisition means for acquiring the map data and the reliability information from the image display device by transmitting the request to the server;
Based on the map data and the reliability information acquired by the information acquisition means, a composite image generation means for generating a composite image in which a special display indicating temporal reliability of the image data is added to the map data; ,
An image providing system comprising: a composite image display unit configured to display the composite image generated by the composite image generation unit on a first display device. An image providing system in which an image acquisition device that acquires image data, a server that manages the image data, and an image display device that displays the image data are connected via a network,
The image acquisition device includes:
Image data acquisition means for acquiring the image data from an imaging means for imaging the periphery of the image acquisition device;
Position information acquisition means for acquiring position information indicating an acquisition position of the image data from position detection means for detecting a current position of the image acquisition device;
Reliability information generating means for generating reliability information indicating temporal reliability of the image data;
The image data acquired by the image data acquisition means, the position information acquired by the position information acquisition means, and the reliability information generated by the reliability information generation means are transmitted to the server. An information transmission means,
The server
Storage control means for associating the image data received from the image acquisition device, the position information, and the reliability information, and storing them in a first storage device;
In response to a request from the image display device, based on the map data stored in the second storage device and the reliability information associated with the position information included in the display range of the map data A composite image generating means for generating a composite image in which a special display indicating temporal reliability of the image data is added to the map data;
An information transmission means for returning the composite image generated by the composite image generation means to the image display device;
The image display device includes:
Information acquisition means for acquiring the composite image from the image display device by transmitting the request to the server;
An image providing system comprising: a composite image display unit that displays the composite image acquired by the information acquisition unit on a first display device. The reliability information is a time stamp indicating the date and time when the image data was acquired,
The special display is a display mode in which the image data having a new acquisition date and time indicated by the time stamp and the image data having an old acquisition date and time indicated by the time stamp can be visually distinguished on the map data. The image providing system according to claim 1 or 2. The reliability information is update frequency data indicating how often the image data is updated,
The special display is a display mode in which the image data having a high update frequency indicated by the update frequency data and the image data having a high update frequency indicated by the update frequency data can be visually distinguished on the map data. The image providing system according to claim 1, wherein the image providing system is provided. The image display device includes:
In the composite image displayed on the first display device, position specifying means for the user to specify a target position;
Target data acquisition means for acquiring target data, which is the image data corresponding to the target position specified by the position specifying means, from the first storage device;
The image providing system according to claim 1, further comprising: target data display means for displaying the target data acquired by the target data acquisition means on a second display device.   The image display device includes warning display means for displaying a warning on the second display device when the temporal reliability indicated by the reliability information corresponding to the target data is lower than a predetermined value. The image providing system according to claim 5. The image display device includes target list display means for displaying the plurality of target data in a list or thumbnail form on the first display device when a plurality of the target data is acquired by the target data acquisition means. ,
6. The target data display unit displays the target data selected by a user among the plurality of target data displayed by the target list display unit on the second display device. 6. The image providing system according to 6. With reference to a storage device that stores map data, map data acquisition means for acquiring the map data;
Referring to a storage device that associates and stores image data, position information indicating the acquisition position of the image data, and reliability information indicating temporal reliability of the image data, and included in the display range of the map data Reliability information acquisition means for acquiring the reliability information associated with the position information,
Based on the map data acquired by the map data acquisition means and the reliability information acquired by the reliability information acquisition means, a special display indicating temporal reliability of the image data is displayed on the map data. A composite image generating means for generating the added composite image;
An image display device comprising: a composite image display unit configured to display the composite image generated by the composite image generation unit on a display device. On the computer,
With reference to a storage device that stores map data, a map data acquisition step of acquiring the map data;
Referring to a storage device that associates and stores image data, position information indicating the acquisition position of the image data, and reliability information indicating temporal reliability of the image data, and included in the display range of the map data A reliability information acquisition step of acquiring the reliability information associated with the position information
Based on the map data acquired by the map data acquisition step and the reliability information acquired by the reliability information acquisition step, a special display indicating temporal reliability of the image data is displayed on the map data. A composite image generation step for generating the added composite image;
An image display program for executing a composite image display step of displaying the composite image generated in the composite image generation step on a display device.

Images