HK1242876B - Imaging device, information acquisition system, program and recording medium - Google Patents

Description

Photographic device, information acquisition system, program, and recording medium

This application is a divisional application of the Chinese patent application with application number 2011800086543, and the invention name is “Photographic device, information acquisition system and program”. The application date of the original application is February 7, 2011.

Technical Field

The present invention relates to a photographic device having the function of obtaining position information of a photographed subject when photographing the subject and establishing a correspondence between the subject and the position information, as well as a photographic device, an information acquisition system and a program for obtaining information about the subject using the information.

This application claims priority based on Japanese Patent Application No. 2010-025998, filed on February 8, 2010, the contents of which are incorporated herein by reference.

Background Art

In the past, among portable information devices such as photographic devices (e.g., digital cameras) with photographic functions, there has been a device that has the following function, namely, detecting the location information of the portable information device and processing the captured image data corresponding to the detected location information (see, for example, Patent Document 1).

[Patent Document 1] Japanese Patent Application Laid-Open No. 2004-15187

Summary of the Invention

However, including the above-mentioned Patent Document 1, the user cannot easily obtain information on what the subject (eg, a building) the user is photographing.

An aspect of the present invention is to provide a photographing device, an information acquisition system, and a program that can enrich the activities of users.

The photographic device of the present invention comprises: a photographing unit for photographing a subject; a position information acquisition unit for acquiring position information of a photographing position; a control unit for acquiring information related to the subject based on the position information, and displaying the image data of the subject and the information related to the subject on a display unit; and a retention control unit for outputting a retention control signal for retaining the image data of the subject and the information related to the subject to the control unit.

A photographic device according to an aspect of the present invention comprises: a latitude and longitude detection unit for detecting the latitude and longitude information of its own position; an azimuth detection unit for detecting the azimuth in captured image data; a control unit for obtaining AR information attached to a building located within the latitude and longitude range in the azimuth direction in the latitude and longitude information by means of the latitude and longitude information and the azimuth, and displaying the information on a display unit; and a retention control unit for outputting a retention control signal to the control unit for storing the AR information and the image data in the storage unit after detecting an action of storing the AR information in the storage unit.

An information acquisition system according to an aspect of the present invention comprises a photographic device and an information retrieval system; the photographic device is any one of the above-mentioned photographic devices.

An information acquisition system of an aspect of the present invention is composed of a photographic device and an information retrieval system, wherein the information retrieval system extracts a building located within the latitude and longitude range of the azimuth direction in the latitude and longitude information by using the latitude and longitude information and the azimuth sent from the aforementioned photographic device, and sends information attached to the extracted building to the aforementioned photographic device; the aforementioned photographic device comprises: a latitude and longitude detection unit for detecting the latitude and longitude information of its own position; an azimuth detection unit for detecting the azimuth in captured image data; a control unit for acquiring AR information attached to the building located within the latitude and longitude range of the azimuth direction in the aforementioned latitude and longitude information by using the latitude and longitude information and the azimuth and displaying the information on a display unit; and a maintenance control unit for storing the AR information in a storage unit after detecting an action of storing the AR information in the storage unit. The information retrieval system outputs to the control unit according to a hold control signal stored in a storage unit; the information retrieval system comprises: a database storing map data in which a building identification number of a building is associated with latitude and longitude information of the building, and a building table in which the building identification number and the AR information of the building displayed by the building identification number are associated; and an information retrieval server, which retrieves from the map data the building identification number of a building located within a latitude and longitude range in the azimuth direction of the latitude and longitude information, based on the latitude and longitude information and the azimuth transmitted from the photographic device; reads from the building table the AR information of the building displayed by the retrieved building identification number, and transmits the read AR information of the building to the photographic device.

A program according to an aspect of the present invention is a program for causing a computer to execute the functions of any of the above-mentioned photographic devices, which is used to cause a computer to execute: a step of inputting position information of a photographic position for photographing a subject; a step of obtaining information related to the subject based on the above-mentioned position information; a step of displaying the image data of the subject and the information related to the subject on a display unit; and a step of outputting a retention control signal for maintaining the image data of the subject and the information related to the subject to a control unit.

A program according to an aspect of the present invention is a program for causing a computer to execute the functions of any of the above-mentioned photographic devices, which is used to cause the computer to execute: the step of inputting the latitude and longitude information of the own position detected by the latitude and longitude detection unit; the step of inputting the azimuth in the captured image data detected by the azimuth detection unit; the step of obtaining and displaying on the display unit AR information attached to a building located within the latitude and longitude range of the aforementioned azimuth direction in the aforementioned latitude and longitude information; and the step of outputting to the aforementioned control unit a maintenance control signal for storing the aforementioned AR information and the aforementioned image data in the storage unit after detecting an action of storing the aforementioned AR information in the storage unit.

According to aspects of the present invention, a photographing device, an information acquisition system, and a program are provided that can enrich user activities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram showing a configuration example of an information acquisition system according to an embodiment of the present invention.

FIG. 2 is a conceptual diagram showing an example of the structure of a building table stored in the database shown in FIG. 1 .

FIG. 3 is a flow chart showing the flow of information retrieval processing of an image captured by a digital camera.

FIG. 4 is a conceptual diagram showing a display image displayed on a digital camera.

FIG. 5 is a conceptual diagram showing a display image displayed on a digital camera.

FIG. 6 is a conceptual diagram showing a display image displayed on a digital camera.

FIG. 7 is a flow chart showing the information retrieval process of an image captured by a digital camera.

FIG8 is a flowchart showing the flow of processing for retrieving information from store information input to a digital camera.

FIG9 is a block diagram showing an information acquisition system having the function of acquiring and storing AR (augmented reality) information.

FIG10 is a diagram showing a composite image on a display unit of a real image of a building and an image of a marker (virtual description) on which AR information of the building is described.

FIG. 11 is a diagram showing an AR information table stored in the AR information storage unit in FIG. 9 .

FIG. 12 is a diagram showing a search range for searching AR information of a building.

FIG13 is a flowchart showing an example of the operation of the information retrieval system in FIG9.

【Main component symbol description】

1 Digital camera 2 Information retrieval system

3 Wireless base stations 4 Information communication network

11 Control Unit 12 Transceiver Unit

13 Photography Department 14 GPS

15 Storage unit 16 Direction sensor

17 Display 18 Timer

19 Navigation Department 21 Information Retrieval Server

22 Database 30 AR Information Storage

31 Maintain Control

DETAILED DESCRIPTION

Hereinafter, an imaging device and an information acquisition system according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of an imaging device and an information acquisition system according to this embodiment.

In Figure 1, the information acquisition system consists of a digital camera 1, an example of a photographic device, and an information retrieval system 2. The digital camera 1 communicates data with a wireless base station 3 via wireless communication. Data is then sent and received to the information retrieval system 2 (or an information retrieval server 21, described later) via this wireless base station 3 and an information communication network 4, such as the Internet. Furthermore, the photographic device of this embodiment may be a portable terminal with a photographic function, such as the digital camera 1 described above, or a mobile phone equipped with a camera. For example, the digital camera 1 has a browsing function and includes a control unit 11, a transceiver unit 12, a photographic unit 13, a GPS (Global Positioning System) 14, a storage unit 15, an azimuth sensor 16 (azimuth angle detection unit), a display unit 17, and a timer 18. The wireless communication of this embodiment does not use wires as a transmission path, but includes communication methods such as radio waves, light, or sound waves.

The imaging unit 13 includes a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor, a lens, and other components, and outputs image data of a subject (the object being photographed) to the control unit 11. In the following description, the subject in this embodiment represents, as an example, a building (commercial buildings including stores, public buildings including schools and hospitals, factories, residential buildings including apartments, towers, bridges, dams, amusement parks, artificial structures such as statues, etc.). When a control signal instructing the user to take a picture (e.g., a control signal output by a peripheral circuit (not shown) that detects that the picture button B1 has been pressed) or a control signal instructing information retrieval (e.g., a control signal output by a peripheral circuit (not shown) that detects that the user has touched an image (marker) displaying a search function and selected an image) is input to the GPS 14 (position information acquisition unit, latitude and longitude detection unit), the GPS 14 acquires the latitude and longitude information (latitude and longitude information, position information) of the digital camera 1 and outputs the information to the control unit 11. The position information acquisition unit (latitude and longitude detection unit) of this embodiment may have other configurations as long as it can acquire the position of the photographing device 13. For example, the wireless base station 3 may be used to acquire the position information of the photographing device 13.

The orientation sensor 16 (or orientation angle detection unit, orientation information acquisition unit) is comprised of an electronic compass or the like. When a control signal instructing the user to take a picture (e.g., a control signal output by a peripheral circuit (not shown) that detects the pressing of the capture button B1) or a control signal instructing information retrieval (e.g., a control signal output by a peripheral circuit (not shown) that detects the user selecting an image by touching an image (marker) displayed for retrieval, etc.) is input, the imaging unit 13, for example, comprises a CCD and lens arranged in series, detects the orientation of the series direction and outputs it as orientation information to the control unit 11. Furthermore, the orientation sensor 16 can also detect the orientation of the subject as viewed from the user (i.e., the imaging direction) and output this orientation information to the control unit 11. Here, when azimuth is used as the direction information, for example, with the latitude and longitude of the above-mentioned location information as the center point, and north as the reference (0 degrees), the azimuth is expressed in degrees, minutes, and seconds in a clockwise direction around east (90 degrees), south (180 degrees), and west (270 degrees).

When a control signal for taking a photo is inputted from a peripheral circuit (not shown) to instruct the user to press the photo button B1, etc., the control unit 11 assigns image identification information to the captured image data, and writes the image data captured for each image identification information into the storage unit 15 in correspondence with the time data, position information and orientation information obtained from the timer 18 in the order of photographing.

The display unit 17 displays the image data captured by the imaging unit 13 or the image data read from the storage unit 15 by the control unit 11 in response to the user selecting an image to be displayed.

Furthermore, the display unit 17 displays image data received from the information retrieval system 2 or text data input by the user under the control of the control unit 11 as will be described later.

The transceiver 12 transmits and receives signals with the wireless base station 3 via wireless communication, and transmits and receives data such as image data, text data, or control signals with the information retrieval system 2 via the information communication network 4 .

Secondly, the information retrieval system 2 has an information retrieval server 21 and a database 22 as shown in Figure 1. In addition, the database 22 can also be located in the storage medium (such as memory or HDD etc.) in the information retrieval server 21, and can also be located in an external storage medium or the storage medium of other terminals.

As shown in FIG2 , the building table stores in the database 22 building identification information for identifying the building, the name of the building, i.e., the building name, building information (information such as address, telephone number, type, and surrounding image data centered on the building), location information such as the latitude and longitude of the building, a description of the building (information recorded by the store if it is a store), and contribution information (e.g., comments such as evaluations by users who have visited the building and image data contributed by users).

Furthermore, map data in which registered buildings are arranged with latitude and longitude as coordinate axes of a two-dimensional plane is stored in the database 22. In this map data, each building is displayed at the latitude and longitude position corresponding to the building identification information.

The information search server 21 uses the input position information and direction information to search for the closest building in the direction indicated by the direction information from the latitude and longitude indicated by the position information, and obtains building identification information of the found building.

The information search server 21 then searches the building table for the building corresponding to the building identification information of the found building and transmits the retrieved building information (building name, building information, etc.) to the digital camera 1. Furthermore, the information search server 21 may selectively transmit the aforementioned building information to the digital camera 1, depending on circumstances such as the required data capacity for transmission and reception. In this case, the information search server 21 may further transmit the remaining building information to the digital camera 1 in response to a predetermined operation (e.g., a request from the digital camera 1).

<Information Retrieval of Images Photographed by Digital Camera 1>

Next, the operation of this embodiment will be described using Figures 1 and 3, as well as Figures 4 to 6. Figure 3 is a flowchart showing an example of the information retrieval process of this embodiment when the digital camera 1 is pointed at a building and is photographing the building and an information retrieval is attempted. Figures 4 to 6 show the screen displayed on the display unit 17 of the digital camera 1 and the arrangement of the capture button B1 of the digital camera.

In the following description, for example, a situation in which a user is walking in a downtown area while searching for a restaurant for dinner will be described.

While wandering in a downtown area, the user points the lens of the digital camera 1 toward a restaurant with a pleasing appearance (step S1). When the user finds a restaurant with an appearance for which he wants to obtain information, as shown in FIG4 , he touches (presses) the browsing image I1 provided in the image display bar MP of the digital camera 1, thereby selecting the browsing image I1 (step S2).

At this time, in the above-described shooting state, the control unit 11 displays the image data of the restaurant being shot, which is input from the shooting unit 13, in the image display field MP via the display unit 17, as shown in FIG4 . Furthermore, the image display field MP (image display unit) in this embodiment is configured as a touch panel (including, for example, a pressure-sensitive type, an electrostatic type, or an acoustic wave type).

In addition, the step S2 can also be performed by pressing the shooting button B1 of the digital camera 1 and viewing the image display bar MP showing the image of the restaurant while touching the browse image I1 to make a selection.

By touching the browsing image I1, a signal indicating that the browsing image I1 has been selected is output from a peripheral circuit (not shown) to the control unit 11. The control unit 11 then detects the selection of the browsing image I1 by the signal indicating that the input browsing image I1 has been selected, assigns image identification information to the image data of the restaurant captured by the camera unit 13, and writes the image data and the time of capture corresponding to the image identification information into the storage unit 15 to store the image data (step S3).

Furthermore, when the browsing icon I1 is touched, the control unit 11 writes the position information obtained by the GPS 14 and the direction information obtained by the direction sensor 16 as related information into the storage unit 15 corresponding to the image recognition information and stores the information (step S4).

Next, after writing the image data and the related information to the storage unit 15, the control unit 11 transmits the camera identification information (user identification information assigned to the user or identification information assigned to the digital camera 1) for identifying the digital camera 1, and a search request signal including the position information and the orientation information to the information search system 2 via the transceiver unit 12 (step S5). Alternatively, in steps S3 through S5, the control unit 11 may transmit the search request signal to the information search system 2 via the transceiver unit 12 without writing the image data and the related information to the storage unit 15.

Next, in the information retrieval system 2, when the information retrieval server 21 receives the search request signal from the digital camera 1, it extracts the target structure from the map data in the database 22 based on the location and orientation information included in the search request signal. The information retrieval server 21 then uses this structure identification information to transmit the structure identification information from the structure table (Figure 2) to the digital camera 1 as search result information (step S7). At this point, for example, the information retrieval server 21 reads the network address of the digital camera 1 that was attached to the search request signal sent from the digital camera 1 via the transceiver 12, and transmits the search result information to this address.

Furthermore, in step S7, the information retrieval server 21 may extract the search target structure from the map data in the database 22 based on the position information and orientation information contained in the search request signal, only when it detects that the camera identification information contained in the search request signal is already registered in the user registration table stored in the database 22. In this case, if the information retrieval server 21 detects that the camera identification information contained in the search request signal is not registered in the user registration table, for example, it may simply transmit information requiring user registration to the digital camera 1 based on the camera identification information.

Next, in the digital camera 1, after the control unit 11 receives the search result information from the information search system 2 via the transceiver unit 12, the display unit 17 displays the building information in the information display bar SP as shown in FIG5 (step S8). For example, if the building is a restaurant, information about the restaurant's type (Chinese, Japanese, French, Italian, etc.), the restaurant's phone number, address, etc., will be displayed. The information display bar SP shown in FIG5 is merely an example of this embodiment; a portion of the information display bar SP may be displayed superimposed on the image of the restaurant. Furthermore, the information display bar SP is part of the image display bar MP and is displayed later by touching and selecting, for example, the browsing image I1 described above or the commercial image I2 described later.

Furthermore, when the user touches the CM image I2 , the peripheral circuit outputs a signal indicating that the CM image I2 has been selected to the control unit 11 .

Thus, upon receiving a signal indicating that the commercial image I2 has been selected, the control unit 11 detects the selection of the commercial image I2 and displays the restaurant advertising information (menu, number of seats, in-store images, chef's recommended dishes, etc.) included in the search result information on the information display section SP via the display unit 17. Furthermore, if the advertising information includes image data such as in-store images, the control unit 11 displays the image data on the information display section SP.

Furthermore, when the user touches the posted browsing image I3 , the peripheral circuit outputs a signal indicating that the posted browsing image I3 has been selected to the control unit 11 .

In response to the input of the signal indicating that the posted viewing image I3 has been selected, the control unit 11 detects the selection of the posted viewing image I3 and displays the posted information (comments) written by other users included in the search result information in the information display field SP via the display unit 17. Furthermore, if there are multiple pieces of posted information, the control unit 11 displays the multiple pieces of posted information in the information display field SP, for example, in the order of the date they were posted.

When the posted information also includes image data, such as photos of dishes or image data of the interior of a restaurant taken by other users, the control unit 11 displays the image data in the information display field SP via the display unit 17 .

Next, in FIG5 , when the control unit 11 detects a signal indicating a touch by sliding rightward from the left end of the information display bar SP, for example, the display of the information display bar SP is terminated, and the image of the restaurant is displayed on the image display bar MP via the display unit 17, as shown in FIG4 . Alternatively, when the control unit 11 detects a signal indicating selection of the posted view image I3 while the posted information is being displayed in the information review bar SP, for example, the display of the information display bar SP or the display of the posted information may be terminated.

Furthermore, the control unit 11 may display the advertisement information together with the building information when displaying the advertisement information (or contribution information) after displaying the building information in the information display field SP, or may display the advertisement information after deleting the building information from the information display field SP.

Furthermore, when the control unit 11 detects a signal indicating that the in-store image displayed in the information display field SP has been touched, it displays an enlarged or reduced image of the in-store image in the information display field SP (or the image display field MP).

Furthermore, when the user touches the mail icon I4 , the peripheral circuit outputs a signal indicating that the mail icon I4 has been selected to the control unit 11 .

In response to the input of a signal indicating that the email image I4 has been selected, the control unit 11 detects the selection of the email image I4 and displays an input field IP as shown in FIG6 on a portion of the image display field MP via the display unit 17. The input field IP comprises a text field T for the user to input information, etc., and a touch screen keyboard portion K. This input field IP is used by the user to input posting information, a store name, a year, etc., as described later.

The user uses the touch-screen keyboard K to enter information such as the dishes they have eaten, the restaurant atmosphere, or the staff's service quality in the text field T of the input field IP. The user then touches the email icon I4 again. The control unit 11 detects that the email icon I4 has been selected and transmits a search request signal containing the text information in the text field T and the image data displayed in the image display field MP, along with the camera identification information and the building identification information, as the posted information to the information retrieval system 2 via the transceiver unit 12.

Next, upon receiving the search request signal from the digital camera 1, the information search server 21 writes the new submission information corresponding to the structure identification information into the submission column of the structure table of the database 22. Furthermore, before writing the submission information written into the submission writing column (text column T) into the submission column of the structure table of the database 22, the information search server 21 may also write the submission information into the submission column of the structure table of the database 22 only when it detects the presence of the camera identification information included in the search request signal or detects that the camera identification information has been registered in the user registration table of the database 22.

Next, the control unit 11 detects whether the end icon E has been selected (step S9 ). Here, when the user touches the end icon E, the peripheral circuit outputs a signal indicating that the end icon E has been selected to the control unit 11 .

Next, when the control unit 11 detects that the end icon E has been selected by receiving a signal indicating that the end icon E has been selected, the control unit 11 ends the information search process.

On the other hand, when the control unit 11 does not receive the signal indicating that the end image E has been selected and does not detect the selection of the end image E, the control unit 11 returns the process to step S1 and continues the photographing process and the information retrieval process of the digital camera 1.

Furthermore, when the browsing image I1 is selected, the control unit 11 may include the captured image data in the search request signal sent to the information search system 2. Subsequently, the information search server 21 may use the building identification information extracted from the map data to compare the surrounding image data of the building information retrieved from the building table with the transmitted image data through image recognition, extract feature quantities from the image data, and compare them with the feature quantities of the stored surrounding image data to detect whether the similarity value is greater than a set value, thereby determining whether the image corresponds to the captured building.

At this time, when it is determined that the building in the surrounding image data is not the same as the building in the image data as a comparison result, the information retrieval server 21 can also extract the next closest building from the map data using the position information and orientation information, compare the feature quantities again, and evaluate the similarity between the surrounding image data of the newly extracted building and the image data being captured.

Furthermore, the browsing image I1 of this embodiment can also be provided as a browsing button (not shown) on the photographing device 1, similar to the shooting button B1 described above. In this case, the function of the browsing button is the same as that of the browsing image I1. Furthermore, the shooting button B1 of this embodiment can also function as a browsing button by operating a switching unit (not shown). In this case, the browsing image I1 or the browsing button is unnecessary.

<Information Retrieval of Images Shot with Digital Camera 1>

Next, the operation of this embodiment will be described using Figures 1 and 7, as well as Figures 4 to 6. Figure 7 is a flowchart showing an example of the information retrieval process of this embodiment, in which a building is photographed by the digital camera 1, image data stored in the storage unit 15 is read, and information retrieval is performed on the image data.

The following description will describe an example of a situation in which a user is wandering around downtown Shinjuku, searching for a restaurant to eat dinner at, and then captures image data of several restaurants and searches for the restaurant to dine at.

While wandering around a downtown area, the user points digital camera 1 toward a restaurant with a desired appearance (step S11). Upon discovering a restaurant with the appearance of which they wish to search for information, they press the camera button B1 (shutter) on digital camera 1 (step S12). The control unit 11 then displays the image data being captured by the camera unit 13 on the image display field MP of the display unit 17, as shown in FIG4 .

Next, when the user presses the capture button B1, the peripheral circuit outputs a signal to the control unit 11 indicating that the capture button has been pressed.

Next, the control unit 11 detects that the photographing button B1 has been pressed by receiving a signal indicating that the photographing button B1 has been pressed, and assigns image identification information to the image data of the above-mentioned restaurant photographed by the photographing unit 13, and writes the image data and the photographing time into the storage unit 15 corresponding to the image identification information and stores them (step S13).

Next, when the capture button B1 is pressed, the control unit 11 writes and stores the position information obtained by the GPS 14 and the direction information obtained by the direction sensor 16 in the storage unit 15 in association with the image recognition information (step S14 ).

Next, when the user wishes to collect information about the restaurant they have photographed, they touch the read image R of the captured image. This causes the peripheral circuit to output a signal to the control unit 11 indicating that the read image R has been selected. The control unit 11 then determines whether the image is stored in the storage unit 15 based on whether the signal indicating the selection of the read image R has been input (step S15).

At this time, when a signal indicating that the read image R has been selected is output, the control unit 11 detects that the read image R has been selected and begins searching for information, thereby advancing the process to step S16. Alternatively, the read image R may be provided as a read button (not shown). In this case, the read button may be provided on the digital camera 1 body, similar to the shooting button B1.

On the other hand, when a signal indicating that the read image R has been selected is not input or when no image is stored in the storage unit 15 , the control unit 11 returns the process to step S11 because new image processing is to be performed.

Next, when the user touches the read image R and a signal indicating that the read image R has been selected is input from the peripheral circuit, the control unit 11 reads out the image data in sequence from the storage unit 15 in the order of time series shooting (step S16), and displays the image data as photo images in the image display bar MP through the display unit 17 according to a pre-set number of frames (or the image data is displayed on a frame-by-frame basis).

Furthermore, when it is difficult to enter a single page of photo images (or when displaying them individually), if the user touches the image displayed in the image display field MP by sliding it in a predetermined direction, the photo images of the previous or next page are displayed in the image display field MP. In this case, the control unit 11 detects a signal indicating that the image displayed in the image display field MP has been touched by sliding it in a predetermined direction, and displays the photo images of the previous or next page in the image display field MP via the display unit 17.

Next, when the user selects an image of a restaurant of interest from the photo images by touching, and then touches the browsing icon I1 , the peripheral circuit outputs a selected image recognition signal and a signal indicating that the browsing icon I1 has been selected to the control unit 11 .

Thus, when the control unit 11 is input with a signal indicating that the browse image I1 has been pressed, it reads the position information and orientation information corresponding to the image identification information of the selected image data from the storage unit 15, and sends the retrieval request signal containing the camera identification information, position information and orientation information to the information retrieval system 2 through the transceiver unit 12 (step S17).

The following steps S6 to S8 are similar to those in FIG3 , and therefore their description is omitted. Next, the control unit 11 detects whether the end image E has been selected (step S18). At this point, if the user touches the end image E and a signal indicating the selection of the end image E is input from the peripheral circuit, the control unit 11 terminates the browsing process. On the other hand, if the control unit 11 does not receive a signal indicating the selection of the end image E, the process returns to step S16 and continues the image selection process from the photo images.

<Information Retrieval from Store Information Input to Digital Camera 1>

Next, the operation of this embodiment will be described using Figures 1 and 8, as well as Figures 4 to 6. Figure 8 is a flowchart showing an example of the information retrieval process of this embodiment, when a user inputs a store name into the digital camera 1 and attempts to retrieve information about restaurants, etc. corresponding to the store name from the building table of Figure 2 stored in the database 22.

In the following description, for example, a user wants to check information about a restaurant he or she has heard about in Harajuku for dinner. When the user touches and selects the search icon I5, the peripheral circuit outputs a signal indicating that the search icon I5 has been selected to the control unit 11.

Thus, the control unit 11 detects the selection of the search image I5 by receiving the signal indicating the selection of the search image I5 and displays the input field IP shown in FIG. 6 on a part of the image display field MP via the display unit 17 .

Next, the user writes the store name of the restaurant to be searched in the text field T of the input field IP using the touch screen keyboard K (step S21 ), and the user touches the search icon I5 again (step S22 ).

As a result, the peripheral circuit outputs a signal indicating that the search image I5 has been selected, along with the text data of the store name entered in the text field T, to the control unit 11. Next, the control unit 11 detects the selection of the search image I5 by receiving the signal indicating that the search image I5 has been selected, reads the store name data entered in the text field T, and transmits this search request signal, along with its own camera identification information, to the information search system 2 via the transceiver unit 12 (step S23).

Next, when the server 21 receives the search request signal, it reads information on the building (store) corresponding to the store name (store information, information in the table of FIG. 2 ) from the building table in the database 22 (step S24 ).

After reading the store information, the server 21 transmits the acquired store information to the digital camera 1 (step S25 ).

After receiving the store information via the transceiver 12, the control unit 11 displays the image data of the store's surrounding area, as included in the building information, in the image display field MP of FIG5 , and displays the building information in the information display field SP (step S26). For example, the display may include information such as the restaurant's type (Chinese, Japanese, French, Italian, etc.) and its phone number.

When the user selects the CM image I2 by touching, the control unit 11 detects the selection of the CM image I2 and displays the restaurant advertising information (menu, chef's recommended dishes, etc.) included in the search result information in the information display column SP via the display unit 17 .

Furthermore, when the user selects the posting and browsing image I3 by touching it, the peripheral circuit outputs a signal indicating that the posting and browsing image I3 has been selected to the control unit 11 .

When a signal indicating that the posted browsing image I3 has been selected is input, the control unit 11 detects the selection of the posted browsing image I3 and displays the posted information written by other users included in the search result information in the information display field SP via the display unit 17 .

When there are multiple submissions, the control unit 11 sequentially displays the submissions in the information display field SP via the display unit 17. Furthermore, when the submissions also include image data, the control unit 11 displays the image data in the information display field SP via the display unit 17.

Next, the control unit 11 detects whether the end image E has been selected (step S27). This detection of the selection of the end image E is the same as the process of step S9 (in FIG. 3 ) in <Information Retrieval of Images Captured by Digital Camera 1>, so its description is omitted.

At this time, if the control unit 11 receives a signal from the peripheral circuit indicating that the user has selected the end icon E by touching the end icon E, it ends the browsing process. On the other hand, if the control unit 11 does not receive a signal indicating that the end icon E has been selected, it returns to step S21 and continues searching for store information.

<Charge Processing for Stores Registered in Database 22>

The building table of the database 22 may also store discount coupon data for each store when the building is a store (including a restaurant or a company).

Next, the server 21 adds the discount coupon information to the search result information and sends it to the digital camera 1 .

When a user uses the above-mentioned discount coupon information to dine or shop, the information retrieval server 21 detects whether the user has dined or shopped based on whether the discount coupon information has been used. For example, when the above-mentioned discount coupon information is displayed as a barcode (including a two-dimensional barcode) in the information display field SP of the digital camera 1, a reader at the store reads the barcode and transmits the camera identification information transmitted from the digital camera 1, the purchase (or meal) fee, and usage information indicating that the discount coupon has been used to the information retrieval system 2.

Thus, upon receiving the above-mentioned usage information, information retrieval server 21 reads the history information corresponding to the camera identification information attached to the usage information from database 22. The user is charged a different information provision fee for each store used, depending on the stage of information usage (whether the store was used for information retrieval, information provided by the store was used for viewing information provided by the store, or information submitted for viewing information). Furthermore, information retrieval server 21 writes the stages of information retrieval performed based on each camera identification information and the history of discount coupon information used into a history table stored in database 22.

The barcode includes the building identification information indicating the store. A charging table is provided in the database 22, which stores the charging history and the accumulated value of the charging corresponding to each store corresponding to the building identification information.

Next, the information search server 21 writes the charge corresponding to the amount used and the stage of use of the information into the charge table as a history corresponding to the building identification information, and adds the new charge to the accumulated value to update the accumulated value.

<Additional Processing of User Points>

The information retrieval server 21 accumulates the points corresponding to the usage amount (for example, calculated by multiplying the usage amount by the point coefficient) each time the user uses the discount coupon information in a store, and the calculated points are corresponding to the camera identification information in the user login table of the database 22.

Furthermore, the information search server 21 accumulates points of a predetermined value for the user who posted the information in association with the camera identification information in the user registration table. The points can also be used in conjunction with discount coupon information in payment instead of money.

<Display order of submission information>

Here, the information search server 21 may also initially transmit the nickname and title as posting information, along with posting identification information, to the digital camera 1. Subsequently, the control unit 11 initially displays only a plurality of nicknames and titles in the information display field SP. The user then touches and selects the nickname and title of the posting information of interest from the displayed plurality. The peripheral circuit then transmits a signal indicating the selection of the nickname and title, along with the posting identification information of the selected posting information, to the control unit 11.

Thereby, the control unit 11 transmits the submission identification information and the submission information transmission request to the information search system 2 .

The information search server 21 then transmits the text data and image data (entire posted information) corresponding to the sent posting identification information to the digital camera 1. The information search server 21 assigns posting identification information for identification to each posted information, writes it into the user registration table in association with the camera identification information, and stores it.

Furthermore, in this embodiment, after receiving text data and image data from the information retrieval system 2 via the communication transceiver 12 , the control unit 11 may display the text data in the information display field SP and the image data in the image display field MP.

Furthermore, the information search server 21 searches the user registration table for the posting identification information corresponding to the referenced posting information, and increases the number of references to the camera identification information corresponding to the posting identification information.

Next, at the initial display stage when the user selects the nickname and title of the submitted information, the information retrieval server 21 will send it to the digital camera 1 in the order of the number of times the submitted information has been selected, that is, the nicknames and titles of users with more reference times will be displayed in the information display bar SP with the nicknames and titles ranked at the top.

Thereby, the control unit 11 displays the nickname and the title in order in the information display field SP according to the display table.

<Display Processing of Past Image Data>

Furthermore, in this embodiment, the database 22 may also have a past image table that stores image data of buildings and landscapes at various latitudes and longitudes captured in each year in correspondence with each longitude and latitude.

When the user selects the past image image I6 by touching, the peripheral circuit outputs a signal indicating that the past image image I6 has been selected to the control unit 11 .

Thus, the control unit 11 detects the selection of the past video image I6 by receiving the signal indicating the selection of the past video image I6 and displays the input field IP shown in FIG. 6 on a part of the video display field MP via the display unit 17 .

Next, the user uses the touch screen keyboard K to enter the year (eg, AD) in the text field T and then touches the past image icon I6 (or the browsing icon I1).

The peripheral circuit thereby transmits a signal indicating that the past image I6 has been selected and the text data of the year to the control unit 11. When the control unit 11 detects the signal indicating that the past image I6 has been selected, it reads the text data (displaying the year written in the text field T).

After reading the annual data, the control unit 11 sends a past image search request together with the read year, position information, direction information, and camera identification information to the information search system 2 .

Next, in the information retrieval system 2, the information retrieval server 21 selects a past image table corresponding to the latitude and longitude based on the building's location information (received from the digital camera 1 displayed in the image display field MP). The information retrieval server 21 then reads the image data corresponding to the orientation information and year from the selected past image table and transmits the read image data to the digital camera 1. The digital camera 1 displays the image data corresponding to the year received from the information retrieval server 21 on the image display field MP via the display unit 17. Furthermore, if the information retrieval server 21 has additional information for the read image data, it transmits the read image data and the information to the digital camera 1.

This allows users to obtain information on what kind of buildings there were in each year, or to obtain only the scenery without other images.

Here, when the information search server 21 cannot search the past image table corresponding to the latitude and longitude, it searches the past image table of the latitude and longitude closest to the direction of the azimuth information and extracts image data from the table.

Furthermore, if the year of the search object is an era without photos, the image data of the photos can be replaced by landscape paintings depicting that era or image data of CG (Computer Graphics) created by imagination, and stored in the above-mentioned past image table of the storage unit 15 corresponding to the year.

<AR (Augmented Reality) Information Display Processing>

As described above, in the configuration of FIG1 , the control unit 11 adds image identification information to the image data of the subject captured by the photographic unit 13, using the latitude and longitude information (position information) indicating the location of the digital camera 1 obtained from the GPS 14, the azimuth information indicating the direction of the optical axis of the digital camera 1 obtained from the azimuth sensor 16 (azimuth detection unit), and the like, and writes the information into the storage unit 15 in the order of the photographs.

1, a retention control unit 31 and an AR information storage unit 30 are newly added as functions for acquiring and storing AR information as shown in FIG9. FIG9 is a block diagram showing an example of the configuration of an information acquisition system having the function of acquiring and storing AR information.

The AR information in this embodiment displays various information (information related to the subject) as shown in the building table shown in Figure 2. For example, AR information includes building identification information for identifying the building, the building name (i.e., building name), building information (such as address, phone number, type, and surrounding image data centered on the building), location information such as the building's latitude and longitude, a description of the building (information recorded by the building if it is a store), and posted information (such as reviews and user-submitted image data). Here, the building identification information may also include a URL displaying a location where the building information is stored.

The control unit 11 may be configured to allow the user to arbitrarily set whether to display any of the AR information as a mark (augmented reality object) superimposed on image data of a through image (an image after capturing a subject).

Here, the so-called through image refers to the image formed on the imaging element by the imaging unit 13 being continuously output to the control unit 11 as image data, and the control unit 11 sequentially displays the image data on the display unit 17.

The retention control unit 31 detects an action requesting the retention of AR information. In other words, the retention control unit 31 detects the retention point at which the image data, which is imaged by the imaging unit 13 on a device such as a CCD and displayed on the display unit 17, and the AR information contained in this image data (within the field of view of the image) are stored in the AR information storage unit 30. Alternatively, the AR information storage unit 30 may be provided within the information retrieval system 2.

Here, the control unit 11 stores a table showing correspondence between focal length, magnification, etc. and angle of view in the storage unit 15 in advance, and obtains the angle of view information of the image from the focal length and magnification when acquiring the image data.

FIG10 shows a composite image (AR image) on the display unit 17 of real images (through images) of buildings T1 to T4 and images of markers TB1, TB2, TB3, and TB4, which are augmented reality objects that record AR information about each of the buildings T1 to T4 (e.g., markers that record augmented reality building information).

Furthermore, the digital camera 1 in this embodiment has an AR information acquisition mode for acquiring AR information or a normal photography mode. The control unit 11 controls the camera 1 to either the AR information acquisition mode or the normal photography mode by pressing a button for acquiring AR information (AR information acquisition button ARB1) provided on the housing of the digital camera 1 or by touching an image for acquiring AR information (AR information acquisition image ARI1) displayed on the display unit 17. The control unit 11 detects the button press by detecting a signal from a switch, or by detecting the touch of the image on the display unit 17 using a touch sensor.

Furthermore, the digital camera 1 of this embodiment has a mode (AR display mode) in which AR information is displayed as a symbol (an augmented reality object that combines image data and AR information) or a mode (AR non-display mode) in which AR information is not displayed (only image data is displayed). The control unit 11 generates a signal from a switch when a display/non-display button (AR switching button ARB2) provided on the housing of the digital camera 1 is pressed, or when a display/non-display image (AR switching image ARI2) displayed on the display unit 17 is touched. The control unit 11 detects the button press or image touch via a sensor and controls whether to display or non-display the image.

Here, a transparent touch sensor is provided on the display element of display unit 17. When an image is touched, the touch sensor is used to detect the coordinates of the touched area and transmit them to control unit 11 as a detection signal. Control unit 11 aligns the coordinates of the touch sensor with the coordinates of the displayed display element and uses the coordinates of the detection signal to determine the image being displayed (e.g., whether the image displayed on display unit 17 indicates the launch of an application, or to identify image data displayed as a thumbnail image).

Whenever the azimuth angle (orientation information) of the digital camera 1 changes, the control unit 11 transmits the latitude and longitude information (position information), orientation information, and viewing angle information following the captured image data displayed on the display unit 17, along with an AR information acquisition request, to the information retrieval system 2 via the transceiver unit 12. Furthermore, the control unit 11 may acquire AR information by transmitting the position information and the AR information acquisition request to the information retrieval system 2 periodically or upon user instruction. Furthermore, the control unit 11 may acquire AR information by transmitting the image data held by the control unit 31 and the AR information acquisition request to the information retrieval system 2 periodically or upon user instruction.

Here, the control unit 11 detects the azimuth angle at a predetermined period with respect to the change in the azimuth angle. When the azimuth angle changes by more than a preset angle from the azimuth angle stored in the azimuth angle storage unit of the storage unit 15, it is detected as a change in the azimuth angle, and the azimuth angle at this point is newly stored in the azimuth angle storage unit.

After receiving the AR information acquisition request from the digital camera 1, the information retrieval server 21 calculates the retrieval distance from the latitude and longitude coordinate position displayed by the calculated latitude and longitude information to the latitude and longitude coordinate position of the retrieved AR information based on the latitude and longitude information, azimuth information and viewing angle information attached to the AR information acquisition request.

Furthermore, the information search server 21 reads the search distance corresponding to the view angle information from the search distance table of the database 22 which records the view angle information and the search distance corresponding to the view angle, thereby obtaining the search distance of the AR information.

FIG. 12 is a diagram showing a search range for searching AR information of a building.

Next, the information search server 21 calculates the search range shown in FIG12 based on the coordinate position based on the latitude and longitude information, the azimuth of the optical axis of the lens of the digital camera 1, and the search distance, and reads the building identification information within the search range from the map data in the database 22.

Furthermore, the information search server 21 reads AR information of the building from the building table in the database 22 using the read building identification information.

The information search server 21 transmits the read AR information together with the corresponding building recognition information to the digital camera 1 .

In this way, the control unit 11 combines the coordinate values based on the latitude and longitude information of the digital camera 1 with the coordinate values of the latitude and longitude information indicated by the latitude and longitude information contained in the AR information, and superimposes the image of the symbol (augmented reality object) of this AR information on the image data (through the image), and displays it on the display unit 17 in the state shown in Figure 10.

Furthermore, after receiving the holding control signal from the holding control unit 31, the control unit 11 processes the latitude and longitude information, azimuth information, and viewing angle information at the time of holding, the AR information sent from the information retrieval system 2 due to the AR information acquisition request, and the building identification information of the building that identifies the AR information, and processes the image identification information of the image data at the time of holding recorded in the AR information storage unit 30.

Fig. 11 is a diagram showing the structure of the AR information table stored in the AR information storage unit 30 shown in Fig. 9. The AR information table is stored in the AR information storage unit 30 for each piece of image recognition information.

The control unit 11 writes and stores the building identification information (the URL of the location where the AR information of the building is stored) generated for each image recognition information as described above, the AR information of the building (including the latitude and longitude information), the latitude and longitude information of the digital camera 1 itself when the hold control signal is input and the digital camera 1 is held, the direction information, and the viewing angle information into the AR information storage unit 30 as an AR information table.

Furthermore, when the hold control signal is input, the control unit 11 writes and stores the image data formed on the imaging element of the imaging unit 13 at that time in the storage unit 15 in association with the image identification information assigned to the image data.

Here, as an action for indicating the input of maintaining AR information, the maintenance control unit 31 has, for example, a detection means (detecting a signal from a switch generated by pressing the AR information acquisition button ARB1 provided on the housing of the digital camera 1), or a detection means for detecting by a touch sensor that the image displayed on the display unit 17 (AR information acquisition image ARI1) has been touched.

Next, when the AR information acquisition button is pressed or the image displaying the AR information acquisition is touched, the hold control unit 31 detects that a signal for recording image data and AR information has been input, and outputs a hold control signal to the control unit 11 for performing hold control. In this way, the hold control unit 31 detects that a signal for recording image data and AR information has been input based on the user's action (movement, operation, or motion) on the digital camera 1, and outputs a hold control signal to the control unit 11 for performing hold control.

In addition, the holding control unit 31, for example, as an action of holding AR information, can also detect through the acceleration sensor that the digital camera 1 is moved rapidly (such as pointing the digital camera 1 downward or waving it to make the picture easier to see), and use this detection time point as the holding time point.

During acceleration detection using the acceleration sensor, upon receiving acceleration data exceeding a predetermined threshold from the acceleration sensor, the hold control unit 31 outputs a hold control signal to the control unit 11 as a control signal instructing a hold. Thus, the hold control unit 31 detects, through the operation of the terminal (e.g., the digital camera 1), that a signal for recording image data and AR information has been input, and outputs a hold control signal to the control unit 11 for performing hold control.

In addition, the maintaining control unit 31, for example, as an action of maintaining AR information, can also detect by moving the detection image feature point within a predetermined detection range and by a predetermined distance of the feature point, or by detecting the change of the image pattern in the image data when the contrast of the image data output by the photographic element exceeds a predetermined difference, and use this detection time point as the maintaining time point.

In the detection of the change in the image pattern, the hold control unit 31 outputs a hold control signal to the control unit 11 as a control signal for performing hold control after detecting a change in the image pattern that is equal to or greater than a preset threshold.

In addition, the holding control unit 31, for example, as an action of holding AR information, can also use an elevation sensor to detect a sudden change in the angle of the digital camera 1 (by moving the digital camera 1 downward, waving it, etc. to make the picture easier to see), and use this detection time point as the holding time point.

In detecting elevation information using the elevation sensor, the hold control unit 31 outputs a hold control signal to the control unit 11 as a control signal for performing hold control after receiving an elevation change exceeding a preset threshold from the elevation sensor. In this manner, the hold control unit 31 detects, through the operation of the terminal (e.g., the digital camera 1), that a signal for recording image data and AR information has been input, and outputs a hold control signal to the control unit 11 for performing hold control.

Furthermore, when the control unit 11 instructs the use of the above-mentioned acceleration sensor, the change of the image pattern, and the maintenance of the elevation sensor, the AR information corresponding to the image data when the maintenance control signal is received is obtained. Therefore, each time the AR information is obtained, the image data required and the building identification information corresponding to the image data, as well as the AR information, latitude and longitude information, azimuth information, and viewing angle information of the building must be stored in the buffer of the AR information storage unit 30 within a predetermined time, for example, 1 second.

Therefore, the control unit 11 stores and caches the image data corresponding to the AR information acquisition request, the building identification information corresponding to the image data, and the AR information, latitude and longitude information, direction information, and viewing angle information of the building in the buffer unit of the AR information storage unit 30 within a predetermined time, for example, 1 second.

Next, after receiving the hold control signal, the control unit 11 receives an AR information acquisition request immediately before the hold control signal or before a preset time, and writes and stores the image data obtained from the information retrieval system 2 and stored in the buffer unit for each AR information acquisition request corresponding to the image recognition information, the building recognition information corresponding to the image data, and the AR information, latitude and longitude information, azimuth information and viewing angle information of the building as an AR information table with time information attached to it into the AR information storage unit 30.

Furthermore, the control unit 11 writes the image data together with the image identification information into the storage unit 15 and stores them.

Furthermore, the control unit 11 and the retention control unit 31 may be configured to perform the acquisition of AR information or the retention process of AR information even when the non-display mode in which the AR display is non-displayed is set.

Through the above-described processing, when displaying AR information for a retained image, the control unit 11 compares the time information in the AR information table stored in the AR information storage unit 30 with the time of the internal clock based on an instruction signal indicating the pressing of the AR display confirmation button (AR switching button ARB2) (provided on the housing of the digital camera 1) or the touching of the AR display confirmation image (AR switching button ARI2) (displayed on the display unit 17). The control unit 11 selects and reads the latest AR information table from the AR information storage unit 15, and displays the image data along with the AR information symbol on the display unit 17, similar to when a signal is transmitted from the information retrieval system 2. Furthermore, the control unit 11 may periodically obtain the latest AR information table from the information retrieval server 21, or may obtain the latest AR information table from the information retrieval server 21 in response to the instruction signal. Furthermore, when in AR information acquisition mode, the control unit 11 may also display the AR information for the retained image by receiving a hold control signal for retaining the image.

Furthermore, the control unit 11 may be configured to read out image data corresponding to the image identification information stored in the AR information storage unit 30 from the storage unit 15 and display a thumbnail image of the read-out image data on the display unit 17. The control unit 11 may also be configured to select image data by a user touching the thumbnail image displayed on the display unit 17. The control unit 11 reads out an AR information table of the image identification information from the AR information storage unit 30 based on the image identification information associated with the selected thumbnail image, and displays the table on the display unit 17.

At this time, as described above, the user can select display/non-display of the AR information.

Here, the control unit 11 only needs to store all the building identification information for identifying the building, the name of the building, i.e., the building name, building information, location information such as the latitude and longitude of the building, the description of the building, and the contribution information recorded in the building table as AR information in the storage unit 15. Even in a place where there is no wireless communication with the wireless base station 3, that is, in an offline situation, the control unit 11 can also add the AR information mark to the image data stored in the AR information storage unit 30 as shown in Figure 1 and display it on the display unit 17.

Furthermore, as long as it is not used offline, the control unit 11 can also be configured to first store the latitude and longitude information, azimuth information and viewing angle information of the digital camera 1 required for AR information acquisition as an AR information table, and then make an AR information acquisition request to the information retrieval system again when selecting image data.

With the above-mentioned configuration, even if the user does not point the shooting direction of the digital camera 1 toward the direction in which the AR information is to be obtained, for example, even when the digital camera 1 is placed on a table and the screen of the display unit 17 is viewed, the user can view the image data in the direction in which the AR information is to be obtained and confirm the AR information.

Furthermore, when the AR information acquisition mode is entered, the control unit 11 receives a hold control signal from the hold control unit 31 and displays a user ID input field together with images of buttons showing alphabetical letters and numbers to urge user ID input.

When the user touches the image of the key displaying English letters and numbers, the control unit 11 detects the corresponding text data and writes it into the internal buffer, and displays the text data in the internal buffer in the user ID writing field.

Next, the control unit 11 detects that the enter key located in the image of the keys displaying English letters and numbers has been touched, thereby fixing the character string in the internal buffer as the user ID.

The control unit 11 adds the user ID stored in the internal buffer to the AR information table and stores the added user ID.

When the user selects image data in order to view AR information, a display prompting the user to input a user ID is displayed along with an image of buttons showing alphabetical letters and numbers, and a user ID input field is displayed.

By detecting that the image of the key displaying English letters and numbers has been touched, the control unit 11 detects the corresponding text data and writes it into the internal buffer, and displays the text data in the internal buffer in the user ID writing field.

Next, the control unit 11 detects that the input key located on the image of the key displaying English letters and numbers has been touched, and compares the user ID of the character string corresponding to the internal buffer and the user ID attached to the AR information table corresponding to the image recognition information of the selected image data.

When the user ID input by the user is equal to the user ID attached to the AR information table, the control unit 11 displays an image (AR image) synthesized by the image data and AR information on the display unit 17. On the other hand, when the user ID is different from the user ID attached to the AR information table, only the image data is displayed.

With this configuration, the digital camera 1 can identify the user (or user ID) based on each image and display AR information.

Furthermore, when the control unit 11 wants to display the AR information logged in by itself or the AR information obtained by itself instead of displaying the AR information of other users logged in to the information retrieval system 2 or the AR information obtained by other users, the control unit 11 stores the users using the digital camera 1 as a user table in the storage unit 15 or the database 22 in advance, such as an address book. After the image to be referenced by the AR information (AR switching image ARI2) is touched, the control unit 11 displays the list of user names (or the list of user IDs) obtained from the user table on the display unit 17.

Next, the control unit 11 may be configured so that, when a user touches their own user ID from the list, the control unit 11 reads the AR information table to which the user with the same user ID as the touched user is attached from the AR information storage unit 30, reads image data having the same image identification information as the image identification information in the AR information table from the storage unit 15, and displays the data as a thumbnail image on the display unit 17. Furthermore, the control unit 11 may be configured so that, while the list is displayed, upon receiving a signal indicating that another user has been selected, the control unit 11 reads the AR information table to which the user with the same user ID as the other user is attached from the AR information storage unit 30, reads the data as the same image identification information as the image identification information in the AR information table from the storage unit 15, and displays the data as a thumbnail image on the display unit 17.

Furthermore, the control unit 11 may store the AR information table of each image recognition information in the AR information storage unit 30, not only the AR information received from the information retrieval system 2 but also the parameters associated with the AR information or the shooting date (time information) as AR associated information, and store them in the AR information table in correspondence with each building recognition information.

In this way, for example, when the user displays image data, AR information and AR-related information from 10 years ago on the display unit 17, after pressing the AR date retrieval image (AR date retrieval image ARI3) displayed on the display unit 17, the control unit 11 detects through the sensor that the AR date retrieval image has been pressed, and displays the input field for entering the date together with the image of the button displaying English letters and numbers on the display unit 17.

Next, when the user inputs a date into the input field using an image of buttons displaying English letters and numbers, the control unit 11 searches the AR information storage unit 30 for an AR information table with the same date as the input date and extracts the date.

Next, the control unit 11 reads the image identification information corresponding to the extracted AR information table, reads the image data of the image identification information from the storage unit 15 , and displays the image data on the display unit 17 as a thumbnail image.

When the user touches the thumbnail image displayed on the display unit 17 , the control unit 11 detects that the thumbnail image has been touched via the sensor of the display unit 17 and detects the image identification information of the touched and selected image data.

Next, the control unit 11 reads out the video data corresponding to the video identification information of the selected video data from the storage unit 15 and displays the video data on the display unit 17 .

Furthermore, the control unit 11 reads the AR information table corresponding to the image recognition information, and displays a synthesized image (AR image) on the display unit 17 by superimposing the AR information and the AR-related information on the previously displayed image data.

With the above configuration, the user can view the image data from ten years ago and the AR information of the building located in the image data from ten years ago through the display unit 17 .

In addition, by touching the current AR information display image (AR switch button image ARI2) displayed on the display unit 17, the control unit 11 detects that the current AR information display image has been touched through the detection signal of the touch sensor, and extracts the AR information table of the image recognition information of this image data from the AR information storage unit 30.

Next, the control unit 11 adds the latitude and longitude information, azimuth information, and viewing angle information obtained from the AR information table during the period from now to the past (in the case of this embodiment, 10 years), and issues an AR information search request to the information search system 2. Alternatively, the control unit 11 may obtain all AR information during the period from now to the past (in the case of this embodiment, 10 years) from the information search system 2.

Upon receiving the AR information retrieval request from the digital camera 1 , the information retrieval server 21 extracts building identification information of buildings within the search range requested by the latitude and longitude information, direction information, and viewing angle information added to the AR information acquisition request from the map data in the database 22 .

Next, the information search server 21 reads AR information of the building corresponding to the building identification information from the building table of the database 22 and transmits the AR information to the digital camera 1 .

Thus, the control unit 11 superimposes AR information of a currently located building on past image data and displays the same.

This structure makes it easy to check the changes of buildings in the past and present.

Furthermore, when AR information of the subject at the time of photography (e.g., 10 years ago) is stored in the database 22 of the information retrieval system 2 in chronological order, the control unit 11 can also obtain the AR information of the time information based on current or past time information (e.g., current, 10 years ago, etc.) and display it on the display unit 17. The user can view the AR information corresponding to the image data in chronological order from the present to the past.

Furthermore, in the configuration described above where a user ID is set for each piece of image identification information in the image data, it is possible to limit the display of AR information to only a number of friends (users) by creating a configuration that allows multiple user IDs to be added. Alternatively, a configuration can be used where a user ID is set for each piece of building identification information and AR information for each building is managed. In this case, a mode for writing a user ID is set, and the control unit 11 prompts the user to enter an ID for each building.

Furthermore, when no user ID is set, the display is public, that is, the viewers (users) are not restricted and any person (user) can view it.

The control unit 11 displays a field for inputting a user ID and an image of buttons displaying alphabetical letters and numbers on the display unit 17 only when a user ID is added to the AR information table corresponding to the selected image data.

As described above, there are the following five types of combinations for displaying AR information using user IDs.

(1) Synthetic image of current (or past) image data and AR information of buildings within the current (or past) image data

A user ID is added, and the display is limited to the user

(2) Synthetic image of current (or past) image data and AR information of buildings within the current (or past) image data

Open display that can be displayed by any user without a user ID

(3) Synthetic image of AR information of current buildings located in the search range of current (or past) image data and current (or past) image data

A user ID is added, and the display is limited to the user

(4) Synthetic image of AR information of current buildings located in the search range of current (or past) image data and current (or past) image data

Open display that can be displayed by any user without a user ID

(5) Display only current (or past) image data (AR information is not displayed)

Furthermore, the control unit 11 can switch the display of the AR information using the user ID in response to a signal based on the selection of the AR user switching image ARI4.

Next, the operation of the information retrieval system of FIG9 according to another embodiment of the present invention will be described using FIG13. FIG13 is a flowchart showing an example of the operation of the information retrieval system according to another embodiment of the present invention.

When the user touches the image in the AR information acquisition mode (AR information acquisition image ARI1 ) displayed on the display unit 17 , the control unit 11 detects that the image in the AR information acquisition mode has been touched and sets the digital camera 1 to the AR information acquisition mode.

The control unit 11 displays the image data continuously supplied from the imaging unit 13 on the display unit 17 (step S31 ).

Next, the control unit 11 reads latitude and longitude information (position information) indicating coordinate values with latitude accuracy and azimuth information indicating the azimuth angle of the optical axis of the digital camera 1 from the GPS 14 and the azimuth sensor 16 (step S32 ).

At this time, the control unit 11 reads the perspective information from the table of corresponding perspective information for the combination of focal length and magnification stored in the storage unit 15 based on the focal length and magnification, and obtains the perspective information in the digital camera 1 when requesting the information retrieval system 2 to obtain AR information.

After obtaining the latitude and longitude information, the direction information and the viewing angle information, the control unit 11 adds the camera identification information, latitude and longitude information, the direction information and the viewing angle information of the digital camera 1 to the AR information acquisition request and sends it to the information retrieval system 2 (step S33).

The information retrieval server 21 calculates a search range for buildings based on the latitude and longitude information, azimuth information, and viewing angle information provided by the digital camera 1, retrieves buildings within the search range from the map data in the database 22, and uses the building identification information of the buildings retrieved within the search range to read AR information from the building table (step S34).

Next, the information search server 21 transmits the read AR information together with the building identification information of the AR information to the digital camera 1 corresponding to the camera identification information.

Next, the control unit 11 calculates the coordinate position of each building in the image data display coordinates on the display unit 17 based on its own latitude and longitude information, azimuth information, viewing angle information, and retrieval distance (sent from the information retrieval system 2 attached to the AR information) and the latitude and longitude information of the building contained in the supplied AR information. After displaying the image data, the control unit 11 displays the AR information of each building in the image data as a mark (step S35).

Next, the control unit 11 determines whether the azimuth angle provided by the azimuth sensor 16 has changed by a predetermined angle (distance) (step S36). Specifically, the control unit 11 calculates the difference between the azimuth angle provided by the azimuth sensor 16 and the azimuth angle stored in the azimuth angle storage unit, and compares this difference with a preset angle (angle threshold).

At this time, when the difference has changed by exceeding (or above) the predetermined angle (angle threshold), the control unit 11 determines that the azimuth angle has changed, and returns the processing to step S31. On the other hand, when the difference has not changed by exceeding (or above) the predetermined angle (angle threshold), the control unit 11 determines that the azimuth angle has not changed, and proceeds to step S37.

Next, the control unit 11 determines whether or not a hold control signal has been supplied from the hold control unit 31 (step S37 ).

At this time, the control unit 11 advances the process to step S38 when the hold control signal is input, and returns the process to step S36 when the hold control signal is not input.

Here, the hold control unit 31 compares the acceleration data from the acceleration sensor with a preset acceleration data threshold, for example. When the acceleration data supplied from the acceleration sensor exceeds (or exceeds) the acceleration data threshold, the hold control unit 31 generates a hold control signal and transmits it to the control unit 11. On the other hand, when the acceleration data supplied from the acceleration sensor is below (or less than) the acceleration data threshold, the hold control unit 31 does not transmit the hold control signal to the control unit 11.

Next, the control unit 11 writes and stores the latitude and longitude information (position information) obtained from the GPS 14 and the azimuth sensor 16, which respectively indicate coordinate values with latitude accuracy, and the azimuth information indicating the azimuth angle of the optical axis of the digital camera 1, into the storage unit 15 (step S38). The control unit 11 then proceeds to step S36 (return).

In the AR information acquisition mode, the flowchart of FIG. 13 is repeated regardless of whether the AR information is displayed or not.

As described above, according to this embodiment, the photographing direction of the digital camera 1 can be changed, that is, the photographing direction can be changed to a direction different from the direction in which the AR information has been obtained, so that the display unit 17 is moved to a position that is easiest for the user to view, and the AR information can be confirmed in the image data when the AR information has been obtained.

For example, AR information for buildings located in image data (e.g., image data from Tokyo Station's North Exit) can be saved, allowing the AR information for buildings in the image data from Tokyo Station's North Exit to be confirmed on a restaurant table. Alternatively, the user can save AR information for buildings pointing north from their current location and, after saving, point the digital camera 1 in a direction other than north (e.g., toward the ground or east) to confirm the AR information for buildings in the saved image data. Furthermore, the digital camera 1 of this embodiment can perform the above-described processing even when offline and not communicating with the wireless base station 3.

Next, the guidance function of the digital camera 1 will be described. The user has entered the AR information acquisition mode, and the image data and AR information are displayed as icons. The same process can be performed in the AR information confirmation mode after the AR information is stored.

After the guide image displayed on the display unit 17 is touched, the control unit 11 detects that the guide image has been touched via the touch sensor and activates the guide program (guidance unit).

The control unit 11 displays a message urging the user to touch (touch the symbol of the building for which guidance is required) on the display unit 17 .

When the user touches the building sign for which guidance is required, the control unit 11 detects building identification information corresponding to the touched building sign and reads an AR information table corresponding to the image identification information of the displayed image data from the AR information storage unit 30 .

Next, the control unit 11 reads the latitude and longitude information of the building from the AR information corresponding to the detected building identification information based on this AR information table, adds the latitude and longitude information of the current position of the digital camera 1 and the latitude and longitude information of the building, and sends it to the information retrieval system 2 as a guidance request.

Upon receiving the guidance request, the information retrieval server 21 retrieves the shortest path (or paths) between the latitude and longitude information of the attached digital camera 1 and the latitude and longitude information of the building from the map data stored in the database 22 .

That is, the information search server 21 extracts from the map data the intersection closest to the coordinates of the digital camera 1 indicated by the latitude and longitude information, and uses this as the starting intersection.

Similarly, the information search server 21 extracts from the map data the intersection closest to the coordinate value where the building indicated by the latitude and longitude information is located, and uses this as the ending intersection.

Next, the information retrieval server 21 generates a combination of multiple road routes connecting the starting intersection to the ending intersection based on the road network of the map data, and selects the path with the smallest distance for each combination. For example, a formula in graph theory that can efficiently solve known shortest path problems, such as Dijkstra's method for extracting the shortest path, is used to extract the combination of road routes with the smallest total distance.

Next, the information retrieval server 21 transmits image data of a map showing a road path from the starting intersection to the ending intersection to the digital camera 1 that has requested guidance. Here, the information retrieval server 21 generates the map image data so that the display color of the guidance path (the road path connecting the starting intersection to the ending intersection) in the map image data is different from that of other roads.

The control unit 11 changes the received map image data to a size that can be displayed on its own display unit 17 , and displays the changed map image data on the display unit 17 .

Next, the control unit 11 superimposes a marker indicating the user's location on the image data of the map displayed on the display unit 17 at a position corresponding to the latitude and longitude position obtained by the GPS 14. This allows the user to receive guidance from the starting intersection to the ending intersection while confirming the movement of the marker.

Furthermore, the digital camera 1 can transmit image data and AR information of a building in the image data to other digital cameras of other users, thereby sharing information with other users.

Furthermore, the image data and the AR information of the building within the image data can be uploaded from the digital camera 1 to a website such as an SNS (Social Networking Service), thereby sharing the information with multiple other digital cameras. In this case, by adding a user ID to the AR information table, only users with the recognized user ID can view or add the AR information corresponding to the image data on the SNS.

Here, the SNS server has an AR information shared database storing image data and AR information.

The user can add user information to the image data that the user wants to register and the AR information corresponding to the image data through the digital camera 1 and register them.

Here, when the user touches the user login image displayed on the display unit 17, the control unit 11 detects the touch of the image via the touch sensor and displays a login screen on the display unit 17. This login screen also includes a thumbnail image of the image data corresponding to the AR information obtained by the user and an image of buttons displaying English letters and numbers.

In this login screen, the user selects the image data to be logged in from the thumbnail images, enters the user ID of the image data and the user's own name (or nickname) and touches the input image, whereby the control unit 11 accesses the SNS server and sends the image data and AR information together with the login request.

After receiving the login request, the SNS server generates a file with the user's name (or nickname), writes the requested image data and AR information into the AR information shared database, and stores it. This file has the same structure as the AR information table in Figure 11.

Next, when the user touches the AR information reference image displayed on the display unit 17 of the digital camera 1, the control unit 11 detects the touch of the image through the touch sensor and displays the reference screen on the display unit 17. This reference screen also includes images of buttons showing English letters and numbers.

Secondly, in this login screen, the user enters the user name (or nickname) and user ID to be referenced by touching the image of buttons displaying English letters and numbers and touches the input image, whereby the control unit 11 accesses the SNS server and sends this name (or nickname) and user ID together with the image data and AR information reference request.

After receiving the reference request, the SNS server searches the AR information sharing database using the name (or nickname) attached to the reference request, and extracts files with the same name (or nickname).

Next, the SNS server reads the image identification information to which the user ID identical to the received user ID is added and the image data corresponding to the image identification information from the AR information in the file, and transmits the read AR information and image data to the digital camera 1 .

The control unit 11 displays the input image data and AR information on the display unit 17 as described above.

Next, when the user touches the AR information change image displayed on the display unit 17 of the digital camera 1, the control unit 11 detects the touch of the image through the touch sensor and displays the change screen on the display unit 17. This login screen also includes a thumbnail image of the image data corresponding to the read AR information, an information recording field, and an image of buttons displaying English letters and numbers.

Next, in this change screen, after the user selects the image data of the AR information to be changed from the thumbnail images, the control unit 11 detects the selected image data through the touch sensor and displays the selected image data on the display unit 17. The AR information mark corresponding to the position of each building is superimposed on the displayed image data.

Next, the user touches the icon of the AR information to be changed, and enters the added or edited text string and user ID in the information writing column by touching the image of the button displaying English letters and numbers, and touches the input key (input image). The control unit 11 then uses the text string entered in the information writing column as the new AR information of the building, and sends the image recognition information and building recognition information together with the changed AR information to the SNS server as a change request including the user name and user ID.

After receiving the change request, the SNS server searches the AR information sharing database using the name (or nickname) attached to the change request, and extracts files with the same name (or nickname).

Next, the SNS server determines whether the user ID added to the image identification information matches the user ID added to the received change request. If they do not match, the SNS server does nothing and notifies the digital camera 1 that the change cannot be made.

On the other hand, when the user ID added to the AR information matches the user ID added to the received change request, the SNS server changes the AR information of the building identification information in the image identification information.

In addition, the program for implementing the AR information holding function of the control unit 11 and the holding control unit 31 of the digital camera 1 in Figure 9 can be recorded on a computer-readable recording medium, and the program recorded on this recording medium can be read into the computer system and executed to control the AR information.

Alternatively, a program for implementing the AR information retrieval function of the information retrieval server 21 in FIG. 9 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform AR information retrieval control.

The term "computer system" herein includes hardware such as an OS and peripheral devices.

Furthermore, the term "computer system" also includes a computer system having a web page providing environment (or display environment) as long as it utilizes the WWW system.

Furthermore, "computer-readable recording medium" refers to removable media such as floppy disks, magnetic optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes media that dynamically retain programs for short periods of time, such as when transmitting programs via networks like the Internet or communication lines like telephone lines, or media that retain programs for a certain period of time, such as in volatile memory (RAM) within a server or client computer system. Furthermore, the aforementioned program may be used to implement a portion of the aforementioned functions, and further, the aforementioned functions may be implemented by combining them with programs already stored in a computer system.

Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the scope of the present invention.

Claims (62)

1. A photographic device, characterized in that it comprises: ontology; Photography Department; First storage unit; Second storage unit; The computer is configured to display the subject image obtained by the aforementioned camera unit and the subject information obtained based on the position information obtained by the position information acquisition unit or the result of image recognition of the aforementioned subject image on the display unit of the aforementioned camera device. Based on the user's action, a control signal is output, wherein the user's action includes at least one of the following actions: detecting that a feature point of the subject image moves a predetermined distance within a predetermined time; detecting that the contrast of the image data of the subject image changes due to a predetermined difference; and detecting a change in an image pattern of the subject exceeding a threshold value. Before the aforementioned user action or before a predetermined time, the computer maintains a composite image of the subject image stored in the first storage unit and the subject information stored in the second storage unit on the display unit, and continues to display the maintained composite image on the display unit even after the user action ends. The operation unit is used to publish submission information during the display of the aforementioned composite image that has been held. 2. The photographic apparatus as claimed in claim 1, wherein the aforementioned computer is capable of switching between displaying and not displaying the aforementioned subject information on the aforementioned display unit. 3. The photographic apparatus as claimed in claim 1 or 2, wherein the aforementioned computer transmits the held subject image and the held subject information to other terminals according to the aforementioned holding control signal. 4. The photographic apparatus according to any one of claims 1 to 3, characterized in that the aforementioned display unit has a touch panel that can be touched and operated during the display of the aforementioned held subject image and the aforementioned held subject information. 5. The photographic apparatus according to any one of claims 1 to 4, characterized in that the aforementioned operation unit has an input unit for inputting the aforementioned submission information during the display of the aforementioned held subject image and the aforementioned held subject information. 6. The photographic apparatus according to any one of claims 1 to 5, wherein at least a portion of the aforementioned operation unit is disposed on the aforementioned display unit. 7. The photographic apparatus according to any one of claims 1 to 6, characterized in that the aforementioned display unit has an icon related to the publication operation of the aforementioned submission information during the display of the aforementioned held subject image and the aforementioned held subject information. 8. The photographic apparatus according to any one of claims 1 to 7, wherein the aforementioned subject information includes coupon information; The aforementioned computer maintains the aforementioned coupon information on the aforementioned display unit according to the aforementioned holding control signal. 9. The photographic apparatus according to any one of claims 1 to 8, wherein the aforementioned user action includes at least one of the following actions: pressing a button provided on the aforementioned body, touching the aforementioned display unit, moving the aforementioned body, and changing the posture of the aforementioned body. 10. The photographic apparatus according to any one of claims 1 to 9, wherein the aforementioned computer, upon detecting the aforementioned holding control signal, immediately sends the aforementioned held subject information to the outside. 11. The photographic apparatus according to any one of claims 1 to 10, characterized in that, after the aforementioned holding, the aforementioned computer further acquires the aforementioned subject information by touching the aforementioned display unit. 12. The photographic apparatus according to any one of claims 1 to 11, characterized in that, after the aforementioned holding, the aforementioned computer obtains second subject information different from the aforementioned subject information by touching the aforementioned display unit. 13. The photographic apparatus according to any one of claims 1 to 12, wherein the aforementioned subject information includes coupon information. 14. The photographic apparatus according to any one of claims 1 to 13, wherein the subject information includes an icon that is displayed over the subject image and continues to be displayed on the display unit after the user's action ends. 15. The photographic apparatus according to any one of claims 1 to 14, wherein the aforementioned subject information includes past information. 16. An information acquisition system, characterized in that it comprises: Photographic devices and information retrieval systems; The aforementioned photographic apparatus is the photographic apparatus of any one of claims 1 to 15. 17. A recording medium readable by a computer, storing a program that enables the computer to perform the functions of a photographic device, characterized in that: The aforementioned computer performs the following steps: The step of obtaining subject information based on the obtained location information or the result of image recognition of the subject image obtained by the photographic unit of the aforementioned photographic device; The step of displaying the aforementioned subject image and the aforementioned subject information on the display unit using AR; The step of caching the aforementioned subject image and the aforementioned subject information; According to the aforementioned step of outputting a hold control signal based on user actions during AR display, the aforementioned user actions during AR display include at least one of the following actions: detecting that feature points of the aforementioned subject image move a predetermined distance within a predetermined time period; detecting that the contrast of the image data of the aforementioned subject image changes due to a predetermined difference; and detecting changes in image patterns above a threshold value in the aforementioned subject. The aforementioned hold control signal is used to hold the composite image of the aforementioned subject image and the aforementioned subject information, which has been cached before the aforementioned user action or before a predetermined time, on the aforementioned display unit, and to continue displaying the aforementioned held composite image on the aforementioned display unit even after the aforementioned user action ends; and The step of publishing submission information during the display of the aforementioned composite image that has been held. 18. A photographic apparatus, characterized in that it comprises: ontology; Photography Department; An accelerometer is installed on the aforementioned body; First storage unit; Second storage unit; and computer; The device is configured to display the subject image obtained by the aforementioned camera unit and the subject information obtained by the position information acquisition unit or the result of image recognition of the aforementioned subject image on the display unit of the aforementioned camera device. Based on the motion output control signal of the aforementioned body accompanying the user's action detected by the aforementioned accelerometer, the aforementioned computer maintains the composite image of the aforementioned subject image stored in the aforementioned first storage unit and the aforementioned subject information stored in the aforementioned second storage unit on the aforementioned display unit before the aforementioned user action or before a preset time, and also maintains the aforementioned maintained composite image on the aforementioned display unit after the aforementioned user action ends. 19. The photographic apparatus as claimed in claim 18, wherein the aforementioned computer is capable of switching between displaying and not displaying the aforementioned subject information on the aforementioned display unit. 20. The photographic apparatus of claim 18 or 19, wherein the aforementioned computer transmits the held subject image and the held subject information to other terminals according to the aforementioned holding control signal. 21. The photographic apparatus according to any one of claims 18 to 20, characterized in that the aforementioned display unit has a touch panel that can be touched and operated during the display of the aforementioned held subject image and the aforementioned held subject information. 22. The photographic apparatus according to any one of claims 18 to 21, characterized in that it includes an input unit for inputting the aforementioned submission information during the display of the aforementioned held subject image and the aforementioned held subject information. 23. The photographic apparatus according to any one of claims 18 to 22, characterized in that it comprises: a storage unit for storing the aforementioned retained subject image and the aforementioned retained subject information after establishing a correspondence. 24. The photographic apparatus according to any one of claims 18 to 23, characterized in that the aforementioned display unit has a mode for displaying an icon related to the publication operation of the aforementioned submission information during the display of the aforementioned held subject image and the aforementioned held subject information. 25. The photographic apparatus according to any one of claims 18 to 24, wherein the aforementioned subject information includes coupon information; The aforementioned computer maintains the aforementioned coupon information on the aforementioned display unit according to the aforementioned holding control signal. 26. The photographic apparatus according to any one of claims 18 to 25, wherein the aforementioned user action includes at least one of the following actions: pressing a button provided on the aforementioned body, touching the aforementioned display unit, moving the aforementioned body, changing the posture of the aforementioned body, detecting that feature points of an image move a predetermined distance within a predetermined time, detecting that the contrast of image data changes when it exceeds a predetermined difference, and detecting that the image pattern changes when it exceeds a threshold value. 27. The photographic apparatus according to any one of claims 18 to 26, wherein the aforementioned computer, upon detecting the aforementioned holding control signal, sends the aforementioned held subject information to the outside. 28. The photographic apparatus according to any one of claims 18 to 27, wherein the computer further acquires the subject information by touching the display unit after the aforementioned holding. 29. The photographic apparatus according to any one of claims 18 to 28, wherein the aforementioned subject information includes coupon information. 30. An information acquisition system, characterized in that it comprises: Photographic devices and information retrieval systems; The aforementioned photographic apparatus is the photographic apparatus of any one of claims 18 to 29. 31. A recording medium readable by a computer, storing a program that enables the computer to perform the functions of a photographic device, characterized in that: The aforementioned computer performs the following steps: The step of obtaining subject information based on the obtained location information or the result of image recognition of the aforementioned subject image; The step of displaying the subject image and subject information obtained by the camera unit of the aforementioned camera device on the display unit using AR; The step of caching the aforementioned subject image and the aforementioned subject information; The step of maintaining the motion output control signal of the aforementioned camera device according to the user's movement detected by the accelerometer during the aforementioned AR display is to maintain the composite image of the subject image and the subject information that was cached a moment before the aforementioned user movement or a preset time on the aforementioned display unit, and to continue to display the aforementioned maintained composite image on the aforementioned display unit after the aforementioned user movement ends. 32. A server, characterized in that: It can receive and transmit signals with the photographic device of any one of claims 1 to 15 and 18 to 29, obtain the aforementioned subject information from the database, and send the aforementioned subject information to the aforementioned photographic device. 33. A photographic apparatus, characterized in that it comprises: Photography department, responsible for photographing subjects; The location information acquisition unit acquires the location information of the photography location; Storage Department; AR Information Storage Department; The control unit, based on a control signal instructing the aforementioned subject to be photographed, causes the aforementioned camera unit to photograph the aforementioned subject, and obtains information related to the aforementioned subject based on the aforementioned position information, and displays the image data of the aforementioned subject and the information related to the aforementioned subject on the display unit; and The holding control unit outputs a holding control signal, which is different from the aforementioned control signal, to the aforementioned control unit based on the user's actions when displaying the image data of the aforementioned subject and information related to the aforementioned subject on the aforementioned display unit. The aforementioned hold control signal is a signal that, before the aforementioned user action or before a preset time, the aforementioned control unit holds the image data of the aforementioned subject stored in the aforementioned storage unit and the composite image of the aforementioned subject related information stored in the aforementioned AR information storage unit, and after outputting the aforementioned hold control signal according to the aforementioned user action, the aforementioned held composite image is also displayed on the aforementioned display unit. The aforementioned user actions include at least one of the following actions: detecting that the feature points of the aforementioned subject image move a predetermined distance within a predetermined time period, detecting that the contrast of the image data of the aforementioned subject image changes when it exceeds a predetermined difference, and detecting changes in the image pattern of the aforementioned subject that exceed a threshold value. 34. The photographic apparatus as claimed in claim 33, wherein the aforementioned control unit is capable of switching between displaying and not displaying information related to the subject on the aforementioned display unit. 35. The photographic apparatus as claimed in claim 33 or 34, wherein the aforementioned control unit transmits information related to the subject held by the aforementioned holding control signal and image data of the subject to other terminals. 36. The photographic apparatus according to any one of claims 33 to 35, wherein the information associated with the subject is AR information. 37. The photographic apparatus according to any one of claims 33 to 36, characterized in that, when the aforementioned control unit has received the aforementioned holding control signal, and when the photographing direction is not directed toward the direction in which information related to the aforementioned subject is to be obtained, the aforementioned held image data of the aforementioned subject and information related to the aforementioned subject are displayed on the aforementioned display unit. 38. The photographic apparatus according to any one of claims 33 to 37, wherein the aforementioned display unit has a touch panel. 39. The photographic apparatus according to any one of claims 33 to 38, wherein the information related to the subject includes coupon information. 40. The photographic apparatus according to any one of claims 33 to 39, characterized in that the aforementioned control unit causes an input field for the user to input submission information to be displayed on the aforementioned display unit. 41. An information acquisition system, characterized in that it comprises: The photographic apparatus of any one of claims 33 to 40; and server. 42. A recording medium readable by a computer, storing a program useful for executing by a computer, characterized in that: The aforementioned computer performs the following steps: The step of causing the photographic unit of the photographic device to photograph the aforementioned subject in accordance with the control signal instructing the photographic subject; The step of inputting the location information of the photographic location of the aforementioned subject; The steps to obtain information related to the aforementioned subject based on the aforementioned location information; The step of displaying the image data of the aforementioned subject and information related to the aforementioned subject on the display unit of the photographic device; The step of caching the image data of the aforementioned subject and information related to the aforementioned subject; Based on user actions when image data of the aforementioned subject and information related to the aforementioned subject are displayed on the aforementioned display unit, a step of outputting a control signal different from the aforementioned control signal is performed. The aforementioned user actions include at least one of the following actions: detecting that feature points of the aforementioned subject image move a predetermined distance within a predetermined time period; detecting that the contrast of the image data of the aforementioned subject image changes due to a predetermined difference; and detecting changes in image patterns in the aforementioned subject that exceed a threshold value. The steps are as follows: holding the composite image of the subject's image data and information related to the subject, which is cached a moment before the user's action or a preset time, on the display unit according to the aforementioned hold control signal, and displaying the held composite image on the display unit after outputting the aforementioned hold control signal according to the user's action. 43. A photographic apparatus, characterized in that it comprises: Photography department, responsible for photographing subjects; The location information acquisition unit acquires the location information of the photography location; Storage Department; AR Information Storage Department; The control unit obtains information related to the subject based on the aforementioned location information, and displays the image data of the subject and the information related to the subject on the display unit; and The holding control unit outputs a holding control signal to the control unit based on the user's actions when displaying the image data of the subject and information related to the subject on the display unit. The user's actions include at least one of the following actions: detecting that the feature points of the subject image move a predetermined distance within a predetermined time; detecting that the contrast of the image data of the subject image changes when it exceeds a predetermined difference; and detecting that the image pattern in the subject changes at a threshold value or above. The aforementioned holding control unit holds the image data of the subject stored in the aforementioned storage unit and the composite image of the subject-related information stored in the aforementioned AR information storage unit on the aforementioned display unit before the aforementioned user's action or before a preset time, and also displays the aforementioned held composite image on the aforementioned display unit after the aforementioned user's action ends. 44. The photographic apparatus as claimed in claim 43, wherein the aforementioned control unit is capable of switching between displaying and not displaying information related to the aforementioned subject on the aforementioned display unit. 45. The photographic apparatus as claimed in claim 43 or 44, wherein the aforementioned control unit, when receiving the aforementioned holding control signal during the photographing of the aforementioned subject, displays the held image data of the aforementioned subject and information related to the aforementioned subject on the aforementioned display unit. 46. The photographic apparatus according to any one of claims 43 to 45, characterized in that the aforementioned control unit transmits information related to the subject held according to the aforementioned holding control signal and image data of the subject to other terminals. 47. The photographic apparatus according to any one of claims 43 to 46, wherein the information related to the subject is AR information. 48. The photographic apparatus according to any one of claims 43 to 47, characterized in that, when the aforementioned control unit has received the aforementioned holding control signal, and when the photographing direction is not oriented toward the direction in which information related to the aforementioned subject is to be obtained, the aforementioned control unit displays the held image data of the aforementioned subject and the information related to the aforementioned subject on the aforementioned display unit. 49. The photographic apparatus according to any one of claims 43 to 48, characterized in that the aforementioned display unit has a touch panel that can be touched and operated while displaying the held image data of the aforementioned subject and information related to the aforementioned subject. 50. The photographic apparatus according to any one of claims 43 to 49, characterized in that it further comprises: an operation unit for publishing the held image data of the subject and/or submission information related to the subject. 51. The photographic apparatus according to any one of claims 43 to 50, characterized in that the aforementioned display unit has an input unit for inputting submission information during the display of the held image data of the aforementioned subject and information related to the aforementioned subject. 52. The photographic apparatus according to any one of claims 43 to 50, characterized in that the aforementioned display unit has a mode for displaying an icon related to the publication operation of the aforementioned submission information while displaying the held image data of the aforementioned subject and information related to the aforementioned subject. 53. The photographic apparatus of claim 1, wherein the aforementioned computer is capable of operating in the following mode: AR display mode displays the aforementioned subject image and the aforementioned subject information on the display unit; and The mode maintains that even after the user's actions have ended and the position and/or posture of the subject changes, the subject image and subject information displayed in the aforementioned AR display mode will continue to be displayed on the aforementioned display unit. 54. The photographic apparatus as claimed in claim 53, characterized in that it comprises: a storage unit that stores the subject image displayed in the aforementioned holding mode after establishing a correspondence with the aforementioned subject information. 55. The photographic apparatus of claim 1, wherein the aforementioned computer is capable of operating in the following mode: AR display mode displays the aforementioned subject image and the aforementioned subject information on the display unit; and The mode remains in effect, meaning that even after the user's actions, the camera unit is no longer facing the subject, but the subject image and subject information displayed in the AR display mode continue to be displayed on the display unit. 56. The photographic apparatus as claimed in claim 55, characterized in that it comprises: a storage unit that stores the subject image displayed in the aforementioned holding mode after establishing a correspondence with the aforementioned subject information. 57. An information acquisition system, characterized in that it comprises: Photographic devices and information retrieval systems; The aforementioned photographic apparatus is the photographic apparatus of any one of claims 43 to 56. 58. The photographic apparatus as claimed in claim 1, characterized in that, The aforementioned subject information includes at least one augmented reality object. 59. The photographic apparatus of claim 58, wherein the control unit displays the image data of the subject and information related to the subject on the display unit in the same manner as at least one of the content of the image data of the subject displayed on the display unit, the number of augmented reality objects displayed on the display unit, and the arrangement of at least one augmented reality object displayed on the display unit during the period when the user performs the action and after the user's action ends. 60. The photographic apparatus as claimed in claim 58 or 59, wherein the aforementioned control unit is capable of switching between displaying and not displaying information related to the aforementioned subject on the aforementioned display unit. 61. The photographic apparatus according to any one of claims 58 to 60, characterized in that the aforementioned control unit transmits information related to the subject held according to the aforementioned holding control signal and image data of the subject to other terminals. 62. An information acquisition system, characterized in that it comprises: Photographic devices and information retrieval systems; The aforementioned photographic apparatus is the photographic apparatus of any one of claims 58 to 61.