JP2019169154A - Terminal device and control method thereof, and program - Google Patents

Abstract

A terminal device capable of displaying a desired character on an image of a real space displayed on a display and changing the display of the desired character according to a change in environment, a control method therefor, and a program I will provide a. A terminal device including an imaging device, wherein the position of the terminal device in a three-dimensional model of a real space is specified, a three-dimensional space image is acquired via the imaging device, and the position of the terminal device in the three-dimensional model is obtained. And the positional relationship between the real objects included in the three-dimensional space image, the environment information of the terminal device is acquired, and the position of the starting point at which the environment information acquired by the second acquisition means is issued in the three-dimensional model On the image displayed on the display, based on the positional relationship between the real object and the terminal device included in the three-dimensional space image, and superimposes and displays a predetermined image to change the acquired environmental information. The display of a predetermined image is changed according to the position where the specified environmental information is issued. [Selection diagram] FIG.

Description

  The present invention relates to a terminal device, a control method thereof, and a program. In particular, the present invention relates to a technique for displaying a predetermined object superimposed on a real space image on a display based on a distance between the real space object and a terminal device.

  2. Description of the Related Art In recent years, when displaying a spatial image in a real space photographed by an imaging means such as a camera on a display, a so-called AR (Augmented Reality) technique for displaying a predetermined object image superimposed on the spatial image is provided. It has become widespread. In addition, with the recent increase in functionality of mobile terminals, the use of AR technology on mobile terminals is increasing.

  Non-Patent Documents 1 and 2 disclose a system in which a sensor and a camera are combined. This system can specify the current position and orientation of the terminal device based on information acquired from the camera and the sensor.

Tango, [online], [Search January 10, 2017], Internet, <URL: https://get.google.com/tango/> The first smartphone “Lenovo PHAB2 Pro” equipped with Google ’s spatial recognition technology “Tango” will have an AR experience like this: [online], [Search January 10, 2017], Internet, <URL: http: //gigazine.net/news/20160630-snapdragon-tango-ar/>

  When a user having a mobile terminal superimposes and displays an image of a real space and a predetermined object image on the display of the mobile terminal, along with the environment that the user is actually experiencing and the environment change, There is a desire to change a predetermined object image being displayed.

  For example, when displaying a user's favorite character as a predetermined object image on the display of the mobile terminal, the character wants to react to the character in accordance with the environmental change around the mobile terminal (that is, the user) There are cases where it is desired to provide predetermined information via the character. The present invention provides a method of dynamically changing an object image superimposed on a real space image on a display of a mobile terminal of the user in accordance with information on a real environment where the user exists and the change thereof. With the goal.

  In order to solve the above problems, the present invention has the following configuration. That is, a terminal device including an imaging device, the first specifying means for specifying the position of the terminal device in a three-dimensional model in real space, and a first one for acquiring a three-dimensional space image via the imaging device. Acquisition means; second specification means for specifying a positional relationship between the position of the terminal device in the three-dimensional model and a real object included in the three-dimensional space image; and second information for acquiring environment information of the terminal device. Acquisition means; in the three-dimensional model, third specification means for specifying the position of the starting point from which the environmental information acquired by the second acquisition means is issued; and on the image displayed on the display The display unit for displaying a predetermined image superimposed on the basis of the positional relationship between the real object included in the three-dimensional space image and the terminal device, and the environment acquired by the second acquisition unit Depending on the location where the environment information identified by distribution changes and the third identification means is issued, and a control means for changing the display of said predetermined image.

  Moreover, it has the following structures as another structure of this invention. That is, a method for controlling a terminal device including an imaging device, the first specifying step of specifying the position of the terminal device in a three-dimensional model of real space, and acquiring a three-dimensional space image via the imaging device A first acquisition step, a second specifying step of specifying a positional relationship between the position of the terminal device in the three-dimensional model and a real object included in the three-dimensional space image, and acquiring environmental information of the terminal device The second acquisition step, the third specification step of specifying the position of the starting point where the environmental information acquired in the second acquisition step is emitted in the three-dimensional model, and the display on the display In the display step of superimposing and displaying a predetermined image based on the positional relationship between the real object included in the three-dimensional space image and the terminal device, and in the second acquisition step Depending on the location where the environment information identified by a specific process variation and the third environmental information is issued, and a control step of changing the display of said predetermined image.

  Moreover, it has the following structures as another structure of this invention. That is, a first program for obtaining a three-dimensional space image via a first specifying means for specifying a position of the computer in a three-dimensional model of a real space, a computer provided with the imaging device. Acquisition means, second specification means for specifying a positional relationship between the position of the computer in the three-dimensional model and a real object included in the three-dimensional space image, second acquisition means for acquiring environment information of the computer, In the three-dimensional model, on the image displayed on the display, the third specifying means for specifying the position of the starting point from which the environmental information acquired by the second acquiring means is emitted. Display means for superimposing and displaying a predetermined image based on the positional relationship between the real object included in the spatial image and the computer; Depending on the location where the environment information identified at the transition and the third specifying means environment information acquired by means emitted, to function as a control means for changing the display of said predetermined image.

  According to the present invention, it is possible to dynamically change the object image to be superimposed and displayed on the real space image on the display of the mobile terminal possessed by the user in accordance with the information of the real environment where the user exists and the change thereof. .

The figure which shows the example of the hardware constitutions of the portable terminal which concerns on this invention. The figure which shows the example of the software structure of the portable terminal which concerns on this invention. The flowchart which shows the processing flow which concerns on this invention. The figure which shows the example of the superimposed display on the display which concerns on this invention.

  Hereinafter, one embodiment according to the present invention will be described with reference to the drawings. In addition, embodiment shown below is an example and is not limited to this. In the following description, a mobile terminal such as a smartphone will be described as an example of the terminal device according to the present embodiment, but the present invention is not limited to this.

[Explanation of terms]
First, terms used in the description of the present invention will be described.

  “Environmental information” is various information related to the surrounding environment of the mobile terminal. For example, temperature, humidity, sound (volume, sound source position), light (light quantity, light source position), time (time), date, longitude / latitude, altitude, and the like are applicable.

  The “operation information” corresponds to the state of each resource included in the mobile terminal, the load, the operation mode, the imaging condition by the imaging unit, or the state of internal processing.

  The “photographed image” is an image (three-dimensional space image) photographed by the photographing unit, and here includes an image displayed in real time on a display.

  The “three-dimensional model” indicates information indicating the characteristics of the three-dimensional space around the mobile terminal. The information acquired by the various sensors includes, for example, a point group (feature point) indicating features of various objects in the image extracted from the image captured by the imaging unit.

  “Depth information” is information indicating the depth (distance) from the origin to the detection target. The origin here is assumed to be provided based on the position of the terminal device.

  The “display object” is an object to be displayed on the display, and examples of the display object include a person, an animal, an object, and an animation. The display object may be a two-dimensional shape or a three-dimensional shape. In the following description, a substance in real space taken through an imaging unit with respect to a display object is simply referred to as “object”.

[Hardware configuration]
FIG. 1 is a diagram illustrating an example of a schematic hardware configuration of a terminal device according to the present embodiment. The terminal device 10 is a portable communication terminal such as a smartphone, for example. The terminal device 10 includes a control unit 100, a speaker 101, a microphone 102, an imaging unit 103, a GPS processing unit 104, a gyro sensor 105, a display 106, a depth sensor 107, a communication unit 108, an acceleration sensor 109, a geomagnetic sensor 110, and a temperature. / Humidity sensor 111 is provided.

  The speaker 101 converts an audio signal output from the control unit 100 into audio. The microphone 102 converts the input sound into a sound signal and outputs the sound signal to the control unit 100. The imaging unit 103 includes a lens and a CCD sensor (not shown), acquires image information, and outputs the image information to the control unit 100. The GPS processing unit 104 receives a signal from a GPS satellite and outputs position information including latitude, longitude, and altitude to the control unit 100 based on the received signal. The gyro sensor 105 measures the angle and the angular velocity of the terminal device 10 and outputs them to the control unit 100 as posture information. In this embodiment, the display 106 is a touch-sensitive display, and functions as an input unit for user operation in addition to presenting information to the user. When the display is not a touch-sensitive display, an input unit is provided separately.

  The depth sensor 107 measures the distance (depth) to the object and outputs the measurement result to the control unit 100 as depth information. For example, an infrared camera can be used as the depth sensor 107. The communication unit 108 performs communication processing via a mobile communication network or a wireless LAN (Local Area Network). The acceleration sensor 109 measures the acceleration of the terminal device 10 and outputs it to the control unit 100 as acceleration information. The geomagnetic sensor 110 detects geomagnetism and outputs azimuth information indicating the current orientation of the reference direction of the terminal to the control unit 100. The temperature / humidity sensor 111 detects the temperature and humidity in the surrounding environment of the terminal device 10 and outputs the detected temperature and humidity to the control unit 100.

  The control unit 100 is a control unit for the terminal device 10 as a whole, and includes one or more processors and a memory unit. The memory unit includes a RAM (Random Access Memory) used for temporary storage of data by the processor, an operating system (OS) and applications executed by the processor, and data to be retained even when the power is turned off. Is stored in the flash memory.

  In addition, the example of the hardware constitutions of said apparatus is an example, You may provide the site | parts other than the above. For example, in order to acquire environmental information and a three-dimensional model around the terminal device 10, a sensor may be provided corresponding to each information.

  In the present embodiment, the three-dimensional model may be a correspondence table that holds a point group of feature points of an image captured by the imaging unit 103 and an imaging position in association with each other. The point cloud is configured so that it can be recognized as the same object even if the imaging location or angle changes slightly. The terminal device 10 reads the feature point information and determines where the imaging position is when the imaging position is unknown. That is, the three-dimensional model is for storing the appearance of the landscape and determining where the imaging position is, and can be generated without using the depth sensor 107. On the other hand, the depth sensor 107 is used when the object stands on the ground or when the size of an actual object is measured.

Software configuration
FIG. 2 is a diagram illustrating an example of a schematic software configuration according to the present embodiment. The functions of the various processing units according to the present embodiment are realized by a processor reading and executing a program stored in a memory unit included in the control unit 100 in a timely manner.

  The terminal device 10 includes a three-dimensional model acquisition unit 201, a captured image acquisition unit 202, an environment information acquisition unit 203, a position information acquisition unit 204, a display control unit 205, an object control unit 206, a positional relationship specifying unit 207, and a voice analysis unit. 208.

  The three-dimensional model acquisition unit 201 acquires a three-dimensional model related to the space around the terminal device 10. The method for acquiring the three-dimensional model is not particularly limited, but as an example, Tango shown in Non-Patent Documents 1 and 2 can be used.

  Here, the three-dimensional model will be described. The three-dimensional model acquisition unit 201 generates location description information based on the technique described in Non-Patent Document 1, for example. The location description information is information for determining the three-dimensional space position of the terminal device 10 and the imaging direction of the imaging unit 103 from the image captured by the imaging unit 103. First, generation of location description information described in Non-Patent Document 2 will be described. Note that the origin of the three-dimensional space position is the position of the terminal device 10 (imaging unit 103) when the location description information is started.

  When the user of the terminal device 10 starts to acquire the location description information, the user changes the position and the imaging direction while capturing an image with the imaging unit 103. The three-dimensional model acquisition unit 201 determines the position of the terminal device 10 and the direction of the imaging direction based on the acceleration information and the angular velocity information acquired by various sensors while acquiring the location description information, using the starting position as the origin. Keep doing. In addition, the three-dimensional model acquisition unit 201 obtains feature points in the image captured by the imaging unit 103. At this time, the three-dimensional model acquisition unit 201 determines the position of the terminal device 10 in the three-dimensional space, the imaging direction, and the position of the same feature point in the image in the camera coordinates. As a result, the 3D model acquisition unit 201 obtains location description information indicating where each feature point can be seen in the image captured by the imaging unit 103 at each position and each imaging direction in the 3D space. Generate. Therefore, the terminal device 10 compares the generated location description information with the feature points of the image captured by the imaging unit 103, so that the 3D spatial position of the terminal device 10 at that time and the imaging of the imaging unit 103 are obtained. The direction can be determined.

  In addition, the 3D model acquisition unit 201 compares the location description information already stored with the feature points of the image captured by the imaging unit 103, and the 3D spatial position of the terminal device and the imaging direction of the imaging unit 103. At the same time, the already-stored location description information is updated based on the feature points of the image captured by the imaging unit 103.

  As described above, the location description information is information indicating the relationship between the position and imaging direction of the terminal device 10 and the position of each feature point of each object in the image captured by the imaging unit 103 in the camera coordinates. Based on the location description information, three-dimensional space information indicating the coordinates of the three-dimensional space position of each feature point can be generated. The position of the feature point is represented by coordinates (x, y, z) in a virtual three-dimensional space that imitates real space.

  The location description information generated here is retained in the storage unit of the terminal device 10 as appropriate. At this time, when there is an already generated 3D model, the newly acquired 3D model may be compared with the already stored 3D model. For example, as a result of the comparison, if feature points of a predetermined ratio or more match, it is recognized that the newly acquired 3D model is acquired at the same location as the already stored 3D model. In this case, the already stored 3D model is updated with the newly acquired 3D model.

  The captured image acquisition unit 202 acquires an image captured via the imaging unit 103. The acquired image is displayed on the display 106 in real time or is transferred to the three-dimensional model acquisition unit 201 described above.

  The environment information acquisition unit 203 acquires temperature information and humidity information in the surrounding environment of the terminal device 10 via the temperature / humidity sensor 111. Furthermore, the environment information acquisition unit 203 analyzes an image captured through the imaging unit 103, and specifies a change in brightness, a light source, and the like in the image. For example, it can determine based on the direction detected by the optical sensor (not shown) with which the terminal device 10 is provided regarding specification of a light source. Furthermore, the environment information acquisition unit 203 acquires various types of audio information analyzed by the audio analysis unit 208. Note that the environment information acquisition unit 208 is not limited to voice such as a person as voice information, and may acquire music, a predetermined sound, or the like. Further, raw voice information before being analyzed by the voice analysis unit 208 may be used as it is as environment information. The environment information acquisition unit 203 may acquire environment information from an external device (not shown) via the communication unit 108. As information acquired from the outside, for example, related information (such as weather information) may be acquired based on audio information.

  The position information acquisition unit 204 specifies the current position of the terminal device 10 in the real space three-dimensional model based on the information acquired by the three-dimensional model acquisition unit 201. The current position is specified by coordinates (x, y, z) in the three-dimensional space. The position information acquisition unit 204 specifies the current position and tilt of the terminal device 10 by motion tracking using the posture information output from the gyro sensor 105 and the acceleration information output from the acceleration sensor 109. For example, the position information acquisition unit 204 first performs matching using feature points obtained as a three-dimensional model, and specifies where the terminal device 10 is in the three-dimensional model. After the identification, the position information acquisition unit 204 can identify the coordinates on the three-dimensional model even if the terminal device 10 is moved by the acceleration sensor 109 and the gyro sensor 105. The position information acquisition unit 204 may correct the accumulation of shifts in motion tracking by area learning based on image information obtained by imaging by the imaging unit 103.

  The display control unit 205 superimposes and displays the image captured via the imaging unit 103 and the display object created by the object control unit 206. The display control unit 205 determines the position of the display object on the display 106 based on position information, a three-dimensional model, environment information, or the like. For example, the display control unit 205 displays the display object on the display 106 based on the correspondence between the coordinates of the object indicated by the three-dimensional model, the position information of the terminal device 10, and the information obtained by the depth sensor 107. Specify the position at. At this time, by using the information obtained from the depth sensor 107, it is possible to control the position and display of the image in consideration of the correspondence with the wall or floor of the real space.

  The object control unit 206 determines a display object to be displayed on the display 106 and its operation, and passes them to the display control unit 205. For example, the action here includes, for example, providing information by voice or displaying it on the display 106 as characters in addition to the action of the display object. In addition, information may be acquired from other applications installed in the terminal device 10 or various types of hardware included in the terminal device 10, and the operation may be determined based on the information.

  Based on the 3D model acquired by the 3D model acquisition unit 201 and the position information acquired by the position information acquisition unit 204, the positional relationship specifying unit 207 applies an image displayed on the display 106 of the terminal device 10. The positional relationship between the various objects included and the terminal device 10 is specified. Based on the positional relationship specified here, the display control unit 205 specifies the position where the predetermined display object is displayed in the image displayed on the display 106.

  The voice analysis unit 208 analyzes the voice input through the microphone 102 and specifies the volume of the sound, the position (sound source), the direction, and the like. As a method of identifying a sound source, for example, there is a method of estimating the position of a sound source based on time differences between the input sounds based on inputs from a plurality of microphones provided in the terminal device. There is also a method for estimating the distance to the sound source using a plurality of microphones, but the method is not particularly limited. The analysis here includes, for example, identification of a voice of a specific user. For example, analysis may be performed so that only a specific sound (voice) is extracted. Various types of audio information analyzed by the audio analysis unit 208 are passed to the environment information acquisition unit 203 as environment information.

  In the configuration illustrated in FIG. 2, the terminal device 10 performs various processes. However, the terminal device 10 may be configured such that after acquiring each information, the data is transmitted to the external device, the processing is performed on the external device side, and the result is acquired.

  For example, the object control unit 206 may download information on a display object associated with a position (coordinates) based on the position of the terminal device 10 specified from the three-dimensional model from a server that is an external device. There are various downloading methods. The user operates the terminal device 10 to select a download icon, and also periodically uploads a captured image or an image of a feature point. The presence or absence may be determined, and the corresponding three-dimensional model may be downloaded. When the GPS information acquired by the GPS processing unit 104 of the terminal device 10 is uploaded to the server, a three-dimensional model corresponding to the location (along with the display of information specifying the corresponding application) is selected and received. May be. Or the terminal device 10 may hold | maintain previously the object information matched with the position (coordinate) based on the position of the terminal device 10 specified from the 3D model memorize | stored and produced | generated and the 3D model.

  Also, the processing modules as shown in FIG. 2 may be integrated and configured, or may be configured as a program that can be further divided and processed by a processor (computer).

[Processing flow]
Hereinafter, the processing flow according to the present embodiment will be described with reference to FIG. In the present embodiment, in the control unit 100, the following processing is realized by the processor reading and executing various programs stored in the memory unit.

  In S <b> 301, the control unit 100 acquires a three-dimensional model around the terminal device 10 using the three-dimensional model acquisition unit 201. At this time, a three-dimensional model created in advance may be acquired, or the imaging unit 103 included in the terminal device 10 may be activated and the three-dimensional model may be configured from the beginning. Good. In addition, you may comprise so that the imaging part 103 may be started with the start of this processing flow.

  In S302, control unit 100 causes position information acquisition unit 204 to specify the current position of terminal device 10 in the three-dimensional model.

  In S303, the control unit 100 uses the positional relationship specifying unit 207 to specify the positional relationship (corresponding relationship) between the positional information of the terminal device 10 and the actual three-dimensional space information (object in the captured image).

  In S304, the control unit 100 determines the display position of the display object desired by the user based on the specified positional relationship by the display control unit 205. The display position determined here may be based on the distance between the position of the terminal device 10 and an object (for example, a wall or a floor) displayed on the screen, or a relative location from the position of the terminal device 10. It may be. Further, as described above, the depth information obtained by the depth sensor 107 can be used as the distance (depth) to the object displayed on the screen.

  In step S <b> 305, the control unit 100 determines a display object using the object control unit 206. Here, the user may select the type of display object in advance, or the display object may be determined based on position information or environment information. Further, the number of display objects is not limited to one, and a plurality of display objects may be displayed simultaneously. In addition, display conditions and priorities based on environment information or the like may be provided for each of the plurality of display objects, and display object candidates may be specified according to the situation. Here, the control part 100 shall acquire and hold | maintain the environmental information and operation | movement information of the terminal device 10 at that time. Furthermore, in determining the display object, the display object may be determined using the environment information and the operation information acquired here.

  In addition, the display object may hold the display settings (parameter settings such as brightness and shadow direction) of the object displayed last time and display it as it is in the next display. Further, during the display of the display object, it is determined whether to change the parameter based on the parameter information of the display object and the newly acquired environment information in the background. The display mode and operation may be changed. At this time, by storing a plurality of display modes and operation patterns of the display object, an object having an appropriate display and operation pattern according to the environment information can be acquired. In addition, although said process was described as a background process in the terminal device 10, this process may be requested | required of the server side and the result may be used. As a result of the determination, new information may be acquired from the server.

  In addition, when the program is started, the location where the terminal device 10 is located is specified indoors / outdoors based on GPS information, etc., and morning / day / night (and season, etc.) is specified based on time information and date information. In addition, a predetermined display pattern may be selected based on any of these pieces of information. As a display pattern, for example, if it is a summer night, the brightness of the night may be controlled with summer clothes, or the setting may be controlled for indoor use. Furthermore, display settings and the like may be corrected based on environmental information in the background during the display. That is, it is possible to control so that information based on display timing is given priority first. With this configuration, the user's stress can be eliminated.

  In S306, the control unit 100 causes the display control unit 205 to display the display object determined in S305 on the display 106 at the display position in the image determined in S304. Whether to display the display object on the display 106 may be determined based on a prior setting (display / non-display at startup) when the application of the terminal device 10 is started. In addition, even when the application is controlled to be hidden when the application is activated, a predetermined operation on the terminal device 10 or voice information input to the terminal device 10 (for example, a call for a name corresponding to a display object) is detected. Accordingly, the display object may be controlled to be displayed on the display 106.

  Further, the display range of the display object may be changed according to the image displayed on the display 106 and the characteristics of the display object. For example, as shown in FIG. 4, it is assumed that the screen displayed on the display 106 includes an area for standing a display object corresponding to the ground or the like. In this case, if the display object is a human character, the display object is displayed so that the display object stands in the area. On the other hand, if there is no such area in the screen (for example, only the wall is displayed), Only the upper body or the part from the neck to the head may be displayed.

  In S <b> 307, the control unit 100 acquires various environment information around the terminal device 10 and operation information of the terminal device 10 by the environment information acquisition unit 203. Further, information on the orientation of the control unit 100 and the imaging unit 103 is acquired. That is, since the image displayed on the display 106 is changed when the orientation of the imaging unit 103 is changed, it means that various information is changed accordingly.

  In S308, the control unit 100 determines whether or not the various information held by the object control unit 206 is different from the various information newly acquired in S307. If they are different (YES in S308), the process proceeds to S309. If they are the same (NO in S308), the process returns to S307.

  In step S309, the control unit 100 causes the object control unit 206 to determine a display object to be newly displayed and a display position based on various information after the change. For example, as a display object to be newly displayed, the size and orientation of the display object displayed so far may be changed, or a completely different display object may be displayed. For example, in the environment information, when the position of the light source or the light amount is changed, the brightness at which the display object is displayed and the direction and size of the shadow may be changed. Further, when audio information is newly input, a position that is a sound source of the audio may be specified, and display of the display object may be changed according to the direction of the position.

  Moreover, when changing, you may make it change continuously and you may make it display switch. Further, as the orientation of the imaging unit 103 is changed, the moving distance (moving amount) of the display object is determined according to the change amount of the orientation, and the movement (for example, a character is walking). May be operated in the manner described above).

  In S310, the display object determined in S309 is displayed at the determined display position on the display 106 by the control unit 100 and the display control unit 205.

  In S311, the control unit 100 determines whether or not to end the display of the predetermined display object. If it is determined that the display is to be ended (YES in S311), the process flow is ended. If it is determined that the display is to be continued (NO in S311), the process returns to S307 and the process is continued. In addition, the case where it complete | finishes corresponds to the case where the instruction | indication from a user is received, the case where the power supply of the terminal device 10 is turned off, etc., but it is not limited to these.

  Note that the past display history (display setting information at the end) may be stored, and the display object may be displayed in the most display manner in the display history at the next display. Alternatively, the display history may be stored for each time zone, and the most frequent display mode may be selected in the next display time zone. Further, the display history may include environment information when the display is being performed. At the next display time, the display mode of the display object may be determined by comparing the display history with the environmental information at that time.

[Display example]
A display example of a display object desired by the user on the display 106 according to the present embodiment will be described. In the following description, a human-type image will be described as an example of a user-desired display object, but the present invention is not limited to this.

  FIG. 4A shows an example in which a display object is displayed superimposed on an image in the real space on the display 106 according to the present embodiment. As shown in the upper part of FIG. 4A, it is assumed that the position of the user is not changed and the direction of the imaging unit 103 is changed to the left and right. In this case, the display object is moved so that the display object is displayed in the image of the three-dimensional space displayed on the display. The display object at this time may be displayed so as to be positioned at the center of the screen, and the display position is adjusted to the front, back, left and right based on the distance (depth) from the object included in the image in the three-dimensional space. You may make it do.

  FIG. 4B shows an example in which a display object is displayed based on a light source or a sound source. For example, an example is shown in which the position of the light source is specified, and the shadow of the display object is displayed from the position of the light source and the position of the display object. Further, based on the input sound information, the position (direction) of the sound source is specified, and the display object performs some operation on the sound source. For example, display is performed so that the display object moves toward the sound source based on the positional relationship between the position of the sound source and the display object.

  Further, the display object may be displayed so as to change the operation according to the content of the audio information. For example, when the audio information is specific music or a predefined rhythm, the character that is the display object may perform a dance or an action in accordance with the music (rhythm). In addition, when the volume indicated by the audio information exceeds a predetermined value, control may be performed so that a character that is a display object is surprised or makes a comment on the sound. When the analyzed voice information is specific information, a display operation associated with the information may be performed. For example, when the name of a character that is a display object is input (called) as voice information, the display object may be displayed so as to react in response to the input. The reaction here may be displayed so as to appear on the screen when, for example, an action of replying is performed or the display object is not displayed on the display. Further, the display object may be displayed so as to perform an action of making the character as a display object angry or sad in accordance with a predetermined wording or voice volume. Furthermore, an operation such as movement and turning may be displayed in response to the sound source (called direction).

  Further, the display object may be displayed so that the hue of the display object is changed based on the specified color of the light source (white, warm color) or the color of the image captured by the photographing unit (luminance, etc.). In addition, information that the display object changes the display setting for the display 106 may be presented in accordance with information such as the color of the light source.

  The environment information may include operation information of the terminal device itself. For example, the operation information includes a photographing operation on the photographing unit of the terminal device. In this case, when the display object is displayed on the display and the shooting operation is performed on the imaging unit (camera) included in the terminal device, the display object may be controlled so that the display object takes a shooting pose. Good. In addition, when the shooting operation is continuously performed, a different operation (shooting pose) may be performed for each shooting operation.

  Further, the environment information may include the current time and the time elapsed without changing the screen displayed on the display, and the display content of the display object may be controlled based on this information. For example, when the current time is midnight, control may be performed so that the character of the display object performs a sleeping action or wears clothes for the night. Further, when it is morning time, a display that performs a time-specific operation such as brushing teeth may be performed. In addition, when the elapsed time without changing the screen displayed on the display exceeds a predetermined threshold value, the display object is controlled not to be displayed, or the user is encouraged to do something. The display of the display object may be controlled.

  Further, the operation and display of the display object may be switched according to the temperature and humidity information included in the environment information. For example, when the temperature exceeds a certain threshold value, display may be performed so that the clothes are thin, or display that presents a message (for example, whether water is being taken) to the user is performed. May be. Further, the position where the user (mobile terminal) is located may be specified according to the position information, and the display corresponding to the position may be performed. For example, the display may be performed so that the national costume of the area and the clothes according to the climate and time of the area.

  As described above, in the present invention, the display object desired by the user is superimposed on the image displayed on the terminal device possessed by the user, and the display of the display object desired by the user is switched according to the environment in which the terminal device is present. Thereby, the user can obtain the display change (reaction by the object) of the object according to the real environment that the user is experiencing via the terminal device. As a result, the user's satisfaction that the user himself and the object (for example, an anime character) are experiencing the same experience in the same environment can be improved, and new entertainment can be provided.

DESCRIPTION OF SYMBOLS 10 ... Terminal device 100 ... Control part 103 ... Imaging part 104 ... GPS processing part 105 ... Gyro sensor 106 ... Display 107 ... Depth sensor 108 ... Communication part 109 ... Acceleration sensor 110 ... Geomagnetic sensor 111 ... temperature / humidity sensor

Claims (13)

A terminal device comprising an imaging device,
First specifying means for specifying a position of the terminal device in a three-dimensional model of real space;
First acquisition means for acquiring a three-dimensional spatial image via the imaging device;
Second specifying means for specifying a positional relationship between the position of the terminal device in the three-dimensional model and a real object included in the three-dimensional space image;
Second acquisition means for acquiring environmental information of the terminal device;
Third specifying means for specifying a position of a starting point from which the environmental information acquired by the second acquiring means is emitted in the three-dimensional model;
Display means for superimposing and displaying a predetermined image on the image displayed on the display, based on the positional relationship between the giz object included in the three-dimensional space image and the terminal device;
Control means for changing the display of the predetermined image in accordance with a change in environmental information acquired by the second acquisition means and a position where the environmental information specified by the third specifying means is emitted; A terminal device comprising: Further comprising third acquisition means for acquiring operation information of the terminal device;
The terminal device according to claim 1, wherein the control unit changes the predetermined image based on the operation information.   In accordance with a change in the image displayed on the display, the control means moves the predetermined image on the image displayed on the display in response to the change. The terminal device according to claim 1 or 2.   The moving distance of the predetermined image on the image displayed on the display is the real object included in the changed three-dimensional spatial image specified by the second specifying means and the three-dimensional of the terminal device. The terminal device according to claim 3, wherein the terminal device is determined based on a positional relationship in the model.   The control means determines the position and size of the predetermined image in the image displayed on the display based on depth information between the actual three-dimensional space and the terminal device. The terminal device according to any one of claims 1 to 4.   The environment information includes at least one of temperature, humidity, sound, light, time, longitude / latitude information in the surrounding environment of the terminal device. The terminal device according to item.   The terminal device according to claim 6, wherein the sound environment information includes a position of a sound source with respect to an image displayed on the display and a sound volume.   The terminal device according to claim 6, wherein the sound environment information is a voice of a user of the terminal device.   The terminal device according to claim 6, wherein the light environment information includes at least one of a position, a light amount, and a color of a light source with respect to the predetermined image displayed by the display unit.   The terminal device according to claim 2, wherein the operation information includes at least one of a position, an orientation, an internal process of the terminal device, and a change with respect to a shooting condition of the imaging device.   The terminal device according to claim 1, wherein the predetermined image includes at least one of a person, an animal, an object, and an animation. A method for controlling a terminal device including an imaging device,
A first specifying step of specifying a position of the terminal device in a three-dimensional model of real space;
A first acquisition step of acquiring a three-dimensional spatial image via the imaging device;
A second specifying step of specifying a positional relationship between the position of the terminal device in the three-dimensional model and a real object included in the three-dimensional space image;
A second acquisition step of acquiring environmental information of the terminal device;
A third specifying step of specifying a position of a starting point from which the environmental information acquired in the second acquiring step is emitted in the three-dimensional model;
On the image displayed on the display, based on the positional relationship between the real object included in the three-dimensional space image and the terminal device, a display step of superimposing and displaying a predetermined image;
A control step of changing the display of the predetermined image in accordance with a change in the environmental information acquired in the second acquisition step and a position where the environmental information specified in the third specification step is emitted; A control method characterized by comprising: A computer comprising an imaging device;
First specifying means for specifying the position of the computer in a three-dimensional model of real space;
First acquisition means for acquiring a three-dimensional spatial image via the imaging device;
Second specifying means for specifying a positional relationship between the position of the computer in the three-dimensional model and a real object included in the three-dimensional space image;
Second acquisition means for acquiring environmental information of the computer;
Third specifying means for specifying the position of the starting point from which the environmental information acquired by the second acquiring means is emitted in the three-dimensional model;
Display means for superimposing and displaying a predetermined image based on the positional relationship between the real object included in the three-dimensional space image and the computer on the image displayed on the display;
Control means for changing the display of the predetermined image in accordance with a change in the environmental information acquired by the second acquisition means and a position where the environmental information specified by the third specification means is emitted. A program to make it work.