WO2025263291A1 - Information processing device, information processing method, and recording medium - Google Patents

Abstract

The present disclosure relates to an information processing device, an information processing method, and a recording medium that make it possible to appropriately display a virtual object superimposed on a real object. Provided is an information processing device comprising a control unit that acquires information related to the states of a real object as a superimposition object, on which a virtual object is to be superimposed, and a second real object different from the real object as the superimposition object, and controls display of the virtual object on the basis of the acquired information. For example, the present disclosure can be applied to an XR HMD or the like.

Description

Information processing device, information processing method, and recording medium

This disclosure relates to an information processing device, an information processing method, and a recording medium, and in particular to an information processing device, an information processing method, and a recording medium that are capable of appropriately displaying virtual objects superimposed on real objects.

Smartphones and XR head-mounted displays (hereafter referred to as XR HMDs) can overlay three-dimensional virtual objects onto the real world. For example, by displaying a product as a virtual object in front of the user's eyes before actually manufacturing or purchasing it, the user can check its size and appearance. Virtual objects are merely virtual objects displayed on a smartphone or XR HMD, and cannot be grasped or moved in the same way as real objects. If virtual objects could be handled like real objects, it would be possible to check the shape and color in more detail during the product manufacturing process or before purchasing, and there is a growing demand for this.

Technologies for overlaying virtual objects on real objects include those disclosed in Patent Documents 1 to 3, for example. Patent Document 1 discloses a technique for blurring the boundaries of a virtual object when overlaying it on a real object. Patent Document 2 discloses a technique for aligning a specified plane of a virtual object so that it is parallel to a plane along the hand or a specified plane of a mockup. Patent Document 3 discloses a technique for overlaying a virtual object of a different size or shape on a real object, and for displaying the virtual object as semi-transparent when the user's hand is not touching the real object.

Japanese Patent Application Laid-Open No. 2009-104249 Japanese Patent Application Laid-Open No. 2016-99839 Japanese Patent Application Publication No. 2018-92313

When superimposing a virtual object on a real object, there are problems such as the user being unable to grasp the desired part of the virtual object due to differences in shape and size between the virtual object and the real object, so there is a need for an appropriate display of the virtual object to be superimposed on the real object.

This disclosure has been made in light of these circumstances, and makes it possible to appropriately display virtual objects superimposed on real objects.

An information processing device according to one aspect of the present disclosure is an information processing device that acquires information about the status of a real object onto which a virtual object is to be superimposed and a second real object different from the real object onto which the virtual object is to be superimposed, and that includes a control unit that controls the display of the virtual object based on the acquired information.

An information processing method according to one aspect of the present disclosure includes an information processing device acquiring information about the status of a real object onto which a virtual object is to be superimposed and a second real object different from the real object onto which the virtual object is to be superimposed, and controlling the display of the virtual object based on the acquired information.

A recording medium according to one aspect of the present disclosure is a computer-readable recording medium having recorded thereon a program for causing a computer to function as a control unit that acquires information regarding the status of a real object onto which a virtual object is to be superimposed and a second real object different from the real object onto which the virtual object is to be superimposed, and controls the display of the virtual object based on the acquired information.

In an information processing device, information processing method, and recording medium according to one aspect of the present disclosure, information is acquired regarding the status of a real object onto which a virtual object is to be superimposed, and a second real object different from the real object onto which the virtual object is to be superimposed, and the display of the virtual object is controlled based on the acquired information.

Note that the information processing device of one aspect of the present disclosure may be an independent device, or may be an internal block constituting a single device.

FIG. 1 is a diagram illustrating a first example of a conventional method. FIG. 10 is a diagram illustrating a second example of a conventional method. FIG. 1 illustrates a first problem of the conventional method. FIG. 10 is a diagram illustrating a second problem of the conventional method. FIG. 10 is a diagram illustrating a third problem of the conventional method. FIG. 10 is a diagram illustrating a fourth problem of the conventional method. 1 is a block diagram illustrating a configuration example of an embodiment of an information processing device to which the present disclosure is applied. A diagram showing an example configuration of an XR HMD to which the present disclosure is applied. FIG. 1 is a diagram illustrating a configuration example of a smartphone to which the present disclosure is applied. FIG. 1 is a diagram illustrating an example of the configuration of a stationary 3D display to which the present disclosure is applied. FIG. 1 is a diagram illustrating an example of the configuration of a photographing camera to which the present disclosure is applied. FIG. 10 is a block diagram illustrating another exemplary configuration of an embodiment of an information processing device to which the present disclosure is applied. 10 is a flowchart showing the flow of processing for displaying a virtual object on a real object to be superimposed. FIG. 10 is a diagram showing an example of a virtual object displayed in a wireframe. FIG. 10 is a diagram illustrating an example of setting a grip origin. 10 is a flowchart showing the flow of a process for setting a tracking target. FIG. 10 is a diagram illustrating an example of visualization of a region to be grasped of a virtual object. 10A and 10B are diagrams illustrating an example of tracking feature points of a real object to be superimposed and a second real object. 10 is a flowchart showing the flow of processing for enlarging, reducing, and rotating a virtual object superimposed on a target real object. FIG. 10 is a diagram illustrating an example of enlarging a virtual object. FIG. 10 is a diagram illustrating an example of rotation of a virtual object. 10 is a flowchart showing a processing flow when a physical object to be superimposed is placed on a plane. FIG. 10 is a diagram showing an example in which a virtual object is rotated so as to be perpendicular to a plane. 10 is a flowchart showing a processing flow when a physical object to be superimposed is changed. 10A and 10B are diagrams illustrating an example of resetting the grip origin when the real object to be superimposed is changed; FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer.

<Conventional method>
Examples of technologies that allow virtual objects displayed on smartphones or XR HMDs to be treated in the same way as real objects include the technologies shown in Figures 1 and 2. XR (Cross Reality) is an abbreviation for Extended Reality/Cross Reality, and is a technology that combines real and virtual spaces to provide new experiences that cannot be perceived in reality. For example, XR includes VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

Figure 1 shows a situation where a virtual object 11 is displayed to a user wearing an XR HMD and a hand gesture 13 is made by the user's hand 12. As shown in Figure 1, by capturing and recognizing hand gesture 13 made by moving hand 12 in the direction of arrow A1 with a camera mounted on the XR HMD, it is possible to virtually grasp virtual object 11 and display it as moving in the direction of arrow A2.

Figure 2 shows how a virtual object 22 is displayed to a user wearing an XR HMD using a physical object such as a marker 21. In this example, the marker 21 is captured and recognized by a camera mounted on the XR HMD, and the virtual object 22 is superimposed according to the position and orientation of the marker 21. In this case, if the user holds the marker 21 in their hand 23 and moves it in the direction of arrow A1, the virtual object 22 can be moved indirectly in conjunction with the movement of the marker 21.

Technology has also been proposed that recognizes real objects in three dimensions from images captured by an XR HMD camera and estimates their position and size. Applying this technology, a virtual object can be superimposed on a camera mockup (a life-size model of a camera that looks exactly like the real thing) of the same shape as the virtual object, allowing the user to grasp the virtual object. However, creating a real object of the same shape each time a virtual object is displayed takes time and money, so it is preferable to use a real object of a similar shape and size so that the user can grasp it without feeling uncomfortable.

The method of indirectly grasping a virtual object by superimposing it on a real object of a different shape or size and then grasping the real object has the following problems:

First, when a virtual object is superimposed on a real object of a different shape or size, there is a problem in that the user may not be able to grasp the desired location with their hand. For example, as shown in Figure 3, consider a case in which virtual object 32 is superimposed on real object 31, which is a rectangular parallelepiped protruding from a plane. In this case, as shown by the tip of arrow A1, when the user tries to grasp virtual object 32 with their hand 33, real object 31 is not present in the desired location, and the user is unable to grasp the desired portion of virtual object 32.

Secondly, when a virtual object superimposed on a real object of a different shape or size can be enlarged, there is a problem in that the virtual object may not be enlarged appropriately. For example, as shown in Figure 4, consider a case in which a user performs hand gesture 45 with their hand 44 while superimposing virtual object 43 on real object 41 on plane 42. In this case, as shown at the tip of arrow A1, in response to hand gesture 45 of separating the index finger and thumb, virtual object 43 can be enlarged in the direction of arrow A2. However, because virtual object 43 is enlarged around a predefined origin, it is not necessarily enlarged to fit the part of real object 41 that the user wants to grasp. Furthermore, as shown in Figure 4, it is possible that virtual object 43 will sink into plane 42 as a result of being enlarged.

Thirdly, when the shape of the real object to be superimposed does not match the shape of the virtual object, and it is possible to place the virtual object not perpendicular to the plane, there is a problem in that the user needs to be conscious of making the virtual object perpendicular to the plane. For example, as shown in Figure 5, when superimposing virtual object 53 on real object 51 on plane 52, the user needs to be conscious of making virtual object 53 perpendicular to plane 52, but as shown at the tip of arrow A1, there are cases in which virtual object 53 is not perpendicular to plane 52. In this example, virtual object 53 is not perpendicular to plane 52, but is perpendicular to real object 54 on plane 52, and is not displayed appropriately.

Fourthly, when the real object to be superimposed is smaller than the virtual object, there is a problem that, particularly when the user holds it in their hand, the real object to be superimposed is hidden by the hand and cannot be captured or recognized by the camera mounted on the XR HMD, making it impossible to display the virtual object. For example, as shown in Figure 6, consider a case in which the user holds a real object 61 with a virtual object 62 superimposed thereon in their hand 63. In this case, real object 61 is smaller than virtual object 62, and as shown at the tip of arrow A1, when real object 61 is hidden by the user's hand 63 and cannot be recognized, it becomes impossible to superimpose virtual object 62 on real object 61 and display it.

This disclosure proposes a method for appropriately displaying a virtual object by changing the display of the virtual object depending on the status of a real object (hereinafter also referred to as a second real object) that is different from the real object that is the superimposed object, in response to issues such as the user being unable to grasp the desired part of the virtual object due to differences in shape or size between the virtual object and the real object that is the superimposed object (hereinafter also referred to as the superimposed real object).

<System Configuration>
FIG. 7 is a block diagram showing a configuration example of an embodiment of an information processing device to which the present disclosure is applied.

In FIG. 7, the information processing device 70 is composed of a sensor unit 71, an object detection unit 72, a trigger detection unit 73, a superimposition target determination unit 74, a gripping area determination unit 75, a data management unit 76, a data storage unit 77, a virtual object position determination unit 78, a drawing unit 79, and a display unit 80.

The sensor unit 71 is composed of various sensors such as an image sensor, a gyro sensor, and an acceleration sensor. The sensor unit 71 performs sensing using the various sensors and outputs the resulting sensor data (image data, etc.) to the object detection unit 72, the virtual object position determination unit 78, and the display unit 80.

For example, the image sensor can capture the real world in front of the device as a video see-through image. An image sensor that captures images to be used as input for environmental recognition processing such as object recognition and position and orientation estimation may be provided separately from the image sensor for capturing video see-through images. An inertial sensor (gyro sensor, acceleration sensor) such as an IMU (Inertial Measurement Unit) may also be provided. If the information processing device 70 is configured as an XR HMD, an image sensor that captures the user's eyeballs may be provided to detect the line of sight of the user wearing the XR HMD.

The object detection unit 72 estimates the position and orientation of any real object, such as a real object onto which a virtual object is to be superimposed or the user's hand holding that real object, based on the image data output from the sensor unit 71. To estimate the position and orientation of any real object, for example, known 3D object recognition technology can be used. The object detection unit 72 outputs the detection results (estimation results) to the trigger detection unit 73, the superimposition target determination unit 74, the gripping area determination unit 75, and the virtual object position determination unit 78.

The trigger detection unit 73 detects the user's decision action based on the detection result output from the object detection unit 72. For example, the trigger detection unit 73 determines whether the user's hand detected by the object detection unit 72 has a specific shape, and if it determines that the hand has a specific shape, it can consider that the user's decision action has been detected. The trigger detection unit 73 outputs the detection result (decision trigger) to the superimposition target determination unit 74, the grip area determination unit 75, and the virtual object position determination unit 78.

The superimposition target determination unit 74 determines the real object to be superimposed based on the detection results output from the object detection unit 72 and the detection results (decision trigger) output from the trigger detection unit 73. Since multiple objects can be detected here, for example, the object that is most central on the screen, or the object the user is gazing at if a gaze detection function is installed, can be selected, and the object that is selected when the decision trigger is detected can be set as the real object to be superimposed. The superimposition target determination unit 74 outputs the determination result (real object to be superimposed) to the gripping area determination unit 75.

Based on the determination result output from the superimposition target determination unit 74, the gripping area determination unit 75 determines whether the distance between the real object to be superimposed and the virtual object drawn by the drawing unit 79 is less than a certain distance. If the gripping area determination unit 75 determines that the distance between the real object to be superimposed and the virtual object is less than a certain distance, it outputs a command to the drawing unit 79 to draw the virtual object semi-transparently or in a wireframe. Furthermore, when the gripping area determination unit 75 receives the detection result (gripping area determination trigger) output from the trigger detection unit 73, it determines the gripping area and grip origin for the real object to be superimposed based on the positional relationship between the real object to be superimposed and the virtual object, and the position of a second real object (e.g., the user's hand) relative to the real object to be superimposed, and outputs (notifies) the determination result to the data management unit 76.

The data management unit 76 stores the determination results notified by the grasping area determination unit 75 (for example, the type of real object to be superimposed, the type of linked virtual object, the positional relationship, and the grasping area and grasping origin of the real object to be superimposed) as data. The area in which the data management unit 76 stores data is typically volatile memory, and data is written to the data storage unit 77 when the device is powered off, an application is terminated, or the device is in standby mode. The data management unit 76 also reads data from the data storage unit 77 when the device is powered on, an application is started, or the like. The data storage unit 77 is made up of non-volatile memory such as NAND flash memory, and stores data from the data management unit 76.

The virtual object position determination unit 78 outputs to the drawing unit 79 a command to draw a virtual object according to the position and setting values specified by the user, based on the sensor data output from the sensor unit 71, the detection results output from the object detection unit 72, the detection results output from the trigger detection unit 73, or data from the data management unit 76.

For example, the user's designation is determined by receiving a decision trigger output from the trigger detection unit 73 while the user is designating any position on the plane detected by the object detection unit 72 with their finger or by directing their gaze. AR applications for XR HMDs and smartphones implement self-position and orientation estimation technologies such as SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry), which make it possible to obtain the three-dimensional coordinates of the shooting camera in real space. This technology may be used to display a virtual object at any relative position to the camera. If the data management unit 76 holds setting values for the real object to be superimposed (setting values related to superimposition), the virtual object position determination unit 78 commands the drawing unit 79 to draw the virtual object in accordance with the setting values received from the data management unit 76.

The rendering unit 79 renders virtual objects using 3DCG (3-Dimensional Computer Graphics) technology based on commands (commands for rendering position, orientation, etc.) output from the virtual object position determination unit 78. The display unit 80 combines an image (an image capturing the real world) based on the image data output from the sensor unit 71 with an image (virtual object) based on the rendering results output from the rendering unit 79, and displays it on a display device. The display device may be configured as an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, or the like. When the information processing device 70 is configured as an XR HMD, the display unit 80 is also equipped with an optical engine.

In the information processing device 70 configured as described above, the object detection unit 72, trigger detection unit 73, superimposition target determination unit 74, gripping area determination unit 75, virtual object position determination unit 78, and drawing unit 79 are provided as functions of the control unit 70A. The functions of the control unit 70A are realized by a processor such as a CPU (Central Processing Unit) executing a program. The number of processors is not limited to one, and two or more processors may be installed. For example, the drawing unit 79 may be configured as a GPU (Graphics Processing Unit), and the object detection unit 72, trigger detection unit 73, superimposition target determination unit 74, gripping area determination unit 75, and virtual object position determination unit 78 may be configured as a CPU.

The information processing device 70 can be configured as a device such as an XR HMD, a smartphone, a 3D display, or a camera for photography.

FIG. 8 is a diagram showing an example configuration of an XR HMD to which the present disclosure is applied. In FIG. 8, an XR HMD 92 is worn on the head of a user 91, with a sensor unit 93 provided on the front of the part covering the head and a display unit 94 on the back. The XR HMD 92 has a configuration similar to that of the information processing device 70 in FIG. 7, with the sensor unit 93 corresponding to the sensor unit 71 in FIG. 7 and the display unit 94 corresponding to the display unit 80 in FIG. 7. Functions such as the object detection unit 72 in FIG. 7 are realized by a processor such as a CPU mounted on the XR HMD 92 executing a program. In FIG. 8, the user 91 is holding a real object 95 to be superimposed in his or her hand (second real object) within the shooting range of the sensor unit 93, and can see a virtual object superimposed on the real object 95 to be superimposed, which is displayed on the display unit 94.

FIG. 9 is a diagram showing an example configuration of a smartphone to which the present disclosure is applied. In FIG. 9, a smartphone 102 is held in the hand of a user 101, and a sensor unit 103 is provided on the back of the housing and a display unit 104 is provided on the front. The smartphone 102 has a configuration similar to that of the information processing device 70 in FIG. 7, with the sensor unit 103 corresponding to the sensor unit 71 in FIG. 7 and the display unit 104 corresponding to the display unit 80 in FIG. 7. Functions such as the object detection unit 72 in FIG. 7 are realized by a processor such as a CPU mounted on the smartphone 102 executing a program. In FIG. 9, the user 101 holds the smartphone 102 in one hand and a real object 105 to be superimposed within the shooting range of the sensor unit 103 with the other hand (second real object), and can view a virtual object superimposed on the real object 105 to be superimposed, which is displayed on the display unit 104.

10 is a diagram showing an example configuration of a stationary 3D display to which the present disclosure is applied. In FIG. 10, a 3D display 112 is installed in an arbitrary location, and a sensor unit 113 and a display unit 114 are provided for a user 111 who is in a predetermined position. The 3D display 112 has a configuration similar to that of the information processing device 70 in FIG. 7, with the sensor unit 113 corresponding to the sensor unit 71 in FIG. 7 and the display unit 114 corresponding to the display unit 80 in FIG. 7. The functions of the object detection unit 72 in FIG. 7 and the like are realized by a processor such as a CPU mounted on the 3D display 112 executing a program. In FIG. 10, the user 111 is holding a real object 115 to be superimposed in his or her hand (second real object) within the shooting range of the sensor unit 113, and can view a virtual object superimposed on the real object 115 to be superimposed, which is displayed in 3D on the display unit 114.

11 is a diagram showing an example configuration of a photographing camera to which the present disclosure is applied. In FIG. 11, photographing camera 122 has a sensor unit 123 and a display unit 124, is installed in an arbitrary location, and is operated by first user 121. Photographing camera 122 has a configuration similar to that of information processing device 70 in FIG. 7, with sensor unit 123 corresponding to sensor unit 71 in FIG. 7 and display unit 124 corresponding to display unit 80 in FIG. 7. Functions such as object detection unit 72 in FIG. 7 are realized by a processor such as a CPU mounted on photographing camera 122 executing a program. In FIG. 11, second user 125 is holding real object 126 to be superimposed in his hand (second real object) within the photographing range of sensor unit 123, and first user 121 can see a virtual object superimposed on real object 126 to be superimposed, which is displayed on display unit 124.

Trigger detection is not limited to gesture determination by the trigger detection unit 73 as shown in the configuration of the information processing device 70 in Figure 7, but may also be, for example, a controller-type device, an input device such as a keyboard connected to a PC (Personal Computer) connected to the XR HMD, or a packet reception program that receives voice input from the user or commands from an external source via a network. Figure 12 shows a configuration for performing trigger detection using such devices and programs.

FIG. 12 is a block diagram showing another example configuration of an embodiment of an information processing device to which the present disclosure is applied. In FIG. 12, the information processing device 130 includes a sensor unit 131, an object detection unit 132, a trigger detection unit 133, a superimposition target determination unit 134, a gripping area determination unit 135, a data management unit 136, a data storage unit 137, a virtual object position determination unit 138, a drawing unit 139, and a display unit 140.

The trigger detection unit 133 can use a button input from a controller or input device as a trigger. The trigger detection unit 133 may also use a command received as a packet over a network such as the Internet as a trigger. The trigger detection unit 133 outputs the detection result (decision trigger) to the superimposition target determination unit 134, the grasped area determination unit 135, and the virtual object position determination unit 138.

In FIG. 12, the configuration other than the trigger detection unit 133 is the same as the configuration shown in FIG. 7, and therefore a description thereof will be omitted. Note that in the information processing device 130 of FIG. 12, the object detection unit 132, trigger detection unit 133, superimposition target determination unit 134, gripping area determination unit 135, virtual object position determination unit 138, and drawing unit 139 are also provided as functions of the control unit 130A. The functions of the control unit 130A are realized by one or more processors executing programs.

<Processing flow>
Fig. 13 is a flowchart showing the flow of processing for displaying a virtual object on a real object to be superimposed. In Fig. 13, the processing is described as being executed by the control unit 70A of the information processing device 70 in Fig. 7, but the processing may also be executed by the control unit 130A of the information processing device 130 in Fig. 12. The same applies to the processing of the other flowcharts shown in the subsequent figures.

In step S11, the data management unit 76 reads setting values (data) from the data storage unit 77. In step S12, the virtual object position determination unit 78 commands the drawing unit 79 to draw a virtual object at a location specified by the user, and the display unit 80 displays the drawing results by the drawing unit 79. In step S13, the superimposition target determination unit 74 selects (determines) a real object to be superimposed based on the detection results by the object detection unit 72 and the detection results by the trigger detection unit 73 (user's decision trigger).

In step S14, the virtual object position determination unit 78 determines whether a setting value exists for the real object to be superimposed. If it is determined in step S14 that a setting value exists, processing proceeds to step S15. In step S15, the virtual object position determination unit 78 commands the rendering unit 79 to superimpose the virtual object on the real object to be superimposed based on the setting value, and the display unit 80 displays the rendering result by the rendering unit 79.

On the other hand, if it is determined in step S14 that a set value does not exist, processing proceeds to step S16. In step S16, the gripping area determination unit 75 checks whether a user decision trigger has been detected by the trigger detection unit 73. If a decision trigger has not been detected in step S16, processing proceeds to step S17. In step S17, the gripping area determination unit 75 measures the distance between the current real object and virtual object to be superimposed, and determines whether the distance between the virtual object and real object is less than a threshold. For example, the distance can be measured by calculating the distance between the origins of the real object and virtual object to be superimposed.

If it is determined in step S17 that the distance is less than the threshold, the process proceeds to step S18. In step S18, the grasping area determination unit 75 commands the drawing unit 79 to display the virtual object in a wireframe, and the display unit 80 displays the drawing result by the drawing unit 79. Figure 14 is a diagram showing an example of a virtual object displayed in a wireframe according to the distance between the virtual object and a real object. Figure 14 shows a virtual object 151, a real object 152 selected as an object to be superimposed, and a user's hand 153 holding the real object 152 to be superimposed. As indicated by the tip of arrow A1, if the real object 152 to be superimposed is located close to the virtual object 151, the virtual object 151 is displayed in a wireframe. In the wireframe display, a shape represented by vertices and lines is displayed without using surfaces.

On the other hand, if it is determined in step S17 that the distance is equal to or greater than the threshold, the process proceeds to step S19. In step S19, the gripping area determination unit 75 instructs the rendering unit 79 not to display the virtual object in a wireframe, and the display unit 80 displays the rendering result by the rendering unit 79. For example, as shown at the base (starting point) of arrow A1 in FIG. 14, if the superimposed real object 152 is located far from the virtual object 151, the wireframe display of the virtual object 151 is terminated. Note that in this example, the virtual object is displayed in a wireframe when the distance between the virtual object and the superimposed real object is less than the threshold before the positional relationship between the virtual object and the superimposed real object is linked. However, this is not limited to a wireframe display, and other display formats may also be used. For example, the virtual object may be displayed semi-transparently with a predetermined transparency. When step S18 or S19 ends, the process returns to step S16, and the subsequent processes are repeated.

If a decision trigger is detected in step S16, processing proceeds to step S20. In step S20, the grasping area determination unit 75 determines whether or not the user is grasping the physical object to be superimposed. If it is determined in step S20 that the user is not grasping the physical object, processing returns to step S16, and the subsequent processing is repeated. If it is determined in step S20 that the user is grasping the physical object, processing proceeds to step S21.

In step S21, the grasping area determination unit 75 determines the grasping origin based on the current positional relationship between the real object and virtual object to be superimposed and the user's grasping position relative to the real object to be superimposed, and notifies the data management unit 76. The data management unit 76 then saves the local origin and grasping origin of the virtual object as setting values. In step S15, the virtual object position determination unit 78 commands the rendering unit 79 to superimpose the virtual object on the real object to be superimposed based on the saved setting values, and the display unit 80 displays the rendering result by the rendering unit 79.

Figure 15 is a diagram showing an example of setting the grip origin. Figure 15A (top row) shows a virtual object 161, a real object 162 to be superimposed that has a rectangular parallelepiped shape, and a user's hand 163 holding the real object 162 to be superimposed. The user's hand 163 is a second real object that is different from the real object to be superimposed. In this case, the grip origin 164 is determined by the positional relationship between the real object 162 to be superimposed and the user's hand 163 holding the real object 162 to be superimposed, and is usually set near the center of the palm of the user near the wrist.

As shown in B (bottom) of FIG. 15, for example, if the real object 165 to be superimposed is a controller, the virtual object 161 will be superimposed on a real object of a different shape. In such a case, as shown on the left side of B of FIG. 15, the virtual object 161 may be displayed at a location (position) a certain distance away from the real object 165 to be superimposed. In this case, based on the positional relationship between the real object 165 to be superimposed and the user's hand 163 holding the real object 165 to be superimposed, the grip origin 166 is usually set near the center of the palm of the user's hand near the wrist, but it can be set as follows. That is, if the virtual object 161 and the user's hand 163 are a certain distance apart and the real object 165 to be superimposed and the virtual object 161 do not intersect, the midpoint thereof may be set as the grip origin 167 (left side of B of FIG. 15).

Alternatively, as shown on the right side of B in Figure 15, if the real object 165 to be superimposed and the virtual object 161 intersect, but the user's hand 163 holding the real object 165 to be superimposed is a certain distance away from the virtual object 161, the grip origin 168 can be set at the center of the intersecting area.

In the flowchart of FIG. 13, when the grasping area determination unit 75 detects a reset trigger by the trigger detection unit 73, it erases the positional relationship between the real object to be superimposed and the virtual object, and the setting of the grasping origin, and the process returns to step S16. When the process of step S15 ends, the series of processes ends. In this way, in the information processing device 70 of FIG. 7, the control unit 70A determines the positional relationship between the virtual object and the real object to be superimposed, as well as the grasping area and grasping origin, and saves them as setting values, and the virtual object is displayed based on the saved setting values.

<Tracking target setting process flow>
Fig. 16 is a flowchart showing the flow of a process for setting a tracking target. This process is performed when a user grasps a virtual object with his/her hand, whose positional relationship between the virtual object and the real object and whose grasping origin have already been set by the process shown in the flowchart of Fig. 13.

In step S41, the virtual object position determination unit 78 determines whether the distance between the user's hand detected by the object detection unit 72 and the real object to be superimposed is less than a threshold (less than a certain distance). If it is determined in step S41 that the distance is less than the threshold, processing proceeds to step S42. In step S42, the virtual object position determination unit 78 commands the drawing unit 79 to draw the gripping area restored from the gripping origin superimposed on the virtual object, making the gripping area visible.

17 is a diagram showing an example of visualizing the area of a virtual object to be grasped. In FIG. 17, a virtual object 171 is displayed superimposed on a real object 172 to be superimposed, but the real object 172 is not visible to the user. In this case, as shown by the tip of arrow A1, when the user's hand 173 approaches the virtual object 171, the display of the virtual object is controlled so that the part of the real object 172 to be superimposed that should be grasped becomes visible. For example, a guide 174 indicating which part of the real object 172 to be superimposed should be grasped may be displayed, or the display of the virtual object 171 may be changed so that the real object 172 to be superimposed becomes visible. For example, when changing the display of the virtual object 171, the grasping area may be displayed transparently or semi-transparently, and the grasping area may be made transparent with a predetermined transparency (transparency). In other words, when the positional relationship between the virtual object 171 and the physical object 172 to be superimposed is linked, if the distance between the physical object 172 to be superimposed and a second physical object (user's hand 173) attempting to grasp the physical object 172 to be superimposed is less than a threshold value, the display of the virtual object 171 or a second virtual object different from the virtual object 171 (e.g., guide 174) is controlled so that the portion of the physical object 172 to be superimposed that is being grasped becomes visible.

In step S43, the object detection unit 72 detects whether the user's hand is grasping the real object to be superimposed. If it is detected in step S43 that the user's hand is grasping the real object to be superimposed, the process proceeds to step S44. In step S44, the object detection unit 72 treats the real object to be superimposed and the hand grasping the real object to be superimposed as a single object and tracks them. In other words, not only the real object to be superimposed but also the grasping hand as a second real object are tracked.

Figure 18 is a diagram showing an example of tracking feature points of a real object to be superimposed and a second real object. In Figure 18, feature points 182 of real object 181 to be superimposed are represented by black squares. Object tracking is a technique for tracking a group of feature points of an object; the more feature points that are tracked, the less likely an object is to be lost (the less likely it is to become untrackable). As shown at the tip of arrow A1, when real object 181 to be superimposed is held by a user's hand 183, the user's hand 183 becomes the second real object, and its feature points 184 are also tracked. In Figure 18, feature points 184 of user's hand 183 are represented by white squares.

As indicated by the tip of arrow A2, superimposed real object 181 and user's hand 183 are considered to be a single object, and tracking is performed by tracking a group of feature points including feature point 182 and feature point 184 as the tracking target. In this way, when the positional relationship between the virtual object and the superimposed real object is linked, when the superimposed real object is grasped by a second real object (the user's hand), the feature points of the superimposed real object and the second real object are tracked, and the second real object is also included in the tracking targets. In this way, even if at least a portion of the superimposed real object is occluded by the second real object and its feature points become invisible, by tracking the feature points of the second real object, the probability of the superimposed real object being lost can be reduced based on the tracking results of at least the feature points of the second real object.

Then, the control unit 70A controls the display of the virtual object to be superimposed on the real object to be superimposed based on the tracking results. For example, the display of the virtual object can be controlled based on the position and orientation of the second real object, and the positional relationship between the second real object and the real object to be superimposed when grasped. Note that the object compared with the second real object (user's hand) in step S41 is not limited to the real object to be superimposed, but may also be, for example, a virtual object or a grasp origin, and it may be determined whether the distance to the virtual object or the grasp origin is less than a threshold. In short, it is sufficient to determine an appropriate timing for making visible the grasp area of the real object to be superimposed, grasped by the second real object (user's hand).

<Flow of processing for enlarging, reducing, and rotating virtual objects>
FIG. 19 is a flowchart showing the flow of processing for enlarging, reducing, and rotating a virtual object superimposed on a target real object.

In step S61, the virtual object position determination unit 78 checks whether an enlargement trigger has been detected by the trigger detection unit 73. If an enlargement trigger is detected in step S61, the process proceeds to step S62. In step S62, the virtual object position determination unit 78 commands the drawing unit 79 to enlarge and draw the virtual object in the positive x, y, and z directions of the coordinate system of the grip origin saved as setting values.

Figure 20 is a diagram showing an example of enlarging a virtual object. Figure 20 shows a virtual object 191, a real object 193 to be superimposed, and a user's hand 194 holding the real object 193 to be superimposed. When an enlargement trigger is detected, the virtual object 191 is enlarged, starting from a grip origin 195 determined according to the state of the real object 193 to be superimposed and the user's hand 194, and virtual object 192 is displayed. In other words, by enlarging the virtual object 191 before enlargement, the enlarged virtual object 192 is displayed. Note that this example shows a case where the virtual object is displayed as a wireframe.

If an enlargement trigger is not detected in step S61, the process proceeds to step S63. In step S63, the virtual object position determination unit 78 checks whether a reduction trigger has been detected by the trigger detection unit 73. If a reduction trigger is detected in step S63, the process proceeds to step S64. In step S64, the virtual object position determination unit 78 commands the drawing unit 79 to reduce and draw the virtual object in the negative x, y, and z directions of the coordinate system of the grip origin saved as the setting values. For example, in FIG. 20, by reducing the virtual object 192 before reduction, the reduced virtual object 191 is displayed.

If a reduction trigger is not detected in step S63, processing proceeds to step S65. In step S65, the virtual object position determination unit 78 checks whether a rotation trigger has been detected by the trigger detection unit 73. If a rotation trigger is detected in step S65, processing proceeds to step S66. In step S66, the virtual object position determination unit 78 commands the drawing unit 79 to rotate and draw the virtual object around the grip origin point saved as a setting value.

FIG. 21 is a diagram showing an example of virtual object rotation. FIG. 21 shows a virtual object 201, a real object 202 to be superimposed, and a user's hand 203 grasping the real object 202 to be superimposed. As indicated by the tip of arrow A1, when a rotation trigger is detected, the virtual object 201 is displayed rotated from a grasping origin 204 determined according to the state of the real object 202 to be superimposed and the user's hand 203. In other words, by rotating the unrotated virtual object 201 indicated by the dashed line in the direction of arrow A2 (counterclockwise), the rotated virtual object 201 indicated by the solid line is displayed.

Once steps S62, S64, and S66 are completed, processing returns to step S61, and the above-described processing is repeated. Furthermore, if no trigger is detected in steps S61, S63, or S65, the series of processing steps ends.

<Processing flow when arranging superimposition targets on a plane>
FIG. 22 is a flowchart showing the flow of processing when a physical object to be superimposed is arranged on a plane.

In step S81, the object detection unit 72 detects that the real object to be superimposed has been placed on the plane. In step S82, the virtual object position determination unit 78 commands the rendering unit 79 to rotate the virtual object around the grip origin saved as a setting value, so that it is perpendicular to the plane, and render it.

FIG. 23 is a diagram showing an example in which a virtual object is rotated so that it is perpendicular to a plane. FIG. 23 shows a virtual object 211, a real object 212 to be superimposed, and a user's hand 213 holding the real object 212 to be superimposed. As shown in FIG. 23, when the real object 212 to be superimposed is placed on a plane 214, the virtual object 211 is displayed rotated so that it is perpendicular to the plane 214, starting from a gripping origin 215 determined according to the state of the real object 212 to be superimposed and the user's hand 213. In other words, by correcting the position and orientation of the virtual object 211 before rotation, indicated by the dashed line, so that it rotates in the direction of arrow A1 (clockwise), the virtual object 211 after rotation, indicated by the solid line, is displayed on the plane 214.

<Processing flow when changing the overlay target>
24 is a flowchart showing the flow of processing when a user changes the grip of a real object to be superimposed. This processing is performed when, for example, the real object to be superimposed and the virtual object are similar but not identical in size or shape, and the user currently holding the real object to be superimposed in one hand changes the real object to be superimposed to another hand.

In step S101, the object detection unit 72 detects that the user has switched the real object to be superimposed, which is currently being held in one hand, to another hand. In step S102, the gripping area determination unit 75 notifies the data management unit 76 of the gripping origin of the switched hand as a setting value and stores it.

In step S103, the virtual object position determination unit 78 updates the position and orientation of the virtual object in accordance with the saved settings, and commands the rendering unit 79 to superimpose the virtual object on the target real object. In step S104, the object detection unit 72 treats the hand that has been swapped with the target real object as a single object to be tracked, and begins tracking it.

Figure 25 is a diagram showing an example of resetting the grip origin when the real object to be superimposed is changed. Figure 25 shows a real object 221 to be superimposed, a virtual object 222, and the user's hand holding the real object 221 to be superimposed. In this example, the real object 221 to be superimposed is configured in the shape of a camera, and is similar in size and shape to the virtual object 222 to be superimposed. The real object 221 to be superimposed is slightly smaller in size than the virtual object 222, and is held in the user's hand.

As shown at the end of arrow A1, it is assumed that the user grasps the superimposed real object 221 that they have been holding with one hand 223 with the other hand 225, and then, as shown at the end of arrow A2, releases the one hand 223 from the superimposed real object 221 and switches the hand holding the superimposed real object 221 from one hand 223 to the other hand 225. In this case, when the superimposed real object 221 was being held with one hand 223, the virtual object 222 was displayed with a gripping origin 224 determined in accordance with the state of the superimposed real object 221 and the user's hand 223. Furthermore, when the superimposed real object 221 is shifted to the other hand 225, the virtual object 222 is displayed with a gripping origin 226 determined in accordance with the state of the superimposed real object 221 and the user's hand 225. In this way, when a virtual object is gripped differently, the grip origin can be reset (redetermined) to correct the position and orientation of the virtual object. Furthermore, by tracking the real object to be superimposed and the hand that grips it, tracking can be performed reliably as described above.

As described above, with the present disclosure, when superimposing a virtual object on a real object to be superimposed, the origin (grasp origin) for displaying the virtual object can be determined depending on the status of the real object to be superimposed (e.g., a controller) and a second real object (e.g., a user's hand), and the display of the virtual object can be controlled depending on the determined origin. That is, with the present disclosure, information is acquired regarding the status of the real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed, and the display of the virtual object is controlled based on the acquired information. This addresses issues such as a user being unable to grasp a desired portion of a virtual object due to differences in shape or size between the virtual object and the real object to be superimposed, by changing the display of the virtual object depending on the status of the real object to be superimposed and the second real object, making it possible to appropriately display the virtual object. By appropriately displaying the virtual object to be superimposed on the real object to be superimposed, for example, the user can grasp a desired portion of the virtual object superimposed on the real object to be superimposed.

Note that the above-mentioned Patent Document 1 proposes a technology that blurs the boundary between a virtual object when it is superimposed on a real object. While this reduces the retinal rivalry that occurs when a real object and a virtual object are simultaneously in the field of view, it does not resolve the discomfort that occurs when grasping a virtual object due to differences in shape or size between the real object and the virtual object. Furthermore, Patent Document 2 proposes a technology that aligns a specified plane of a virtual object so that it is parallel to a plane along the hand or a specified plane of a mockup, but does not propose which part of a real object on which a virtual object should be grasped, nor does it mention any criteria for enlarging, reducing, or rotating the virtual object. Furthermore, Patent Document 3 proposes a technology that, when superimposing a virtual object of a different size or shape on a real object, displays the virtual object semi-transparently when the user's hand is not touching the real object, but does not mention any criteria for enlarging, reducing, or rotating the virtual object.

<Modification>
The information processing device 70 of FIG. 7 and the information processing device 130 of FIG. 12 are not limited to the above-described configurations and may employ other configurations. For example, if the information processing device 70 of FIG. 7 is configured as a server connected to a network such as the Internet, it may be configured without the sensor unit 71 and the display unit 80, i.e., with only a control unit 70A and a communication unit. Furthermore, a terminal device (e.g., an XR HMD, a smartphone, etc.) may include the sensor unit 71, the display unit 80, and a communication unit. In this case, the information processing device 70 (server) may receive sensor data transmitted from the terminal device via the network, the control unit 70A may process the received data, and the resulting processed data may be transmitted to the terminal device via the network. Meanwhile, the terminal device may transmit sensor data acquired by the sensor unit 71 to the information processing device 70 (server) via the network. Furthermore, the terminal device may receive processed data from the information processing device 70 (server) via the network, and a virtual object based on the received data may be displayed on the display unit 80. In the information processing device 70 (server), the control unit 70A only needs to have at least some of the blocks among the object detection unit 72, the trigger detection unit 73, the superimposition target determination unit 74, the gripping area determination unit 75, the data management unit 76, the data storage unit 77, the virtual object position determination unit 78, and the drawing unit 79, and the remaining blocks may be provided on the terminal device side. For example, the drawing unit 79 may be provided on the terminal device side instead of the server side.

In the above example, a camera is displayed as a virtual object superimposed on the real object to be superimposed, but other objects (e.g., products that have not yet been manufactured or purchased) may also be displayed as virtual objects. Furthermore, in the above example, the user's hand is used as an example of a second real object that is different from the real object to be superimposed, but other objects such as objects around the user (e.g., a table) may also be used.

<Computer Configuration>
The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, a program constituting the software is installed in a computer. Fig. 26 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

In a computer, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are interconnected by a bus 304. An input/output interface 305 is also connected to the bus 304. An input unit 306, an output unit 307, a memory unit 308, a communication unit 309, and a drive 310 are connected to the input/output interface 305.

The input unit 306 includes a keyboard, mouse, microphone, etc. The output unit 307 includes a display, speaker, etc. The storage unit 308 includes a hard disk, non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a semiconductor memory, magnetic disk, optical disk, or magneto-optical disk.

In a computer configured as described above, the CPU 301 loads programs stored in the ROM 302 or memory unit 308 into the RAM 303 via the input/output interface 305 and bus 304, and executes them, thereby performing the series of processes described above.

The program executed by the computer (CPU 301) can be provided by being recorded on a removable recording medium 311, such as a packaged medium. The program can also be provided via a wired or wireless transmission medium, such as a local area network, the Internet, or digital satellite broadcasting.

In a computer, a program can be installed in the storage unit 308 via the input/output interface 305 by inserting the removable recording medium 311 into the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be pre-installed in the ROM 302 or storage unit 308.

The processing performed by a computer according to a program includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing). Furthermore, a program may be processed by a single computer (processor), or may be distributed across multiple computers.

Note that the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present disclosure. For example, the present disclosure may take the form of a cloud computing configuration in which a single function is shared and processed collaboratively by multiple devices via a network.

Each step described in the above flowchart can be executed by a single device, or can be shared and executed by multiple devices. Furthermore, if a single step includes multiple processes, the multiple processes included in that single step can be executed by a single device, or can be shared and executed by multiple devices. Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

Furthermore, the present disclosure can be configured as follows:

(1)
acquiring information about a situation of a real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed;
An information processing device comprising: a control unit that controls display of the virtual object based on the acquired information.
(2)
The control unit
determining a starting point for displaying the virtual object in accordance with the situations of the real object to be superimposed and the second real object;
The information processing device according to (1), wherein display of the virtual object is controlled in accordance with the determined starting point.
(3)
The information processing device according to (2), wherein the control unit determines a starting point when enlarging or reducing the virtual object according to a state of the real object to be superimposed and the second real object.
(4)
The information processing device according to (2) or (3), wherein the control unit determines a starting point when rotating the virtual object according to a state of the real object to be superimposed and the second real object.
(5)
The information processing device according to (4), wherein, when placing the physical object to be superimposed on a plane, the control unit rotates the virtual object according to the origin and displays the virtual object so as to be perpendicular to the plane.
(6)
the real object to be superimposed is an object to be held by a user,
The information processing device according to any one of (2) to (5), wherein the second physical object is a hand of the user.
(7)
The control unit
when the user holds the physical object to be superimposed with one hand, determining the starting point in accordance with the physical object to be superimposed and the one hand holding the physical object to be superimposed;
The information processing device described in (6), wherein when the user switches the hand holding the physical object to be superimposed from one hand to the other, the starting point is re-determined based on the physical object to be superimposed and the other hand holding the physical object to be superimposed.
(8)
The information processing device according to (1) or (6), wherein the control unit displays the virtual object semi-transparently or in a wireframe when a distance between the virtual object and the real object to be superimposed is less than a threshold before linking a positional relationship between the virtual object and the real object to be superimposed.
(9)
The information processing device according to (1) or (6), wherein, when the positional relationship between the virtual object and the real object to be superimposed is linked, if a distance between the real object to be superimposed and the second real object attempting to grasp the real object to be superimposed is less than a threshold, the control unit controls display of the virtual object or a second virtual object different from the virtual object so that a grasped portion of the real object to be superimposed is visible.
(10)
The control unit
when the virtual object and the real object to be superimposed are associated in positional relationship and the real object to be superimposed is grasped by the second real object, feature points of the real object to be superimposed and the second real object are targeted for tracking;
The information processing device according to (1) or (6), wherein when at least a part of the real object to be superimposed is occluded by the second real object, the display of the virtual object is controlled based on at least a tracking result of the second real object.
(11)
The information processing device according to (9), wherein the control unit displays a guide indicating a portion of the real object to be superimposed that is to be grasped.
(12)
The information processing device according to (10), wherein the control unit controls the display of the virtual object based on the position and orientation of the second physical object and the positional relationship between the second physical object and the physical object to be superimposed when grasped.
(13)
The information processing device
acquiring information about a situation of a real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed;
and controlling display of the virtual object based on the acquired information.
(14)
determining a starting point for displaying the virtual object in accordance with the situations of the real object to be superimposed and the second real object;
The information processing method according to (13), further comprising: controlling display of the virtual object in accordance with the determined starting point.
(15)
The information processing method according to (14), further comprising determining a starting point when enlarging or reducing the virtual object according to the situations of the real object to be superimposed and the second real object.
(16)
The information processing method according to (14) or (15), further comprising determining a starting point for rotating the virtual object according to the situations of the real object to be superimposed and the second real object.
(17)
The information processing method according to any one of (13) to (16), further comprising displaying the virtual object semi-transparently or in a wireframe when a distance between the virtual object and the real object to be superimposed is less than a threshold value before linking the positional relationship between the virtual object and the real object to be superimposed.
(18)
The information processing method according to any one of (13) to (16), further comprising, when the positional relationship between the virtual object and the real object to be superimposed is linked and the distance between the real object to be superimposed and the second real object attempting to grasp the real object to be superimposed is less than a threshold, controlling display of the virtual object or a second virtual object different from the virtual object so that the grasped portion of the real object to be superimposed becomes visible.
(19)
when the virtual object and the real object to be superimposed are linked in positional relationship and the real object to be superimposed is held by the second real object, feature points of the real object to be superimposed and the second real object are targeted for tracking;
The information processing method according to any one of (13) to (16), further comprising: when at least a part of the real object to be superimposed is occluded by the second real object, controlling the display of the virtual object based on at least a tracking result of the second real object.
(20)
Computer,
acquiring information about a situation of a real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed;
A computer-readable recording medium on which a program is recorded for causing the device to function as a control unit that controls the display of the virtual object based on the acquired information.

70 Information processing device, 70A Control unit, 71 Sensor unit, 72 Object detection unit, 73 Trigger detection unit, 74 Superimposition target determination unit, 75 Grasping area determination unit, 76 Data management unit, 77 Data storage unit, 78 Virtual object position determination unit, 79 Rendering unit, 80 Display unit, 130 Information processing device, 130A Control unit, 131 Sensor unit, 132 Object detection unit, 133 Trigger detection unit, 134 Superimposition target determination unit, 135 Grasping area determination unit, 136 Data management unit, 137 Data storage unit, 138 Virtual object position determination unit, 139 Rendering unit, 140 Display unit

Claims (20)

acquiring information about a situation of a real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed;
An information processing device comprising: a control unit that controls display of the virtual object based on the acquired information. The control unit
determining a starting point for displaying the virtual object in accordance with the situations of the real object to be superimposed and the second real object;
The information processing apparatus according to claim 1 , wherein the display of the virtual object is controlled in accordance with the determined starting point. The information processing device according to claim 2 , wherein the control unit determines a starting point when enlarging or reducing the virtual object depending on the status of the real object to be superimposed and the second real object. The information processing device according to claim 2 , wherein the control unit determines a starting point for rotating the virtual object depending on the situations of the real object to be superimposed and the second real object. The information processing device according to claim 4 , wherein when placing the physical object to be superimposed on a plane, the control unit rotates the virtual object according to the origin and displays the virtual object so as to be perpendicular to the plane. the real object to be superimposed is an object to be held by a user,
The information processing apparatus according to claim 2 , wherein the second physical object is a hand of the user. The control unit
when the user holds the physical object to be superimposed with one hand, determining the starting point in accordance with the physical object to be superimposed and the one hand holding the physical object to be superimposed;
The information processing device according to claim 6, wherein when the user switches the hand holding the physical object to be superimposed from one hand to the other, the starting point is re-determined depending on the physical object to be superimposed and the other hand holding the physical object to be superimposed. 2. The information processing device according to claim 1, wherein, before a positional relationship between the virtual object and the real object to be superimposed is linked, if a distance between the virtual object and the real object to be superimposed is less than a threshold, the control unit displays the virtual object semi-transparently or in a wireframe. 2. The information processing device according to claim 1, wherein, when a positional relationship between the virtual object and the real object to be superimposed is linked and a distance between the real object to be superimposed and the second real object attempting to grasp the real object to be superimposed is less than a threshold, the control unit controls display of the virtual object or a second virtual object different from the virtual object so that a grasped portion of the real object to be superimposed is visible. The control unit
when the virtual object and the real object to be superimposed are associated in positional relationship and the real object to be superimposed is grasped by the second real object, feature points of the real object to be superimposed and the second real object are targeted for tracking;
The information processing apparatus according to claim 1 , wherein when at least a part of the real object to be superimposed is occluded by the second real object, the display of the virtual object is controlled based on at least a tracking result of the second real object. The information processing device according to claim 9 , wherein the control unit displays a guide indicating a portion of the real object to be superimposed that is to be grasped. The information processing device according to claim 10 , wherein the control unit controls the display of the virtual object based on a position and orientation of the second physical object and a positional relationship between the second physical object and the physical object to be superimposed when held. The information processing device
acquiring information about a situation of a real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed;
and controlling display of the virtual object based on the acquired information. determining a starting point for displaying the virtual object in accordance with the situations of the real object to be superimposed and the second real object;
The information processing method according to claim 13 , further comprising: controlling display of the virtual object in accordance with the determined starting point. The information processing method according to claim 14 , further comprising determining a starting point for enlarging or reducing the virtual object according to the situations of the real object to be superimposed and the second real object. The information processing method according to claim 14 , further comprising determining a starting point for rotating the virtual object according to the situations of the real object to be superimposed and the second real object. The information processing method according to claim 13 , further comprising: displaying the virtual object semi-transparently or in a wireframe when a distance between the virtual object and the real object to be superimposed is less than a threshold value before the positional relationship between the virtual object and the real object to be superimposed is linked. 14. The information processing method according to claim 13, further comprising: controlling display of the virtual object or a second virtual object different from the virtual object so that a portion of the real object to be superimposed that is to be grasped becomes visible when a distance between the virtual object to be superimposed and the second real object attempting to grasp the real object to be superimposed is less than a threshold value in a case where a positional relationship between the virtual object and the real object to be superimposed is linked. when the virtual object and the real object to be superimposed are linked in positional relationship and the real object to be superimposed is held by the second real object, feature points of the real object to be superimposed and the second real object are targeted for tracking;
14. The information processing method according to claim 13, further comprising: when at least a part of the real object to be superimposed is occluded by the second real object, controlling display of the virtual object based on at least a tracking result of the second real object. Computer,
acquiring information about a situation of a real object to be superimposed on which the virtual object is to be superimposed and a second real object different from the real object to be superimposed;
A computer-readable recording medium on which a program is recorded to function as a control unit that controls the display of the virtual object based on the acquired information.