JP2020071394A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a feeling of strangeness. SOLUTION: The second image is mounted on an image display device including an image pickup means for acquiring a first image of an image of a real space and a display means for displaying a second image generated by using the first image. A second information processing apparatus to supply, which generates a background image in which a region of a specific object included in the first image is complemented by using image features obtained from the periphery of the region included in the first image. Based on the result of estimating the position of a specific object included in the first image after a predetermined time with one generation means, an image depicting the object at a position based on the estimation result in the background image was synthesized. The second generation means for generating the second image, a delay time due to at least a part of the processing time in the processing from imaging to display, and the amount of change in the position or posture of the image display device. It has a display position control means for shifting a position for displaying a second image supplied to the display means. [Selection diagram] Fig. 2

Description

  The present invention relates to virtual reality and mixed reality systems.

  Virtual reality (VR) and mixed reality (MR) systems are used for the purpose of shortening the period of the trial manufacturing process and reducing costs in the field of design and manufacturing. In these systems, design (shape / design) data created by a CAD (Computer Aided Design) system can be used to evaluate ease of assembly and maintainability without making a prototype of a real object. In these systems, a head mounted display (HMD: HEAD Mounted Display) is used. The HMD covers the entire field of view by mounting it directly on the head, and displays an image that allows the user to experience virtual reality from a viewpoint according to the position and posture of the user. When the position and orientation of the HMD change due to the movement of the experience person, it takes time to generate the image to be displayed on the HMD, so that the image before the experience person moves is displayed on the HMD. At this time, an uncomfortable feeling (uncomfortable feeling due to perceptual delay) occurs in the image due to the display of the image different from the image assumed by the experience person. Therefore, in Patent Document 1, the display image is shifted within the screen by an amount corresponding to the posture change amount of the delay time for the purpose of easing the perceptual delay.

JP, 2004-109994, A

  In Patent Document 1, even when the moving body exists in the physical space, it is not estimated in which direction the moving body moves, and only the image before the position / posture abruptly moves is shifted. Therefore, the moving body may be displayed in a direction opposite to the direction in which the position at which the moving body should be displayed moves, as the position and orientation of the HMD change. That is, since the method disclosed in Patent Document 1 does not consider the motion of the moving body in the physical space, it is not possible to reduce the perceived discomfort in the image including the moving body. The present invention has been made in view of the above problems, and even when a moving body exists in a scene, when the HMD wearer suddenly changes the position and orientation of the HMD, the moving body is not displayed at the position where it should be displayed. The purpose is to reduce the discomfort.

  An information processing apparatus according to the present invention that solves the above problem includes an image capturing unit that acquires a first image obtained by capturing an image of a physical space, and a display unit that displays a second image generated using the first image. An information processing apparatus that supplies the second image to an image display apparatus, wherein image characteristics obtained by surrounding a region of a specific object included in the first image with a periphery of the region included in the first image are provided. First generation means for generating a background image complemented by using the first image generation means, and a position based on the estimation result in the background image based on a result of estimating a position of a specific object included in the first image after a predetermined time. Second generation means for generating the second image in which the images depicting the object are combined with each other, a delay time caused by at least a part of the processing time in the processing from the imaging to the display, and the position of the image display device. Also Based on the amount of change of the posture, and a display position control means for shifting the position of displaying the second image to be supplied to the display means.

  According to the present invention, according to the present invention, even when a moving object is present in a scene, when a sudden change in the position and orientation of the HMD of the HMD wearer occurs, the moving object is not displayed at the position where it should be displayed. Can be reduced.

The figure which shows the example of a display of the delayed image which can occur in virtual reality space. FIG. 6 is a diagram showing a display example of an image generated by the information processing device. The figure which shows the example of the coordinate system which predicts an object. The block diagram which shows the function structural example of an information processing system. 6 is a flowchart showing processing executed by the information processing device. The sequence diagram which shows the process which an information processing system performs. The block diagram which shows the function structural example of an information processing system. 6 is a flowchart showing processing executed by the information processing device. The sequence diagram which shows the process which an information processing system performs. The block diagram which shows the function structural example of an information processing system. FIG. 2 is a block diagram showing a hardware configuration example of an information processing device.

(First embodiment)
Hereinafter, with reference to the accompanying drawings, a detailed description will be given according to preferred embodiments to which the present invention is applied.

  A head mounted display (hereinafter, HMD) has a built-in display for both eyes and headphones for both ears. By mounting the HMD on the head, the user can watch a still image or a moving image displayed on the display and can listen to voice or music output from the headphones. In this embodiment, an example of an HMD used in a mixed reality system (MR system) will be described. At this time, the HMD synthesizes the image of the real space and the image of the virtual space captured from the image pickup unit in the HMD, and outputs the image as a video that allows the user to experience mixed reality on the display unit in the HMD. Further, it is possible to measure the position information of the head of the user who wears the HMD and the posture information such as the rotation angle and the inclination of the head by a gyro sensor or an acceleration sensor built in or attached to the HMD. The posture information is the result of measuring the relative movement of the HMD by a posture sensor such as a gyro sensor. The images or videos described below are examples of images generated by the information processing device. Some or all of the generated images are presented to the user wearing the HMD. In this embodiment, a method of observing the moving body 110 and arranging the moving body 110 at a predicted position to reduce a feeling of physical discomfort will be described. In this embodiment, a moving object is observed by a stereo camera, and an image for displaying a two-dimensional moving object region extracted from a captured image at a predicted position is generated.

  First, an example in which a video delay including a specific object (for example, a moving object or an object that may move) occurs in the virtual space in the virtual space will be described with reference to FIG. FIG. 1A shows the positional relationship between the HMD, a specific object (moving object 110) and the stationary object 120 in the physical space at a predetermined time t. Here, it is assumed that the specific object is the hand of the HMD wearer as a representative of the objects that may move. The positional relationship at the next time t + Δt from the state of (A) of FIG. 1 is shown in (B) of FIG. The time Δt is a delay time, and is assumed to be a difference between the time taken by the information processing apparatus and the time measured at the position of the HMD immediately before the image displayed on the HMD is generated (for example, 100 msec.).

  FIG. 1B shows a state in which the HMD rotates to the right by ΔΘ during Δt seconds, and the moving body moves to the right by Δh with respect to the stationary object 120. The change Δθ in the position and orientation of the HMD can be measured by the position and orientation sensor built in the HMD. 1C and 1D show ideal images without delay at time t and time t + Δt as viewed from the HMD viewpoint. FIG. 1E illustrates an image 1020 in which a delay has occurred in the virtual space drawing process. The image 1020 is the result of the image processing of the image 1000 at time t, but is displayed with a delay of the time required for the image processing regardless of the movement of the HMD. Here, with respect to the ideal display position of the moving body in FIG. 1D without delay, the moving body is depicted at a position horizontally displaced by the length of arrow 150 in FIG. 1E. The object 130 is a CG that depicts the CG model in real space. In proportion to the size of the arrow, the HMD wearer can cause a sense of discomfort in the image in the virtual space. It should be noted that the dotted lines passing through FIGS. 1D, 1E, and 1F show the field of view in which the image 1000 was visible, and indicate that the image 1000 is displaced by ΔX as the HMD moves.

  On the other hand, FIG. 1 (F) shows an image when the image generated at time t is horizontally shifted by a size ΔX in consideration of the rotation amount ΔΘ of the HMD. Details of the process of shifting the image will be described later. The black area of the image 1030 indicates that the area that cannot be captured because the image is shifted is displayed in black. The position of the stationary object 120 in FIG. 1 (F) is displayed at the same position on the screen of the HMD as compared with the position of the stationary object 120 in FIG. 1 (D). However, with respect to the moving body, the length of the arrow 160 causes a deviation from the position of the moving body in real time (time t + Δt) (ideal display position of the moving body in FIG. 1D), so that the moving body is delayed. I feel that I am doing it. This occurs because the captured image 1000 obtained at time t is shifted by ΔX, ignoring the moving distance Δh of the moving body that has moved in Δt seconds. Note that the arrow 160 is longer than the arrow 150 which is the difference between the position of the moving object in the case where the processing of shifting the image in FIG. 1E is not included and the ideal display position in FIG. .. Therefore, the HMD wearer may feel more uncomfortable. An example is a case where the HMD rotates to the right and the wearer's hand moves in parallel to the right. That is, even if an image is displayed in consideration of a change in the position or orientation of the HMD, the movement of a specific object cannot be predicted and further processing time is required. Therefore, an image that is uncomfortable for the HMD wearer is displayed. There is a possibility that Further, in the video see-through HMD for providing mixed reality, the time for capturing and exposing the real space by the HMD-equipped camera and the time for transmitting the image information are added before the generation of the display image described above. It That is, the processing time required for image generation is further lengthened, and there is a possibility that discomfort may be further increased due to changes in the posture of the HMD or the moving body.

  FIG. 2 is a diagram showing a display example of an image generated by the information processing apparatus according to the present invention. In FIGS. 2A and 2B, the shooting time is t and the display time is t + Δt. Similar to FIG. 1, at time t, the hand 210 and the object 220 that are moving bodies are observed from the HMD 200. Further, at time t + Δt, the positional relationship between the HMD, the object, and the hand is as in the HMD 201, the hand 211, and the object 220. The HMD rotates and directs its line of sight to the right. Also, the hand moves from the time t to the time t + Δt from the position 210 to the position 211.

  At time t, the HMD acquires an image 2000 captured by the attached image capturing device. The image 230 shows the hand 230 of the HMD wearer. Next, as shown in FIG. 2D, a two-dimensional moving body area 232 is detected from the image 2000. Here, the skin color area is extracted from the captured image. In this embodiment, all the color information of the skin color area when the skin color is imaged is recorded in advance and stored in the table. At this time, the color information may be represented by the three primary colors of RGB, or may be represented by the luminance and tint information of YCbCr. It should be noted that the moving object detection method can be selected from color detection, edge detection, inter-frame difference, area division by SuperPixel, pattern matching, and object detection method based on learning, but the present invention is limited to a specific method. is not. For example, Yuji Yamauchi, Takayoshi Yamashita, Hironobu Fujiyoshi, “[Survey Paper] Human Detection by Statistical Learning Method”, IEICE Technical Report on Pattern Recognition and Media (PRMU), pp. The method of 113-126 (2012) may be used. After the area is detected, the position of the moving body at time t + Δt is predicted by the method described later in FIG. 2E, and the predicted moving body position 234 is obtained.

  After detecting the moving body area, a background area 241 is obtained as shown in FIG. Next, in FIG. 2G, the moving body area 241 is filled in based on the image feature of the background area, and a background image 2040 having no moving body is generated. In the present embodiment, the average of the colors of the peripheral area of the moving body area is acquired, and the moving body area is complemented by the color. As for the background filling method, there are a method of filling with a color similar to that of the peripheral area and a method of filling from a past image, but the present invention is not limited to a particular method. For example, Mori et al. (Shohei Mori, Ryosuke Ichikari, Fumihisa Shibata, Asako Kimura, Hideyuki Tamura, "Technical framework and problems of hidden reality" -visually conceal, erase and see through real objects in the real world. About technology ~ ", the journal of the Virtual Reality Society of Japan) may be used. Further, based on the CG model data, the object 130 is rendered at a corresponding predetermined position in the real space in the background image.

  Here, the original captured image at time t + Δt is the image 2060 in FIG. 2 (i), but the most recent captured image on the system is the image 2000 in FIG. 2 (c). Therefore, the image 2070 for display is generated using FIG. This is called a composite image. As in the case of Patent Document 1, the composite image moves the image 2040 of FIG. 2G in parallel in the direction opposite to the rotation by ΔX in accordance with the viewpoint rotation angle ΔΘ. As a result, the image 2050 of FIG. 2H is generated. Next, the moving body region 232 is combined with the moving body predicted position in FIG. 2 (h) to generate the image 2070 shown in FIG. 2 (j). Finally, the composite image 2070 is displayed on the HMD. As a result of the above processing, it is possible to display a composite image 2070 that is close to the desired image 2060.

  Alternatively, the method described below may be used. FIG. 2K is an image 2080 in which the image 2020 depicting the moving body 235 at the predicted position and the background image 2040 complementing the moving body region are combined. FIG. 2 (l) is an image 2090 obtained by shifting the image 2080 in accordance with changes in the delay time and the position and orientation of the HMD. At this time, the image 2090 may be displayed on the HMD. With this method as well, since the motion of the moving body is used for prediction, it is possible to display an image with less physical discomfort.

  An example of the moving body prediction method according to the present invention will be described with reference to FIG. The moving body 301 shows the position of the moving body at time t in FIG. 2, and the moving body 302 shows the position of the moving body at time t + Δt in FIG. Using the coordinates 301 and 302 in FIG. 3A, Δh in FIG. 2B is predicted. The coordinates of the moving body are acquired from the image. For example, the coordinates of the moving body 301 at time t are acquired from the image 2000 of FIG. 2C from the coordinates of the center of gravity of the area of the moving body 230 in the image coordinate system having the lower left corner of the image as the origin. Motion prediction is performed by a prediction filter. Examples of the prediction filter include an α-β tracker and a Kalman filter, but the present invention is not limited to a specific prediction filter. The prediction filter predicts and estimates the current moving body observation position from the past moving body observation position based on the observation model and the motion model. As a motion model handled by the filter, as shown in FIG. 3A, a position on the image coordinates, a velocity, and an acceleration are internally provided, and a position of a moving body on the image can be observed as an observation model. In this case, the moving position (x ′, y ′) after Δt is predicted and estimated from the observation (x, y). Alternatively, as shown in FIG. 3B, a motion model may be used in which the direction (θ, φ) in which the object is viewed from the HMD is observable and the direction and its displacement are internally updated.

  FIG. 4 is a block diagram showing a functional configuration example of the information processing system. Here, a configuration is shown in which all the processing is performed inside the HMD. The HMD 200 has an imaging unit 410, an imaging optical system correction unit 430, a display optical system correction unit 440, and a display unit 420. Further, the information processing device 401 mounted on the HMD 200 has the following functional configuration. That is, the image acquisition unit 441, the moving body region detection unit 433, the moving body position prediction unit 434, the posture acquisition unit 431, the position / posture estimation unit 432, the background image generation unit 435, the CG image generation unit 437, the holding unit 438, the combined image generation unit. 439 and a display position control unit 442.

  The image capturing unit 410 has an image capturing optical system 411 and an image capturing sensor 412, and outputs the captured image to the information processing device 401. In the present embodiment, specifically, the image pickup unit 410 is a color camera, and the obtained image is a color image. The imaging unit 410 may be a stereo camera.

  The image acquisition unit 441 acquires from time to time an image (first image) obtained by the imaging unit 410 capturing an image of a moving body in the physical space. The acquired image is sent to the imaging optical system correction unit 430. The imaging optical system correction unit 430 corrects various aberrations such as the color of the captured image. On the other hand, the display unit 420 has a display optical system 421 and a display panel 422, and displays a corrected image. The correction at the time of display is performed by the display optical system correction unit 440. This is the reverse process of the imaging optical system correction unit 430.

  The moving body region detection unit 433 receives the captured image (first image) as input, and detects a moving body region in the image based on a predetermined image feature indicating the moving body. For example, when the moving body is a hand, the moving body region is extracted based on the image feature of the hand. Specifically, the skin color area may be extracted. The moving body position predicting unit 434 predicts the position of the moving body after a predetermined time according to the method using the prediction filter. In the present embodiment, Δt is a fixed value, and the moving space of the motion model is within the imaging screen. Therefore, in the moving body position prediction 434, it is not necessary to input the position and orientation of the HMD. Since Δt is a very short time of about 100 msec, it can be approximated as “the moving space of the moving body is within the captured image”. The use of a fixed value is expected to reduce the processing time and reduce the discomfort in the image. The background image generation unit 435 receives the moving body region detection result as an input and complements the moving body region in the captured image using the image feature of the background region to generate a background image. Specifically, the skin color area is complemented with the color of the background area included in the captured image.

  The posture acquisition unit 431 includes a gyro sensor and an accelerometer, and acquires a result of measuring relative posture information of the HMD. The posture information acquired here is used to predict the position of a specific object. The attitude acquisition unit 431 preferably uses an attitude sensor that can acquire attitude data at 100 Hz or higher and that has a small delay. The posture acquisition unit 431 transmits the measured posture information to the delay acquisition unit 1450 and the CG image generation unit 437. In addition, the posture acquisition unit 1300 transmits the latest posture information currently measured immediately before the synthetic image generation unit 439 generates an image to the synthetic image generation unit 439. Further, the posture acquisition unit 431 holds the posture measurement time and the posture measurement value in association with each other. When the posture measurement time is input, the posture acquisition unit 431 performs a process of returning the corresponding posture measurement value.

  The position / orientation estimation unit 432 estimates position / orientation information indicating the absolute position and orientation of the HMD used for superimposing the CG image depicting the CG model in correspondence with the real space. The position / orientation information indicates the absolute three-dimensional position and orientation of the HMD in the physical space. The position and orientation information is used to superimpose the CG image in correspondence with the real space. The position and orientation of the HMD is based on the world coordinate system set in the marker calibration step when setting the system. As a position / orientation estimation method, an index (marker) or a position / orientation estimation method using Simultaneous Localization And Mapping (SLAM) can be used. SLAM is a technique for estimating three-dimensional position / orientation information in the physical space of the image capturing apparatus from the markers shown in the captured image. Alternatively, it is possible to improve the accuracy by using a posture sensor (not shown), or the value may be directly obtained from an external position / posture sensor. The present invention is not limited to a particular HMD position / orientation acquisition method. Further, the position / orientation estimation unit 432 predicts the position / orientation of the HMD at time t + Δt from the position / orientation acquisition result. In estimating the position and orientation of the HMD, a prediction filter similar to that for predicting a moving object can be used.

  The CG image generation unit 437 receives the HMD position / orientation information obtained from the holding unit 438 and the attitude acquisition unit 431 as an input, and generates a CG image depicting a hand that is a moving body at a position visible from the position / orientation of the HMD at time t + Δt. Further, the CG image generation unit 437 generates a CG image in which the object 130 is rendered at a corresponding predetermined position in the real space in the background image based on the CG model data held in the holding unit 438. Based on the CG model data, a CG image is generated by rendering the object 130 at a corresponding predetermined position in the real space in the background image. Using the HMD position / orientation information obtained from the holding unit 438 and the attitude acquisition unit 431 as input, a CG image depicting a CG model at a position visible from the position / orientation of the HMD at time t + Δt is generated.

  The holding unit 438 holds the CG model data. In addition, a log of position and orientation measurement results is held. Further, it holds the image acquired by the image capturing unit 410.

  The composite image generation unit 439, based on the result of estimating the position of a specific object included in the background image (first image) after a predetermined time, draws a moving object at a position based on the estimation result in the background image. And a background image are combined to generate a combined image (second image). Alternatively, the moving body position prediction result based on the prediction filter, the HMD position / orientation prediction result, the background image, and the CG image are input. The shift amount of the image is calculated based on the HMD position / orientation prediction result, the background image is shifted, and the CG image and the moving body region are superimposed to generate the image 2070 of FIG. Here, regarding the superimposing order of the moving body and the CG image, when the distance from the HMD to the moving body is known, the drawing order is changed based on the front-rear relationship with the CG. The image generated by the composite image generation unit 439 is displayed on the display unit 420 via the display optical system correction unit 440.

  The display position control unit 442, based on the delay time caused by at least a part of the processing time in the processing from the image pickup to the display, and the amount of change in the position or orientation of the image display device, the composite image (first 2) shift the display position. The shifted image is displayed as shown in FIG. Here, a fixed value Δt is used as the delay time. It should be noted that when the synthesized image generation unit 439 synthesizes the shifted background image and the CG image, the processing here is omitted.

  FIG. 5 is a flowchart showing processing executed by the information processing device. The flow of processing performed by the information processing device 401 will be briefly described with reference to FIG. Hereinafter, it is assumed that the flowchart is realized by the CPU executing the control program. In the following description, each process (step) will be described by adding S to the beginning, and the description of the process (step) will be omitted. The processing shown in the flowchart of FIG. 5 is executed by the CPU 901 of FIG. 11, which is a computer, according to a computer program stored in the external storage device 906 or the like. However, the information processing device 401 does not necessarily have to perform all the steps described in this flowchart. Further, a plurality of processes may be simultaneously executed in parallel as in a sequence diagram described later.

  In S500, the information processing device 401 is initialized. Specifically, the CG image generation unit 1700 reads from the model data storage unit 1750 CG model data for rendering a given CG image at a predetermined position of a three-dimensional CG model. In step S501, the image acquisition unit 441 determines whether the image capturing process of the image capturing unit 410 is completed. If imaging has been completed and a new image can be acquired, a captured image of a specific object in the physical space is acquired and the process proceeds to S502. If imaging has not been completed, the process returns to S501. The captured image corresponds to the image 2000 shown in FIG. In S502, the imaging optical system correction unit 430 corrects various aberrations including the color of the captured image acquired from the imaging unit 401. In S503, the posture acquisition unit 431 acquires the posture information measured by the posture measurement sensor. The position / orientation estimation unit 432 may acquire the result of estimating the position / orientation information of the HMD at the time of imaging from the captured image. In S504, the position / orientation estimation unit 432 estimates the position / orientation of the HMD at time t + Δt from the position / orientation acquisition result in S503 and the past position / orientation information. In S505, the moving body region detection unit 433 acquires an object region indicating an object and a background region in which the object region is separated from the captured image (first image). The object area is the area 232 in FIG. The background area is the image 2030 in FIG.

  In S506, the moving body position prediction unit 434 estimates the position of the object after a predetermined time (after Δt) based on the object region in the captured image (first image). For example, in the image 2020 of FIG. 2E, a change from the position of the moving body 233 to the position of the moving body 234 after Δt seconds is predicted. Here, the position of the moving body at time t + Δt is predicted according to the method using the above-described prediction filter. Here, the predetermined time is the delay time Δt. In this embodiment, a fixed value of Δt prepared in advance is used. In S507, the CG image generation unit 437 receives the position and orientation information of the HMD obtained from the holding unit 438 and the orientation acquisition unit 431 as an input, and moves the moving body to the position predicted in S506 to a predetermined position corresponding to the real space. Generate a CG image depicting a CG model. The image 2080 in FIG. 2K is an image of the image of the hand 235 extracted from the image 2000 at the predicted position, and the object 130 that is the CG model on the object 220. In step S <b> 508, the background image generation unit 435 generates a background image by complementing the object area with the image characteristics (color and brightness) of the background area obtained from the periphery of the object area of the captured image (first generation). .. The image 2040 in FIG. 2G is a background image in which the area 242 is complemented by using the image feature of the background. For example, the object area is filled with the color of the background area. S507 and S508 may be processed in reverse order.

  In step S <b> 509, the synthetic image generation unit 439 draws the moving object at a position based on the estimation result in the background image based on the result of estimating the position of the specific object included in the background image (first image) after a predetermined time. A combined image (second image) is generated by combining the CG image and the background image. The image 2080 in FIG. 2K corresponds to the image generated in S509. It is depicted by synthesizing the moving body region at the corresponding position in the background image based on the position (estimation result) predicted in S506. In S510, the display optical system correction unit 440 converts the combined image into an image suitable for the display optical system. In step S511, the display position control unit 442 supplies to the display unit based on the delay time caused by at least a part of the processing time in the processing from the imaging to the display and the amount of change in the position or orientation of the image display device. The position for displaying the image (second image) is shifted. The procedure of S510 and S511 may be reversed. In S512, the information processing apparatus 401 determines whether or not there is a termination instruction from the user. If there is an end instruction, the process ends. If there is no end instruction, the process proceeds to S501.

  Alternatively, the processing after S508 may be performed as follows. In step S508, the background image generation unit 435 complements the object area using the image characteristics (color and brightness) of the background area, and generates a background image shifted based on the change in the position and orientation of the HMD. The image 2050 in FIG. 2H is a background image in which the area 242 is complemented using the image characteristics of the background and further shifted according to the change in the position and orientation of the HMD. In step S <b> 509, the composite image generation unit 439 uses the background image after the shift and the second image that depicts the object at the estimated position as the foreground, based on the moving body position prediction result and the HMD position / orientation prediction result. To generate a composite image (second image). In S510, the display optical system correction unit 440 converts the combined image into an image suitable for the display optical system. In S511, the display position control unit 442 displays the combined image at the position as it is. In S512, the information processing apparatus 401 determines whether or not there is a termination instruction from the user. If there is an end instruction, the process ends. If there is no end instruction, the process proceeds to S501.

  FIG. 6 is a sequence diagram showing processing executed by the information processing system. In FIG. 6, each processing of (a) photographing, (b) position / orientation acquisition, (c) CG image generation, (d) moving body region detection, (e) display image generation, and (f) display operates simultaneously in parallel. To do. The above processes are repeatedly performed. Note that the description of the end of the activation of each process is omitted.

  In the shooting process of FIG. 6A, the imaging unit 410 shoots the first image in S600. This corresponds to S501 in FIG. In step S601, the correction process of the first image of the imaging optical system correction unit 430 is performed. This corresponds to S502 in FIG. In the position and orientation acquisition process of FIG. 6B, the orientation acquisition unit 431 of S610 detects the position and orientation of the HMD at time t from the captured image. This corresponds to S503 in FIG. Next, in step S611, the position and orientation estimation unit 432 predicts the position and orientation of the HMD at t + Δt from the detected position and orientation. This corresponds to S504 in FIG. In this embodiment, Δt uses a fixed value. In the CG image generation process of FIG. 6C, in S620, the CG image generation unit 437 generates a CG image based on the position and orientation information of the HMD measured in S610. This corresponds to S507 in FIG. In the moving body area detection process of FIG. 6D, the moving body area detection unit 433 detects a moving body area from the captured image in S630. This corresponds to S505 in FIG. In S631, the moving body position predicting unit 434 predicts the moving body position at time t + Δt. This corresponds to S506 in FIG. In this embodiment, since the prediction in the screen of FIG. 3A is adopted, the position / orientation information of the HMD is unnecessary.

  In the background & display image generation process of FIG. 6E, in response to the moving body region detection result of S630, the background image generation unit 435 generates a background image in which the moving body region is filled in in S640. This corresponds to S508 in FIG. After that, in S643, the composite image generation unit 439 performs a composite image generation process. This corresponds to S509 in FIG. This process receives the results of the moving body position prediction S631, the HMD position / orientation prediction S611, and the CG image generation S620, and generates a composite image to be displayed on the HMD according to the latest position / orientation information. FIG. 6F is a display process. Upon receiving the composite image generated in S643, the display optical system correction unit 440 corrects the composite image in S650. This corresponds to S510 in FIG. In step S652, the display position control unit 442 supplies the composite to the display unit based on the delay time caused by at least a part of the processing time in the processing from the imaging to the display and the amount of change in the position or orientation of the image display device. The position for displaying the image (second image) is shifted. In step S553, the display unit 420 sequentially performs image display processing to display an image. This corresponds to S511 in FIG.

  As described above, according to the present embodiment, the discomfort of the HMD user can be improved by considering the motion of the moving body in the time warp image generation of the image shift method for the purpose of reducing the perceptual delay of the HMD. ..

(Modification 1)
As a modified example of the present invention, it is possible to improve the prediction accuracy by setting the coordinates used for the prediction of the moving body in the θφ space which is the posture of the moving body as viewed from the HMD. In addition, the prediction accuracy can be improved by making the delay time Δt from photographing to display variable. It is more accurate to set Δt based on the measurement result of the delay time between the actual shooting and the display when performing the prediction based on the motion model, so that the discomfort of the image can be further reduced. FIG. 7 shows a functional configuration example of this modification. Hereinafter, the difference from FIG. 4 will be mainly described.

  In this modification, a delay measuring unit 641 is provided to measure the difference Δt between the shooting time and the display time. The delay measuring unit 641 has a timer inside. The delay time Δt is the difference between the time when the imaging optical system correction unit 630 acquires the captured image (first time) and the time when the display optical system correction unit 640 completes the transmission of the display image (second time). It is decided based on. The delay time Δt is attached to the captured image and used for prediction in the moving body position predicting unit 634 and the HMD position / posture estimating unit 632. The position / orientation acquisition unit 631 is the same as in the first embodiment 431. The position / orientation acquisition unit 631 transmits to the delay acquisition unit 641 the measurement time of the latest orientation that is currently being measured.

  The moving body region detection unit 633 detects the moving body position (orientation) in the θφ space as shown in FIG. This indicates in which direction the moving body can be seen from the HMD. The moving body position predicting unit 634 predicts the moving body position (azimuth) in the θφ space. The background image generation unit 635 is the same as 435. The predicted background image generation unit 636 calculates the shift amount of the background image at time t + Δt based on the position / orientation prediction of the HMD, and shifts the image. Similarly, the CG image generation unit 637 also generates a CG model image viewed from the predicted position and orientation of the HMD at time t + Δt. The synthetic image generation unit 639 generates a predicted background image, a CG image, and an image in which moving objects are combined. If the moving object is found to be in front of the CG as a result of distance measurement, the moving object is superimposed last. This applies, for example, when the moving body is the hand of the HMD wearer.

  FIG. 8 is a flowchart showing processing executed by the information processing device. In S500, the information processing device 401 is initialized. In S501, the image pickup optical system correction unit 630 determines whether the image pickup process of the image pickup unit 410 is completed. If imaging has been completed and a new image can be acquired, a captured image of a specific object in the physical space is acquired and the process proceeds to S800. If imaging has not been completed, the process returns to S501. In S800, the delay measuring unit 641 measures the difference Δt between the shooting time and the display time. In S502, the imaging optical system correction unit 630 corrects various aberrations such as the color of the captured image acquired from the imaging unit 401. In S503, the position and orientation acquisition unit 631 acquires the orientation information of the HMD at the time of image capturing from the captured image. In S504, the position / orientation estimation unit 632 estimates the position / orientation of the HMD at time t + Δt from the position / orientation acquisition result in S503 and the past position / orientation information. In step S505, the moving body region detection unit 633 acquires an object region indicating an object and a background region in which the object region is separated from the captured image (first image). In S506, the moving body position prediction unit 634 estimates the position of the object after the delay time based on the object region in the captured image (first image). In S507, the CG image generation unit 437 receives the position and orientation information of the HMD obtained from the holding unit 438 and the orientation acquisition unit 431 as input, and sets the moving body at the position predicted in S506 and the CG at a predetermined position corresponding to the physical space. The model and the CG image are generated. In step S508, the background image generation unit 635 generates a background image by complementing the object area with the image features (color and brightness) of the background area. In step S <b> 509, the composite image generation unit 639 draws the moving object at the position based on the estimation result in the background image based on the result of estimating the position of the specific object included in the background image (first image) after a predetermined time. A combined image (second image) is generated by combining the CG image and the background image. In S510, the display optical system correction unit 640 converts the composite image into an image suitable for the display optical system. In step S511, the display position control unit 642 supplies the composite to the display unit based on the delay time caused by at least a part of the processing time in the processing from the imaging to the display and the amount of change in the position or orientation of the image display device. The position for displaying the image (second image) is shifted. In S801, the delay measuring unit 641 records the delay time required for the current process. The delay time is held in the holding unit 638. In S512, the information processing device 601 determines whether or not there is an end instruction from the user. If there is an end instruction, the process ends. If there is no end instruction, the process proceeds to S501.

  FIG. 9 is a sequence showing a process executed by the information processing system. Similar to FIG. 6, a plurality of processes operate in parallel. Hereinafter, description will be made focusing on parts different from FIG. In order to measure the delay time Δt between the shooting and the display, the imaging optical system correction unit 630 acquires the shooting time in S702 of the shooting process (a). In addition to the display time acquisition information in S751, the delay measuring unit 641 performs delay measurement in S703. In the position and orientation acquisition process (b), the position and orientation prediction process is performed in S711. In S720 of the CG image generation processing (c), the position / orientation prediction result is received, and a CG image from the predicted position / orientation is generated. In S731 of the moving body area detection process (d), the moving body position in the θφ space is predicted based on the position and orientation measurement result of the HMD. In S742 of the background & display image generation process (e), the background image at the time t + Δt is generated in response to the result of the S711 position / orientation prediction process. In S750 of the display process (f), the display position control unit 642 and the display optical system correction unit 640 convert the composite image into an image suitable for the display optical system. In step S752, the display position control unit 642 supplies the composite image to the display unit based on the delay time caused by at least a part of the processing time in the process from imaging to display and the amount of change in the position or orientation of the image display device. The position for displaying (second image) is shifted. In S751, the delay measuring unit 641 acquires the display time. In S753, the display unit displays an image.

(Modification 2)
As another modification related to the present invention, as shown in FIG. 10, it is also possible to realize a configuration in which the HMD and the control device are separated. The difference from FIG. 6 will be mainly described below. The system consists of HMD 800 and controller 850, which communicate with each other. The HMD 800 has a data transmission unit 840 and a data reception unit 841. The control device is composed of, for example, a commercially available PC, and has a data receiving unit 832 and a data transmitting unit 853. The data transmission unit 840 of the HMD collectively transmits the captured image, the delay measurement result, and the posture information to the control device 850, and the reception unit 852 of the control device 850 receives this. On the other hand, the data transmitting unit 853 of the control device 850 transmits the composite image generated by the composite image generating unit 862 to the HMD 800, and the data receiving unit 841 receives the composite image. The communication between the HMD 800 and the control device 850 can be selected from USB, IEEE1394, and LAN, and is not limited to a specific communication method. In the software configuration of this modification, in the software configuration of Modification 1, the shooting process (a) and the display process (f) are arranged on the HMD side. Further, the position / orientation acquisition process (b), the CG generation process (c), the moving body region detection process (d), and the background & display image generation process (e) are arranged on the control device side to communicate with each other. It can be realized by

(Other embodiments)
FIG. 11 is a schematic diagram showing hardware for realizing the information processing apparatus of the above embodiment. The CPU 901 controls the entire computer using computer programs and data stored in the RAM 907 and the ROM 202. Further, the CPU also executes each processing described as being executed by the information processing apparatus in each of the following embodiments. The RAM 907 temporarily stores the computer programs and data loaded from the external storage device 906 and the storage medium drive 905. Further, the RAM 907 has an area for temporarily storing data received from the outside. Further, the RAM 907 also has a work area used when the CPU 901 executes each process. That is, the RAM 907 can appropriately provide various areas. Further, the ROM 902 stores computer setting data, a boot program, and the like. The keyboard 909 and the mouse 908 are examples of an operation input device, and various instructions can be input to the CPU 901 by being operated by a computer user. The display unit 904 is composed of a CRT, a liquid crystal screen, or the like, and can display the processing result by the CPU 901 as an image or characters. For example, the display unit 904 can display a combined image obtained by combining the image of the physical space captured by the imaging device 410 and the virtual image. The external storage device 906 is a large capacity information storage device represented by a hard disk drive device. The external storage device 906 stores an OS (operating system) and programs and data for causing the CPU 901 to execute each process performed by the information processing device. The computer programs and data stored in the external storage device 906 are appropriately loaded into the RAM 907 under the control of the CPU 901. The CPU 901 executes each process using the loaded program and data, thereby executing each process performed by the information processing apparatus. The storage medium drive 905 reads programs and data recorded in a storage medium such as a CD-ROM and a DVD-ROM, and writes a computer program and data in the storage medium. Incidentally, some or all of the programs and data described as being stored in the external storage device 906 may be recorded in this storage medium. The computer program and data read from the storage medium by the storage medium drive 905 are output to the external storage device 906 and the RAM 907. The I / F 903 is configured by an analog video port for connecting the imaging device 410 or a digital input / output port such as IEEE1394. The data received via the I / F 903 is input to the RAM 907 or the external storage device 906. The bus 910 connects the above-described components with a bus signal.

  The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

400 Information Processing System 100 Information Processing Device 200 Head Mounted Display

Claims (10)

Information processing that supplies the second image to an image display device that includes an image capturing unit that acquires a first image obtained by capturing an image of a physical space and a display unit that displays a second image generated using the first image A device,
First generation means for generating a background image in which a region of a specific object included in the first image is complemented using image features obtained from the periphery of the region included in the first image;
Based on the result of estimating the position of the specific object included in the first image after a predetermined time, the second image obtained by synthesizing the image depicting the object at the position based on the estimation result in the background image is displayed. Second generating means for generating,
Based on the delay time caused by at least a part of the processing time in the processing from the image pickup to the display, and the amount of change in the position or orientation of the image display device, the position for displaying the second image to be supplied to the display means is set. An information processing apparatus, comprising: a display position control unit that shifts.   The information processing according to claim 1, wherein the predetermined time is a difference between a first time when an image is captured by the image capturing device and a second time when the position or orientation of the image display device is measured. apparatus.   The information processing apparatus according to claim 1, wherein the first generation unit generates the area having a predetermined image feature from the first image.   The information processing apparatus according to claim 3, wherein the first generation unit generates the area having an image feature of a hand from the first image. The object is a hand,
The said 1st production | generation means produces | generates the said background image by complementing the skin color area | region in the said 1st image with the color of the background area contained in the said 1st image. Processing equipment.   The second generation unit generates the second image by synthesizing the image obtained by extracting the region of the specific object generated by the first generation unit, at a position based on the estimation result in the background image. The information processing apparatus according to any one of claims 1 to 5, characterized in that:   7. The information processing device according to claim 1, wherein the image display device is a head mounted display. Information processing that supplies the second image to an image display device that includes an image capturing unit that acquires a first image obtained by capturing an image of a physical space and a display unit that displays a second image generated using the first image A device,
At least a part of the processing time in the processing from the image pickup to the display is performed by complementing the area of the specific object included in the first image with the image feature obtained from the periphery of the area included in the first image. First generation means for generating a background image obtained by shifting the first image on the basis of the delay time caused by and the amount of change in the position or orientation of the image display device,
An image in which the object is depicted at a position in the background image based on the estimation result, based on a result of estimating a position of the specific object included in the first image after a predetermined time, and the first generation means. An information processing device, comprising: a second generation unit configured to generate the second image by combining the generated background image.   A program for causing a computer to function as each unit of the information processing apparatus according to claim 1. Information processing that supplies the second image to an image display device that includes an image capturing unit that acquires a first image obtained by capturing an image of a physical space and a display unit that displays a second image generated using the first image Method,
A first generation step of generating a background image in which a region of a specific object included in the first image is complemented using image features obtained from the periphery of the region included in the first image;
Based on the result of estimating the position of the specific object included in the first image after a predetermined time, the second image obtained by synthesizing the image depicting the object at the position based on the estimation result in the background image is displayed. A second generation step for generating,
Based on the delay time caused by at least a part of the processing time in the processing from the image pickup to the display, and the amount of change in the position or orientation of the image display device, the position for displaying the second image to be supplied to the display means is set. A display position control step of shifting.

Images