JP2019062397A - IMAGE PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, AND IMAGE PROCESSING METHOD - Google Patents

Abstract

To achieve reduction in data volume without increasing a processing delay in various types of image processing by executing frame decimation processing of moving image data.SOLUTION: A plurality of moving image data with an optional frame rate are input, decimation processing of frame number is executed for each of the plurality of moving image data without changing a frame period, and decimation moving image data created by reducing data volume by means of the decimation processing is output. At that time, the frame decimation processing is controlled so that a total of the data volume of the moving image data subjected to decimation processing is accommodated within a range of data volume capable of being output.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technology for reducing the amount of transmission of moving image data.

  In recent years, so-called MR (Mixed Reality) technology is known as a technology for seamlessly merging the real world and the virtual world in real time and seamlessly. One of the MR techniques is a video see-through HMD (Head Mounted Display: hereinafter referred to as "HMD"). In this system, an object substantially corresponding to the object observed from the pupil position of the HMD wearer is imaged by a video camera or the like, and the HMD wearer can observe an image in which CG (Computer Graphics) is superimposed and displayed on the imaged image.

  FIG. 15 is a functional block diagram of a general mixed reality system (hereinafter referred to as an MR system) using a video see-through HMD. The outline is explained using this figure.

  The video see-through HMD 1501 includes an imaging unit 1503, a display unit 1504, a position and orientation sensor 1505, and an HMD side interface 1506. The image processing apparatus 1502 includes an apparatus-side interface 1507, a position and orientation information generation unit 1508, a content DB (database) 1509, and a CG rendering and combining unit 1510. The image processing apparatus 1502 draws CG based on the captured image and three-dimensional position and orientation information received from the HMD 1501, and performs composition processing with the captured image. In general, a device having a high performance arithmetic processing function and a graphic display function such as a personal computer and a work station is used.

  The imaging unit 1503 of the HMD 1501 captures an observation image of the external world that substantially matches the gaze position of the HMD user. Generally, in order to generate a stereo image, it comprises two imaging elements for the right eye and for the left eye, an optical system, and a DSP (Digital Signal Processor) for performing image processing in the subsequent stage.

  The display unit 1504 displays the synthesized MR image. This also consists of two display devices for the right eye and for the left eye and an optical system. As a display device, a small liquid crystal display or a device of a retina scan type by MEMS is used.

  The position and orientation sensor 1505 acquires sensor information for obtaining the position and orientation information of the HMD user. As the position and orientation sensor, a magnetic sensor, a gyro sensor (acceleration, angular velocity) or the like is used.

  The HMD side interface 1506 transmits the image captured by the imaging unit 1503 and the position and orientation sensor information acquired by the position and orientation sensor 1505 to the image processing apparatus 1502, and transmits the synthesized MR image to the HMD 1501.

  The apparatus side interface 1507 is an interface on the image processing apparatus side, and performs data transmission with the HMD side interface 1506.

  The position and orientation information generation unit 1508 generates position and orientation information of the HMD user based on the received captured image and the position and orientation sensor information.

  The content DB 1509 is a database storing the content of the virtual image.

  The CG rendering composition 1510 renders CG based on the position and orientation information and the information of the content DB, and composes the CG with the captured image. The MR image obtained here is sent to the HMD 1501 via the apparatus side interface 1507 and the HMD side interface 1506, and is displayed on the display unit 1504.

  Generally, for transmission between the HMD side interface 1506 and the device side interface 1507, a multi-core cable using metal wires, optical fibers, etc. in order to realize real-time performance required for the MR system and large-capacity data transmission. Wired connection is used. In order to allow the HMD wearer to experience the MR space more freely without being restricted by the movement by the cable, the thinning of the cable used for data transmission between the HMD side interface and the device side interface and wireless transmission Is desired. For example, by combining data amount reduction methods such as image coding and frame rate conversion, it is possible to thin cables, and there is also an improvement in wireless transmission technology in recent years. It has also become possible to carry out transmissions.

  The method for generating left-eye image data and right-eye image data from moving image data having a frame rate of 1/2 of the display frame rate, and outputting left-eye image data and right-eye image data in time division is as follows: Patent Document 1 discloses the following.

JP, 2015-39076, A

  However, in the method of Patent Document 1, by using moving image data of a frame rate of 1/2 of the frame rate of the display, accumulation of processing delay when performing a plurality of image processing on the moving image data It will be huge. For example, when the frame rate of the display is 60 fps, the transmission time per frame is 1/60 seconds, but for 30 fps moving image data which is a half frame rate, the transmission time per frame is 1 It will be / 30 seconds. That is, when performing image processing using a frame memory, the difference in time until one frame's worth of data accumulates in the frame memory is 1/60 seconds, which is the process until it is finally displayed on the display It will appear as a delay. Therefore, the more the image processing using the frame memory, the line memory, etc., the larger the difference in processing delay in the image processing between the 60 fps moving image data and the 30 fps moving image data.

  Therefore, an object of the present invention is to realize reduction of the data amount without increasing processing delay in various image processing by performing frame thinning processing of moving image data.

  In order to solve the above problems, according to the present invention, first force means for inputting at least two or more pieces of moving image data having an arbitrary frame rate, and the plurality of moving image data, Image processing means for performing image processing by a plurality of image processing circuits, frame thinning means for performing a thinning process on the number of frames without changing the frame period for each of the plurality of moving image data, and the image Processing means, control means for controlling the frame thinning-out means, and first output means for outputting thinned-out moving image data whose data amount has been reduced by the frame thinning-out means, the control The means is arranged such that the sum of the data amounts of the plurality of motion picture data subjected to thinning processing falls within the range of data amounts that can be output by the first output means. Controls image processing unit, and the frame thinning means, said first output means, characterized by time-division outputs each frame decimation processing said plurality of moving image data.

  According to the above configuration, by performing frame thinning processing of moving image data, the present invention can realize reduction of the data amount without increasing processing delay in various image processing.

It is a figure which shows the apparatus structure of MR system using the display apparatus with an imaging function. It is a figure explaining the image composition in MR system. BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows an example of a structure of the image processing system in the 1st Embodiment of this invention. It is a figure explaining accumulation of a delay in image processing to moving picture data in which frame rates differ. FIG. 6 is a diagram for explaining frame thinning processing and accumulation of delay in processing from the imaging unit to the position and orientation information generation unit of the image processing system according to the first embodiment of the present invention. It is a figure explaining the frame thinning-out process in the process from the CG drawing synthetic | combination part of the image processing system in the 1st Embodiment of this invention to a display part, and accumulation of a delay. It is a figure which shows an example of a structure of the image processing system in the 2nd Embodiment of this invention. It is a figure explaining the image composition which considered depth information. It is a figure explaining depth information generation of the arbitrary subjects contained in the image pick-up picture for left eyes and right eyes. It is a figure explaining accumulation of frame thinning-out processing and delay in processing from an imaging part of an image processing system to a depth information generation part in a 2nd embodiment of the present invention. It is a figure explaining the frame thinning-out process in the process to the display part from the CG drawing synthetic | combination part of the image processing system in the 2nd Embodiment of this invention, and the accumulation | storage of a delay. It is a figure explaining the composition which performs data transmission between HMD of a graphics processing system in a 3rd embodiment of the present invention, and a graphics processing device by cable transmission using a cable. It is a figure explaining the composition which performs data transmission between HMD of an image processing system in a 3rd embodiment of the present invention, and an image processing device by radio transmission via an HMD side radio box and an apparatus side radio box. It is a figure which shows an example of a structure of the image processing system in the 4th Embodiment of this invention. It is a functional block diagram of the common MR system using video see-through HMD.

First Embodiment
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an apparatus configuration of an MR system using a display device with an imaging function.

  An MR system using a display device with an imaging function includes an HMD (Head Mounted Display) 101, a controller 102, and a PC 103. The HMD 101 includes an imaging unit for acquiring a captured image of a real space of a field of view observed by the user. The display unit also has a display unit for providing a display image such as a captured image, an input image from the PC 103, or a composite image obtained by superimposing a CG image generated by the PC 103 on the captured image. Although in FIG. 1 the HMD 101 and the controller 102 are connected by wire, they may be connected wirelessly. The HMD 101 can be driven by receiving power supply from the controller 102 or can be driven by a battery. One of the HMD 101, the controller 102, and the PC 103 includes a position and orientation information generation unit for generating three-dimensional position and orientation information of the HMD user. The three-dimensional position and orientation information is generated using image information acquired by an imaging unit of the HMD or an objective camera separately installed, and various sensor information such as an acceleration sensor, a gyro sensor, and a magnetic sensor. The PC 103 connected to the controller 102 has a CG drawing / synthesizing unit that draws and synthesizes a CG image observed from the viewpoint of the HMD user based on three-dimensional position and orientation information. The controller 102 has various image processing functions such as image resolution conversion, color space conversion, distortion correction of an optical system, and image coding, and a transmission function. Although the controller 102 and the PC 103 have different hardware configurations in FIG. 1, the functions of the controller 102 and the PC 103 can be collected and configured as a dedicated image processing apparatus 104. In the following description, a combination of functions of the controller 102 and the PC 103 will be described as the image processing apparatus 104 from the functional viewpoint.

  FIG. 2 is a diagram for explaining image synthesis in the MR system. The captured image 201 is an image obtained by capturing an image of a physical space by an imaging unit of the HMD. In the HMD for the MR system, it is preferable that the central optical axis of the imaging range of the imaging unit be arranged so as to substantially coincide with the line of sight of the HMD user. The marker 202 in the captured image is an index used to calculate the position and orientation of the HMD user. The CG image 203 is an image drawn based on the position and orientation information by the CG drawing and combining unit in the image processing apparatus, and the CG object 204 in the virtual space is drawn. The CG image composition unit superimposes the CG image 203 on the captured image 201 to generate the MR image 205, and displays the image on the display unit of the HMD, thereby providing an MR space to the HMD user. It is also possible to generate a translucent composite image through which a CG image is transmitted by using information on depth in a three-dimensional space at the time of image composition.

  FIG. 3 is a diagram showing an example of the configuration of the image processing system according to the first embodiment of the present invention. The image processing system according to the first embodiment of the present invention includes an HMD 101 and an image processing apparatus 104 having the following configuration. The HMD 101 includes an imaging unit 301, a position and orientation sensor 302, an HMD side captured image processing unit 303, an HMD side decimation processing unit 304, an HMD side captured image control unit 305, and an HMD side common captured image processing unit 306. Further, it has an HMD side I / F (interface) 307, an HMD side common display image processing unit 320, an HMD side display image control unit 321, a display frame memory 322, an HMD side display image processing unit 323, and a display unit 324.

  The image processing apparatus 104 includes an apparatus-side I / F 308, an apparatus-side shared captured image processing unit 309, and an apparatus-side captured image control unit 310. The imaging frame memory 311, the apparatus-side captured image processing unit 312, the position and orientation information generation unit 313, the content DB (database) 314, the CG rendering / composition unit 315, and the apparatus-side display image processing unit 316. Further, the apparatus-side thinning processing unit 317, the apparatus-side display image control unit 318, and the apparatus-side common display image processing unit 319 are included.

  The HMD-side I / F 307 and the device-side I / F 308 each have a transmitter and a receiver. Further, the notation L or R at the end of each block name means that processing of left-eye moving image data and processing of right-eye moving image data are performed, respectively.

  The imaging unit 301 of the HMD 101 acquires a captured image of the physical space as a stereo image having a parallax that substantially matches the viewing positions of the left eye and the right eye of the HMD user. The imaging unit includes an imaging device and an imaging optical system. As the imaging device, for example, a CMOS image sensor, a CCD image sensor, or the like is used.

  The position and orientation sensor 302 acquires information for calculating three-dimensional position and orientation information regarding the position and direction of the viewpoint of the HMD user. As the position and orientation sensor 503, for example, an objective camera, a magnetic sensor, an acceleration sensor, an angular velocity sensor, or the like is used. The position and orientation sensor 503 does not necessarily have to be provided in the HMD 101. For example, the necessary information may be obtained from a captured image of an objective camera installed around the HMD user.

  The HMD-side captured image processing unit 303 performs image processing on the left-eye captured image and the right-eye captured image acquired by the imaging unit using a plurality of image processing circuits. The image processing on the left-eye captured image and the image processing on the right-eye captured image are executed by parallel processing by separate processing circuits. As the image processing performed here, processing using different parameters for the left-eye captured image and the right-eye captured image is desirable. For example, processing for correcting individual variation of an imaging device or an imaging optical system such as pixel defect correction, gamma correction, or distortion correction of an imaging optical system may be mentioned.

  The HMD-side decimation processing unit 304 performs frame decimation processing for reducing the amount of data on the left-eye captured image and the right-eye captured image. In the frame skipping process, the frame period itself is not changed simultaneously with the reduction of data amount as in the so-called frame rate conversion, and the frame period does not change between input and output, and reduces the number of frames sent to the latter stage. Thus, the amount of moving image data is reduced.

  The HMD-side captured image control unit 305 controls the intervals and timings of frame thinning processing by the HMD-side thinning processing unit, and performs time-division of the captured images for the left eye and the right eye after frame thinning processing in frame units. Control to send to the subsequent HMD side common captured image processing unit 306 is performed. Here, the total number of frames of the left-eye and right-eye captured images after frame thinning falls within the number of frames that can be processed by the image processing unit in the subsequent stage and the number of frames that can be transmitted by the HMD I / F 307 To be controlled.

  The HMD-side common captured image processing unit 306 performs image processing in time division on the captured images for the left eye and the right eye after the frame thinning processing. As the image processing performed here, for example, processing that can use the same parameter for the left-eye captured image and the right-eye captured image, such as resolution conversion, color space conversion, or image coding, is desirable.

  The HMD I / F 307 communicates with the device I / F 308 of the image processing device 104 to transmit the left-eye captured image, the right-eye captured image, and sensor information acquired by the position and orientation sensor 302. The display unit 324 also receives a display image to be displayed, and transmits and receives other control signals.

  The HMD-side common display image processing unit 320 performs image processing in time division on the display images for the left eye and the right eye after the frame thinning processing. As the image processing performed here, for example, processing capable of using the same parameter for the display image for the left eye and the display image for the right eye, such as resolution conversion, color space conversion, or image decoding, is desirable.

  The HMD display image control unit 321 reconstructs display images for the left eye and the right eye having a frame rate that can be displayed on the display unit 324 from the display images for the left eye and the right eye after the frame thinning process. Specifically, the left-eye and right-eye display images after frame thinning processing are distributed and stored in the left-eye and right-eye display frame memories 322, respectively, and are updated according to the frame rate of the display unit 324. Control of outputting the display image of the display frame from the display frame memory.

  The display frame memory 322 stores the display image for the left eye and the display image for the right eye, and outputs the display image to the HMD display image processing unit 323 under the control of the HMD display image control unit 321.

  The HMD-side display image processing unit 323 performs image processing on the display image for the left eye and the display image for the right eye by a plurality of image processing circuits. The image processing on the left-eye display image and the image processing on the right-eye display image are executed by parallel processing by separate processing circuits. As image processing performed here, processing using separate parameters for the display image for the left eye and the display image for the right eye is desirable. For example, processing for correcting individual variation of a display device or a display optical system such as offset adjustment, gain adjustment, or distortion correction of a display optical system can be mentioned.

  The display unit 324 displays an MR image on which CG is superimposed on the left eye and the right eye of the HMD user. Although not shown in FIG. 3, the display unit 324 includes a display device and a display optical system. As this display device, for example, a small liquid crystal display, an organic EL display, a device of a retinal scan type by MEMS, or the like is used.

  The device-side I / F 308 of the image processing device 104 communicates with the HMD-side I / F 307 of the HMD 101 to receive the left-eye captured image, the right-eye captured image, and the sensor information acquired by the position and orientation sensor 302. Further, transmission of a display image to be displayed on the display unit 324 and transmission and reception of other control signals are performed.

  The device-side common captured image processing unit 309 performs image processing on the captured images for the left eye and the right eye after frame thinning processing in a time division manner. As the image processing performed here, for example, processing capable of using the same parameter in the captured image for the left eye and the captured image for the right eye, such as resolution conversion, color space conversion, or image decoding, is desirable.

  The apparatus-side captured image control unit 310 distributes the left-eye and right-eye captured images after the frame thinning processing to the left-eye and right-eye imaging frame memories 311, respectively. Then, control is performed to output a captured image from the imaging frame memory 311 in accordance with a frame rate that can be displayed on the display unit 324.

  The imaging frame memory 311 stores the left-eye imaging image and the right-eye imaging image, and outputs the imaging image to the apparatus-side imaging image processing unit 312 based on the control of the apparatus-side imaging image control unit 310.

  The device-side captured image processing unit 312 performs image processing on the left-eye captured image and the right-eye captured image output from the imaging frame memory 311 by a plurality of image processing circuits. The image processing on the left-eye captured image and the image processing on the right-eye captured image are executed by parallel processing by separate processing circuits. As the image processing performed here, similarly to the HMD-side captured image processing unit 303, a process using separate parameters for the left-eye captured image and the right-eye captured image is desirable. Further, in consideration of the processing load and the processing order, the image processing to be executed between the apparatus-side captured image processing unit 312 and the HMD-side captured image processing unit 303 may be dispersed.

  The position and orientation information generation unit 313 calculates three-dimensional position and orientation information of the HMD 101 from the sensor information acquired by the position and orientation sensor 302, the captured image information acquired by the imaging unit 301, or both information.

  The content DB 314 is a database that stores content for CG drawing. The CG rendering / synthesizing unit 315 renders a CG image and synthesizes a captured image and a CG image based on the three-dimensional position and orientation information, and generates a display image for MR experience.

  The device-side display image processing unit 316 performs image processing on the display image for the left eye and the display image for the right eye generated by the CG rendering and combining unit by a plurality of image processing circuits. The image processing on the left-eye display image and the image processing on the right-eye display image are executed by parallel processing by separate processing circuits. As the image processing to be performed here, similarly to the HMD-side display image processing unit 323, it is desirable to use different parameters for the display image for the left eye and the display image for the right eye. Further, in consideration of the processing load and the processing order, the image processing to be executed between the apparatus-side display image processing unit 316 and the HMD-side display image processing unit 323 may be dispersed.

  The device-side decimation processing unit 317 performs frame decimation processing for reducing the data amount on the display image for the left eye and the display image for the right eye.

  The apparatus-side display image control unit 318 controls the intervals and timings of frame thinning processing by the device-side thinning processing unit, and displays the display images for the left eye and the right eye after frame thinning processing in time division in frame units. Control is performed to send to the device-side common display image processing unit 319 in the latter stage. Here, the total number of frames of the display image for the left eye and the right eye after the frame thinning process falls within the number of frames that can be processed by the image processing unit in the subsequent stage and the number of frames that can be transmitted by the device-side I / F 308 To be controlled.

  The device-side common display image processing unit 319 performs time-division image processing on the display images for the left eye and the right eye after the frame thinning processing. As the image processing performed here, processing that can use the same parameter for the display image for the left eye and the display image for the right eye is desirable as in the case of the HMD side common display image processing unit 320. Further, in consideration of the processing load and the processing order, the image processing to be executed between the device-side common display image processing unit 319 and the HMD-side common display image processing unit 320 may be dispersed.

  With the above configuration, the wearer of the HMD 101 can experience a highly realistic MR image in which a captured image and a CG image are combined.

  In addition, although the component of an image processing system exists also except the above, since it is not the main point of this invention, description is abbreviate | omitted.

  FIG. 4 is a diagram for explaining accumulation of delays in image processing for moving image data having different frame rates.

  In FIG. 4, a case will be described in which a process of displaying on the display unit is performed via the image processing A, the image processing B, the transmission, and the image processing C on the captured image acquired by the imaging unit.

FIG. 4A shows accumulation of delay in image processing for moving image data at a frame rate of 60 fps. If set to 0 the start time of the imaging of the captured image C ₁ of the imaging unit, the completion time of the acquisition of the captured image C ₁ is the post 1/60 seconds. Subsequent image processing A If the image processing which requires a frame memory of one plane image, the image processing A is started, i.e. the processed image of PA ₁ is output after acquisition completion of the captured image C ₁ It will be from. Image processing B If a process that requires a line memory of the image 1/3 screen portion, the processing by the image processing B image PB ₁ is output, the processed image PA ₁ after 1/3 screen portion output It becomes. Similarly, when each processing delay is accumulated in consideration of frame memory, line memory, etc. necessary for transmission, image processing C, and display, respectively, the captured image C ₁ is finally displayed on the display unit as a display image. time displayed initiated as D ₁ becomes T1.

FIG. 4B shows accumulation of delay in image processing for moving image data of frame rate 30 fps. If set to 0 the start time of the imaging of the captured image C ₁ of the imaging unit, the completion time of the acquisition of the captured image C ₁ is the post 1/30 seconds. Subsequently, in the image processing A, since it requires a frame memory of the image one plane, as described above, it becomes 1/30 after the processed image PA ₁ is output. As in the case of 60 fps, when each processing delay is accumulated in consideration of the image processing B, transmission, image processing C, frame memory required for display, line memory, etc., the captured image C ₁ is final display start is the time as the display image D ₁ becomes T2 in displaying unit.

  As described above, it is possible to confirm that the difference ΔT between T2 and T1 is generated by performing various image processing on moving image data having different frame rates. Further, although not shown in FIG. 4, in fact, in addition to the delay for the memory required for each process, the processing time required for each process is also accumulated as a delay time. For example, when processing is performed using a pixel clock of moving image data in each image processing, and processing delay in pixel clock units occurs, the value of ΔT becomes larger.

  FIG. 5 is a diagram for explaining frame thinning processing and accumulation of delay in processing from the imaging unit to the position and orientation information generation unit of the image processing system according to the first embodiment of the present invention.

  The image processing system in the present embodiment will be described assuming that the frame rate of the imaging unit 301 of the HMD 101 and the frame rate of the display unit 324 are 60 fps.

In the imaging unit 301, imaging of a left-eye captured image CL _n and a right-eye captured image CR _n (n is an integer of 1 or more) is performed. Assuming that the imaging start time of the _first left-eye captured image CL ₁ and the first right-eye captured image CR ₁ is 0, the acquisition completion time of the captured images CL ₁ and CR ₁ is 1/60 seconds later. Become.

Subsequently, in the HMD-side captured image processing unit 303, the captured image processing for the left eye and the right eye is performed in parallel, and processed images P ₁ L _n and P ₁ R _n (n is an integer of 1 or more) are generated.

In the HMD side thinning processing unit 304, even-numbered frames from the left-eye processed image P ₁ L _n and the right-eye processed image P ₁ R _n to reduce the transmission data amount of the captured image from the HMD 101 to the image processing apparatus 104 Processing to thin out each odd frame is performed. The processed images P ₁ L _{(2 n -1)} and P ₁ R _{2 n} (n is an integer of 1 or more ₎ thinned out by this process are output to the subsequent processing blocks in time division. Here, in order to reduce the data amount to 1⁄2, the even and odd frames are thinned out on the left and right, respectively, and output is performed alternately, but the thinning processing in the present embodiment is not limited to this, Arbitrary frame skipping may be performed in a range where time-division output is possible.

The HMD-side common captured image processing unit 306 executes common image processing on the left-eye captured image and the right-eye captured image that are time-divisionally output, and the processed image P ₂ L _(2n−1) , P ₂ R _2n (n is an integer of 1 or more) is generated. In this way, by performing image processing commonly executed on the left-eye captured image and the right-eye captured image in the latter stage of thinning processing, it is possible to realize commonality of circuits by time division processing and power saving. Is possible.

Subsequently, transmission of transmission images T ₁ L _{(2 n -1)} and T ₁ R _{2 n} (n is an integer of 1 or more) is performed via the HMD side interface 307 and the device side interface 308.

In the device-side common captured image processing unit 309, T ₁ L _{(2 n -1)} and T ₁ R _{2 n} (n is an integer of 1 or more ₎ transmitted by time division as in the HMD-side common captured image processing unit 306 Common image processing is performed. As a result, processed images P ₃ L _{(2 n -1)} and P ₃ R _{2 n} (n is an integer of 1 or more) are generated.

Subsequently, the left-eye processed image P ₃ L _(2n−1) is stored in the imaging frame memory 311L, and the right-eye processed image P ₃ R _2n is stored in the imaging frame memory 311R. Since the processing image is output from the imaging frame memory 311 at a cycle of 60 fps, the same processing image is output twice consecutively to the processing block in the subsequent stage.

In the apparatus-side captured image processing unit 312, similarly to the HMD-side captured image processing unit 303, the captured image processing for the left eye and the right eye is performed in parallel, and the processed images P ₄ L _{(2 n -1)} and P ₄ R _{2 n} (N is an integer of 1 or more) is generated.

The position and orientation information generation unit 313 calculates three-dimensional position and orientation information of the HMD 101 using the processed images P ₄ L _(2n−1) and P ₄ R _2n and the sensor information acquired by the position and orientation sensor 302. Then, the processed images P ₅ L _{(2 n -1)} and P ₅ R _{2 n} (n is an integer of 1 or more) are output to the subsequent stage.

After acquisition of the left-eye captured image CL ₁ is started by the imaging unit according to the flow of the above-described captured image processing, the output of the processed image P ₅ L ₁ correlated with the position and orientation information by the position and orientation information generation unit The treatment delay before starting is T _c .

  FIG. 6 is a diagram for explaining frame thinning processing and accumulation of delay in processing from the CG rendering / synthesizing unit to the display unit of the image processing system according to the first embodiment of the present invention.

The CG rendering composition unit 315 performs rendering processing of rendering of a CG image based on three-dimensional position and orientation information, and processing images P ₅ L _{(2 n -1)} and P ₅ R _{2 n} for the left eye and the right eye. An MR image MRL _(2n-1) and a right-eye MR image MRR _2n (n is an integer of 1 or more) are generated.

In the apparatus-side display image processing unit 316, display image processing for the left eye and right eye is performed in parallel to generate processed images P ₆ L _{(2 n -1)} and P ₆ R _{2 n} (n is an integer of 1 or more). Be done.

In the device-side thinning process section 317, for the left eye in order to reduce the transmission data amount of the display image from the image processing apparatus 104 to HMD101 processing image _{P 6 L (2n-1)} , from the right eye processing image _P 6 _{R 2n} Perform processing to thin out each even frame. The processed images P ₆ L _{(2 n -1)} and P ₆ R _{2 n} thinned out by this process are output to the processing blocks of the subsequent stage in time division.

In the device-side common display image processing unit 319, common image processing is performed on the display image for the left eye and the display image for the right eye output in time division, and the processed images P ₇ L _{(2 n -1)} and P ₇ R _2n (n is an integer of 1 or more) is generated.

Subsequently, transmission of transmission images T ₂ L _{(2 n -1)} and T ₂ R _{2 n} (n is an integer of 1 or more) is performed via the device side interface 308 and the HMD side interface 307.

The HMD-side common display image processing unit 320 executes common image processing on T ₂ L _{(2 n -1)} and T ₂ R _{2 n} transmitted by time division as in the device-side common display image processing unit 319. As a result, processed images P ₈ L _{(2 n -1)} and P ₈ R _{2 n} (n is an integer of 1 or more) are generated.

Subsequently, the left-eye processed image P ₈ L _(2n−1) is stored in the display frame memory 322L, and the right-eye processed image P ₈ R _2n is stored in the display frame memory 322R. Since the processing image is output from the display frame memory 322 at a cycle of 60 fps, the same processing image is output twice in succession to the processing block in the subsequent stage.

In the HMD-side display image processing unit 323, as in the device-side display image processing unit 316, display image processing for the left eye and the right eye is performed in parallel, and the processed images P ₉ L _{(2 n-1)} and P ₉ R _{2 n} (N is an integer of 1 or more) is generated.

In the display unit 324, the processed images P ₉ L _{(2 n -1)} and P ₉ R _{2 n} are displayed as the left-eye display image DL _{(2 n-1)} and the right-eye display image DR _{2 n} (n is an integer of 1 or more). Be done.

By the flow of the above display image processing, from the acquired captured image CL ₁ for the left eye is started by the imaging unit, treatment delay to display as a display image DL ₁ in the final display unit is started T _D It becomes.

  It should be noted here that the amount of image data transmitted between the HMD 101 and the image processing apparatus 104 is dropped equal to 30 fps in both the captured image and the display image. However, since all processing from imaging to display is performed at 60 fps, accumulation of delay time by frame memory and line memory can be suppressed as compared to the case where image processing is performed at 30 fps after frame rate conversion. It is possible.

  By having the above configuration, the image processing system according to the present embodiment performs frame thinning processing and time division output between left-eye and right-eye images, while suppressing accumulation of delay time in each processing. It is possible to reduce the amount of transmission data of moving image data. In addition, since moving image data after thinning processing is output in time division, it is possible to share the image processing of the latter stage with the image for the left eye and the image for the right eye, thereby realizing reduction in circuit size and power saving. be able to.

Second Embodiment
A second embodiment suitable for realizing the present invention will be described with reference to the drawings. In the first embodiment, the even frames of the left-eye image data and the odd frames of the right-eye image data are respectively thinned out. However, in the first embodiment, the left-eye captured image and the right-eye captured image used for calculating the three-dimensional position and orientation information in the position and orientation information generation unit, the left-eye display image displayed on the display unit, and the right-eye display There is a problem that a shift occurs in the generation time of the image. As a means for solving this, in the present embodiment, an example will be described in which the timing of the thinning process is synchronized between the image for the left eye and the image for the right eye. The description of the parts overlapping with the first embodiment will be omitted.

  FIG. 7 is a diagram showing an example of the configuration of an image processing system according to the second embodiment of the present invention. The image processing system according to the second embodiment of the present invention includes an HMD 101 and an image processing apparatus 104 having the following configuration. The HMD 101 includes an imaging unit 301, a position and orientation sensor 302, an HMD side captured image processing unit 303, an HMD side decimation processing unit 304, an HMD side captured image control unit 305, and an HMD side common captured image processing unit 306. Further, it has an HMD side I / F (interface) 307, an HMD side common display image processing unit 320, an HMD side display image control unit 321, a display frame memory 322, an HMD side display image processing unit 323, and a display unit 324.

  The image processing apparatus 104 includes an apparatus-side I / F 308, an apparatus-side shared captured image processing unit 309, an apparatus-side captured image control unit 310, an imaging frame memory 311, an apparatus-side captured image processing unit 312, and a position and orientation information generation unit 313. . In addition, a depth information generation unit 701, a content DB (database) 314, a CG rendering synthesis unit 315, an apparatus side display image processing unit 316, an apparatus side reduction processing unit 317, an apparatus side display image control unit 318, and an apparatus side common display image A processing unit 319 is provided.

  The HMD-side I / F 307 and the device-side I / F 308 each have a transmitter and a receiver. Further, the notation L or R at the end of each block name means that processing of left-eye moving image data and processing of right-eye moving image data are performed, respectively.

  The image processing system according to the present embodiment has a depth information generation unit 701 in the image processing apparatus 104, and is characterized in that the captured image and the CG image are combined in consideration of the depth information. Hereinafter, this point will be described.

  The position and orientation information generation unit 313 of the image processing apparatus 104 calculates three-dimensional position and orientation information of the HMD 101 from the sensor information acquired by the position and orientation sensor 302 of the HMD 101, the captured image acquired by the imaging unit 301, or both information. .

  The depth information generation unit 701 extracts an arbitrary subject to be subjected to depth detection from the captured images for the left eye and the right eye, and calculates depth information of the subject by a stereo method using the principle of triangulation. Methods of detecting an object from a captured image include a method of detecting and using marker shapes and characteristic points, and a method of detecting and using color information such as color markers, clothes, skin color, etc. It is also possible to further improve the detection accuracy.

  The CG rendering / synthesizing unit 315 performs rendering of a CG image and synthesis with a captured image based on three-dimensional position and orientation information and depth information to generate a display image for MR experience.

  With the above configuration, the wearer of the HMD 101 can experience a highly realistic MR image synthesized in consideration of the depth relationship between an arbitrary subject in a captured image and a CG image.

  In addition, although the component of an image processing system exists also except the above, since it is not the main point of this invention, description is abbreviate | omitted.

  FIG. 8 is a diagram for explaining image composition in consideration of depth information. FIG. 8A is a view schematically showing a scene to be subjected to the MR experience. The image capturing unit of the HMD 101 captures an image of a portion where the experience person interacts with the CG object 802 by moving the hand 801. In the following description, it is described that the subject that interacts with the CG object 802 is the hand 801, but the present invention is not limited to this, and can be applied to any subject.

  FIGS. 8B and 8C are diagrams schematically showing an image displayed on the display unit of the HMD 101 in the MR experience. In the MR experience, in order for the user to feel that the CG object 802 actually exists, the CG image of the CG object 802 is drawn and synthesized in consideration of the context of the hand 801 and the CG object 802. There is a need. That is, when viewed from the HMD 101, when the hand 801 is in front of the CG object 802, it is necessary to hide and display the CG object 802 with the hand 801 as shown in FIG. 2B. On the other hand, when the hand 801 is behind the CG object 802, the hand 801 needs to be hidden and displayed by the CG object 802 as shown in FIG. 2C. In the present invention, the area of the hand 801 is extracted from the captured image acquired by the imaging unit of the HMD 101 in order to perform such display, and CG of the CG object 802 is extracted in the area according to the depth information of the area. It is determined whether the image is drawn by overwriting. In the extraction of the area of the hand 801, the color area of the hand 801 is registered in advance, and when a pixel of the registered color appears in the image, the pixel is extracted as the hand 801.

  FIG. 9 is a diagram for explaining depth information generation of an arbitrary subject included in captured images for the left eye and the right eye. FIG. 9 shows a left-eye captured image 901L acquired by the left-eye imaging unit 301L of the HMD 101 and a right-eye captured image 901R acquired by the right-eye imaging unit 301R of the HMD 101, respectively.

  Here, it is assumed that coordinates of a point Pl1 of an arbitrary subject 902 in the captured image for the left eye 901L in the captured image plane are (X11, Y11). Similarly, a corresponding point corresponding to the point P11 in the captured image for the left eye in the captured image for right eye 901R is a point Pr1, and coordinates in the captured image plane of the point Pr1 are (Xr1, Yr1). The corresponding points can be searched by using various stereo matching methods by image processing. Here, the base length between the left-eye imaging unit 301L and the right-eye imaging unit 301R is B, and the focal distance of the imaging optical system forming an optical image on each imaging device is f. At this time, the depth information Z1 of the points P11 and Pr1 can be obtained by the relational expression of Z1 = Bf / | X11-Xr1 | by the stereo method using the principle of triangulation. In order to perform the above-described depth information generation, it is required that acquisition timings of the left-eye captured image 901L and the right-eye captured image 901R be simultaneous.

  FIG. 10 is a diagram for explaining frame skipping processing and accumulation of delay in processing from the imaging unit to the depth information generation unit of the image processing system according to the second embodiment of the present invention.

  As in the first embodiment, the image processing system according to the present embodiment will be described assuming that the frame rate of the imaging unit 301 of the HMD 101 and the frame rate of the display unit 324 are 60 fps.

  The processes in the imaging unit 301 and the HMD side captured image processing unit 303 are the same as in the first embodiment, and thus the description thereof is omitted.

The HMD-side thinning processing unit 304 respectively processes even-numbered frames from the left-eye processed image P ₁ L _n and the right-eye processed image P ₁ R _n in order to reduce the transmission data amount of the captured image from the HMD 101 to the image processing apparatus 104. Perform thinning processing. Processed images P ₁ L _{(2 n -1)} and P ₁ R ₍₂ n _-1) (n is an integer of 1 or more ₎ thinned out by this process are output to the subsequent processing blocks in time division. Here, the left-eye processed image P ₁ L _(2n−1) and the right-eye processed image P ₁ R _(2n−1) are processed at the same timing. Therefore, the right-eye processed image P ₁ R _{(2 n -1)} is stored in a frame memory ₍ not shown ₎ during output of the left-eye processed image P ₁ L _{(2 n -1)} for time division output to the subsequent processing block. Do the process. Then, after outputting the left-eye processed image P ₁ L _(2n−1) , the right-eye processed image P ₁ R _(2n−1) is read from the frame memory and output. Implement time-division output. In addition, even frames on both the left and right sides are thinned out and output is alternately performed in order to reduce the data amount to 1⁄2, but the thinning processing in the present embodiment is not limited to this, and time-division output is Any frame skipping process may be performed to the extent possible.

The HMD-side common captured image processing unit 306 executes common image processing on the left-eye processed image P ₁ L _{(2 n -1)} and the right-eye processed image P ₁ R _{(2 n-1)} output in time division. As a result, processed images P ₂ L _{(2 n -1)} and P ₂ R ₍₂ n _-1) (n is an integer of 1 or more) are generated. In this way, by performing image processing commonly executed on the left-eye captured image and the right-eye captured image in the latter stage of thinning processing, it is possible to realize commonality of circuits by time division processing and power saving. Is possible.

Subsequently, transmission of transmission images T ₁ L _{(2 n -1)} and T ₁ R ₍₂ n _-1) (n is an integer of 1 or more) is performed via the HMD side interface 307 and the device side interface 308.

In the device-side common captured image processing unit 309, T ₁ L _{(2 n-1)} and T ₁ R _{(2 n-1)} (n is 1 or more ₎ transmitted in time division as in the HMD-side common captured image processing unit 306 Common image processing is performed on integers. As a result, processed images P ₃ L _{(2 n -1)} and P ₃ R ₍₂ n _-1) (n is an integer of 1 or more) are generated.

Subsequently, the left-eye processed image P ₃ L _(2n−1) is stored in the imaging frame memory 311L, and the right-eye processed image P ₃ R _(2n−1) is stored in the imaging frame memory 311R. Since the processing image is output from the imaging frame memory 311 at a cycle of 60 fps, the same processing image is output twice consecutively to the processing block in the subsequent stage.

In the apparatus-side captured image processing unit 312, similarly to the HMD-side captured image processing unit 303, the captured image processing for the left eye and the right eye is performed in parallel, and the processed images P ₄ L _{(2 n -1)} and P ₄ R _{( 2n-1)} (n is an integer of 1 or more) is generated.

The position and orientation information generation unit 313 uses the processed images P ₄ L _{(2 n −1)} and P ₄ R _{(2 n −1)} and the sensor information acquired by the position and orientation sensor 302 to generate three-dimensional position and orientation information of the HMD 101. calculate. Then, the processed images P ₅ L _{(2 n -1)} and P ₅ R ₍₂ n _-1) (n is an integer of 1 or more) are output to the subsequent stage.

The depth information generation unit 701 uses the processing images P ₅ L _{(2 n -1)} and P ₅ R _{(2 n -1)} for the left eye and the right eye to calculate depth information of an arbitrary subject in the captured image, Processed images P ₆ L _{(2 n -1)} and P ₆ R ₍₂ n _-1) (n is an integer of 1 or more) are output to the subsequent stage.

After the acquisition of the left-eye captured image CL ₁ is started in the imaging unit by the flow of the above-described captured image processing, the output of the processed image P ₆ L _{1 associated} with the depth information is started in the depth information generation unit The treatment delay until t is T _c .

  FIG. 11 is a diagram for explaining frame skipping processing and delay accumulation in processing from the CG rendering / synthesis unit to the display unit of the image processing system according to the second embodiment of the present invention.

The CG rendering / combining unit 315 composes a rendering of a CG image based on three-dimensional position and orientation information, and processing of the left-eye and right-eye processed images P ₆ L _{(2 n -1)} and P ₆ R _{(2 n -1)} Do. As a result, the left-eye MR image MRL _(2n-1) and the right-eye MR image MRR _(2n-1) (where n is an integer of 1 or more) are generated.

In the apparatus-side display image processing unit 316, display image processing for the left eye and right eye is performed in parallel, and the processed images P ₇ L _{(2 n -1)} and P ₇ R ₍₂ n _-1) (n is 1 or more ₎ Integer) is generated.

The apparatus-side decimation unit 317 processes the left-eye processed image P ₇ L _(2n−1) and the right-eye processed image P ₇ R _{(2n )} to reduce the amount of transmission data of the display image from the image processing apparatus 104 to the HMD 101. _A process of thinning out even frames from _-1) is performed. The processed images P ₇ L _{(2 n -1)} and P ₇ R _{(2 n -1)} thinned out by this process are output to the processing blocks of the subsequent stage in a time division manner.

In the device-side common display image processing unit 319, common image processing is performed on the display image for the left eye and the display image for the right eye output in time division, and the processed images P ₈ L _{(2 n -1)} and P ₈ R _(2n-1) (n is an integer of 1 or more) is generated.

Subsequently, transmission of transmission images T ₂ L _{(2 n -1)} and T ₂ R ₍₂ n _-1) (n is an integer of 1 or more) is performed via the apparatus side interface 308 and the HMD side interface 307.

In the HMD side common display image processing unit 320, as in the device side common display image processing unit 319, T ₂ L _{(2 n -1)} and T ₂ R _{(2 n -1)} transmitted in time division are common. Image processing is performed. As a result, processed images P ₉ L _{(2 n -1)} and P ₉ R ₍₂ n _-1) (n is an integer of 1 or more) are generated.

Subsequently, the left-eye processed image P ₉ L _(2n−1) is stored in the display frame memory 322L, and the right-eye processed image P ₉ R _(2n−1) is stored in the display frame memory 322R. Since the processing image is output from the display frame memory 322 at a cycle of 60 fps, the same processing image is output twice in succession to the processing block in the subsequent stage.

In the HMD-side display image processing unit 323, as in the device-side display image processing unit 316, the display image processing for the left eye and the right eye is performed in parallel, and the processed images P ₁₀ L _{(2 n -1)} , P ₁₀ R _{( 2n-1)} (n is an integer of 1 or more) is generated.

In the display unit 324, the processing image _{P 10 L (2n-1)} , P 10 R (2n-1) , respectively displaying the left-eye image _{DL (2n-1),} the right-eye display image _{DR (2n-1) (n} Is displayed as an integer of 1 or more).

By the flow of the above display image processing, from the acquired captured image CL ₁ for the left eye is started by the imaging unit, treatment delay to display as a display image DL ₁ in the final display unit is started T _D It becomes.

  Here, as in the image processing system in the first embodiment, the amount of image data transmitted between the HMD 101 and the image processing apparatus 104 is dropped to be equal to 30 fps in both the captured image and the display image. However, since all processing from imaging to display is performed at 60 fps, accumulation of delay time by the frame memory and the line memory can be suppressed as compared with the case where image processing is performed at 30 fps after frame rate conversion.

Furthermore, the image processing system according to the present embodiment is characterized in that the timing of frame thinning processing is synchronized between the image for the left eye and the image for the right eye. Thereby, the generation times of the left-eye and right-eye captured images used for information calculation in the position and orientation information generation unit and the depth information generation unit, and the left-eye and right-eye display images displayed on the display unit It can be made to coincide in a cycle of 30 seconds. When the generation times of the captured image for the left eye and for the right eye coincide with each other, depth information of any subject in the captured image can be calculated, and the wearer of the HMD 101 can
By having the above configuration, the image processing system according to the present embodiment, like the image processing system according to the first embodiment, reduces the amount of transmission data of moving image data while suppressing the accumulation of delay time in each process. It is possible to reduce. In addition, since moving image data after thinning processing is output in time division, it is possible to share the image processing of the latter stage with the image for the left eye and the image for the right eye, thereby realizing reduction in circuit size and power saving. be able to. Furthermore, by performing timing control of frame thinning processing so that the generation timings of left-eye and right-eye image data coincide, a highly realistic MR image in which the depth relationship between the subject and the CG object is considered. It becomes possible to experience.

Third Embodiment
A third embodiment suitable for realizing the present invention will be described with reference to the drawings. The description of the parts overlapping with the first and second embodiments will be omitted.

  In the first and second embodiments, regarding data transmission between the HMD 101 and the image processing apparatus 104, regardless of the wired or wireless connection method, in order to reduce the amount of transmission data, the HMD 101 and the image processing apparatus 104 interframe, respectively. The configuration for performing the pulling process has been described. In the present embodiment, a configuration in which connection between the HMD 101 and the image processing apparatus 104 can be both wired connection using a cable and wireless connection via a wireless box will be described. In the wireless connection, a cable for directly connecting the HMD 101 and the image processing apparatus 104 is unnecessary, and there is an advantage that the freedom of movement of the HMD user can be improved. On the other hand, in wired connection, broadband transmission is generally possible compared to wireless connection, so providing a higher quality MR image without using a data amount reduction method such as image coding or frame rate conversion. It has the merit of being able to

  12 and 13 are diagrams showing an example of the configuration of an image processing system according to the third embodiment of the present invention.

  The image processing system according to the third embodiment of the present invention includes an HMD 101, an HMD wireless box 1301, an apparatus wireless box 1302, and an image processing apparatus 104. Each of the HMD 101, the HMD wireless box 1301, the apparatus wireless box 1302, and the image processing apparatus 104 has the following configuration.

  The HMD 101 includes an imaging unit 301, a position and orientation sensor 302, an intra-HMD captured image processing unit 1201, an intra-HMD wired I / F 1202, an intra-HMD display image processing unit 1203, and a display unit 324.

  The HMD-side wireless box 1301 includes an HMD-side wired I / F 1303, an HMD-side captured image processing unit 303, an HMD-side decimation processing unit 304, an HMD-side captured image control unit 305, and an HMD-side common captured image processing unit 306. Further, it has an HMD side wireless I / F 1304, an HMD side common display image processing unit 320, an HMD side display image control unit 321, a display frame memory 322, an HMD side display image processing unit 323, and a display unit 324.

  The device-side wireless box 1302 includes a device-side wireless I / F 1305, a device-side shared-captured image processing unit 309, a device-side captured-image control unit 310, an imaging frame memory 311, a device-side captured image processing unit 312, and a device-side wired I / F 1306 Have. Further, it has an apparatus side display image processing unit 316, an apparatus side thinning processing unit 317, an apparatus side display image control unit 318, and an apparatus side common display image processing unit 319.

  The image processing apparatus 104 includes an in-apparatus wired I / F 1204, an in-apparatus captured image processing unit 1205, a position and orientation information generating unit 313, a content DB 314, a CG rendering and combining unit 315, and an in-apparatus display image processing unit 1206.

  The HMD internal I / F 1202, the internal I / F 1204, the HMD side I / F 1303, the HMD side wireless I / F 1304, the device side wireless I / F 1305, and the device side I / F 1306 respectively have a transmitter and a receiver. Have. Further, the notation L or R at the end of each block name means that processing of left-eye moving image data and processing of right-eye moving image data are performed, respectively.

  FIG. 12 is a diagram for explaining a configuration in which data transmission between the HMD and the image processing apparatus of the image processing system according to the third embodiment of the present invention is performed by wired transmission using a cable.

  The imaging unit 301 of the HMD 101 acquires a captured image of the physical space as a stereo image having a parallax that substantially matches the viewing positions of the left eye and the right eye of the HMD user.

  The position and orientation sensor 302 acquires information for calculating three-dimensional position and orientation information regarding the position and direction of the viewpoint of the HMD user.

  The in-HMD captured image processing unit 1201 performs image processing on the left-eye captured image and the right-eye captured image acquired by the imaging unit using a plurality of image processing circuits.

  The in-HMD wired I / F 1202 transmits the captured image for the left eye, the captured image for the right eye, and the sensor information acquired by the position and orientation sensor 302 by communication with the in-device wired I / F 1204 of the image processing device 104. In addition, reception of a display image to be displayed on the display unit 324 and transmission and reception of other control signals are performed.

  The in-HMD display image processing unit 1203 performs image processing on the display image for the left eye and the display image for the right eye by a plurality of image processing circuits.

  The display unit 324 displays an MR image on which CG is superimposed on the left eye and the right eye of the HMD user.

  The intra-apparatus wired I / F 1204 of the image processing apparatus 104 receives the captured image for the left eye and the right eye and the sensor information and communicates with the display unit 324 by communication with the intra-HMD wired I / F 1202 of the HMD 101. Sends display images and sends and receives other control signals.

  The in-device captured image processing unit 1205 performs image processing on the captured image for the left eye and the captured image for the right eye by a plurality of image processing circuits. It is desirable that the image processing performed by the in-device captured image processing unit 1205 and the in-HMD captured image processing unit 1201 be distributed to each in consideration of the processing load and the processing order.

  The position and orientation information generation unit 313 calculates three-dimensional position and orientation information of the HMD 101 from the sensor information acquired by the position and orientation sensor 302, the captured image information acquired by the imaging unit 301, or both information.

  The content DB 314 is a database that stores content for CG drawing. The CG rendering / synthesizing unit 315 renders a CG image and synthesizes a captured image and a CG image based on the three-dimensional position and orientation information, and generates a display image for MR experience.

  The in-apparatus display image processing unit 1206 performs image processing on the display image for the left eye and the display image for the right eye generated by the CG rendering synthesis unit by a plurality of image processing circuits. It is desirable that image processing performed between the in-device display image processing unit 1206 and the in-HMD display image processing unit 1203 be dispersed in consideration of processing load and processing order.

  With the above configuration, data transmission between the HMD 101 and the image processing apparatus 104 can be realized by broadband wired transmission using a cable. This enables transmission, for example, of the captured image having a frame rate of 60 fps and the frame rate of the display image without lowering the frame rate, and the wearer of the HMD 101 can experience a more realistic MR image It becomes possible.

  The image processing system in the present embodiment may be configured to have a depth information generation unit in the image processing apparatus 104 as in the image processing system in the second embodiment. Further, although the components of the image processing system exist other than the above, since they are not the gist of the present invention, the description will be omitted.

  FIG. 13 is a diagram for explaining a configuration in which data transmission between the HMD and the image processing apparatus of the image processing system in the third embodiment of the present invention is performed by wireless transmission via the HMD side radio box and the apparatus side radio box. . With regard to the HMD 101 and the image processing apparatus 104, the description similar to that of the configuration in FIG. 12 is omitted.

  The HMD internal wired I / F 1202 of the HMD 101 transmits the captured image for the left eye and the right eye, and the sensor information by communication with the HMD side wired I / F 1303 of the HMD side wireless box 1301. In addition, reception of a display image to be displayed on the display unit 324 and transmission and reception of other control signals are performed.

  The HMD side wired I / F 1303 of the HMD side wireless box 1301 receives the captured image for the left eye and the right eye, and the sensor information by communication with the HMD wired I / F 1202 of the HMD 101. Further, transmission of a display image to be displayed on the display unit 324 and transmission and reception of other control signals are performed.

  The HMD-side captured image processing unit 303 performs image processing on the left-eye captured image and the right-eye captured image acquired by the imaging unit using a plurality of image processing circuits. It is desirable that the image processing performed here is processing unique to the HMD side wireless box 1301 and the device side wireless box 1302 and using different parameters for the left-eye captured image and the right-eye captured image.

  The HMD-side decimation processing unit 304 performs frame decimation processing for reducing the amount of data on the left-eye captured image and the right-eye captured image.

  The HMD-side captured image control unit 305 controls the intervals and timings of frame thinning processing by the HMD-side thinning processing unit, and performs time-division of the captured images for the left eye and the right eye after frame thinning processing in frame units. Control to send to the subsequent HMD side common captured image processing unit 306 is performed.

  The HMD-side common captured image processing unit 306 performs image processing in time division on the captured images for the left eye and the right eye after the frame thinning processing. As the image processing performed here, processing that is unique to the HMD-side wireless box 1301 and the device-side wireless box 1302 and that can use the same parameter for the left-eye captured image and the right-eye captured image is desirable. For example, processing such as resolution conversion, color space conversion, or image coding may be mentioned.

  The HMD-side wireless I / F 1304 transmits the captured image for the left eye, the captured image for the right eye, and the sensor information by wireless communication with the device-side wireless I / F 1305 of the device-side wireless box 1302 and displays on the display unit 324 Reception of other display images, and transmission and reception of other control signals.

  The HMD-side common display image processing unit 320 performs image processing in time division on the display images for the left eye and the right eye after the frame thinning processing. As the image processing performed here, processing that is unique to the HMD-side wireless box 1301 and the device-side wireless box 1302 and that can use the same parameter for the display image for the left eye and the display image for the right eye is desirable. For example, processing such as resolution conversion, color space conversion, or image decoding may be mentioned.

  The HMD-side display image control unit 321 distributes the left-eye and right-eye display images after frame thinning processing to the left-eye and right-eye display frame memories 322, respectively. Then, control is performed to output a display image from the display frame memory in accordance with the displayable frame rate on the display unit 324.

  The display frame memory 322 stores the display image for the left eye and the display image for the right eye, and outputs the display image to the HMD display image processing unit 323 under the control of the HMD display image control unit 321.

  The HMD-side display image processing unit 323 performs image processing on the display image for the left eye and the display image for the right eye by a plurality of image processing circuits. As image processing performed here, processing specific to the HMD side wireless box 1301 and the device side wireless box 1302 and using different parameters for the display image for the left eye and the display image for the right eye is desirable.

  The device-side wireless I / F 1305 of the device-side wireless box 1302 receives the captured image for the left eye and the right eye and the sensor information by wireless communication with the HMD-side wireless I / F 1304 of the HMD-side wireless box 1301. Further, transmission of a display image to be displayed on the display unit 324 and transmission and reception of other control signals are performed.

  The device-side common captured image processing unit 309 performs image processing on the captured images for the left eye and the right eye after frame thinning processing in a time division manner. As the image processing performed here, processing that is unique to the HMD-side wireless box 1301 and the device-side wireless box 1302 and that can use the same parameter for the left-eye captured image and the right-eye captured image is desirable.

  The apparatus-side captured image control unit 310 distributes the left-eye and right-eye captured images after the frame thinning processing to the left-eye and right-eye imaging frame memories 311, respectively. Then, control is performed to output a captured image from the imaging frame memory 311 in accordance with a frame rate that can be displayed on the display unit 324.

  The imaging frame memory 311 stores the left-eye imaging image and the right-eye imaging image, and outputs the imaging image to the apparatus-side imaging image processing unit 312 based on the control of the apparatus-side imaging image control unit 310.

  The device-side captured image processing unit 312 performs image processing on the left-eye captured image and the right-eye captured image output from the imaging frame memory 311 by a plurality of image processing circuits. The image processing performed here is processing unique to the HMD-side wireless box 1301 and the device-side wireless box 1302 as in the HMD-side captured image processing unit 303, and separately for the left-eye captured image and the right-eye captured image It is desirable that the process be performed using the parameters of Further, in consideration of the processing load and the processing order, the image processing to be executed between the apparatus-side captured image processing unit 312 and the HMD-side captured image processing unit 303 may be dispersed.

  The device-side wired I / F 1306 transmits the captured image for the left eye and the right eye and the sensor information by communication with the in-device wired I / F 1204 of the image processing device 104, and displays the left for displaying on the display unit 324. It receives images and sends and receives other control signals.

  The device-side display image processing unit 316 performs image processing on the display image for the left eye and the display image for the right eye generated by the CG rendering and combining unit by a plurality of image processing circuits. The image processing performed here is processing unique to the HMD-side wireless box 1301 and the device-side wireless box 1302 as in the HMD-side display image processing unit 323, and separately for the display image for the left eye and the display image for the right eye It is desirable to process using the parameters of Further, in consideration of the processing load and the processing order, the image processing to be executed between the apparatus-side display image processing unit 316 and the HMD-side display image processing unit 323 may be dispersed.

  The device-side decimation processing unit 317 performs frame decimation processing for reducing the data amount on the display image for the left eye and the display image for the right eye.

  The apparatus-side display image control unit 318 controls the intervals and timings of frame thinning processing by the device-side thinning processing unit, and displays the display images for the left eye and the right eye after frame thinning processing in time division in frame units. Control is performed to send to the device-side common display image processing unit 319 in the latter stage.

  The device-side common display image processing unit 319 performs time-division image processing on the display images for the left eye and the right eye after the frame thinning processing. The image processing performed here is processing unique to the HMD-side wireless box 1301 and the device-side wireless box 1302 as in the HMD-side common display image processing unit 320, and is a display image for the left eye and a display image for the right eye. A process that can use the same parameters is desirable. Further, in consideration of the processing load and the processing order, the image processing to be executed between the device-side common display image processing unit 319 and the HMD-side common display image processing unit 320 may be dispersed.

  The in-device wired I / F 1204 of the image processing device 104 communicates with the device-side wired I / F 1306 of the device wireless box 1302 to receive the captured image for the left eye and the right eye and the sensor information. Further, transmission of a display image to be displayed on the display unit 324 and transmission and reception of other control signals are performed.

  By having the above configuration, the image processing system according to the present embodiment can perform both wired connection using a cable and wireless connection via a wireless box between the HMD 101 and the image processing apparatus 104. It becomes possible. In the wireless connection, a cable for directly connecting the HMD 101 and the image processing apparatus 104 is unnecessary, and there is an advantage that the freedom of movement of the HMD user can be improved. On the other hand, in wired connection, broadband transmission is generally possible compared to wireless connection, so providing a higher quality MR image without using a data amount reduction method such as image coding or frame rate conversion. It has the merit of being able to

Fourth Embodiment
A fourth embodiment suitable for realizing the present invention will be described with reference to the drawings. In addition, description is abbreviate | omitted about the part which overlaps with the 1st-3rd embodiment.

  In the first to third embodiments, the configuration for performing thinning processing of captured images acquired by the left-eye imaging unit 301L and the right-eye imaging unit 301R of the HMD 101 has been described. In the present embodiment, an optional imaging unit is additionally provided to the HMD 101, and a configuration for performing thinning processing of three captured images acquired by the left-eye imaging unit, the right-eye imaging unit, and the optional imaging unit will be described.

  FIG. 14 is a diagram showing an example of the configuration of an image processing system according to the fourth embodiment of the present invention.

  The image processing system according to the fourth embodiment of the present invention includes an HMD 101 and an image processing apparatus 104 having the following configuration. The HMD 101 includes an imaging unit 301, a position and orientation sensor 302, an HMD side captured image processing unit 303, an HMD side decimation processing unit 304, an HMD side captured image control unit 305, and an HMD side common captured image processing unit 306. Further, it has an HMD side I / F (interface) 307, an HMD side common display image processing unit 320, an HMD side display image control unit 321, a display frame memory 322, an HMD side display image processing unit 323, and a display unit 324. Further, it has an option imaging unit 1401, an option HMD side captured image processing unit 1402, and an option HMD side thinning processing unit 1403.

  The image processing apparatus 104 includes an apparatus-side I / F 308, an apparatus-side shared-captured image processing unit 309, an apparatus-side captured image control unit 310, an imaging frame memory 311, an apparatus-side captured image processing unit 312, a position and orientation information generation unit 313, content It has a DB (database) 314. The CG rendering / combining unit 315, device-side display image processing unit 316, device-side thinning-out processing unit 317, device-side display image control unit 318, device-side common display image processing unit 319, option imaging frame memory 1404, for option The apparatus-side captured image processing unit 1405 is provided.

  The HMD-side I / F 307 and the device-side I / F 308 each have a transmitter and a receiver. In addition, the notation L, R or OP at the end of each block name means performing processing of moving image data for the left eye, processing of moving image data for the right eye, and processing of moving image data for an optional imaging unit.

  The option imaging unit 1401 of the HMD 101 acquires an option captured image in a range different from the range in real space captured by the left-eye imaging unit 301L and the right-eye imaging unit 301R. The option imaging unit 1401 may use the same imaging device and imaging optical system as the left-eye and right-eye imaging units 301, and differs from the left-eye and right-eye imaging units 301 in order to make the resolution and imaging range different. An imaging device or an imaging optical system may be used. Further, the range in the physical space imaged by the option imaging unit 1401 may include part or all of the areas imaged by the left-eye imaging unit 301L and the right-eye imaging unit 301R. Furthermore, the optional imaging unit 1401 may be built in the HMD 101, or may be externally attached to the HMD 101.

  The option HMD side captured image processing unit 1402 performs image processing on the option captured image acquired by the option imaging unit 1401. As the image processing performed here, processing using different parameters in the captured image for the left eye and the captured image for the right eye and the option captured image is desirable. For example, processing for correcting individual variation of an imaging device or an imaging optical system such as pixel defect correction, gamma correction, or distortion correction of an imaging optical system may be mentioned.

  The option HMD side decimation processing unit 1403 performs a frame decimation process on the option captured image to reduce the data amount.

  The HMD side captured image control unit 305 controls an interval and timing of frame thinning processing by the HMD side thinning processing unit 304 and the option HMD side thinning processing unit 1403. Also, control is performed to send each captured image after the frame thinning process to the HMD side common captured image processing unit on the subsequent stage by time division. As an example of frame thinning process control of a captured image, a method of sequentially thinning out and outputting frames at different times as in the first embodiment, and thinning of frames of the same time as in the second embodiment And a method of outputting in time division using a frame memory. Further, the frame thinning process control is not limited to the above, and the thinning is performed according to the priority in consideration of the left eye imaging image, the right eye imaging image, the application of the option imaging image, the required number of frames, etc. The processing intervals may be different.

  The HMD side common captured image processing unit 306 performs image processing on the captured image for the left eye and the right eye after the frame thinning process and the option captured image in a time division manner. As the image processing performed here, for example, processing that can use the same parameter for the left-eye captured image and the right-eye captured image, such as resolution conversion, color space conversion, or image coding, is desirable.

  The HMD I / F 307 transmits the captured image for the left eye and the right eye, the optional captured image, and the sensor information by communication with the device I / F 308 of the image processing device 104 and displays the image on the display unit 324. It receives display images and sends and receives other control signals.

  The device-side I / F 308 of the image processing device 104 receives the left-eye and right-eye captured images, option captured images, and sensor information by communication with the HMD-side I / F 307 of the HMD 101 and displays the information on the display unit 324 Transmission of display images, and transmission and reception of other control signals.

  The device-side common captured image processing unit 309 performs image processing on the captured image for the left eye and the right eye after the frame thinning processing and the option captured image in a time division manner. As the image processing performed here, for example, processing capable of using the same parameter in the captured image for the left eye and the captured image for the right eye, such as resolution conversion, color space conversion, or image decoding, is desirable.

  The apparatus-side captured image control unit 310 distributes the left-eye and right-eye captured images and option captured images after frame thinning processing to the left-eye and right-eye imaging frame memory 311 and option imaging frame memory 1404, respectively. Store. Further, control is performed to output a captured image from the imaging frame memory 311 and the option frame memory 1404 in accordance with a frame rate that can be displayed on the display unit 324.

  The option imaging frame memory 1404 stores the option imaging image, and outputs the imaging image to the option apparatus imaging image processor 1405 based on the control of the apparatus imaging image control unit 310.

  The option-apparatus side captured image processing unit 1405 performs image processing on the option captured image output from the option imaging frame memory 1404. As the image processing performed here, similarly to the option HMD-side captured image processing unit 1402, processing using different parameters for the left-eye and right-eye captured images and the option captured image is desirable. Further, in consideration of the processing load and the processing order, the image processing to be executed between the option device side captured image processing unit 1405 and the option HMD side captured image processing unit 1402 may be dispersed.

  The position and orientation information generation unit 313 calculates the three-dimensional position and orientation information of the HMD 101 from the sensor information acquired by the position and orientation sensor 302, the captured image information acquired by the imaging unit 301 and the option imaging unit 1401, or both information. Do.

  With the above configuration, in the image processing system according to the present embodiment, the position and orientation information generation unit 313 of the image processing apparatus 104 can use the option captured image acquired by the option imaging unit 1401 of the HMD 101. Become. Thus, information such as markers existing outside the range of vision of the HMD wearer, that is, outside the imaging range of the left-eye imaging unit 301L and the right-eye imaging unit 302R can also be used to generate position and orientation information. The accuracy of information and the freedom of setting markers etc. are improved.

Other Embodiments
Further, the present invention supplies software (program) for realizing the functions of the above embodiments to a system or apparatus via a network or various storage media, and a computer (CPU, MPU or the like) of the system or apparatus It is processing to read out and execute. Further, the present invention may be applied to a system constituted by a plurality of devices or to an apparatus comprising a single device. The present invention is not limited to the above embodiments, and various modifications (including organic combinations of the respective embodiments) are possible based on the spirit of the present invention, which are excluded from the scope of the present invention is not. That is, the configuration in which each of the above-described embodiments and their modifications are combined is also included in the present invention.

101 HMD
301 imaging unit 302 position and orientation sensor 303 HMD-side captured image processing unit 304 HMD-side decimation processing unit 305 HMD-side captured image control unit 306 HMD-side common captured image processing unit 307 HMD-side I / F
320 HMD side common display image processing unit 321 HMD side display image control unit 322 display frame memory 323 HMD side display image processing unit 324 display unit 104 image processing apparatus 308 apparatus side I / F
309 device-side shared captured image processing unit 310 device-side captured image control unit 311 imaging frame memory 312 device-side captured image processing unit 313 position and orientation information generation unit 314 content DB
315 CG rendering / combining unit 316 device-side display image processing unit 317 device-side thinning processing unit 318 device-side display image control unit 319 device-side common display image processing unit

Claims (13)

First input means for inputting at least two or more pieces of moving image data having an arbitrary frame rate;
Image processing means for performing image processing on the plurality of moving image data by a plurality of image processing circuits;
Frame thinning means for thinning the number of frames without changing the frame period for each of the plurality of moving image data;
The image processing means, and control means for controlling the frame thinning means;
And first output means for outputting thinned-out moving image data whose data amount has been reduced by the frame thinning-out means,
The control means is the image processing means and the frame thinning so that the sum of the data amount of the plurality of moving image data subjected to the thinning process falls within the range of the data amount which can be output by the first output means. Control means,
The image processing apparatus according to claim 1, wherein the first output unit outputs each frame of the plurality of moving image data subjected to thinning processing in a time division manner.   The control means performs parallel processing using different image processing circuits on moving image data before the frame thinning-out process is performed by the frame thinning-out means, and a moving image after frame thinning-out process is performed The image processing apparatus according to claim 1, wherein the image processing unit is controlled to perform time division processing by a common image processing circuit on data.   The image processing apparatus according to claim 1, wherein the plurality of moving image data are stereo images having parallaxes for the left eye and the right eye, respectively.   The thinning means performs a thinning process of thinning out either an odd frame or an even frame on moving image data for the left eye, and a frame thinned out with moving image data for the left eye on moving image data for the right eye The image processing apparatus according to any one of claims 1 to 3, wherein thinning out processing is performed to thin out frames different from H.   The thinning means performs a thinning process of thinning out either an odd frame or an even frame on moving image data for the left eye, and a frame thinned out with moving image data for the left eye on moving image data for the right eye The image processing apparatus according to any one of claims 1 to 4, wherein thinning processing is performed to thin the same frames as in (4). Second input means for inputting time-divided moving image data having an arbitrary frame rate, in which a plurality of moving image data are arranged in time division on a frame basis;
Image processing means for performing image processing on the plurality of moving image data and the time-divided moving image data by a plurality of image processing circuits;
Control means for performing control of the image processing means and control of distributing time-division frames of moving picture data as respective moving picture data;
An image processing apparatus comprising: second output means for outputting each frame distributed by the control means as moving image data;
When the control means distributes the frames of the time-divided moving image data as each moving image data, the moving image data is input when the frame rate of each moving image data is divided in time. An image processing apparatus which is controlled to be equal to a frame rate. An image processing system comprising a first image processing apparatus and a second image processing apparatus, comprising:
The first image processing apparatus
First input means for inputting at least two or more pieces of moving image data having an arbitrary frame rate;
First image processing means for performing image processing on the plurality of moving image data by a plurality of image processing circuits;
Frame thinning means for thinning the number of frames without changing the frame period for each of the plurality of moving image data;
The first image processing means, and a first control means for controlling the frame thinning means;
And first output means for outputting thinned-out moving image data whose data amount has been reduced by the frame thinning-out means,
The first control means performs the first image processing such that a total of data amounts of the plurality of moving image data subjected to thinning processing falls within a range of data amounts that can be output by the first output means. Control means and the frame thinning means,
The first output unit outputs decimated moving image data in which frames of the plurality of decimated moving image data are arranged in time division on a frame basis,
The second image processing apparatus
Second input means for inputting the thinned moving image data output from the first image processing apparatus;
Second image processing means for performing image processing on the plurality of moving image data and the thinned moving image data by a plurality of image processing circuits;
Second control means for performing control of the second image processing means and control of distributing frames of thinned moving image data as respective moving image data;
And second output means for outputting each frame distributed by the second control means as respective moving image data,
When the second control means distributes the frames of the thinned moving image data as respective moving image data, the frame rate of each moving image data is equal to the frame rate at the time of input of the thinned moving image data. An image processing system characterized by controlling to become. An image processing system for transmitting image data between a first image processing apparatus and a second image processing apparatus, comprising:
The first and second image processing devices respectively
First input means for inputting at least two or more pieces of moving image data having an arbitrary frame rate;
Frame thinning means for thinning the number of frames without changing the frame period for each of the plurality of moving image data;
First output means for outputting thinned moving image data whose data amount has been reduced by said frame thinning means to the other image processing apparatus;
Second input means for inputting the thinned moving image data from the other image processing apparatus;
Second output means for distributing the frames of the thinned moving image data as respective moving image data and outputting a plurality of reconstructed moving image data;
Image processing means for performing image processing on the plurality of moving image data and thinned moving image data by a plurality of image processing circuits;
Control means for controlling the image processing means, controlling the frame thinning means, and controlling to distribute frames of the thinning moving image data as respective moving image data,
The control means is the image processing means and the frame so that a total of data amounts of the plurality of thinned moving image data subjected to thinning processing falls within a range of data amounts that can be output by the first output means. Control the thinning means,
Further, the control means further distributes the frames of the thinned moving image data as respective moving image data, and when reconstructing, the frame rate of each moving image data corresponds to the input of the thinned moving image data. An image processing system that is controlled to be equal to a frame rate.   The control means distributes the moving image data before the frame thinning process is performed by the frame thinning means and the frames of the thinned moving image data as respective moving image data, and generates reconstructed moving image data. The image processing circuit performs parallel processing using different image processing circuits, and performs time division processing by a common image processing circuit on thinned moving image data after frame thinning processing is performed. The image processing system according to claim 7, which controls the processing means.   The image processing system according to any one of claims 7 to 9, wherein each of the plurality of moving image data is a stereo image having parallax for left eye and right eye.   The thinning means performs a thinning process of thinning out either an odd frame or an even frame on moving image data for the left eye, and a frame thinned out with moving image data for the left eye on moving image data for the right eye The image processing system according to any one of claims 7 to 10, wherein thinning processing is performed to thin a frame different from the image data.   The thinning means performs a thinning process of thinning out either an odd frame or an even frame on moving image data for the left eye, and a frame thinned out with moving image data for the left eye on moving image data for the right eye The image processing system according to any one of claims 7 to 11, wherein thinning processing is performed to thin the same frames as in (4). Inputting at least two or more pieces of moving image data having an arbitrary frame rate;
Performing image processing on the plurality of moving image data by a plurality of image processing circuits;
Performing decimation processing on the number of frames without changing the frame period for each of the plurality of moving image data;
Outputting the decimated moving image data of which the data amount has been reduced;
The sum of the data amount of the plurality of moving image data subjected to thinning processing is controlled so as to fall within the range of the data amount capable of the output;
An image processing method comprising time-divisionally outputting each frame of the plurality of moving image data subjected to the thinning process.

Images