JP2017010119A - Information processing apparatus, image processing apparatus, control method thereof, and program - Google Patents

Abstract

The amount of video data to be transmitted is reduced while suppressing a decrease in position and orientation detection accuracy.
An information processing apparatus capable of communicating with an image processing apparatus, an acquisition unit for acquiring a movement rotation amount including at least one of a movement amount and a rotation amount of the information processing device, and a movement acquired by the acquisition unit A control unit that reduces a data amount of a captured image transmitted to the image processing apparatus when the rotation amount is equal to or less than a predetermined value;
[Selection] Figure 2

Description

  The present invention relates to an information processing apparatus, an image processing apparatus, a control method thereof, and a program.

  Mixed reality (MR) technology is known as a technology that seamlessly fuses the real world and the virtual world in real time. One of the methods for realizing the MR system is to use a video see-through head-mounted display (HMD: Head Mounted Display). In this MR system, the field of view of the HMD wearer is imaged by a camera, and an image obtained by synthesizing the captured image with CG (Computer Graphics) is presented to the HMD wearer.

  In Patent Document 1, as an MR system using a video see-through HMD, a first camera that captures a field of view of an HMD wearer and a second camera that captures an image for detecting the position and orientation of the HMD A technique using the provided HMD has been proposed.

  The processing for acquiring the position and orientation of the HMD from the captured image and the processing for generating a CG corresponding to the position and orientation of the HMD have a high calculation processing cost. For this reason, the HMD is connected to an external image processing apparatus such as a PC (Personal Computer), and the captured video is transmitted to the image processing apparatus. Then, the position and orientation of the HMD are detected on the image processing apparatus, the CG is generated according to the position and orientation of the HMD, and the CG is synthesized with the captured image. The CG is sent back to the HMD and then displayed on the HMD. . In such a case, in order to allow the HMD wearer to experience the MR space while freely moving, the HMD and the image processing apparatus are often connected wirelessly. However, in general, wireless communication has a narrower communication band than wired communication, so it is necessary to reduce the amount of video data to be transmitted wirelessly. Further, since the HMD is driven by a battery or the like, it is necessary to reduce the amount of video data to be wirelessly transmitted from the viewpoint of power consumption.

  As a known technique for reducing the amount of video data to be wirelessly transmitted, a method of compressing data using a video coding technique is known. Patent Document 2 proposes a technique for detecting a difference between video frames and extracting and transmitting only a region changed between frames.

  In the MR system using the video see-through HMD described above, the image processing apparatus detects the position and orientation of the HMD by extracting an index such as a marker or a natural feature point from the captured image by image analysis.

JP 2004-205711 A JP 2001-320707 A

  However, this detection process requires high-definition video data, and if the video data is significantly compressed, the necessary and sufficient video quality cannot be obtained, and the accuracy of the position and orientation detection of the HMD decreases. There is a problem. Further, as in the technique described in Patent Document 1, in the method of extracting and transmitting only the region that has changed between frames, when the HMD wearer changes the line-of-sight direction, many change regions occur, and the amount of data is reduced. There is a problem that it is difficult to reduce.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for reducing the amount of video data to be transmitted while suppressing a decrease in position and orientation detection accuracy.

In order to achieve the above object, an information processing apparatus according to an aspect of the present invention includes the following arrangement. That is,
An information processing apparatus capable of communicating with an image processing apparatus,
An acquisition means for acquiring a movement rotation amount including at least one of a movement amount and a rotation amount of the information processing apparatus;
Control means for reducing the amount of data of a captured image to be transmitted to the image processing device when the amount of movement rotation acquired by the acquisition unit is a predetermined value or less.

  According to the present invention, it is possible to reduce the amount of video data to be transmitted while suppressing a decrease in position and orientation detection accuracy.

It is a figure which shows the structural example of the image processing system which concerns on one Embodiment of this invention. It is a figure which shows the structural example of the functional block of the information processing apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the structural example of the functional block of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a sequence diagram which shows the operation example of the information processing apparatus and image processing apparatus which concern on Embodiment 1 of this invention. It is a sequence diagram which shows the operation example of the information processing apparatus and image processing apparatus which concern on Embodiment 1 of this invention. It is a figure which shows the structural example of the functional block of the information processing apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the structural example of the functional block of the image processing apparatus which concerns on Embodiment 2 of this invention. It is a sequence diagram which shows the operation example of the information processing apparatus which concerns on Embodiment 2 of this invention, and an image processing apparatus. It is a sequence diagram which shows the operation example of the information processing apparatus which concerns on Embodiment 2 of this invention, and an image processing apparatus. It is a figure which shows the operation example of each part of the information processing apparatus which concerns on one Embodiment of this invention, and an image processing apparatus.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
<1. Configuration of image processing system>
FIG. 1 is a diagram illustrating a configuration example of an image processing system (for example, an MR system) according to an embodiment of the present invention. An image processing system (MR system) according to an embodiment of the present invention includes a video see-through information processing device (head-mounted display device (HMD100)) worn by a user and an image processing device (PC101). And can communicate with each other. In the space where the user experiences MR, an index for the PC 101 to detect the position and orientation of the HMD 100 is installed. For example, the index may be constituted by circular markers each having a different color, or may be constituted by feature points such as natural features each having a different feature. It is also possible to use a square index formed by a square area having a certain area. Any form may be used as long as the image coordinates can be detected on the captured image and the pattern indicating which index is identifiable.

  The HMD 100 is attached with a main camera and a side camera. The main camera video is used to generate a video to be displayed on the HMD 100. Further, the side camera image is used for detecting the position and orientation of the HMD 100. In the present embodiment, in order to store a large number of indexes in the video, a camera having a wider angle of view than the main camera is used as the side camera. Note that the configurations of the main camera and the side camera are not limited to this, and may be any configuration depending on the purpose, such as using a camera having a different focal length as in the technique described in Patent Document 1. Further, the HMD 100 is provided with a three-dimensional position and orientation sensor. The main camera video data 108, the side camera video data 109, and the detection result of the three-dimensional position and orientation sensor are transmitted to the PC 101. As a transmission method, an arbitrary wireless method such as IEEE802.11n or IEEE802.11ac or a wired method such as Gigabit Ethernet (registered trademark) or IEEE1394 can be used. However, the transmission method is not limited to these communication methods. Absent.

  The PC 101 analyzes the side camera video data 109 received from the HMD 100. In the side camera image, indices 102 to 107 are imaged, and after detecting these indices, the position and orientation of the HMD 100 are acquired from the shape. Further, as described later, the position and orientation of the HMD 100 are acquired based on the detection result of the three-dimensional position and orientation sensor. The PC 101 generates a CG 110 corresponding to the position and orientation of the HMD 100 and combines the generated CG 110 on the main camera video data 108 to generate a CG composite video 111. Then, the PC 101 transmits the CG composite video 111 to the HMD 100, and the HMD 100 displays the received CG composite video 111.

<2. Functional block of information processing apparatus>
FIG. 2 is a diagram illustrating a configuration example of functional blocks of the information processing apparatus (HMD 100) according to the present embodiment. FIG. 10 is a diagram showing the operation timing of each part of the HMD 100 and the PC 101. Hereinafter, the configuration and operation of the HMD 100 will be described with reference to FIGS. 2 and 10.

  First, as shown in FIG. 2, the HMD 100 includes a main camera 200, a side camera 201, a three-dimensional position and orientation sensor 202, a timing control unit 203, a movement rotation amount acquisition unit 204, and a data amount control unit 205. A transmission unit 206, an antenna 207, a reception unit 208, and a display unit 209.

  The main camera 200 and the side camera 201 are constituted by a charge coupled device such as a CCD (Charge Coupled Device) and an optical system. The main camera 200 captures a real space image that substantially matches the field of view of the HMD wearer. The side camera 201 captures a real space image having a wider angle of view than the main camera image.

  The three-dimensional position / orientation sensor 202 includes a magnetic sensor and a gyro sensor (acceleration, angular velocity), and detects the acceleration in the three-dimensional direction of the HMD 100 and the angular velocity in the pan / tilt / yaw direction. The timing control unit 203 outputs an imaging timing signal to the main camera 200, the side camera 201, and the moving rotation amount acquisition unit 204 at every predetermined period. In addition, a detection timing signal is output to the three-dimensional position / orientation sensor 202 at predetermined intervals.

  In the present embodiment, it is assumed that the frame rate of the main camera 200 and the side camera 201 is controlled to 60 fps and the sampling frequency of the three-dimensional position and orientation sensor 202 is controlled to 600 Hz by these timing signals. Note that these frame rate and sampling frequency are not limited to these, and can be set to arbitrary values.

  The movement rotation amount acquisition unit 204 operates based on the imaging timing signal output from the timing control unit 203 and integrates the output value of the three-dimensional position / orientation sensor 202 so that the HMD 100 between the imaging frames in the three-dimensional direction. At least one of the movement amount and the rotation amount in the pan / tilt / yaw direction (hereinafter referred to as the movement rotation amount) is acquired. In the present embodiment, the three-dimensional position and orientation sensor 202 outputs acceleration / angular velocity data 10 times between imaging frames (1/60 seconds). The movement rotation amount acquisition unit 204 acquires the movement rotation amount of the HMD 100 between imaging frames by integrating the ten acceleration / angular velocity data.

  The data amount control unit 205 controls the amount of video data output from the side camera 201 based on the movement rotation amount of the HMD 100 acquired by the movement rotation amount acquisition unit 204. Specifically, when the moving rotation amount of the HMD 100 is equal to or less than a predetermined value, the operation is performed so as to thin out the imaging frames output from the side camera 201.

  In the timing chart shown in FIG. 10, frame N-1 and frame N + 1 show an operation example when the amount of movement rotation of the HMD 100 exceeds a predetermined value. On the other hand, the frame N shows an operation example when the moving rotation amount of the HMD 100 is equal to or less than a predetermined value. The transmission unit 206 receives the captured video output from the main camera 200, the captured video output from the data amount control unit 205, and the movement rotation amount output from the movement rotation amount acquisition unit 204. The transmission unit 206 adds a header indicating the data type of the transmission data, and generates a radio signal by performing modulation processing according to the communication method to be used. In a frame in which the amount of movement rotation of the HMD 100 exceeds a predetermined value, the transmission unit 206 generates a radio signal including the header 1003, the main camera video 1000, and the side camera video 1001. At this time, as the header 1003, information indicating that the main camera video and the side camera video are transmitted in the frame is added.

  On the other hand, in a frame in which the movement rotation amount of the HMD 100 is equal to or less than a predetermined value, the transmission unit 206 generates a wireless signal including the header 1003, the movement rotation amount 1002, and the main camera image 1000. At this time, as the header 1003, information indicating that the moving rotation amount and the main camera image are transmitted in the frame is added. The radio signal generated by the transmission unit 206 is transmitted to the PC 101 via the antenna 207.

  In order to reduce the amount of wireless transmission data, the main camera video and the side camera video are Motion-JPEG or H.264. Alternatively, compression may be performed using H.264 / AVC. For example, when 60 fps video data is compressed to 25 Mbps, the amount of video data per frame is about 420 kbit. On the other hand, the amount of movement rotation is 96 bits when the resolution is 16 bits, and the amount of data is very small compared to the video data.

  A CG composite video is transmitted from the PC 101, and the HMD 100 receives this via the antenna 207. The receiving unit 208 obtains a CG composite video by demodulating the received wireless signal and outputs it to the display unit 209. The display unit 209 is configured by a liquid crystal display, an organic EL (Electro-Luminescence), or the like, and presents an MR space to the HMD wearer by displaying a CG composite image transmitted from the PC 101.

  Note that each functional unit shown in FIG. 2 is implemented in the apparatus as hardware or software. When the functional unit is implemented as software, a program for executing these functions is stored in a storage unit (memory such as ROM or RAM) provided in the HMD 100. Then, the CPU provided in the HMD 100 executes the program stored in the storage unit, thereby realizing the functions provided by these functional units.

<3. Functional Block of Image Processing Device>
FIG. 3 is a diagram illustrating a configuration example of functional blocks of the image processing apparatus (PC 101) according to the present embodiment. The PC 101 includes an antenna 300, a reception unit 301, an image analysis unit 302, a position / orientation acquisition unit 303, a CG generation unit 304, a CG synthesis unit 305, and a transmission unit 306.

  A radio signal transmitted from the HMD 100 is received by the antenna 300 and output to the receiving unit 301. The receiving unit 301 demodulates the radio signal and outputs the received data to the subsequent processing unit. The image analysis unit 302 obtains the position and orientation of the HMD 100 by detecting an index in the side camera video. The number of indices used for acquiring the position and orientation may be one, but a plurality of indices may be used simultaneously. When using a plurality of indices simultaneously, the positional relationship between the indices can be defined in advance, and the position and orientation of the HMD 100 with respect to the indices can be acquired from the positional relationship.

  In this embodiment, the position and orientation of the HMD 100 are acquired using the indices 102 to 107 in the side camera image. However, the position and orientation are detected by detecting feature points in the image and using them in combination. It is also possible to improve the detection accuracy. Note that the image analysis unit 302 does not execute the position / orientation acquisition process in a frame in which the side camera video is not transmitted from the HMD 100 to the PC 101.

  When the position / orientation acquisition result is input from the image analysis unit 302, the position / orientation acquisition unit 303 outputs the value to the CG generation unit 304. When the position / orientation acquisition result is not input from the image analysis unit 302, that is, in a frame in which the side camera video is not transmitted, the position / orientation acquisition unit 303 detects the position / orientation using the movement rotation amount transmitted from the HMD 100. Specifically, the position and orientation of the HMD 100 in this frame is acquired by adding the amount of movement rotation transmitted from the HMD 100 to the position and orientation acquired in the frame that received the side camera video.

  The CG generation unit 304 stores data related to a virtual object constituting the virtual space, and generates an image when the virtual space is viewed from a viewpoint having the position and orientation of the HMD 100 as a CG image. The CG combining unit 305 generates a CG combined image by combining the main camera image and the CG image generated by the CG generating unit 304. The CG composite video is transmitted to the HMD 100 via the transmission unit 306.

  Note that each functional unit shown in FIG. 3 is implemented in the apparatus as hardware or software. When the functional unit is implemented as software, a program for executing these functions is stored in a storage unit (memory such as ROM or RAM) provided in the PC 101. Then, the CPU provided in the PC 101 executes the program stored in the storage unit, thereby realizing the functions provided by these functional units.

<4. Sequence showing operation examples of information processing apparatus (HMD100) and image processing apparatus (PC101)>
FIG. 4 is a sequence diagram illustrating an operation example of the HMD 100 and the PC 101 when the amount of movement rotation acquired by the HMD 100 exceeds a predetermined value. FIG. 4 shows processing for one video frame.

  First, in step S400, the HMD 100 acquires the main camera video and the side camera video based on the control of the timing control unit 203. Subsequently, in step S401, the movement rotation amount of the HMD 100 is acquired by integrating the output value of the three-dimensional position and orientation sensor 202. In step S402, the HMD 100 transmits the main camera video and the side camera video to the PC 101 because the amount of movement rotation exceeds a predetermined value.

  In step S <b> 403, the PC 101 analyzes the side camera video received from the HMD 100 and obtains the position and orientation of the HMD 100. In step S404, the PC 101 generates a CG image based on the position and orientation of the HMD 100 acquired in step S403. In step S405, the PC 101 generates a CG composite video by combining the generated CG image with the main camera video. In step S <b> 406, the PC 101 transmits the CG composite video to the HMD 100. In step S407, the HMD 100 displays the received CG composite video.

  FIG. 5 is a sequence diagram illustrating an operation example of the HMD 100 and the PC 101 when the amount of movement rotation acquired by the HMD 100 is equal to or less than a predetermined value. FIG. 5 also shows processing for one video frame, as in FIG. Also, the same processing flow as that in FIG. 4 is given the same reference numeral, and only the parts different from the processing flow in FIG. 4 will be described.

  In step S500, since the moving rotation amount is equal to or less than the predetermined value, the HMD 100 thins out the side camera image and transmits only the main camera image and the moving rotation amount to the PC 101. As described above, since the data amount of the moving rotation amount is very small compared to the video data, the amount of wireless transmission data is greatly reduced in the frame obtained by thinning the side camera video.

  In step S501, the PC 101 obtains the position and orientation of the HMD 100 in this frame by adding the amount of movement rotation received from the HMD 100 to the position and orientation of the HMD 100 acquired in the previous frame. Since the output of the three-dimensional position / orientation sensor 202 includes an error, the movement rotation amount acquired by the HMD 100 does not necessarily match the actual movement rotation amount. When this error is large, problems such as a shift in the position of the virtual object generated by the CG generation unit 304 occur.

  However, in this embodiment, since the position and orientation of the HMD 100 are acquired using the sensor output only when the amount of movement rotation acquired by the HMD 100 is small, that is, when the absolute error of the sensor is small, the position and orientation of the HMD 100 is large. There is no error. Therefore, it is possible to reduce the amount of data transmitted from the HMD 100 to the PC 101 while suppressing a decrease in detection accuracy of the position and orientation of the HMD 100.

  That is, according to the present embodiment, it is possible to provide an MR system that can efficiently use radio frequency resources. Moreover, since power consumption can be suppressed by reducing the radio transmission time, an MR system suitable for battery driving of the HMD can be provided.

  Note that the side camera video data reduction process described above is configured to operate only under specific conditions such as when the available communication band is narrowed or when the battery level of the HMD 100 is low. Also good. Alternatively, the HMD 100 may be provided with an operation mode such as a power saving mode, and the operation of this embodiment may be performed only when the HMD wearer selects this mode.

  In the present embodiment, the PC 101 does not perform the position / orientation calculation using the movement rotation amount in a frame in which the movement rotation amount of the HMD 100 is equal to or greater than a predetermined value, so that the movement rotation amount is not transmitted from the HMD 100 to the PC 101. . However, since the amount of movement rotation data is small, the amount of movement rotation may be transmitted in all frames.

  In the present embodiment, the HMD 100 operates so that the side camera image is thinned and wirelessly transmitted in a frame in which the movement rotation amount is equal to or less than the predetermined value. However, the average moving rotation speed is obtained and the side camera image is obtained according to this value. May be thinned out. For example, when the moving rotation speed exceeds a predetermined value, the side camera image is transmitted every frame, and when the moving rotation speed is equal to or less than the predetermined value, the side camera image is transmitted every N frames (N is a positive integer). To be configured. By this operation, when the amount of movement rotation of the HMD 100 is small, the operation is performed to thin out the side camera image. Therefore, the amount of data transmitted from the HMD 100 to the PC 101 while suppressing the decrease in the position and orientation detection accuracy of the HMD 100 as in the present embodiment. Can be reduced.

  Further, the HMD 100 operates to reduce the data amount by thinning out the side camera video, but the data amount may be reduced by increasing the image compression rate of the side camera video by an image compression unit (not shown). . In this case, the PC 101 uses the position / orientation detection result output from the image analysis unit 302 when the index can be detected from the side camera video. On the other hand, when the index cannot be detected from the side camera image, the position and orientation are acquired using the movement rotation amount transmitted from the HMD 100. By controlling the compression rate in this way, the same effect as in the present embodiment can be obtained.

  If a wireless transmission error of the moving rotation amount occurs in a frame in which the side camera image is thinned out, the PC 101 cannot detect the position and orientation of the HMD 100, and the position of the CG is greatly shifted in that frame. In order to prevent this, in the frame in which the HMD 100 thins out the side camera video, the transmission unit 206 may reduce the modulation and coding rate of the movement rotation amount. Specifically, for example, modulation processing is normally performed using 802.11n MCS (Modulation and Coding Scheme) 7, but MCS 0 is used only for the movement rotation amount modulation processing in a frame in which the side camera video is thinned out. . By this operation, it is possible to reduce the wireless transmission error of the movement rotation amount in the frame in which the side camera image is thinned out.

  In the present embodiment, the HMD 100 includes a main camera and a side camera, and the side camera video data is reduced. However, the present invention is not limited to this, and can be applied to any system that transmits video for detecting an index. For example, the PC 101 may be configured to detect the index from the main camera video. In this case, the HMD 100 includes only the main camera 200 as an imaging unit, and is configured to reduce main camera video data to be transmitted to the PC 101 based on the acquisition result of the movement rotation amount acquisition unit 204. Further, the PC 101 is configured to transmit the CG 110 to the HMD 100 and generate the CG composite video 111 in the HMD 100. With this configuration, even when the HMD 100 thins out the main camera video based on the movement rotation amount acquisition result, the CG composite video 111 can be displayed on the HMD 100.

  In this embodiment, the movement rotation amount acquisition unit 204 of the HMD 100 acquires the movement rotation amount by integrating the output value of the three-dimensional position and orientation sensor, but transmits the sensor output value from the HMD 100 to the PC 101. The PC 101 may acquire the movement rotation amount.

  As described above, according to the present embodiment, it is possible to efficiently use radio frequency resources by reducing the amount of video data transmitted from the HMD to the image processing device while suppressing a decrease in the accuracy of HMD position and orientation detection. MR system can be provided. Further, since power consumption can be suppressed by reducing the radio transmission time, it is possible to provide an MR system suitable for battery driving of the HMD.

(Embodiment 2)
In the first embodiment, the HMD 100 determines whether or not to reduce the side camera video data. However, the PC 101 may determine the side camera video data. In the present embodiment, the PC 101 acquires the movement rotation amount of the HMD 100 based on the analysis result of the side camera image, and compares the acquisition result with the movement rotation amount acquired by the sensor output. As a result of this comparison, if the error in the amount of movement rotation obtained from the sensor output is less than or equal to a predetermined value, the HMD 100 is instructed to reduce side camera video data.

  Hereinafter, the configuration and operation of the functional blocks of the HMD 100 and the PC 101 in this embodiment will be described. In the figure, the same reference numerals as those in the first embodiment are given to the same blocks as those in the first embodiment, and the description thereof will be omitted. To do. Note that each function unit is implemented in the apparatus as hardware or software, as described in the first embodiment.

  FIG. 6 is a diagram illustrating a configuration example of functional blocks of the HMD 100 in the present embodiment. The functional block configuration of the HMD 100 in the first embodiment is different in that the data amount control unit 605 reduces the side camera video data based on the data amount control instruction information received by the receiving unit 608 from the PC 101. In the first embodiment, the moving rotation amount is not transmitted from the HMD 100 to the PC 101 in the frame in which the side camera image is not thinned out. However, in the present embodiment, the transmission unit 206 transmits the moving rotation amount in all frames. It shall be.

  FIG. 7 is a diagram illustrating a configuration example of functional blocks of the PC 101 in the present embodiment. Compared to the first embodiment, the PC 101 of this embodiment further includes a moving rotation amount acquisition unit 400 and a data amount control instruction information generation unit 401.

  The movement rotation amount acquisition unit 400 operates when the position / orientation acquisition result of the HMD 100 is output from the image analysis unit 302 in successive frames. The movement rotation amount acquisition unit 400 acquires the movement rotation amount of the HMD 100 between frames based on the position / orientation acquisition result of the image analysis unit 302. Specifically, the movement rotation amount of the HMD 100 is acquired by taking the difference between the position and orientation acquisition result of the previous frame and the current frame position and orientation acquisition result.

  The data amount control instruction information generation unit 401 compares the movement rotation amount output from the movement rotation amount acquisition unit 400 with the movement rotation amount transmitted from the HMD 100. If the difference between the two is equal to or smaller than the predetermined value, it is determined that the sensor error is small, and the HMD 100 is instructed to reduce the side camera video. If the difference between the two exceeds a predetermined value, it is determined that the sensor error is large, and the HMD 100 is instructed to transmit the side camera video.

  FIG. 8 and FIG. 9 are sequence diagrams showing an operation example of the HMD 100 and the PC 101 in the present embodiment. In the figure, the same processing flow as that of the first embodiment is given the same reference numeral, and the description thereof is omitted.

  FIG. 8 is a sequence diagram illustrating an operation example of the HMD 100 and the PC 101 in a frame in which a side camera video is transmitted from the HMD 100 to the PC 101. In step S800, the HMD 100 transmits data based on the data amount control instruction information received from the PC 101 in the previous frame to the PC 101. In FIG. 8, information instructing transmission of the side camera video is received from the PC 101, and the HMD 100 transmits the main camera video, the side camera video, and the movement rotation amount to the PC 101.

  When the PC 101 receives a side camera image from the HMD 100, the PC 101 compares the amount of movement rotation of the HMD 100 acquired by image analysis with the amount of movement rotation received from the HMD 100 in step S801. In step S802, if the difference between the two is equal to or smaller than the predetermined value, the PC 101 determines that the sensor error is small, generates data amount control instruction information that instructs the HMD 100 to reduce the side camera video, In step 803, the data amount control instruction information is transmitted to the HMD 100 together with the CG composite video.

  FIG. 9 is a sequence diagram illustrating an operation example of the HMD 100 and the PC 101 in a frame in which no side camera video is transmitted from the HMD 100 to the PC 101. In step S900, the HMD 100 transmits data based on the data amount control instruction information received from the PC 101 in the previous frame to the PC 101. In FIG. 9, information instructing reduction of the side camera video is received from the PC 101, and the HMD 100 thins out the side camera video and transmits the main camera video and the moving rotation amount to the PC 101.

  In step S <b> 901, the PC 101 generates data amount control instruction information to be transmitted to the HMD 100. In the present embodiment, when the inter-frame movement rotation amount of the HMD 100 exceeds a predetermined value, information that instructs the HMD 100 to transmit the side camera video is generated. On the other hand, when the inter-frame movement rotation amount of the HMD 100 is equal to or less than a predetermined value, information for instructing the HMD 100 to reduce the side camera video is generated. When the side camera video is reduced over a plurality of consecutive frames, the PC 101 cumulatively adds the inter-frame movement rotation amount of each frame, and when this value exceeds a predetermined value, instructs the HMD 100 to transmit the side camera video. Information may be generated. In addition, the HMD 100 may always generate information instructing transmission of the side camera video for the frame next to the frame from which the side camera video is reduced.

  In the present embodiment, the PC 101 compares the inter-frame movement rotation amount of the HMD 100 acquired by image analysis with the movement rotation amount acquired based on the output of the three-dimensional position / orientation sensor 202 of the HMD 100, and the comparison result is obtained. The side camera image is reduced accordingly. That is, since the position / orientation of the HMD 100 is acquired using the sensor output only when the sensor error is small, it is possible to reduce the amount of data transmitted from the HMD 100 to the PC 101 while suppressing a decrease in the position / orientation detection accuracy of the HMD 100. It becomes.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  200: main camera, 201: side camera, 202: three-dimensional position and orientation sensor, 203: timing control unit, 204: movement rotation amount acquisition unit, 205: data amount control unit, 206: transmission unit, 207: antenna, 208: Reception unit, 209: display unit, 300: antenna, 301: reception unit, 302: image analysis unit, 303: position and orientation acquisition unit, 304: CG generation unit, 305: CG synthesis unit, 306: transmission unit, 400: movement Rotation amount acquisition unit 401: Data amount instruction information generation unit

Claims (15)

An information processing apparatus capable of communicating with an image processing apparatus,
An acquisition means for acquiring a movement rotation amount including at least one of a movement amount and a rotation amount of the information processing apparatus;
An information processing apparatus comprising: a control unit that reduces a data amount of a captured image to be transmitted to the image processing device when a movement rotation amount acquired by the acquisition unit is equal to or less than a predetermined value.   The information processing apparatus according to claim 1, wherein the control unit reduces the data amount by thinning out frames of the captured image when the movement rotation amount is equal to or less than a predetermined value.   The control means reduces the data amount by thinning out frames of a captured image captured for acquiring the position and orientation of the information processing apparatus when the amount of rotation of rotation is equal to or less than a predetermined value. The information processing apparatus according to claim 1 or 2.   When the movement rotation amount is equal to or less than a predetermined value, the control means transmits the captured image captured from the viewpoint of the wearer of the information processing device and the movement rotation amount to the image processing device. The information processing apparatus according to claim 3.   The information processing apparatus according to claim 4, wherein the control unit lowers a modulation and coding rate of the movement rotation amount transmitted to the image processing apparatus. Further comprising image compression means for compressing the captured image;
The information processing apparatus according to claim 1, wherein the control unit reduces the data amount by increasing an image compression rate of the captured image when the movement rotation amount is equal to or less than a predetermined value. First imaging means for imaging from a viewpoint of a wearer of the information processing apparatus;
The information processing apparatus according to claim 1, further comprising: a second imaging unit configured to perform imaging for acquiring the position and orientation of the information processing apparatus. Receiving means for receiving a CG image generated based on information acquired from the information processing apparatus from the image processing apparatus;
The information processing apparatus according to claim 1, further comprising display means for displaying the CG image.   The information processing apparatus according to claim 1, wherein the information processing apparatus is a head-mounted display device. An image processing apparatus capable of communicating with an information processing apparatus,
Receiving means for receiving a movement rotation amount including at least one of a movement amount and a rotation amount of the information processing apparatus, and a captured image captured by the information processing apparatus;
Obtaining means for obtaining a movement rotation amount of the information processing device based on the captured image;
Generating means for generating instruction information for controlling the amount of data transmitted by the information processing device based on the movement rotation amount received by the reception means and the movement rotation amount acquired by the acquisition means;
An image processing apparatus comprising: a transmission unit configured to transmit the instruction information to the information processing apparatus.   The generation unit reduces the amount of data transmitted by the information processing apparatus when a difference between the movement rotation amount received by the reception unit and the movement rotation amount acquired by the acquisition unit is a predetermined value or less. The image processing apparatus according to claim 10, wherein instruction information for generating the instruction information is generated. An information processing apparatus control method capable of communicating with an image processing apparatus,
An acquisition step of acquiring an amount of movement rotation including at least one of an amount of movement and an amount of rotation of the information processing apparatus;
An information processing apparatus comprising: a control unit that reduces a data amount of a captured image to be transmitted to the image processing apparatus when the movement rotation amount acquired by the acquisition unit is equal to or less than a predetermined value; Control method. An image processing apparatus control method capable of communicating with an information processing apparatus,
A receiving step for receiving a movement rotation amount including at least one of a movement amount and a rotation amount of the information processing apparatus, and a captured image captured by the information processing apparatus;
An acquisition step of acquiring an amount of movement rotation of the information processing device based on the captured image;
The generation unit generates instruction information for controlling the amount of data transmitted by the information processing device based on the movement rotation amount received in the reception step and the movement rotation amount acquired in the acquisition step. Generation process;
And a transmission step of transmitting the instruction information to the information processing apparatus.   A program for causing a computer to execute each step of the control method for the information processing apparatus according to claim 12.   A program for causing a computer to execute each step of the control method for the image processing apparatus according to claim 13.

Images