JP2016082528A - System, program, display device, image processing device, control method for system, control method for display device, and control method for image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system, a program, devices and control methods capable of suppressing influences of marker detection rate reduction caused by image quality deterioration while reducing a transmission band through compression.SOLUTION: A display device (HMD part 100) detects first index information for each of a plurality of regions which are obtained by dividing a captured image, generates a compression image for each region on the basis of a detection result of first index information detection means and transmits to an image processing device 110 the first index information and the compression image regarding a region in which the first index information is detected, and the compression image regarding a region in which the first index information is not detected. The image processing device receives the first index information and the compression image from the display device, performs image decompression, detects second index information from the region in which the first index information is not detected, in a decompression image, performs position measurement to generate positional attitude information of the display device from the first and second index information, and generates a combination image by combining CG which is generated based on the positional attitude information, with the decompression image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system for synthesizing two images, a program, an image processing apparatus, and a display apparatus that receives and displays an image from such an image processing apparatus, and in particular, virtual reality technology and mixed reality technology using HMD (Head Mounted Display). About.

  In recent years, a mixed reality, so-called MR (Mixed Reality) technology has been known as a technology that seamlessly fuses the real world and the virtual world in real time. A technique using video see-through HMD is known as one of MR techniques. This is a technique that allows an HMD user to observe an image obtained by capturing a subject that substantially matches the subject observed from the pupil position of the HMD user with a video camera or the like and superimposing and displaying CG (Computer Graphics) on the captured image. is there.

  FIG. 2 is a functional block diagram of a general video see-through mixed reality system that performs wireless image transmission. This system is composed of an HMD that captures images from the outside world and displays images to the user, and an image processing device that creates images superimposed on CG. In order to experience the MR space while moving freely using the HMD, it is better to communicate between the HMD and an external device such as a PC by wireless transmission. However, in general, wireless communication has a narrow communication band and is unstable as compared with wired communication. As a method for solving this, in Patent Document 1, in addition to compressing and transmitting image data, a stable image can be obtained by changing the compression rate of the image and adjusting the data amount according to the situation of the communication network. Data transmission is possible.

In FIG. 2, the video see-through mixed reality system includes a head-mounted display device unit (hereinafter, HMD unit 200) and an image processing unit 210. The image processing unit 210 generally uses a device having a high-performance arithmetic processing function or graphic display function such as a personal computer or a workstation. The HMD unit 200 includes an imaging unit 201, an image compression unit 203, a first network communication unit 204, and an image display unit 205.
In the configuration of FIG. 2, index information is detected from a decompressed image, that is, a stretched captured image. However, consider a configuration in which index information detection processing is performed on an image before compression, that is, index information is detected on the HMD side. Is also possible.

  FIG. 3 is a functional block diagram of a general video see-through mixed reality system that detects index information on the HMD side and performs wireless image transmission. In FIG. 3, the video see-through mixed reality system includes an HMD unit 300 and an image processing unit 310. The HMD unit 300 includes an imaging unit 301, an index information detection unit 302, an image compression unit 303, a first network communication unit 304, and an image display unit 305. The image processing unit 310 includes a second network communication unit 311, an image expansion unit 312, a position measurement unit 314, and a CG composition unit 315.

By wearing the video see-through HMD shown in FIG. 2 or 3, it is possible to experience a mixed reality world where the real world and the virtual world are seamlessly fused in real time.
A general MR technique and system configuration are disclosed in Patent Document 2.

A concept of generating position and orientation information from a captured image using a marker as index information will be described with reference to FIG.
It is assumed that the marker 703 has a positional relationship with the imaging device in advance. When the marker 703 is displayed in the real space image 701, the index information detection unit and the position measurement unit detect the marker 703 from the image data, and acquire information such as the size and shape of the marker 703, a fill pattern, and the like. . From this information, it is possible to calculate the relative positional relationship between the marker 703 and the imaging means main body (HMD) and three-dimensional position and orientation information regarding the direction in which the HMD user is observing the marker.

  In FIG. 7, as an example, a three-dimensional coordinate system with the central portion of the marker 703 as the origin is assumed. However, the origin of the coordinate system does not need to be set on the marker 703, and can be set to an arbitrary position by associating the relative positional relationship between the origin of the coordinate system and the marker 703. Moreover, the marker used for position and orientation information generation is not limited to a single marker, and a plurality of markers can be used simultaneously. When a plurality of markers are used at the same time, it is possible to calculate the direction in which the marker is observed from the relative positional relationship by defining the positional relationship between the markers in advance.

Therefore, it is not a marker that can be identified up to the direction by the internal fill pattern as shown in FIG. 7, but one-dimensional information that does not have directionality information such as a color marker or a light emitting element such as an LED is used. It is also possible to use the marker that you have. It is also possible to extract feature points in the image such as the contour line 704 of the table, a specific color in the image, and the like, and calculate position / orientation information using them. By using a plurality of the same type of markers, using several types of markers at the same time, or combining the marker information and the feature point information in the image, higher-accuracy position and orientation information can be generated. Furthermore, since the positional relationship of multiple markers and feature points is associated, even if not all markers or feature points are displayed in the image, the position of each marker or feature point can be estimated. Is possible.
Realization of MR technology using markers is disclosed in Non-Patent Document 1.

Japanese Patent No. 3944726 JP-A-11-88913

Kato, H.C. Billinghurst, M .; (1999) Marker Tracking and HMD Calibration for a video-based Augmented Reality Conferencing System. In Proceedings of the 2nd International Workshop on Augmented Reality (IWAR 99). October, San Francisco, USA

  The method of Patent Document 1 is premised on irreversible compression of image data, and some information is lost from the original image in the transmission image, and the image quality deteriorates. Therefore, when detecting index information such as a marker from a stretched image, that is, a stretched captured image, as in the configuration of FIG. 2, the detection accuracy may be reduced, and an accurate position and orientation may not be calculated.

  The influence of image quality deterioration in the index information detection process will be described with reference to FIG. In FIG. 4, four markers (401, 402, 403, 404) are arranged in the captured image 400 before compression. In the compressed captured image 410 generated by compressing the captured image 400, the marker corresponding to the marker 401 is 411 as is apparent from FIG. Markers (enlarged) 405 and markers (enlarged) 415 are obtained by enlarging the markers 401 and 411, respectively. It can be seen that the image quality of the marker (enlarged) 415 is clearly deteriorated with respect to the marker (enlarged) 405. Due to such image quality deterioration, there is a possibility that the index information in the captured image after compression represented by the marker 411 is not correctly recognized and may not be detected as a result.

  As in the configuration of FIG. 3, by performing the index information detection process using the pre-compressed captured image on the HMD side, it is not necessary to consider a decrease in detection accuracy due to image compression transmission. However, considering that the HMD is a device attached to the human body and that light weight and compactness are important, it is difficult to perform all of the index information detection processing actually performed by the image processing unit on the HMD side. . That is, even when the index information is detected only on the HMD side, the detection accuracy and the detection speed are lowered.

In order to achieve the above object, the system of the present invention comprises the following arrangement. That is, a system including a display device and an image processing device,
The display device
Imaging means for imaging a real space and generating a captured image;
First index information detection means for performing detection processing of first index information for each of a plurality of regions obtained by dividing the captured image;
An image compression unit that generates a compressed image by compressing the captured image for each region based on a detection result of the first index information detection unit;
The first index information and the compressed image are transmitted for the area where the first index information is detected, and the compressed image is transmitted for the area where the first index information is not detected. Network communication means;
Display means;
With
The image processing apparatus includes:
Second network communication means for receiving the compressed image from the first network communication means;
Image expansion means for expanding the compressed image to generate an expanded image;
Second index information detecting means different from the first index information detecting means for detecting second index information from an area where the first index information of the decompressed image is not detected;
Position measuring means for generating position and orientation information of the display device from the first index information and the second index information;
CG combining means for generating a combined image by combining CG generated based on the position and orientation information with the expanded image;
With
The second network communication means transmits the composite image to the first network communication means;
The first network communication means receives the composite image from the second network communication means;
The display means displays the composite image.

  According to the present invention, it is possible to suppress the influence of a decrease in the marker detection rate due to image quality degradation while reducing the transmission band by compression.

1 is a block diagram showing a configuration example of an image processing apparatus according to an embodiment of the present invention. The block diagram which shows the structural example of the image processing apparatus in a prior art example. The block diagram which shows the structural example of the image processing apparatus in a prior art example. The figure which shows the influence of image quality degradation in the detection process of the index information in a prior art example. 5 is a flowchart illustrating a method for compressing a captured image according to the first embodiment. 6 is a flowchart illustrating a method for compressing a captured image according to the second embodiment. The figure which shows an example of a marker. FIG. 10 is a block diagram illustrating a configuration example of an image processing apparatus according to a fourth embodiment. The flowchart which showed the compression method in 4th Embodiment. FIG. 10 is a block diagram illustrating a configuration example of an image processing apparatus according to a fifth embodiment. The flowchart which showed the compression method in 5th Embodiment. The figure which shows an example of the state transition of each area | region of the captured image in 1st Embodiment. The figure which shows an example of the state transition of each area | region of the captured image in 2nd Embodiment. The figure which shows an example of the state transition of each area | region of the captured image in 4th Embodiment. The figure which shows an example of the state transition of each area | region of the captured image in 5th Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment]
A first embodiment realized by the present invention will be described with reference to FIG.
FIG. 1 is a functional block diagram of a video see-through mixed reality system. In FIG. 1, the image processing apparatus according to the present embodiment includes a head-mounted display device unit (hereinafter, HMD unit 100) and an image processing unit 110. Although the present embodiment will be described as being worn on the user's head, it may be configured to be worn on the user's face, such as a glasses-type display device.
The HMD unit 100 includes an imaging unit 101, a first index information detection unit 102, an image compression unit 103, a first network communication unit 104, and an image display unit 105.
The image processing unit 110 includes a second network communication unit 111, an image expansion unit 112, a second index information detection unit 113, a position measurement unit 114, and a CG synthesis unit 115.
The HMD unit 100 and the image processing unit 110 are connected via the first network communication unit 104 and the second network communication unit 111.
The HMD unit 100 and the image processing unit 110 communicate via a small network such as a WLAN (Wireless Local Area Network) or a WPAN (Wireless Personal Area Network). The HMD unit 100 and the image processing unit 110 are not limited to wireless communication, and a wired communication method may be used.

The imaging unit 101 captures an external observation image that substantially matches the line-of-sight position of the HMD user, and generates a captured image. It comprises two sets of image sensors for the right eye and left eye for generating a stereo image, an optical system, and a signal processing circuit such as a DSP for performing subsequent image processing.
The first index information detection unit 102 detects index information necessary for generating position and orientation information of the HMD user from the captured image.
The image compression unit 103 compresses the captured image and creates a compressed image in which the data amount is reduced with respect to the original captured image.
The first network communication unit 104 transmits the compressed image (captured image) and the index information to the image processing unit 110 via the wireless access point, and the composition generated by the image processing unit 110 via the wireless access point. Receive an image.
The display unit 105 displays the synthesized MR image. Similarly to the imaging unit 101, this is composed of two sets of display devices for right eye and left eye and an optical system. As the display device, a small liquid crystal display or a retina scan type device using MEMS is used.

The second network communication unit 111 transmits the combined image generated by the CG combining unit 115 to the HMD unit 100 via the wireless access point, and wirelessly transmits the compressed image (captured image) and the index information from the HMD unit 100. Receive via an access point.
The image decompression unit 112 decompresses (decodes) the compressed image transmitted from the HMD unit 100, and generates a decompressed image, that is, a decompressed captured image.
The second index information detection unit 113 detects index information necessary for generating position and orientation information of the HMD user from the expanded image, that is, the expanded captured image.
The position measurement unit 114 generates position and orientation information of the HMD user from the first and second index information. Since the position and orientation measurement using the index information is the same as that described in the background art section, the description is omitted here.
The CG synthesis unit 115 creates a synthesized image obtained by synthesizing the CG with the captured image from the captured image and the position and orientation information.

  A compressed image is generated by the image compression unit 103 from the captured image captured by the imaging unit 101 in the HMD unit 100. Similarly, from the captured image, the first index information detection unit 102 detects the first index information, and the compressed image and the first index information are transmitted to the image processing unit 110 by the first network communication unit 104. The

  In the image processing unit 110, the compressed image received by the second network communication unit 111 is decompressed (decoded) by the image decompressing unit 112, and a decompressed captured image is generated. The second index information detection unit 113 detects second index information from the expanded captured image, and the position measurement unit 114 generates the position and orientation information of the HMD user from the first index information and the second index information. To do. The CG synthesizing unit 115 generates a synthesized image obtained by synthesizing the CG with the captured image from the expanded captured image HMD user's position and orientation information, and sends the synthesized image to the HMD unit 100 via the second network communication unit 111. Send. The composite image received by the first network communication unit 104 in the HMD unit 100 is displayed to the HMD user via the image display unit 105.

  Here, in consideration of the characteristics of the HMD unit that is required to be lightweight and compact, the process of the first index information detection unit 102 needs to be simpler than the process of the second index information detection unit 113. There is. For example, when searching for index information, it is conceivable that deformation or rotation of the index information is not taken into account (omitted), or a range to be considered is limited (searching only for an index that is erect in front, etc.).

  The flow of image compression processing in the HMD unit of the first embodiment will be described with reference to FIG. MPEG-2 and H.264 In a standard image encoding / compression method such as H.264, an image is divided into regions (macroblocks) each having 16 pixels in the vertical direction and 16 pixels in the horizontal direction, and the image can be compressed at a desired compression rate in units of macroblocks. Similarly in this embodiment, the captured image is divided into macroblocks and compressed.

First, in S501, one captured image divided into regions (macroblocks) is united (usually from the upper left corner to the lower right corner, from left to right, and when reaching the right edge, returns to the left edge and moves down by one macroblock row. Select the raster scan order to move).
Next, in S502, it is checked whether or not the first index information detected by the first index information detection unit 102 is included in the selected macroblock. If included, the process proceeds to S503. The process proceeds to S504.

In S503, since index information is included, the image processing unit 110 determines that it is not necessary to perform further index information detection processing on the macroblock, and sets the image quality parameter to Q2 (low image quality). That is, the compression condition is set to low image quality.
On the other hand, in S504, since index information is not included (index information cannot be detected) and it is necessary to perform index detection processing in the image processing unit 110, the image quality parameter Q1 is set to suppress the influence of image degradation due to compression. Set to (High Quality). That is, the compression condition is set to high image quality. In the present embodiment, a quantized scaling value is assumed as the image quality parameter (compression condition), but the present invention is not limited to this.

Next, in S505, the selected macroblock is compressed using the image quality parameters set in S503 and S504.
Thereafter, S501 to S505 are repeated until it is determined in S506 that the compression processing for all macroblocks has been completed.

The image quality control for each area (macroblock) will be described with reference to FIG.
If the marker 401 is not detected by the first index information detection process using the pre-compression captured image 400, the image quality parameter is Q1 (high image quality) in S504 when the macroblocks around the marker 401 are encoded. Therefore, image quality deterioration after compression is suppressed. That is, it is possible to avoid compression with a low image quality such as the marker (enlarged) 415 that cannot be detected by the second index information detection process.

  On the other hand, if the marker 401 is detected by the first index information detection process, the image quality parameter of the macro block around the marker 401 is set to Q2 (low image quality) in S503. In other words, the data rate after compression is suppressed, and it is possible to reduce the network bandwidth. Since the marker has already been detected, there is no need to worry about the influence of the image quality deterioration due to compression on the second index information detection process.

FIG. 12 shows an example of state transition of each region in the first embodiment. In order to simplify the explanation, it is assumed that the captured image 1210 is divided into four regions 1211 to 1214 and processed as shown in FIG.
(A1) to (a3) are diagrams for explaining the processing in the HMD unit, and (a4) to (a5) are diagrams for explaining the processing in the image processing unit.

First, the area 1211 is selected, the first index information detection process is performed (FIG. 12 (a1)), the first index information is detected from the area 1211 (FIG. 12 (a2)), and the image quality parameter Q2 (low) Image quality) (FIG. 12 (a3)).
Next, the area 1212 is selected, the first index information detection process is performed (FIG. 12 (a1)), the first index information is not detected from the area 1212 (FIG. 12 (a2)), and the image quality parameter is set. The image is compressed with Q1 (high image quality) (FIG. 12 (a3)).
Similarly, the first index information detection process is performed for the areas 1213 and 1214 (FIG. 12 (a1)), and the first index information is not detected from the area 1212 (FIG. 12 (a2)). It is compressed with the parameter Q1 (high image quality) (FIG. 12 (a3)).

  In the image processing unit, a second index information detection process is performed on areas 1212 to 1214 where the first index information has not been detected (areas compressed with high image quality) (FIG. 12 (a4)). The second index information is detected from the region 1214 (FIG. 12 (a5)).

  As described above, according to the present embodiment, a part of the index information is detected by the HMD unit, the area where the index information is detected is compressed with low image quality, and the area where the index information is not detected is compressed with high image quality. And sent to the image processing unit. By compressing the captured image in this way, it is possible to suppress a decrease in index information detection processing accuracy due to deterioration of the compressed image while reducing the network bandwidth.

[Second Embodiment]
In the first embodiment, image quality parameters are controlled according to whether or not index information is detected by the first index information detection unit 102. That is, the image quality parameter is not controlled for the area where the index information is not detected by the first index information detection unit 102. However, the area where the index information is not detected by the first index information detection unit 102 can be divided into the following two types. That is, an area where the index information is not detected because the index information does not exist, and an area where the index information is not detected although the index information exists.

In the present embodiment, a method for controlling the image quality parameter even for an area where index information is not detected by the first index information detection unit 102 will be described.
For example, an area in which index information is not detected by the first index information detection unit, and an area estimated to include index information as a result of estimation based on index information detected from an area other than the area Is the first region.
The first area is likely to be an area where the first index information detection unit cannot detect the index information even though the index information exists.

As a result of estimation based on the index information detected from the area other than the area, the area where the index information is not detected by the first index information detection unit, 2 area.
Since the index information is not present in the second area, there is a high possibility that the first index information detection unit cannot detect the index information.

The flow of image compression processing in the HMD unit of the second embodiment will be described with reference to FIG.
Since S601 is the same as S501 of FIG.
In S602, it is checked whether or not the first index information detected by the first index information detection unit 102 is included in the selected macroblock.
If included, the process proceeds to S604, and the image quality parameter is set to Q3 (low image quality). That is, the compression condition is set to low image quality.
If the index information is not included (the first index information could not be detected), the process proceeds to S603.

  In S603, it is determined whether or not index information is included in the selected macroblock. As a determination method, it is conceivable to calculate the positional relationship from index information detected in the past and index information detected in the current captured image. For example, when some index information is already detected from the current captured image in the HMD unit 100, the index information is compared with the index information detected in the past, and the index information is not detected this time. Predictable points can be predicted.

For example, the index information 1-1-1, 1-1-2, and 1-1-3 are included in the set of index information detected by the first index information detection unit 102 at time T1,
The position of the index information 1-1-3 is assumed to be between the “position of the index information 1-1-1” and the “position of the index information 1-1-2”.
Then, the index information 1-2-1 and 1-2-2 are included in the set of index information detected by the first index information detection unit 102 at time T2,
The “position of the index information 1-1-1” and the “position of the index information 1-2-1” are substantially the same, and the “position of the index information 1-1-2” and the “index information 1-2— It is assumed that “position 2” is substantially the same.
However, the set of index information detected at time T2 includes index information positioned between “the position of the index information 1-2-1” and “the position of the index information 1-2-2”. Suppose not.
In this case, the set of index information detected at time T2 includes index information positioned between “the position of the index information 1-2-1” and “the position of the index information 1-2-2”. However, it can be determined (estimated) that there is a high possibility that the index information exists in the middle.

  In this way, it is possible to estimate the position of the index information and set the compression condition of the corresponding region according to the estimation result. For past index information, previous detection results and calibration data used during initialization / setup of the HMD unit may be stored in the HMD unit 100 or may be transmitted from the image processing unit 110.

When it is determined (estimated) that the index information is unlikely to be included in the selected macroblock, the process proceeds to S605 and the image quality parameter is set to Q2 (medium image quality). That is, the compression condition is set between the low image quality and the high image quality, or equivalent to the low image quality.
If it is determined that there is a high possibility that the index information is included in the macroblock selected by the prediction, the process proceeds to S606, and the image quality parameter is set to Q1 (high image quality). That is, the compression condition is set to high image quality.

In the present embodiment, a quantized scaling value is assumed as the image quality parameter, but the present invention is not limited to this. Here, the relationship between the image quality parameters when the quantized scaling value is assumed is assumed to be Q1 ≦ Q2 ≦ Q3.
In step S607, the selected macroblock is compressed using the image quality parameters set in steps S604, S605, and S606. Thereafter, S601 to S607 are repeated until it is determined in S608 that the compression processing for all the macroblocks has been completed.

The image quality control for each area (macroblock) will be described with reference to FIG.
It is assumed that index information detection processing is performed by the first index information detection unit using the pre-compression captured image 400 captured at time T1, and the markers 401, 402, 403, and 404 are detected.
On the other hand, using the pre-compression captured image (not shown) captured at time T2 (T1 + ΔT = T2) under the condition that the position and orientation of the image capturing unit are substantially the same as the captured image 400 before compression. Index image detection processing was performed by one index information detection unit. It is assumed that the markers 402, 403, and 404 are detected by the index image detection process, and the marker 401 is not detected.
A set of index information detected from an image captured at time T1 (past) (set of index information at time T1) and a set of index information detected from an image captured at time T2 (current) (time T2) With a set of index information). Then, from the difference between the set at time T1 and the set at time T2, an area where the marker 401 “is supposed to exist” (around the marker 401 in FIG. 4) is estimated and set to the image quality parameter Q1 (high image quality). The area where the marker 401 “is supposed to exist” is compressed.
Similarly, the region where the marker is “not considered to exist”, that is, the region other than the periphery of the markers 401, 402, 403, 404 is set to the image quality parameter Q2 (medium image quality) and compressed.
The area around the markers 402, 403, and 404 is compressed by setting the image quality parameter Q3 (low image quality).

  By performing such image quality control, the marker 401 (periphery) has an image quality that can be detected even from a compressed image, and the image quality is excessively increased for regions other than the periphery of the markers 401 to 404. An increase in data rate can be suppressed.

FIG. 13 shows an example of state transition of each region in the second embodiment. In order to simplify the explanation, it is assumed that the captured image 1310 is divided into four regions 1311 to 1314 and processed as shown in FIG.
(B1) to (b4) are diagrams for explaining processing in the HMD unit, and (b5) to (b6) are diagrams for explaining processing in the image processing unit.

First, the area 1311 is selected, the first index information detection process is performed (FIG. 13 (b1)), the first index information is detected from the area 1311 (FIG. 13 (b2)), and then the area 1312 and It is estimated that the index information is included in 1314 (FIG. 13 (b3)).
The area 1311 where the first index information is detected is compressed with low image quality, and the areas 1312 and 1314 estimated to include the index information are compressed with high image quality. Then, the area 1313 in which the first index information is not detected and the index information is not estimated to be included is compressed with medium image quality (FIG. 13 (b4)).

  In the image processing unit, the second index information detection process is performed on the areas 1312 to 1314 in which the first index information is not detected (FIG. 13 (b5)). Is detected (FIG. 13 (b6)).

  As described above, according to the present embodiment, image compression is performed according to whether or not index information is estimated to be included in an area in which index information is not detected by the first index information detection unit 102. Change the image quality parameter. In this way, in the video see-through mixed reality system, by compressing the captured image, the bandwidth of the network is reduced, and the accuracy of the index information detection processing due to the deterioration of the compressed image is further reduced. It becomes possible to deter.

[Third Embodiment]
In the first and second embodiments, marker detection is used as the first and second index information detection units. In the third embodiment, a method using natural features as index information will be described. Since the functional block configuration and the flowchart of the compression method are the same as those in the first and second embodiments, the description thereof is omitted.
In the real space, one or a plurality of indices P for imaging by the imaging unit 101 in the HMD unit 100 are arranged. Here, a coordinate system in which one point in the real space is defined as an origin and three axes orthogonal to each other are defined as an X axis, a Y axis, and a Z axis is referred to as a world coordinate system. An indicator Pk (k = 1,..., K) whose position in the world coordinate system is known is arranged. The index Pk is preferably installed so that information of at least three points can be observed by the imaging unit 101. For example, the indicator Pk may be configured by circular markers each having a different color, or may be configured by feature points such as natural features each having a different feature. It is also possible to use a square index formed by a square area having a certain area. Any form may be used as long as the image coordinates on the captured image can be detected and the index can be identified.
As described above, according to the present embodiment, it is possible to detect index information using natural features in a video see-through mixed reality system.

[Fourth Embodiment]
In the first and second embodiments, the same independent method is assumed as the first and second index information detection units, although there is a difference in processing capability. That is, the first index information detection unit 102 and the second index information detection unit 113 each independently detect the index information. Specifically, the second index information detection unit 113 detects the index information from the area in which the index information is included, but the first index information detection unit 102 cannot detect the index information. It was.
However, in the fourth embodiment, the first index information detection unit 802 only performs a process that is a pre-process of the second index information detection unit 813 for a region that is estimated to include the index information. .
For example, the first index information detection unit 802 detects only the outer frame of the marker as part of the index information. Then, the second index information detection unit 813 detects a pattern in the outer frame of the marker as the remainder of the index information.
In the fourth embodiment, a part of the index information detected by the first index information detection unit 802 (for example, the outer frame of the marker) is referred to as first index information, and the second index information detection unit 813 detects the index information. The remaining part of the index information (for example, the pattern in the outer frame of the marker) is referred to as second index information.

FIG. 8 is a functional block diagram of a video see-through mixed reality system according to the fourth embodiment, and includes an HMD unit 800 and an image processing unit 810.
The HMD unit 800 includes an imaging unit 801, a first index information detection unit 802, an image compression unit 803, a first network communication unit 804, and an image display unit 805. The functions of the imaging unit 801, the image compression unit 803, the first network communication unit 804, and the image display unit 805 are described in the imaging unit 101, the image compression unit 103, and the first network communication in FIG. Since it is the same as the part 104 and the image display part 105, description is abbreviate | omitted.

As described above, the first index information detection unit 802 only performs a process that is a pre-process of the second index information detection unit for a region estimated to include the index information. For example, the first index information detection unit 802 detects only part of the index information. The part of the index information is, for example, an outer portion (rectangular area) of the index information (marker). The first index information detection unit 802 detects only the outer shape portion (rectangular region) of the index information (marker) from the pre-compression captured image as preprocessing of index information detection processing in the second index information detection unit 813. .
The result of the preprocessing by the first index information detection unit 802 is sent to the second index information detection unit 813.
The second index information detection unit 813 can omit a part of the index information detection process by using the result of the preprocessing by the first index information detection unit 802.

The image compression unit 803 compresses the image of the area estimated by the first index information detection unit 802 to include the index information by setting the image quality parameter to high image quality. The image of the region estimated to include the index information is subjected to index information detection processing by the second index information detection unit 813.
On the other hand, the image compression unit 803 compresses an image of an area that has not been estimated that index information is included by the first index information detection unit 802 by setting the image quality parameter to medium image quality. The second index information detection unit 813 does not perform the index information detection process on the image of the area that is not estimated to include the index information.

The image processing unit 810 includes a second network communication unit 811, an image expansion unit 812, a second index information detection unit 813, a position measurement unit 814, and a CG composition unit 815. The functions of the second network communication unit 811, the image decompression unit 812, and the CG composition unit 815 are the same as those of the second network communication unit 111, the image decompression unit 112, and the CG composition unit 115 in FIG. 1. Therefore, the description is omitted.
However, the second index information detection unit 813 detects the second index information using the first index information in addition to the expanded captured image, and the position measurement unit 814 detects only the second index information. The position and orientation information is generated from the difference.

A compressed image is generated by the image compression unit 803 from the captured image captured by the imaging unit 801 in the HMD unit 800.
Similarly, from the captured image, the first index information detection unit 802 performs a first index information detection process. Here, the first index information detection process is a process in the previous stage in which the index information is actually determined, for example, detection of the outer portion (rectangular region) of the marker.
The compressed image and the first index information are transmitted to the image processing unit 810 by the first network communication unit 804.

In the image processing unit 810, the compressed image received by the second network communication unit 811 is expanded (decoded) by the image expansion unit 812, and an expanded captured image is generated.
The second index information detection unit 813 detects the second index information from the expanded captured image and the first index information. For example, a process for determining whether or not the rectangular area is a marker is performed on an area including the rectangular area detected as the first index information.
The position measurement unit 814 generates position and orientation information of the HMD user from the second index information.
The CG synthesizing unit 815 generates a synthesized image obtained by synthesizing CG viewed from the position and posture of the HMD user with the expanded captured image.
The second network communication unit 811 transmits the combined image generated by the CG combining unit 815 to the HMD unit 800. The composite image received by the first network communication unit 804 in the HMD unit 800 is displayed to the HMD user via the image display unit 805.

  The flow of image compression processing in the HMD unit of the fourth embodiment will be described with reference to FIG. The description of the same parts as those in FIG. 5 used in the first embodiment will be omitted, and different parts will be described. Specifically, S901 is S501, S903 is S503, S904 is S504, and S906 is S506 because the processing contents are the same.

  In S902, whether the area selected in S901 is “an area estimated to include index information”, that is, part of the index information by the first index information detection process (pre-processing of the second index information detection process). It is determined whether the area is detected. If it is an area estimated to include the index information, the process proceeds to S904, and if it is not an area estimated to include the index information, the process proceeds to S903.

  For example, an area where a figure having the same shape as the outer shape of the index information (marker) is detected is estimated as an area including the index information. Only when a graphic having the same shape as the outer shape of the index information is detected, it cannot be determined whether the outer shape is a part of the index information or a part of another graphic. The first index information detection unit 802 detects only the outer shape, and the second index information detection unit 813 determines whether the outer shape is a part of the index information or a part of another figure.

FIG. 14 shows an example of state transition of each region in the fourth embodiment. In order to simplify the explanation, it is assumed that the captured image 1410 is divided into four regions 1411 to 1414 and processed as shown in FIG.
(C1) to (c3) are diagrams for explaining processing in the HMD unit, and (c4) to (c5) are diagrams for explaining processing in the image processing unit.

First, an area 1411 is selected, and first index information detection processing (preprocessing for second index information detection) is performed (FIG. 14C1), and first index information is detected from the area 1411 (FIG. 14). 14 (c2)), and compression is performed with the image quality parameter Q1 (high image quality) (FIG. 14 (c3)).
Next, an area 1412 is selected, and first index information detection processing (preprocessing for second index information detection) is performed (FIG. 14 (c1)), but first index information is detected from the area 1412. It is not compressed (FIG. 14 (c2)) and is compressed with the image quality parameter Q2 (medium image quality) (FIG. 14 (c3)).

Next, an area 1413 is selected, and first index information detection processing (preprocessing for second index information detection) is performed (FIG. 14 (c1)), but first index information is also detected from the area 1413. It is not compressed (FIG. 14 (c2)) and is compressed with the image quality parameter Q2 (medium image quality) (FIG. 14 (c3)).
Next, the area 1414 is selected, and first index information detection processing (preprocessing for second index information detection) is performed (FIG. 14C1), and first index information is detected from the area 1414 ( FIG. 14 (c2)), and compression is performed with the image quality parameter Q1 (high image quality) (FIG. 14 (c3)).

  In the image processing unit, second index information detection processing (post-processing) is performed on the regions (regions compressed with high image quality) 1411 and 1414 where the first index information is detected (FIG. 14 (c4)). ), The second index information is detected from the areas 1411 and 1414 (FIG. 14 (c5)).

As described above, according to the present embodiment, the first index information detection unit performs the preprocessing of the detection process in the second index information detection unit, and the second index information detection unit performs the first index information detection. The index information detection process is performed using the result of the preprocessing of the detection unit. The compression condition is set to high image quality for the area where the first index information is detected, and the compression condition is set to medium image quality for the area where the first index information is not estimated to be included.
By doing so, the HMD unit performs only simple index information detection processing considering the characteristics of the HMD unit that is required to be lightweight and compact, and the index information by compression while reducing the network bandwidth. It becomes possible to further suppress the deterioration of the detection processing accuracy.

[Fifth Embodiment]
In the present embodiment, the priority of the target area for the first index information detection according to the image characteristics of the captured image will be described.
FIG. 10 is a functional block diagram of a video see-through mixed reality system according to the fifth embodiment, and includes a HMD unit 1000 and an image processing unit 1010.
The HMD unit 1000 includes an image capturing unit 1001, a first index information detection unit 1002, an image compression unit 1003, a first network communication unit 1004, and an image display unit 1005. The first index information detection unit 1002 includes a spatial frequency acquisition unit.
The functions of the imaging unit 1001, the image compression unit 1003, the first network communication unit 1004, and the image display unit 1005 are the same as the imaging unit 101, the image compression unit 103, and the first network communication unit 104 in FIG. Since it is the same as that of the image display unit 105, description thereof is omitted.

The spatial frequency acquisition unit obtains the spatial frequency of the captured image before compression for each macroblock. The determined spatial frequency is compared with a threshold value.
When the spatial frequency obtained by the spatial frequency acquisition unit is equal to or greater than the threshold, the first index information detection unit 1002 performs index information detection processing with priority on the macroblock, and when the spatial frequency is less than the threshold, the macro A normal index information detection process is performed for the block.
“Perform index information detection processing preferentially” means, for example, that index information detection processing is performed over a longer time compared to normal (that is, normal) index information detection processing.

The flow of image compression processing in the HMD unit of the fifth embodiment will be described with reference to FIG.
In S1101, one of the units (macroblocks) obtained by dividing the captured image (usually, from the upper left corner to the lower right corner, from left to right, when reaching the right edge, returns to the left edge and moves down by one macroblock row. Raster scan order).

In S1102, the spatial frequency of each macroblock unit is acquired, the acquired spatial frequency is compared with a threshold value, and it is checked whether the acquired spatial frequency is the above or not. If the acquired spatial frequency is above, the process proceeds to S1103, and if the acquired spatial frequency is less, the process proceeds to S1104.
In S1103, priority index information detection processing is performed, and in S1104, normal (that is, non-priority) index information detection processing is performed.
S1105 is the same as S501, S1106 is the same as S503, S1107 is the same as S504, S1108 is the same as S505, and S1109 is the same as S506.

  FIG. 7 shows an example of the marker. The marker is often represented by a black and white fill pattern as indicated by the marker 703 in FIG. In such a region with high spatial frequency including edge information, sharpness is likely to be lost by image compression. Therefore, there is a concern that the detection accuracy may deteriorate when second index information is detected after compression.

FIG. 15 shows an example of state transition of each region in the fifth embodiment. In order to simplify the description, it is assumed that the captured image 1510 is divided into four areas 1511 to 1514 and processed as shown in FIG.
(D1) to (d4) are diagrams for explaining processing in the HMD unit, and (d5) to (d6) are diagrams for explaining processing in the image processing unit.

First, the area 1511 is selected, it is determined that the spatial frequency is equal to or higher than the threshold (FIG. 15 (d1)), and the first index information detection process preferentially performed on the area 1511 is performed (FIG. 15 (d2)). ), First index information is detected (FIG. 15 (d3)). The region 1511 where the first index information is detected is compressed with the image quality parameter Q2 (low image quality) (FIG. 15 (d4)).
Next, the region 1512 is selected, it is determined that the spatial frequency is less than the threshold (FIG. 15 (d1)), and the normal first index information detection processing is performed on the region 1512 (FIG. 15 (d2)). ), The first index information is not detected (FIG. 15 (d3)). The area 1512 in which the first index information is not detected is compressed with the image quality parameter Q1 (high image quality) (FIG. 15 (d4)).

Next, the region 1513 is selected, it is determined that the spatial frequency is less than the threshold (FIG. 15 (d1)), and the normal first index information detection process is performed on the region 1513 (FIG. 15 (d2)). ), The first index information is not detected (FIG. 15 (d3)). The area 1513 in which the first index information is not detected is compressed with the image quality parameter Q1 (high image quality) (FIG. 15 (d4)).
Next, the area 1514 is selected, and it is determined that the spatial frequency is equal to or higher than the threshold (FIG. 15 (d1)), and the first index information detection process preferentially performed on the area 1514 (FIG. 15). (D2)), the first index information was not detected (FIG. 15 (d3)). The area 1514 in which the first index information is not detected is compressed with the image quality parameter Q1 (high image quality) (FIG. 15 (d4)).

  In the image processing unit, the second index information detection process is performed on the areas 1512 to 1314 in which the first index information is not detected (FIG. 15 (d5)). Is detected (FIG. 15 (d6)).

In the fifth embodiment, based on whether or not the spatial frequency is equal to or higher than a threshold value, it is determined whether or not the region where the sharpness is lost by the compression. The index information detection process is preferentially performed by the first index information detection unit of the HMD unit.
As described above, index information detection processing is preferentially performed on an uncompressed image in an area where degradation of the image due to compression is expected (an area where it is difficult to detect index information from the compressed image). By doing in this way, it becomes possible to suppress the fall of index information detection processing accuracy, reducing the use zone of a network.

As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.
For example, the present invention can be applied to a mixed reality system using HUD (Head Up Display) or HHD (Hand Held Display) instead of HMD.
The present invention is also applicable to a mixed reality system using an optical see-through HMD. In that case, the image processing unit 110 transmits only the CG portion of the composite image to the HMD unit 100. Alternatively, information for identifying the captured image (background) and CG in the composite image is transmitted to the HMD unit 100 together with the composite image. The display unit 105 is visible through the real space corresponding to the captured image, and displays the received composite image (CG) superimposed on the real space.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

100 HMD unit 101 Imaging unit 102 First index information detection unit 103 Image compression unit 104 Network communication unit 105 Image display unit 110 Image processing unit 111 Network communication unit 112 Image expansion unit 113 Second index information detection unit 114 Position measurement unit 115 CG synthesis unit

Claims (16)

A system comprising a display device and an image processing device,
The display device
Imaging means for imaging a real space and generating a captured image;
First index information detection means for performing detection processing of first index information for each of a plurality of regions obtained by dividing the captured image;
An image compression unit that generates a compressed image by compressing the captured image for each region based on a detection result of the first index information detection unit;
The first index information and the compressed image are transmitted for the area where the first index information is detected, and the compressed image is transmitted for the area where the first index information is not detected. Network communication means;
Display means;
With
The image processing apparatus includes:
Second network communication means for receiving the compressed image from the first network communication means;
Image expansion means for expanding the compressed image to generate an expanded image;
Second index information detecting means different from the first index information detecting means for detecting second index information from an area where the first index information of the decompressed image is not detected;
Position measuring means for generating position and orientation information of the display device from the first index information and the second index information;
CG combining means for generating a combined image by combining CG generated based on the position and orientation information with the expanded image;
With
The second network communication means transmits the composite image to the first network communication means;
The first network communication means receives the composite image from the second network communication means;
The display means displays the composite image. The first network communication means includes:
For the region in which the first index information is detected, the first index information and the compressed image are transmitted,
2. The system according to claim 1, wherein only the compressed image is transmitted without transmitting the first index information for an area in which the first index information is not detected. The image compression means includes
The compression ratio of the area where the index information is detected by the first index information detecting means is
3. The system according to claim 1, wherein the compression condition is set to be higher than a compression rate of a region where the index information is not detected by the first index information detection unit. The image compression means includes
When index information is included as a result of estimation based on index information detected from a region other than the first region, which is a first region where index information is not detected by the first index information detection unit The estimated compression ratio of the first region is
When index information is included as a result of estimation based on index information detected from an area other than the second area, which is a second area where index information is not detected by the first index information detecting means 4. The system according to claim 3, wherein the compression condition is set to be lower than a compression rate of the second area that is not estimated. The first index information detecting means includes
Spatial frequency acquisition means,
The system according to claim 3, wherein the index information detection process is preferentially performed on a region where the spatial frequency acquired by the spatial frequency acquisition unit is equal to or greater than a threshold value. The first index information detecting means includes
Performing pre-processing for detecting index information for each of a plurality of regions obtained by dividing the captured image, detecting a part of the index information as the first index information,
The image compression means includes
The compression ratio of the area where the first index information is detected is
Setting a compression condition to be lower than the compression ratio of the area where the first index information is not detected,
The second index information detecting means includes
The first index information is detected by performing post-processing after preprocessing performed by the first index information detecting unit on the area where the first index information detecting unit detects the first index information. The remainder of the index information not detected by the detecting means is detected as the second index information, and one index information is generated by the first index information and the second index information. Item 4. The system according to Item 1.   The system according to any one of claims 1 to 6, wherein the first and second index information are markers arranged in a real space.   The system according to any one of claims 1 to 6, wherein the first and second index information are natural features arranged in a real space.   The process performed by the first index information detecting unit when searching for index information is a simple process in which a part of the process performed by the second index information detecting unit when searching for index information is omitted or restricted. The system according to any one of claims 1 to 8.   The system according to claim 1, wherein the display device is mounted on a user's head or face.   The program for functioning a computer as each means of the system of any one of Claims 1 thru | or 8. A second network communication means for receiving a compressed image; an image decompressing means for decompressing the compressed image to generate a decompressed image; and a first index information detecting means detected by the first index information detecting means among the decompressed images. Second index information detection means different from the first index information detection means for detecting second index information from an area where index information has not been detected, the first index information and the second index An image processing apparatus comprising: position measuring means for generating position and orientation information of a display device from information; and CG combining means for generating a composite image by combining CG generated based on the position and orientation information with the expanded image. A connected display device,
Imaging means for imaging a real space and generating a captured image;
The first index information detecting means for detecting the first index information for each of a plurality of areas obtained by dividing the captured image;
An image compression unit that generates a compressed image by compressing the captured image for each region based on a detection result of the first index information detection unit;
The first index information and the compressed image for the area where the first index information is detected, the compressed image for the area where the first index information is not detected, and the second network communication First network communication means for transmitting to the means;
Display means for acquiring and displaying the composite image;
A display device comprising: Imaging means for capturing a real space and generating a captured image; first index information detecting means for performing first index information detection processing for each of a plurality of regions obtained by dividing the captured image; The image compression means for generating a compressed image by compressing the captured image for each region based on the detection result of the first index information detection means, and the compressed image as a second network communication means An image processing apparatus connected to a display device comprising first network communication means for transmitting,
The first index information and the compressed image for the area where the first index information is detected, the compressed image for the area where the first index information is not detected, and the first network communication. Said second network communication means for receiving from means;
Image expansion means for expanding the compressed image to generate an expanded image;
Second index information detecting means different from the first index information detecting means for detecting second index information from an area where the first index information of the decompressed image is not detected;
Position measuring means for generating position and orientation information of the display device from the first index information and the second index information;
CG combining means for generating a combined image by combining CG generated based on the position and orientation information with the expanded image;
An image processing apparatus comprising: A control method for a system including a display device and an image processing device,
In the display device,
An imaging step in which an imaging unit images a real space to generate a captured image;
A first index information detection step in which a first index information detection means performs a first index information detection process on each of a plurality of regions obtained by dividing the captured image;
An image compression step in which an image compression unit generates a compressed image by compressing the captured image for each region based on a detection result of the first index information detection unit;
The first network communication means transmits the first index information and the compressed image for the area in which the first index information is detected, and the area for which the first index information is not detected. A first network communication step of transmitting the compressed image;
Have
In the image processing apparatus,
A second network communication step in which a second network communication means receives the compressed image from the first network communication means;
An image expansion means for generating an expanded image by expanding the compressed image;
A second index different from the first index information detecting step, wherein the second index information detecting means detects second index information from a region where the first index information of the expanded image is not detected. An information detection process;
A position measuring unit that generates position and orientation information of the display device from the first index information and the second index information;
A CG combining step for generating a combined image by combining a CG generated based on the position and orientation information with the expanded image;
Have
The second network communication means transmits the composite image to the first network communication means;
The first network communication means receives the composite image from the second network communication means;
In the display device, a display means displays the composite image. A second network communication means for receiving a compressed image; an image decompressing means for decompressing the compressed image to generate a decompressed image; and a first index information detecting means detected by the first index information detecting means among the decompressed images. Second index information detection means different from the first index information detection means for detecting second index information from an area where index information has not been detected, the first index information and the second index An image processing apparatus comprising: position measuring means for generating position and orientation information of a display device from information; and CG combining means for generating a composite image by combining CG generated based on the position and orientation information with the expanded image. A method for controlling a connected display device, comprising:
An imaging step in which an imaging unit images a real space to generate a captured image;
A first index information detection step in which the first index information detection means performs a first index information detection process on each of a plurality of regions obtained by dividing the captured image;
An image compression step in which an image compression unit generates a compressed image by compressing the captured image for each region based on a detection result of the first index information detection unit;
The first network communication means is
The first index information and the compressed image for the area where the first index information is detected, the compressed image for the area where the first index information is not detected, and the second network communication Transmitting to the means;
Receiving the composite image from the second network communication means;
A display step in which the display means displays the received composite image;
A control method for a display device, comprising: Imaging means for capturing a real space and generating a captured image; first index information detecting means for performing first index information detection processing for each of a plurality of regions obtained by dividing the captured image; The image compression means for generating a compressed image by compressing the captured image for each region based on the detection result of the first index information detection means, and the compressed image as a second network communication means A control method for an image processing apparatus connected to a display device comprising first network communication means for transmitting,
The second network communication means comprises:
The first index information and the compressed image for the area where the first index information is detected, the compressed image for the area where the first index information is not detected, and the first network communication. A second network communication step for receiving from the means;
An image expansion means for generating an expanded image by expanding the compressed image;
This is different from the detection step performed by the first index information detecting unit, wherein the second index information detecting unit detects the second index information from a region where the first index information of the decompressed image is not detected. A second index information detection step;
A position measuring unit that generates position and orientation information of the display device from the first index information and the second index information;
A CG combining step for generating a combined image by combining a CG generated based on the position and orientation information with the expanded image;
The second network communication means transmitting the composite image to the first network communication means;
A control method for an image processing apparatus, comprising: