JP2003111101A - Method, apparatus and system for processing stereoscopic image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem wherein an infrastructure for accelerating distribution of stereoscopic image is not arranged. SOLUTION: An image processor 100 comprises an image acquisition section 102, and an information adding section 104. The image acquisition section 102 acquires a basic image capable of stereoscopic display externally and further acquires an original image being converted into a basic image. The information adding section 104 records information being referred during stereoscopic viewing, e.g. the arrangement and the estimated number of viewpoints, on the header of image data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic image processing technique, and more particularly to a method for processing or displaying a stereoscopic image,
The invention relates to devices, systems and related computer programs and data structures.

[0002]

2. Description of the Related Art In recent years, the population of users of the Internet has rapidly increased, and the broadband era, which can be said to be a new stage of Internet usage, is about to enter. In broadband communication, the communication band is remarkably widened, so heavy image data, which has been often shunned in the past, is becoming popular. Concepts such as "multimedia" and "video on demand" have long been proposed, but in the broadband era,
For the first time, it became the situation that ordinary users could actually experience these words.

When the distribution of images, especially moving images, becomes widespread, users naturally demand the enhancement of contents and the improvement of image quality. These are largely due to the digitization of existing video software, the development of authoring tools therefor, and the pursuit of highly efficient and lossless image coding technology.

[0004]

Under these circumstances, as one form of image distribution service in the near future, the distribution of pseudo three-dimensional images (hereinafter also simply referred to as "stereoscopic images") has received technical attention and is considerable. It is possible to win the market. Stereoscopic images fulfill users' desires for more realistic images, and are particularly attractive for applications such as movies and games that seek a sense of presence. Further, the three-dimensional image is also useful for realistic display of products in product presentation in EC (electronic commerce) which is considered to be one of the standards of commerce in the 21st century.

However, when considering a new net business of distribution of stereoscopic images, it can be said that neither an infrastructure therefor nor a model for business promotion exists yet. The present inventor has made the present invention paying attention to such a current situation, and an object thereof is to provide a stereoscopic image processing technique for facilitating distribution of a stereoscopic image from a technical aspect.

[0006]

To understand the present invention,
First, the following concepts in this specification will be defined. “Stereoscopic image”: It is not the image data itself but the image displayed as a stereoscopic image, which conceptually refers to the image that the user sees. The image data that can be displayed as a stereoscopic image is
It is called "multiplex image" described later. That is, when a multiplex image is displayed, a stereoscopic image can be seen. “Parallax image”: Generally, for stereoscopic viewing with a sense of depth, an image that should be cast on the right eye (hereinafter simply referred to as the right eye image) and an image that should be cast on the left eye (hereinafter simply referred to as the left eye image) so that parallax occurs. You need to prepare. A pair of images that cause parallax, such as a right-eye image and a left-eye image, may be collectively referred to as a parallax image, but in the present specification, each image that causes parallax is referred to as a parallax image. That is, both the right-eye image and the left-eye image are parallax images. In addition to these, generally, images from respective viewpoints assumed in a stereoscopic image are parallax images.

"Basic image": An image which is a target for which processing necessary for stereoscopic viewing is performed in order to display a stereoscopic image, or an image which has already been processed. As a specific example, in addition to separate parallax images to be described later, images such as multiplex format or side-by-side format that have already been composed in some form by combining multiple parallax images (also referred to as “composite images”) including. “Side-by-side format”: One of the basic image configuration modes. A format in which multiple parallax images are placed side by side in the horizontal or vertical direction or in both directions. Usually, thinned parallax images are juxtaposed. For example, when two parallax images are arranged side by side in the horizontal direction, each parallax image is thinned out pixel by pixel in the horizontal direction. The side-by-side format basic image is also simply called "side-by-side image".

"Multiplex format": One of the basic image construction modes. The final image data format for displaying stereoscopic images. The basic image in the multiplex format is also simply called “multiplex image”. “Separate format”: One of the basic image configuration modes. Although it is a single two-dimensional image, it is supposed to be combined with other two-dimensional images for stereoscopic viewing, and refers to each of the plurality of two-dimensional images. The basic image in “separate format” is also simply referred to as “separate image”. Separate images are not composite images, unlike multiplex images and side-by-side images.

"Viewpoint": A viewpoint for viewing a stereoscopic image is assumed. The number of viewpoints and the number of parallax images are usually equal. When there are two parallax images, the left-eye image and the right-eye image, the number of viewpoints is “2”. However, even if there are two viewpoints, the estimated position of the user's head is one. Similarly, when displaying a stereoscopic image in consideration of the movement of the user in the left and right direction, for example, four viewpoints va, vb, vc, and v in the left and right direction.
Assuming d, the parallax images seen from each are Ia, I
If b, Ic, and Id, for example, (Ia, Ib) (I
A stereoscopic image with a sense of depth can be displayed by three sets of parallax images of (b, Ic) (Ic, Id). In this state, to generate a stereoscopic image that wraps around in the vertical direction,
If four images viewed from a relatively upper direction and four images viewed from a lower direction are used, the number of viewpoints is “8”.

Based on the above definition, one embodiment of the present invention is
The present invention relates to a stereoscopic image processing method. This method can be understood as a technique on the encoding side that should be called the starting point of distribution of stereoscopic images. This method adds information (hereinafter also referred to as “stereoscopic information”) to be referred to in a predetermined scene in a series of processing for displaying the stereoscopic image to a basic image for displaying the stereoscopic image. is there.

"Add" may be incorporated in the basic image, may be incorporated in the header of the basic image or other areas, or may be incorporated in a separate file or the like associated with the basic image. It suffices to establish a correspondence relationship with. An example of the “predetermined scene in a series of processes for displaying a stereoscopic image” is, for example, a scene in which a side-by-side image is converted into a multiplex image.

According to this method, the stereoscopic image can be displayed by an appropriate method by referring to the stereoscopic information. If you prepare a large number of basic images in this way, various information terminals can take out the data and display it in 3D.
This method serves as the basic technology for the distribution of stereoscopic images.
This method can be used, for example, in a stereoscopic image server.

Another aspect of the present invention relates to the structure of the image data produced by the method described above. This data structure is the main data of the basic image for displaying a stereoscopic image,
It is formed as a combination with information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image, that is, sub-data holding stereoscopic information. The main data may be a basic image compressed by a predetermined method. The “combination” may be a combination of the two, or a combination of the two. According to this data structure, as described above, stereoscopic display is easily realized on the display side.

Yet another aspect of the present invention also relates to a stereoscopic image processing method. This method can be understood as a method that interprets and uses the above-mentioned data structure, that is, a technique on the decoding side that generally displays a stereoscopic image. This method detects information that is added to a basic image for displaying a stereoscopic image and should be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image, that is, stereoscopic information. To facilitate the detection, the stereoscopic information may be added according to a predetermined format previously agreed with the encoding side. The method may further convert the aspect of the configuration of the basic image into another one based on the detected stereoscopic information.

Yet another aspect of the present invention also relates to a stereoscopic image processing method. In this method, when a stereoscopic image is handled by a device having a memory, a basic image of a mode different from that of the structure of the basic image finally displayed on the screen is held in the memory, and the basic image is read out as appropriate. To use.
For example, even if a multiplex image is used to display a stereoscopic image, the side-by-side format may be more convenient than the multiplex format when the stereoscopic image is to be enlarged or otherwise processed. In that case, if the side-by-side image is stored in the memory, the processing speed can be increased.

Yet another aspect of the present invention also relates to a stereoscopic image processing method. This method detects information that is added to a basic image for displaying a stereoscopic image and should be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image, that is, stereoscopic information, and Originally, the brightness of the display screen of the stereoscopic image is adjusted. For example, as three-dimensional information,
Information about the number of assumed viewpoints may be put in the stereoscopic image, and the brightness may be adjusted according to the number.

If the number of viewpoints is "4", four parallax images are combined to form a multiplex image. The number of pixels that can be seen from any one of the four viewpoints is only 1/4 that of a normal two-dimensional image. Therefore, the screen brightness is theoretically 1/4 of the normal level. Therefore, the process of increasing the brightness of the screen of the display device according to the number of viewpoints is effective. This processing is performed by the cooperation of software for detecting the number of viewpoints and a circuit for adjusting the brightness, for example.

Yet another aspect of the present invention relates to a stereoscopic image processing apparatus. This device is on the encoding side that prepares a stereoscopic image, and includes an image acquisition unit that acquires a basic image for displaying the stereoscopic image, and a series for displaying the stereoscopic image on the acquired basic image. The information adding section for adding information to be referred to in a predetermined scene in the processing of 1, ie, stereoscopic information. The image acquisition unit may generate the basic image by itself, or may input the ready-made basic image.

Yet another aspect of the present invention also relates to a stereoscopic image processing apparatus. This device is for actually displaying a stereoscopic image, or for performing preprocessing therefor, and an image acquisition unit for acquiring a basic image for displaying a stereoscopic image, and added to the acquired basic image, It includes information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image, that is, an information detection unit for detecting stereoscopic information.
The image acquisition unit acquires or inputs a basic image via, for example, a network or a recording medium. The image acquisition unit may include an image input unit that inputs image data that has been compressed in advance, and an image expansion unit that generates a basic image by expanding the input image data. In addition, a brightness adjusting unit for adjusting the brightness of the display screen of the stereoscopic image may be included.

Yet another aspect of the present invention relates to a stereoscopic image processing system. This system includes a stereoscopic image synthesizing device and a display device, and the synthesizing device is a basic image for displaying a stereoscopic image, and information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image. That is, the stereoscopic information is incorporated, and the display device displays the stereoscopic image by detecting the stereoscopic information and appropriately performing image processing on the basic image based on the stereoscopic information. As an example of image processing, there is a change in the form of the basic image configuration. The system may be a server / client system. This system can contribute to the promotion of distribution of stereoscopic images.

Yet another aspect of the present invention relates to a stereoscopic image processing method. This method obtains a basic image for displaying a stereoscopic image, and inspects a part of the basic image, information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image, that is, It is for estimating stereoscopic information. Unlike the case described so far, here, the processing when the stereoscopic information is not explicitly added is considered. Therefore, a part of the basic image is actually inspected. As an example, by examining some regions on the base image, it is estimated whether this is a side-by-side image.

According to this method, even when the stereoscopic information is not explicitly given, it can be known from the basic image. Therefore, the past image created by the general method can be used, and the software assets can be effectively used.

It is to be noted that any combination of the above constituent elements, and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium, a transmission medium, etc. are also effective as an aspect of the present invention. is there.

[0024]

BEST MODE FOR CARRYING OUT THE INVENTION When displaying an image on an LCD, the minimum display unit is usually a dot. However, one pixel is formed by collecting three dots corresponding to RGB, and a pixel is recognized as the minimum unit of processing in normal image display or image processing.

However, when displaying a stereoscopic image on the LCD, another consideration is required. The right-eye image and the left-eye image reach the user's eye with parallax through an optical filter such as a lenticular lens or a parallax barrier.
When the left and right images are alternately arranged in pixel units, that is, in 3 dot units, a region in which both images are visible occurs between the region in which only the right eye image is visible and the region in which only the left eye image is visible, and colors are mixed, Very hard to see. Therefore, it is desirable that the dots are alternately arranged, which is the minimum physical display element. Therefore, a multiplex image in which a right-eye image and a left-eye image are alternately arranged in dot units is often used as a basic image to be stereoscopically displayed.

When the parallax image is composed of only two images, the right-eye image and the left-eye image, that is, when the number of horizontal viewpoints is "2", it suffices to arrange the right-eye image and the left-eye image in stripes for the multiplex image. . However, when the number of viewpoints is “4” and a stereoscopic image considering horizontal viewpoint movement is displayed using four parallax images, a parallax barrier placed in front of the screen 10 as shown in FIG. 12, only the dots of the corresponding parallax images can be seen from the first to fourth viewpoints VP1 to VP4. On screen 10, the first
The dot of the first parallax image corresponding to the viewpoint VP1 is indicated by adding “1”, and the same applies to the following viewpoints. In this example, the first to fourth parallax images are sequentially arranged in stripes in dot units, and a multiplex image is formed.

Further, when the viewpoint movement is considered in the vertical direction, the parallax barrier 12 does not form a stripe shape but a pinhole arranged in a matrix form, and the multiplex image also becomes a matrix form in which dots are interchanged. Figure 2
Shows an example of the multiplex image 20 when both the number of horizontal viewpoints and the number of vertical viewpoints are “4”. Here, the areas denoted by (i, j) indicate dots that should be seen from the i-th viewpoint in the horizontal direction and the j-th viewpoint in the vertical direction, respectively. In the horizontal direction, i is 1, 2, 3, 4, 1, ...
.. changes cyclically, and similarly in the vertical direction, j
Cyclically change to 1, 2, 3, 4, 1, ...

Considering promotion of use of stereoscopic images, the multiplex image 20 shown in FIG. 2 may be transmitted to a required terminal. Since the multiplex image 20 is already in the final format for stereoscopic viewing, the terminal side only has to display it. Of course, at this time, it is assumed that an optical filter such as a parallax barrier is present for stereoscopic viewing.

However, the present inventor has recognized that one problem arises here. That is, image data should of course be compressed at the time of transmission, but in the case of the multiplexed image 20, JPEG (Joint Photographic Exper
The usual lossy compression represented by t Group) is virtually unavailable. Because multiple parallax images that have been multiplexed are images with different viewpoints, they are essentially irrelevant at the pixel level, and when compressed by a method that relies on spatial frequencies such as JPEG, each Despite the use of parallax images independent from the viewpoint, high-frequency components are cut off between those images, and as a result, correct stereoscopic display cannot be performed. Especially, when the independent images are alternately arranged on a pixel-by-pixel basis, a large number of very fine high-frequency components are generated, and this problem is sometimes fatal. Even though the bandwidth of the network is expanding, when normal images can be compressed without problems,
If only basic images for stereoscopic images cannot be compressed, it will be a hindrance to widespread use.

Therefore, it is conceivable that the side-by-side image is highly used as a compressible format for transmission and storage. FIG. 3 shows a side-by-side image 30 having four viewpoints both horizontally and vertically. Where (i,
The area denoted by j) indicates one parallax image that should be seen from the i-th viewpoint in the horizontal direction and the j-th viewpoint in the vertical direction. That is, the side-by-side image 30 is a combination of parallax images arranged side by side in one direction or both in the horizontal direction or the vertical direction, and each parallax image functions as one image when cut from the side-by-side image 30.

However, since each parallax image needs to correspond to only one of 4 × 4 = 16 viewpoints, the parallax image may have a size of 1/16 of the image size to be displayed as a stereoscopic image, and is usually the same size as the stereoscopic image. From the original image of 1), 1 dot is selected every 4 dots in each of the horizontal direction and the vertical direction. As an easy-to-understand example, in the case of a multiplex image consisting of right-eye images and left-eye images with a viewpoint number of "2", only the odd-numbered rows of dots need to be visible from the right eye, and only the even-numbered rows of dots can be seen from the left eye. Good. Therefore, the right-eye image only needs to be obtained by extracting only odd-numbered columns from the original image in advance and decimating it to 1/2 in the horizontal direction, and similarly for the left-eye image, only even-numbered columns may be extracted. In general, if the number of viewpoints is “n”, each parallax image forming the side-by-side image may be 1 / n of the original image size, and if all parallax images are juxtaposed like tiles, they will return to the original image size.

In the case of the side-by-side image 30, since each parallax image is independent except for its boundary, even if the lossy compression is performed, the adverse effect occurs only at the boundary portion.
Therefore, normally, the side-by-side 30 is compressed by JPEG or the like so that it can be easily transmitted through the network or can be stored in a large number in a small storage.
As described above, the side-by-side image 30 is suitable for widespread use, but on the contrary, it has a defect that a special viewer is required. That is, in any of the display devices, stereoscopic display cannot be performed unless it is finally converted into a multiplex image, and conversion processing from the side-by-side image 30 into a multiplex image is required.

In addition to the above two formats having advantages and disadvantages, a separate image is considered as a third format from the viewpoint of popularization, particularly from the viewpoint of image preparation. The separate image is
Although a stereoscopic image can be formed as an aggregate, it is only a normal two-dimensional image by itself. FIG. 4 shows the relationship between 16 separate images and a stereoscopic image. Of the 16 images, for example, a separate image 32 described as “image of viewpoint (4, 2)”
Is based on the viewpoint (4, 2), and its image size is the same as the original image. Therefore, the 16 separate images can be easily understood by considering them as camera images taken by the user while moving.

As described above, since the size of the separate image may be the same as that at the time of photographing, it is easy to prepare without requiring the processing such as thinning and composition. Also, since each image remains in its original state, it can be used separately by itself. However, in the case of stereoscopic display, a total of 16 parallax images are required, which is disadvantageous in terms of transmission and storage.
After all, a special viewer is needed.

The above are the three major formats that can be considered for the spread of stereoscopic images. These modifications will be described last, and a stereoscopic image processing method for technically realizing widespread promotion and stereoscopic display when the basic image is represented in any of these three formats will be described below. Hereinafter, for simplicity, a case where the number of horizontal viewpoints is “2” and the number of vertical viewpoints is “1” will be exemplified.

5, 6, and 7 are side-by-side images 40 and 40 among the basic images according to the present embodiment, respectively.
The data structure of the multiplex image 50 and the separate image 60, 62 of a set of 2 sheets is shown typically.

As shown in FIG. 5, the side-by-side image 40 is a horizontally synthesized image of the first parallax image 44, which is the left-eye image, and the second parallax image 46, which is the right-eye image. 42 is added. Similarly, a header area 42 according to the same format is added to the multiplex image 50 of FIG.
A header area 42 is added to each of the two separate images 60 and 62 in FIG. 7. In either case,
This data structure is a combination of main data that is a basic image for displaying a stereoscopic image and sub data that holds information to be referred to in a predetermined scene in a series of processing for displaying the stereoscopic image. I can think. When the main data is obtained by compressing the basic image by a predetermined method, the sub data may be stored in the header area defined by the compression method. When the definition of the header area already exists, for example, a user-defined area can be used among the areas.

FIG. 8 schematically shows the detailed structure of the header area 42. In the figure, each area holds the following stereoscopic information. (1) DIM area 70: 3 bits indicate the dimension and structure of the basic image. 000: Separate image or general two-dimensional image that cannot be stereoscopically viewed 001: Multiplex image of three-dimensional image 010: Side-by-side image of three-dimensional image 011: Reserve 1xx: As an example of a format that is reserved, like a side-by-side image A "joint image" in which multiple parallax images are juxtaposed, but the parallax images are juxtaposed without changing the original images, or a parallax image in which even-numbered fields and odd-numbered fields should be displayed alternately in time division. A "field sequential image" indicating that The "joint image" is often observed by the parallel method or the cross method, but it is effective as one format because it can be thinned out by the viewer and converted into a side-by-side image, or can be directly converted into a multiplex image. .

(2) BDL area 72: 1 bit indicates presence / absence of boundary processing of the side-by-side image. It has a meaning when DIM is "01x". 0: No boundary processing 1: With boundary processing As described above, when the side-by-side image is lossy compressed, the image is adversely affected at the boundary portion. In order to reduce this, it indicates whether or not the processing shown in the next section is performed.

(3) HDL area 74: The content of the boundary processing of the side-by-side image is indicated by 2 bits. It has meaning when BDL is "1". 00: Insert white frame 01: Insert black frame 10: Copy edge pixels and insert 11: In order to reduce the adverse effect of reserve compression, insert white frame, black frame, etc. at the boundary to mix multiple parallax images. cut back. Copying the edge pixels has the same effect.

(4) WDT area 76: 2 bits specify the number of pixels for boundary processing of the side-by-side image. BD
It has a meaning when L is "1". 00-11: Number of pixels (5) VPH area 78: Indicates the number of horizontal viewpoints assumed for a stereoscopic image in 8 bits. It may be manually described when the basic image is created, or may be automatically created by software that creates the basic image. 000000: Unknown or reserve 00000001 to 11111111: Number of horizontal viewpoints (6) VPV area 80: Indicates the number of vertical viewpoints assumed for a stereoscopic image in 8 bits. 00000000: Unknown or reserve 00000001 to 11111111: Number of vertical viewpoints When VPH and VPV are both 00000001, the basic image may be determined as a normal two-dimensional image that cannot be stereoscopically viewed.

(7) ODH area 82: 1 bit indicates the arrangement of a plurality of parallax images in the horizontal direction. 0: Same as camera arrangement at the time of shooting 1: Opposite to camera arrangement at the time of shooting In other words, when ODH is "0", the parallax image taken by the leftmost camera at the time of shooting is the leftmost even in the basic image. It has been recorded in, and has been recorded in order thereafter. Normally, it is difficult to think that the arrangement is random, so two types of rules are sufficient.

(8) ODV area 84: 1 bit indicates the arrangement of a plurality of parallax images in the vertical direction. 0: Same as the arrangement of cameras at the time of shooting 1: Reverse of the arrangement of cameras at the time of shooting Note that both VPH and VPV related to the number of viewpoints described above are 8
These most significant bits may be assigned to ODH and ODV, respectively, as they are bits and are usually considered to be more than sufficient.

(9) PSH area 86: 8 bits indicate which viewpoint position each separate image is in the horizontal direction. It has meaning when DIM is “000”. 00000000: Unknown or reserve 00000001 to 11111111: Horizontal position Note that the origin determined on each separate image, for example, an absolute value such as the coordinates of the point at the upper left corner of the image may be separately included in the header area 42. In that case, it leads to speeding up of processing.

(10) PSV area 88: 8 bits indicating the position of the viewpoint of each separate image in the vertical direction. 00000000: Unknown or reserve 00000001 to 11111111: Vertical position, for example, a separate image described as “image of viewpoint (4, 2)” in FIG. 4 has a description of PSH = 4 and PSV = 2.

The above is an example of the header area 42. An apparatus for realizing the distribution of a stereoscopic image using this area will be described. FIG. 9 shows an image processing apparatus 10 for generating this area.
The structure of 0 is shown. This device 100 includes an image acquisition unit 102 that acquires a basic image for displaying a stereoscopic image, and a predetermined scene in a series of processes for displaying a stereoscopic image on the acquired basic image, for example, on the display side later. An information adding unit 104 for adding stereoscopic information to be referred to when a multiplex image is generated by the apparatus. In terms of hardware, this configuration is the CP of any computer.
U, memory, and other LSIs can be realized, and in terms of software, it is realized by a program having a memory-loaded basic image generation function and a stereoscopic information addition function. I'm drawing a block. Therefore, these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.
It will be understood by those skilled in the art. Therefore, in the following, if the name of the configuration is not explicitly shown, for example, CP
It may be considered to be performed by a control unit centered on U.

The image acquisition unit 102 inputs an original image from an image source such as a network or a user's digital camera, and uses the original image as it is, or processes it to generate a basic image. For example, when a separate image is required, the original image may be used as the basic image as it is. On the other hand, when side-by-side images are required, a plurality of original images are juxtaposed and combined. When a multiplex image is required, the parallax image from each viewpoint is reconstructed in a stripe shape or a matrix shape.

The image acquisition unit 102 further compresses the obtained basic image as needed. Prior to that, it is determined whether or not the image quality of the stereoscopic image is affected by the compression, and if it is determined that the image quality is output, the compression may be prohibited. For example, when compressing a multiplexed image in terms of spatial frequency components,
The compression may be prohibited, or the image may be once converted into a side-by-side image and then compressed.

The information adding unit 104 adds the above-mentioned header information to the basic image thus obtained, and records the resulting image data for stereoscopic display in a storage device (not shown) or via a network. Deliver to a specified location with.
As described above, according to this device 100, a basic image with stereoscopic information can be prepared in advance for a person who desires stereoscopic display.

The image acquisition unit 102 does not always obtain the original image first. By inputting what is already a multiplex image from a network, etc., detecting the stereoscopic information, determining that it is a multiplex image, recognizing that it is not suitable for compression in that state, It is also possible to convert the image into a side-by-side image, compress it, and rewrite stereoscopic information. In this case, the device 100 can also be used as a relay point for the distribution of stereoscopic images.

On the other hand, FIG. 10 shows the configuration of the image processing apparatus 200 on the decoding side that actually performs stereoscopic display. This device 20
0 is an image acquisition unit 202 that acquires a basic image for displaying a stereoscopic image, and is added to the acquired basic image.
Information detecting unit 204 for detecting stereoscopic information to be referred to in a predetermined scene in a series of processing for displaying a stereoscopic image
Including and This device 200 is typically a terminal on the user side, and the image acquisition unit 202 acquires a basic image to which stereoscopic information has already been added. When the image data that has been compressed in advance is input, the image acquisition unit 202 may decompress the image data to generate or reproduce the basic image.

Subsequently, the information detecting unit 204 parses the header area added to the basic image and detects the stereoscopic information. If it is determined from the detected stereoscopic information that the basic image is not a multiplex image, the device 200 converts the basic image into a multiplex image and displays the stereoscopic image. As an optional function of the device 200, according to the above-described consideration regarding the brightness based on the number of horizontal viewpoints and the number of vertical viewpoints of the detected stereoscopic information, the display screen (not shown) of the device 200 is displayed. The brightness may be increased.

This device 200 can not only display a stereoscopic image, but can of course store and edit a basic image. When saving, if the basic image is a multiplex image, it may be converted into a side-by-side image or the like, and the stereoscopic information may be rewritten and then saved. At the time of editing, for example, it is sometimes desired to enlarge or reduce the image. At that time, since the processing is complicated for a multiplex image, the processing may be once converted into a side-by-side image, then subjected to predetermined image processing, and finally returned to the multiplex image for display.

In order to facilitate such editing and other image processing, the apparatus 200 always stores images in formats other than multiplex images, especially side-by-side images, in a memory or other storage device, and if necessary, stores them. It may be read out and used as appropriate.

As an additional configuration of the device 200, if a characteristic acquisition unit for acquiring characteristics such as the number of viewpoints and the optimum observation distance of the display device as data is provided, the convenience is further enhanced. For example, when the assumed number of viewpoints of the basic image is different from that of the display device, it is possible to provide a display image selection unit that automatically selects a parallax image to be displayed from the basic image based on the above characteristics. If the assumed number of viewpoints of the basic image is “4” and that of the display device is “2”, two of the four parallax images are selected. These two parallax images do not have to be from continuous viewpoints, and two images from different viewpoints may be selected to enhance the stereoscopic effect. If the number of viewpoints of the basic image is “2” and that of the display device is “4”,
By displaying the same parallax image twice, it is possible to secure a stereoscopic viewable area in front of the screen.

Further, if the image processing apparatus 200 is provided with a position detecting section for detecting the head position of the observer looking at the display screen, the display image selecting section will select the parallax image to be selected according to the head position. Can also be changed to show the circling image to the observer.

When the optical filter is replaceable, for example, the display section of the apparatus 200 may be provided with a reading section for optically reading a display including information such as the number of viewpoints pattern-printed on the optical filter. When the read data suggests a disagreement with the number of viewpoints of the parallax image, the parallax image may be appropriately optimally selected and displayed as described above, or a display prompting replacement of the optical filter may be performed.

FIG. 11 shows the configuration of a network system 300 for distributing a stereoscopic image. Here, the synthesizer 30
Reference numeral 2 denotes the image processing apparatus 100 in FIG. 9, which acts as a starting point of distribution. On the other hand, the display device 304 is the image processing device 200 of FIG. 10 and acts as the end point of distribution. As shown in the figure, the synthesizing device 302 has a storage device 306 for recording a large number of basic images and acts as an image server, so that the user can easily obtain a desired stereoscopic image via the Internet or other network 308. .

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that these embodiments are mere examples, and that various modifications can be made to the combinations of the respective constituent elements and the respective processing processes, and such modifications are also within the scope of the present invention. By the way. Here are some examples.

The image processing apparatuses 100 and 20 shown in FIG. 9 and FIG.
Each of the 0 functions can be provided to the user in the form of a computer program. When the user wants to generate the basic image by himself, the function of the image processing apparatus 100 in FIG. 9 may be prepared as an authoring tool and then provided to the user.

The structure of the header area 42 shown in FIG. 8 naturally has a high degree of freedom including the number of bits. For example, copyright information of a basic image observed by a user as a stereoscopic image, conditions to be satisfied by an optical filter such as a parallax barrier or a lenticular lens suitable for viewing the basic image can be further incorporated. As an example of “conditions that the optical filter should satisfy”, the distance between viewpoints of the parallax image,
That is, there are the distance between the eyes and the angle of view of the camera at the time of shooting.
Such a condition is an essential parameter for designing an optical filter when reproducing a stereoscopic image on the correct scale in the front-back direction. Further, even in the case of selecting an image when the number of viewpoints of the basic image is larger than the number of viewpoints of the display device, it is possible to refer to automatically select an image that gives a more natural stereoscopic effect.

In the embodiment, the stereoscopic image finally displayed is the multiplex image, but the image to be displayed changes depending on the observation method. Therefore, various images may be converted to images suitable for the observation method and displayed. For example, when liquid crystal shutter glasses are used, the image to be displayed is a field sequential image. Also,
In a head-mounted display of a type having separate display means corresponding to the left and right eyes, the stereoscopic image to be displayed is a separate image, which is sent to each display means by separate image output means. Side-by-side images can also be displayed on a head-mounted display with a single display device. In this case, the image may be divided into the left half and the right half by optical means, and may be enlarged in the horizontal direction for observation. Further, the joint image may be displayed by an observation method such as a crossing method or a parallel method.

In the embodiment, the apparatus for displaying the stereoscopic image, that is, the image processing apparatus 200 in FIG. 10, starts the processing on the assumption that the stereoscopic information is added to the basic image.
However, if there is an existing basic image that is not according to the present embodiment, it does not have a header area that is characteristic of the present embodiment, so an inspection / estimation processing unit that inspects the basic image is provided, The stereoscopic information may be estimated on the side of the processing device 200. For example, whether or not the image is a side-by-side image is determined by dividing the image horizontally and vertically into m equal parts and n parts, respectively.
The image may be equally divided and the degree of approximation of the image in each region may be evaluated while changing the values of m and n. If the degree of approximation of each region or a part of the set of m and n is high, it can be estimated that this is a side-by-side image with m horizontal viewpoints and n vertical viewpoints. The evaluation of the degree of approximation is based on, for example, the sum of squared differences of pixel values. In addition to this, it is also possible to apply a differential filter to the basic image to make the boundary line of the region stand out,
It may be possible to estimate whether or not the image is a side-by-side image.

[0064]

According to the present invention, the distribution of stereoscopic images can be promoted.

[Brief description of drawings]

FIG. 1: 4 users are multiplexed horizontally
It is a figure which shows the state which carries out the stereoscopic vision of the parallax image of 1 sheet.

FIG. 2 is a diagram showing a multiplex image.

FIG. 3 is a diagram showing a side-by-side image.

FIG. 4 is a diagram showing a set of a plurality of separate images.

FIG. 5 is a diagram schematically showing a configuration of a side-by-side image according to the embodiment.

FIG. 6 is a diagram schematically showing a configuration of a multiplexed image according to the embodiment.

FIG. 7 is a diagram schematically showing a configuration of a separate image according to the embodiment.

FIG. 8 is a configuration diagram of a header area added to a basic image according to the embodiment.

FIG. 9 is a configuration diagram of an image processing apparatus serving as a starting point of image distribution according to the embodiment.

FIG. 10 is a configuration diagram of an image processing apparatus serving as an end point of image distribution according to the embodiment.

11 is a diagram showing a configuration of a network system for image distribution in which the image processing apparatus of FIG. 9 is a synthesizing apparatus and the image processing apparatus of FIG. 10 is a display apparatus.

[Explanation of symbols]

12 Parallax barrier, 20,40 multiplex image, 30,50 side-by-side image, 3
2, 60, 62 separate image, 42 header area, 100, 200 image processing device, 102, 20
2 image acquisition unit, 104 information addition unit, 204 information detection unit, 300 network system, 302
Synthesizing device, 304 Display device, 306 Storage device.

Continued front page    F-term (reference) 5B050 AA09 BA06 DA07 EA17 FA02                 5C061 AA03 AA06 AA07 AA08 AA13                       AB08 AB20 AB21 AB24

Claims (28)

[Claims] 1. A basic image for displaying a stereoscopic image,
A stereoscopic image processing method, comprising adding information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image. 2. The information to be referred to includes information about a dimension of the basic image.
The method described in. 3. The method according to claim 1, wherein the information to be referred to includes information regarding a configuration aspect of the basic image. 4. The information regarding the aspect is a side-by-side format in which the basic image is composed by juxtaposing a plurality of parallax images, or a multiplex format in which a plurality of parallax images are finally combined in a stereoscopically displayable form. The method according to claim 3, characterized in that it includes information indicating whether or not it is in a format other than those. 5. The method according to claim 1, wherein the information to be referred to includes information regarding the number of viewpoints assumed in the stereoscopic image. 6. The information to be referred to includes information about a sequence of the basic images.
The method described in. 7. The method according to claim 1, wherein the information to be referred to includes information regarding a position of the basic image in the stereoscopic image. 8. A combination of main data of a basic image for displaying a stereoscopic image and sub-data holding information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image. A data structure for a stereoscopic image characterized by: 9. The data according to claim 8, wherein the main data is obtained by compressing the basic image by a predetermined method, and the sub-data is stored in a header area for the compressed data. Construction. 10. A stereoscopic image characterized by detecting information added to a basic image for displaying a stereoscopic image, which should be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image. Processing method. 11. The method according to claim 10, further comprising converting the aspect of the structure of the basic image into another one based on the detected information to be referred. 12. From the detected information to be referred to,
12. The method according to claim 11, further comprising converting the basic image into a multiplex format when it is found that the basic image is not in a multiplex format in which the basic image can be stereoscopically displayed. 13. When compressing the basic image, it is determined whether or not compressing the basic image as it is affects the stereoscopic image based on the detected information to be referred to. The method according to claim 10, characterized in that 14. When it is determined that compressing the basic image as it is affects the stereoscopic image, another aspect of the configuration of the basic image is based on the detected information to be referred to. 14. It is converted into
The method described in. 15. The method according to claim 10, wherein the basic image is acquired by acquiring image data that has been compressed in advance and expanding the image data. 16. When performing a predetermined image processing on the basic image, it is found from the detected information to be referenced that the basic image is a side-by-side format in which a plurality of images are juxtaposed and combined. It is characterized in that the processing is performed as it is, and if it is found that the basic image is a multiplex format in which a plurality of images are combined as they are in a stereoscopic display form, the basic image is once converted into the side-by-side format and then the processing is performed. The method according to claim 10. 17. When handling a stereoscopic image in a device having a memory, a basic image of a mode different from that of the structure of the basic image finally displayed on the screen is held in the memory, and the basic image is appropriately stored. A three-dimensional image processing method characterized by reading and using. 18. The method according to claim 17, wherein the different aspect is a side-by-side format in which a plurality of parallax images are juxtaposed and combined. 19. Information which is added to a basic image for displaying a stereoscopic image and should be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image is detected based on the information. A three-dimensional image processing method comprising adjusting the brightness of the display screen of the three-dimensional image. 20. The information to be referred to includes information regarding the number of viewpoints assumed in the stereoscopic image, and the brightness is adjusted according to the number.
The method described in. 21. An image acquisition unit for acquiring a basic image for displaying a stereoscopic image, and information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image on the acquired basic image. A stereoscopic image processing device, comprising: an information adding unit for adding. 22. An image acquisition unit for acquiring a basic image for displaying a stereoscopic image, and reference in a predetermined scene in a series of processing for displaying the stereoscopic image, which is added to the acquired basic image. A stereoscopic image processing apparatus comprising: an information detection unit that detects information to be processed. 23. The apparatus according to claim 22, wherein the image acquisition unit inputs precompressed image data and decompresses the image data to generate the basic image. 24. A system including a stereoscopic image synthesizing device and a display device, wherein the synthesizing device provides a basic image for displaying the stereoscopic image with a predetermined image in a series of processes for displaying the stereoscopic image. Incorporating information to be referenced in a scene, the display device is configured to detect the information to be referenced, perform appropriate image processing on the basic image based on the information, and display the stereoscopic image. 3D image processing system. 25. A basic image for displaying a stereoscopic image is acquired, and the basic image is inspected to estimate information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image. A stereoscopic image processing method characterized by the above. 26. A computer is caused to execute a process of adding information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image to a basic image for displaying the stereoscopic image. A computer program to do. 27. A computer is caused to detect information added to a basic image for displaying a stereoscopic image, which should be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image. A computer program to do. 28. A computer acquires information to be referred to in a predetermined scene in a series of processing for displaying the stereoscopic image by acquiring a basic image for displaying the stereoscopic image and inspecting the basic image. A computer program characterized by: