JP2012114589A - Stereoscopic video processing method and device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simplify adjustment of a parallax amount of a stereoscopic video content at the time of production and viewing of stereoscopic video content, and to create and display a stereoscopic video content based on an appropriate amount of parallax.
SOLUTION: The present invention sets an adjusted parallax amount or a ratio between a pre-adjustment parallax amount and an adjusted parallax amount (parallax amount ratio), and inputs a right-eye video and a left-eye video. From the (input stereoscopic video), a right eye image and a left eye video (adjusted stereoscopic video) in which the parallax amount is adjusted based on the set parallax amount or parallax amount ratio are generated. At this time, based on the amount of parallax specified by the user, by adjusting the amount of parallax of the simple stereoscopic video content by the user by creating an intermediate video of an intermediate viewpoint between the right eye viewpoint and the left eye viewpoint Can do.
[Selection] Figure 2

Description

  The present invention relates to a stereoscopic image processing method, apparatus, and program, and more particularly to a method for stereoscopic viewing by presenting a right-eye image to the right eye and a left-eye image to the left eye. The present invention relates to a stereoscopic video processing method, apparatus, and program for adjusting the amount of parallax between video and left-eye video.

  Conventionally, techniques for displaying stereoscopic images have been widely studied. In particular, stereoscopic viewing methods (binocular stereoscopic viewing) using parallax between the right eye and left eye (binocular parallax) are widely used. In recent years, services using binocular stereoscopic viewing (movies, amusements, etc.) have become widespread. ing. The binocular stereoscopic technology realizes stereoscopic vision by presenting a right-eye image to the right eye and a left-eye image to the left eye. There are methods for presenting left-eye images to each eye independently.

  On the other hand, stereoscopic vision based on binocular parallax is different from stereoscopic vision normally performed by humans, and therefore, there is a problem of causing eye strain or the like depending on a person or content. In particular, it is said that when there is an excessive amount of binocular parallax (parallax amount), eye strain is likely to occur. In order to reduce or eliminate eye strain and the like, it is necessary to appropriately set the degree of projection of a stereoscopic image and the degree of depth (stereoscopic effect).

  Here, the stereoscopic effect will be described. FIG. 7 shows a binocular stereoscopic mechanism. FIG. 7A shows a case where an object is perceived in front of the monitor screen. If the object presented to the right eye is located to the left of the same object presented to the left eye, the object is perceived in front of the monitor. In this case, the larger the amount of parallax is perceived in front. On the other hand, FIG. 1B shows a case where an object is perceived behind the monitor screen. If the object presented to the right eye is located to the right of the same object presented to the left eye, the object is perceived behind the monitor. In this case, the larger the amount of parallax, the deeper the perception. Thus, the greater the amount of parallax, the higher the stereoscopic effect.

  For this reason, the producer of stereoscopic video content sets the amount of parallax so that the stereoscopic effect is appropriate for the user to view comfortably.

  However, the stereoscopic effect may be different even with the same amount of parallax. The stereoscopic effect also varies depending on the distance (viewing distance) between the monitor screen and the viewer. FIG. 8 shows the relationship between the viewing distance and the stereoscopic effect. FIG. 8A shows the case where the viewing distance is long, and FIG. 8B shows the case where the viewing distance is short. When the viewing distance is longer, the object is perceived away from the monitor screen.

  Furthermore, the way of feeling the three-dimensional effect varies depending on the individual. For this reason, it is difficult to completely control the stereoscopic effect on the stereoscopic image production side, and it is effective to adjust the parallax amount to an appropriate amount according to the user's environment in order to view the stereoscopic image comfortably. .

  As a method for adjusting the amount of parallax, as shown in FIG. 9, a technique for changing the amount of parallax by shifting the right-eye image and the left-eye image left and right is disclosed (first method).

  Also, as shown in FIG. 10, a technique for changing the amount of parallax by enlarging and reducing the right-eye image and the left-eye image is disclosed (second method).

  However, the amount of parallax occurs due to the difference between the viewpoints of the right and left eyes, and the main method for creating a stereoscopic image using binocular parallax is a method of shooting using a binocular camera. There is. Before photographing, the amount of parallax can be adjusted by changing the distance between the two cameras built in the twin-lens camera. Apply this principle to generate a video with the distance between the two cameras changed, and adjust the amount of parallax by using it as an alternative to the video for the right eye or the video for the left eye, or both Is possible.

The adjustment of the amount of parallax between the left and right eyes, which is necessary in the production of stereoscopic video content, depends on the environment and subjectivity of the user viewing the content, so it must be adjusted to an appropriate amount of parallax according to the viewer at the time of viewing. Is desirable. In the conventional technique for adjusting the amount of parallax,
(1) A method for adjusting the amount of parallax by shifting the right-eye image and the left-eye image left and right (first method); (2) the right-eye image and the left-eye image are respectively enlarged and reduced. Adjusting the amount of parallax by changing the value (second method);
In the method (1), the left and right images are not shifted due to the shift of the left and right images, and the images are deleted without being stereoscopically viewed. In the method (2), the left and right images are enlarged and reduced. In some cases, such as a case that is not suitable, the amount of parallax can be easily adjusted to create a stereoscopic video content.

  The present invention has been made in view of the above points, and simplifies adjustment of the parallax amount of stereoscopic video content when producing or viewing stereoscopic video content, and creates and displays stereoscopic video content based on an appropriate amount of parallax. It is an object of the present invention to provide a stereoscopic image processing method, apparatus, and program capable of performing the above.

In order to solve the above-described problems, the present invention provides a right eye in an apparatus for presenting a right-eye image to the right eye, presenting a left-eye image to the left eye, and stereoscopically viewing with binocular parallax. A stereoscopic image processing method for adjusting the amount of parallax between an eye image and a left eye image,
A parallax amount setting means in which the parallax amount setting means sets the parallax amount after adjustment or the ratio between the parallax amount before adjustment and the parallax amount after adjustment (parallax amount ratio);
The parallax amount adjusting means adjusts the parallax amount from the input right-eye video and left-eye video (input stereoscopic video) based on the parallax amount or the parallax amount ratio set in the parallax amount setting step. A parallax adjustment step for generating right-eye and left-eye images (adjusted stereoscopic images).

Further, according to the present invention (Claim 2), in the parallax amount adjusting step according to Claim 1,
Using an existing morphing technique, and generating an intermediate image of images included in each video based on the input right-eye video and left-eye video, and adjusting the amount of parallax,
A parameter indicating the degree of change necessary for generating an intermediate image by the morphing technique is determined based on the parallax amount ratio.

  According to the present invention, a part of the screen of stereoscopic video content is deleted by simplifying the adjustment of the parallax amount and creating an intermediate video of the stereoscopic video content based on the specified parallax amount by applying an existing morphing technique. It is possible to adjust the amount of parallax without changing the display size of the video. By adjusting the amount of parallax, it becomes possible to realize the amount of parallax suitable for the user and the user environment, and it is possible to enjoy 3D video safely and comfortably. Therefore, it is possible to increase the added value of a product such as a monitor equipped with the present invention or a service using the product.

It is a figure which shows the morphing technique used by this invention. 1 is a configuration example of a stereoscopic video processing apparatus according to an embodiment of the present invention. It is an example of a structure of the parallax amount setting part in one embodiment of this invention. It is an example of the corresponding | compatible table of parallax amount information and setting parallax amount ratio in one embodiment of this invention. It is an example of the correspondence table of parallax information and set parallax amount in an embodiment of the present invention. It is an example of the correspondence table of the user information and setting parallax amount in an embodiment of the present invention. It is a figure which shows the mechanism of binocular stereoscopic vision. It is a figure which shows the relationship between a visual distance and a three-dimensional effect. It is a figure which shows the technique (1st method) which adjusts the amount of conventional parallax. It is a figure which shows the technique (2nd method) which adjusts the amount of conventional parallax.

  Embodiments of the present invention will be described below with reference to the drawings.

  In the present invention, by applying an existing morphing technique, an image of a viewpoint between the right eye viewpoint and the left eye viewpoint is generated using a parameter based on the designated amount of parallax, and the amount of parallax is adjusted.

  As shown in FIG. 1, the morphing technique is a technique for generating an image (intermediate image) in the middle of changing when changing from one image to another. If the shapes of the two images are significantly different or if the change in the intermediate image is not linear, the intermediate image may not be generated correctly. However, the right-eye video and the left-eye video are similar in image. In addition, since the change is linear, an intermediate image can be created with high accuracy.

  FIG. 2 shows a configuration example of a stereoscopic video processing apparatus according to an embodiment of the present invention.

  The stereoscopic video processing apparatus shown in FIG. 1 includes a parallax setting unit 1 and a parallax amount adjustment unit 2.

  The parallax amount setting unit 1 sets a parallax amount to be adjusted or a ratio between a parallax amount to be adjusted (set parallax amount) and an input parallax amount (set parallax amount ratio) and outputs the parallax amount to the parallax amount adjusting unit 2.

  When the right-eye and left-eye video (input stereoscopic video) is input, the parallax amount adjustment unit 2 adjusts the parallax amount based on the parallax amount or the parallax amount ratio set by the parallax amount setting unit 1. And an image for the left eye (adjusted stereoscopic image).

  Hereinafter, the parallax amount setting unit 1 will be described in detail.

  FIG. 3 shows a configuration example of the parallax amount setting unit in one embodiment of the present invention. As shown in the figure, the parallax amount setting unit 1 has two configurations depending on the difference in output (set parallax amount ratio / set parallax amount).

  The parallax amount setting unit 1A illustrated in FIG. 3A includes a user input unit 10 and a parallax amount ratio calculation unit 11, and outputs the set parallax amount ratio with the parallax amount information as an input. In the user input unit 10, parallax amount information is input by the user via an input device such as a remote controller. The parallax amount information can be selected or input step by step according to a word or a numerical value indicating the degree of the amount, or a unit of the numerical value, for example, “strong” “weak” “100%” “ The parallax amount is expressed in a manner that is easy for the user to understand, such as “50%”. The input parallax amount information is output to the parallax amount calculation unit 11 directly or via a memory. The parallax amount calculation unit 11 calculates the set parallax amount ratio from the input parallax amount information based on the correspondence table (FIG. 4) of the parallax amount information and the set parallax amount ratio stored in advance in a memory or the like, and the parallax amount Output to the adjustment unit 2.

  The parallax amount setting unit 1B illustrated in FIG. 3B includes a user input unit 10 and a parallax amount calculation unit 12, and outputs the set parallax amount with the parallax amount information as an input. In the user input unit 10, the parallax amount information is input by the user via an input device such as a remote controller as in the case of the parallax amount setting unit 1A in FIG. The input parallax amount information is output to the parallax amount calculation unit 11 directly or via a memory. As shown in FIG. 5, the parallax amount calculation unit 11 uses the correspondence table in which the parallax amount information stored in advance in a memory or the like and the set parallax amount are associated with each other, based on the set parallax amount information. Is calculated.

  4 and 5 are stored in the storage unit such as a memory connected to the parallax amount setting unit 1 or the parallax amount setting unit 1.

The parallax amount adjusting unit 2 calculates a set parallax amount ratio based on the input from the parallax amount setting unit 1, adjusts the parallax amount of the input stereoscopic video based on the set parallax amount ratio, and outputs the adjusted stereoscopic video. First, a method for calculating the set parallax amount ratio will be described. The method for calculating the set parallax amount ratio differs depending on the information input from the parallax amount setting unit 1 to the parallax amount adjustment unit 2. Specifically, if the input is
1) Set parallax amount ratio;
2) Set parallax amount;
3) None;
It is divided into three ways. These types of input information need to be set in the system in advance. Each calculation method will be described in detail below.

1) When input is set parallax amount ratio:
The input set parallax amount ratio is used as a calculation result.

2) When input is set parallax amount:
The parallax amount of the input stereoscopic video (input parallax amount) is calculated, and the ratio of the input parallax amount and the set parallax amount is calculated as the parallax amount ratio as shown in the following equation.

上記の式の入力視差量は任意のアルゴリズムによって算出される。例えば、文献１「野尻、山之上、花里、岡野、"位相相関法を用いた立体ハイビジョンの視差量測定、"映像情報メディア学会技術報告、26（51），37−42，200-07−19」に記載の位相相関法を用いた視差量算出アルゴリズムや、文献２「"２眼式立体映像のコンテンツ評価システムの試作、" 映像情報メディア学会誌 : 映像情報メディア 60（6）， 934−942， 2006−06−01」に記載の、グラディアント法もしくはブロッキング法を用いた視差量算出アルゴリズムを用いてもよい。

The input parallax amount of the above equation is calculated by an arbitrary algorithm. For example, reference 1 “Nojiri, Yamanoue, Hanasato, Okano,“ Measurement of parallax in stereoscopic hi-vision using the phase correlation method, ”Video Information Media Society Technical Report, 26 (51), 37-42, 200-07-19 "The parallax calculation algorithm using the phase correlation method" and reference 2 "Prototype of a content evaluation system for two-lens stereoscopic video," The Journal of the Institute of Image Information and Television Engineers: 60 (6), 934-942 , 2006-06-01 ”, a parallax calculation algorithm using a gradient method or a blocking method may be used.

3) Without input:
As shown in FIG. 6, the set parallax amount is calculated using a correspondence table between user information and the set parallax amount stored in advance in a memory or the like. User information is information about user attributes, viewing distance, and viewing environment. It is necessary to set an optimal amount of parallax for user information in advance. Then, the ratio between the input parallax amount and the set parallax amount is calculated as the parallax amount ratio as shown in the following equation.

上記の式で算出された視差量比に基づき、入力立体映像の視差量を調整した調整立体映像を生成する。視差量の調整には、モーフィング技術を用いる。モーフィング技術とは、図１に示すように、ある画像から、別の画像に変化していく場合の、変化途中の画像（中間画像）を生成する技術であり、文献３「松延、長原、岩井、谷内田、鈴木、"モーフィングによる高解像度高フレームレート動画像の生成、"電子情報通信学会論文誌。Ｄ、情報・システム Ｊ90-D（4），1073-1084，2007-04-01」や文献４「ハッサニエン，高橋，中嶋，"フェイシヤルアニメーションを目的に開発したエラスティックボディースプラインに基づいたイメージワーピング、"電子情報通信学会技術研究報告.ＩＥ，画像工学 97（123），27-32，1997-06-20」など様々な手法が提案されており、任意の手法を利用可能である。モーフィング技術は、パラメータの設定により、２つの画像の中間位置を調整することができる。図１に示すように、例えば、パラメータＰが０.５の場合は、画像Ａが５０％、画像Ｂが５０％の画像が生成される。パラメータＰが０.２５の場合は、画像Ａが２５％、画像Ｂが７５％の画像が生成される。入力された視差量比をパラメータＰとして、モーフィング技術を適用することで、設定した視差量の右眼用映像と左眼用映像の中間映像が生成することができる。右眼もしくは左眼用映像と生成された中間映像を調整立体映像として出力する。

Based on the parallax amount ratio calculated by the above formula, an adjusted stereoscopic video in which the parallax amount of the input stereoscopic video is adjusted is generated. A morphing technique is used to adjust the amount of parallax. As shown in FIG. 1, the morphing technique is a technique for generating an image (intermediate image) in the middle of changing from one image to another image. Reference 3 “Matsunobu, Nagahara, Iwai” , Taniuchida, Suzuki, "Generation of high-resolution and high-frame-rate moving images by morphing," IEICE Transactions. D, Information and Systems J90-D (4), 1073-1084, 2007-04-01 4 “Hassanien, Takahashi, Nakajima,“ Image warping based on elastic body splines developed for facial animation, ”IEICE Technical Report. IE, Image Engineering 97 (123), 27-32, 1997 Various methods such as “-06-20” have been proposed, and any method can be used. The morphing technique can adjust the intermediate position between two images by setting parameters. As shown in FIG. 1, for example, when the parameter P is 0.5, an image in which the image A is 50% and the image B is 50% is generated. When the parameter P is 0.25, an image in which the image A is 25% and the image B is 75% is generated. By applying the morphing technique using the input parallax amount ratio as the parameter P, it is possible to generate an intermediate image between the right-eye video and the left-eye video having the set parallax amount. The right-eye or left-eye video and the generated intermediate video are output as adjusted stereoscopic video.

  It is possible to construct the operation of the above-described components of the stereoscopic video processing device shown in FIG. 2 as a program and install it on a computer used as the stereoscopic video processing device, or distribute it via a network. It is.

  In addition, the constructed program can be stored in a storage medium such as a hard disk or a flexible disk, and can be installed or distributed in a computer.

  The present invention can be used at the time of viewing as well as being able to be used at the time of producing stereoscopic video content, and can provide a parallax amount suitable for individual viewers. For example, it is incorporated in a 3D TV monitor or the like, and the user can set the parallax amount as if the user adjusts the volume.

1 parallax amount setting unit 2 parallax amount adjusting unit 10 user input unit 11 parallax amount ratio calculating unit 12 parallax amount calculating unit

Claims (10)

Adjust the amount of parallax between the right-eye video and left-eye video in a device that presents the right-eye video to the right eye, the left-eye video to the left eye, and allows stereoscopic viewing using binocular parallax. 3D image processing method
A parallax amount setting means in which the parallax amount setting means sets the parallax amount after adjustment or the ratio between the parallax amount before adjustment and the parallax amount after adjustment (parallax amount ratio);
The parallax amount adjusting means adjusts the parallax amount from the input right-eye video and left-eye video (input stereoscopic video) based on the parallax amount or the parallax amount ratio set in the parallax amount setting step. A parallax adjustment step for generating right-eye and left-eye video (adjusted stereoscopic video);
3D image processing method characterized in that In the parallax adjustment step,
Using an existing morphing technique, and generating an intermediate image of images included in each video based on the input right-eye video and left-eye video, and adjusting the amount of parallax,
The stereoscopic video processing method according to claim 1, wherein a parameter indicating a degree of change required when generating an intermediate image by the morphing technique is determined based on a parallax amount ratio. In the parallax adjustment step,
The stereoscopic video processing method according to claim 1, wherein a parallax amount (input parallax amount) of the input stereoscopic video is calculated from the input stereoscopic video input by an existing arbitrary algorithm. In the parallax amount setting step,
From the correspondence table in which the parallax amount information and the parallax amount ratio are stored in association with each other, the parallax amount selected or input by the user in stages is expressed by a word, a numerical value, or a numerical value and a unit indicating the degree of the amount. A parallax amount ratio calculating step for obtaining a parallax amount ratio corresponding to the parallax amount information
Or
A parallax amount calculating step for acquiring a set parallax amount corresponding to the parallax amount information from a correspondence table in which the parallax amount information and the set parallax amount are stored in association with each other;
Or
When there is no selection or input from the user, it corresponds to the user information indicating the characteristics of the user set in advance from the correspondence table stored in association with the user information stored in advance and the set parallax amount A set parallax amount obtaining step for obtaining a set parallax amount to be performed;
Or
The stereoscopic video processing method according to claim 1, wherein processing is performed by all of the parallax amount ratio calculating step, the parallax amount calculating step, the set parallax amount acquiring step, or a combination of the steps. Three-dimensional video that adjusts the amount of parallax between the right-eye video and the left-eye video in order to present the right-eye video to the right eye, the left-eye video to the left eye, and stereoscopic viewing with binocular parallax A processing device comprising:
A parallax amount setting means for setting a parallax amount after adjustment or a ratio between a parallax amount before adjustment and a parallax amount after adjustment (parallax amount ratio);
From the input right-eye video and left-eye video (input stereoscopic video), for the right eye and left eye whose parallax amount is adjusted based on the previous parallax amount or the parallax amount ratio set by the parallax amount setting means Parallax amount adjusting means for generating a video (adjusted stereoscopic video);
A stereoscopic video processing apparatus comprising: The parallax amount adjusting means includes
Using an existing morphing technique, including intermediate image generation means for adjusting the amount of parallax by generating an intermediate image of an image included in each video based on the input video for the right eye and the video for the left eye;
The intermediate image generating means
The stereoscopic image processing apparatus according to claim 5, wherein a parameter indicating a degree of change required when generating an intermediate image by the morphing technique is determined based on a parallax amount ratio. The parallax amount adjusting means includes
The stereoscopic video processing apparatus according to claim 5, further comprising means for calculating a parallax amount (input parallax amount) of the input stereoscopic video from the input stereoscopic video input by an existing arbitrary algorithm. The parallax amount setting means includes
From the correspondence table in which the parallax amount information and the parallax amount ratio are stored in association with each other, the parallax amount selected or input by the user in stages is expressed by a word, a numerical value, or a numerical value and a unit indicating the degree of the amount. Parallax amount ratio calculating means for acquiring a parallax amount ratio corresponding to the parallax amount information
Or
A parallax amount calculating means for acquiring a set parallax amount corresponding to the parallax amount information selected or input from the user from a correspondence table in which the parallax amount information and the set parallax amount are stored in association with each other;
Or
When there is no selection or input from the user, the set parallax corresponding to the user information indicating the characteristics of the user set in advance from the correspondence table stored in association with the user information and the set parallax amount in advance Set parallax amount acquisition means for acquiring the amount;
Or
The stereoscopic image processing apparatus according to claim 5, comprising all of the parallax amount ratio calculating unit, the parallax amount calculating unit, the set parallax amount acquiring unit, or a combination of any of the units. The parallax amount setting means includes
Means for outputting either the set parallax amount or the set parallax amount ratio to the parallax amount adjusting means;
The parallax amount adjusting means includes
When the input value from the parallax amount setting means is the set parallax amount ratio, the input value is the calculated parallax amount ratio, and when the input value is the set parallax amount, the parallax amount of the input stereoscopic video ( (Input parallax amount) is calculated from an existing arbitrary algorithm, and includes means for calculating a ratio with the set parallax amount registered in the correspondence table in advance as a parallax amount ratio,
The stereoscopic image processing apparatus according to claim 8, wherein the calculation formula is parallax amount ratio = set parallax amount ÷ input parallax amount.   The program for functioning a computer as each means which comprises the three-dimensional video processing apparatus of any one of Claims 5 thru | or 9.

Images