WO2012043003A1 - Three-dimensional image display device, and three-dimensional image display method - Google Patents

Abstract

A three-dimensional image display device is provided with: a display unit capable of displaying, in three dimensions, a set of images having parallax in a predetermined display parallax direction; a storage unit capable of storing both a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions; a selection unit for selecting a desired set of images from among the sets of images stored in the storage unit; a determination unit for determining whether or not the display parallax direction matches the image parallax direction, which is the direction in which the selected set of images selected by the selection unit has a single parallax; and a display control unit for controlling so that each image in the aforementioned selected set of images is displayed in three dimensions in the displaying unit after rotating each image of the aforementioned selected set of images by an angle such that the display parallax direction and the image parallax direction match when the determination unit determines that the display parallax direction and the image parallax direction do not match.

Description

Stereoscopic image display apparatus and stereoscopic image display method

The present invention relates to a stereoscopic image display device and a stereoscopic image display method, and in particular, based on four parallax images obtained by forming, on an imaging element, subject images that have passed through different regions in four directions of a photographing lens. The present invention relates to display technology.

Patent Document 1 discloses an imaging apparatus that arranges a single microlens for a plurality of captured images and generates a method of right and left parallax images and upper and lower parallax images.

Further, Patent Document 2 discloses a display device that can simultaneously generate both a left and right parallax image and an upper and lower parallax image.

In Patent Document 3, pixels of a stereoscopic image display device that expresses an image in three dimensions are divided and formed into pairs of odd-numbered pixels and even-numbered pixels, and the light rays are horizontally placed in front of the odd-numbered pixels and even-numbered pixels. And a stereoscopic image display device provided with polarizing means for passing only one of the vertical components.

JP 2009-165115 A JP 2006-285247 A JP-A-6-258733

A parallax image in the vertical and horizontal directions can be obtained using a Bayer array type imaging device having a configuration of one microlens for four photodiodes as in Patent Document 1.

When a parallax image in the vertical and horizontal directions is acquired, the parallax image can be rotated and displayed so as to obtain an appropriate stereoscopic view according to the parallax direction of the viewer and the orientation of the display device. For example, when a stereoscopic display device having a display parallax (a direction in which binocular parallax occurs on the display surface) in the left-right direction stereoscopically displays a vertical parallax image, if the parallax image is rotated by + 90 ° or −90 °, The pupil division direction matches the display parallax direction, and stereoscopic viewing is possible. Note that three-dimensional display methods include a parallax barrier method, a lenticular method, and a liquid crystal shutter method.

However, when only a vertical parallax image or a horizontal parallax image is acquired, stereoscopic display becomes impossible when the image or display is rotated. For example, when binocular parallax is generated by placing the display of the display device in the horizontal direction (horizontal placement), the parallax image in the left-right direction is stereoscopically displayed horizontally. However, if the display is rotated by 90 ° (vertically placed), the left-right parallax image cannot be recognized as a stereoscopic display. In addition, since the vertical parallax image has not been acquired, the vertical parallax image cannot be stereoscopically displayed on the vertical display.

Suppose that an image set consisting only of left and right parallax images and an image set consisting of vertical parallax images are acquired and mixed on the same recording medium. When the display device stereoscopically displays these images sequentially by frame advance, the parallax image is not recognized as a stereoscopic display unless the parallax direction of the parallax image matches the binocular parallax direction of the display device viewed from the observer. On the other hand, when the parallax direction of the parallax image matches the binocular parallax direction of the display device viewed from the observer, it is recognized as stereoscopic display. If the observer cannot recognize the stereoscopic effect by the frame during frame advance, the observer feels uncomfortable and tires the eyes of the observer.

The present invention has been made in view of such circumstances, and an object thereof is to enable appropriate display of a parallax image corresponding to the parallax direction of a display device.

The present invention provides both a display unit capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction, and a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions. A storage unit that can be stored, a selection unit that selects a desired set of images from the set of images in the storage unit, and a single selection image set that is a combination of the display parallax direction and the selection unit. A determination unit that determines whether or not an image parallax direction that is a parallax direction matches, and a display parallax direction and an image that are determined by the determination unit determining that the display parallax direction and the image parallax direction do not match There is provided a stereoscopic image display device including a display control unit that controls to rotate each image of a selected image set by an angle that matches a parallax direction and stereoscopically display it on a display unit.

When the display parallax direction is the horizontal direction and the image parallax direction is the vertical direction, the display control unit controls to rotate each image of the selected image set by 90 ° and display the image on the display unit in a stereoscopic manner.

The display control unit includes a setting unit that sets whether to rotate the selected image set. When the setting unit is set to rotate the image, the display control unit selects the selected image set by an angle at which the display parallax direction matches the image parallax direction. These images are rotated so as to be displayed in a stereoscopic manner on the display unit.

The display control unit displays information indicating the rotation angle of the image or the rotation direction of the display unit on the display unit so that the display parallax direction and the image parallax direction match when the setting unit is configured not to rotate the image. Control.

The present invention provides both a display unit capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction, and a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions. A storage unit that can be stored, a selection unit that selects a desired set of images from the set of images in the storage unit, and a single selection image set that is a combination of the display parallax direction and the selection unit. A determination unit that determines whether or not an image parallax direction that is a parallax direction matches, and a display parallax direction and an image that are determined by the determination unit determining that the display parallax direction and the image parallax direction do not match There is provided a stereoscopic image display device comprising: a display control unit that controls to display information indicating a rotation angle of an image or a rotation direction of a display unit such that a parallax direction coincides with the display unit.

Information is a figure and / or a character indicating the rotation angle of the image or the rotation direction of the display unit.

The display parallax direction is horizontal, the set of images with parallax in a single direction has parallax in the horizontal or vertical direction, and the set of images with parallax in multiple directions is parallax in the horizontal and vertical directions Have

When the selected image set has parallax in the horizontal and vertical directions, the display control unit stereoscopically displays the set of images having parallax in the horizontal direction on the display unit, and stereoscopically displays the set of images having parallax in the vertical direction. Control is performed so that information indicating that the information can be displayed is displayed on the display unit.

When the selected image set has parallax in the horizontal and vertical directions, the display control unit stereoscopically displays the set of images having parallax in the horizontal direction on the display unit, and displays the set of images having parallax in the vertical direction. To display on the display unit a representative image showing a state of rotation so as to match.

When the selected image set has parallax in the horizontal and vertical directions, the display control unit converts the set of images having parallax in the vertical direction by rotating the set of rotated images by an angle that matches the display parallax direction and the parallax in the horizontal direction. Control is performed to stereoscopically display the set of images on the display unit sequentially or simultaneously.

The light beam from the subject is divided into four light beams by pupil division in the vertical and horizontal directions, and the four light beams are imaged and photoelectrically converted into four photoelectric conversion element groups corresponding to the color filters in a predetermined arrangement. An imaging unit that outputs a set of four viewpoint image signals having color components corresponding to each color filter, and a set of four viewpoint image signals having color components corresponding to each color filter output by the imaging unit By combining the two viewpoint image signals obtained from the photoelectric conversion elements adjacent in the vertical direction or the two viewpoint image signals obtained from the photoelectric conversion elements adjacent in the horizontal direction, there is a parallax in the horizontal direction. A composition unit that creates a set of images or a set of images having a parallax in the vertical direction, and the storage unit creates a set of images having a parallax in the horizontal direction created by the synthesis unit or in the vertical direction. The set of images having the difference is stored as a set of images having a parallax in a single direction.

The combining unit includes two viewpoint image signals obtained from photoelectric conversion elements adjacent in the vertical direction and a horizontal direction among a set of four viewpoint image signals having color components corresponding to the color filters output from the imaging unit. By combining two viewpoint image signals obtained from the photoelectric conversion elements adjacent to each other, a set of images having parallax in the horizontal direction and a set of images having parallax in the vertical direction are created. The set of images having the parallax in the horizontal direction and the set of images having the parallax in the vertical direction are stored as a set of images having the parallax in a plurality of directions.

When the selected image set has parallax in the horizontal and vertical directions, the display control unit selects a set of images having parallax that matches the display parallax direction at the time of output of the imaging unit, and stereoscopically displays the selected set of images To do.

A posture information detection unit that detects posture information at the time of outputting a set of four viewpoint image signals of the imaging unit, and the combining unit is a photoelectric conversion element adjacent in the vertical direction based on the posture information detected by the posture information detection unit The left viewpoint image signal set and the right viewpoint image signal set obtained from the above are discriminated, the discriminated left viewpoint image signal set is synthesized, and the discriminated right viewpoint image signal set is synthesized Thus, a set of images having parallax in the horizontal direction is created.

According to the present invention, a stereoscopic image display device capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction includes a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions. A step of storing both, a step of selecting a desired set of images from the set of images, and an image parallax which is a single parallax direction of a selected image set which is a set of a display parallax direction and a selected image And determining whether the display parallax direction and the image parallax direction do not coincide with each other in response to determining whether the display parallax direction and the image parallax direction do not coincide with each other. And a step of controlling each of the images to be stereoscopically displayed on a display unit.

A stereoscopic image display device capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction stores both a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions The step, the step of selecting a desired image set from the set of images, and the display parallax direction and the image parallax direction that is the single parallax direction of the selected image set that is the selected image set match. The rotation angle of the image or the display unit such that the display parallax direction and the image parallax direction match in accordance with the step of determining whether or not the display parallax direction and the image parallax direction are mismatched And a step of controlling to display information indicating the rotation direction of the three-dimensional image display method.

In the present invention, when a parallax image that cannot be stereoscopically displayed as it is is selected, the selected image is rotated in an appropriate direction and stereoscopically displayed, so that the user does not feel uncomfortable. In the present invention, the direction in which the selected parallax image can be viewed stereoscopically is displayed as information such as a figure. If the user recognizes this information and rotates the image, the user can understand that the image is displayed three-dimensionally.

Block diagram of monocular stereoscopic imaging device The figure which shows the structural example of the light-receiving surface of CCD in the case of vertical installation of 0 degree The figure which shows the structural example of the light-receiving surface of CCD in the case of +90 degrees vertical installation The figure which shows the structural example of the light-receiving surface of CCD in the case of -90 degrees vertical installation Compositing process flowchart The figure which illustrates the mode that the monocular three-dimensional imaging device 1 is "normal horizontal placement" The figure which shows the parallax image synthesize | combined corresponding to the arrangement | positioning direction of a monocular three-dimensional imaging device. The figure which shows the example of the frame recorded according to photography date The figure which shows the example of the flame | frame of the group of four images which have parallax in a horizontal / vertical direction The figure which shows the example of the flame | frame of the group of two images which have parallax in a horizontal direction The figure which shows the example of the flame | frame of the group of two images which have parallax in a perpendicular direction Flow chart of rotation process Diagram showing an example of rotation of a composite image Notification process flowchart The figure which shows the example of a display of the parallax image which consists of synthetic | combination image R * L The figure which shows the example of a display of the information explaining the rotation direction in which a three-dimensional display is attained Flow chart of rotation processing that can be selected on / off Flowchart of notification processing according to the fourth embodiment The figure which shows the example of a display of the information which shows direction of the display apparatus in which a stereoscopic vision is possible The figure which shows the example of a display of the information which shows the direction which can be displayed in three dimensions The figure which shows the example of a display of the information which shows that a plane display is possible Flowchart of notification processing according to the fifth embodiment The figure which shows the example of a display of the icon which shows that the stereo image of an up-down direction can be displayed The figure which shows the example of a display of the thumbnail of the image which has the parallax of an up-down direction The figure which shows the example which displays sequentially each of the stereoscopic image which rotated the image set Image1-1 which has parallax in the left-right direction, and the image set Image1-2 which has binocular parallax in the vertical direction by 90 degrees The figure which shows the example which each displays the image set Image1-1 which has binocular parallax in a horizontal direction, and each of the three-dimensional image Image1-2 which rotated the set of images which have binocular parallax in a vertical direction 90 degree | times. Block diagram of 3D display device

<First Embodiment>
FIG. 1 is a block diagram showing an embodiment of a monocular stereoscopic imaging device 1.

This monocular stereoscopic imaging device 1 records a captured image on a recording medium 57, and the operation of the entire device is centrally controlled by a central processing unit (CPU) 40. Further, the operation power supply of each block of the apparatus is supplied from the power supply unit 60.

The monocular stereoscopic imaging device 1 is provided with operation units 38 such as a shutter button, a mode dial, a playback button, a MENU / OK key, a cross key, and a BACK key. A signal from the operation unit 38 is input to the CPU 40, and the CPU 40 controls each circuit of the monocular stereoscopic imaging device 1 based on the input signal. For example, lens driving control, aperture driving control, photographing operation control, image processing control, Image data recording / reproduction control, display control of the display device 30 for stereoscopic display, power on / off, and the like are performed.

The shutter button is an operation button for inputting an instruction to start shooting, and is configured by a two-stroke switch having an S1 switch that is turned on when half-pressed and an S2 switch that is turned on when fully pressed. The mode dial is selection means for selecting one of an auto shooting mode for shooting a still image, a manual shooting mode, a scene position such as a person, a landscape, a night view, and a moving image mode for shooting a moving image.

The playback button is a button for switching to a playback mode in which a still image or a moving image of a stereoscopic image (3D image) or a planar image (2D image) that has been captured and recorded is displayed on the display device 30. The MENU / OK key is an operation key having both a function as a menu button for instructing to display a menu on the screen of the display device 30 and a function as an OK button for instructing confirmation and execution of selection contents. It is. The cross key is an operation unit for inputting instructions in four directions, up, down, left, and right, and functions as a button (cursor moving operation means) for selecting an item from the menu screen or instructing selection of various setting items from each menu. To do. The up / down key of the cross key functions as a zoom switch for shooting or a playback zoom switch in playback mode, and the left / right key functions as a frame advance (forward / reverse feed) button in playback mode. . The BACK key is used to delete a desired object such as a selection item, cancel an instruction content, or return to the previous operation state.

In the shooting mode, the image light indicating the subject corresponds to four types of viewpoints, up, down, left, and right, of a parallax image obtained by pupil division in the vertical and horizontal directions through the imaging lens 12, the micro lens L, and a diaphragm (not shown). An image is formed on the light receiving surface of a solid-state imaging device (hereinafter referred to as “CCD”) 16 which is a phase difference image sensor that can be acquired by the pixels 16a to 16d. The imaging lens 12 is driven by a motor driver 36 controlled by the CPU 40, and performs focus control, zoom (focal length) control, and the like. The diaphragm is composed of, for example, five diaphragm blades and is driven by a diaphragm driving unit (not shown) controlled by the CPU 40. For example, a diaphragm value (F value) F2.8 to F11 is controlled in five steps in increments of 1AV. Is done.

Further, the CPU 40 performs charge accumulation time (shutter speed) in the CCD 16 and reading control of an image signal from the CCD 16 through the timing generator 37.

The signal charge accumulated in the CCD 16 is read out as a voltage signal corresponding to the signal charge based on the readout signal applied from the timing generator 37. From the voltage signals read from the pixels 16a to 16d of the CCD 16, R, G, and B signals for each pixel are sampled and held, amplified, and then applied to the A / D converters 20a to 20d, respectively. The A / D converter 20 converts R, G, and B signals that are sequentially input into digital R, G, and B signals and outputs them to the image input controller 22.

The image signal processing unit 24 performs predetermined processing such as offset control, gain control processing including white balance correction and sensitivity correction, gamma correction processing, YC processing, etc., on the digital image signal input via the image input controller 22. Perform signal processing.

The top / bottom / left / right viewpoint image data (3D image data) processed by the image signal processing unit 24 is input to the RAM 50. The RAM 50 includes an A area and a B area each storing 3D image data representing a 3D image for one frame. In the RAM 50, 3D image data representing a 3D image for one frame is rewritten alternately in the A area and the B area. The written 3D image data is read from an area other than the area where the 3D image data is rewritten in the A area and the B area of the RAM 50. The 3D image data read from the RAM 50 is converted into an image signal for stereoscopic display by the 3D image signal processing unit 34, encoded by the video encoder 28, and provided for the stereoscopic display provided on the back of the camera. This is output to the display device 30, whereby a 3D subject image is displayed on the display screen of the display device 30.

This display device 30 is a stereoscopic display means capable of displaying stereoscopic images (left viewpoint image and right viewpoint image) as directional images each having a directivity in a predetermined direction, here the horizontal direction, by a parallax barrier. . However, the stereoscopic image display method adopted by the display device 30 is not limited to this, and each viewpoint image can be viewed individually by using a lenticular lens or wearing special glasses such as polarized glasses or liquid crystal shutter glasses. You can do it.

When the shutter button of the operation unit 38 is pressed (half-pressed) in the first stage, the CPU 40 starts the AF operation and the AE operation, and the focus lens in the imaging lens 12 is brought into the in-focus position via the motor driver 36. Control it to come. The image data output from the A / D converter 20 when the shutter button is half-pressed is taken into the AE / AWB detection circuit 44.

The AE / AWB detection circuit 44 integrates the G signals of the entire screen or integrates the G signals that are weighted differently in the central portion and the peripheral portion of the screen, and outputs the integrated value to the CPU 40. The CPU 40 calculates the brightness of the subject (shooting Ev value) from the integrated value input from the AE / AWB detection circuit 44, and the aperture value of the aperture for obtaining an appropriate exposure based on this shooting Ev value and the electronic shutter of the CCD 16 ( (Shutter speed) is determined according to a predetermined program diagram, the aperture is controlled via the aperture drive unit based on the determined aperture value, and the CCD 16 is controlled via the timing generator 37 based on the determined shutter speed. Control charge accumulation time. Note that the brightness of the subject may be calculated based on an external photometric sensor.

The AF processing unit 42 is a part that performs contrast AF processing or phase difference AF processing. When performing the contrast AF process, by extracting the high frequency component of the image data in the predetermined focus area from the image data of at least one of the left viewpoint image data and the right viewpoint image data, and integrating the high frequency component An AF evaluation value indicating the in-focus state is calculated. The AF control is performed by controlling the focus lens in the imaging lens 12 so that the AF evaluation value is maximized. Further, when performing the phase difference AF process, the phase difference between the image data corresponding to the main pixel and the sub pixel in the predetermined focus area in the left viewpoint image data and the right viewpoint image data is detected, and this phase difference is detected. The defocus amount is obtained based on the information indicating. AF control is performed by controlling the focus lens in the imaging lens 12 so that the defocus amount becomes zero.

When the AE operation and the AF operation are completed and the shutter button is pressed in the second stage (full press), the vertical and horizontal viewpoint images corresponding to the pixels output from the A / D converter 20 in response to the press are pressed. Image data is input from the image input controller 22 to the RAM 50 and temporarily stored.

The image data temporarily stored in the RAM 50 is appropriately read out by the image signal processing unit 24, where predetermined signal processing including luminance data and color difference data generation processing (YC processing) is performed. The YC processed image data (YC data) is stored in the RAM 50 again. Subsequently, the YC data is output to the image signal processing unit 24, and after a predetermined compression process such as JPEG (joint photographic photographic experts group) is executed, it is stored in the memory 48 again.

A multi-picture file (MP file: a file in which a plurality of images are connected) is generated from YC data (compressed data) of the upper, lower, left, and right viewpoint images stored in the RAM 50, and the MP file is a media recording control unit. 52 and is recorded on the recording medium 57.

The speaker 53 emits sound according to the control of the sound input / output processing unit 54. The contents of the voice are stored in the ROM 55. The microphone 56 acquires sound in conjunction with image recording and converts it into an analog sound signal. The analog audio signal is converted into compressed digital audio data via the audio input / output processing unit 54 and recorded in the RAM 50 or the recording medium 57.

The parallax correction unit 63 deconvolves the small area with a restoration filter corresponding to the small area covering each viewpoint image, and restores the corresponding small area of the viewpoint image. Specifically, the parallax correction unit 63 obtains a defocus amount corresponding to each of the small areas based on the phase difference calculated for each of the small areas. A set of defocus amounts corresponding to each of the small areas obtained over the entire effective pixel area is referred to as a defocus map. The parallax correction unit 63 temporarily stores the obtained defocus map in the RAM 50 or the like. The parallax correction unit 63 may detect feature points and corresponding points between the viewpoint images, and create a defocus map based on a difference in position information between the feature points and the corresponding points.

The ROM 55 restores the image height (distance from the image center, typically the distance from the optical axis center L of the imaging lens 12) and the defocus amount (or subject distance) of each small area in each viewpoint image. Saving the filter. The parallax correction unit 63 deconvolves the small area with the restoration filter selected for each small area of each viewpoint image, and restores the corresponding small area of the viewpoint image. Thereby, parallax according to the defocus amount (blur amount) can be given to the image.

When the stereoscopic image display mode is selected from the operation unit 38, the 3D image signal processing unit 34 performs stereoscopic image data for the display device 30 to stereoscopically display the right-eye image and the left-eye image stored in the RAM 50. To synthesize. For example, when the display device 30 adopts the parallax barrier method, the 3D image signal processing unit 34 generates stereoscopic image data in which the image for the right eye and the image for the left eye are divided into strips and the strip images are alternately arranged. And output to the video encoder 28. The image selection unit 71 selects a viewpoint image to be combined with the right-eye image and the left-eye image from the four viewpoint images. The image rotation processing unit 72 rotates the image selected by the image selection unit 71 by an angle at which the image is stereoscopically displayed. The line-of-sight display control unit 73 displays information indicating the orientation of the display device on the display device 30 such that the image selected by the image selection unit 71 is stereoscopically displayed.

FIG. 2A is a diagram showing a configuration example of the light receiving surface of the CCD 16. As shown in FIG. 2A, on the light receiving surface of the CCD 16, a red, blue, or green color filter, a pixel group of four viewpoints of pixels A, B, C, and D corresponding to the color filter, and the four pixel group. A microlens ML is disposed. The light receiving units including the color filter, the four pixel group, and the microlens are arranged in a Bayer shape.

The Bayer array has three primary colors of red (R, r), green (G, g), and blue (B, b) on the four pixel groups formed in a square lattice pattern on the semiconductor substrate surface of the CCD 16. When the color filters are arranged, a row in which the red and green filters are alternately arranged and a row in which the green and blue filters are alternately arranged are alternately provided in the column direction.

In FIGS. 2A to 2C, the XY plane is the light receiving surface of the CCD 16, X is the row direction, and Y direction is the column direction. From the viewpoint of the photographer, the pixels A and C are located on the left side, and the pixels B and D are located on the right side. Z is the optical axis direction of the lens 12, and the direction toward the subject (in this figure, the direction from the front to the back of the page) is defined as the positive direction. Hereinafter, when X is in the horizontal direction and Y is in the vertical direction, it is assumed that the monocular three-dimensional imaging device 1 is “horizontal”. When X is in the vertical direction and Y is in the horizontal direction, the monocular stereoscopic imaging device 1 is assumed to be “vertically placed”.

As shown in FIG. 2A, when the monocular stereoscopic imaging device 1 is placed horizontally and the positive direction of Y (the direction from the column C toward the column A) is vertically upward, the monocular stereoscopic imaging device 1 It is assumed that it is “landscape” or “0 ° landscape”.

In addition, although the monocular stereoscopic imaging device 1 is placed horizontally, if the positive direction of Y is directed vertically downward, it is assumed that the monocular stereoscopic imaging device 1 is “laterally placed at + 180 °”.

In addition, as shown in FIG. 2B, the monocular stereoscopic imaging device 1 is placed vertically, but when the positive direction of X (the direction from the pixel A toward the pixel B) is vertically upward, that is, the monocular stereoscopic imaging device 1 is When the photographer rotates 90 ° counterclockwise, it is assumed that the monocular stereoscopic imaging device 1 is “vertically placed at + 90 °”.

In addition, as shown in FIG. 2C, the monocular stereoscopic imaging device 1 is placed vertically, but when the positive direction of X is directed vertically downward, that is, the monocular stereoscopic imaging device 1 is 90 ° clockwise as viewed from the photographer. In the case of rotation, it is assumed that the monocular stereoscopic imaging device 1 is “-90 ° portrait”.

Attitude information including the rotation direction and rotation angle of the CCD 16 around the Z axis is detected by an attitude sensor 70 such as a gyroscope. The CPU 40 controls the posture sensor 70 to detect posture information at the timing when imaging starts when the shutter button is half-pressed or fully pressed. As a result of this control, the posture information obtained from the posture sensor 70 is recorded on the recording medium 57 in association with the vertical, horizontal, and parallax images. For example, this posture information is recorded as tag information (such as an Exif rotation tag) of an image file that stores vertical, horizontal, and parallax images. By referring to this posture information, the monocular stereoscopic imaging device 1 at the time of capturing the four-viewpoint image can perform “normal horizontal placement”, “+ 180 ° horizontal placement”, “−90 ° vertical placement”, “+ 90 °”. It is possible to determine which state is “vertically placed”.

The subject light incident on the pixel group is divided in the horizontal direction (left-right direction) and the vertical direction (up-down direction) by pupil dividing means such as a light shielding member and a mirror (not shown). Accordingly, a subject image having parallax in the vertical (up and down) direction and the horizontal (left and right) direction is formed on each pixel constituting the four pixel group.

FIG. 3 is a flowchart of the synthesis process executed by the monocular stereoscopic imaging device 1. A program for causing the monocular stereoscopic imaging device 1 to execute the following processing is recorded in a computer-readable recording medium such as the ROM 55. This process starts in response to the selection of the stereoscopic display mode from the operation unit 38. This process can also be executed by an information processing apparatus other than the monocular stereoscopic imaging apparatus 1, such as a stereoscopic display apparatus 10 or a personal computer described later.

In S <b> 1, the image selection unit 71 reads a set of four parallax images and posture information at the time of acquisition of the parallax images from the RAM 50 and the recording medium 57. In the recording medium 57, a set of four parallax images is recorded as an MP file. The image selection unit 71 determines whether or not the parallax image is captured “0 ° horizontally” from the read posture information. If Yes, the process proceeds to S2. If No, the process proceeds to S3.

In S <b> 2, the image selection unit 71 selects the image signal from the pixel A and the image signal from the pixel C as a synthesis source image of the synthesized image L, and synthesizes the image signal from the pixel B and the image signal from the pixel D. Select the image from which the image R is to be synthesized. The 3D image signal processing unit 34 combines the image signal from the pixel A and the image signal from the pixel C, the image signal from the pixel B, and the image signal from the pixel D according to the selection of the image selection unit 71. A composite image R is created.

In S3, the 3D image signal processing unit 34 determines whether the parallax image was shot with “+ 90 ° portrait” or the parallax image was taken with “−90 ° portrait” from the read posture information. . If Yes (the parallax image was captured with “+ 90 ° portrait”), the process proceeds to S4. If No (the parallax image was captured with “−90 ° portrait”), the process proceeds to S5.

In S4, the image selection unit 71 selects the image signal from the pixel A and the image signal from the pixel B as a synthesis source image of the synthesized image L, and synthesizes the image signal from the pixel C and the image signal from the pixel D. Select the image from which the image R is to be synthesized. The 3D image signal processing unit 34 creates a composite image L of the pixels A and B and a composite image R of the pixels C and D according to the selection of the image selection unit 71.

In S <b> 5, the image selection unit 71 selects the image signal from the pixel C and the image signal from the pixel D as the source image of the composite image L, and combines the image signal from the pixel A and the image signal from the pixel B. Select the image from which the image R is to be synthesized. The 3D image signal processing unit 34 creates a composite image L of the pixels C and D and a composite image R of the pixels A and B according to the selection by the image selection unit 71.

In S6, the 3D image signal processing unit 34 controls the video encoder 28 to display the composite image L on the display device 30 as the left-eye display image and the composite image R as the right-eye display image.

FIG. 4B shows a parallax image synthesized corresponding to the arrangement direction of the monocular stereoscopic imaging device 1. As shown in FIG. 4A, when the monocular stereoscopic imaging device 1 is in the “normal horizontal position” as viewed from the observer, the orientation of the imaging surface is as shown in FIG. 2A. In this case, a left-right parallax exists between the composite image L of the pixels A and C generated in S2 and the composite image R of the pixels B and D. Therefore, the composite image L is an image for left eye display, and the composite image R is an image for right eye display.

When the monocular three-dimensional imaging device 1 changes from “normal horizontal placement” to “+ 90 ° vertical placement” (FIG. 2B) rotated by + 90 ° counterclockwise, the composite image of the pixels A and B generated in S4 There is a left-right parallax between L and the composite image R of the pixels C and D. Therefore, the composite image L is an image for left eye display, and the composite image R is an image for right eye display.

When the monocular stereoscopic imaging device 1 changes from “normal horizontal placement” to “−90 ° vertical placement” ((FIG. 2C) rotated −90 ° counterclockwise, the pixels C and D generated in S5 Left and right parallax exists between the synthesized image L and the synthesized image R of the pixels A and B. Therefore, the synthesized image L is an image for left eye display, and the synthesized image R is for right eye display. It becomes the image of.

Although not shown, when the XY plane of FIG. 2 is rotated by + 180 ° about the Z axis, and the monocular stereoscopic imaging device 1 is rotated by + 180 ° from “horizontal placement” to “+ 180 ° landscape orientation”, A parallax in the left-right direction exists between the composite image L of the pixels B and D and the composite image R of the pixels A and C. Therefore, the composite image L is an image for left eye display, and the composite image R is an image for right eye display. However, since it is not normally assumed that an image is captured in “+ 180 ° landscape”, the description is omitted.

Thus, the composite images L and R can be generated in accordance with the orientation of the monocular stereoscopic imaging device 1 during imaging, and the composite images L and R can be displayed as stereoscopic images.

In the above composition process, composite images L and R for left eye display and right eye display were created and displayed. The monocular three-dimensional imaging device 1 can also record the combination of the composite images L and R on the recording medium 57 instead of displaying the composite images L and R. In this way, in the monocular three-dimensional imaging apparatus 1 capable of acquiring the vertical and horizontal parallax images, if only the vertical parallax image or the horizontal parallax image is combined and recorded, all the vertical and horizontal parallax images are recorded. Data amount is halved compared to recording.

However, in this case, the recording medium 57 may include a frame composed of the composite images L and R (two viewpoint images) and a frame composed of the upper, lower, left, and right parallax images that are not combined (four viewpoint images). Further, since the orientation of the monocular stereoscopic imaging device 1 at the time of imaging does not necessarily match the orientation of the display device 30 at the time of stereoscopic display, even if a frame composed of the composite images L and R is displayed as it is, stereoscopic display may not be obtained. .

For example, as shown in FIG. 5A, frames Image1 to Image6 are recorded on the recording medium 57 according to the shooting date and time. Here, as shown in FIG. 5B, it is assumed that frames Image1 to Image3 are a set of four images having parallax in the horizontal and vertical directions. As shown in FIG. 5C, the frames Image4 and 5 are a set of two images having parallax in the horizontal direction. Also, as shown in FIG. 5D, Image6 is a set of images having parallax in the vertical direction. Image4 · 5 is generated in S4 of the above-described combining process, and Image6 is generated in S5 of the above-described combining process.

Image1 to Image5 are recognized as stereoscopic images if a set of images having a parallax in the horizontal direction is placed on the display device 30 placed horizontally at 0 ° as it is. This is the same as S2 in the above synthesis process.

However, for Image6, the orientation of the monocular stereoscopic imaging device 1 at the time of imaging is vertical at −90 ° and does not coincide with the horizontal orientation of 0 ° at the time of viewing the image. For this reason, when the image 6 is displayed as it is with the monocular stereoscopic imaging device 1 placed horizontally at 0 °, an image rotated by −90 ° is displayed. In addition, since the parallax direction of Image6 is the vertical direction, Image6 is not stereoscopically displayed.

When the user browses the images Image1 to Image6 continuously, the images 1 to 5 are displayed in 3D, but the image 6 is not displayed in 3D. This makes the user feel uncomfortable and causes eye fatigue.

Therefore, the next rotation processing is performed for the frames of the two viewpoint images that cannot be browsed as a three-dimensional image as it is.

FIG. 6 shows a flowchart of the rotation process. This process starts in response to the selection of the stereoscopic display mode from the operation unit 38. It is assumed that the monocular three-dimensional imaging device 1 is horizontally placed at 0 ° when this process is executed. The same processing steps as those already described are given the same reference numerals.

In S11, the 3D image signal processing unit 34 reads the selected parallax image frame from the recording medium 57 in accordance with the frame selection operation to the operation unit 38. The 3D image signal processing unit 34 determines whether the read frame is a four-viewpoint image having parallax in the vertical and horizontal directions, or a two-viewpoint image having parallax in the vertical and horizontal directions. In the case of Yes (four viewpoint images), the process proceeds to the composition process. If No (two viewpoint images), the process proceeds to S13.

In S13, the 3D image signal processing unit 34 determines whether the read frame is a two-viewpoint image having parallax in the left-right direction or a two-viewpoint image having parallax in the vertical direction. In the case of Yes (parallax in the left-right direction), the process proceeds to S14. In the case of No (vertical parallax), the process proceeds to S15.

In S14, the 3D image signal processing unit 34 synthesizes stereoscopic image data for performing stereoscopic display from the read parallax images of the frames. The display device 30 performs stereoscopic display based on the stereoscopic image data.

In S15, the image rotation processing unit 72 calculates the angle difference from the posture information at the time of obtaining the parallax image to the current posture information, and rotates the parallax image of the read frame in a direction to eliminate the angle difference. The 3D image signal processing unit 34 synthesizes stereoscopic image data for performing stereoscopic display from the parallax image after rotation. The display device 30 performs stereoscopic display based on the stereoscopic image data.

Through the above processing, a two-viewpoint image that cannot be stereoscopically displayed as it is can be stereoscopically displayed by being rotated in an appropriate direction, and the user does not feel uncomfortable.

For example, when Image 6 that is a frame imaged by the monocular stereoscopic imaging device 1 in “−90 ° portrait” is displayed in “normal landscape”, the parallax direction of the parallax image does not match the display parallax direction. . Therefore, as shown in FIG. 7, by rotating the composite image L • R of Image6 by + 90 °, the parallax direction of the composite image L • R of Image6 matches the display parallax direction of the monocular stereoscopic imaging device 1 at the time of viewing. Image6 can also be viewed as a stereoscopic image.

Second Embodiment
In the rotation processing of the first embodiment, the two-viewpoint image is automatically rotated in the direction in which stereoscopic display is possible. However, instead of the rotation processing, processing that indicates information indicating the direction in which stereoscopic display is possible to the user is executed. May be.

FIG. 8 shows a flowchart of notification processing according to the second embodiment. This process starts in response to the selection of the stereoscopic display mode from the operation unit 38. Hereinafter, the same processing steps as those described above are denoted by the same reference numerals.

S11 to S13 are the same as the rotation process. However, when it determines with No in S13, it progresses to S21.

In S21, the 3D image signal processing unit 34 notifies that the parallax direction of the parallax image is different from the display parallax direction. For example, it is assumed that a parallax image Image6 composed of the composite images R and L as shown in FIG. 9A is displayed. The composite images R and L are created in S4 of the composite process. In this case, as shown in FIG. 9B, the posture information at the time of shooting (the top and bottom direction at the time of shooting the image is represented by a graphic, and the text explaining the rotation direction in which stereoscopic display is possible are also displayed. If this notification is recognized and the image is rotated, it can be understood that the image is displayed three-dimensionally.After this notification process, the rotation process may be executed in accordance with an instruction from the user.

<Third Embodiment>
In the first and second embodiments, the rotation process or the notification process may be executed in response to an instruction to turn on or off the function of rotating the viewpoint image. FIG. 10 shows the flowchart.

In S31, it is determined whether or not the operation unit 38 has instructed to turn on or off the function of automatically rotating the viewpoint image. When on is instructed, the process proceeds to the rotation process of the first embodiment, and when off is instructed, the process proceeds to the notification process of the second embodiment.

Through the above processing, notification that the image is automatically rotated or the parallax direction of the parallax image is different from the display parallax direction is performed in accordance with an instruction from the user.

<Fourth embodiment>
When the display device 30 can stereoscopically display parallax not only in the horizontal direction but also in the vertical direction (for example, Patent Document 3), in the notification processing of the second and third embodiments, instead of indicating the rotation direction of the image, the stereoscopic view is displayed. You may notify the information which shows the rotation direction of the display apparatus 30 which becomes possible.

FIG. 11 is a flowchart of notification processing according to the fourth embodiment.

S11 to S13 are the same as above. However, if No in S13, the process proceeds to S61.

In S61, the line-of-sight display control unit 73 displays information indicating the rotation direction of the display device 30 that enables stereoscopic viewing.

Suppose, for example, that Image1 to Image6 as shown in Fig. 5A are selected and displayed by frame advance. Image6 is created in S4 of the synthesis process and has vertical parallax.

In this case, since Image 6 has parallax in the vertical direction, if the display device 30 is placed 90 ° vertically rotated by 90 ° from the normal landscape, the parallax direction of Image 6 matches the binocular parallax direction on the left and right. Is recognized as a stereoscopic image. Therefore, as shown in FIG. 12, information I such as graphics and characters indicating the orientation of the display device 30 capable of stereoscopic viewing is displayed. This process is performed instead of S22 of the notification process.

When the user rotates the display device 30 by + 90 ° according to this information I, the parallax direction of Image6 and the display parallax direction of the display device 30 coincide with each other, so that the user can visually recognize the stereoscopic image. The image may be automatically rotated as in the above rotation process, but if the user himself / herself rotates the display device 30 in an appropriate direction and stereoscopic display is established, the user is shown the appropriate rotation direction, The actual rotation may be left to the user.

Note that the 3D image signal processing unit 34 may indicate information such as a figure and characters in a direction in which stereoscopic display is possible when a set of images having parallax in the vertical and horizontal directions is selected (FIG. 13). For a set of planar images that cannot be stereoscopically displayed, information such as graphics and characters indicating that planar display is possible may be displayed (FIG. 14).

<Fifth Embodiment>
In the first to fourth embodiments, information indicating that a stereoscopic image in the vertical direction can be displayed may be indicated when a set of images having parallax in both the vertical and horizontal directions is selected.

FIG. 15 is a flowchart of the notification process according to the fifth embodiment.

S11 to S15 are the same as above, but if S11 is Yes, the process proceeds to the synthesis process, and then proceeds to S41.

In S41, the line-of-sight display control unit 73 displays an icon M indicating that a vertical stereoscopic image can be displayed on the selected frame. FIG. 16 shows an example.

When the parallax images in the vertical and horizontal directions are acquired, both the image set consisting only of the parallax images in the horizontal direction and the image set consisting of the parallax images in the vertical direction can be stereoscopically displayed, but only one image set is stereoscopically displayed. The viewer may not be aware of the presence of the other image set. However, the user can know the presence of a parallax image that can display a stereoscopic image by rotating by visually recognizing the icon M.

In the above processing, instead of the icon M as shown in FIG. 16, a representative image (planar image, thumbnail image, etc.) of an image having vertical parallax that can be stereoscopically displayed may be displayed.

For example, as shown in FIG. 17, a thumbnail J1 of an image having parallax in the vertical direction of Image1 is displayed together with Image1. This thumbnail is created by reducing Image1-U or Image1-D having the vertical parallax in FIG. 5B.

<Sixth Embodiment>
In the fifth embodiment, the representative image of the image having the parallax in the vertical direction of the selected frame is displayed. Instead, all the parallax images in each direction that can be stereoscopically displayed may be displayed.

For example, it is assumed that a frame Image1 having parallax images in the vertical and horizontal directions is selected from Image1 to Image6 as shown in FIG. 5A. As shown in FIG. 5B, Image1 includes Image1-1 and Image1-2. In addition, Image1-1 and Image1-2 are a set of two parallax images each having a parallax in the horizontal direction and a parallax in the vertical direction. In this case, as shown in FIG. 18, stereoscopic images obtained by rotating a set of images Image1-1 having parallax in the left-right direction and a set of images Image1-2 having binocular parallax in the vertical direction by 90 ° are sequentially displayed.

This allows the user to sequentially view stereoscopic images in different parallax directions, and to compare and examine their favorite stereoscopic images.

Alternatively, all images in each direction that can be stereoscopically displayed may be displayed on one screen.

For example, as shown in FIG. 19, each of an image set Image1-1 having binocular parallax in the horizontal direction and a stereoscopic image Image1-2 obtained by rotating a set of images having binocular parallax in the vertical direction by 90 ° on one screen indicate.

In the first frame display, both stereoscopic images with different parallax directions may be displayed, and in the second frame display, only the one selected from both stereoscopic images by the operation unit 38 may be displayed. .

This allows the user to view stereoscopic images in different parallax directions at the same time, and to compare and examine their favorite stereoscopic images.

It should be noted that the same 3D display as Image 1 is performed even when a frame Image 2 or Image 3 having parallax in the vertical and horizontal directions is selected. When a frame Image4 / Image5 having a parallax image in the horizontal direction is selected, a set of images having binocular parallax in the horizontal direction is displayed as it is. For Image6, the rotation processing and notification processing of the above embodiment are performed.

<Seventh embodiment>
The various processes (compositing process, rotation process, notification process) described in the first to sixth embodiments can be executed by a display device having blocks related to display equivalent to the monocular stereoscopic imaging device 1.

FIG. 20 is a block diagram of the stereoscopic display device 10. Blocks relating to display of a stereoscopic image equivalent to the monocular stereoscopic imaging device 1 are denoted by the same reference numerals as those of the monocular stereoscopic imaging device 1. The recording medium 57 records the parallax image captured by the monocular stereoscopic imaging device 1. The operation unit 38 mainly accepts operations related to image display and does not need to accept operations related to imaging.

The above processing can be repeated every time a set of parallax images (frames) capable of displaying a composite image is selected from the recording medium 57.

30: Display device, 34: 3D image signal processing unit, 71: Image selection unit, 72: Image rotation processing unit, 73: Gaze display control unit

Claims (16)

A display unit capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction;
A storage unit capable of storing both a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions;
A selection unit for selecting a desired set of images from the set of images in the storage unit;
A determination unit that determines whether or not the display parallax direction matches an image parallax direction that is a single parallax direction of a selected image set that is a set of images selected by the selection unit;
In response to determining that the display parallax direction and the image parallax direction do not coincide with each other, the determination unit rotates each image of the selected image set by an angle at which the display parallax direction and the image parallax direction coincide with each other. A display control unit for controlling the display unit to perform stereoscopic display;
A stereoscopic image display device comprising: When the display parallax direction is a horizontal direction and the image parallax direction is a vertical direction, the display control unit performs control to rotate each image of the selected image set by 90 ° and to perform stereoscopic display on the display unit. The stereoscopic image display apparatus according to claim 1. A setting unit for setting whether to rotate the selected image set;
When the setting unit is configured to rotate the image, the display control unit rotates each image of the selected image set by an angle at which the display parallax direction and the image parallax direction coincide with each other on the display unit. The stereoscopic image display device according to claim 1, wherein the stereoscopic image display device controls the stereoscopic display. The display control unit, when the setting unit is set not to rotate the image, information indicating a rotation angle of the image or a rotation direction of the display unit such that the display parallax direction matches the image parallax direction. The stereoscopic image display device according to claim 3, wherein the stereoscopic image display device is controlled to display on the display unit. A display unit capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction;
A storage unit capable of storing both a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions;
A selection unit for selecting a desired set of images from the set of images in the storage unit;
A determination unit that determines whether or not the display parallax direction matches an image parallax direction that is a single parallax direction of a selected image set that is a set of images selected by the selection unit;
When the determination unit determines that the display parallax direction and the image parallax direction do not coincide with each other, the rotation angle of the image or the display unit such that the display parallax direction coincides with the image parallax direction is determined. A display control unit that controls to display information indicating a rotation direction on the display unit;
A stereoscopic image display device comprising: The stereoscopic image display device according to claim 5, wherein the information is a figure and / or a character indicating a rotation angle of the image or a rotation direction of the display unit. The display parallax direction is a horizontal direction;
The set of images having parallax in the single direction has parallax in the horizontal or vertical direction;
The stereoscopic image display device according to any one of claims 1 to 6, wherein the set of images having parallax in the plurality of directions has parallax in a horizontal direction and a vertical direction. When the selected image set has parallax in the horizontal and vertical directions, the display control unit stereoscopically displays the set of images having parallax in the horizontal direction on the display unit, and also displays the parallax in the vertical direction. The stereoscopic image display apparatus according to claim 7, wherein information indicating that the group can be stereoscopically displayed is controlled to be displayed on the display unit. When the selected image set has parallax in the horizontal and vertical directions, the display control unit stereoscopically displays the set of images having parallax in the horizontal direction on the display unit, and also displays the parallax in the vertical direction. The stereoscopic image display apparatus according to claim 7, wherein a control is performed to display a representative image indicating a state in which the set is rotated so as to match the display parallax direction on the display unit. The display control unit, when the selected image set has parallax in the horizontal and vertical directions, a set of rotated images obtained by rotating the set of images having parallax in the vertical direction by an angle that matches the display parallax direction; The stereoscopic image display apparatus according to claim 8, wherein a set of images having parallax in the horizontal direction is controlled to be stereoscopically displayed on the display unit sequentially or simultaneously. The light beam from the subject is divided into four light beams by pupil division in the vertical and horizontal directions, and the four light beams are imaged and photoelectrically converted into four photoelectric conversion element groups each corresponding to a predetermined color filter array. An imaging unit that outputs a set of four viewpoint image signals having color components corresponding to each color filter,
Out of a set of four viewpoint image signals having color components corresponding to each color filter output from the imaging unit, two viewpoint image signals obtained from photoelectric conversion elements adjacent in the vertical direction or adjacent in the horizontal direction. A synthesis unit that creates a set of images having parallax in the horizontal direction or a set of images having parallax in the vertical direction by synthesizing two viewpoint image signals obtained from the photoelectric conversion elements;
With
The storage unit stores a set of images having parallax in the horizontal direction or a set of images having parallax in the vertical direction created by the combining unit as a set of images having parallax in the single direction. The stereoscopic image display device according to any one of to 10. The synthesizing unit includes two viewpoint image signals obtained from photoelectric conversion elements adjacent in the vertical direction among a set of four viewpoint image signals having color components corresponding to the color filters output from the imaging unit, and By combining two viewpoint image signals obtained from photoelectric conversion elements adjacent in the horizontal direction, a set of images having parallax in the horizontal direction and a set of images having parallax in the vertical direction are created,
The storage unit stores the set of images having parallax in the horizontal direction and the set of images having parallax in the vertical direction created by the combining unit as a set of images having parallax in the plurality of directions. The stereoscopic image display device described. When the selected image set has a parallax in the horizontal and vertical directions, the display control unit selects a set of images having a parallax that matches a display parallax direction at the time of output of the imaging unit, and The three-dimensional image display apparatus according to claim 12, wherein the group is three-dimensionally displayed. A posture information detection unit for detecting posture information at the time of outputting a set of four viewpoint image signals of the imaging unit;
The combining unit determines a set of left viewpoint image signals and a set of right viewpoint image signals obtained from the photoelectric conversion elements adjacent in the vertical direction based on the posture information detected by the posture information detection unit, 12. A set of images having a parallax in the horizontal direction is created by combining the determined set of left viewpoint image signals and combining the determined set of right viewpoint image signals. The stereoscopic image display device according to any one of 13. A stereoscopic image display device capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction,
Storing both a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions;
Selecting a desired set of images from the set of images;
Determining whether the display parallax direction matches an image parallax direction that is a single parallax direction of a selected image set that is a set of the selected images; and
In response to determining that the display parallax direction and the image parallax direction do not coincide with each other, the respective images of the selected image set are rotated and displayed in three dimensions by an angle at which the display parallax direction and the image parallax direction coincide with each other. Controlling to
3D image display method for executing A stereoscopic image display device capable of stereoscopically displaying a set of images having parallax in a predetermined display parallax direction,
Storing both a set of images having parallax in a single direction and a set of images having parallax in a plurality of directions;
Selecting a desired set of images from the set of images;
Determining whether the display parallax direction matches an image parallax direction that is a single parallax direction of a selected image set that is a set of the selected images; and
In response to determining that the display parallax direction and the image parallax direction do not coincide with each other, the rotation angle of the image or the rotation direction of the display unit is indicated so that the display parallax direction and the image parallax direction coincide with each other. Controlling to display information;
3D image display method for executing

Images