HK1229101B - Method for preparing 3d image - Google Patents

Description

Method for preparing three-dimensional images

This application is a divisional application of the invention patent application with the application date of May 11, 2012, application number "201210147475.4", and invention name "Method and device for preparing three-dimensional video images".

Technical Field

The present invention relates to 3D displays and, in particular, to a mechanism for eliminating crosstalk in 3D displays.

Background Art

There are various technologies available for displaying 3D images to a viewer. These technologies present separate images to the viewer's left and right eyes. These images can show different perspectives of the same scene or object. The viewer's brain combines and interprets the left and right eye images to perceive a single 3D image with a sense of depth.

Some non-limiting example technologies used in 3D displays include:

Polarize the left-eye image and the right-eye image differently. The viewer can wear polarized glasses to block the right eye from viewing the left-eye image, and block the left eye from viewing the right-eye image;

- Displaying left-eye and right-eye images in an alternating sequence. The viewer can view the images through glasses containing controlled light shutters that open and close in time with the display of the images;

A technique for providing left-eye and right-eye images with different spectral characteristics. The viewer can view the images through glasses containing a spectral filter that passes one image but blocks the other.

- Multiview technology that directs different images to different locations in space (so that the viewer's eyes see different images).

One problem with 3D imaging displays is the potential for crosstalk between the images viewed by a viewer's left and right eyes. These images can be referred to as "views." Crosstalk occurs when light intended for viewing by only one eye is also visible to some extent to the viewer's other eye. Crosstalk can occur for any of a variety of reasons. For example, a shutter, polarizer, or filter worn on each of the viewer's eyes may not completely prevent light from an image intended for viewing by only one eye from reaching the other eye, or vice versa.

Crosstalk can cause viewers to perceive "ghosting" in 3D images, the effect of seeing double images, especially at high-contrast edges in the image. Crosstalk can significantly degrade the perceived 3D image quality and can prevent viewers from blending the images and perceiving depth.

Crosstalk may be particularly noticeable where a pixel in the left-eye image is very bright and a corresponding pixel in the corresponding right-eye image is very dark, or where a pixel in the right-eye image is very bright and a corresponding pixel in the corresponding left-eye image is very dark.

There is a need for an efficient and practical way to eliminate crosstalk in 3D displays.

Summary of the Invention

This summary is provided to introduce a selection of representative concepts and aspects of the invention in a simplified form that are further described below in the specification. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way that might limit the scope of the claimed subject matter.

The present invention provides methods and apparatus for processing and displaying 3D images. One aspect relates to eliminating crosstalk in 3D images. The present invention can be implemented as a method and apparatus. The apparatus can comprise a standalone device or be integrated with a display or other device in a channel used to transmit 3D image data to the display.

One exemplary aspect of the present invention provides a method for preparing a 3D image comprising a left-eye view and a right-eye view for display. The exemplary method comprises: determining, based on image data, an amount by which pixel values are to be increased to allow for full subtractive crosstalk cancellation; determining a maximum value for the amount; and globally increasing the pixel values by one or both of addition and scaling based on the maximum value. The 3D image may comprise a video frame from a video sequence. In some embodiments, the method comprises: applying a temporal low-pass filter to the maximum value or to an amount derived from or related to the maximum value.

Another example aspect provides a method for preparing a 3D video image comprising a left-eye view and a right-eye view for each frame in a series of frames for display. The method includes: identifying a local region in the view where pixel values are too small to allow full subtractive crosstalk cancellation; determining an intensity of a luma patch to be applied to the local region; temporally low-pass filtering the intensity; and generating a luma patch based on the temporally filtered intensity.

In some embodiments, the method includes: for each of the luma patches, generating a corresponding luma patch for the other view. Generating the corresponding luma patch may include: locating a center of the luma patch; performing block matching to estimate an inconsistency between the views at a location corresponding to the center of the local region; and adding a copy of the luma patch to the other view that is offset in the x-direction by the estimated inconsistency.

Another aspect provides an example method for preparing a 3D video image that includes a left-eye view and a right-eye view for each frame in a series of frames for display. The method includes: identifying a local region in the view where pixel values are too small to allow full subtractive crosstalk cancellation; determining the intensity of a luminance patch applied to the local region; linking corresponding local regions across frames of the video image to provide one or more series of linked local regions; identifying a first linked local region in one of the series of corresponding linked local regions; and applying a faded-in luminance patch corresponding to the first linked local region to a series of frames preceding the frame corresponding to the first linked local region.

Another aspect of the present invention provides a device as described herein. The device can be configured and/or operable to perform a method according to the present invention.

Other aspects of the invention provide a computer program product comprising a non-volatile medium carrying computer-readable instructions which, when executed by one or more data processors, cause the data processors to perform a method according to the invention.

According to one embodiment, a method for preparing a three-dimensional image is provided, the three-dimensional image including a left-eye view and a right-eye view for display, the method comprising: identifying, based on image data of the left-eye view and the right-eye view, a set of pixels in each of the left-eye view and the right-eye view whose pixel values are insufficient to allow complete subtractive crosstalk cancellation; selectively determining, for the pixels in the set, an amount by which the pixel values of the pixels are to be increased to allow complete subtractive crosstalk cancellation; establishing one or more amounts by which the pixel values are to be increased based on the amounts determined for the pixels in the set; and increasing the pixel values of at least some of the pixels in the set by one or both of addition and scaling performed on the established one or more amounts.

According to another embodiment, a method for preparing a three-dimensional image is provided, the three-dimensional image including a left-eye view and a right-eye view for display, the method comprising: preprocessing image data of the left-eye view and the right-eye view to determine regions in the left-eye view and the right-eye view for which a contrast between pixel values of the left-eye view and the right-eye view exceeds a threshold; generating metadata identifying the determined regions; and storing the metadata in association with the image data.

In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and by study of the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate non-limiting example embodiments.

Figure 1 is a flow chart of a method according to an exemplary embodiment of the present invention. Figures 1A-1B are examples of two alternative specific implementations of block 100 shown in Figure 1 .

FIG2 is a block diagram illustrating an apparatus according to an example embodiment of the present invention.

FIG3 is a flowchart of a method according to another example embodiment of the present invention.

FIG4 is a graph showing smooth spatial variations of luminance patches.

5A , 5B, and 5C illustrate the addition of luma patches in one view corresponding to luma patches added to increase foot room in another view.

6 is a flow chart illustrating an example method for processing image data to add luminance patches that fade in and out for reducing flicker.

7A and 7B compare the process of performing crosstalk cancellation on each frame of video individually and fading the added brightness to assist in crosstalk cancellation in and out over time.

Like reference numerals are used throughout the drawings and the description to refer to like elements.

DETAILED DESCRIPTION

In the following description, specific details are described to provide a more thorough understanding to those skilled in the art. However, well-known elements are not shown or described in detail to avoid unnecessarily obscuring the present disclosure. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.

For example, suppose the left-eye image has pixel values for a certain color channel given by V _L (x, y), and the corresponding right-eye image has pixel values for that color channel given by _VR (x, y). Without compensating for crosstalk, the viewer will see an image with the following pixel values for that color channel in their left eye: V _L (x, y) + A _RL × _VR (x, y), while the viewer will see an image with the following pixel values for that color channel in their right eye: _VR (x, y) + A _LR × V _L (x, y). Here, A _RL is the portion of light from the right-eye image that reaches the viewer's left eye, and A _LR is the portion of light from the left-eye image that reaches the viewer's right eye. In many cases, A _LR = A _RL .

Crosstalk can be eliminated by processing the image data to subtract the estimated crosstalk. For example, the intensity of each pixel can be reduced based on the amount of expected crosstalk from the corresponding pixel in the other view. Therefore, when the images are displayed, the crosstalk should be eliminated, and the viewer should perceive the left and right images as if there was no crosstalk. However, because negative light intensities are impossible, complete subtractive crosstalk cancellation requires that the value of the pixel in each image is greater than the expected crosstalk. This is especially not the case if the pixel in one view is dark and the corresponding pixel in the other view is very bright.

Consider the following example, where the image data includes 8-bit gamma-encoded red, green, and blue values for each pixel. These values can be converted to linear values as follows:

Where I ₈ is the 8-bit gamma-encoded input for one color channel (red, green, or blue), γ is the display gamma value (typically 2.2), and I _lin is the linear value for that color channel scaled to the range of 0 to 1. The views reaching the viewer's eye, each including crosstalk from the other view, can be modeled as:

I _L,eye =I _L,lin +cI _R,lin (2)

as well as

I _R,eye =I _R,lin +cI _L,lin (3)

where IL _,lin and IR _,lin are the linear values of the input left and right views, _IL,eye and _IR,eye are the linear values exposed to the viewer's left and right eyes, and c is the amount of crosstalk.

Subtractive crosstalk cancellation works by modifying the displayed left and right view image data so that after adding the crosstalk, the signal reaching the viewer is the desired image (without crosstalk). This can be done, for example, by processing the image data so that the value of the input to the display is given by the following equation:

as well as

where the subscript proc indicates the processed input value. By substituting the processed values of equations (4) and (5) for IL _,lin and IR _,lin in equations (2) and (3), it can be verified that the outputs IL _,eye and IR _,eye will be the original images without crosstalk. Note that applying equations (4) and (5) mainly involves subtracting the right image from the left image, and subtracting the left image from the right image, while applying a small gain factor of 1/(1-c ² ) to all terms. Equations (4) or (5) will produce negative results unless the following two conditions are met:

I _L,lin ≥cI _R,lin (6)

as well as

I _R,lin ≥cI _L,lin (7)

In some embodiments, light intensity is added to each view in an amount sufficient to allow subtractive cancellation of crosstalk. The cancellation may be complete or partial. The amount of light intensity added may be based on the image data so that no more light intensity is added than is required to allow the desired degree of crosstalk cancellation.

In some embodiments, processing the image data includes compressing the image data and/or shifting the image data toward higher intensities such that the black level is raised. The amount by which the black level is raised can be calculated based on the image data.

In an exemplary embodiment, processing includes applying a function that linearly scales the color channels (e.g., red, green, and blue channels) to compress their range to [b, 255]. This function may be given, for example, by the following equation:

I'＝b+(1-b/255)I (8)

Where I represents a value from any one of the color channels I∈(R,G,B). The same scaling can be applied to each color channel to avoid hue shift. Larger values of b will further compress the dynamic range of the image, which reduces image contrast. Therefore, it may be desirable to keep the value of b small while still allowing a desired degree of crosstalk cancellation.

It can be shown that complete crosstalk cancellation is possible if b satisfies the following equation:

For the case of 8-bit values, where the subscript S indicates the signal image (i.e., the currently considered view) and the subscript C indicates the crosstalk image (i.e., the view that is the source of crosstalk in the currently considered view), Equation (9) can be applied to provide constraints on the value of b for each color channel (e.g., R, G, and B) of each pixel in each view.

In an exemplary embodiment, equation (9) is applied to each color channel of each pixel in each view. A maximum value of b is determined and used in equation (8) to apply the same image data compression to all color channels for all pixels in both views. This ensures that the conditions of equations (6) and (7) hold for all pixels in the processed image data.

In some embodiments, the value of b is chosen to be slightly smaller than the value determined by equation (9). Although this means that full subtractive crosstalk cancellation cannot be performed, a small amount of uncancelled crosstalk does not significantly affect image quality, and reducing b advantageously reduces the amount of compression.

The processing can be adjusted to avoid the detailed calculation of b for each color channel of each pixel. For example, b does not need to be calculated for pixel values that satisfy the following equation:

I _S ≥c ^1/γ I _C (10)

This is because for these pixel values, full compensation for any crosstalk can always be performed.In many typical images, 90% or more of the pixel values satisfy equation (10).

In the case where the images comprise video frames, it may be desirable to provide temporal filtering to avoid distracting the viewer due to sudden changes in compression. This can be relatively straightforward if the compression is defined by a single parameter such as b, as described above. In this case, b can be low-pass filtered so that b does not change suddenly between frames.

In some embodiments, the filtering is bidirectional in time, so that b begins to increase several frames before the frame where the abrupt jump would otherwise occur.

In some alternative embodiments, compression using the corresponding b-values can be performed separately for different color channels. However, this may distort the colors and is generally undesirable when color fidelity is a concern.

The processing described above is not limited to representing pixel values as 8-bit values. Pixel values may be represented in any suitable manner, for example, as 10-bit, 11-bit, or 12-bit values, within any suitable range, such as 0 to 1. Furthermore, processing may be performed using color models other than RGB or other additive color models.

In some embodiments, processing is performed using a color model that provides pixel values representing light intensity separately from pixel values representing hue (or chroma). An example of such a color model is the YCbCr color model. Many other color models exist, such as YUV, xvYCC, YPbPr, YIQ, and the like.

Scaling can be applied to the luminance (luma) component of the YCbCr color model (or the corresponding component on other similar color models) without changing the pixel values that define hue (chrominance). Therefore, processing image data whose pixel values are represented by the YCbCr color model (or similar color model) can provide the advantage of better color preservation.

In the ITU-R BT.709 standard commonly used for HDTV, luminance (Y) has a range of 16 to 235. The luminance range can be compressed by increasing the minimum level while keeping the maximum value at 235. This can be achieved, for example, by scaling the luminance as follows:

Y'＝b+(1-b/235)Y (11)

Where Y' is the compressed brightness and b is the compression parameter. As above, it is advantageous to choose the smallest possible b value while still allowing crosstalk to be eliminated. A set of conditions that guarantee complete crosstalk compensation is:

R′ _S ≥c ^1/γ R′ _C

G′ _S ≥c ^1/γ G′ _C

B′ _S ≥c ^1/γ B′ _C (12)

In the case of conversion between RGB color space and YCbCr color space given by:

R＝1.164(Y-16)+1.793(Cr-128)

G＝1.164(Y-16)-0.213(Cb-128)-0.534(Cr-128)

B＝1.164(Y-16)+2.112(Cb-128) (13)

By scaling the brightness as follows according to equation (11), and applying condition (12), we can solve for the minimum value of b that will result in condition (12) being satisfied in all color channels. When the transformation is expressed as (13), it can be shown that it produces the following equation for the red channel:

Similarly, the condition for the green channel is given by:

The condition for the blue channel is given by:

The calculations shown in equations (14) to (16) can be performed for all pixels of both views to determine a b value that is large enough to achieve the desired degree of crosstalk reduction. Once the b value is determined, the luminance channels of the two views can be scaled, for example, according to equation (11), converted to RGB space according to equation (13), and then crosstalk compensation can be performed according to equations (4) and (5).

In some embodiments, the video can be analyzed at the source or at another point in the distribution path upstream of the display. The analysis can include comparing pixel values in two views that both correspond to the same frame to identify the most restrictive pixel pairs for the frame (e.g., the pixel pairs that exhibit the greatest contrast between the views). The analysis does not require knowledge of the crosstalk c that typically depends on the display displaying the image. The results of the analysis can be distributed with the image (e.g., as metadata associated with the image). The metadata can be extracted and used when determining the appropriate scaling value for display using subtractive crosstalk cancellation. This approach reduces computational complexity at the display and can allow image processing and crosstalk cancellation to be performed at the display using simpler hardware than would otherwise be required.

Where a display performs image processing (eg, applying a sharpening filter) prior to displaying an image as described herein, it may be desirable to perform such image processing prior to performing image processing to prepare the image for subtractive crosstalk cancellation as described herein.

FIG1 illustrates a method 100 according to an example embodiment of the present invention. At block 102, the method 100 determines the amount of additional floor space required for subtractive crosstalk cancellation for each pixel in each of the left-eye view and the right-eye view. As shown in FIG1A , the required additional floor space may be determined separately for each of a plurality of color channels (as shown in blocks 101R, 101G, and 101B (which process the left-eye view) and blocks 102R, 102G, and 102B (which process the right-eye view)).

In an optional embodiment shown in Figure 1B, the pixel values are represented in a color model with a separate luminance value, or the pixel values are transformed into a color model with a separate luminance value, and in blocks 103L and 103R that process the left eye view and the right eye view, respectively, the required additional bottom space is determined based on the luminance value.

In block 104, a value for the amount of compression (e.g., b) or luma to be added to each color channel or luma channel is determined. Block 104 may be based, for example, on a maximum or nearly maximum value of the additional headroom required for the color channels of both views, or a maximum or nearly maximum value of the additional headroom required for the luma channels of both views. The nearly maximum value may be, for example, the 90th, 95th, or 99th percentile value.

In block 106, where the image is a video frame, a low-pass temporal filter is applied to the amount of compression or brightness from block 104.

In block 108 , the image is compressed (or brightness is added) based on the temporally filtered amount from block 106 .

In block 110, subtractive crosstalk cancellation is performed.

In block 112, the image is displayed.

FIG2 illustrates a device 200 according to an example embodiment. Device 200 includes a computation block 202 that determines the amount of additional headroom required for subtractive crosstalk cancellation. Computation block 202 may include multiple components that operate in parallel on different color channels or luminance channels for the left and right views of an image. Computation block 202 may include a programmed data processor, such as one or more microprocessors, graphics processors, or digital signal processors, programmable logic circuits such as FPGAs, hardwired logic circuits, combinations thereof, and the like.

Compression determination block 204 determines the desired compression based on the output of block 202. Temporal filter 206 filters the output of block 204. Compressor 208 compresses the image data of the two views based on the output of temporal filter 206. Block 210 performs subtractive crosstalk cancellation. Block 212 is a display that displays the image.

Example

The method described above was applied to three 3D stereo images. Assuming a display gamma of 2.2 and a crosstalk of 7% (c=0.07). For comparison, the same images were processed by worst-case RGB domain linear scaling (raising the black level to 255c ^1/r =76).

An objective comparison of the results is presented in Table 1, which shows the compression range for each image and the chromaticity difference between the original and compressed images, measured using the CIEDE2000 color difference formula. CIEDE2000 color difference is a perceptual metric for comparing two colors based on the CIELAB color space. To prevent the comparison from being affected by the added brightness caused by applying these methods, the lightness term (L) in the CIEDE2000 formula was set to zero, while retaining the terms related to chromaticity and hue. This produces a metric that only measures color change.

Comparing the images shows that the worst-case RGB scaling makes the image appear washed out. The optimal RGB scaling as described above produces an image with better contrast, but is still slightly washed out compared to the original image. Scaling in YCbCr space produces an image with more saturated colors that more accurately reflect the colors of the input image (as indicated by the significantly lower CIEDE2000 chromaticity difference values).

Because crosstalk does not pose a problem in all parts of the image, it is not mandatory to perform the processing that allows subtractive crosstalk cancellation to be performed globally. Some embodiments identify areas where crosstalk cancellation is needed or desired and perform processing on these local areas to ensure that the desired degree of subtractive crosstalk cancellation can be performed within these local areas. This processing can increase the brightness level in the local area by adding and/or scaling the brightness. As described above, scaling can be performed on the brightness channel or on each color channel.

FIG3 illustrates a method 300 according to an example partial embodiment. In method 300, in block 302, a pair of views of a 3D image are processed to detect regions (local regions) where subtractive crosstalk cancellation would fail. Optionally, in block 304, these regions are analyzed based on criteria to determine whether crosstalk cancellation is necessary or desirable in the region. For example, if uncancelled crosstalk is confined to a very small region or occurs only for a very short period of time (e.g., in the case of video, for only one frame or a few consecutive frames in the video), uncancelled crosstalk does not produce noticeable image artifacts. For the regions selected by block 304, processing continues at blocks 305, 306, and 307. Block 305 determines the amount of headroom to be added to each region, block 306 increases the headroom for subtractive crosstalk cancellation by summing luminance and/or scaling, and block 307 adds the headroom to the corresponding region in the other view. In some embodiments, the luminance or scaling amount to be added is determined separately for each region. Areas not selected at block 304 may be ignored. Subtractive crosstalk cancellation is performed at block 308, and at block 310 the image is displayed.

It is advantageous to vary the added brightness smoothly. For example, in a border region around a local region, the added brightness can be slowly increased from zero to the amount added to the local region. In the case where an image has multiple local regions to be processed to improve subtractive crosstalk cancellation, different added brightness and/or proportional amounts can be applied to different regions within the local region.

Here, i(x,y) represents any of the red, green, and blue channels of the image. The same signal α(x,y) can be added to all channels to avoid excessive pixel color changes. Separate versions of α(x,y) can be generated for the left-eye and right-eye views. To construct these signals for a local region, it is possible to determine how much floor space is required for full subtractive crosstalk cancellation in that local region, and then generate a smooth signal that sufficiently boosts brightness for effective crosstalk cancellation.

Local areas where improved crosstalk cancellation may be desirable can be identified based on the calculated amount R _K that each pixel in the view needs to be improved to provide sufficient floor space for crosstalk cancellation. R _K can be given by:

R _K (x,y)＝max(0,c×I _C,K (x,y)-I _S,K (x,y)) (18)

Wherein, the subscript S indicates the "signal" view being considered, the subscript C indicates the "crosstalk" view that is the source of crosstalk added to the signal view when displaying the image, and the subscript K indicates the color channel (which can be, for example, one of red, green, and blue). Because any negative value indicates that the signal view has sufficient brightness for subtractive crosstalk cancellation, the result in equation (18) is clamped to zero by the "max" function.

Identifying local regions may be based on a modified version of R _K where values less than a threshold are set to zero. The threshold may be based on, for example, maximum display brightness. In an exemplary embodiment, values less than 1% of the maximum display brightness are set to zero.

For example, a small area that is less visually noticeable can be removed by using a small mask to erode and dilate R _K. For example, a circular mask of 8 pixels can be used to remove the small area.

For example, local regions to be processed to improve crosstalk cancellation can be identified by dividing R _K (x, y) (as modified by any preprocessing) into a plurality of connected regions. This can be performed, for example, using a binary labeling algorithm. A suitable example binary labeling algorithm is described in R M Haralick and L G Shapiro. Computer and Robot Vision, Volume 1. Addison-Wesley, 1992, pp. 40-48, which is incorporated herein by reference.

For each of these local areas, a patch that smoothly changes the brightness can be added, which has the maximum value of the pixels in the local area and gradually decreases within the area of surrounding pixels based on their distance from the local area. As an example, the brightness patch to be added to a local area can be calculated as follows:

Where subscript j is the label of the local region under consideration, _Mj is the maximum value of _Rk in local region j, w is the width of the transition region outside local region j, and d is the distance of the pixel at x,y from local region j (d=0 for pixels within local region j). Figure 4 is a curve 400 of a section of α(x,y) that smoothly varies by example through the center of a local region 402. Within region 402, α(x,y) has a constant value 403. At the boundary or "transition" region 404 (portions 404A and 404B of transition region 404 are shown in Figure 4), α(x,y) decreases smoothly (and in this example, linearly) from value 403.

For HD resolution video, a suitable transition width (w) is 200 pixels. A larger width will result in a less noticeable transition and also cause a larger portion of the image to have lower contrast.

Other forms of α(x,y) may also be used. For example, α(x,y) is not required to decrease linearly with distance from local region j. Instead, α(x,y) may decrease smoothly according to an optional function such as a Gaussian or sigmoid function, a polynomial function, a spline curve, or the like. α(x,y) is also not required to have the same value 403 throughout local region j. Instead, α(x,y) may gently peak within local region j or vary in some other smooth manner.

α(x, y) can be determined based on the amount R _K that each pixel in the local region needs to be increased to provide sufficient floor space for crosstalk cancellation. Equation (18) can be performed for each color channel. α(x, y) can be based on the maximum value of R _K over all color channels in the local region. The same signal α(x, y) can be added to all color channels in the local region. Different amounts of brightness can be added to different local regions.

When adding brightness to a local area of one view, the same brightness should be added to the corresponding local area of the other view to prevent retinal rivalry. Identifying the corresponding local area in the other view may take into account estimating inconsistencies between views. For example, in one embodiment, identifying the corresponding local area in the other view includes calculating the centroid of the local area determined for one view and then performing block matching to estimate inconsistencies between views at the centroid of the local area. A relatively large block size (e.g., 16 pixels) may be used for block matching.

The added brightness for the corresponding local area in the other view can be generated by adding a copy of α(x,y) to the other view that is offset in the x direction by the estimated inconsistency. Figures 5A and 5B illustrate this. In Figure 5A, patches _αj (x,y) for various local areas in the left and right images are shown. In Figure 5B, additional patches corresponding to the patches from the other view and offset by the estimated inconsistency are added to each view.

The entire process described here is performed twice, once with the left view considered as the signal (S) and the right view as the source of crosstalk (C), and once with the right view considered as the signal and the left view as the source of crosstalk (C). To generate the final brightness to be added to each image, the maximum value of all the individual patches α(x,y) covering any pixel (x,y) can be taken. An example of this is shown in Figure 5C.

A smoothed final luminance signal is added to the input image, and then subtractive crosstalk cancellation can be applied (e.g., as described with respect to equations (4) and (5)). If necessary or desired, the final image can be converted from linear space to gamma-coded space, or transformed into a desired image format and/or color model representation.

In the case where the images are frames in a video sequence, temporal filtering may be performed to reduce flicker due to the appearance and disappearance of luma patches. In addition, patches of short duration may optionally be removed. In some embodiments, the luma patches are gradually faded in and out. For example, where a luma patch α(x,y) appears for the first time in a particular video frame, a series of attenuated copies of α(x,y) may be added to the series of video frames immediately preceding the particular video frame. Where a luma patch α(x,y) appears for the last time in a particular video frame, a series of attenuated copies of α(x,y) may be added to the series of video frames immediately following the particular video frame. The attenuated copies may cause the patch α(x, y) to increase linearly upwards or according to some other function.

FIG6 shows a method 600 for processing video data to fade luma patches in or out. In block 602, regions in temporally adjacent frames are matched. This can be done, for example, by starting at the first frame and advancing through each frame in the video, each time matching the local region providing the luma patch with the corresponding local region in the next frame. Matching can include, for each local region in frame N, comparing the centroid with the centroid of the local region in frame N+1. If the distance between the center of the local region in frame N and the center of the local region in frame N+1 is less than a threshold amount (e.g., less than a certain number of pixels), the local regions are considered matched and "linked" together. If more than one local region meets the matching criteria, the best matching local region can be linked. The best match can be based on center-to-center distance and/or the size and shape of the local region. For example, the matching local region with the closest center-to-center distance can be linked.

Linking may be achieved by providing a data structure that identifies the local regions in each frame and includes, for each frame, pointers to any matching local regions in the next and previous frames.

In optional block 603, local regions with very short durations are discarded. A viewer is less likely to notice crosstalk that occurs over a short period of time. To achieve this, block 603 may be the number of frames covered by a group of linked local regions (excluding any duplicated regions). If the count of a group of temporally linked regions is less than a threshold (e.g., the number of frames corresponding to 1/2 second or one second, such as 30 frames for a 30 fps video), all local regions in the group may be deleted.

If no match is found in frame N+1 for the local region in frame N, some embodiments copy the local region in frame N+1. In method 600, block 604 performs this function. This copying can be used for one or both of the following purposes: allowing the brightness patch of the local region to fade out, thereby preventing visually obvious sudden drops in brightness; and filling in time gaps in local regions that were not detected in one or more frames but were detected in later frames. Block 605 applies a fade after the last linked region. When copying the brightness patch from frame N to frame N+1, the value of Mj can be reduced by a small amount so that the brightness patch fades out over time. For example, an example embodiment reduces Mj in the copy of the brightness patch by 0.1% of the displayed brightness in each frame. This causes a typical brightness patch to fade out in 2-3 seconds.

Block 608 identifies local areas that appear for the first time in a frame that is not at the beginning of the video. To prevent brightness from jumping suddenly when a local area appears for the first time, block 609 fades in the brightness patches applied to these local areas. A way to achieve this is to go through the frames and check if there are no local areas in frame N that have backward links to the corresponding local areas in frame N-1. If any such local areas are found, a copy of the brightness patch can be added to the previous frame, first added to frame N-1, then to N-2, etc. Each time a brightness patch is copied, the value of Mj can be reduced by a small amount so that the brightness patch will fade in when the video is played.

As in the case of the fade-out discussed above, each copy of the brightness patch can be lowered or attenuated relative to the copy in the next frame by a small amount, such as 0.1% of the displayed brightness, to achieve a gradual transition time of, for example, a few seconds (e.g., 2-3 seconds in some embodiments).

To prevent flicker, temporal filtering is applied to ensure that the quantity Mj is consistent between frames and changes slowly over time. For example, a low-pass filter with a cutoff frequency of 0.5 Hz can be applied to the added luminance patch to eliminate flicker in the video (based on the transient frequency response of the human visual system). In an example embodiment, an 80-tap low-pass filter with a cutoff frequency of 0.5 Hz is applied to the value of Mj for each region over time. After filtering, a luminance patch is created for each local region using the temporally filtered value of _Mj in equation (19). In FIG6 , temporal filtering is performed in block 610.

As described above, once one or more luma patches are created for one view, corresponding luma patches can be created for addition to the other view. This can be done by performing an inconsistency search using block matching for each local region and applying an offset version of the corresponding luma patch to the other view. To save computational effort, the inconsistency of the region in the previous frame can be used as an initial estimate, with a smaller inconsistency search used for most frames (and searching for example +/- 2 pixels). The final luma added to each view can be obtained by taking the maximum value of all patches covering the pixel. The appropriate final image is then added to each view. Subtractive crosstalk cancellation can then be performed. In Figure 6, crosstalk cancellation is performed in block 611. The resulting 3D image is displayed in block 612.

FIG7A shows the intensity of patches added over the course of a 10-second video using the method described herein. For comparison, FIG7B shows the intensity of patches calculated for each frame individually. As shown in FIG7B , the patches determined individually for each frame may be completely inconsistent from frame to frame. Sometimes local regions to which processing should be applied for crosstalk cancellation are not detected in every frame. Only a few local regions are detected in one or two frames. This produces annoying, very noticeable flickering in the output video.

By contrast, as shown in FIG. 7A , the method as described herein gives a much smoother temporal transition and reduces or avoids flicker.

The methods described herein can be performed in real time or in a non-real time pre-processing step. These methods can be applied, for example, in 3D computer displays, 3D televisions, and 3D cinema displays.

In some embodiments where processing is applied to the signal in real time as it is received, when a new local region is detected that was not in the previous frame, a fade-in for the new region is started at the beginning of the frame. Since crosstalk may have visible effects during the fade-in time, a relatively fast fade-in may be used.

The method described herein can be adapted to be suitable for crosstalk models other than the linear model exemplified by equations (2) and (3). To use a different crosstalk model, R _K can be calculated using a function that gives the minimum amount of light required to compensate for the crosstalk from another view according to the crosstalk model. The crosstalk model can, for example, provide nonlinear crosstalk, different amounts of crosstalk in different color channels, and/or spatially varying crosstalk.

For example, a device configured to perform the method described herein can be provided in a display or a driver for the display, in an attached computer for driving a 3D display, in a video player or media player, in a 3D digital cinema projector, or the like. The device can be independent or integrated with the display. In some embodiments, the device is configured to obtain information characterizing crosstalk in the display and process image data for display on the display based on the information characterizing the display crosstalk. The device can be configured to obtain the information characterizing the crosstalk, for example, from the display itself or from an online database.

The systems and modules described herein can include software, firmware, hardware, or any combination thereof suitable for the purposes described herein. A system or module can perform one or more functions, and a software, firmware, or hardware can perform the functions of one or more systems or modules. The systems and modules can reside on general-purpose computers such as servers, workstations, personal computers, computerized tablet computers, personal digital assistants (PDAs), and other devices suitable for the purposes described herein. Those skilled in the relevant art will understand that various aspects of the system can be implemented by other communications, data processing, or computer system configurations, including: Internet appliances, handheld devices (including PDAs), wearable computers, all types of cellular phones or mobile phones, multiprocessor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, minicomputers, mainframe computers, etc. In fact, the terms "computer," "server," "host," "host system," etc. are usually used interchangeably here, and these terms refer to any one of the above-mentioned devices and systems and any data processor. In addition, various aspects of the system can be implemented in a special-purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described in detail herein.

Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a computer driving a 3D display, a 3D projector, a 3D display, a graphics card, a source unit connected to drive a 3D projector or display, a set-top box for a cable television system, or other data distribution system, can implement the methods described herein by executing software instructions in a program memory accessible to the processor. The present invention can also be provided in the form of a program product. The program product can include any non-volatile medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. The program product according to the present invention can be in any of a variety of different forms. For example, the program product can include physical media such as magnetic data storage media including floppy disks, hard drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, etc. The computer-readable signals on the program product can optionally be compressed or encrypted.

Examples of the present technology may be stored or distributed on computer-readable media, including magnetically or optically readable computer disks, hard-wired or pre-programmed chips (e.g., EEPROM semiconductor chips), nanotechnology memories, biological memories, or other data storage media. In practice, computer-implemented instructions, data structures, screen displays, and other data for aspects of the system may be distributed over a period of time over the Internet or other networks (including wireless networks), on propagated signals over propagation media (e.g., electromagnetic waves, acoustic waves, etc.), or may be provided over any analog or digital network (packet-switched, circuit-switched, or other schemes).

Where a component (e.g., assembly, device, etc.) is referred to as such, unless otherwise indicated, reference to the component (including reference to the device) should be interpreted as an equivalent of the component, including any component that performs the same function as the described component, including components that are not structurally equivalent to the disclosed structure for performing the function of the exemplary embodiments of the invention shown.

Processes, methods, lists, etc. are presented in a given order. Optional examples may be performed in a different order, and some elements may be deleted, moved, added, subdivided, combined, and/or modified to provide alternatives or sub-combinations. Each of these elements may be implemented in a variety of ways. Furthermore, although elements are sometimes shown as being performed serially, they may instead be performed in parallel, or may be performed at different times. Some elements may be conditional, which is not shown for brevity.

Unless the context clearly requires otherwise, throughout the specification and claims, the words "including", "comprising", etc. should be interpreted in an inclusive sense, that is, in the sense of "including, but not limited to". As used herein, the terms "connected", "coupled" or any variation thereof means any connection or coupling, direct or indirect, between two or more elements; the coupling or connection between elements can be physical, logical, or a combination thereof. In addition, the words "herein", "above", "below" and words of similar meaning should refer to the entire document and not to any particular part. Where the context permits, words using the singular or plural number may also include the plural or singular number, respectively. The word "or" with respect to a list of two or more items covers all of the following interpretations of the word: any one of the items in the list; all the items in the list; and any combination of the items in the list.

Although various exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, permutations, additions, and sub-combinations thereof. It is therefore intended that the following appended claims and claims hereafter introduced be interpreted as including all such variations, permutations, additions, and sub-combinations as are within their true spirit and scope.

The present invention also includes the following embodiments:

(1) A method for preparing a 3D image, the 3D image including a left-eye view and a right-eye view for display, the method comprising:

determining, based on the image data, an amount to increase pixel values to allow for complete subtractive crosstalk cancellation;

determining a maximum value for said amount;

The pixel values are globally increased by one or both of addition and scaling in amounts based on the maximum value.

(2) The method of (1), wherein the 3D image comprises a video frame in a video sequence, and the method comprises applying a temporal low-pass filter to the maximum value or a quantity derived from the maximum value.

(3) The method according to (2), wherein the temporal filter comprises a bidirectional filter.

(4) The method according to any one of (1) to (3), wherein determining the amount to increase the pixel value comprises calculating:

R _K (x,y)=max(0,F(I _C,K (x,y))-I _S,K (x,y))

where _Rk is the amount to increase the pixel at position (x,y), F() is the crosstalk function, I _S,K is the value of color channel K of the pixel in one view of the image, and I _C,K is the value of color channel K of the pixel in the other view of the image.

(5) The method of (4), wherein F() includes multiplication by a crosstalk factor c.

(6) The method of any one of (1) to (5), wherein globally increasing the pixel value comprises globally scaling and/or adding values in a luminance channel in a color model having separate luminance and chrominance channels.

(7) The method according to (6), wherein the color model includes a YCbCr color model.

(8) A method for preparing a 3D video image, the 3D video image comprising a left-eye view and a right-eye view for each frame in a series of frames for display, the method comprising:

identifying local regions in the view where pixel values are too small to allow full subtractive crosstalk cancellation;

determining an intensity of a brightness patch applied to the local area;

temporally low-pass filtering the intensities; and

The luminance patch is generated from the temporally filtered intensity.

(9) The method of (8), comprising: for each of the luma patches, generating a corresponding luma patch for another view.

(10) A method according to (9), wherein generating the corresponding luminance patch includes: locating the center of the luminance patch; performing block matching to estimate the inconsistency between the views at a position corresponding to the center of the local area; and adding a copy of the luminance patch offset in the x direction by the estimated inconsistency to the other view.

(11) The method of (8), comprising: applying a fade-out to each luma patch in a frame following the last frame associated with the luma patch.

(12) The method of (8), comprising fading in each luma patch in a frame preceding a first frame associated with the luma patch.

(13) The method according to (10), wherein the block matching is performed using a block size of 16 pixels or larger.

(14) A method for preparing a 3D video image, the 3D video image comprising a left-eye view and a right-eye view for each frame in a series of frames for display, the method comprising:

identifying local regions in the view where pixel values are too small to allow full subtractive crosstalk cancellation;

determining an intensity of a brightness patch applied to the local area;

linking corresponding local regions across frames of the video image to provide one or more series of linked local regions;

identifying a first linked local area of a series of the series of corresponding linked local areas; and

A faded-in luminance patch corresponding to the first linked local area is added to a series of frames preceding the frame corresponding to the first linked local area.

(15) The method according to (14) includes: identifying the last linked local area of the series of corresponding linked local areas; and adding a faded brightness patch corresponding to the last linked local area to a series of frames following the frame corresponding to the last linked local area.

(16) A non-volatile medium carrying a set of computer-readable signals, the computer-readable signals comprising instructions that, when executed by a data processor, cause the data processor to perform the method for preparing a 3D image according to any one of (1) to (15).

(17) An apparatus for preparing a 3D image, the 3D image comprising a left-eye view and a right-eye view for display, the apparatus comprising:

means for processing the image data to determine an amount by which pixel values are to be increased to allow for complete subtractive crosstalk cancellation;

means for determining a maximum value of said quantity;

Means for globally increasing the pixel values by one or both of adding or scaling in amounts based on the maximum value.

(18) The apparatus of (17), wherein the 3D image comprises a video frame in a video sequence, and the apparatus comprises means for applying a temporal low-pass filter to the maximum value or a quantity derived from the maximum value.

(19) The apparatus according to (18), wherein the temporal filter comprises a bidirectional filter.

(20) The apparatus according to any one of (17) to (19), wherein the means for processing the image data to determine the amount to increase the pixel value is configured to calculate:

R _K (x,y)=max(0,F(I _C,K (x,y))-I _S,K (x,y))

where _Rk is the amount to increase the pixel at position (x,y), F() is the crosstalk function, I _S,K is the value of color channel K of the pixel in one view of the image, and I _C,K is the value of color channel K of the pixel in the other view of the image.

(21) The apparatus of (20), wherein F() includes multiplication by a crosstalk factor c.

(22) An apparatus according to any one of (17) to (21), wherein the device for globally increasing the pixel value includes: a device for globally scaling and/or adding values in a luminance channel in a color model having separate luminance and chrominance channels.

(23) The apparatus according to (22), wherein the color model includes a YCbCr color model.

(24) An apparatus for preparing a 3D video image comprising a left-eye view and a right-eye view for each frame in a series of frames for display, the apparatus comprising:

means for identifying local regions in the view having pixel values too small to permit complete subtractive crosstalk cancellation;

means for determining an intensity of a brightness patch applied to said local area;

means for temporally low-pass filtering said intensities; and

means for generating the luminance patch from the temporally filtered intensity.

(25) The apparatus according to (23), wherein the means for generating the luma patches is configured to generate a corresponding luma patch for another view for each of the luma patches.

(26) The apparatus according to (24), comprising: means for locating the center of the luminance patch; means for performing block matching to estimate an inconsistency between the views at a position corresponding to the center of the luminance patch; and means for adding a copy of the luminance patch offset in the x direction by the estimated inconsistency to the other view.

(27) The apparatus of (24), comprising means for fading each luma patch in a frame subsequent to the last frame associated with the luma patch.

(28) The apparatus of (24), comprising means for fading in each luma patch in a frame preceding a first frame associated with the luma patch.

(29) An apparatus for preparing a 3D video image comprising a left-eye view and a right-eye view for each frame in a series of frames for display, the apparatus comprising:

means for identifying local regions in the view where pixel values are too small to permit full subtractive crosstalk cancellation;

means for determining the intensity of a brightness patch applied to said local area;

means for linking corresponding local regions across frames of said video image to provide one or more series of linked local regions;

means for identifying a first linked local area of one of said series of corresponding linked local areas; and

Means for adding a faded-in luminance patch corresponding to said first linked local area to a series of frames preceding the frame corresponding to said first linked local area.

(30) The apparatus according to (29) comprises: a device for identifying the last linked local area in the series of corresponding linked local areas and adding a faded brightness patch corresponding to the last linked local area to a series of frames following the frame corresponding to the last linked local area.

(31) A display, a graphics card, a 3D projector, a video player, a media player or a 3D digital cinema projector provided with the device according to any one of (17) to (30).

Claims (19)

1. A method for preparing a three-dimensional image, the three-dimensional image including a left-eye view and a right-eye view for display, the method comprising: Based on the image data of the left eye view and the right eye view, identify the set of pixels in each of the left eye view and the right eye view whose pixel values are insufficient to allow for complete subtraction crosstalk cancellation; Selectively determine the amount by which the pixel value of the pixel in the set is increased to allow for complete subtractive crosstalk cancellation; One or more quantities are established to increase the pixel value based on the quantities determined for the pixels in the set; The pixel values of at least some pixels in the set are increased by adding to one or more of the established quantities and by proportionally determining one or two of them. 2. The method according to claim 1, wherein identifying the set of pixels comprises: identifying pixels with _IS ≥ ^{c 1/γ} _IC , wherein _IS is the pixel value of the currently considered view in the left-eye view and the right-eye view, _IC is the corresponding pixel value of another view in the left-eye view and the right-eye view, c is the amount of crosstalk between the left-eye view and the right-eye view when displayed, and γ is the gamma value. 3. The method of claim 1, wherein establishing the one or more quantities that increase the pixel value comprises: taking the maximum value of the determined quantities. 4. The method of claim 3, wherein establishing the one or more quantities that increase the pixel value comprises: taking the percentile value of the determined quantity. 5. The method of claim 4, wherein the percentile value is at least the 90th percentile of the determined quantity. 6. The method of claim 1, further comprising: globally increasing the pixel values in the left-eye view and the right-eye view by applying the same summation and/or proportional determination to all pixels in the left-eye view and the right-eye view. 7. The method of claim 1, wherein the pixel value is an R, G, B pixel value, and the method comprises: applying the same summation and/or proportional determination to all the R, G, B pixel values of the pixel for each of the pixels. 8. The method of claim 1, wherein the three-dimensional image comprises video frames in a video sequence, and the method comprises: applying a time low-pass filter to one or more quantities that increase the pixel values. 9. The method of claim 8, wherein the time low-pass filter comprises a bidirectional filter. 10. The method of claim 1, wherein the pixel value is represented by more than 8 bits. 11. The method of claim 1, further comprising: applying a sharpening filter before identifying a set of pixels in each of the left-eye and right-eye views whose pixel values are insufficient to allow for complete subtraction crosstalk cancellation. 12. The method of claim 1, further comprising: individually determining, for different regions within the left-eye view and the right-eye view, the amount by which the pixel value of the pixel is increased. 13. The method of claim 1, further comprising: identifying a plurality of regions within one of the left-eye view and the right-eye view, wherein the plurality of regions includes pixels whose pixel values are insufficient to allow for complete subtraction crosstalk cancellation; and for each of the plurality of regions, selectively increasing or not increasing the pixel values of pixels within that region based on whether the region is larger than a threshold size. 14. The method of claim 1, wherein the three-dimensional image comprises video frames in a video sequence, and the method comprises: identifying a plurality of regions within one of the left-eye view and the right-eye view, wherein the plurality of regions comprises pixels whose pixel values are insufficient to allow for complete subtraction crosstalk cancellation; and for each of the plurality of regions, selectively increasing or not increasing the pixel values of pixels in the region based on whether the region persists for more than a threshold number of frames in the video sequence. 15. The method of claim 14, wherein selectively increasing the pixel value of the pixels in the region or not increasing the pixel value of the pixels in the region further depends on whether the region is larger than a threshold size. 16. The method of claim 1, further comprising: identifying a plurality of regions within one of the left-eye view and the right-eye view by applying a binary labeling algorithm, wherein the plurality of regions include pixels whose pixel values are insufficient to allow for complete subtraction crosstalk cancellation. 17. The method of claim 1, further comprising: identifying a plurality of regions within one of the left-eye view and the right-eye view, wherein the plurality of regions includes pixels with pixel values insufficient to allow for complete subtraction crosstalk cancellation, wherein the established quantity includes a quantity for each of the plurality of regions, wherein the method further comprises: increasing pixel values in transition regions along the boundaries of the plurality of regions by the amount that decreases with distance from the boundaries of the plurality of regions. 18. The method of claim 17, wherein, within the transition region, the amount of reduction decreases non-linearly with distance from the boundary of the plurality of regions. 19. The method of claim 17, further comprising: increasing the pixel value in each of the plurality of regions by a different amount, such that the amount of brightness added to the region changes with position within the region to provide a gentle peak.