HK1217068B - System and methods for generating scene stabilized metadata - Google Patents

Description

System and method for generating scene-stable metadata

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 61/859,956, filed July 30, 2013, the entire contents of which are hereby incorporated by reference.

Technical Field

The present invention relates to video content creation and presentation, and in particular, to systems and methods for generating associated metadata for improving the presentation of video data on a target display.

Background Art

Metadata in video files is typically generated per frame or for keyframes. However, in many cases, video playback may have artifacts that are objectionable to viewers of the video content. These artifacts may be noticeable between scenes (e.g., for scenes that may have certain common features). For example, a camera may be capturing video of a single actor moving in space and time—e.g., in a dimly lit room and moving to a bright, sunny space outdoors at one moment.

Such changes in ambient conditions may cause noticeable artifacts to the viewer (e.g., changing the facial tone of the actor mentioned above). This may be particularly true when the video content is to be displayed on a target display that may have limitations on its performance (e.g., brightness, color gamut rendering, etc.). For content creators (such as directors or post-production personnel), such artifacts can be mitigated by generating scene-based metadata.

Summary of the Invention

Disclosed herein are methods and systems for generating and applying scene-stabilizing metadata for a desired video data stream. Systems and/or methods are provided in which a video data stream is divided or segmented into scenes, and a first set of metadata can be generated for a given scene of the video data. The first set of metadata can be any metadata known to be a desired function of the video content (e.g., brightness, color gamut, etc.). The first set of metadata can be generated on a frame-by-frame basis. In one embodiment, scene-stabilizing metadata can be generated that is different from the first set of metadata for the scene. The scene-stabilizing metadata can be generated by monitoring desired characteristics of the scene, and the scene-stabilizing metadata can be used to keep the desired characteristics within an acceptable range of values. This can help avoid noticeable and potentially objectionable visual artifacts when presenting the video data.

In one embodiment, a method for using scene-stabilizing metadata in a video data stream includes: dividing the video data stream into a group of scenes; generating first metadata associated with a first scene within the group of scenes; generating scene-stabilizing metadata; and associating the scene-stabilizing metadata with the first scene.

In another embodiment, a system for using scene-stable metadata for video data includes: a processor; a memory associated with the processor, the memory further including processor-readable instructions so that when the processor reads the processor-readable instructions, the processor executes the following instructions: receiving a video data stream, the video data stream including a set of scenes; for the set of scenes, generating first metadata associated with the set of scenes; generating a set of scene-stable metadata; and for at least one scene, associating the scene-stable metadata with the at least one scene.

In another embodiment, a video processor includes: a processor; a memory associated with the processor, the memory further including processor-readable instructions so that when the processor reads the processor-readable instructions, the processor executes the following instructions: receiving an incoming video data stream, the video data stream including a set of scenes; receiving a first set of metadata associated with at least one scene; receiving an indication that the scene cut is substantially the next frame of the incoming video data stream; receiving scene stabilization metadata; and associating the scene stabilization metadata substantially with the next frame of the incoming video data stream.

Other features and advantages of the present system are presented below in the detailed description when the detailed description is read in conjunction with the drawings presented within this application.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments are illustrated in the referenced figures of the accompanying drawings.The embodiments and figures disclosed herein are to be considered illustrative, not restrictive.

FIG1 depicts one embodiment of the environment and architecture of a video pipeline system constructed according to the principles of the present invention.

2A and 2B depict two embodiments of video pipeline flow diagrams that may be suitable for the purposes of the present application.

FIG3 depicts one embodiment of a high-level flow diagram of video processing that may occur during display management of an exemplary target display.

FIG4 is one embodiment of a video process for the generation and association of scene-stabilizing metadata for a video file.

FIG5 is one embodiment of a flow diagram for incorporating advanced notification of scene changes into a video pipeline.

FIG6 depicts an exemplary video file segmented into scenes and a frame within a scene including an indication of a scene change.

DETAILED DESCRIPTION

As used herein, the terms "component," "system," "interface," and the like are intended to refer to a computer-related entity, which is hardware, (e.g., in execution) software, and/or firmware. For example, a component can be a process, a processor, an object, an executable instruction, a program, and/or a computer running on a processor. By way of example, both an application running on a server and the server can be components. One or more components can reside within a process, and a component can be localized on one computer and/or distributed between two or more computers. A component can also be intended to refer to a communication-related entity, which is hardware, (e.g., in execution) software, and/or firmware, and can also include wired or wireless hardware sufficient to enable communication.

Throughout the following description, certain details are set forth to provide a more thorough understanding to those skilled in the art. However, well-known elements are not shown or described in detail to avoid unnecessarily obscuring the present disclosure. Therefore, the description and drawings are to be viewed in an illustrative, non-restrictive sense.

introduction

In order to ensure temporal stability (e.g., no flickering, pulsating, flickering, etc.) in video playback on the target display and to mitigate potentially unpleasant visual artifacts, it may be desirable for metadata associated with the video data to be substantially stable over time. In several embodiments, this can be achieved by enforcing the metadata to be stable over the duration of a scene. Such metadata may be allowed to change at each scene cut. In such cases, changes in the metadata to accommodate content may not be noticeable to the viewer.

As just one example, video/image data metadata can be estimated frame by frame just before display. However, this can result in unpleasantly noticeable changes to the appearance of a scene—perhaps in the middle of a scene.

In several embodiments of the present application, systems and methods for generating or otherwise creating metadata related and/or associated with video data are described. In many embodiments of the present application, as will be described in more detail herein, associated metadata can be generated scene by scene. Such metadata can be created at the front end of the video data stream - or created at any other suitable portion of the video data stream. The video data can be transmitted and/or sent to the user/consumer/viewer of the video data - whether in a movie theater, a home viewing environment, a video conference, or anywhere else the video data will be viewed and/or consumed.

A number of metadata-generating and/or metadata-consuming techniques are described in the following commonly owned patents and/or patent applications:

(1) U.S. Patent Application 20130076763 to Messmer, published on March 28, 2013, entitled “TONE AND GAMUT MAPPING METHODS AND APPARATURS”;

(2) U.S. Patent Application 20130162666 to Messmer et al., published on June 27, 2013, entitled “DATA TRANSMISSION USING OUT-OF-GAMUT COLOR COORINATES”;

(3) U.S. Patent Application 20130141647, entitled “MEATDATA FOR USE IN COLORGRADING,” published on June 6, 2013, by Longhurst et al.; and

(4) U.S. Patent Application 20120315011 to Messmer et al., published on December 13, 2012, entitled “VIDEO DELIVERY AND CONTROL BY OVERWRITING VIDEO DATA.”

- All of these applications are hereby incorporated by reference in their entirety.

Figures 1, 2A, and 2B depict several generalized environmental systems (100, 200, and 206, respectively) in which the systems and/or methods of the present application may reside. These systems represent possible end-to-end video production/transmission/viewing video pipelines—for example, in which video may be captured, metadata extracted on a scene-by-scene basis, and placed into a video stream for distribution and sent to a target display for viewing.

In FIG1 , system 100 (or portions thereof) can be configured to perform one or more of the methods described herein. Components of system 100 can be implemented as software, firmware, hardware, and/or a combination thereof. System 100 includes a video capture subsystem 102, a post-production subsystem 103, and a display subsystem 104. Video data stream 123 (more specifically, 123-1, 123-2, and 123-3 at different points in the video stream pipeline) is generated by video capture subsystem 102 and delivered to post-production subsystem 103 for processing and editing. During the editing of video data 123, video images can be displayed and viewed on a reference display 111 of post-production subsystem 103. The edited video data 123 is delivered (via encoder 127A and decoder 127B via distribution medium 125) to display subsystem 104 for further processing and display. Each of subsystems 102, 103, and 104 (and encoder 127A) can be configured to encode metadata 225 in video data 123. Downstream subsystems can be configured to receive video data 123 from upstream devices and decode metadata 225 embedded therein. Metadata 225 can be used by downstream subsystems (e.g., subsystems 103 and 104) to direct processing and/or display of video data 123. Metadata 225 can be used by display subsystem 104, along with display characterization parameters 120, to control and/or direct video playback on display 118 of display subsystem 104.

As shown in FIG1 , subsystems 102, 103, and 104 may include processors 106, 108, and 116, respectively, and program memories 107, 109, and 117 accessible to the respective processors. Each processor (described herein and elsewhere) may include a central processing unit (CPU), one or more microprocessors, one or more field programmable gate arrays (FPGAs), or any combination thereof, or any other suitable processing unit(s) including hardware and/or software capable of operating as described herein. In each subsystem, the processor executes instructions provided by software stored in the program memory. The software may include routines that perform the metadata generation, encoding, decoding, and processing steps described herein, such as, for example, routines that perform the following operations:

(1) generating metadata 225 or receiving parameters related to metadata 225;

(2) encoding metadata 225 in the video data 123 before the video data 123 is transmitted to a downstream device;

(3) decoding metadata 225 from video data 123 received from an upstream device;

(4) processing and applying metadata 225 to guide processing and/or display of video data;

(5) selecting a method for encoding metadata 225 based on the image and/or video data 123;

(6) Etc.

System 100 may include a repository 110 accessible to subsystems 102, 103, and 104. Repository 110 may include a metadata definition library 112 (e.g., a library that informs metadata encoders and decoders how to generate and/or read metadata) and a reserved word list 114 (e.g., protected pixel values or reserved metadata words). Metadata definition library 112 may be accessed by subsystems 102, 103, and 104 when generating, encoding, and/or processing metadata. When encoding or decoding metadata 225, reserved words 114 may be compared with encoded/decoded metadata bits to identify a sequence of protection bits to be inserted (or already inserted) into the metadata stream to prevent the transmission of reserved words. While a shared repository 110 is shown in the illustrated embodiment of FIG. 1 , in other embodiments, each of subsystems 102, 103, and 104 may incorporate a local repository 110 stored on a storage medium accessible to that subsystem.

FIG2A is a flow chart illustrating the flow of data through a video delivery pipeline 200 according to certain embodiments. The video delivery pipeline 200 has stages similar to those depicted in the video delivery pipeline 100 of FIG1 . At one or more stages of the video delivery pipeline 200, metadata 225 may be generated and embedded in the video data stream 123 for use at downstream stages. The metadata 225 is transmitted along with the video data 123 through the video delivery pipeline 200 to guide processing of the video data by downstream devices and/or to guide video playback at a display subsystem at block 210. At block 206, the video data 123, including the embedded metadata 225, may be delivered to the display subsystem using systems, devices, and methods appropriate to the type of video content delivery (e.g., television broadcast via satellite, cable, or high-definition networks; streaming multimedia over IP or wireless networks; playback from a DVD or other storage medium; etc.).

In the embodiment of FIG2A , at block 202 , camera metadata 225A may be generated and embedded in the video data 123 - 1 . The camera metadata 225A may be generated based on camera settings and the video frame capture environment. The camera metadata 225A may include, for example, camera parameters that provide a snapshot of the camera settings during video frame capture. Such camera parameters may include aperture (f-stop), lens, shutter speed, sensitivity (ISO rating), etc. These camera parameters may be used to guide subsequent steps in the video delivery pipeline 200 , such as color adjustment (e.g., color tuning) during post-production editing at block 204 or display configuration at block 210 .

At block 204, post-production metadata 225B is generated and embedded in the video data 123-2. The post-production metadata 225B may include reference display and environment metadata _225B1 and source video content representation metadata _225B2 . The post-production metadata 225B may be used to guide subsequent steps in the video delivery pipeline 200, such as display configuration at block 210.

The reference display and environment metadata _225B1 may describe the reference display configuration and studio or viewing environment used in post-production editing at block 204. For example, regarding the reference display used to display the video data 123 during post-production editing at block 204, the reference display and environment metadata _225B1 may include parameters such as:

(1) A 3D gamut map that describes the hues and gamut boundaries of a reference display at fine resolution;

(2) A reduced set of parameters defining the hue and gamut boundaries of a reference display (which can be used to estimate a 3D gamut mapping);

(3) System tone response parameters describing the tone response of the reference display for each chromaticity channel;

(4) Screen size

(5) Etc.

The reference display and environment metadata _225B1 may also include parameters describing the studio environment in which the video content is color-adjusted or edited on the reference display during post-production editing at block 204. Such parameters may include ambient brightness and ambient color temperature.

The source video content representation metadata _225B2 may describe the video content that has been edited after post-production, and may include information that may identify or provide the following:

(1) Tone mapping (e.g., custom tone mapping parameters or curves that can be used to guide tone expansion at a display); and gamut mapping (e.g., custom gamut mapping parameters that can be used to guide color gamut expansion at a display);

(2) the minimum black level that is considered important in the scene (e.g., the shadow under the car);

(3) the level corresponding to the most prominent part of the scene (e.g., the actor's face);

(4) the level of the maximum white level considered important in the scene (e.g., the center of a light bulb);

(5) the most colorful color in the scene (e.g., neon lights);

(6) a map of the positions of light sources in the image or of reflecting or emitting objects in the image;

(7) The color gamut of the video source content;

(8) The image is intentionally color-adjusted to areas outside the color gamut of the reference display;

(9) protected colors that should not be changed during pre-display processing in the video processor or during display configuration;

(10) an image histogram that characterizes the image in terms of brightness or color gamut (e.g., such information can be used by downstream devices to determine average brightness to improve tonal and color gamut mapping);

(11) Scene change or reset flags that alert downstream devices that any statistics or lags from previous video frames are no longer valid;

(12) a motion map representing the video content to identify objects in motion, which can be used by downstream devices in combination with a light source position map to guide tone and color gamut mapping;

(13) an indication of the source of the color-modulated content (e.g., directly from the camera or in post-production editing);

(14) Director's creative intent settings, which can be used to control downstream devices such as decoders/TVs or other displays. For example, such settings may include: display mode controls that provide the ability to control the display to operate in a specific mode (e.g., vivid, cinema, standard, professional, etc.); content types (e.g., animation, drama, sports, games, etc.) that can be used to determine appropriate color gamut or tone mapping, etc.;

(15) and so on.

In square frame 206, video data 123-2 is delivered to the display subsystem. As seen in Figure 2B, delivery pipeline 206 may include encoder stage 127A, which is used to drive the video data 123 to be released through video distribution medium 125 (such as satellite, cable or high-definition network), IP or wireless network, or DVD or other storage medium, etc., broadcast or transmission. Decoder stage 127B may be located at the display end of square frame 206 to decode the video data 123 released through medium 125. Decoder stage 127B may be implemented with, for example, a set-top box or with a decoder in the display subsystem. In square frames 206 and/or 208, viewing environment metadata 225C and/or other metadata 225 may be embedded in the video data 123. Viewing environment metadata 225C may include, for example:

Advanced Video Coding (AVC) VDR encoder data that provides a reference monitor tone mapping or color gamut curve or the ambient luminance of the reference environment. At least some of this information can be determined by the video processor with knowledge of the display characteristics (e.g., by reading the display's Extended Display Identification Data (EDID)) and the environment of the display subsystem. In some embodiments, at least some of this information can be determined in the studio during post-production processing of the video data.

Parameters that describe the environment that the display of the display subsystem is in. Such parameters may include, for example, ambient brightness and/or hue or color temperature.

The viewing environment metadata 225C may be used to guide processing of the video data at block 208 and/or configuration of the display at block 210 .

The display subsystem includes a video processor for processing the incoming video data 123-3 at block 208. The display subsystem's video processor may perform signal processing on the video data 123-3 based on metadata 225 (e.g., metadata 225A) extracted from the video data 123 and/or known display characteristics associated with the display of the display subsystem. The video data 123 may be processed and adjusted for the display based on the display characterization parameters 226 and/or the metadata 225.

Other metadata 225 that may be embedded in the video data 123 at blocks 206 and/or 208 or other stages of the video delivery pipeline 200 includes housekeeping metadata 225D (for managing publishing rights, etc.), such as, for example:

(1) Watermark data indicating where the video content was generated, published, modified, etc.

(2) Fingerprinting data that provides a description of video content for search or indexing purposes, etc.;

(3) protection data indicating who owns the video content and/or who can access it;

(4) Etc.

The viewing environment metadata 255C may be generated based at least in part on the display characterization parameters 206 associated with the display of the display subsystem. In some embodiments, the viewing environment metadata 255C, the source video content characterization metadata _225B2 , and/or the governance metadata 225D may be created or provided by analysis of the video data 103 at the encoder stage 127A, the decoder stage 127B, and/or created or provided by the video processor at block 208.

At block 210, display configuration can be performed on the display of the display subsystem. Appropriate parameters for display configuration can be determined based on display characterization parameters 226 and/or metadata 225, such as camera metadata 225A, post-production metadata 225B (including reference display and environment metadata _225B1 and source video content characterization metadata _225B2 ), and viewing environment metadata 225C. The display is configured according to these parameters. Video data 123 is output to the display.

The metadata 225 used for processing the video data 123 at block 208 and for display configuration at block 210 is delivered in the video data stream so that the metadata 225 is received at the display subsystem (including the video processor and display) before it is applied. In some embodiments, the metadata 225 is delivered so that it is received by the display subsystem at least one video frame ahead of the frame to which the metadata 225 is to be applied. In certain embodiments, the metadata 225 is delivered one video frame ahead, and application of the metadata 225 at blocks 208 and/or 210 can be triggered when a new video frame is detected in the incoming video stream.

"Stable" metadata by scene

As mentioned above, it may be desirable to capture metadata in video files on a scene-by-scene basis. As described herein, several embodiments of the present application can capture metadata on a scene-by-scene basis (e.g., based on brightness, color gamut, etc.). In particular, one embodiment can provide a set of "stable" metadata that can be applied across identical and/or similar scenes.

In one embodiment, each scene can be associated with global scene metadata, which can be generated in response to frame-related characteristics within the scene (e.g., such as the minimum, maximum, and median luminance values in each frame). It can also be forced that scenes with similar characteristics share the same metadata so that they maintain the same look and feel during display. In another embodiment, the receiver can also receive "Advanced Notice" metadata - that is, metadata for future scenes, so it can prepare parameters related to the DM process in advance.

To appreciate the concept of "stabilizing" scene metadata, the following description is provided for illustrative purposes only and is not intended to limit the scope of this application. It may be desirable to stabilize color and brightness over the course of several scenes. In one example, suppose there are two actors in a "scene," but the camera switches to one actor and then to the other actor in a set sequence of video frames—for example, during a prolonged conversation between the two actors. Even though this may constitute a "scene" dramatically, the two different camera switches may cause color and/or brightness shifts that are both noticeable and objectionable to the viewer. In some embodiments, different metadata may be used for each switch—for example, to generate a stable appearance for the entire scene.

As another example, consider a scene in which there is a single actor—but the actor is in motion, and the camera follows the actor. Again, even though this may be a single scene dramatically, there may be brightness and/or color shifts that are both noticeable and objectionable to the viewer. As another example, a director may utilize a "dissolve" (or "fade") technique in which one scene reduces its brightness (perhaps to zero), while another scene may start at a low (e.g., zero) brightness and increase to maximum brightness over a period of several frames. Such a dissolve or fade may be used to illustrate a flashback to an on-screen actor or for other purposes.

These situations may become relevant in situations where a director may be involved in post-production processing of captured video. Such a director may color grade and brightness map the video on a professional-grade monitor (e.g., with a brightness of up to approximately 5000 nits). However, the movie may be viewed on a home video device or some other target display that may have a much lower brightness. Knowing this in advance may allow the director or other content creator the opportunity to improve the viewer's experience of the content.

Based on these several examples (and others not mentioned herein), it may be desirable from a viewer's perspective (if not that of the content creator/director of the video) to apply metadata on a scene-by-scene basis, and/or to have a process in place that can determine when to apply "stable" metadata to a scene and/or frame sequence—a process that may already be utilizing different, possibly frame-based, metadata for the current scene/frame.

For home video situations, there may often be a situation where a display management (DM) processor exists that attempts to provide the "best" (or "better") mapping of video data to the home display. The DM often provides a dynamic range map to provide a good luminance mapping from the available video data to the target display. The dynamic range map can use metadata based on luminance statistics (e.g., maximum luminance, mean luminance, and/or minimum luminance) to provide the mapping.

Several commonly owned patent applications disclose display management (DM) systems and techniques and may be useful with the systems and methods of the present application.

(1) U.S. Patent Application 20110194618, entitled “COMPATIBLE COMPRESSION OF HIGH DYNAMIC RANGE, VISUAL DYNAMIC RANGE, AND WIDE COLOR GAMUT VIDEO,” to Gish et al., published on August 11, 2011;

(2) U.S. Patent Application No. 20120229495, entitled “INTERPOLATION OF COLOR GAMUTFOR DISPLAY ON TARGET DISPLAY,” published on September 13, 2012, to Longhurst;

(3) U.S. Patent Application 20120321273 to Messmer, published on December 20, 2012, entitled “VIDEO DISPLAY CONTROL USING EMBEDDED METADATA”; and

(4) U.S. Patent Application 20130038790, published on February 14, 2013, to Seetzen et al., entitled “DISPLAY MANAGEMENT METHODS AND APPARATUS”;

- All of these applications are hereby incorporated by reference in their entirety.

An embodiment of stable scene metadata

FIG3 depicts one embodiment of a high-level block flow diagram of the present application. A video pipeline 300 may receive an encoded bitstream 301, which may further include video/image data along with metadata, which may be in a usable format—for example, frame-by-frame, scene-by-scene, and including metadata based on brightness statistics, color mapping metadata, etc.

The encoded bitstream 301 can be received by a decoder 302, which can further include a parser 304. The decoder 302 can decode the incoming bitstream, which can be encrypted, compressed, or otherwise encoded in any manner known in the art. Once decoded, the incoming bitstream can be parsed by the parser 304. The parser 304 can separate metadata from the video/image data.

The extracted video/image data may be emitted along with its associated metadata as an intermediate bitstream 303. As will be described further herein, the bitstream 303 may also include one or more flags (or some other indication, signal, etc.) 305 that may inform downstream processor(s) what metadata to apply, etc.

The intermediate bitstream 303 and/or any flags 305 may be received by a display management (DM) module 306. The DM module 306 may apply any desired image/video mapping before the final image/video data is sent to a target display 308. The target display 308 may be any suitable device that can display image and/or video data to a viewer. To name a few, such a target display 308 may be an HD television, a movie projector, a desktop monitor, a laptop computer, a tablet computer, a smart device, etc.

As mentioned, several embodiments of the present application may involve the computation and/or derivation of per-scene metadata (e.g., possibly a set of "stable" scene metadata). Such stable scene metadata may be pipeline-wise utilized during times when stable scene metadata may be employed (possibly in place of other available metadata, whether scene-based or frame-based) to mitigate artifacts that may be noticeable and/or objectionable to a viewer.

As just one example, consider a scene in a dark cave. The image might show all the dark details of the cave. However, if the camera pans to the cave's mouth (which is bright), the adaptive mapping can adjust the image accordingly—for example, dark details of the cave wall can be reduced to accommodate the new, brighter pixels. By generating and using scene-stabilizing metadata, the mapping can be optimized for the entire scene—for example, so there are no intermediate scenes that change significantly.

FIG4 is one embodiment of a high-level flow diagram 400 for stable scene metadata processing. At 402, video data may be divided into a set of scenes. This division and/or segmentation of the video into a set of scenes may be accomplished in several ways. First, the segmentation may be performed by a human user (e.g., a director, a film editor, someone in post-production, etc.). For example, in one embodiment, scene cuts may already be known from an edit decision list (EDL) (which may be used to create a movie from several different shots). In one embodiment, this EDL may be extracted and used to eliminate scene boundaries. In this way, little or no additional effort is required. Additionally, the user has the option of overriding automatically determined (or extracted) scene cuts.

Alternatively, the identification of scene segments can be performed automatically by a video processor, which can make such determinations by analyzing the video data on a scene-by-scene basis. For example, if there is a measurable large change in brightness data, color data, or other image data metric between frames, the video processor can determine that the difference likely marks the boundary between two scenes. Such automatic determinations can be enhanced in a look-ahead or multi-pass process—whereby several frames can be analyzed, and if an initial difference in an image data metric is marked, and if the metric in many subsequent frames is substantially consistent with such initial difference, then it can be assessed that a scene change is likely to have occurred.

For the purposes of this application, scenes can be identified in the video data in any known manner. At 404, metadata can be calculated, measured, or otherwise extracted on a scene-by-scene basis. For example, if there are 50 frames containing a given scene, luminance data can be measured and extracted for the entire scene. Scene metadata such as minimum luminance, mean and/or average luminance, and maximum luminance can be calculated. Other image/video metrics can similarly be measured and/or extracted to form other scene-based metadata.

The following is one embodiment of generating scene stabilization metadata within a video stream:

(1) Calculate the MIN, MID, and MAX brightness for each frame within the scene. Then combine the results for the entire scene.

a. For MIN, take the minimum of all the minimum values of all frames in the scene;

b. For MID, take the median (average) of all the median values of all frames in the scene;

c. For MAX, take the maximum of all maxima in all frames of the scene.

It will be appreciated that similar statistics may be derived for other video/image metrics (eg, color gamut data, etc.) In another embodiment, other scene-related metadata may be generated—eg, how much sharpening or smoothing to apply to image data within a scene.

At 406, a set of "stable" metadata can be calculated for the scene. The stable metadata can be different from previously calculated scene-based (or frame-based) metadata, depending on the use of such metadata. Scene-stable metadata can be calculated and/or generated for the scene, possibly based on some monitored features, aspects, and/or metrics that may generate noticeable and/or objectionable changes in the video data (e.g., even if the previously calculated scene-based metadata is to be used to render the scene for viewing). For example, in the case of an actor, moving across different backgrounds in space and time (e.g., from a dark, enclosed room to a bright, sunny outdoor environment in a single cut) can generate noticeable and/or objectionable changes in the color or tint or skin tone of the actor's face. In some embodiments, if the first scene and the second scene (e.g., different from the second scene) can be considered perceptually similar based on the monitored features, aspects, and/or metrics, the metadata for the second scene can also be replaced by the metadata calculated for the first scene. The second scene can be after or before the first scene.

Other features, aspects, and/or metrics are possible—for example, skin tones, light features/objects, dark features/objects, colored features/objects, and the like. Such changes can be mitigated using stable scene metadata. Scene stabilization metadata can be calculated and/or generated to return the monitored features, aspects, and/or metrics to and/or remain within a range of acceptable values during the course of the scene. At 408, the process can associate the stable scene metadata with any other metadata that may or may not have been previously associated with the scene and/or replace the other metadata with the stable scene metadata. This association and/or replacement of the stable scene metadata can be provided to return such features, aspects, and/or metrics to within an acceptable range—for example, where other metadata would allow such features, aspects, and/or metrics to exceed an acceptable range. The range of acceptable values for features, aspects, and/or metrics can be determined manually (for example, by a director and/or film editor) or based on certain rules and/or heuristics related to image processing/rendering and/or film editing.

It should be appreciated that the processing illustrated in FIG4 can occur at many different points in the video/image pipeline. For example, segmenting the video into scenes can be performed by a human in post-production—or elsewhere in the pipeline by a processor. Additionally, the calculation and/or extraction of scene-based metadata can be performed in post-production or elsewhere in the pipeline. Similarly, the association of "stable" scene metadata can occur in post-production or further downstream before the final video/image data is sent to the target display for presentation, for example, by a DM or other video processor.

Alternative Embodiments

In some embodiments, the mapping operation can be image content dependent in order to achieve maximum performance. Such image dependent mapping can be controlled by metadata generated from the source content. To ensure temporal stability (e.g., no flickering, pulsation, flashing, etc.), it may be desirable that the metadata be substantially stable over time. In one embodiment, this can be achieved by forcing the metadata to be stable for the duration of the scene. The metadata may be allowed to change at each scene cut. In such a case, a sudden change in the metadata to accommodate the content may not be noticeable to the viewer.

In one embodiment, the steps for generating scene stabilization metadata may include the following:

(1) Obtain the location of scene cuts in the video data. In one embodiment, this can be derived from an edit decision list (EDL). Alternatively, this can be manually entered by a human—or automatically detected by a processor.

(2) Calculate and/or generate metadata for each frame in the scene:

a. Optionally, downsample the image. (This may tend to make processing slower.

speed and minimize the influence of some distant pixel values.)

b. Convert the image to the desired color space (e.g., IPT-PQ)

c. Calculate the minimum value of the image (for example, I channel)

d. Calculate the maximum value of the image (for example, I channel)

e. Calculate the mean of the image (e.g., I channel)

(3) Combine the results of each frame into the results of each scene:

a. Calculate the minimum value of each of the frame minima.

b. Calculate the maximum value of each of the frame maxima.

c. Calculate the mean of each of the frame means.

(4) Associating metadata with the scene—or alternatively, with each frame within the scene.

It will be appreciated that variations of the above embodiments are possible and are contemplated within the scope of the present application. For example, instead of analyzing every frame of the scene in step (2), a single representative frame may be selected and used to calculate and/or generate metadata that is then associated with the entire scene.

Additionally, gradients can be supported by indicating metadata for the scene on either side of the gradient and then interpolating for the intermediate frames. Such interpolation can be linear or asymptotic on both ends via a cosine or similar function.

After step (4), the metadata can be inserted into the coded bitstream in proper synchronization with the correct video frames. The metadata can be repeated regularly to allow random input into the stream.

In yet another embodiment, some pre-calculated values can be included in the metadata to assist in converting the decoded video into the desired color space (e.g., IPT-PQ). This may be desirable because conversions are often performed on devices with fixed-point processors that may not handle certain mathematical operations well, such as division and exponential operations. The use of pre-calculated values and their embedding in the metadata stream can be beneficial.

Decoding scene stabilization metadata/"advance notice" metadata

At the video decoder, new scene metadata may arrive in the same frame as the first frame of the new scene. Alternatively, the metadata may arrive before the first frame of the scene to provide time for decoding and interpreting the metadata in time for it to be applied to the processing of the video. This "pre-notification metadata" and other techniques may be desirable to improve the robustness of scene-stable metadata over bitstream transmission. Several improvements may include the following (either individually or in combination):

(1) Repeat metadata for every frame in the same scene;

(2) adding an indicator/flag in the metadata body that the scene change will occur in the next frame;

(3) Add an indicator/flag in the metadata body that the scene change occurs in the current frame;

(4) adding an indicator/flag that the metadata for the next frame is substantially the same (or significantly different) than the metadata for the current frame; and/or

(5) Add data integrity fields in the metadata ontology for error checking (e.g., CRC32).

FIG5 depicts one embodiment of a flowchart 500 for such pre-notification metadata. At 502, the system/pipeline may calculate and/or generate metadata for each scene. At 504, this metadata—stable scene metadata or otherwise—may be associated with the scene in the video data bitstream. Then, at 506, the system/pipeline may add an indication of an impending scene change a desired number (e.g., one or more) frames before the actual first frame of the new scene. This indication and/or flag may comprise a portion of the bitstream and be notified by the DM (or other suitable processor in the pipeline). At 508, the system may allow the DM (or other suitable processor) time to install parameters and/or mappings before the scene change. This additional time may allow the system an opportunity to avoid any noticeable artifacts that may be objectionable to viewers of the video content.

This can tend to be an improvement over conventional metadata stabilization methods, where the locations of scene cuts may not be known in advance. For example, if scene cuts are not known in advance, metadata can be estimated on the fly through analysis of the video data, or allowed to change smoothly over time. This can cause image artifacts such as flickering, pulsation, or flickering. In another embodiment, by calculating metadata at the source (e.g., before video compression), computation can be reduced, thereby reducing the cost required for less capable consumer devices.

FIG6 is an example of video data 600, which is divided into several scenes (scene 1 to scene N), each of which includes several frames (e.g., frame 602a). Frame 602m of scene 1 may have a pre-notification flag associated with the frame—so that the DM can have time to set parameters and/or mapping to better present the following scene 2.

A detailed description of one or more embodiments of the present invention, to be read in conjunction with the accompanying drawings, illustrating the principles of the present invention, has now been given. It will be appreciated that the present invention is described in conjunction with such embodiments, but the invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses many alternatives, modifications, and equivalents. Many specific details have been set forth in this description in order to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention may be implemented according to the claims without some or all of these specific details. For the sake of clarity, technical material known in the technical field related to the present invention has not been described in detail so as not to unnecessarily obscure the present invention.

Claims (31)

1. A method for using scene-stabilized metadata in a video data stream, the method comprising: The video data stream is divided into a set of scenes; Generate first metadata associated with a first scene within the set of scenes, wherein generating the first metadata associated with the first scene within the set of scenes includes: Calculate the minimum, intermediate, and maximum brightness for each frame in the scene; and Calculate the minimum, intermediate, and maximum brightness of the scene; By monitoring features within the scene to generate scene-stabilized metadata; and Associating scene-stabilized metadata with the first scene, wherein associating scene-stabilized metadata with the first scene further includes: The scene-stabilized metadata is repeated for each frame within the first scene. 2. The method according to claim 1, wherein dividing the video data stream into a group of scenes further includes: The location of the scene switch is deduced from the edit decision list. 3. The method according to claim 2, wherein deriving the scene switching location from the editing decision list further includes: The location of scene switching can be determined manually or automatically by the processor. 4. The method according to claim 1, wherein generating scene-stabilized metadata further comprises: The computation ensures that the monitored features within the scene remain within acceptable ranges, resulting in scene-stable metadata. 5. The method of claim 4, wherein the monitored feature is one of a group comprising: skin tone, highlighting features, dark features, and tinted features. 6. The method according to claim 5, wherein associating the scene stability metadata with the first scene further includes: Associate scenario-stable metadata, which is different from the first metadata generated for the first scenario, with the first scenario. 7. The method according to claim 5, wherein associating the scene stability metadata with the first scene further includes: If the scene-stabilized metadata keeps the monitored features within the first scene within acceptable ranges, then the scene-stabilized metadata is associated with the first scene. 8. The method according to claim 1, wherein associating the scene stability metadata with the first scene further comprises: Add an indicator for scene transitions in the current frame or the next frame. 9. The method according to claim 1, wherein associating the scene stability metadata with the first scene further comprises: An indication that the metadata of the next frame is substantially the same as or significantly different from the metadata of the current frame. 10. The method of claim 1, wherein associating the scene stability metadata with the first scene further comprises: Add a data integrity field to the metadata for error checking. 11. A system for stabilizing metadata for video data usage scenarios, the system comprising: processor; The memory associated with the processor, the memory further including processor-readable instructions, such that when the processor reads the processor-readable instructions, the processor performs the following processing: Receive a video data stream, the video data stream comprising a set of scenes; Generate first metadata associated with a first scene within the set of scenes, wherein generating the first metadata associated with the first scene within the set of scenes includes: Calculate the minimum, intermediate, and maximum brightness for each frame in the scene; and Calculate the minimum, intermediate, and maximum brightness of the scene; By monitoring features within the scene to generate scene-stabilized metadata; and Associating the scene-stabilized metadata with the first scene, wherein associating the scene-stabilized metadata with the first scene further includes: The scene-stabilized metadata is repeated for each frame within the first scene. 12. The system of claim 11, wherein generating scene-stabilized metadata further comprises: The computation ensures that the monitored features within the scene remain within acceptable ranges, resulting in scene-stable metadata. 13. The system of claim 12, wherein the monitored feature is one of a group comprising: skin tone, glowing features, dark features, and colored features. 14. An apparatus for using scene-stabilized metadata in a video data stream, the apparatus comprising: A means for dividing the video data stream into a set of scenes; A means for generating first metadata associated with a first scene within the set of scenes, wherein the means for generating first metadata associated with the first scene within the set of scenes includes: A device for calculating the minimum, intermediate, and maximum brightness of each frame in a scene; and A device for calculating the minimum, intermediate, and maximum brightness of the scene; A means for generating scene-stable metadata by monitoring features within the scene; and A means for associating scene-stabilized metadata with the first scene, wherein the means for associating scene-stabilized metadata with the first scene further includes: A means for repeating the scene-stabilized metadata for each frame within the first scene. 15. The apparatus of claim 14, wherein the means for dividing the video data stream into a set of scenes further comprises: A device for deriving the location of scene switching from an edit decision list. 16. The device of claim 15, wherein the means for deriving the location of the scene switch from the editing decision list further comprises: A device for manually or automatically deriving the location of scene transitions via a processor. 17. The apparatus of claim 14, wherein the means for generating scene-stabilized metadata further comprises: A device for calculating scene-stable metadata that keeps monitored features within a scene within acceptable ranges. 18. The device of claim 17, wherein the monitored feature is one of a group comprising: skin tone, glowing features, dark features, and colored features. 19. The apparatus of claim 18, wherein the means for associating scene-stabilized metadata with the first scene further comprises: A means for associating scenario-stable metadata, which is different from the first metadata generated for the first scenario, with the first scenario. 20. The apparatus of claim 18, wherein the means for associating scene-stabilized metadata with the first scene further comprises: A means for associating scene-stabilized metadata with the first scene if the scene-stabilized metadata keeps the monitored features within the first scene within an acceptable range. 21. The apparatus of claim 14, wherein the means for associating scene-stabilized metadata with the first scene further comprises: A device for adding an indication of scene transition in the current frame or the next frame. 22. The apparatus of claim 14, wherein the means for associating scene-stabilized metadata with the first scene further comprises: A means for adding an indication that the metadata of the next frame is substantially the same as or significantly different from the metadata of the current frame. 23. The apparatus of claim 14, wherein the means for associating scene-stabilized metadata with the first scene further comprises: A device for adding a data integrity field to the metadata for error checking. 24. A system for using scene-stabilized metadata in a video data stream, the system comprising: processor; The memory associated with the processor, the memory further including processor-readable instructions, such that when the processor reads the processor-readable instructions, the processor performs the following processing: The video data stream is divided into a set of scenes; Generate first metadata associated with a first scene within the set of scenes, wherein generating the first metadata associated with the first scene within the set of scenes includes: Calculate the minimum, intermediate, and maximum brightness for each frame in the scene; and Calculate the minimum, intermediate, and maximum brightness of the scene; By monitoring features within the scene to generate scene-stabilized metadata; and Associating scene-stabilized metadata with the first scene, wherein associating scene-stabilized metadata with the first scene further includes: The scene-stabilized metadata is repeated for each frame within the first scene. 25. A device for stabilizing metadata for video data usage scenarios, comprising: A device for receiving a video data stream, the video data stream comprising a set of scenes; A means for generating first metadata associated with a first scene within the set of scenes, wherein the means for generating first metadata associated with the first scene within the set of scenes includes: A device for calculating the minimum, intermediate, and maximum brightness of each frame in a scene; and A device for calculating the minimum, intermediate, and maximum brightness of the scene; A means for generating scene-stable metadata by monitoring features within the scene; and A means for associating the scene-stabilized metadata with the first scene, wherein the means for associating the scene-stabilized metadata with the first scene further includes: A means for repeating the scene-stabilized metadata for each frame within the first scene. 26. The apparatus of claim 25, wherein the means for generating scene-stabilized metadata further comprises: A device for calculating scene-stable metadata that keeps monitored features within a scene within acceptable ranges. 27. The device of claim 26, wherein the monitored feature is one of a group comprising: skin tone, glowing features, dark features, and colored features. 28. A method for stabilizing metadata for video data usage scenarios, comprising: Receive a video data stream, the video data stream comprising a set of scenes; Generate first metadata associated with a first scene within the set of scenes, wherein generating the first metadata associated with the first scene within the set of scenes includes: Calculate the minimum, intermediate, and maximum brightness for each frame in the scene; and Calculate the minimum, intermediate, and maximum brightness of the scene; By monitoring features within the scene to generate scene-stabilized metadata; and Associating the scene-stabilized metadata with the first scene, wherein associating the scene-stabilized metadata with the first scene further includes: The scene-stabilized metadata is repeated for each frame within the first scene. 29. The method of claim 28, wherein generating scene-stabilized metadata further comprises: The computation ensures that the monitored features within the scene remain within acceptable ranges, resulting in scene-stable metadata. 30. The method of claim 29, wherein the monitored feature is one of a group comprising: skin tone, highlighting features, dark features, and tinted features. 31. A storage medium comprising readable instructions that, when executed by a processor, cause the method according to any one of claims 1-10 and 28-30 to be performed.