HK1228142A1 - Methods for encoding and decoding high dynamic range images - Google Patents

Description

techniques for encoding, decoding, and representing high dynamic range images

Cross Reference to Related Applications

This application claims priority to U.S. provisional patent application No.61/924,345, filed on 7/1/2014, the entire contents of which are hereby incorporated by reference.

Technical Field

The present invention relates generally to high dynamic range digital images. The present invention relates in particular to methods and devices for encoding and decoding high dynamic range images, whether still pictures or moving pictures, and data structures containing digital high dynamic range images.

Background

Human vision can realize contrast ratios as high as 1:10,000. That is, a person can see the following scene and see details in both the brightest and darkest parts of the scene: in this scene, some parts of the scene are 10,000 times brighter than other parts of the scene. Furthermore, human vision can adapt its sensitivity to brighter or darker scenes further exceeding 6 orders of magnitude.

Most conventional digital image formats (so-called 24-bit formats) use up to 24 bits to store color and brightness (luminance) information for each pixel in the image. For example, each of the red, green, and blue (RGB) values of a pixel may be stored in one byte (8 bits). Such a format is capable of representing brightness (brightness) changes only over about 2 orders of magnitude (each byte may store one of 256 possible values). There are several standard formats for representing digital images, including both still images and video images. These include JPEG (joint photographic experts group), MPEG (moving picture experts group), AVI (audio video interleave), TIFF (tagged image file format), BMP (bitmap), PNG (portable network graphics), GIF (graphics interchange format), and others. Such formats may be referred to as "output reference standards" because they do not attempt to retain image information beyond that which can be reproduced by the most commonly available types of electronic displays. Until recently, displays such as computer displays, televisions, digital motion image projectors, etc. have been unable to accurately reproduce images with contrast ratios better than about 1: 1000.

Display technologies and others being developed by the assignee are capable of rendering images with High Dynamic Range (HDR). Such displays can reproduce images that represent real-world scenes more faithfully than conventional displays. There is a need for a format for storing HDR images for rendering on these displays, as well as other HDR displays that will become available in the future.

Several formats have been proposed for storing HDR images as digital data. These formats have various disadvantages. Several of these formats produce oversized image files that can only be viewed using specialized software. Some manufacturers of digital cameras provide proprietary RAW formats. These formats tend to be camera specific and are excessive in data storage requirements.

A convenient framework for storing, swapping, and rendering high dynamic range images is needed. There is a particular need for such a framework that is backward compatible with existing image viewer technology.

Drawings

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 illustrates an exemplary decoding process according to an embodiment of the present invention;

FIG. 2 illustrates an exemplary decoding process according to another embodiment of the present invention;

FIG. 3 illustrates exemplary data contained in an APP11 header (segment) in accordance with an embodiment of the present invention;

4A-4B illustrate exemplary segments of a residual ratio image (residual ratio image);

FIG. 5 illustrates an example hardware platform on which a computer or computing device described herein may be implemented.

Detailed Description

Example possible embodiments related to HDR encoding, decoding, and data structures are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in detail so as not to unnecessarily obscure, or obscure the present invention. That is, U.S. patent No.8,514,934 entitled "Apparatus and methods for encoding, decoding, and decoding high dynamic range images" is incorporated herein by reference for all purposes.

According to one embodiment of the invention, an HDR data structure is configured to be readable by legacy (legacy) image viewers. Legacy image viewers can read tone map (tone map) information and ignore HDR information, such as ratio data (explained later). In some embodiments, the data structure comprises a JFIF file and the tone map information comprises a JPEG image. In some embodiments, the data structure comprises an MPEG file and the tone map information comprises frames of MPEG video.

Another aspect of the invention provides a data structure for representing a high dynamic range image having an initial dynamic range. The data structure includes a tone map portion and a high dynamic range information portion. The tone map portion contains tone map information representing an image and has a dynamic range smaller than the initial dynamic range. The high dynamic range information portion contains information describing a ratio of (luminance) values in the tone map portion to luminance values of the high dynamic range image.

Residual ratio image

One aspect of the present invention provides a method for encoding high dynamic range image data. The method involves obtaining or otherwise generating tone map information corresponding to high dynamic range image data. The tone map information has a dynamic range lower than that of the high dynamic range image data. The method calculates ratio data comprising a ratio of a value in the high dynamic range image data to a corresponding value in the tone map information. The ratio data (or information derived therefrom) and the tone map information may be stored and transmitted for decoding.

Another aspect of the invention provides a method for decoding a codestream to reconstruct a high dynamic range image. The method involves receiving or otherwise accessing tone map information and corresponding ratio data (or information derived therefrom). The method includes calculating a high dynamic range image using values in tone map information and corresponding ratio data.

Ratio data as referred to throughout this application may be calculated as, but is not limited to, (i) a mathematical division of the numerator value by the denominator value, including but not limited to further mathematical operations such as the logarithm of the ratio, or (ii) alternatively, a subtraction of two logarithmic values, including but not limited to further mathematical operations. Typically, the ratio data describes luminance, but may also be used for chrominance channels (e.g., Cr, Cb). For clarity, the ratio data is sometimes described herein as or included with the residual data.

FIG. 1 illustrates an exemplary decoding process according to an embodiment of the present invention. The process starts with a legacy decoder block that reconstructs the base image (base image). The image is then optionally chroma upsampled, followed by an inverse decorrelation block. The output of this transformation is a low dynamic range, backward compatible image of, for example, eight bits per sample in an RGB type color space.

The low dynamic range component is further mapped to a floating point image, referred to as a predecessor (presensor) image, by a base mapping and color space conversion block. The precursor image is optionally converted to the HDR color space, and luminance may be calculated. The noise level can be used to avoid nulls and to reduce compression artifacts (artifacts) that can be amplified in subsequent blocks.

The residual decoder path uses residual data embedded in the codestream in the APP11 mark (marker). The data is reconstructed and then optionally up-sampled. It is then processed by the residual mapping and inverse decorrelation block. This block maps the residual data to a floating-point domain, which is optionally inverse decorrelated. The mapping may use the luminance calculated by the base mapping and color space conversion block. The mapped residual data and the precursor image are processed by an HDR reconstruction block to generate a reconstructed HDR image.

Fig. 2 illustrates an exemplary decoding process according to another embodiment of the present invention. The decoding process relies on a layered approach that decomposes the HDR image into a base layer and an HDR residual ratio layer. The base layer is a tone-mapped image tone-mapped from the original floating point HDR with a local or global tone mapper. The code stream will be backward compatible with legacy decoders and accessible by legacy decoders. The residual ratio layer contains the HDR quantized log luminance ratio and the chrominance residual, which are put together and represented as a single residual ratio image.

Since the residual data is hidden in the APP11 flag, legacy decoders can skip the residual picture and access only the base picture code stream, so the decoding process is backward compatible. However, a decoder implementing the present invention may combine these two layers to reconstruct an HDR image.

In fig. 2, the upper path, including blocks B1, B2, and B3, may be a standard flow for legacy decoders and outputs backward compatible Lower Dynamic Range (LDR) images in the typical sRGB space. This base image data is then mapped into the linear HDR space in block B4 and processed by a color space conversion operation. This block converts the LDR image into the color space of the original HDR image, and it also maps this image to floating point values, called linear pre (pre) RGB2, which may also be called "LP _ RGB 2". Noise floor (floor) values specified in the parameter code stream are added to the luminance component of LP _ RGB2 to avoid dividing by 0 and to avoid amplifying any noise that may occur due to operations on small values downstream of this block B4.

In FIG. 2, the lower path from B5 begins with the residual data of the high dynamic range image and is represented in ISO/IEC 10918-1 codestream format (which is incorporated by reference for all purposes and shows the desired format). The codestream is embedded in the APP11 tag as a residual data segment as described below. After being decoded by the decoder, the chroma upsampling step is performed by B6 to bring all components to full resolution, e.g., 4:4: 4.

The residual ratio data is then split by B7 into a floating point linear ratio luminance value and a linear residual color difference value. The incoming residual luminance values are inverse quantized according to parameters in the code stream. In particular embodiments, this is provided by an explicit look-up table in the parameter section of the codestream. If the table does not exist, min and max (referred to in the parameter section as ln1, ln0) are used and the inverse log map is computed. Similarly, incoming chroma residual sample values are inverse quantized according to the minimum and maximum parameters stored as cb0, cb1, and cr0, cr1 (if present) in the parameter section of the codestream.

The chroma values are then processed by a B8, YCbCr to RGB2 block, and the linearly dequantized YCbCr is converted to a linear residual RGB2 in the HDR color space, which linear residual RGB2 may alternatively be referred to as "LR _ RGB 2". Finally, blocks B9 and B10 reconstruct the HDR image by first adding linear pre-RGB 2 to linear residual RGB2 in B9 and then multiplying the result by linear ratio luminance in B10.

APP11 marker

As shown in fig. 3, the APP11 marker segment is divided into a parameter data segment and a data segment. The parameter section has two or more (e.g., 3) types of sections, such as a parameter ASCII type section, a residual section, and a parameter binary type section. This structure for the APP11 marker segment may be used in conjunction with any of the embodiments of the invention described herein, including but not limited to the exemplary embodiments reflected in fig. 1 and 2.

Checksum for edit (edit) detection

The Parameter Data Section (PDS) carries parameters encoded as ASCII or binary text, payload data. The last parameter in this segment is the checksum of the base layer code stream. In particular embodiments, the ckb (ASCII) or chksum (binary, 16-bit) parameter is a checksum of the base layer code stream, which is calculated by summing all bytes in the base layer code stream. The checksum includes the first SOF (e.g., start of frame) marker following the last APP11 marker segment and includes all subsequent bytes up to and including the EOI (e.g., end of frame) marker. It can be used by the decoder to detect edits of the base layer that may cause undesirable artifacts when High Dynamic Range (HDR) images are decoded. In particular embodiments, the checksum is location (or order) dependent, such as a Fletcher checksum (e.g., Fletcher-16, Fletcher-32, Fletcher-64). See Fletcher, J.G (month 1 1982). For additional information see "An ArithmeticChecksum for Serial Transmissions," IEEE Transactions on communications, COM-30(1): 247-.

In an alternative embodiment, the PDS may indicate the use of a more complex hash algorithm than a checksum. More complex hashing algorithms reduce the likelihood of hash collisions (e.g., undetectable changes in data) when different input data results in the same hash value. Thus, if the base layer is altered, the hash values generated for the original base layer should be unlikely to match probabilistically. An exemplary hash function may be or may be implemented by:

(i) a non-linear look-up table;

(ii) cryptographic hash functions (e.g., HAIFA, Merkle-Unique block iterations, etc.);

(iii) non-cryptographic hash functions (exclusive or, product, addition, rotation);

(iv) selecting a randomization of a hash function among a predefined set;

(v) cyclic redundancy check; and

(vi) checksum-e.g., Fletcher, Adler-32.

In still other alternative embodiments, fingerprinting or media watermarking techniques may be signaled by the PDS and verified during decoding or image rendering/rendering.

Checksums, hash functions, or other described alternatives for base layer edit detection may be used in conjunction with any of the embodiments of the invention described herein, including but not limited to the exemplary embodiments reflected in fig. 1 and 2. Additionally, checksums, hash functions, or alternatives may also be used for edit detection of the residual ratio layer based on the teachings herein.

Encryption/decryption of residual layer based on per-segment implementation

Another parameter within the PDS or elsewhere may be an encryption parameter, such as an encryption key. This information may be used, for example, to decrypt the rate residual layer on a per segment basis of the codestream. A segment may be an independently decodable sequence of entropy encoded bytes of compressed image data. In other words, according to embodiments of the present invention, different encryption parameters may be provided and used for each segment. The encryption parameters and associated processing may be used in conjunction with any of the embodiments of the invention described herein, including but not limited to the exemplary embodiments reflected in fig. 1 and 2.

Inverse tone mapping in DEGAMMA (DEGAMMA) LUT/mapping LUT

The degamma look-up table (LUT) described above (as block B4 in FIG. 2) is via a default Rec.601 table (at block B4)http://www.itu.int/rec/R-REC-BT.601-7-201103-I/enAvailable ITU-R Recommendation bt.601, incorporated by reference) a loaded 256 entry table, the rec.601 table is typically an inverse linear power function of 2.4. If it is in an alternate color space (such as Adobe RGB from Adobe Systems, Inc.), a look-up table may be sent in the header information. In addition, the degamma LUT may include inverse tone mapping functions/curves such as for inverse histogram equalization or inverse Reinhard tone mapper. In some cases, a degamma LUT with inverse tone mapping may reduce memory for residual ratio layers. For additional information on Reinhard tone mapper, seehttp://www.cs.utah.edu/～reinhard/cdrom/tonemap.pdf(“Photographic Tone Reproduction for DigitalImages "), which is incorporated herein by reference for all purposes.

Binary header

The APP11 marker segment may include binary parameter data as shown as "type 3" in fig. 3. This type 3 segment and its associated processing may be used in conjunction with any of the embodiments of the invention described herein, including but not limited to the exemplary embodiments reflected in fig. 1 and 2.

Segment index and start position for the segment

In embodiments of the present invention, the span (span) and extent (extent) of the segments for the residual ratio image need to be consistent with the base layer image. For example, the residual ratio image may be divided into a plurality of continuous and discontinuous segments. The set of segments of the residual ratio image need not correspond to the complete image, but may define one or more portions of the image. This functionality allows HDR reconstruction from a portion of the base layer image, rather than the entire base layer image. For example, encryption parameters may be provided for one segment (e.g., left half image, top half image) for HDR reconstruction, while the residual ratio information for another segment (e.g., right half image, bottom half image) remains encrypted for limited base layer rendering.

Each segment of the residual ratio image may be specified by a coordinate reference (e.g., an x-coordinate and a y-coordinate of one of the four corners, if a rectangular segment) and its length and width. If the segment is of a different geometry, it may be defined by a central location and a radius/diameter, etc. Fig. 4A-4B illustrate exemplary segments of residual ratio images that may be used in conjunction with any embodiment of the present invention, including, but not limited to, the exemplary embodiments reflected in fig. 1 and 2.

Implementation mechanisms-hardware overview

According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special purpose computing device may be hardwired to perform the techniques, or may include a digital electronic device such as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such special purpose computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement these techniques. A special-purpose computing device may be a desktop computer system, portable computer system, handheld device, networked device, or any other device that incorporates hardwired logic and/or program logic to implement the techniques.

For example, FIG. 5 is a block diagram that illustrates a computer system 1600 upon which an embodiment of the invention may be implemented. Computer system 1600 includes a bus 1602 or other communication mechanism for communicating information, and a hardware processor 1604 coupled with bus 1602 for processing information. The hardware processor 1604 may be, for example, a general purpose microprocessor.

Computer system 1600 also includes a main memory 1606, such as a Random Access Memory (RAM) or other dynamic storage device, coupled to bus 1602 for storing information and instructions to be executed by processor 1604. Main memory 1606 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1604. Such instructions, when stored in a non-transitory storage medium accessible to processor 1604, cause computer system 1600 to become a special-purpose machine customized to perform the operations specified in the instructions.

Computer system 1600 also includes a Read Only Memory (ROM)1608 or other static storage device coupled to bus 1602 for storing static information and instructions for processor 1604. A storage device 1610, such as a magnetic disk or optical disk, is provided and coupled to bus 1602 for storing information and instructions.

Computer system 1600 can be coupled via bus 1602 to a display 1612, such as a liquid crystal display, for displaying information to a computer user. An input device 1614, including alphanumeric and other keys, is coupled to bus 1602 for communicating information and command selections to processor 1604. Another type of user input device is cursor control 1616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 1604 and for controlling cursor movement on display 1612. The input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to specify positions in a plane.

Computer system 1600 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and/or program logic that in combination with the computer system make computer system 1600 a special-purpose machine or program computer system 1600 a special-purpose machine. According to one embodiment, the techniques described herein are performed by computer system 1600 in response to processor 1604 executing one or more sequences of one or more instructions contained in main memory 1606. Such instructions may be read into main memory 1606 from another storage medium, such as storage device 1610. Execution of the sequences of instructions contained in main memory 1606 causes processor 1604 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

The term "storage medium" as used herein refers to any non-transitory medium that stores instructions and/or data that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 1610. Volatile media includes dynamic memory, such as main memory 1606. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

A storage medium is different from, but may be used in conjunction with, a transmission medium. Transmission media participate in the transfer of information between storage media. Transmission media includes, for example, coaxial cables, copper wire and fiber optics, including the wires that comprise bus 1602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 1604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 1600 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 1602. The bus 1602 carries the data to the main memory 1606, from which the processor 1604 retrieves and executes the instructions. The instructions received by main memory 1606 may optionally be stored on storage device 1610 either before or after execution by processor 1604.

Computer system 1600 also includes a communication interface 1618 coupled to bus 1602. Communication interface 1618 provides a two-way data communication coupling to network link 1620, network link 1620 being connected to local network 1622. For example, communication interface 1618 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 1618 may be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface 1618 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Network link 1620 typically provides data communication through one or more networks to other data devices. For example, network link 1620 may provide a connection through local network 1622 to a host computer 1624 or to data equipment operated by an Internet Service Provider (ISP) 1626. ISP 1626 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 1628. Local network 1622 and internet 1628 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 1620 through communication interface 1618, which carry the digital data to computer system 1600 and from computer system 1600, are example forms of transmission media.

Computer system 1600 can send messages and receive data, including program code, through the network(s), network link 1620 and communication interface 1618. In the Internet example, a server 1630 might transmit a requested code for an application program through Internet 1628, ISP 1626, local network 1622 and communication interface 1618.

The received code may be executed by processor 1604 as it is received, and/or stored in storage device 1610, or other non-volatile storage for later execution.

Equivalents, extensions, substitutions and others

In the foregoing specification, possible embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Additional references

In addition to the references cited above, the following references are also incorporated herein by reference for all purposes:

(i)ITU-T Rec.T.81|ISO/IEC 10918-1:Information Technology–Digital Compression and Coding of Continuous Tone Still Images–Requirements and Guidelines

(ii)ITU-T Rec.T.86|ISO/IEC 10918-4:Information technology--Digital compression and coding of continuous-tone still images:Registration of JPEG profiles,SPIFF profiles,SPIFF tags,SPIFFcolour spaces,APPn markers,SPIFF compression types,andRegistration Authorities

(iii)ITU-T Rec.T.871|ISO/IEC 10918-5:Informationtechnology--Digital compression and coding of continuous-tone stillimages:JPEG File Interchange Format

(iv) ITU-T Rec.T.801| ISO/IEC 15444-1: informationchnocology-JPEG 2000Image Coding System; and

(v)IEC 60559Binary floating-point arithmetic formicroprocessor systems。

Claims (9)

1. A method for decoding, the method comprising:

receiving base layer data for a High Dynamic Range (HDR) image;

receiving, by a decoder, a first checksum parameter;

receiving residual ratio data for the HDR image;

calculating, by the decoder, a second checksum parameter of the base layer data, the second checksum parameter being calculated based on the start marker of the first frame following the last APP11 marker segment and including all subsequent bytes up to and including the end marker of the picture; and

comparing the first checksum parameter to a second checksum parameter.

2. The method of claim 1, further comprising:

receiving encryption parameters in the APP11 token segment; and

and decrypting the residual ratio data by using the encryption parameter.

3. The method of claim 2, wherein decrypting the residual ratio data using encryption parameters is performed on a per segment basis.

4. A method for encoding, the method comprising:

receiving a High Dynamic Range (HDR) image;

determining base layer data and residual ratio data for the High Dynamic Range (HDR) image;

calculating, by an encoder, checksum parameters of the base layer data, the checksum parameters calculated based on a start marker of a first frame following a last APP11 marker segment and including all subsequent bytes up to and including an end marker of a picture; and

and storing the checksum parameter.

5. The method of claim 4, further comprising storing encryption parameters in the APP11 tag segment for decryption of the residual ratio data.

6. A decoder apparatus comprising a processor implementing any one of the methods of claims 1 to 3.

7. An encoder apparatus comprising a processor implementing any one of the methods of claims 4 to 5.

8. A computer program product stored on a non-transitory computer readable medium for encoding an HDR picture, the computer program being operable with an image system comprising at least an encoder, the computer program comprising instructions for causing a computer to implement any of the methods of claims 4 to 5.

9. A computer program product stored on a non-transitory computer readable medium for decoding HDR pictures, the computer program being operable with an image system comprising at least a decoder, the computer program comprising instructions for causing a computer to implement any of the methods of claims 1 to 3.