HK1114987B - Film grain generation and addition - Google Patents

Description

Film grain generation and addition

Technical Field

The present invention relates generally to image and video processing and, more particularly, to film grain generation in image signals.

Background

The amount of storage required to store uncompressed 90 minute long motion picture feature film (i.e., motion picture) is typically around 90 gigabits. However, DVD media typically have a storage capacity of 4.7 gigabits. Therefore, storing a complete movie on a single DVD requires a compression rate of about 20: 1. The data is further compressed in order to accommodate the audio data on the same storage medium. A relatively high compression rate can be achieved, for example, by using the MPEG2 compression standard. However, compression artifacts (artifacts) such as "blockiness" and "mosquito noise" are often seen when decoding and playing back a movie.

The appreciation of the present invention and its improvements can be obtained by reference to the accompanying drawings, which are briefly summarized below, to the following detailed description of illustrated embodiments of the invention, and to the appended claims.

Disclosure of Invention

According to a first aspect of the present invention, there is provided a method of providing film grain for a digital image frame, comprising: receiving pixels of an incoming video data stream associated with the digital image frame; using a high pass filter to generate a mask of noise values independent of the incoming video data stream, wherein the mask represents film grain; selecting an entry from the mask of noise values; combining the selected entry with the received pixel to add film grain to the digital image frame to produce a grain-perceived digital image frame.

According to a second aspect of the present invention, there is provided a system for providing film grain for digital image frames, comprising: an input configured to receive pixels of an incoming video data stream associated with the digital image frame; a high pass filter configured to generate a mask of noise values independent of said incoming video data stream, wherein said mask represents film grain; a mask memory configured to store the noise value mask representative of film grain; circuitry configured to select an entry from the mask memory; and circuitry configured to combine the selected entries with the received pixels to add film grain to the digital image frames to produce a grain-sensed digital image frame.

According to a third aspect of the present invention, there is provided a system for providing film grain for a digital image frame, comprising: means for receiving pixels of an incoming video data stream associated with the digital image frame; means for generating a noise value mask independent of the incoming video data stream using a high pass filter, wherein the mask represents film grain; means for selecting an entry from the noise value mask; and means for combining the selected entry with the received pixel to add film grain to the digital image frame to produce a grain-sensed digital image frame.

Drawings

Fig. 1 is a block diagram of an example film grain generation system.

Fig. 2 is a block diagram of an example film grain generator and mechanism for adding film grain data to incoming video data.

Fig. 3 is a flow chart illustrating example processing steps for mask data generation by the film grain generator.

Fig. 4 is a graph illustrating the frequency response of an example high pass filter used to filter pseudo random data to achieve film grain generation.

Fig. 5 is a simplified block diagram illustrating a film grain generator in an example video processing circuit implemented with an integrated circuit.

Detailed Description

In the following detailed description of exemplary embodiments of the invention, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific exemplary embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that other changes may be made without departing from the spirit or scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The meaning of "a", "an" and "the" includes reference to a plurality, and the meaning of "in. The term "coupled" means a direct electrical connection between the items coupled, without any intervening devices. The term "coupled" means either a direct electrical connection between the items connected, or an indirect connection via one or more passive or active intermediary devices. The term "circuit" means a single component or a plurality of components, active and/or passive, coupled together to provide a desired function. The term "signal" means at least one current, voltage or data signal. Like reference numerals refer to like parts throughout the several views of the drawings. The present invention may be embodied in a system such as that described in U.S. patent application 11/296,185 entitled Color Management Unit, filed on 7.12.2005, which is incorporated herein by reference.

The present invention is directed to generating high frequency noise for digital video signals that approximates the traditional "film grain" look. By adding a relatively small amount of film grain noise, the video can be made to appear more natural and more pleasing to human viewers. Digital film grain generation can be used to mask unnatural smoothing artifacts such as "blocking" and "contouring" in digital video in the case of compressed video and/or to provide visual enhancement or trick effects to any digital video stream. The digital film grain generator is capable of controlling the grain size and the amount of film grain to be added.

Many cinematographers have this opinion: film can produce a more aesthetically pleasing look and feel than digital video, even when very high resolution digital sensors are used. The "film look" is sometimes described as being more "subtle" than the comparative "hard" look of digital video. This aesthetically pleasing attribute of the film is caused, at least in part, by the random occurrence of continuously moving high frequency film grain (as compared to the fixed pixel grid of the digital sensor). Thus, the generation of high frequency noise with the visual attribute of "film grain" can be used to mask unnatural smoothing artifacts like "blocking" and "contouring" in the case of compressed video. Moreover, simply adding film-granular high frequency noise can provide visual enhancement or special effects to any digital video stream.

Certain in-loop and post-processing algorithms may be used to reduce "blocking" and "mosquito noise". In the processes of compression, decompression and artifact removal, the video signal often loses its natural looking appearance and instead may get a "patchy" appearance. By adding a relatively small amount of film grain noise, the audio film can be made to appear more natural and more pleasing to human viewers. The addition of film grain noise can also provide a slightly textured look to the regions of the image that appear to be stitching-like.

Fig. 1 is a block diagram of an example film grain generation system. Film grain generation system 100 includes a digital video signal source (110), an optional MPEG blocking and mosquito noise reducer (120), an optional multiplexer (130), a film grain generator (140), and a final display device (150). The digital video signal source may be an MPEG decoder for decoding MPEG encoded data on a DVD. The decoded stream (in digital format) is fed to an MPEG blocking and mosquito noise reducer 120. The MPEG noise reducer is provided to reduce blocking artifacts and mosquito noise artifacts present in the decoded stream. The noise reduced data is then fed to a film grain generator (140). The bypass path indicates that film grain data can also be added directly to the MPEG decoded data. In the bypass mode of operation, the film grain data can still be added to the decoded stream, which can also serve to create a masking of blocking artifacts.

In one embodiment, film grain noise is added to the luminance data path while the luminance data path is also MPEG noise reduced. To compensate for the additional processing delay, the chrominance data path may be delay equalized accordingly (to keep, for example, luminance and chrominance information synchronized). In an alternative embodiment, film grain generation can also be added to the chrominance path. As shown, the input to the film grain generator 140 can thus include luminance and/or chrominance information.

To help control the extent of film grain generation (e.g., to optimize the aesthetic quality of any given digital video stream), the attributes of the film grain (e.g., the size of the grain and the amount of film grain that needs to be added) can be controlled.

Fig. 2 is a block diagram of an example film grain generator and an example adder for adding film grain data to incoming Y data (e.g., luminance data). The film grain generator 200 includes an address generator 210, a memory 220, a sealer 230, rounding logic 240, a summing node 250, and saturation logic 260. In various embodiments, incoming data to the film grain addition module can come from the MPEG noise reducer (120) or directly input (110) (such as from an MPEG decoder integrated within the same system or remotely located). The film grain data can be initially written to the memory 220 by the host processor. (the generation of film grain data will be discussed below with respect to fig. 3.) the memory 220 can be an on-chip static RAM so that the customer can supply user-specific film grain entries and have the flexibility to change these entries at a later time.

The address generator 210 accesses the memory 220 to provide the film grain data stored in the static RAM. The film grain data can be derived using a 2D (two-dimensional) data mask of size 32 x 32. The 2D mask data in memory may be stored in a 1D (one-dimensional) format by mapping each incoming pixel map to a unique address in memory and computing the address by running horizontal and vertical counters. The counter may cycle in response to the control signals (hde and vde). The formula for calculating the address of the memory 220 is given by the following equation (1):

sram_address＝32×(vcount modulo 32)+(hcount modulo 32)(1)

the corresponding entry is selected from memory in response to the pixel location and added to the incoming pixel value at summing node 250 after being appropriately scaled by sealer 230. The film grain entries in the static RAM can be either positive or negative and can vary from-127 to 127.

In addition to the film grain data from memory (providing information about the size of the film grain), a mechanism is used to control how much film grain noise needs to be added. To achieve this control, a multiplier (e.g., scaler 230) is used having film grain data as a first input and a scaling factor as a second input, the scaling factor being programmable via a register. The bit precision of the scaling factor is, for example, "1.7", so that the scaling factor comprises one integer bit and seven decimal bits. After scaling the film grain data, the output of the multiplier has a 9.7 bit format. The 9.7 bit format is converted to an 8.0 bit format by restriction (bounding) logic 240.

At summing node 250, the scaled and limited entry is added to the incoming pixel. By adding controlled film grain noise (which can be positive and negative) to the incoming Y data (8-bit precision in this example), care is taken that overflow and underflow of data does not occur. Saturation (or clipping clamp) logic 260 may be used to minimize unwanted data overflow and underflow problems.

The generation of the film grain data or mask is typically processed off-line. The generation may be implemented using a commonly available mathematical program such as "Matlab". In order to design the filter coefficients of a two-dimensional high-pass filter (hpf), some functions are used in this example, which are available in the mathematical program Matlab.

FIG. 3 is a flow diagram illustrating example processing steps for mask data generation. In step 310, a one-dimensional filter is designed based on frequency specifications. The one-dimensional filter is transformed into a two-dimensional set of coefficients using a function called "ftrans 2". Specific requirements for designing an example high pass filter are given below in table 1. The user can change the passband frequency specification to achieve different film grain sizes. In this example, the film grain size is larger if the spatial sampling frequency is smaller (the film grain size is smaller if the spatial sampling frequency is larger).

TABLE 1

Parameter(s)	Specific requirements	Unit of
			Spatial sampling frequency (Fs)	512	Number of cycles/width
Stop band frequency (fs)	0.1*Fs	Number of cycles/width
			Passband frequency (fp)	0.5*Fs	Number of cycles/width
Passband ripple (Ap)	0.01	Decibel
			Stopband attenuation (As)	60	Decibel

The one-dimensional filter is sixth order and has seven coefficients in total, according to the frequency specifications listed in table 1. The (Matlab) function "ftrans 2" is used to transform the one-dimensional filter into a two-dimensional circularly symmetric filter. The set of coefficients for the two-dimensional circularly symmetric filter is listed below in table 2.

TABLE 2

	Hc0	Hc1	Hc2	Hc3	Hc4	Hc5	Hc6
								Vc0	-0.0003	-0.0017	-0.0042	-0.0057	-0.0042	-0.0017	-0.0003
Vc1	-0.0017	0.0027	0.0263	0.0437	0.0263	0.0027	-0.0017
								Vc2	-0.0042	0.0263	-0.0537	-0.1685	-0.0537	0.0263	-0.0042
Vc3	-0.0057	0.0437	-0.1685	0.5642	-0.1685	0.0437	-0.0057
								Vc4	-0.0042	0.0263	0.0537	-0.1685	-0.0537	0.0263	-00042

	Hc0	Hc1	Hc2	Hc3	Hc4	Hc5	Hc6
								Vc5	-0.0017	0.0027	0.0263	0.0437	0.0263	0.0027	-0.0017
Vc6	-0.0003	-0.0017	-0.0042	-0.0057	-0.0042	-0.0017	-0.0003

In step 320, a two-dimensional matrix of pseudo-random (or "random") data of 512 words by 512 words is generated. In step 330, the data is applied to a two-dimensional filter (the two-dimensional filter determined in step 310). The two-dimensional filter applies a high-pass shaping function to the incoming data, as will be described below in fig. 4.

Fig. 4 is a graph illustrating the frequency response (amplitude response) of an example high pass filter. Graph 400 illustrates a filter that is circularly symmetric about an origin and provides a high pass shaping function to incoming data. Pseudo-random data with a flat spectrum is thus modified according to the high-pass filter transfer function.

Referring again to fig. 3, in step 340, a mask for a 32-word by 32-word data entry is selected from the 512-word by 512-word filtered data set. The data in the middle of the data set is chosen to minimize boundary errors in the filtered data. In step 350, the selected film grain data mask is stored in, for example, a 1024 x 8 bit static RAM so that the film grain data stored in the static RAM can be later retrieved for combination with the image data stream.

Fig. 5 is a simplified block diagram illustrating a film grain generator in an example video processing circuit implemented with an integrated circuit. For clarity and conciseness, the modules in the figures are illustrated as having more than one function within a single module. (in various embodiments, these functions may be implemented using a single module, or even distributed among several modules.) furthermore, various modules in the figure have bypass functionality so that functions within the module can be bypassed so that processing that is not desired for the application can be bypassed.

Module 510 allows selection of a video data stream from between, for example, standard definition video (NTSC or PAL) and high definition video. The module 520 implements color space conversion and is capable of downsampling a video input stream. The module 520 can also provide MPEG noise reduction as needed. The module 530 may be used to provide three-dimensional gaussian noise reduction and may also be used to de-interleave image frames. The module 540 may be used to scale image frames within a video signal and provide frame rate conversion. The module 550 may be used to perform edge enhancement in the video signal while also performing the upsampling task.

Film grain generation can occur in block 560 such that film grain is added to the luminance portion of the video signal being processed. Module 570 provides color processing functions such as color enhancement and maintenance. The module 580 may be used to dither output video data that is sent to, for example, a display that has a lower resolution than the output video data. Module 590 can be used to interleave frames associated with output video data, if desired.

Various embodiments of the invention are possible without departing from the spirit and scope of the invention. The foregoing specification, examples and data provide a complete description of the manufacture and use of the composition of the invention. For example, the architecture may be implemented using hardware, software, or a combination of both. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.

Claims (20)

1. A method of providing film grain for a digital image frame, comprising:

receiving pixels of an incoming video data stream associated with the digital image frame;

using a high pass filter to generate a mask of noise values independent of the incoming video data stream, wherein the mask represents film grain;

selecting an entry from the noise value mask;

combining the selected entry with the received pixel to add film grain to the digital image frame to produce a grain-perceived digital image frame.

2. The method of claim 1, wherein the selected entry is combined with a luminance signal associated with the received pixel.

3. The method of claim 1, wherein the incoming video data stream contains image information that has been decompressed.

4. The method of claim 1, wherein the high pass filter is a two-dimensional filter.

5. The method of claim 4 wherein the size of the film grain is controlled by changing the coefficients of said two-dimensional filter.

6. The method of claim 1 wherein the amount of film grain to be incorporated is selected in response to a scaling factor.

7. The method of claim 1, wherein the selected entry and the received pixel taken together comprise values bounded by a minimum threshold and a maximum threshold.

8. The method of claim 1, wherein a counter is used to determine an address associated with the location of the pixel.

9. A system for providing film grain for digital image frames, comprising:

an input configured to receive pixels of an incoming video data stream associated with the digital image frame;

a high pass filter configured to generate a mask of noise values independent of said incoming video data stream, wherein said mask represents film grain;

a mask memory configured to store the noise value mask representative of film grain;

circuitry configured to select an entry from the mask memory; and

circuitry configured to combine the selected entries with the received pixels to add film grain to the digital image frame to produce a grain-sensed digital image frame.

10. The system of claim 9, wherein the selected entry is combined with a luminance signal associated with the received pixel.

11. The system of claim 9, wherein the incoming video data stream contains image information that has been decompressed.

12. The system of claim 9, wherein the high pass filter is a two-dimensional filter.

13. The system of claim 12, wherein the size of the film grain is controlled by changing coefficients of said two-dimensional filter.

14. The system of claim 9 wherein the amount of film grain to be incorporated is selected in response to a scaling factor.

15. The system of claim 9, wherein the selected entry and the received pixel taken together comprise values bounded by a minimum threshold and a maximum threshold.

16. The system of claim 9, wherein the address associated with the pixel location is determined using a counter.

17. A system for providing film grain for digital image frames, comprising:

means for receiving pixels of an incoming video data stream associated with the digital image frame;

means for generating a noise value mask independent of the incoming video data stream using a high pass filter, wherein the mask represents film grain;

means for selecting an entry from the noise value mask; and

means for combining the selected entry with the received pixel to add film grain to the digital image frame to produce a grain-sensed digital image frame.

18. The system of claim 17, wherein the selected entry is combined with a luminance signal associated with the received pixel.

19. The system of claim 17, wherein the incoming video data stream contains image information that has been decompressed.

20. The system of claim 17, wherein the high pass filter is a two-dimensional filter.