HK1170614A

HK1170614A - A method for quality-aware video optimization and a video optimizer

Info

Publication number: HK1170614A
Application number: HK12111166.1A
Authority: HK
Inventors: Miguel A. Melnyk; Andrew Penner; Jeremy Tidemann
Original assignee: Citrix Systems, Inc.
Priority date: 2009-03-31
Filing date: 2010-03-31
Publication date: 2013-03-01

Description

Framework for quality-aware video optimization

Cross reference to related patent

The present application claims priority and benefit from U.S. provisional application No. 61/165,224 entitled "a frame for Quality-Aware Video Optimization" filed 3, 31, 2009, the entire contents of which are incorporated herein by reference.

Background

Quantification versus perceptual Quality (perceptual Quality)

In video processing, quantization is a lossy compression technique implemented by compressing a range of values into a single quantum value. When video frames are quantized in any system, information is lost. For example, a typical video encoder (such as MPEG-2) may compress video frames by discarding information that does not contribute to the reconstruction of the image representing the original image during decoding. The amount of information discarded during encoding depends on how each video frame is quantized. Each video compression format defines a discrete set of quantization settings, and each quantization setting has an abstract identifier, denoted as Quantization Parameter (QP). The QP may be arbitrarily defined as, for example, an integer that indexes an array of quantization settings such that the quantization noise introduced by a smaller QP value X is less than the quantization noise introduced by a larger QP value X + 1. The quantization settings indexed by a given QP value may be different for each video codec.

If too much information is discarded during quantization, the video frame may exhibit distortion when decompressed during playback. This captures the relationship between quantization and perceptual quality. Thus, the QP may be used as an indicator of perceptual quality, since the QP indicates how much information is discarded when encoding a video frame.

To illustrate that QP is only a heuristic algorithm for estimating the perceived quality of a video frame, one should consider a "low quality" video frame quantized with a fine quantization setting. If the video frame is encoded with a fine quantization setting (which means that very little information is discarded), the video frame reconstructed by the decoder will match the original video frame very closely. The quantization noise in the reconstructed video frame is very low, but the reconstructed video frame still appears to be of low quality, since the original video frame is of low quality. This is only one example where QP does not accurately indicate the perceptual quality of the video.

Quantization versus bit rate

In video processing, the bit rate refers to the number of bits used per playback time unit for representing a continuous video after encoding (data compression). Of course, different images or video frames may require different numbers of bits to represent even when they share the same pixel size and are encoded with the same QP. However, for a given video frame, the relationship between QP and compressed byte size is more predictable. If a higher QP value represents a coarser quantization (more information loss), a frame quantized with a higher QP value of X +1 will never require more bits than the same frame quantized with a smaller QP value of X (which ignores the corner cases in the entropy coding schemes typically applied to quantize coefficient data). In practice, this means: when the QP used to quantize a frame is increased, the average bit rate requirement of the frame decreases. Fig. 3 shows how the bit rate of a video sequence decreases with increasing QP. The number of bit rates and QP values in fig. 3 are examples only. In real life, these numbers and values may be different, and the correlation between QP value and bit rate may vary for different video sequences.

Drawings

FIG. 1 illustrates a block diagram of an exemplary system.

Fig. 2 is a block diagram illustrating an embodiment of an exemplary quality aware video optimizer in the exemplary system of fig. 1.

Fig. 3 illustrates an exemplary correlation between bitrate and QP for various sequences of video frames.

FIG. 4 illustrates slave QP_InputExemplary mapping to Δ QP.

Fig. 5 is a functional diagram illustrating an exemplary process flow in the embodiment of fig. 2.

Fig. 6 is a flow diagram representing an exemplary method of quality-aware video optimization.

Fig. 7 is a block diagram illustrating another embodiment of an exemplary quality aware video optimizer in the exemplary system of fig. 1.

Fig. 8 is a functional diagram illustrating an exemplary process flow in the embodiment of fig. 7.

Fig. 9 is a flow diagram representing another exemplary method of quality-aware video optimization.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

Example embodiments disclosed herein relate to methods and systems for quality-aware video optimization. Quality-aware video optimization is a technique for modifying a sequence of video frames such that the byte size of the video stream is reduced as much as possible while at the same time limiting the perceived quality degradation to a controllable level. Quality-aware video optimization techniques may require only a single pass (single pass) on a previously encoded video frame sequence in order to optimize the video frame sequence. The problems of the existing single-pass rate control technology are as follows: the relationship between the compressed byte size of a video frame and its quantization parameter is known only after encoding the frame. To achieve byte reduction and controllable quality degradation in a single pass, a quality-aware video optimizer may refer to information about how video frames are quantized during initial encoding of the video frames and use this information to optimize a sequence of video frames.

The framework for quality-aware video optimization can adjust image quantization to allow fine control of quality degradation in byte-reduced content through experimental testing. Through further experimental testing, it is also possible to apply quantization adjustment optimization to transcoding scenarios in which the input and output compression formats are different.

Fig. 1 illustrates a block diagram of an exemplary system 100. Exemplary system 100 may be any type of system that provides video content over a local connection or network (e.g., a wireless network, the internet, a broadcast network, etc.). Exemplary system 100 may include, among other things, a user device 102, a gateway 104, one or more networks 106 and 110, a video optimizer 108, and one or more media servers 112 through 114.

User device 102 may be a hardware device such as a computer, PDA, cell phone, laptop computer, desktop computer, VCR, laser disk player, DVD player, blu-ray disk player, television including a broadcast tuner, or any electronic device capable of accessing a data network and/or capable of playing video. The user equipment 102 may include: a software application that allows the user device 102 to communicate with and receive video content from a data network, such as network 106. For example, user device 102 may send request data to a media server to download a particular video streaming file, and the media server may transmit the video streaming file to user device 102. In some embodiments, the video streaming file may be routed through the video optimizer 108. The user device 102 may provide a display for displaying the video stream to a user of the user device and one or more software applications, such as a media player.

In some embodiments, the user device 102 may also be or include a storage device for storing or archiving video content. For example, the storage device may be provided as a video CD, DVD, blu-ray disc, hard disk, magnetic tape, flash memory card/drive, volatile or non-volatile memory, holographic data storage, and any other storage medium.

The gateway 104 is one or more devices that act as portals or access devices and may or may not convert formatted data provided in one type of network into a particular format required by another type of network. For example, the gateway 104 may be a server, a router, a firewall server, a host, or a video optimizer. For example, gateway 104 may have the following capabilities: transforms signals received from the user device 102 into signals that the network 106 or 110 can understand, and vice versa. However, this translation capability is not required in some data networks, such as fourth generation (4G) or Worldwide Interoperability for Microwave Access (WiMAX). Gateway 104 may be capable of handling audio, video, and t.120 transmissions, either individually or in any combination, and capable of full duplex media conversion. In addition, the gateway 104 may include a video optimizer 108 for quality-aware video optimization.

Networks 106 and 110 may comprise any combination of a Wide Area Network (WAN), a Local Area Network (LAN), or a wireless network suitable for packet-type communications, such as internet communications, or a broadcast network suitable for distributing video content.

The video optimizer 108 may be implemented as a software program and/or hardware device that provides quality-aware video optimization. The video optimizer 108 is a server that provides communication between the user equipment 102 and the media servers 112 to 114. The video optimizer 108 may perform a single pass, instantaneous quality-aware optimization of a sequence of previously encoded video frames, as described below. Since the video optimization method requires only one pass (one pass) over the sequence of video frames, the video optimizer 108 is well suited to operate in a wide variety of environments. For example, the video optimizer 108 may be any of the media servers 112-114 or part of any network device between the user device 102 and the media servers 112-114. In this way, the video optimizer 108 optimizes the live video feed before traversing the low capacity network segment. In addition, the video optimizer 108 may also be installed at the front of the storage array to optimize surveillance video prior to archiving it and thereby save storage space.

The media servers 112 to 114 may be the following computer servers: it receives a request for video content from the user device 102, processes the request, and in some embodiments provides the video content to the user device 102 through the video optimizer 108. For example, media servers 112-114 may be web servers, enterprise servers, or any other type of computer server. Media servers 112-114 may be computers programmed to accept requests (e.g., HTTP, RTSP, or other protocols that may initiate video sessions) from user device 102 and provide services to user device 102 using video streams. Further, media servers 112-114 may be PDAs, cellular phones, laptops, desktops, or any device configured to deliver video content to user device 102 over one or more networks 106 and 110 in some embodiments. Further, media servers 112-114 may be broadcast facilities, such as free programs, cables, satellites, and other broadcast facilities, for distributing video content to user equipment 102 through video optimizer 108 in some embodiments. Further, the media servers 112 to 114 may be: a video source, such as a monitoring device, configured to capture video and, in some embodiments, transmit the captured video to the user device 102 through the video optimizer 108. Further, as an alternative to the configuration of the system 100 shown in fig. 1, any of the media servers 112 to 114 may include a video optimizer 108 for quality-aware video optimization.

Fig. 2 is a block diagram illustrating an embodiment of an exemplary quality aware video optimizer 108 in the exemplary system of fig. 1. For simplicity, fig. 2 illustrates only the video optimizer 108, with compressed video frames as input (202) and optimized compressed video frames as output (204). The video optimizer 108 may include, among other things, a decoder 210, a QP adjustment module 220, and an encoder 230. The illustrated configuration of the video optimizer 108 is merely exemplary and may be modified to provide discrete elements or to combine elements into a single element and provided as any combination of hardware and software components.

Referring to fig. 2, the decoder 210 may be a software program and/or hardware device that intercepts and decodes (decompresses) incoming compressed video frames. Each incoming compressed video frame may include, among other things, a header and compressed frame video. The header may include, among other things, the input QP (QP) that has been used to compress the frame_Input）。

The decoder 210 may include, among other things, an inverse quantizer 212. For each decompressed video frame, inverse quantizer 212 may extract QP-_InputAnd will QP_InputAnd to QP adjustment module 220. The decoder 210 may then transmit the decompressed video frames to the encoder 230. In some embodiments, the decompressed video frames may be stored in a database (e.g., a buffer)Or a queue). Furthermore, as an alternative to the configuration of the video optimizer 108 shown in fig. 2, the inverse quantizer 212 may be located outside the decoder 210.

Quality-aware video optimization: QP adjustment

The QP adjustment module 220 may be for QP based_InputDeriving an output QP (QP) for compressing decompressed video frames_Output) Software program and/or hardware device. As previously described, the QP adjustment module 220 may use QP since a relationship may be established between the QP value and the perceived quality of the video frame and also between the QP and the bitrate of the video frame_InputAnd QP_OutputThe difference (Δ QP) between to simultaneously control the bit rate reduction and perceptual quality degradation of the output video frame that optimizes the compressed video frame (204).

The parameter Δ QP may be defined as, for example:

QP_Input + ΔQP = QP_Output

wherein QP_InputIs a quantization parameter used when initially encoding an input (compressed) video frame (202), and QP_OutputIs a quantization parameter derived by QP adjustment module 220 for use in requantizing the decompressed frame. For each input video frame, the QP is known_InputAnd a compressed byte size. QP adjustment module 220 may select a delta QP value to derive a QP for quantizing an output video frame that can meet the requirements of byte size reduction and maintain the original perceptual quality_Output. The smaller the Δ QP value, QP_OutputThe smaller the value and thus the less information will be discarded during encoding-the less degradation in perceived quality. Conversely, the greater the Δ QP value, the QP_OutputThe larger the value and thus more information will be discarded during encoding-the degradation in perceived quality is more.

The two attributes of the QP versus bitrate curve shown in fig. 3 can be utilized to achieve optimization of the input video frame. The first attribute is to take advantage of the fact that: when in useQP_InputHigh, a small Δ QP has little effect on the bit rate. This is shown in the rightmost flat portion of the curve in fig. 3. Using high QP_OutputQuantizing the video frame discards more information from the original video frame. To maintain the original perceptual quality of the video frame, a high QP cannot be used_OutputThe value is obtained. This contradicts the goal of reducing byte size, since when QP_OutputLess than QP_InputThe byte size of the frame can only be increased. Given the shape of the QP versus bit rate curve in fig. 3, then when QP is given_IntputWhen high, a negative Δ QP may be used to derive a relatively small QP that discards less information and maintains perceptual quality_OutputWhile increasing the byte size by only a small amount.

The second attribute to be utilized is: when QP_InputLow, a small Δ QP has a large impact on the bit rate. This is shown in the left most steep part of the curve in fig. 3. When QP_InputAt low, even a small positive Δ QP will result in a large byte reduction — a large amount of information is discarded during encoding.

These observations lead to a new function F, in terms of QP_InputTo define Δ QP. Δ QP and QP_InputThis relationship between can be written as, for example:

ΔQP = F(QP_Input)

the function F may be derived experimentally from the above observations and from sensory analysis of the actual video source. The function F may be, for example, QP_InputAnd Δ QP. The function F may also be based on the inclusion of, for example, QP_InputAnd Δ QP.

Referring back to fig. 2, the QP adjustment module 220 may utilize a function F to obtain Δ QP and, in turn, use the Δ QP to derive a QP for quantizing the decompressed video frame_Output. For example, when QP_InputWhen low or below a predetermined threshold, QP adjustment module 220 may select a positive Δ QP. Positive delta QP may be desirable for high qualityA significant byte savings results when the frame is re-quantized. Based on positive Δ QP, QP adjustment module 220 may derive a ratio QP_InputHigher QP_OutputValue to substantially reduce the byte size while minimally degrading the original perceived quality of high quality input video frames. On the other hand, when QP_InputAt or above a predetermined threshold, the QP adjustment module 220 may select a negative delta QP when a large amount of information is discarded during the original compression of the input video frame. Based on negative Δ QP, QP adjustment module 220 may derive a ratio QP_InputLower QP_OutputValue to preserve the original perceptual quality of the input video frame while only minimally enlarging the byte size. The net byte reduction may be positive when the method is used to optimize a normal sequence of video frames (202) comprising frames of various qualities.

Quality-aware video optimization: controlling quality degradation

In some embodiments, the QP adjustment module 220 may also obtain a quality degradation offset (which is a parameter indicative of the amount of quality degradation) to control the quality degradation during optimization of the decompressed video frames. QP that can be used to re-quantize video frames_InputAnd a quality degradation is observed in the mapping between delta QP values. FIG. 4 shows QP_InputAnd Δ QP. Δ QP and QP in FIG. 4_InputThe values of (a) are merely examples. In real life, these values may be different for different video sequences, and QP_InputThe mapping to Δ QP may vary. In fig. 4, a higher curve yields more quality degradation, since a higher Δ QP means that more information is discarded when the frame is re-quantized.

Referring back to fig. 2, the QP adjustment module 220 may add a quality degradation offset to the Δ QP to adjust for the constant quality degradation introduced during optimization. For example, QP adjustment module 220 may obtain Δ QP using the following modification function:

ΔQP = F(QP_Input) + D

as mentioned above, F is public QP_InputAnd the correlation between Δ QP. The quality degradation deviation D is a parameter indicating the quality degradation amount. The parameter may be obtained based on, for example, historical data analysis of the optimized video, among other things. The default value of the quality degradation offset D may be set to 0 if the quality degradation introduced during optimization is to be minimized. The QP adjustment module 220 can obtain the quality degradation offset from, for example, a database containing, for example, historical data analysis of the optimized video. QP-based_InputAnd a quality degradation offset, QP adjustment module 220 may derive Δ QP. Then, QP adjustment module 220 may use Δ QP to calculate and QP_InputCorresponding QP_Output。

Quality-aware video optimization: mapping quantization parameters between codecs

In some embodiments, QP adjustment module 220 may also take into account differences between any combination of input and output compression formats (codecs) and calculate a codec adjustment that takes into account differences between the input and output compression formats (codecs). The quantization settings indicated by a given QP may be different for each video codec. Equivalent or corresponding quantization settings of two different video codecs that exhibit the same perceptual quality may be represented by two different QP values with different scales. For example, the quantization setting for a video frame in codec a is represented by QP value 20 and the corresponding quantization setting in codec B is represented by QP value 15. To obtain the quantization setting in codec B that corresponds to the quantization setting in codec a represented by the QP value (e.g., 20), the QP adjustment module 220 may convert the QP value (e.g., 20) from a QP scale in codec a to a QP scale in codec B and obtain a converted QP value (e.g., 15). By having this functionality, the quality-aware video optimizer 108 can provide optimization benefits in any transcoding scenario.

If the compression format (codec) of the input (compressed) video frame (202) is detected by the decoder 210 and used by the encoder 230 to decompress the video frameIf the compression format is different, the QP adjustment module 220 may adjust the QP before the Δ QP can be calculated_InputThe QP scale for the decoder is converted to the QP scale for the encoder. This can be done, for example, by: a mapping between QP scales for each combination of decoder and encoder supported by the quality-aware video optimizer is maintained. For example, the mapping between QP scales correlates all QPs from codec a with those of codec B so that video frames encoded with codec a can yield the same perceptual quality as when encoded with codec B. These mappings may be determined experimentally and may be implemented in a QP adjustment step by QP adjustment module 220. QP may not be converted if the input codec is the same as the output codec_InputOr QP after conversion_InputAnd the QP_InputThe same is true.

Based on converted QP_InputQuality degradation offset, QP adjustment module 220 may derive Δ QP. Alternatively, QP adjustment module 220 may obtain a codec adjustment that takes into account the difference between the input and output compression formats (codecs) based on a determination of whether the input and output compression formats are the same. QP-based_InputCodec adjustment and quality degradation offset, QP adjustment module 220 may calculate Δ QP. QP adjustment module 220 may then calculate QP based on delta QP_OutputAnd will QP_OutputAnd forwarded to encoder 230.

The encoder 230 may be a software program and/or hardware device that receives decompressed video frames and encodes (compresses) the video frames. The encoder 230 may include, among other things, a quantizer 232. Quantizer 232 receives the QP_OutputAnd corresponding decompressed video frame, and by QP_OutputThe quantization settings of the representation are applied to the encoding (compression) of the video frame to optimize the video frame. The video optimizer 108 then outputs and transmits 204 the optimized compressed video frames to the destination. Further, as an alternative to the configuration of the video optimizer 108 shown in fig. 2, the quantizer 232 may be located external to the encoder 230.

The video optimizer may process the incoming compressed video frames one by one (202) and apply the above-described video quality-aware optimization process to each video frame in a single pass, instantaneously. The video optimizer can be installed at any location when needed, and can apply quality-aware video optimization whenever needed.

Fig. 5 is a functional diagram illustrating an exemplary process flow in the embodiment of fig. 2. The exemplary process flow may be altered to delete steps, change the order of steps, or include additional steps without departing from the exemplary embodiments.

After receiving (202) the compressed video frames from the data network, the video optimizer 108 may direct the compressed video frames to the decoder 210. The decoder 210 may intercept and decode (decompress) the incoming compressed video frames. The inverse quantizer 212 of the decoder 210 may extract the QP from the decompressed video frame that was originally used to compress the incoming compressed video frame_Input. The decoder 210 or inverse quantizer 212 may then extract the QP_InputPasses 502 to the QP adjustment module 220 and transmits 504 the decompressed video frames to the encoder 230.

QP based if the input compressed video frame (202) and the output compressed video frame (204) have different compression formats (codecs)_InputThe QP adjustment module 220 may adjust the QP_InputFrom the QP scale of the decoder 210 to the QP scale of the encoder 230. The conversion may be based on a mapping between QP scales for each combination of input and output codecs supported by the quality-aware video optimizer. QP may not be converted if the input codec is the same as the output codec_InputOr QP after conversion_InputAnd the QP_InputThe same is true.

Further, in some embodiments, the QP adjustment module 220 may obtain a quality degradation offset that indicates an amount of quality degradation to control the quality degradation introduced during optimization of the decompressed video frames. Can be used forTo set the default value of the quality degradation offset to 0, which indicates that a minimum quality degradation is introduced. The quality degradation offset may be obtained, for example, from a database containing, for example, historical data analysis of the optimized video. Based on converted QP_InputAnd a quality degradation offset, the QP adjustment module 220 may calculate a QP for derivation and entry_InputCorresponding QP_OutputAnd will QP_OutputForwarded 506 to encoder 230.

Upon receipt (504, 506) of the decompressed video frame and corresponding QP_OutputThe quantizer 232 of the encoder 230 then passes the signal through the QP_OutputThe indicated quantization settings are applied to the encoding (compression) of the decompressed video frames to optimize the video frames. The optimization may include: for example, the byte size of a compressed video frame is reduced while attempting to maintain the original perceived quality of the compressed video frame. After the optimization, the video optimizer 108 outputs and transmits 204 the optimized compressed video frames to the destination (e.g., the user device 102).

Fig. 6 is a flow diagram representing an exemplary method of quality-aware video optimization. The exemplary process flow may be altered to delete steps, change the order of steps, or include additional steps without departing from the exemplary embodiments.

Upon receiving (602) the compressed video frame, the video optimizer decodes (decompresses) (604) the video frame and extracts (606) the input QP value from the decoded (decompressed) video frame. In addition, the video optimizer may obtain (608) a codec adjustment that takes into account the difference between the input and output compression formats (codecs). The video optimizer may perform this step based on a determination of whether the encoding codec (output video compression format) is different from the decoding codec (input video compression format). If the input codec is the same as the output codec, the default value for codec adjustment may be set to 0.

At step 610, the video optimizer may obtain a quality degradation offset indicating an amount of quality degradation to control the quality degradation introduced during optimization of the decompressed video frames. If the quality degradation is to be minimized, the default value of the quality degradation offset may be set to 0. The quality degradation offset may be obtained based on, for example, an analysis of historical data of the optimized video. Based on the input QP, codec adjustment, and quality degradation offset, the video optimizer may calculate Δ QP. Δ QP corresponds to the difference between the input QP and the output QP used to optimize the decompressed video frame. The video optimizer may then derive 612 an output QP based on the delta QP.

Alternatively, the video optimizer may convert the input QP value from a decoded (input) QP scale to a coded (output) QP scale based on a determination of whether the coding codec is different from the decoding codec. If the decoding codec is the same as the encoding codec, the input QP may not be converted or the converted input QP may be the same as the input QP. Based on the converted input QP and quality degradation offset, the video optimizer may calculate a Δ QP and thereby use the Δ QP to derive 612 an output QP for optimizing the decompressed video frame.

After the output QP has been derived, the video optimizer may optimize (614) the decoded (decompressed) video frame by encoding (compressing) the decoded (decompressed) video frame based on the quantization setting corresponding to the derived output QP. The optimization can modify the decompressed video frames to reduce the byte size of the compressed video frames as much as possible while at the same time limiting the perceptual quality degradation to a controllable level. The video optimizer may transmit (616) the optimized video frame to the destination. The method may then end (618).

Fig. 7 is a block diagram illustrating another embodiment of the example quality aware video optimizer 108 in the example system of fig. 1. The system in fig. 7 has similar components to those in fig. 2, except that in fig. 7, the video optimizer 108 has an expansion prevention module 710 for compensating for byte size expansion that may be introduced during optimization. The illustrated configuration of the video optimizer 108 in fig. 7 is merely exemplary and may be modified to provide discrete elements or to combine multiple elements into a single element, and may be provided as any combination of hardware and software.

Quality-aware video optimization: applying compression feedback to prevent expansion

The following situations may occur: wherein QPs for at least some of the input video frames of the video stream (202)_InputThe value is high and thus the QP adjustment module 220 can select a negative delta QP value for these frames to maintain the original perceptual quality of the video stream. High QP_InputThe value indicates that the input video frame (202) has lost a lot of information during its original compression. During the quality-aware video optimization process, it is not desirable to additionally discard more information from the input video frames. QP adjustment module 220 may select to derive a low QP_OutputA negative delta QP value for the value to maintain the original perceived quality of the video stream. However, negative Δ QP values and low QP_OutputThe value may expand the byte size of the video stream. Although the extent to which the byte size expands may be small, it is generally undesirable to increase the byte storage requirements of the video stream.

The dilation prevention module 710 may utilize a feedback mechanism to adjust any potentially expanded delta QP values. In some embodiments, for a video stream currently being processed, the expansion prevention module 710 may utilize a compensation function that takes into account the total byte size of at least some video frames encoded so far and the total subsection size of at least some video frames decoded so far. In some embodiments, for a video stream currently being processed, the dilation prevention module 710 may utilize a compensation function that takes into account the total byte size of all video frames encoded so far and the total byte size of all video frames decoded so far. The compensation function may output a positive value to compensate for the byte size expansion. The enlargement prevention module 710 may then also adjust the Δ QP value based on the byte size enlargement adjustment (output of the compensation function) to maintain the original perceptual quality of the input video frame while avoiding or minimizing byte size enlargement.

For each video frame to be encoded, Δ QP may be calculated as, for example:

ΔQP = F(QP_Input) + D + C(InputBytes, OutputBytes)

as previously described herein, F is the public QP_InputA function of the correlation with Δ QP, and the parameter D is a quality degradation offset indicating the amount of quality degradation to be introduced during re-quantization. Function C is a compensation function. In some embodiments, the symbol InputBytes represents the total byte size of at least some frames previously decoded for the video stream currently being processed, and the symbol OutputBytes represents the total byte size of at least some frames previously encoded for the video stream currently being processed. In some embodiments, the symbol InputBytes represents the total byte size of all frames previously decoded for the video stream currently being processed, and the symbol OutputBytes represents the total byte size of all frames previously encoded for the video stream currently being processed. The video optimizer 108 may track the InputBytes and OutputBytes of the stream of video frames currently being processed. The InputBytes and OutputBytes may also be obtained from a database containing, for example, the total byte size of the video frames encoded so far and the total byte size of the video frames decoded so far for the stream of video frames currently being processed. If no byte size expansion occurs, the compensation function may output a 0 value.

Already based on QP at QP adjustment module 220_InputAnd the quality degradation offset, the expansion prevention module 710 may adjust the Δ QP based on the byte size expansion adjustment. Then, based on the adjusted Δ QP, the expansion prevention module 710 may derive a QP for optimizing the decompressed video frame being processed_Output. In some embodiments, the QP adjustment module 220 and the expansion prevention module 710 may be combined into one component. The composition component may be based on QP_InputThe quality degradation offset and the byte size expansion adjustment are used to calculate delta QP and also used to derive a delta QP for optimizing the decompressed video frameQP of (2)_Output。

The remainder of the components operate in a similar manner to the components provided above in fig. 2. The dilation prevention mechanism may help to maintain the original perceived quality of a video stream that has lost a large amount of information during the original compression, while reducing the byte size of the video stream as much as possible.

Fig. 8 is a functional diagram illustrating an exemplary process flow in the embodiment of fig. 7. The system in fig. 8 has similar components to those in fig. 5, except that in fig. 8, the video optimizer 108 provides an expansion prevention module 710 for compensating for byte size expansion when calculating Δ QP. The exemplary process flow may be altered to delete steps, change the order of steps, or include additional steps without departing from the exemplary embodiments.

After the delta QP for the input compressed video frame has been calculated, QP adjustment module 220 provides 802 the calculated delta QP to expansion prevention module 710. The enlargement prevention module 710 may retrieve historical byte size data or historical enlargement data. The historical byte size data may include, for example, Inputbytes (the total byte size of all or some frames previously decoded for the stream of video frames currently being processed) and Outputbytes (the total byte size of all or some frames previously encoded for the stream of video frames currently being processed) that are saved by the video optimizer 108 or in a database. The history enlargement data may comprise, for example, byte size enlargement that has been introduced during video optimization of some frames of the stream of video frames currently being processed. Based on historical byte size data or historical enlargement data, enlargement prevention module 710 can estimate the byte size enlargement to be introduced during video optimization based on the Δ QP provided by QP adjustment module 220. The expansion prevention module 710 may then adjust Δ QP by compensating for the estimated byte size expansion. The adjusted delta QP can help maintain the original perceptual quality of the input compressed video frame while reducing the byte size of the video frame as much as possible. Based on the adjusted Δ QP, the dilation prevention module 710 may derive a QP that is related to the incoming_InputCorresponding QP_OutputAnd will QP_OutputForwarded 506 to encoder 230.

The remainder of the communication flow is similar to that provided above in fig. 5.

Fig. 9 is a flow diagram representing another exemplary method of quality-aware video optimization. The exemplary process flow may be altered to delete steps, change the order of steps, or include additional steps without departing from the exemplary embodiments.

Upon receiving (902) the compressed video frame, the video optimizer decodes (decompresses) (904) the video frame and extracts (906) the input QP value from the decoded (decompressed) video frame. Further, the video optimizer may convert (908) the input QP value from a decoded QP scale to an encoded QP scale based on a determination that the encoding codec (output video compression format) is different from the decoding codec (input video compression format). The conversion takes into account the difference between the input and output compression formats (codecs). If the encoding codec is the same as the decoding codec, the input QP value is not converted or the converted input QP value is the same as the input QP value.

At step 910, the video optimizer may also obtain a quality degradation offset that indicates an amount of quality degradation to be introduced during optimization of the decompressed video frames. If the quality degradation is to be minimized, the default value of the quality degradation offset may be set to 0. The quality degradation offset may be obtained based on, for example, an analysis of historical data of the optimized video.

The video optimizer may also obtain (912) a byte-size expansion adjustment based on historical data of the video stream currently being processed. The historical data may include historical byte size data or historical expansion data for the video stream. The historical byte size data may include, for example, a total byte size of all or some frames previously decoded for the video stream and a total subsection size of all or some frames previously encoded for the video stream. The historical enlargement data may include, for example, byte size enlargement that has been introduced during video optimization of some frames of the stream of video frames currently being processed. In some embodiments, historical data may be retrieved from a database. The byte size expansion adjustment may compensate for byte size expansion to be introduced during optimization of the decompressed video frames. If no byte size expansion is introduced, the default value for byte size expansion adjustment may be set to 0.

Based on the converted input QP, quality degradation offset, and byte size expansion adjustment, the video optimizer may calculate Δ QP. Δ QP corresponds to the difference between the input QP and the output QP used to optimize the decompressed video frame. The video optimizer may then derive 914 an output QP based on the delta QP.

Alternatively, the video optimizer may first calculate Δ QP based on the converted input QP and quality degradation offset, and then adjust Δ QP based on the byte size expansion adjustment. Based on the adjusted delta QP, the video optimizer may derive 914 an output QP for optimizing the decompressed video frame.

After the output QP has been derived, the video optimizer may optimize (916) the decoded (decompressed) video frame by encoding (compressing) the decoded (decompressed) video frame based on the quantization setting corresponding to the derived output QP. The optimization can modify the decompressed video frames to reduce the byte size of the compressed video frames as much as possible while at the same time limiting the perceptual quality degradation to a controllable level. The video optimizer may transmit (918) the optimized video frames to the destination. The method may then end (920).

The methods disclosed herein may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Some or all of the methods disclosed herein may also be implemented by an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Complex Programmable Logic Device (CPLD), a Printed Circuit Board (PCB), a Digital Signal Processor (DSP), a combination of programmable logic components and programmable interconnects, a single Central Processing Unit (CPU) chip, a CPU chip combined on a motherboard, a general purpose computer, or any other combination of devices or modules capable of performing the quality-aware video optimization disclosed herein.

In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims that follow. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein.

Claims

1. A computer-implemented method, comprising:

receiving an encoded video frame;

decompressing the received encoded video frame;

extracting a first Quantization Parameter (QP) from the decompressed video frame, wherein the first QP corresponds to a quantization setting originally used to compress the encoded video frame;

obtaining a delta QP based on the first QP;

obtaining a second QP based on the delta QP and the first QP, wherein the second QP corresponds to quantization settings used for compressing decompressed video frames;

compressing the decompressed video frame based on the second QP; and

compressed video frames are provided.

2. The method of claim 1, further comprising:

determining whether a compression format of the received encoded video frame is different from a compression format to be used for compressing the decompressed video frame;

based on the determination, converting the first QP from an input QP scale to an output QP scale, wherein the input QP scale corresponds to a compression format of the received encoded video frame and the output QP scale corresponds to the compression format to be used for compressing the decompressed video frame; and

obtaining the second QP based on the converted first QP and the delta QP.

3. The method of claim 2, wherein converting the first QP is further based on a mapping between the input QP scale and the output QP scale.

4. The method of claim 1, wherein obtaining Δ QP is further based on a mapping between the first QP and the Δ QP.

5. The method of claim 1, wherein obtaining Δ QP comprises:

selecting a positive value for the Δ QP if the value of the first QP is below a threshold.

6. The method of claim 1, wherein obtaining Δ QP comprises:

selecting a negative value for the delta QP if the value of the first QP is above a threshold.

7. The method of claim 1, further comprising:

obtaining a quality degradation offset for controlling an amount of video quality degradation to be introduced during compression of decompressed video frames; and

obtaining the second QP based on the quality degradation offset, the first QP, and the delta QP.

8. The method of claim 7, wherein obtaining the quality degradation offset can be based on an analysis of historical data of the compressed video.

9. The method of claim 7, wherein the default value of the quality degradation offset can be set to 0 if a minimum quality degradation is introduced during compression.

10. The method of claim 1, further comprising:

obtaining an expansion adjustment based on a comparison between a total byte size of a previously decompressed frame and a total byte size of a frame after the previous compression, wherein the received encoded video frame and the previously decompressed frame are part of a sequence comprising a plurality of video frames; and

obtaining the second QP based on the delta QP, the first QP, and the expansion adjustment.

11. A computer-implemented method, comprising:

decompressing a received encoded video frame that is part of a sequence comprising a plurality of video frames;

obtaining a delta QP based on the first QP;

obtaining an augmentation adjustment based on historical data about the sequence;

obtaining a second QP based on the delta QP and the expansion adjustment, wherein the second QP corresponds to quantization settings used for compressing decompressed video frames; and

compressing the decompressed video frame based on the second QP.

12. The method of claim 11, wherein the historical data regarding the sequence comprises byte size expansion that has been introduced during compression of at least some of the sequence frames.

13. The method of claim 11, wherein the historical data regarding the sequence comprises byte sizes of at least some of the previously decompressed sequence frames and byte sizes of at least some of the sequence frames after the previous compression.

14. The method of claim 11, wherein the acquisition dilation adjustment is based on a comparison between byte sizes of at least some of the previously decompressed sequence frames and byte sizes of at least some of the sequence frames after the previous compression.

15. A video optimizer comprising:

a decoder configured to decompress a received encoded video frame, wherein the decompressed video frame comprises a first Quantization Parameter (QP) corresponding to a quantization setting originally used to compress the encoded video frame;

a QP adjustment module configured to obtain a delta QP based on a first QP and obtain a second QP based on the delta QP and the first QP, wherein the second QP corresponds to quantization settings used for compressing decompressed video frames; and

an encoder configured to compress the decompressed video frame based on the second QP.

16. The video optimizer of claim 15, further comprising: an inverse quantizer configured to decompress the received encoded video frame and extract the first QP from the decompressed video frame.

17. The video optimizer of claim 15, wherein the QP adjustment module is further configured to:

obtaining a compression format adjustment that takes into account differences between a compression format of a received encoded video frame and a compression format to be used for compressing a decompressed video frame;

converting the first QP from an input QP scale to an output QP scale based on the compression format adjustment, wherein the input QP scale corresponds to a compression format of the received encoded video frame and the output QP scale corresponds to the compression format to be used for compressing the decompressed video frame; and

obtaining the second QP based on the converted first QP and the delta QP.

18. The video optimizer of claim 15, wherein the QP adjustment module is further configured to obtain the compression format adjustment from a database containing a mapping between the input QP scale and the output QP scale.

19. The video optimizer of claim 15, wherein the QP adjustment module is further configured to obtain the delta QP from a database containing a mapping between the first QP and the delta QP.

20. The video optimizer of claim 15, wherein the QP adjustment module is further configured to:

obtaining the second QP based on the quality degradation offset and the delta QP.

21. The video optimizer of claim 20, wherein the QP adjustment module is further configured to obtain the quality degradation offset from a database containing historical data analysis of compressed video.

22. The video optimizer of claim 15, further comprising an expansion prevention module configured to:

23. The video optimizer of claim 15, further comprising a quantizer configured to compress the decompressed video frames by applying the quantization settings represented by the second QP to the compression of the decompressed video frames.

24. A video optimizer comprising:

a decoder configured to decompress a received encoded video frame that is part of a sequence of a plurality of video frames, wherein the decompressed video frame comprises a first Quantization Parameter (QP) corresponding to a quantization setting originally used to compress the encoded video frame;

a QP adjustment module configured to obtain a delta QP based on the first QP;

an expansion prevention module configured to obtain an expansion adjustment based on historical data regarding the sequence and obtain a second QP based on the delta QP and the expansion adjustment, wherein the second QP corresponds to quantization settings used for compressing decompressed video frames; and

25. The video optimizer of claim 24, wherein the historical data about the sequence comprises byte size expansions that have been introduced during compression of at least some of the sequence frames.

26. The video optimizer of claim 24, wherein the historical data regarding the sequence comprises byte sizes of at least some of the sequence frames previously decompressed and byte sizes of at least some of the sequence frames after previous compression.

27. The video optimizer of claim 24, wherein the dilation prevention module is further configured to: the dilation adjustment is obtained based on a comparison between a byte size of at least some of the sequence frames previously decompressed and a byte size of at least some of the sequence frames after previous compression.

28. A computer-readable medium storing instructions that, when executed, cause a computer to perform a method for video optimization, the method comprising:

receiving an encoded video frame;

decompressing the received encoded video frame;

obtaining a delta QP based on the first QP;

compressing the decompressed video frame based on the second QP; and

compressed video frames are provided.

29. A computer-readable medium storing instructions that, when executed, cause a computer to perform a method for video optimization, the method comprising:

decompressing a received encoded video frame that is part of a sequence of a plurality of video frames;

obtaining a delta QP based on the first QP;

obtaining a second QP based on the delta QP and the byte size expansion adjustment, wherein the second QP corresponds to quantization settings used for compressing decompressed video frames; and

compressing the decompressed video frame based on the second QP.