GB2540242A

GB2540242A - Method and apparatus for rate control subjective optimisation

Info

Publication number: GB2540242A
Application number: GB1607083.1A
Authority: GB
Inventors: Patrick Compton Matthew
Original assignee: Riovideo Ltd
Current assignee: Riovideo Ltd
Priority date: 2015-02-22
Filing date: 2016-04-22
Publication date: 2017-01-11
Anticipated expiration: 2036-04-22
Also published as: GB2540242B; GB201502915D0

Abstract

A method of video compression (coding) comprises determining a video picture encoding metric, providing the data encoding metric as an input feed to a bitrate controller, and regulating the extent of compression based at least in part on the data encoding metric. The estimated or predicted encoding metric or parameter may be a measure of data cost or magnitude, or picture complexity or entropy. The bit rate control measure may be determined by: an inter-picture metric; picture (image) difference value; correlation between a picture to be coded and a previous picture; or relative to multiple previous pictures. An optimum compression control parameter, such as a quantisation parameter (QP), may be determined, taking account of a bitrate constraint. Predicted resulting picture encoding metrics for different compression control parameters may be determined, with an optimum compression control parameter selected by interpolation. A combination of inter and intra picture coding metrics may be used. Buffer fill may be determined, and scene cuts may be detected based on a metric threshold being exceeded.

Description

METHOD AND APPARATUS FOR RATE CONTROL SUBJECTIVE OPTIMISATION

Technical field

The present invention is related to video coding in general and in particular, although not exclusively, to a method and apparatus for deciding (quantisation parameter) QP during video coding in order to achieve a specific or pre-determined number of coded bits or bitrate whilst maximising subjective video quality.

Background

In conventional video compression encoders each picture is divided into blocks (so called macroblocks in MPEG2/AVC and Coding Units in HEVC). For each block it is usually possible to adapt or vary the QP (quantisation parameter) in order to achieve a specific number of coded bits or a specific visual effect (for example to lower or increase distortion). In addition, QP may also be changed at (overall) picture level for the same aforementioned reasons.

It is generally necessary to change the QP during video encoding of video content in order to achieve a target coded bitrate (or bits per picture). For example CBR (Constant Bitrate) applications require a fixed target bit rate and VBR (Variable Bitrate) applications require a variable target bitrate, where the target bitrate is defined by an external source. Target bitrates typically arise because of a requirement to deliver a smooth constant bitrate encoded video signal, and all pictures are placed in a "rate buffer" which is required to buffer the bursty nature of encoded pictures vs stream delivery. A rate buffer can be thought of as a leaky bucket with variable input rate and constant output rate. Using the analogy of a leaky bucket, care is required not to overfill the bucket or data will overflow. However, care is also required not to under fill the bucket otherwise a constant output stream will stop. Many TV systems require that the leaky bucket conforms to these rules to maintain a consistent viewing experience. The task of rate control is therefore to manage the fullness of the buffer whilst maintaining optimum video quality (VQ).

Video content (for example a football match) generally has rapid changes in complexity, such as scene transitions or changes in motion or detail. These changes in complexity will in turn change the number of coded bits generated by the video coder, hence the need to adapt QP to maintain a target bitrate. For video encoding (algorithms) that deploy intra predicted (I) and intra predicted (P/B) pictures, the change in complexity can be very pronounced in the number of compressed bits generated. In video encoding algorithms that deploy only intra predicted (I) pictures, only spatial complexity will determine the number of compressed bits generated.

Video sequences are generally encoded as a series of picture types I, B or P. I pictures use only the intra prediction mode (for each block of the picture) and act as an entry point into a stream. Intra pictures only have internal (intra) dependencies. Whereas, P and B pictures are Inter Predictive and Inter Bi-Predictive and hence have an Inter dependency on other pictures. It should be noted that P and B pictures can have a mix of Intra and Inter prediction blocks internally. A typical method of rate control is the use of a closed loop proportional controller algorithm, where the number of generated bits from a video encoding algorithm is used to directly control the current operating QP value. Whilst this method may achieve the desired bitrate, it does not take any subjective visual consequences into account as the algorithm implemented by the rate controller reacts to change from one picture to the next.

An alternative known method is to use a combination of a closed loop proportional controller and plurality of look-ahead data to make visually optimised decisions. In this instance, the look-ahead data may come from a first pass video coding algorithm in order to steer a second pass video coding algorithm for optimum visual quality whilst maintaining target bitrate.

Typically, the human visual system can be sensitive to rapid changes in QP. Hence it is usually desirable to align QP changes with a video content complexity change (such as a scene cut) in order to mask the QP change. This usually requires pre-emptive knowledge of the video content or so called look-ahead data. However, to generate the additional look-ahead data extra processing units or processing power is required. This reduces overall processing efficiency and/or increases cost of a video encoding implementations. A known method of rate control is shown schematically in Figure 3. A Rate Control unit counts the number of bits generated by the video coding process in order to maintain a target bitrate as specified by an external source. The Rate Control unit controls the QP into the process effected by the Transform and Quantisation module to regulate the number of bits generated, typically as a closed loop proportional controller.

Summary

According to a first aspect of the invention there is provided a method as claimed in claim 1.

The method may comprise determining the data cost metric by way of determining an extent or measure of correlation between the picture to be encoded and the previous picture. The result of this may be termed a residual signal. A second aspect of the invention relates to an apparatus as claimed in claim 24.

The compression control parameter may be termed a coefficient.

The picture encoding metric may be termed look-ahead data.

The method may comprise determining a ratio of Intra Picture size compared to Inter Picture sizes for a picture to be encoded, such as a new scene.

The method may comprise setting a predetermined delay to the encoding of pictures. The delay may be proportional or equivalent to one or more picture times.

The method may comprise determining a measure of entropy.

The method may comprise determining data encoding metric by determining a residual, performing a transformation on the residual and quantising the transformation. These processing steps may be implemented by one or both of a inter picture estimator and an intra picture estimator.

The method and/or apparatus may comprise one, some or all of the following processing, functionalities and functional modules: motion estimation, motion compensation, mode decision, difference (residual) calculator, transformation and quantisation, inverse quantisation and transform, sum (reconstruction) and entropy encoder.

The method and apparatus of rate control may be viewed as providing an improved method of look-ahead that gives similar results to a two-pass look-ahead without the need for a first pass process. Embodiments of the invention advantageously comprise the generation of look-ahead data which is generated as part of a single pass video encoding process or algorithm with minimal additional processing.

The method and apparatus may be viewed as relating to rate control subjective optimisation in digital video coding, comprising determining look-ahead metrics for a plurality of QP points for Intra and Inter encoded video pictures, and selecting QP for rate control operation.

An aspect of the invention may be viewed as a method of managing a rate buffer of a video encoder. This may include determining a measure of current rate buffer fill.

An aspect of the invention may be viewed as a method of data processing for encoding a video signal. A further aspect of the invention may comprise identifying scene cuts or transitions in a video, the method comprising determining if a picture encoding metric of a picture to be encoded exceeds a predetermined value.

Embodiments of the present invention are particularly beneficial to the HEVC compression standard, as HEVC generally requires significantly more processing power and the invention improves subjective visual quality without adding significant processing overhead. However, the invention is not limited to the HEVC standard and is equally as applicable to other compression standards or algorithms (for example A VC and MPEG2).

The method and apparatus may comprise one or more features as described in description and/or as shown in the drawings, either singularly or in combination.

Brief description of the drawings

Various embodiments of the drawings will now be described, by way of example only, with reference to the following drawings in which:

Figure 1 shows a plot of an example video sequence encoded in IPPP GOP structure. The sequence has high spatial detail but low motion complexity demonstrating the difference in I and P coded pictures;

Figure 2 shows a plot of an example video sequence encoded in IPPP GOP structure. The sequence has medium spatial detail and high motion complexity demonstrating the difference in I and P coded pictures;

Figure 3 shows a schematic block diagram of an example intra/inter video coder algorithm with proportional feedback for controlling QP;

Figure 4 shows a schematic block diagram of a rate control apparatus according to the embodiment of the invention in which different functional blocks are depicted;

Figure 5 shows a schematic block diagram of an Intra picture estimator apparatus according to the embodiment of the invention in which its sequence of processing steps is depicted schematically;

Figure 6 shows a schematic block diagram of an Inter picture estimator apparatus according to the embodiment of the invention in which its sequence of processing steps is depicted schematically; and

Figure 7 shows a plot of Inter Picture Estimator scores for a sequence of video pictures with scene cuts.

Detailed description

Reference is made initially to Figure 4 in which, in comparison to the prior art arrangement of Figure 3, the real-time encoding apparatus is enhanced with two additional look-ahead metrics, namely an Intra (I) picture estimator and an Inter (P/B) picture estimator . Both of the Intra and Inter picture estimators are arranged to operate in advance of the main picture encoding processor in order to give preemptive knowledge of the video complexity. This may be a delay of one or more video pictures (denoted by ‘delay n’ in Figure 4). In broad terms, the estimators derive cost metrics (or scores) relating to a picture to be encoded, which are made available ahead of the time at which the picture is encoded. This may be based on previous pictures (in the temporal sequence of the pictures) which are then used in determining a suitable measure of compression as applied to a picture to be encoded.

In addition, and advantageously, the estimators are not required to generate an exact prediction of how big/complex (i.e. their respective data cost) an Intra or Inter predicted picture will be, but rather the estimation provides a meaningful guide in order to maintain a visual continuity over the sequence of pictures forming the video. Although, it is to be noted that the more accurate the prediction and costing function, the more accurate the overall estimate will be.

As such, the intra estimator could use a closed loop (i.e. generate prediction from recon samples) or an open loop (i.e. generation from source samples) in order to make a prediction for a block. In addition both HEVC and AVC offer various modes of prediction from (multiple) neighbouring video samples, a single mode or a simplification of a mode may be used to create a predicted candidate.

Likewise, the inter estimator can be a simplification of the specified inter prediction process of the main video coding algorithm. The Motion Estimation (ME) module generates motion vector candidates for (sub-)blocks within the picture (maybe a pixel resolution or sub pixel resolution). In order for the Motion Estimation algorithm to operate in advance of the main encoding algorithm it is likely to use source video picture for source and reference. The best motion vector (MV) candidate for a specific block is used to create a predicted candidate.

For both the Intra and Inter estimators a per block residual signal is created (source-predicted) which is then transformed, quantised and costed (into a metric). The costing process may include a full entropy encoding algorithm of the main video coder or a simplification thereof. A sum is then created of all blocks in a picture to yield a final picture estimate score or costing. Figure 5 shows the process for Intra picture estimation and Figure 6 shows the process for Inter picture estimation. The total costing of a picture, which takes account of both inter and intra encoding metrics, in combination, provides an (estimated) indication of the complexity of that picture, which, as is described below, can be used determine an optimum quantisation parameter.

Both the Intra and Inter estimators use at least one fixed QP point to generate an estimated metric. For example, AVC and HEVC compression standards typically use a QP range of 0-51, therefore an appropriate fixed QP point might be 32, but of course other QP points could be used.

It may be desirable to have a plurality of QP points spread across the QP operating range for a given compression standard. This is denoted by QPn in Figures 5 and 6.

For most compression standards there is a relationship between QP step size and the number of coded bits generated. For example, AVC and HEVC compression standards a reduction of 6 QP points yields approximately a factor of two increase in coded bits generated.

Using a QP relationship and plurality of QP point estimations it is possible to extrapolate an estimated picture size for all QP operating points. This is calculated in the Rate Control unit shown in Figure 4.

As can be seen in Figure 4, an output from an Entropy Encoder module, which implements the compression to each picture, is fed into the Rate Control module, and so is used in the determination of a suitable QP value, together with the data encoding metrics from the Inter and Intra Picture Estimators, and the input Target Rate.

The described method of rate control enhancement is particularly useful for scene transitions in video content. In this instance the Rate Control module is able to choose a QP operating point for a new scene that is maintainable for visual stability (and perceptible continuity) (i.e. using the look-ahead metrics it is possible to select a QP that does not require significant correction (relative to neighbouring pictures) through the closed loop proportional controller). Then once the scene has established, the Rate Control may revert to using closed loop proportional controller for target bitrate control.

Figure 1 shows an example of a test sequence “rain fruits” which has high spatial detail but low motion complexity. The sequence has been encoded in HEVC at a fixed QP of 35. As such the coded picture sizes are uniform which a clear relationship between Intra (I) and Inter (P) pictures.

Figure 2 shows an alternative test sequence “sintel” which has medium spatial detail but very high motion complexity (a fight scene). As such the Intra (I) and Inter (P) pictures are nearly indistinguishable in size and the plot is generally erratic.

The sequences in Figures 1 and 2 are typical examples of the type of content that Rate Control algorithms must contend with, and serve to demonstrate the benefits of the described method.

The Rate Controller is typically given a bitrate target for duration of time ‘t’, or a set of targets for a duration of time ‘t\ In addition, time ‘t’ might represent a GOP of a fraction of a GOP (Group of Pictures). A scene cut will typically start with an Intra (I) picture followed by Inter (P/B) pictures, thus using Intra/Inter picture estimation information a suitable starting QP for the new scene can be derived from that information, ultimately in order to minimise the visual impact or artefacts during the scene transition. Note that the equation below is given as an example and other variations may be used.

In the instance where t>n (i.e. the bitrate target interval is greater than the look-ahead delay ‘n’) the following pseudo code demonstrates the QP selection implemented by the Rate Control module:

For simplification ‘t’ and ‘n’ are assumed to be in whole units of picture times. targetBits = £ targets over time Ί ’ ipRatio = Intra Picture Estimate QP 1 / Average (Inter Picture Estimates QP 1 for n-1 pictures) ipRatio is an average ratio of Intra Picture size compared to Inter Picture sizes for the new scene. QP1 is assumed to be a middle QP estimate that is representative of the whole QP range. targetlntraSize = targetBits / (1 + ((t-1)/ipRatio)) targetlntraSize assumes that the ratio of Intra to Inter pictures persists for time ‘t’ pictures such that it is possible to predict how bit the new scene cut Intra picture can be.

Then, by using the interpolated range of Intra Picture Estimates QP points described previously, it is possible to translate the targetlntraSize into a QP value to use in the main encoder algorithm.

An additional benefit of the Inter Picture Estimator data is that it may also be used to detect scene cuts in advance of encoding. As in general a scene cut in video content represents a visual discontinuity which is reflected as a very high value in the overall picture score. Figure 7 shows a plot of Inter Picture Estimator scores for a sequence of pictures that contain scene cuts. Typically a single picture with a score greater than a threshold X would be considered a scene cut. With this knowledge the QP value can be suitably controlled, and maintain the bitrate requirements as well as VQ, by ensuring that the transition is smoothly effected.

The above described method maybe carried out by any suitably adapted or designed hardware. Portions of the method may also be embodied in a set of instructions stored on a computer readable medium, which when loaded into a computer, Digital Signal Processor (DSP) or similar, causes the computer or data processor to implement the described method.

Equally, the method may be embodied as a specially programmed or hardware designed integrated circuit (for example an FPGA or ASIC), or other circuitry, which operates to carry out the described method when loaded into said integrated circuit. The integrated circuit may be formed as part of a general purpose computing device, such as a PC, and the like, or it may be formed as part of a more specialised device, such as a hardware video encoder or the like. Such circuitry may be incorporated into a television or visual display unit, for example.

Claims

1. A method of video compression comprising determining a picture encoding metric of a picture to be encoded of a video, the method comprising providing the data encoding metric as an input feed to a bitrate controller, and the method further comprising regulating the extent of compression based at least in part on the data encoding metric.

2. A method as claimed in claim 1 in which the picture encoding metric comprises a measure of a data cost in encoding a picture.

3. A method as claimed in claim 1 or claim 2 in which the picture encoding metric is an estimated value or a predicted value.

4. A method as claimed in any preceding claim which comprises determining an inter-picture data encoding metric of a picture to be encoded relative to a previous picture of a sequence of pictures of a video.

5. A method as claimed in any preceding claim in which the step of determining the data encoding metric comprises determining a difference value between at least one previous picture and a picture to be encoded.

6. A method as claimed in any preceding claim wherein the step of determining the data encoding metric comprises determining an extent or measure of correlation between the picture to be encoded and the previous picture.

7. A method as claimed in any previous claim in which the picture encoding metric is determined relative to at least one or multiple previous pictures of the video.

8. A method as claimed in any preceding claim which comprises determining an optimum compression control parameter to regulate the compression, which takes account of a bitrate constraint.

9. A method as claimed in claim 8 in which the compression control parameter comprises a quantisation parameter.

10. A method as claimed in any preceding claim which comprises determining predicted resulting picture encoding metrics for different compression control parameters of a picture to be encoded.

11. A method as claimed in claim 10 which comprises selecting an optimum compression control parameter using the multiple compression control parameters to determine an optimum compression parameter.

12. A method as claimed in claim 11 in which the step of selecting an optimum compression parameter comprises interpolating from determined compression control parameter values.

13. A method as claimed in any preceding claim in which the data encoding metric is determined in advance of the picture encoding.

14. A method as claimed in any preceding claim in which the data encoding metric provides a measure of encoding complexity of a picture.

15. A method as claimed in any preceding claim in which the data encoding metric provides a measure of data magnitude of encoding a picture.

16. A method as claimed in any preceding claim which comprises generating picture encoding metrics on an inter picture basis for a picture to be encoded.

17. A method as claimed in any preceding claim which comprises using data from a motion estimator, operative to determine motion vectors that describe the transformation from one picture to a subsequent picture, for use in determining the picture encoding metric.

18. A method as claimed in any preceding claim wherein the picture encoding metric is a measure of approximate entropy per picture.

19. A method as claimed in any preceding claim in which the step of regulating the extent of compression comprises using a combination of an inter picture encoding metric and an intra picture encoding metric.

20. A method as claimed in any preceding claim which comprises determining a measure of current rate buffer fill.

21. A method as claimed in any preceding claim which is a single-pass video compression method.

22. A method as claimed in any preceding claim comprising determining whether a picture encoding metric of a picture to be encoded exceeds a threshold value.

23. A method as claimed in claim 20 comprising determining a scene cut in the video as a result of the threshold value being exceeded.

24. A video compression apparatus comprising a picture estimator, the picture estimator arranged to provide a picture encoding metric of picture to be encoded, and the apparatus comprises a bitrate controller for controlling the data encoding rate, wherein the picture estimator arranged to input the picture encoding metric into the bitrate controller, and the bitrate controller arranged to determine an extent of compression based at least in part on the picture encoding metric.

25. A method substantially as herein described, with reference to the drawings.

26. A video compression apparatus substantially as herein described, with reference to the drawings.