CN1674677A - Improved FGS coding method and coder/decoder thereof - Google Patents

Abstract

The invention relates to an improved FGS video coding method and codec. Its method is based on the FGS structure proposed by MPEG-4, which has carried out motion compensation on the basic layer, and also makes motion compensation on the enhancement layer, forming a MC+FGS structure. This structure greatly improves the efficiency of the FGS structure video coding scheme. On the basis of the MC+FGS structure, extend its structure to the space-time domain, and obtain a video coding scheme with the structure of MC+FGSST, which not only obtains the fine and scalable ability of hybrid time domain + space domain + SNR, but also Get higher coding efficiency. The invention designs a codec for the MC+FGSST structure, and develops a new algorithm for how to determine the number of bit planes used for motion compensation in the coder. Experimental results show that the encoding performance of the MC+FGSST structure based on the enhancement layer for motion compensation proposed by the present invention is significantly better than the original FGSST structure for motion compensation using the base layer.

Description

Improved FGS Video Coding Method and Its Codec

technical field

The invention relates to a video coding method and a codec applicable to network streaming media, in particular to an improved FGS video coding method and a codec.

Background technique

In recent years, network streaming video communication on the Internet has become more and more popular. Scalable coding is an effective method to solve the constant fluctuation of network bandwidth in Internet streaming video applications. The traditional SNR scalable coding scheme is also layered, but the bit rate of each layer is pre-determined during coding, so it can only provide fixed grading or rough scalability, and cannot finely match the network. Immediate bandwidth, and the image quality transition is not smooth enough at the critical point where the code rate of the enhancement layer jumps. In order to obtain a finely scalable coded stream, the FGS coding scheme is adopted in the MPEG-4 standard. MPEG-4 FGS technology is based on video coding with scalable image quality, which compresses the original video sequence into two code streams - basic layer code stream and enhancement layer code stream. The base layer provides the lowest image quality acceptable to the user. The enhancement layer is coded with bit-plane technology to provide embedded SNR scalability. It allows the interception and transmission of any code rate for the code stream of the enhancement layer. The more code streams received by the user, the better the image quality, so it provides Very fine image quality gradability. In terms of robustness, since the bit rate of the base layer is lower than the minimum available bandwidth of the network, the probability of packet loss in the base layer stream is greatly reduced. It is well guaranteed, and different levels of error recovery mechanisms can be used for the enhanced layer code stream according to the specific requirements of the application. However, the image quality SNR fine gradability and robustness obtained by the FGS technology are at the cost of the image quality degradation after user decoding, that is to say, the video coding efficiency is reduced. This is mainly because the structure of FGS uses the image reconstructed from the base layer as the motion reference frame, which fails to remove the temporal correlation of the enhancement layer.

In order to improve the coding efficiency of FGS technology, Wu Feng et al. proposed a PFGS (Progress FGS) technology. PFGS compresses the original sequence into a base layer and multiple (3-4) enhancement layers. The reference frame used for base layer coding is the image reconstructed from the base layer, while the reference frame used for enhancement layer coding is from one or more The reconstructed image in the enhancement layer. Since the quality of the image reconstructed from the enhancement layer is higher than that of the image reconstructed from the base layer, using the reconstructed image of the enhancement layer as a reference frame improves the prediction accuracy, that is, improves the coding efficiency. The advantage of PFGS is that it takes into account the anti-error recovery capability of the code stream while improving the coding efficiency. Its disadvantage is that the implementation structure is complex, the calculation complexity is high, and more physical caches and more calculation time are required.

The bit stream encoded by FGS technology video only provides the SNR fine grading capability. However, in different application scenarios, different scalability capabilities may be required, such as time-domain scalability, space-domain scalability, and mixed time-space domain scalability. Therefore, it is necessary to extend the SNR fine-scale scalability provided by the FGS structure to the time-space domain, and obtain a FGSST structure (FGS spatial temporal) that mixes SNR and time-space domains, so as to realize the fine-scale scalability of mixed SNR+space domain+time domain. Since the FGSST structure is extended from the FGS structure, the FGSST structure inherits the defect of the FGS structure, that is, the coding efficiency is low. Therefore, it is necessary to find an improved FGS structure. This structure must not only improve the coding efficiency, but also be simple in structure implementation, so that the improved FGSST structure obtained after extending to the space-time domain will not be difficult to be practical due to the overly complex structure.

Contents of the invention

The purpose of the present invention is to provide an improved FGS video coding method and its codec, which can provide multiple selectable bit streams for network stream video transmission, and these selectable bit streams all have fine and scalable SNR Ability, users can obtain the best video quality allowed by the network according to their own network conditions.

For achieving the above object, design of the present invention is as follows:

First of all, the present invention proposes an improved single-loop MC+FGS structure based on the enhancement layer for motion compensation. Compared with the original FGS structure, the improved single-loop MC+FGS structure has higher coding efficiency; compared with PFGS, the improved The structure of single-loop MC+FGS is simpler, the physical overhead is smaller, and the calculation time is shorter. The original FGS structure uses the image reconstructed from the base layer as a reference frame in the coding process. Since the quality of the base layer image is poorer than that of the enhancement layer image, the motion prediction is not accurate enough, resulting in low coding efficiency. Obviously, if the reconstructed enhancement layer image is used as a motion reference, the coding efficiency will definitely be greatly improved. A single-loop MC+FGS structure proposed by the present invention is based on this design idea. MC+FGS compresses the original video into a base layer code stream and an enhancement layer code stream. Both the base layer code stream and the enhancement layer code stream are encoded using images reconstructed from the extended base layer (base layer + enhancement layer) ( The basic image reconstructed from the base layer code stream + the residual image after decoding the enhancement layer code stream) is used as a motion reference. Compared with PFGS, the coding efficiency of MC+FGS is higher than that of PFGS. However, a defect of the MC+FGS structure is that when decoding, if the bandwidth of the user network is not enough to allow the user to receive all the enhancement layers, the MC+FGS structure is more likely to cause image quality degradation than PFGS, and this weakening will continue to affect to the prediction reference for subsequent frames until the end of the current GOP, resulting in prediction drift. In order to solve this problem, the present invention takes some measures to suppress the defects of this structure.

Secondly, on the basis of the proposed improved MC+FGS structure, the present invention extends the MC+FGS structure to the space-time domain, thereby obtaining a mixed MC+FGSST structure. The MC+FGSST structure not only obtains higher coding efficiency, but also provides fine and scalable capabilities of mixed time domain + air domain + SNR. Users can selectively configure according to the specific conditions of their own network bandwidth to obtain what they need, such as better Image quality, higher image resolution (spatial domain), higher video frame rate (temporal domain) and other services.

Finally, the present invention designs a codec for the proposed MC+FGSST structure, and develops a new algorithm for how to determine how many bit planes are needed for motion compensation in the coder. It has a very good effect in the application occasion.

According to above-mentioned inventive design, the present invention adopts following technical scheme:

An improved FGS video coding method, based on the FGS structure that the base layer has been motion compensated based on the MPEG-4 suggestion, is characterized in that the enhancement layer is also motion compensated to form a single-loop MC+FGS structure; and adopts the following three One measure suppresses the defects of the monocyclic MC+FGS structure:

1) In order to control the prediction wine drift generated by the decoder at low bit rate, for the I frame and P frame that will affect the prediction accuracy, the number of bit planes used for motion reference during encoding is 2 to 3 bit planes. For B frames that affect the accuracy of subsequent motion parameters, use as many bit planes as possible for motion reference during encoding to improve encoding efficiency;

2) When packet loss or error occurs in the enhancement layer, in order to reduce its impact on the image quality of the base layer, a frame memory is added in the single-ring MC+FGS structure to save the image reconstructed from the base layer, to obtain a conservative image quality equivalent to that in the original FGS structure;

3) The base layer selectively uses the enhancement layer to reconstruct the image, or the base layer reconstructs the image as a motion reference, depending on the specific situation, to improve the error resistance of the coded stream; for a more reliable network, such as a local area network, then The base layer uses the reconstructed image of the enhancement layer as motion reference to improve coding efficiency; for general networks, such as the Internet, the reconstructed image of the base layer and the reconstructed image of the enhancement layer are basically mixed for motion reference.

On the basis of the above-mentioned single-loop MC+FGS structure, its structure is extended to the time-space domain to form the MC+FGSST structure, and at the same time, the fine gradability of time domain + space domain + SNR is obtained; the time-domain gradability is achieved through the following two It can be obtained by any of the following methods: ①Introduce a new independent time-domain FGST enhancement layer to provide the bit stream of the time-domain FGST enhanced frame; ②Modify the original FGS layer to the FGST shared layer, so that the original FGS layer only provides SNR enhancement A single bitstream is extended to a mixed bitstream containing two bitstreams. These two bit streams provide SNR enhanced bit streams for FGS frames and SNR enhanced bit streams for FGST frames respectively; for the spatial domain scalability, a new FGSS spatial domain enhancement layer is introduced.

A kind of codec adopting above-mentioned improved video encoding method, comprises encoder and decoder, is characterized in that encoder is made of three sub-encoders, and three sub-encoders share most modules; Three sub-encoders are:

1. Encoder of this layer: adopts traditional DCT technology encoding based on motion compensation, which encodes the residual of the original frame with low spatial resolution and the reconstructed video frame with low spatial resolution into the base layer bit stream;

2. Time-domain FGST enhancement layer sub-encoder: use bit-plane technology to encode the DCT residual coefficients after motion compensation. The residual system consists of two parts: one is the DCT residual coefficients of the FGS frame for SNR enhancement, and the other is is the DCT residual coefficient of the FGST frame;

3. Space-domain FGSS enhancement layer encoder: use bit-plane technology to encode the DCT residual coefficients of the motion-compensated FGSS frame;

The symbols in the encoder are defined as follows:

LSPi: Low spatial resolution motion prediction images, including prediction images of FGS frames and prediction images of FGST frames

LSd: The original low spatial resolution image and the residual image of LSPi

LSD: DCT residual coefficient obtained after LSd is transformed by DCT

LSTD: DCT residual coefficients for low spatial resolution FGST frames

LSBD: DCT residual coefficients for low spatial resolution FGS frames

LSBDR: Low spatial resolution residual image reconstructed from the base layer coded code stream

LSBR: Reconstructed prediction reference image for the next low spatial resolution FGS frame

LSTR: Reconstructed prediction reference image for the next low spatial resolution FGST frame

HSPi: High Spatial Resolution Motion Prediction Imagery

HSPd: original high spatial resolution image and HSPi residual image

HSPD: DCT residual coefficient of the high spatial resolution image obtained by HSPd after DCT transformation

HSBR: Reconstructed prediction reference image for next high spatial resolution FGSS frame

BMVs: Base layer motion vectors for FGS frames

TMVs: Enhancement layer motion vectors for FGST frames

SMVs: enhancement layer motion vectors for FGSS frames

The design of the decoder is based on the structure of the encoder, which is used correspondingly with the encoder;

The symbols in the decoder are defined as follows:

HSRD: Residual images of high spatial resolution FGSS frames after decoding for display

LSRD: SNR-enhanced residual image of low spatial resolution FGS frames after decoding for display

LSTRD: Residual images of low spatial resolution FGST frames after decoding for display

HSRPD: Residual images of high spatial resolution FGSS frames after decoding for motion reference

LSPRD: SNR-Enhanced Residual Images of Low Spatial Resolution FGS Frames After Decoding for Motion Reference

LSTPRD: Residual images of low spatial resolution FGST frames after decoding for motion reference

LSBR: Reconstructed prediction reference image for the next low spatial resolution FGS frame

LSTR: Reconstructed prediction reference image for the next low spatial resolution FGST frame

HSBR: Reconstructed prediction reference image for next high spatial resolution FGSS frame

BMVs: Base layer motion vectors for FGS frames

TMVs: Enhancement layer motion vectors for FGST frames

SMVs: Enhancement layer motion vectors for FGSS frames.

Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages: the improved FGS video coding method provided by the present invention is based on the FGS structure, and motion compensation is also performed on the enhancement layer, forming a single-loop MC+FGS structure , which improves the accuracy and coding efficiency of the motion reference; on the basis of the structure of the single-loop MC+FGS, its structure is extended to the space-time domain, and the MC+FGSST structure is obtained. In addition to grading capability, higher coding efficiency is obtained. The invention also designs a set of encoder for MC+FGSST structure, and develops a new algorithm for how to determine the number of bit planes used for motion compensation in the encoder. Users can obtain the best video quality allowed by the network according to their own network conditions.

Description of drawings

Figure 1 is a comparison between the single-ring MC+FGS structure and the original FGS structure. Figure (a) in the figure is the original structure diagram of FGS, and figure (b) is the structure diagram of the single-ring MC+FGS.

Fig. 2 is a block diagram of an encoder with a single-loop MC+FGS structure.

Figure 3 is a block diagram of the improved single-ring MC+FGS structure.

Figure 4 is a schematic diagram of the structure of MC+FGSST.

Figure 5 is a schematic diagram of the CODEC implementation of the MC+FGSST structure, in which Figure (a) is a block diagram of an encoder, and Figure (b) is a block diagram of a decoder.

Figure 6 is a coding structure diagram of the simulation experiment.

Figure 7 is a comparison of the PSNR values of the MC+FGSST structure and the FGSST structure. In the figure (a) is a comparison of the FGS frame PSNR of the two structures, and figure (b) is a comparison of the FGST frame PSNR of the two structures, and the figure ( c) The comparison of FGSS frame PSNR of the two structures is.

Detailed ways

The MC+FGSST structure video coding method based on the enhancement layer for motion compensation of the present invention will be analyzed and described below in three sections in conjunction with the accompanying drawings. In the following expressions, the FGS frame represents a non-temporal enhanced frame enhanced on the SNR, the FGST frame represents the enhanced frame in the temporal resolution, the FGSS frame represents the enhanced frame in the spatial resolution, and the FGSST frame Represents frames enhanced at both temporal and spatial resolution.

1. Single ring MC+FGS structure

Accompanying drawing 1 (a) is the original FGS structure suggested by MPEG-4, and the gray-scale part in the figure represents the part that is actually transmitted and received for enhancement. It can be seen from the figure that the original FGS only performs motion compensation on the base layer, and removes the correlation in the time domain of the base layer, but does not use any processing to remove the correlation in the time domain for the enhancement layer, which will inevitably reduce the compression efficiency. For this reason, this paper proposes a single-loop MC+FGS structure that also performs motion compensation on the enhancement layer. The idea is to use more accurate motion references to obtain more accurate predictions to obtain more accurate motion compensation, thereby improving coding efficiency. Accompanying drawing 1 (b) is the schematic diagram of the MC+FGS structure, the base layer and the enhancement layer (called the extended base layer) are used for motion prediction and reference to the frames of the following base layer. The advantages of this structure are: 1) Since the extended base layer is used as the motion reference, the accuracy and coding efficiency of the motion reference are improved, so it can make up for the defect that the original FGS will reduce the coding efficiency; 2) the complexity is low, and the encoder The structure is simple and the realization is easy. Fig. 2 is a block diagram of an encoder with a single-loop MC+FGS structure. Since both the base layer code stream and the enhancement layer code stream use the extended base layer for motion reference and motion compensation, this single-loop structure also has disadvantages: 1) During decoding, packet loss in the enhancement layer will affect the image of the base layer frame quality. 2) When decoding, when the bit rate allowed by the user network bandwidth is lower than the bit rate of the extended base layer, the image quality will be weakened, and this weakening will continue to affect the prediction reference for subsequent frames until the end of the current GOP, thus Generate prediction drift (drift).

The defects of the monocyclic structure can be suppressed by taking the following measures:

●In order to control the prediction drift generated by the decoder when the bit rate is low, for the I frame and P frame that will affect the prediction accuracy, the number of bit planes used for motion reference during encoding should not be too much (such as 2-3 bit planes) ), appropriately sacrificing coding efficiency in exchange for better error resistance. For B frames that do not affect the accuracy of subsequent motion references, as many bit planes as possible are used for motion references during encoding to improve encoding efficiency.

●When packet loss or error occurs in the enhancement layer, in order to reduce its impact on the image quality of the base layer, a frame memory is added in the single-ring structure to save the reconstructed image from the base layer to obtain an image equivalent to the original The conservative image quality in the FGS structure can reduce its impact on the error accumulation caused by the subsequent prediction reference. In this way at least the image quality of the original FGS can be obtained even when the allowable bit rate is very low.

●Using the improved single-loop MC+FGS structure shown in Figure 3, the base layer selectively uses the reconstructed image of the enhancement layer or the reconstructed image of the base layer as shown by the dotted line in the figure as a motion reference depending on the specific situation. For error-prone networks such as wireless networks, the base layer uses images reconstructed from the base layer as motion references to improve the error resistance of the coded stream; while for more reliable networks such as local area networks, the base layer uses images from enhanced The image reconstructed from the base layer is used as a motion reference to improve coding efficiency; for a general network such as the Internet, the base layer mixes the image reconstructed from the base layer and the image reconstructed from the enhancement layer as a motion reference, specifically how to mix and use the present invention No in-depth discussion.

The above three measures to suppress the weak error resistance of the single-loop MC+FGS structure are at the cost of sacrificing part of the coding efficiency and computational complexity, and can be selected according to the specific application environment to meet the needs of users.

In order to verify the coding performance of single-loop MC+FGS, the following simulation experiments are carried out, and the experimental comparison is made with the FGS scheme of MPEG-4 (ie, the original FGS in the table). The test sequence of the experiment is Foreman, Coastguard, Akiyo (CIF format) sequence, using TM5 rate control mode, the frame rate is 10f/s. The GOP structure is NI=1, N _P =4, and N _B =12. Set the base layer bit rate R _BL =128kbits/s, and adopt the MPEG-4 video compression coding scheme. The code rate of the enhancement layer can be intercepted at any code rate according to the available bandwidth of the network. In this experiment, the code rates of the enhancement layer are intercepted at 64kbits/s, 172kbits/s, ... until 872kbits/s. Table 1 lists the PSNR value of the Y luminance component of the fifth frame of the three sequences in the original FGS structure and the single-loop MC+FGS structure as the bit rate of the code stream changes. It should be pointed out that this PSNR value is the best image quality PSNR (non-reconstructed base layer image quality), that is, the PSNR value of the image obtained when all code streams of the enhancement layer are received and correctly decoded. It can be seen from the experimental results in the table that the PSNR of the proposed scheme can be improved by 1.23-5.83dB for Forman, 0.66-3.05dB for Coastguard, and 4.43-9.64dB for Akiyo compared with the MPEG-4 FGS scheme.

Table 1 PSNR (dB) values of Forman, Coastguard, and Akiyo sequence Y luminance components at different code rates bit rate (kbits/s) Foreman Coastguard Akiyo Raw FGS MC+FGS Raw FGS MC+FGS Raw FGS MC+FGS 192 37.40 42.02 32.34 33.27 43.24 52.04 300 40.31 43.87 33.24 34.57 46.88 56.52 450 41.36 47.20 35.22 36.95 47.95 56.91 600 43.04 47.12 36.42 37.08 49.70 56.91 1000 46.39 47.62 38.84 41.89 52.54 56.91

2. MC+FGSST structure

The structure of MC+FGS is the same as that of FGS, which can only provide SNR fine grading. In order to make the MC+FGS structure obtain the fine scalability of temporal+spatial+SNR at the same time, more enhancement layers must be introduced. Time-domain scalability can be obtained in two ways: 1) Introducing a new independent time-domain FGST enhancement layer to provide the bit stream of time-domain FGST enhanced frames; 2) Modifying the original FGS layer to a FGST shared layer, making the original FGS A single bitstream providing only SNR enhancement is extended to a mixed bitstream containing two bitstreams, namely a bitstream providing SNR enhancement for FGS frames and a bitstream providing SNR enhancement for FGST frames. The introduction of an independent enhancement layer provides very good flexibility for the decoding of the server and the decoder. It does not need to decode all FGS frames as a prediction reference for FGST frame decoding. Therefore, the image quality of the FGS enhancement layer and the time-domain FGST enhancement frame are independent of each other and do not affect each other. But it needs to transmit one more enhancement layer, which will increase the complexity of the structure implementation, and the server needs to provide the client with one more time-domain enhancement layer bit stream service, which increases the burden on the server. For spatial scalability, a new FGSS spatial enhancement layer has to be introduced because this enhancement layer has a different spatial resolution than the other layers. Considering the complexity of the entire MC+FGSST structure implementation, we implement time-domain scalability by modifying the FGS layer into an FGST shared layer. In this way, the structure of MC+FGSST is shown in Figure 4, and the whole structure is divided into three layers. The basic layer provides images with low SNR, low frame rate, and low spatial resolution; the first enhancement layer provides images with enhanced SNR and FGST frames with SNR fine scalability; the second enhancement layer provides enhanced SNR fine scalability Scalable images of FGSS frames, and as shown by the dotted line in Figure 4, the enhancement layer can be further selectively configured so that this layer also provides images of FGSST frames with SNR fine scalability. The MC+FGSST structure not only obtains the fine gradability of mixed time domain + space domain + SNR, but also overcomes the defect of low coding efficiency of the original FGS structure and obtains higher coding efficiency. Not only that, the MC+FGSST structure adopts a simplified design idea, so it is not complicated to implement and has good practicability.

3. CODEC design of MC+FGSST structure

According to the MC+FGSST structure diagram in Figure 4, we designed the CODEC shown in Figure 5 for MC+FGSST. The symbols in the encoder in the figure are defined as follows:

LSPi: Low spatial resolution motion prediction images, including prediction images of FGS frames and prediction images of FGST frames

LSd: The original low spatial resolution image and the residual image of LSPi

LSD: DCT residual coefficient obtained after LSd is transformed by DCT

LSTD: DCT residual coefficients for low spatial resolution FGST frames

LSBD: DCT residual coefficients for low spatial resolution FGS frames

LSBDR: Low spatial resolution residual image reconstructed from the base layer coded code stream

LSBR: Reconstructed prediction reference image for the next low spatial resolution FGS frame

LSTR: Reconstructed prediction reference image for the next low spatial resolution FGST frame

HSPi: High Spatial Resolution Motion Prediction Imagery

HSPd: original high spatial resolution image and HSPi residual image

HSPD: DCT residual coefficient of the high spatial resolution image obtained by HSPd after DCT transformation

HSBR: Reconstructed prediction reference image for next high spatial resolution FGSS frame

BMVs: Base layer motion vectors for FGS frames

TMVs: Enhancement layer motion vectors for FGST frames

SMVs: enhancement layer motion vectors for FGSS frames

The design of the decoder is based on the structure of the encoder, which is used correspondingly with the encoder;

The symbols in the decoder are defined as follows:

HSRD: Residual images of high spatial resolution FGSS frames after decoding for display

LSRD: SNR-enhanced residual image of low spatial resolution FGS frames after decoding for display

LSTRD: Residual images of low spatial resolution FGST frames after decoding for display

HSRPD: Residual images of high spatial resolution FGSS frames after decoding for motion reference

LSPRD: SNR-Enhanced Residual Images of Low Spatial Resolution FGS Frames After Decoding for Motion Reference

LSTPRD: Residual images of low spatial resolution FGST frames after decoding for motion reference

LSBR: Reconstructed prediction reference image for the next low spatial resolution FGS frame

LSTR: Reconstructed prediction reference image for the next low spatial resolution FGST frame

HSBR: Reconstructed prediction reference image for next high spatial resolution FGSS frame

BMVs: Base layer motion vectors for FGS frames

TMVs: Enhancement layer motion vectors for FGST frames

SMVs: enhancement layer motion vectors for FGSS frames

The encoder includes three sub-encoders, and it can be seen from Figure 5(a) that the three sub-encoders share most of the modules. The base layer encoder adopts the traditional DCT technique based on motion compensation to encode the residual of the low spatial resolution original video frame and the low spatial resolution reconstructed video frame into the base layer bit stream. Practice has proved that the coding efficiency of bit plane is higher than that of run-level, and it provides embedded fine scalability capability for coded stream. Therefore, the time-domain FGST enhancement layer sub-encoder uses bit-plane technology to encode the motion-compensated DCT residual coefficients. The residual coefficients are composed of two parts: one is the DCT residual coefficients of the FGS frame for SNR enhancement, and the other is The DCT residual coefficients of the FGST frame. Bit-plane technology provides high coding efficiency and SNR fine-scale scalability for FGST frames. The space-domain FGSS enhancement layer encoder also uses bit-plane technology to encode the DCT residual coefficients of the motion-compensated FGSS frame, which also provides high coding efficiency and fine SNR scalability for the FGSS frame. Since the three sub-encoders share most of the modules, the encoder we designed saves a lot of physical overhead and reduces the cost. The two enhancement layer bit streams encoded by the MC+FGSST structure provide space-domain scalability and time-domain + SNR fine-scale scalability respectively. Users can selectively configure to receive an enhancement layer bit stream according to the specific conditions of their own network bandwidth. Or receive two enhancement layer bit streams at the same time, and combine and decode with the basic layer bit stream that must be received to obtain the service required by the user.

In the block diagram of the encoder shown in Figure 5, how to determine the number of bit-planes used for motion compensation is a key issue. If too many bit-planes are used, the user cannot fully receive all the If the bitstream data used for motion compensation in the enhancement layer is used, the quality of the decoded image will be weakened, and this weakened image will continue to affect the motion prediction accuracy of subsequent frames, thereby causing prediction drift (drift). Therefore, if the target code rate of the service object provided by the server is low, it is not appropriate to use too many bit planes for I frames and P frames that will affect the accuracy of subsequent prediction references (we recommend using 2-3 bit planes); and for B frame, the normal number of bit-planes is used for motion compensation to improve coding efficiency. If the target code rate of the service object provided by the server is relatively high, all I, P or B will use the normal number of bit planes for motion compensation. A key issue now is how to determine the normal number of bit-planes used for motion compensation. To this end, we have developed a set of algorithms to solve this technical problem. A prerequisite for its work is that we already know the available network bandwidth of all current users accessing the server. In the encoder, the current available bandwidth of all users passes through the module "Bandwidth Estimation and Available Bandwidth Calculation" to obtain. Another advantage of this algorithm is that it not only determines the number of bit planes N _bitplaneused used for motion compensation, but also determines the total number of blocks (blocks) that can be used for motion compensation in the N _bitplaneused +1 plane M _blockused . Since more enhancement layer bit streams are used for motion compensation, the accuracy of motion compensation is further improved, and thus the coding efficiency is further improved.

Assuming that there are currently N users accessing the streaming server at the same time, the current available bandwidth of these N users can be known from "Bandwidth Estimation and Available Bandwidth Calculation" as: {B ₁ , B ₂ , B ₃ ,...B _N }, take The minimum bandwidth B _min of all N users, B _min is the target code rate for motion compensation.

algorithm:

           
Nbits=0; //Nbits: the number of bits entropy-encoded by the current bit-plane encoding

Nbitplaneused=Nmax; //Nbitplaneused: the number of bit planes used for motion compensation

Mblockused=Nblock; //Mblockused: the number of blocks used for motion compensation in the lowest bit plane
T＝Min{B1, B2, B3,...BN} //T: Target bit rate for motion compensation

for(i=0; i<Nmax; i++){ //Nmax: maximum number of bit planes

for(j=0; j<Nblock; j++){ //Nblock: the total number of blocks in the frame image

if(Nbits<(T-RBL)/f //RBL: base layer code rate, f. frame rate

Nbits+＝BlockBitplaneEncoder(); //BlockBitplaneEncoder(): The return value is the coded bitplane

The number of bits spent by the current "block" on the layer

Else {

Mblockused=j;

Break;

}

}

if(Nbits<(B-RBL)/f){

Nbitplaneused=i;
        <!-- SIPO <DP n="10"> -->
        <dp n="d10"/>
Break;
}
}

        

This algorithm is especially suitable for occasions that require real-time encoding, such as live video broadcasting. For the currently commonly used pre-stored video encoding, the current minimum available network bandwidth of all users in the algorithm can be initialized to a predetermined value. How to determine it depends on the specific application. and network conditions, and will not be discussed in depth here.

In order to verify the coding performance of the MC+FGSST structure, we did the following simulation experiments. Experimentally tested the first 24 frames of Foreman and Coastguard video sequences. The experimental test environment is as follows: High temporal resolution: 30 frames/second Low temporal resolution: 15 frames/second High spatial resolution: CIF (352×288 ) Low spatial domain resolution: QCIF (176×144) base layer bit rate: R _BL =75kbits/sMC+FGST enhancement layer bit rate: R _FGST =150kbit/s where 90kbits/s is used for SNR enhancement of FGS frame,

60kbit/s is used for the SNR enhancement MC+FGSS enhancement layer bit rate of FGST frame: R _FGSS =90kbits/s is used for the SNR enhancement of FGSS frame and uses encoder that the present invention proposes to encode, and encoding structure is as shown in accompanying drawing 6, After encoding, a three-layer bit stream is obtained. Then, use the decoder of the corresponding coder that the present invention proposes to the three-layer bit stream that coding obtains again, obtain the PSNR value of FGS frame, FGST frame and FGSS frame after decoding, and these PSNR values and original FGSST (by FGSST) The structure is extended to the space-time domain) and the PSNR value obtained by structure decoding is compared. The experimental results are shown in Figure 7. It can be seen from Figure 7(a)(b)(c) that the coding performance of the MC+FGSST structure is significantly better than that of the FGSST structure as a whole. For the 14th frame of the Coastguard sequence in Figure 7(b), the performance of the MC+FGSST structure is not as good as that of the FGSST structure, which may be caused by the own error of motion estimation; for the 17th frame of the Foreman sequence in Figure 7(c) , the performance of the MC+FGSST structure is not as good as that of the FGSST structure, which may be caused by the prediction distortion caused by the low spatial resolution image upsampling.

Claims (3)

1. An improved FGS video coding method, based on the FGS structure that the base layer has been motion compensated based on MPEG-4 suggestion, is characterized in that the enhancement layer is also motion compensated to form a single loop MC+FGS structure; and adopts the following The above three measures suppress the defects of the monocyclic MC+FGS structure: 1) In order to control the prediction wine drift generated by the decoder at low bit rate, for the I frame and P frame that will affect the prediction accuracy, the number of bit planes used for motion reference during encoding is 2 to 3 bit planes. For B frames that affect the accuracy of subsequent motion parameters, use as many bit planes as possible for motion reference during encoding to improve encoding efficiency; 2) When packet loss or error occurs in the enhancement layer, in order to reduce its impact on the image quality of the base layer, a frame memory is added in the single-ring MC+FGS structure to save the image reconstructed from the base layer, to obtain a conservative image quality equivalent to that in the original FGS structure; 3) The base layer selectively uses the enhancement layer to reconstruct the image, or the base layer reconstructs the image as a motion reference, depending on the specific situation, to improve the error resistance of the coded stream; for a more reliable network, such as a local area network, then The base layer uses the reconstructed image of the enhancement layer as motion reference to improve coding efficiency; for general networks, such as the Internet, the reconstructed image of the base layer and the reconstructed image of the enhancement layer are basically mixed for motion reference. 2. the improved FGS video coding method according to claim 1 is characterized in that on the basis of the described single-ring MC+FGS structure, its structure is extended to the space-time domain, constitutes MC+FGSST structure, obtains time simultaneously Domain + space domain + SNR fine scalability; its time domain scalability is obtained by either of the following two ways: ① Introduce a new independent time domain FGST enhancement layer to provide bit stream of time domain FGST enhanced frame ; ② Modify the original FGS layer to the FGST shared layer, so that the original FGS layer only provides SNR-enhanced single bit stream and expands to a mixed bit stream containing two bit streams. These two bit streams provide SNR enhanced bit streams for FGS frames and SNR enhanced bit streams for FGST frames respectively; for the spatial domain scalability, a new FGSS spatial domain enhancement layer is introduced. 3. A codec adopting the improved FGS video coding method of claim 1, comprising a coder and a decoder, is characterized in that the coder is made of three sub-coders, and the three sub-coders share most of the modules; The sub-encoders are: 1) Base layer encoder: adopt traditional motion compensation-based DCT technology encoding, which encodes the residual of low spatial resolution original frame and low spatial resolution reconstructed video frame into base layer bit stream; 2) Time-domain FGST enhancement layer sub-encoder: bit-plane technology is used to encode DCT residual coefficients after motion compensation. The residual system consists of two parts: one is the DCT residual coefficients of the FGS frame for SNR enhancement, and the other is the DCT residual coefficients for SNR enhancement of the FGS frame. is the DCT residual coefficient of the FGST frame; 3) Spatial FGSS enhancement layer encoder: use bit-plane technology to encode the DCT residual coefficients of the motion-compensated FGSS frame; The symbols in the encoder are defined as follows: LSPi: Low spatial resolution motion prediction images, including prediction images of FGS frames and prediction images of FGST frames LSd: The original low spatial resolution image and the residual image of LSPi LSD: DCT residual coefficient obtained after LSd is transformed by DCT LSTD: DCT residual coefficients for low spatial resolution FGST frames LSBD: DCT residual coefficients for low spatial resolution FGS frames LSBDR: Low spatial resolution residual image reconstructed from the base layer coded code stream LSBR: Reconstructed prediction reference image for the next low spatial resolution FGS frame LSTR: Reconstructed prediction reference image for the next low spatial resolution FGST frame HSPi: High Spatial Resolution Motion Prediction Imagery HSPd: original high spatial resolution image and HSPi residual image HSPD: DCT residual coefficient of the high spatial resolution image obtained by HSPd after DCT transformation HSBR: Reconstructed prediction reference image for next high spatial resolution FGSS frame BMVs: Base layer motion vectors for FGS frames TMVs: Enhancement layer motion vectors for FGST frames SMVs: enhancement layer motion vectors for FGSS frames The design of the decoder is based on the structure of the encoder, which is used correspondingly with the encoder; The symbols in the decoder are defined as follows: HSRD: Residual images of high spatial resolution FGSS frames after decoding for display LSRD: SNR-enhanced residual image of low spatial resolution FGS frames after decoding for display LSTRD: Residual images of low spatial resolution FGST frames after decoding for display HSRPD: Residual images of high spatial resolution FGSS frames after decoding for motion reference LSPRD: SNR-Enhanced Residual Images of Low Spatial Resolution FGS Frames After Decoding for Motion Reference LSTPRD: Residual images of low spatial resolution FGST frames after decoding for motion reference LSBR: Reconstructed prediction reference image for the next low spatial resolution FGS frame LSTR: Reconstructed prediction reference image for the next low spatial resolution FGST frame HSBR: Reconstructed prediction reference image for next high spatial resolution FGSS frame BMVs: Base layer motion vectors for FGS frames TMVs: Enhancement layer motion vectors for FGST frames SMVs: Enhancement layer motion vectors for FGSS frames.

Images