HK1229105B - Methods and apparatus for incorporating video usability information within a multi-view video coding system - Google Patents

Description

Method and apparatus for incorporating video availability information into a multi-view video coding system

This application is a divisional application of the invention patent application No. 200880119540.4 filed on September 16, 2008, entitled "Method and Apparatus for Incorporating Video Usability Information (VUI) into a Multi-View Video (MVC) Coding System."

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Serial No. 60/977,709, filed October 5, 2007, which is hereby incorporated by reference in its entirety. In addition, this application is related to a commonly assigned non-provisional application, attorney docket No. PU080155, entitled “METHODS AND APPARATUS FOR INCORPORATIONG VIDEO USABILITY (VUI) WITHIN A MULTI-VIEW VIDEO (MVC) CODING SYSTEM,” filed concurrently with this application, which also claims the benefit of U.S. Provisional Application Serial No. 60/977,709, filed October 5, 2007.

Technical Field

The present principles relate generally to video encoding and decoding, and more particularly to methods and apparatus for incorporating video usability information (VUI) into multi-view video coding (MVC).

Background Art

The International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4 (MPEG-4) Part 10 Advanced Video Coding (AVC) standard/International Telecommunication Union-Telecommunication Sector (ITU-T) H.264 Recommendation (hereinafter referred to as the "MPEG-4 AVC standard") specifies the syntax and semantics of the Video Usability Information (VUI) parameter of the sequence parameter set. The VUI includes the following information: aspect ratio, over-scanning, video signal type, chroma position, timing, Network Abstraction Layer (NAL) Hypothetical Reference Decoder (HRD) parameters, Video Coding Layer (VCL) Hypothetical Reference Decoder parameters, bitstream constraints, etc. The VUI provides additional information of the corresponding bitstream to allow a wider range of user applications. For example, in the bitstream constraint information, the VUI specifies: (1) whether motion exceeds the picture boundary; (2) the maximum bytes per picture; (3) the maximum bits per macroblock; (4) the maximum motion vector length (horizontal and vertical); (5) the number of reordered frames; and (6) the maximum decoded frame buffer size. When a decoder sees this information, instead of using the "level" information to set decoding requirements (which are typically higher than what the bitstream actually requires), the decoder can tailor its decoding operations based on tighter limits.

Multi-view video coding (MVC) is an extension of the MPEG-4 AVC standard. In multi-view video coding, multi-view video images can be encoded by using the correlation between the views. Among all views, one view is a base view, which is compatible with the MPEG-4 AVC standard and cannot be predicted from other views. Other views are called non-base views. Non-standard views can be predictively encoded from the base view and other non-base views. Each view can be subsampled in time. The temporal subset of the view can be identified by the temporal_id syntax element. The temporal level of the view is a representation of the video signal. In the encoded bitstream of the multi-view video, there are different combinations of views and temporal levels. Each combination is called an operation point. The sub-bitstream corresponding to each operation point can be extracted from the bitstream.

Summary of the Invention

These and other deficiencies and shortcomings of the prior art are addressed by the present principles, which are directed to methods and apparatus for incorporating video usability information (VUI) into multi-view video coding (MVC).

According to an aspect of the present principles, there is provided an apparatus comprising an encoder for encoding multi-view video content by specifying video usability information for at least one of respective views, respective temporal levels in a view, and respective operating points.

According to another aspect of the present principles, there is provided a method comprising encoding multi-view video content by specifying video usability information for at least one of respective views, respective temporal levels within a view, and respective operating points.

According to another aspect of the present principles, there is provided an apparatus comprising a decoder for decoding multi-view video content by specifying video usability information for at least one of respective views, respective temporal levels in a view, and respective operating points.

According to another aspect of the present principles, there is provided a method comprising decoding multi-view video content by specifying video usability information for at least one of respective views, respective temporal levels within a view, and respective operating points.

According to another aspect of the present principles, there is provided a coding method comprising: encoding multi-view video content by specifying bitstream limitation information for respective operation points in multi-view video coding, wherein the operation point corresponds to one of different combinations of view and temporal level.

According to another aspect of the present principles, there is provided a decoding method, comprising decoding multi-view video content by decoding bitstream restriction information specified for respective operation points in multi-view video encoding, wherein the operation point corresponds to one of different combinations of view and temporal level.

According to another aspect of the present principles, there is provided a decoding apparatus comprising: a decoder for decoding multi-view video content by decoding bitstream restriction information specified for respective operation points in multi-view video coding, wherein the operation point corresponds to one of different combinations of view and temporal level.

According to another aspect of the present principles, there is provided an apparatus for transmitting a video signal, comprising: means for transmitting multi-view video content by specifying bitstream limitation information for respective operation points in multi-view video coding, wherein the operation point corresponds to one of different combinations of view and temporal level.

According to another aspect of the present principles, there is provided a decoding method, comprising: decoding multi-view video content by decoding bitstream restriction information specified for each temporal level in a view of multi-view video encoding, wherein the bitstream restriction information indicates one or more of the following:

(1) Whether the motion vector exceeds the frame boundary;

(2) Maximum bytes per frame;

(3) Maximum bits per macroblock;

(4) Maximum horizontal and vertical motion vector lengths;

(5) the number of reordered frames; and

(6) Maximum decoded frame buffer size.

These and other aspects, features and advantages of the present principles will become apparent from the following detailed description of example embodiments, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

This principle can be better understood with reference to the following example diagram, where:

1 is a block diagram of an example multi-view video coding (MVC) encoder to which the present principles may be applied, in accordance with an embodiment of the present principles;

FIG2 is a block diagram of an example Multi-View Video Coding (MVC) decoder to which the present principles may be applied, in accordance with an embodiment of the present principles;

3 is a flowchart of an example method for encoding per-view bitstream restriction parameters using the mvc_vui_parameters_extension() syntax element, in accordance with an embodiment of the present principles;

FIG4 is a flowchart of an example method for decoding per-view bitstream restriction parameters using the mvc_vui_parameters_extension() syntax element, in accordance with an embodiment of the present principles;

5 is a flowchart of an example method for encoding bitstream restriction parameters for each temporal level in each view using the mvc_vui_parameters_extension() syntax element, in accordance with an embodiment of the present principles;

6 is a flowchart of an example method for decoding bitstream restriction parameters for each temporal level in each view using the mvc_vui_parameters_extension() syntax element, in accordance with an embodiment of the present principles;

FIG7 is a flowchart of an example method for encoding bitstream restriction parameters for each operation point using the view_scalability_parameters_extension() syntax element, in accordance with an embodiment of the present principles; and

8 is a flowchart of an example method for decoding bitstream restriction parameters for each operation point using the view_scalability_parameters_extension() syntax element, in accordance with an embodiment of the present principles.

DETAILED DESCRIPTION

The present principles are directed to methods and apparatus for incorporating video usability information (VUI) into multi-view video coding (MVC).

This description illustrates the present principles. Thus, it will be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the present principles and are included within its spirit and scope.

All examples and conditional language recited herein are intended for teaching purposes to aid the reader in understanding the present principles and concepts contributed by the inventors to advance the art, and should not be construed as limiting such specifically recited examples and conditions.

In addition, all statements herein reciting principles, aspects, and embodiments of the present principles, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

Thus, for example, those skilled in the art will recognize that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the present principles. Similarly, it will be recognized that any flow charts, flow diagrams, state transition diagrams, pseudo-code, and the like may substantially represent various processes that are embodied in a computer-readable medium and thereby executed by a computer or processor, whether or not such computer or processor is explicitly shown.

The functions of the various elements shown in the figures can be provided by using dedicated hardware and hardware that can execute software in association with appropriate software. When a processor is utilized to provide the functions, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of independent processors some of which can be shared. In addition, the explicit use of the terms "processor" or "controller" should not be interpreted as referring exclusively to hardware that can execute software, but rather can implicitly include, without limitation, digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), and non-volatile memory.

Other conventional and/or custom hardware may also be included. Similarly, any switches shown in the figures are conceptual only. Their functions may be performed through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, as will be more specifically understood from the context, and the specific technique may be selected by the implementer.

In its claims, any element represented as a means for performing a specified function is intended to encompass any means of performing that function, including, for example, a) a combination of circuit elements that perform that function or b) any form of software, such as firmware or microcode, combined with appropriate circuitry for executing the software to perform the function. The present principle defined by such claims resides in the fact that the functionality provided by the various recited components is combined and brought together in the manner required by the claims. Therefore, any component that can provide those functions is considered equivalent to those shown herein.

Reference throughout this specification to "one embodiment" or "an embodiment" of the present principles means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrases "in one embodiment" and "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

It should be appreciated that the use of the terms "and/or" and "at least one of" (e.g., in the context of "A and/or B" and "at least one of A and B") is intended to include selecting only the first listed option (A), only the second listed option (B), or both options (A and B). As another example, in the context of "A, B, and/or C" and "at least one of A, B, and C," such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be readily appreciated by those of ordinary skill in this and related arts, this can be extended to many listed items.

Multi-view video coding (MVC) is a compression framework for encoding multi-view sequences. A multi-view video coding (MVC) sequence is a set of two or more video sequences that capture the same scene from different viewpoints.

As used interchangeably herein, "cross-view" and "inter-view" both refer to pictures belonging to a view other than the current view.

Furthermore, as used herein, "high-level syntax" refers to syntax that appears in a bitstream that resides hierarchically at the macroblock layer. For example, high-level syntax (as used herein) may refer to, but is not limited to, syntax at the slice header level, the supplemental enhancement information (SEI) level, the picture parameter set (PPS) level, the sequence parameter set (SPS) level, and the network abstraction layer (NAL) unit header level.

Furthermore, it should be appreciated that although one or more embodiments of the present principles are described herein with respect to the multi-view video coding extension of the MPEG-4 AVC standard for illustrative purposes, the present principles are not limited to that extension and/or that standard and, thus, the present principles may be utilized with respect to other video coding standards, proposals, and extensions thereof while maintaining the spirit of the present principles.

Additionally, it should be appreciated that although one or more embodiments of the present principles are described herein with respect to bitstream restriction information for illustrative purposes, the present principles are not limited to the use of bitstream restriction information as a type of video availability information, and thus other types of video availability information that may be extended for use with respect to multi-view video coding may also be used in accordance with the present principles while maintaining the spirit of the present principles.

1 , an exemplary multi-view video coding (MVC) encoder is generally indicated by the reference numeral 100. Encoder 100 includes a combiner 105 having an output connected in signal communication with an input of a transformer 110. The output of transformer 110 is connected in signal communication with an input of a quantizer 115. The output of quantizer 115 is connected in signal communication with an input of an entropy encoder 120 and an input of an inverse quantizer 125. The output of inverse quantizer 125 is connected in signal communication with an input of an inverse transformer 130. The output of inverse transformer 130 is connected in signal communication with a first non-inverting input of combiner 135. The output of combiner 135 is connected in signal communication with an input of an intra predictor 145 and an input of a deblocking filter 150. The output of deblocking filter 150 is connected in signal communication with an input of a reference picture storage (of view i) 155. An output of the reference picture storage component 155 is connected in signal communication with a first input of a motion compensator 175 and a first input of a motion estimator 180. An output of the motion estimator 180 is connected in signal communication with a second input of the motion compensator 175.

An output of the reference picture storage (of other views) 160 is connected in signal communication with a first input of a disparity/illumination estimator 170 and a first input of a disparity/illumination compensator 165. An output of the disparity/illumination estimator 170 is connected in signal communication with a second input of the disparity/illumination compensator 165.

The output of the entropy decoder 120 is available as an output of the encoder 100. A non-inverting input of the combiner 105 is available as an input of the encoder 100 and is connected in signal communication with a second input of the disparity/illumination estimator 170 and a second input of the motion estimator 180. An output of the switch 185 is connected in signal communication with a second non-inverting input of the combiner 135 and an inverting input of the combiner 105. The switch 185 includes a first input connected in signal communication with an output of the motion compensator 175, a second input connected in signal communication with an output of the disparity/illumination compensator 165, and a third input connected in signal communication with an output of the intra predictor 145.

The mode decision module 140 has an output connected to the switch 185 for controlling which input is selected by the switch 185 .

2 , an exemplary multi-view video coding (MVC) decoder is generally indicated by the reference numeral 200. The decoder 200 includes an entropy decoder 205 having an output connected in signal communication with an input of an inverse quantizer 210. The output of the inverse quantizer is connected in signal communication with an input of an inverse transformer 215. The output of the inverse transformer 215 is connected in signal communication with a first non-inverting input of a combiner 220. The output of the combiner 220 is connected in signal communication with an input of a deblocking filter 225 and an input of an intra-frame predictor 230. The output of the deblocking filter 225 is connected in signal communication with an input of a reference picture storage component 240 (for view i). The output of the reference picture storage component 240 is connected in signal communication with a first input of a motion compensator 235.

An output of the reference picture storage (of other views) 245 is connected in signal communication with a first input of the disparity/illumination compensator 250 .

An input to the entropy decoder 205 is available as an input to the decoder 200 for receiving the residual bitstream. Furthermore, an input to the mode module 260 is also available as an input to the decoder 200 for receiving control syntax to control which input is selected via the switch 255. Furthermore, a second input to the motion compensator 235 is available as an input to the decoder 200 for receiving motion vectors. Furthermore, a second input to the disparity/illumination compensator 250 is available as an input to the decoder 200 for receiving disparity vectors and illumination compensation syntax.

An output of the switch 255 is connected in signal communication with a second non-inverting input of the combiner 220. A first input of the switch 255 is connected in signal communication with an output of the disparity/illumination compensator 250. A second input of the switch 255 is connected in signal communication with an output of the motion compensator 235. A third input of the switch 255 is connected in signal communication with an output of the intra predictor 230. An output of the mode module 260 is connected in signal communication with the switch 255 to control which input is selected by the switch 255. The output of the deblocking filter 225 is available as an output of the decoder.

In the MPEG-4 AVC standard, the syntax and semantic parameters of the sequence parameter set are specified for video usability information (VUI). This represents additional information that can be inserted into the bitstream to enhance the usability of the video for various purposes. VUI includes information such as aspect ratio, overscan, video signal type, chroma position, timing, network abstraction layer (NAL) hypothetical reference decoder (HRD) parameters, video coding layer (VCL) hypothetical reference decoder parameters, bitstream constraints, and more.

According to one or more embodiments of the present principles, we use the existing video availability information field for a new and different purpose compared to the prior art, and further extend its use to multi-view video coding (MVC). In our multi-view video coding mechanism, the video availability information is extended so that it can be different, for example, between different views, different temporal levels within a view, or different operating points. Thus, according to an embodiment, we specify the video availability information according to one or more of the following (but not limited to): specifying the video availability information for each view separately; specifying the video availability information for each temporal level within a view separately; and specifying the video availability information for each operating point separately.

In the MPEG-4 AVC standard, a set including video usability information (VUI) can be transmitted in a sequence parameter set (SPS). According to an embodiment, we extend the concept of video usability information to be used in the context of multi-view video coding (MVC). Advantageously, this allows different video usability information to be specified for different views, different temporal levels within a view, or different operating points in multi-view video coding. In an embodiment, we provide novel ways to consider, modify, and use bitstream constraint information in the video usability information for multi-view video coding.

The bitstream restriction information in the MPEG-4 AVC standard is specified in the vui_parameters() syntax element which is part of the sequence_parameter_set(). Table 1 illustrates the MPEG-4 AVC standard syntax of vui_parameters().

Table 1

The semantics of the syntax elements of bitstream constraint information are as follows:

bitstream_restriction_flag equal to 1 specifies that the following coded video sequence bitstream restriction parameters are present.

bitstream_restriction_flag equal to 0 specifies that the following coded video sequence bitstream restriction parameters are not present.

motion_vectors_over_pic_boundaries_flag equal to 0 indicates that samples outside the picture boundaries and samples at partial sample positions whose values are derived using one or more samples outside the picture boundaries are not used for inter prediction of any sample.

motion_vectors_over_pic_boundaries_flag equal to 1 indicates that one or more samples outside picture boundaries may be used in inter prediction.

When the motion_vectors_over_pic_boundaries_flag syntax element is not present, the value of motion_vectors_over_pic_boundaries_flag shall be inferred to be equal to 1.

max_bytes_per_pic_denom indicates the number of bytes that the sum of the sizes of Virtual Coding Layer (VCL) Network Abstraction Layer (NAL) units associated with any coded picture in the coded video sequence does not exceed.

For this purpose, the number of bytes representing a picture in the NAL unit stream is specified as the total number of bytes of the virtual coding layer NAL unit data for that picture (i.e., the sum of the NumBytesInNALunit variables of the virtual coding layer NAL unit). The value of max_bytes_per_pic_denom should be in the range of 0 to 16, inclusive.

Depending on max_bytes_per_pic_denom, the following applies:

- If max_bytes_per_pic_denom is equal to 0, no limit is indicated.

Otherwise (max_bytes_per_pic_denom is not equal to 0), the uncoded picture is represented in the coded video sequence by more than the following number of bits:

(PicSizelnMbs*RawMbBits)÷(8*max_bytes_per_pic_denom)

When the max_bytes_per_pic_denom syntax element is not present, the value of max_bytes_per_pic_denom shall be inferred to be equal to 2. The variable PicSizeInMbs is the number of macroblocks in the picture. The variable RawMbBits is derived as in subclause 7.4.2.1 of the MPEG-4 AVC standard.

max_bits_per_mb_denom indicates the maximum number of coded bits of macroblock_layer() data for any macroblock in any picture of the coded video sequence. The value of max_bits_per_mb_denom shall be in the range of 0 to 16, inclusive.

Depending on max_bits_per_mb_denom, the following applies:

- If max_bits_per_mb_denom is equal to 0, no limit is specified.

Otherwise (max_bits_per_mb_denom is not equal to 0), uncoded macroblock_layer() shall be represented in the bitstream by more than the following number of bits.

(128+RawMbBits)÷max_bits_per_mb_denom

Depending on entropy_coding_mode_flag, the bits of macroblock_layer() data are counted as follows:

- If entropy_coding_mode_flag is equal to 0, the number of bits of the macroblock_layer() data is given by the number of bits in the macroblock_layer() syntax structure of the macroblock.

- Otherwise (entropy_coding_mode_flag is equal to 1), when parsing the macroblock_layer() associated with the macroblock, the number of bits of the macroblock_layer() data for the macroblock is given by the number of calls to read_bits(1) in subclauses 9.3.3.2.2 and 9.3.3.2.3 of the MPEG-4 AVC standard.

When max_bits_per_mb_denom is not present, the value of max_bits_per_mb_denom should be inferred to be equal to 1.

log2_max_mv_length_horizontal and log2_max_mv_length_vertical indicate the maximum absolute values of the decoded horizontal and vertical motion vector components, respectively, in 1/4 luma sample units, for all pictures in the coded video sequence. The value n declares that no motion vector component will exceed the range ^-2n to ^{2n - 1} , inclusive, in units of 1/4 luma sample displacement. The value of log2_max_mv_length_horizontal shall be in the range 0 to 16, inclusive. The value of log2_max_mv_length_vertical shall be in the range 0 to 16, inclusive. When log2_max_mv_length_horizontal is not present, the values of log2_max_mv_length_horizontal and log2_max_mv_length_vertical shall be inferred to be equal to 16. It should be noted that the maximum absolute value of a decoded vertical or horizontal motion vector component is also limited by the profile and level limits as specified in Annex A of the MPEG-4 AVC standard.

num_reorder_frames indicates the maximum number of frames, supplementary field pairs, or unpaired fields that precede in decoding order and follow in output order any frame, supplementary field pair, or unpaired field in the coded video sequence, respectively. The value of num_reorder_frames shall be in the range of 0 to max_dec_fram_buffering, inclusive. When the num_reorder_frames syntax element is not present, the value of num_reorder_frames shall be inferred as follows:

If profile_idc is equal to 44, 100, 110, 122, or 244, and constraint_set3_flag is equal to 1, then the value of num_reorder_frames shall be inferred to be equal to 0.

Otherwise (profile_idc is not equal to 44, 100, 110, 122 or 244, or constraint_set3_flag is equal to 0), the value of num_reorder_frames shall be inferred to be equal to max_dec_fram_bufferingMaxDpbSize.

max_dec_fram_buffering specifies the required size of the decoded picture buffer (DPB) of the assumed reference decoder in units of frame buffers. The encoded video sequence should not require a decoded picture buffer with a size larger than Max(1, max_dec_fram_buffering) frame buffers so that the output of the decoded pictures is at the output time specified by the dpb_output_delay of the picture timing supplementary enhancement information (SEI) message. The value of max_dec_fram_buffering should be in the range of num_ref_frames to MaxDpbSize, including num_ref_frames and MaxDpbSize (as specified in subclause A.3.1 or A.3.2 of the MPEG-4 AVC standard). When the max_dec_fram_buffering syntax element is not present, the value of max_dec_fram_buffering should be inferred as follows:

If profile_idc is equal to 44 or 244, and constraint_set3_flag is equal to 1, then the value of max_dec_fram_buffering shall be inferred to be equal to 0.

Otherwise (profile_idc is not equal to 44 or 244, or constraint_set3_flag is equal to 0), the value of max_dec_frame_buffering shall be inferred to be equal to MaxDpbSize.

In multi-view video coding, bitstream constraint parameters customize the decoding operation of substreams based on tighter bounds. Therefore, it should be possible to specify bitstream constraint parameters for each extractable substream of a multi-view video coding bitstream. According to an embodiment, we propose to specify bitstream constraint information per view, per temporal level within a view, and/or per operation point.

Specify bitstream restriction parameters for each view

Bitstream restriction parameters can be specified for each view. We propose the mvc_vui_parameters_extension syntax, which is part of subset_sequence_parameter_set. Table 2 illustrates the mvc_vui_parameters_extension syntax.

mvc_vui_parameters_extension() loops over all views associated with this subset_sequence_parameter_set, specifying the view_id of each view and the bitstream restriction parameters of each view within the loop.

Table 2

The semantics of the bitstream constraint syntax element are as follows:

bitstream_restriction_flag[i] specifies the value of bitstream_restriction_flag for the view with view_id[i] equal to view_id.

motion_vectors_over_pic_boundaries_flag[i] specifies the value of motion_vectors_over_pic_boundaries_flag for the view with view_id[i] equal to view_id. When the motion_vectors_over_pic_boundaries_flag[i] syntax element is not present, the value of motion_vectors_over_pic_boundaries_flag for the view with view_id[i] equal to view_id shall be inferred to be equal to 1.

max_bytes_per_pic_denom[i] specifies the value of max_bytes_per_pic_denom for the view with view_id[i] equal to view_id. When the max_bytes_per_pic_denom[i] syntax element is not present, the value of max_bytes_per_pic_denom for the view with view_id[i] equal to view_id shall be inferred to be equal to 2.

max_bits_per_mb_denom[i] specifies the max_bits_per_mb_denom value for the view with view_id[i] equal to view_id. When max_bits_per_mb_denom[i] is not present, the max_bits_per_mb_denom value for the view with view_id[i] equal to view_id shall be inferred to be equal to 1.

log2_max_mv_length_horizontal[i] and log2_max_mv_length_vertical[i] specify the values of log2_max_mv_length_horizontal and log2_max_mv_length_vertical, respectively, for the view with view_id[i] equal to view_id. When log2_max_mv_length_horizontal[i] is not present, the values of log2_max_mv_length_horizontal and log2_max_mv_length_vertical for the view with view_id[i] equal to view_id shall be inferred to be equal to 16.

num_reorder_frames[i] specifies the value of num_reorder_frames for the view with view_id[i] equal to view_id. The value of num_reorder_frames[i] should be in the range of 0 to max_dec_frame_buffering, inclusive. When the num_reorder_frames[i] syntax element is not present, the value of num_reorder_frames for the view with view_id[i] equal to view_id should be inferred to be equal to max_dec_frame_buffering.

max_dec_frame_buffering[i] specifies the value of max_dec_frame_buffering for the view with view_id[i] equal to view_id. The value of max_dec_frame_buffering[i] shall be in the range of num_ref_frames[i] to MaxDpbSize, inclusive, of num_ref_frames[i] and MaxDpbSize (as specified in subclauses A.3.1 or A.3.2 in the MPEG-4 AVC Standard). When the max_dec_frame_buffering[i] syntax element is not present, the value of max_dec_frame_buffering for the view with view_id[i] equal to view_id shall be inferred to be equal to MaxDpbSize.

3 , an example method for encoding per-view bitstream restriction parameters using the mvc_vui_parameters_extension() syntax element is indicated generally by the reference numeral 300 .

Method 300 includes a start block 305, which passes control to a function block 310. Function block 310 sets a variable M equal to the number of views minus one and passes control to a function block 315. Function block 315 writes the variable M to the bitstream and passes control to a function block 320. Function block 320 sets a variable i equal to 0 and passes control to a function block 325. Function block 325 writes the view_id[i] syntax element and passes control to a function block 330. Function block 330 writes the bitstream_restriction_flag[i] syntax element and passes control to a decision block 335. Decision block 335 determines whether the bitstream_restriction_flag[i] syntax element is equal to 0. If so, control is passed to a decision block 345. Otherwise, control is passed to a function block 340.

Function block 340 writes the bitstream restriction parameters for view i and passes control to decision block 345. Decision block 345 determines whether variable i is equal to variable M. If so, control is passed to end block 399. Otherwise, control is passed to function block 350.

Function block 350 sets the variable i equal to i plus one and returns control to function block 325 .

4 , an example method for decoding per-view bitstream restriction parameters using the mvc_vui_parameters_extension() syntax element is indicated generally by the reference numeral 400 .

Method 400 includes a start block 405, which passes control to a function block 407. Function block 407 reads a variable M from the bitstream and passes control to a function block 410. Function block 410 sets the number of views equal to the variable M plus one and passes control to a function block 420. Function block 420 sets a variable i equal to 0 and passes control to a function block 425. Function block 425 reads the view_id[i] syntax element and passes control to a function block 430. Function block 430 reads the bitstream_restriction_flag[i] syntax element and passes control to a decision block 435. Decision block 435 determines whether the bitstream_restriction_flag[i] syntax element is equal to 0. If so, control is passed to a decision block 445. Otherwise, control is passed to a function block 440.

Function block 440 reads the bitstream restriction parameters for view i and passes control to decision block 445. Decision block 445 determines whether variable i is equal to variable M. If so, control is passed to end block 499. Otherwise, control is passed to function block 450.

Function block 450 sets the variable i equal to i plus one and returns control to function block 425 .

Specify bitstream constraint parameters for each temporal level of each view

The bitstream restriction parameters may be specified for each temporal level of each view. We propose the mvc_vui_parameters_extention syntax as part of subset_sequence_parameter_set. Table 3 illustrates the mvc_vui_parameters_extention syntax.

Table 3

The semantics of the bitstream constraint syntax element are as follows:

bitstream_restriction_flag[i][j] specifies the value of bitstream_restriction_flag for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id.

motion_vectors_over_pic_boundaries_flag[i][j] specifies the value of motion_vectors_over_pic_boundaries_flag for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id. When the motion_vectors_over_pic_boundaries_flag[i] syntax element is not present, the value of motion_vectors_over_pic_boundaries_flag for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id shall be inferred to be equal to 1.

max_bytes_per_pic_denom[i][j] specifies the value of max_bytes_per_pic_denom for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id. When the max_bytes_per_pic_denom[i] syntax element is not present, the value of max_bytes_per_pic_denom for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id shall be inferred to be equal to 2.

max_bits_per_mb_denom[i][j] specifies the value of max_bits_per_mb_denom for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id. When max_bits_per_mb_denom[i] is not present, the value of max_bits_per_mb_denom for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id shall be inferred to be equal to 1.

log2_max_mv_length_horizontal[i][j] and log2_max_mv_length_vertical[i][j] specify the values of log2_max_mv_length_horizontal and log2_max_mv_length_vertical, respectively, for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id. When log2_max_mv_length_horizontal[i][j] is not present, the values of log2_max_mv_length_horizontal and log2_max_mv_length_vertical for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id shall be inferred to be equal to 16.

num_reorder_frames[i][j] specifies the value of num_reorder_frames for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id. The value of num_reorder_frames[i] shall be in the range of 0 to max_dec_frame_buffering, inclusive. When the num_reorder_frames[i] syntax element is not present, the value of num_reorder_frames for the temporal level with temporal_id[i][j] equal to temporal_id in the view with view_id[i] equal to view_id shall be inferred to be equal to max_dec_frame_buffering.

max_dec_frame_buffering[i][j] specifies the value of max_dec_frame_buffering for the temporal level with temporal_id[i][j] equal to temporal_id, in the view with view_id[i] equal to view_id. The value of max_dec_frame_buffering[i] shall be in the range of num_ref_frames[i] to MaxDpbSize, inclusive, of num_ref_frames[i] and MaxDpbSize (as specified in subclauses A.3.1 or A.3.2 in the MPEG-4 AVC Standard). When the max_dec_frame_buffering[i] syntax element is not present, the value of max_dec_frame_buffering for the temporal level with temporal_id[i][j] equal to temporal_id, in the view with view_id[i] equal to view_id, shall be inferred to be equal to MaxDpbSize.

In mvc_vui_parameters_extension(), two loops are executed. The outer loop loops over all views associated with subset_sequence_parameter_set. The view_id, which specifies the number of temporal levels for each view, is specified in the outer loop. The inner loop loops over all temporal levels of the view. The bitstream restriction information is specified in the inner loop.

5 , an example method for encoding bitstream restriction parameters for each temporal level in each view using the mvc_vui_parameters_extension() syntax element is indicated generally by the reference numeral 500 .

Method 500 includes a start block 505, which passes control to a function block 510. Function block 510 sets a variable M equal to the number of views minus one and passes control to a function block 515. Function block 515 writes the variable M to the bitstream and passes control to a function block 520. Function block 520 sets a variable i equal to 0 and passes control to a function block 525. Function block 525 writes the view_id[i] syntax element and passes control to a function block 530. Function block 530 sets a variable N equal to the number of temporal levels in view i minus one and passes control to a function block 535. Function block 535 writes the variable N to the bitstream and passes control to a function block 540. Function block 540 sets a variable j equal to 0 and passes control to a function block 545. Function block 545 writes the temporal_id[i][j] syntax element and passes control to a function block 550. Function block 550 writes the bitstream_restriction_flag[i][j] syntax element and passes control to decision block 555. Decision block 555 determines whether the bitstream_restriction_flag[i][j] syntax element is equal to 0. If so, control is passed to decision block 565. Otherwise, control is passed to function block 560.

Function block 560 writes the bitstream restriction parameters for temporal level j in view i and passes control to decision block 565. Decision block 565 determines whether variable j is equal to variable N. If so, control is passed to decision block 570. Otherwise, control is passed to function block 575.

Decision block 570 determines whether the variable i is equal to the variable M. If so, control is passed to end block 599. Otherwise, control is passed to function block 580.

Function block 580 sets the variable i equal to i plus one and returns control to function block 525 .

Function block 575 sets the variable j equal to j plus one and returns control to function block 545 .

6 , an example method for decoding bitstream restriction parameters for each temporal level in each view using the mvc_vui_parameters_extension() syntax element is indicated generally by the reference numeral 600 .

Method 600 includes a start block 605, which passes control to a function block 607. Function block 607 reads a variable M from the bitstream and passes control to a function block 610. Function block 610 sets the number of views to M plus one and passes control to a function block 620. Function block 620 sets a variable i to 0 and passes control to a function block 625. Function block 625 reads a view_id[i] syntax element and passes control to a function block 627. Function block 627 reads a variable N from the bitstream and passes control to a function block 630. Function block 630 sets the number of temporal levels in view i to N plus one and passes control to a function block 640. Function block 640 sets a variable j to 0 and passes control to a function block 645. Function block 645 reads a temporal_id[i][j] syntax element and passes control to a function block 650. Function block 650 reads the bitstream_restriction_flag[i][j] syntax element and passes control to decision block 655. Decision block 655 determines whether the bitstream_restriction_flag[i][j] syntax element is equal to 0. If so, control is passed to decision block 665. Otherwise, control is passed to function block 660.

Function block 660 reads the bitstream restriction parameters for temporal level j in view i and passes control to decision block 665. Decision block 665 determines whether variable j is equal to variable N. If so, control is passed to decision block 670. Otherwise, control is passed to function block 675.

Decision block 670 determines whether the variable i is equal to the variable M. If so, control is passed to end block 699. Otherwise, control is passed to function block 680.

Function block 680 sets the variable i equal to i plus one and returns control to function block 625 .

Function block 675 sets the variable j equal to j plus one and returns control to function block 645 .

Specify bitstream constraint information for each operation point

Bitstream restriction parameters can be specified for each operation point. We propose to convey the bitstream restriction parameters for each operation point in the view scalability information SEI message. The syntax of the view scalability information SEI message can be modified as shown in Table 4. The syntax of the bitstream restriction information is inserted in a loop that loops over all operation points.

Table 4

The semantics of the bitstream constraint syntax element are as follows:

bitstream_restriction_flag[i] specifies the value of bitstream_restriction_flag for the operation point having operation_point_id[i] equal to operation_point_id.

motion_vectors_over_pic_boundaries_flag[i] specifies the value of motion_vectors_over_pic_boundaries_flag for the operation point with operation_point_id[i] equal to operation_point_id. When the motion_vectors_over_pic_boundaries_flag[i] syntax element is not present, the value of motion_vectors_over_pic_boundaries_flag for the operation point with operation_point_id[i] equal to operation_point_id shall be inferred to be equal to 1.

max_bytes_per_pic_denom[i] specifies the max_bytes_per_pic_denom value for the operation point with operation_point_id[i] equal to operation_point_id. When the max_bytes_per_pic_denom[i] syntax element is not present, the max_bytes_per_pic_denom value for the operation point with operation_point_id[i] equal to operation_point_id shall be inferred to be equal to 2.

max_bits_per_mb_denom[i] specifies the max_bits_per_mb_denom value for the operation point with operation_point_id[i] equal to operation_point_id. When max_bits_per_mb_denom[i] is not present, the max_bits_per_mb_denom value for the operation point with operation_point_id[i] equal to operation_point_id shall be inferred to be equal to 1.

log2_max_mv_length_horizontal[i] and log2_max_mv_length_vertical[i] specify the value of log2_max_mv_length_horizontal and the value of log2_max_mv_length_vertical, respectively, for the operation point with operation_point_id[i] equal to operation_point_id. When log2_max_mv_length_horizontal[i] is not present, the values of log2_max_mv_length_horizontal and log2_max_mv_length_vertical for the operation point with operation_point_id[i] equal to operation_point_id shall be inferred to be equal to 16.

num_reorder_frames[i] specifies the value of num_reorder_frames for the operation point with operation_point_id[i] equal to operation_point_id. The value of num_reorder_frames[i] should be in the range of 0 to max_dec_frame_buffering, inclusive. When the num_reorder_frames[i] syntax element is not present, the value of num_reorder_frames for the operation point with operation_point_id[i] equal to operation_point_id should be inferred to be equal to max_dec_frame_buffering.

max_dec_frame_buffering[i] specifies the value of max_dec_frame_buffering for the operation point with operation_point_id[i] equal to operation_point_id. The value of max_dec_frame_buffering[i] shall be in the range of num_ref_frames[i] to MaxDpbSize, inclusive, of num_ref_frames[i] and MaxDpbSize (as specified in subclauses A.3.1 or A.3.2 in the MPEG-4 AVC Standard). When the max_dec_frame_buffering[i] syntax element is not present, the value of max_dec_frame_buffering for the operation point with operation_point_id[i] equal to operation_point_id shall be inferred to be equal to MaxDpbSize.

Turning to FIG. 7 , an example method for encoding bitstream restriction parameters per operation point using the view_scalability_parameters_extension() syntax element is indicated generally by the reference numeral 700 .

Method 700 includes a start block 705, which passes control to a function block 710. Function block 710 sets a variable M equal to the number of operation points minus one and passes control to a function block 715. Function block 715 writes the variable M to the bitstream and passes control to a function block 720. Function block 720 sets a variable i equal to 0 and passes control to a function block 725. Function block 725 writes the operation_point_id[i] syntax element and passes control to a function block 730. Function block 730 writes the bitstream_restriction_flag[i] syntax element and passes control to a decision block 735. Decision block 735 determines whether the bitstream_restriction_flag[i] syntax element is equal to 0. If so, control is passed to a decision block 745. Otherwise, control is passed to a function block 740.

Function block 740 writes the bitstream constraint parameters for operation point i and passes control to decision block 745. Decision block 745 determines whether variable i is equal to variable M. If so, control is passed to end block 799. Otherwise, control is passed to function block 750.

Function block 750 sets the variable i equal to i plus one and returns control to function block 725 .

Turning to FIG. 8 , an example method for decoding bitstream restriction parameters per operation point using the view_scalability_parameters_extension() syntax element is indicated generally by the reference numeral 800 .

Method 800 includes a start block 805, which passes control to a function block 807. Function block 807 reads a variable M from the bitstream and passes control to a function block 810. Function block 810 sets the number of operation points equal to the variable M plus one and passes control to a function block 820. Function block 820 sets a variable i equal to 0 and passes control to a function block 825. Function block 825 reads the operation_point_id[i] syntax element and passes control to a function block 830. Function block 830 reads the bitstream_restriction_flag[i] syntax element and passes control to a decision block 835. Decision block 835 determines whether the bitstream_restriction_flag[i] syntax element is equal to 0. If so, control is passed to a decision block 845. Otherwise, control is passed to a function block 840.

Function block 840 reads the bitstream constraint parameters for operation point i and passes control to decision block 845. Decision block 845 determines whether variable i is equal to variable M. If so, control is passed to end block 899. Otherwise, control is passed to function block 850.

Function block 850 sets the variable i equal to i plus one and returns control to function block 825 .

A description will now be given of some of the many accompanying advantages/features of the present invention, some of which have been mentioned above. For example, one advantage/feature is an apparatus comprising an encoder for encoding multi-view video content by specifying video usability information for at least one of individual views, individual temporal levels within a view, and individual operating points.

Another advantage/feature is the apparatus having the encoder as described above, wherein the parameters are specified in at least one high-level syntax element.

Furthermore, another advantage/feature is the apparatus having the encoder as described above, wherein the at least one high-level syntax element includes at least one of: an mvc_vui_parameters_extension() syntax element, an mvc_scalability_info supplemental enhancement information syntax message, at least a portion of a sequence parameter set, a picture parameter set, and supplemental enhancement information.

Additionally, another advantage/feature is the apparatus having the encoder as described above, wherein at least a portion of the video availability information includes a bitstream restriction parameter.

Based on the teachings herein, those skilled in the relevant art can easily determine these and other features and advantages of the present principles.It should be understood that the teachings of the present principles can be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof.

Most preferably, the teachings of the present principles are implemented as a combination of hardware and software. In addition, the software can be implemented as an application program tangibly embodied on a program storage unit. The application program can be uploaded to a machine comprising any suitable structure and executed by the machine. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPU"), random access memory ("RAM"), and input/output ("I/O") interfaces. The computer platform can also include an operating system and microinstruction code. The various processes and functions described herein can be part of the microinstruction code that can be executed by the CPU or part of the application program, or any combination thereof. In addition, various other peripheral units can be connected to the computer platform, such as additional data storage units and printing units.

It should also be understood that since some of the system components and methods shown in the drawings are preferably implemented in software, the actual connections between these system components or processing function blocks may vary depending on how the present principles are programmed. Given the teachings herein, one of ordinary skill in the relevant art will be able to contemplate these and similar implementations or configurations of the present principles.

Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the present principles are not limited to those precise embodiments, and that various changes and modifications may be made therein by one of ordinary skill in the relevant art without departing from the scope and spirit of the present principles. All such changes and modifications are intended to be included within the scope of the present principles as set forth in the appended claims.

Claims (11)

1. An encoding method, comprising: Multi-view video content is encoded by specifying bitstream limiting information for each operation point in multi-view video encoding, where the operation point corresponds to one of different combinations of views and time levels, and the bitstream limiting information indicates one or more of the following: (1) Does the motion vector exceed the screen boundary? (2) Maximum number of bytes per frame; (3) The maximum number of bits in each macroblock; (4) Maximum horizontal and vertical motion vector lengths; (5) The number of frames to be reordered; and (6) Maximum decoding frame buffer size. 2. The method of claim 1, wherein the bitstream restriction information is specified in the video availability information (“VUI”) syntax structure. 3. The method of claim 1, wherein the bitstream restriction information is transmitted in the supplementary enhancement information message. 4. A decoding method, comprising: Multi-view video content is decoded by decoding bitstream constraint information specified for each operation point in multi-view video encoding, where the operation point corresponds to one of different combinations of views and time levels, and the bitstream constraint information indicates one or more of the following: (1) Does the motion vector exceed the screen boundary? (2) Maximum number of bytes per frame; (3) The maximum number of bits in each macroblock; (4) Maximum horizontal and vertical motion vector lengths; (5) The number of frames to be reordered; and (6) Maximum decoding frame buffer size. 5. The method of claim 4, wherein the bitstream restriction information is specified in the video availability information (“VUI”) syntax structure. 6. The method of claim 4, wherein the bitstream restriction information is transmitted in a supplementary enhancement information message. 7. A decoding device, comprising: A decoder is used to decode multi-view video content by decoding bitstream limiting information specified for each operation point in multi-view video encoding, wherein the operation point corresponds to one of different combinations of views and time levels, and wherein the bitstream limiting information indicates one or more of the following: (1) Does the motion vector exceed the screen boundary? (2) Maximum number of bytes per frame; (3) The maximum number of bits in each macroblock; (4) Maximum horizontal and vertical motion vector lengths; (5) The number of frames to be reordered; and (6) Maximum decoding frame buffer size. 8. The apparatus of claim 7, wherein the bitstream restriction information is specified in the video availability information (“VUI”) syntax structure. 9. The apparatus of claim 7, wherein the bitstream restriction information is transmitted in a supplemental enhancement information message. 10. A decoding method, comprising: Multiview video content is decoded by decoding bitstream limiting information specified for each time level in the view encoded as multiview video, wherein the bitstream limiting information indicates one or more of the following: (1) Does the motion vector exceed the screen boundary? (2) Maximum number of bytes per frame; (3) The maximum number of bits in each macroblock; (4) Maximum horizontal and vertical motion vector lengths; (5) The number of frames to be reordered; and (6) Maximum decoding frame buffer size. 11. A computer-readable storage medium storing program instructions, which, when executed by a processor, cause the processor to perform the encoding method according to any one of claims 1-3 or the decoding method according to any one of claims 4-6 and 10.