HK40110265A - Multi-view decoder - Google Patents

Description

Multi-view decoder

This application is a divisional application of patent application 201910419460.0, filed on April 8, 2014, entitled “Encoding Concept Allowing Effective Multi-View/Layer Encoding”, the entire contents of which are incorporated herein by reference.

Technical Field

This application relates to an encoding concept that allows for effective multi-view/layer encoding (e.g., multi-view image/video encoding).

Background Technology

The concept of scalable coding is well-known in this field. In video coding, for example, H.264 allows the base-level encoded video data stream to be accompanied by additional enhancement layer data to improve the reconstruction quality of the base-level quality video in various ways, such as spatial resolution, signal-to-noise ratio (SNR), and/or last but equally important, the number of views. The recently finalized HEVC standard is also extended by the SVC/MVC framework (SVC = Scalable Video Coding, MVC = Multi-View Coding). HEVC differs from its predecessor H.264 in many ways, for example, it is suitable for parallel decoding/encoding and low-latency transmission. Regarding parallel decoding/encoding, HEVC supports WPP (Wavefront Parallel Processing) encoding/decoding and the concept of tile parallel processing. According to the WPP concept, individual pictures are divided into substreams in a line-by-line manner. The encoding order within each substream is from left to right. The substreams have a decoding order defined within them, leading from the top substream to the bottom substream. Entropy coding of the substreams is performed using probabilistic adaptation (adaptive). Individually or based on an initial fit state for entropy coding, such as the probabilities of the substream immediately preceding the left edge of the preceding substream at the end of the second CTB (coding tree block) up to a certain position, each substream is probabilistically initialized. Spatial prediction is not restricted. That is, spatial prediction can span boundaries between immediately following substreams. In this way, these substreams can be encoded/decoded in parallel with the current encoding/decoding position forming the wavefront, which operates in a tiled manner from bottom left to top right and from left to right. According to the tile concept, the image is divided into tiles, and in order to render the encoding/decoding of these tiles, parallel processing and possible themes of spatial prediction above tile boundaries are prohibited. Only loop filtering above tile boundaries is allowed. To support low-latency processing, the slab concept has been extended: allowing slabs to be switched to reinitialize entropy probabilities using the entropy probabilities saved during processing of the previous substream, i.e., the substream preceding the current slab's starting point, and using continuously updated entropy probabilities until the immediately preceding slab ends. This approach makes WPP and the tile concept more suitable for low-latency processing.

However, it is preferable to have a concept under consideration, which further enhances the concept of multi-view/layer coding.

Summary of the Invention

Therefore, the objective of this invention is to provide a concept that improves the concept of multi-view/layer encoding.

This goal is achieved by the subject matter of the pending independent claims.

The first aspect of this application relates to multi-view coding. Specifically, the underlying concept of this first aspect is as follows: On the one hand, inter-view prediction helps to utilize redundancy among multiple views capturing a scene, thereby improving coding efficiency. On the other hand, inter-view prediction prevents multiple views from being independently decodeable/encodeable, i.e., in parallel, so as to utilize, for example, multi-core processors. More precisely, inter-view prediction makes a portion of the second view dependent on a corresponding reference portion of the first view, and this relationship between portions of the first and second views needs to satisfy a certain inter-view decoding/encoding offset/delay when decoding/encoding the first and second views in parallel. The underlying concept of this first aspect is that this inter-view coding offset can be significantly reduced by changing the way coding and/or decoding are performed, by altering the manner in which inter-view prediction is performed on the spatial segment boundaries of the spatial segments into which the first/reference view is divided, thus reducing coding efficiency only slightly. Variations can be made so that the inter-view prediction from the first view to the second view does not combine any information from different spatial segments of the first view, but predicts the second view and its syntax elements separately only from information originating from a spatial segment of the first view. According to one implementation, even more stringent variations are made so that inter-view predictions do not even cross spatial segment boundaries, i.e., a spatial segment boundary is a spatial segment that includes a co-located location or a co-located portion. The advantages arising from the variation at the segment boundaries become clear when considering the result of combining information from two or more spatial segments originating from the first view in inter-view prediction. In this case, encoding/decoding of any part of the second view involving such a combination in inter-view prediction must be postponed until the encoding/decoding of all spatial segments of the first view is combined by inter-layer prediction. However, the variation in inter-view prediction at the spatial segment boundaries of the first view solves this problem, and once a spatial segment of the first view is decoded/encoded, each part of the second view becomes readily encodeable/decodeable. However, the encoding efficiency is only slightly reduced because inter-layer prediction is still largely permitted, with the limitation only applying to the spatial segment boundaries of the first view. According to one implementation, the encoder takes note of the variation in inter-layer prediction at the spatial segment boundaries of the spatial segments of the first view to avoid two or more spatial segments of the first view having the aforementioned combination, and sends this avoidance/situation signaling to the decoder, which in turn uses the signaling as a guarantee to, for example, reduce the inter-view decoding latency in response to the signaling. According to another implementation, the decoder also changes the way inter-layer prediction is triggered by signaling within the data stream so that, since the amount of edge information required to control inter-layer prediction with respect to these spatial segment boundaries can be reduced, the limitations of the inter-layer prediction parameter settings on the spatial segment boundaries of the first view can be utilized when forming the data stream.

The second aspect of this application relates to multi-layer video coding and the following situation: Typically, NAL units encoded from images of multiple layers are collected into an access unit so that an access unit is formed with NAL units associated with a time step, independent of the layer associated with the corresponding NAL unit, or so that there is one access unit for each different pair of time steps and layers. However, regardless of the chosen possibility, the NAL units of each time-to-layer pair are processed separately and arranged non-interleaved. That is, NAL units belonging to a certain time step and layer are issued before continuing with NAL units of another pair of time steps and layers. Interleaving is not allowed. However, this prevents further reduction of end-to-end latency because it prevents the encoder from issuing NAL units belonging to the relevant layer between NAL units belonging to the base layer, which is caused by inter-layer parallel processing. The second aspect of this application abandons the strict sequential non-interleaved arrangement of NAL units in the transmitted wake stream and, for this purpose, reuses the first possibility of defining access units, collecting all NAL units of a time step: collecting all NAL units of a time step within an access unit, and the access unit is still arranged in a non-interleaved manner in the transmitted bit stream. However, NAL units of an access unit are allowed to be interleaved, so that NAL units of one layer are scattered with NAL units of another layer. The execution of NAL units belonging to one layer within an access unit forms a decoding unit. Interleaving is permitted so that for each NAL unit within an access unit, the information needed for inter-layer prediction is contained within any preceding NAL units within that access unit. The encoder can signal within the bitstream whether interleaving is applied, and conversely, the decoder can (e.g.) use multiple buffers to reclassify the interleaved NAL units of different layers for each access unit, or, depending on the signaling, use only one buffer without interleaving. No coding efficiency loss occurs; however, end-to-end latency is reduced.

The third aspect of this application relates to signaling for the layer index of each bitstream data packet (e.g., each NAL unit). According to the third aspect, the inventors recognize that applications primarily fall into one of two types. Normal applications require a moderate number of layers, therefore, there is no layer ID field within each data packet, which is configured to fully cover the moderate number of layers in total. Only rarely do more complex applications, in turn, require an excessive number of layers. Therefore, according to the third aspect, a layer identification extension mechanism signaling is used within a multi-layer video signal, such that the signaling of the layer identification syntax element within each data packet, either entirely or only partially, together with the layer identification extension within the multi-layer data stream, determines whether the layer of the corresponding data packet is determined or entirely replaced/dominated by the layer identification extension. With this measure, layer identification extension is only needed in rare applications, and this layer identification extension consumes bit rate, while in most cases, effective signaling associated with the layer is feasible.

A fourth aspect of this application relates to signaling for predicting inter-layer dependencies between different levels of information content, encoding video data into multi-layer video data streams. According to the fourth aspect, a first syntactic structure defines the number of dependency dimensions and the maximum N_{_i} of the ordinal rank of each dependency dimension i, and a bijective mapping maps each rank to a corresponding up-rank of at least a subset of available points in the dependency space, and for dependency dimension i, a second syntactic structure. The latter defines the dependencies between layers. Each syntactic structure describes the dependencies within the N_{_i} ordinal rank of the dependency dimension i to which the corresponding second syntactic structure belongs. Thus, the effort to define dependencies increases only linearly with the number of dependency dimensions, while the constraint imposed by this signaling on inter-layer dependencies is relatively low.

Naturally, all of the above aspects can be combined in pairs, threes, or all of them.

Attached Figure Description

The preferred embodiments of this application are described below with reference to the accompanying drawings, wherein:

Figure 1 shows a video encoder as an illustrative example of any multilayer encoder used to implement the following figure, which is further outlined below.

Figure 2 shows a schematic block diagram of a video decoder that works in conjunction with the video encoder of Figure 1;

Figure 3 shows a schematic diagram of an image subdivided into substreams for WPP processing;

Figure 4 shows a schematic diagram of an image displaying any layer subdivided into blocks, with further subdivision of the image into spatial segments;

Figure 5 shows a schematic diagram of any layer of images subdivided into blocks and tiles;

Figure 6 shows a schematic diagram of the image subdivided into blocks and sub-streams;

Figure 7 shows a base view and a related view according to one embodiment, wherein the related view image is arranged in front of the base view image, and the two and the related view image are schematic diagrams. The related view image is set in front of the base view image. The base and related views are aligned with each other to show the domain of possible values of the disparity vector of the base view block near the spatial segment boundary relative to the base view block farther from the spatial segment boundary.

Figure 8 shows a schematic block diagram of an encoder supporting inter-view prediction constraints and spatial segment boundaries according to one embodiment;

Figure 9 shows a schematic block diagram of a decoder that works in conjunction with the encoder of Figure 8;

Figure 10 illustrates a schematic diagram of inter-view prediction using disparity vectors to illustrate the limitations of the domain for determining the possible values of the disparity vectors and/or aspects of possible modifications to the inter-view prediction processing of related view blocks.

Figure 11 shows a schematic diagram of inter-view prediction, illustrating the parameters used to predict related views, to demonstrate the application of inter-view prediction;

Figure 12 shows a schematic diagram of an image subdivided into code blocks and tiles, where tiles are composed of integer multiples of code blocks, and the decoding order is defined within the code blocks after the image is subdivided into tiles.

Figure 13A shows a portion of the modified VPS syntax as an example of constructing HEVC for the implementation of Figures 8 through 11.

Figure 13B shows the part corresponding to the part in Figure 13A; however, this part belongs to the SPS syntax.

Figure 13C shows another exemplary section in the modified VPS syntax;

Figure 13D shows an example of the modified VPS syntax for implementing inter-view predictive change signaling;

Figure 13E shows an even further example of a portion of the modified VPS syntax for predicting changes between signaling views at space segment boundaries;

Figure 13F shows even further examples of a portion of the modified VPS syntax;

Figure 13G illustrates a portion of the modified VPS syntax as a further possibility for predicting changes/limitations between signaling views;

Figure 14 shows an example of a portion of the modified VPS syntax for predicting changes/limitations, and even further implementations, between signaling views;

Figure 15 illustrates a schematic diagram of the overlay of related view images and base view images displayed as overlapping each other according to one embodiment, so as to show possible modifications to the base layer filtering process displayed on the spatial segment boundary, which can be triggered simultaneously with the predicted changes/constraints between views;

Figure 16 shows a schematic diagram of a multi-layer video data stream, schematically including three layers. In the lower half of Figure 16, options 1 and 2 are shown for setting NAL units belonging to the corresponding time and corresponding layer within the data stream.

Figure 17 shows a schematic diagram of a portion of the data flow, illustrating the two options in an exemplary case of two layers;

Figure 18 shows a schematic block diagram of the decoder, which is configured to process multi-layer video data streams according to Figures 16 and 17 according to Option 1;

Figure 19 shows a schematic block diagram of an encoder that works in conjunction with the decoder of Figure 18;

Figure 20 shows a schematic diagram of an image subdivided into substreams according to one embodiment for WPP processing, and also illustrates the wavefront generated when using WPP to decode/encode the image in parallel.

Figure 21 illustrates a multi-layer video data stream involving three views with three decoding units, each decoding unit in a non-interleaved state;

Figure 22 shows a schematic diagram of the configuration of the multi-layer video data stream according to Figure 21, but the views are interleaved;

Figure 23 shows a schematic diagram of a portion of a multi-layer video data stream within the internal sequence of a NAL unit, to illustrate the constraints that may be observed when accessing interleaved layers within the unit.

Figure 24 shows an example of a portion of the modified VPS syntax to illustrate the possibility of signaling decoding unit interleaving;

Figure 25 shows a portion of the NAL unit header, which exemplarily includes a fixed-length layer identification syntax element;

Figure 26 shows a portion of the VPS syntax, illustrating the possibility of the implementation layer recognizing extended mechanism signaling;

Figure 27 illustrates a portion of the VPS syntax, representing another possibility for implementation layer recognition of extended mechanism signaling;

Figure 28 shows a portion of the VPS syntax to illustrate even further instances of implementation layer recognition of extended mechanism signaling;

Figure 29 shows a portion of the thin fragment header, used to demonstrate the possibility of implementing layer identification extensions within the data stream;

Figure 30 shows a portion of the thin fragment header to illustrate a further example of the implementation layer identification extension;

Figure 31 shows a portion of the VPS syntax used to demonstrate implementation layer identification extensions;

Figure 32 illustrates a portion of the dataflow syntax to demonstrate another possibility for implementation layer recognition extensions;

Figure 33 schematically illustrates a camera setup according to one embodiment to demonstrate the possibility of combining layer recognition syntax elements with layer recognition extensions;

Figures 34A and 34B illustrate a portion of the VPS extended syntax, outlining the framework for signaling layer extension mechanisms within the data stream;

Figure 35 shows a schematic diagram of a decoder configured to process multi-layer video data streams with signaling involving layers of data streams, with these layers set in a dependency space and having dependencies between layers.

Figure 36 shows a schematic diagram of the dependency space, where the direct dependency structure of the layers is in a two-dimensional space, with each dimension in the space using a specific prediction structure;

Figure 37 shows a schematic diagram of an array of direct dependency flags that specify dependencies between different layers;

Figure 38 shows a schematic diagram of two arrays of direct dependency markers that specify dependencies between different locations and dimensions;

Figure 39 shows a portion of the data flow syntax, illustrating how a part of the first syntactic structure for signaling-limited dependency space is presented;

Figure 40 illustrates a portion of the data flow, showing the possibility that signaling involves a portion of the first syntactic structure of mapping between layers of data flow and available points within the dependency space;

Figure 41 illustrates a portion of the data flow, demonstrating the possibility of a second syntactic structure that describes dependencies in terms of dependency-dimension; and

Figure 42 illustrates another possibility for defining the second grammatical structure.

Detailed Implementation

First, as an overview, an embodiment of an encoder/decoder architecture is presented, which is suitable for any of the concepts presented later.

Figure 1 illustrates the general structure of an encoder according to an embodiment. The encoder 10 can be implemented to operate in a multi-threaded or non-multi-threaded manner (i.e., a single-threaded manner only). That is, for example, the encoder 10 can be implemented using multiple CPU cores. In other words, the encoder 10 can support parallel processing, but does not necessarily do so. The generated bitstream can also be generated/decoded by a single-threaded encoder/decoder. However, the encoding concept of this application enables a parallel processing encoder to effectively apply parallel processing without compromising compression efficiency. A similar statement regarding parallel processing capabilities applies to the decoder described later in Figure 2.

Encoder 10 is a video encoder, but typically, encoder 10 can also be an image encoder. Image 12 of video 14 is shown as entering encoder 10 at input 16. Image 12 shows a scene, i.e., image content. However, encoder 10 also receives another image 15 belonging to the same moment at its input 16, and these two images 12 and 15 belong to different layers. For illustrative purposes only, image 12 is shown as belonging to layer 0, while image 15 is shown as belonging to layer 1. Figure 1 shows that layer 1 can have a higher spatial resolution relative to layer 0, i.e., more image samples of the same scene can be displayed, but this is for illustrative purposes only, and alternatively, image 15 of layer 1 can have the same spatial resolution, but, for example, can be different in the view orientation relative to layer 0, i.e., images 12 and 15 can be captured from different perspectives. It should be noted that the terms base layer and enhancement layer used in this document can refer to any set of reference layers and related layers in a layer hierarchy.

Encoder 10 is a hybrid encoder, i.e., images 12 and 15 are predicted by predictor 18, and the prediction residual 20 obtained by residual determiner 22 undergoes transformation (e.g., spectral decomposition, e.g., DCT) and quantization within transform/quantization module 24. The transformed and quantized prediction residual 26 thus obtained is entropy encoded within entropy encoder 28 using, for example, context adaptation, arithmetic coding, or variable-length coding. A reconstructable version of the residual can be used in decoder, i.e., the dequantized and retransformed residual signal 30 is recovered by retransform/requantization module 31 and recombined with the prediction signal 32 of predictor 18 via combiner 33 to generate reconstructions 34 of images 12 and 15, respectively. However, encoder 10 operates on a block-based basis. Accordingly, the reconstructed signal 34 suffers discontinuities at block boundaries; therefore, in order to generate a reference image 38 for images 12 and 15, filter 36 can be applied to the reconstructed signal 34, based on which predictor 18 predicts images of different layers subsequently encoded. However, as shown by the dashed line in Figure 1, predictor 18 can also directly utilize the reconstructed signal 34 without filter 36 or intermediate versions using other prediction modes (e.g., spatial prediction mode).

Predictor 18 can select from different prediction modes to predict certain blocks of image 12. One such block 39 of image 12 is exemplarily shown in FIG. 1. A temporal prediction mode can be used, according to which block 39 representing any block of image 12, divided from image 12, is predicted based on previously encoded images of the same layer (e.g., image 12'). A spatial prediction mode can also be used, according to which block 39 is predicted based on previously encoded portions of the same image 12 (adjacent blocks 39). Block 41 of image 15 is also illustratively shown in FIG. 1 to represent any other block of image 15. For block 41, predictor 18 can support the prediction modes just discussed, namely, the temporal and spatial prediction modes. Furthermore, predictor 18 can provide an inter-layer prediction mode, according to which block 41 is predicted based on corresponding portions of the lower-layer image 12. "Corresponding" in "corresponding portion" indicates spatial correspondence, that is, the portion within image 12 represents the same scene portion as block 41 to be predicted in FIG. 15.

Essentially, the predictions of predictor 18 are not limited to image samples. Predictions can also be applied to any encoding parameters, i.e., prediction modes, temporally predicted motion vectors, disparity vectors predicted between views, etc. Only the residuals can then be encoded in bitstream 40. That is, encoding parameters can be predictively encoded/decoded using spatial and/or inter-layer predictions. Even here, disparity compensation can be used.

A certain syntax is used to compile the quantized residual data 26, i.e., the transform coefficient levels and other residual data, as well as encoding parameters, such as the prediction modes and prediction parameters of the individual blocks 39 and 41 of images 12 and 15 determined by predictor 18, and the syntax elements of this syntax are subjected to entropy encoding by entropy encoder 28. The resulting data stream 40 output by entropy encoder 28 forms the bit stream 40 output by encoder 10.

Figure 2 illustrates a decoder that works in conjunction with the video encoder of Figure 1, i.e., capable of decoding bitstream 40. The decoder of Figure 2 is generally indicated by reference numeral 50 and includes an entropy decoder, a re-transform/dequantization module 54, a combiner 56, a filter 58, and a predictor 60. The entropy decoder 42 receives the bitstream and performs entropy decoding to recover residual data 62 and encoding parameters 64. The re-transform/dequantization module 54 dequantizes and re-transforms the residual data 62 and forwards the resulting residual signal to the combiner 56. The combiner 56 also receives a prediction signal 66 from the predictor 60, which then forms the prediction signal 66 using the encoding parameters 64 based on the reconstructed signal 68 determined by the combiner 56 combining the prediction signal 66 and the residual signal 65. This prediction reflects the prediction ultimately selected by the predictor 18; i.e., the same prediction modes are available and are selected for the respective blocks of images 12 and 15, controlled according to the prediction parameters. As described above with respect to Figure 1, alternatively or additionally, the predictor 60 may use a filtered version of the reconstructed signal 68 or an intermediate version thereof. Similarly, the images of different layers to be ultimately reproduced and output on the output 70 of the decoder 50 may be determined based on an unfiltered version of the combined signal 68 or some of its filtered versions.

Based on the tile concept, images 12 and 15 are subdivided into tiles 80 and 82, respectively. At least, the predictions of blocks 39 and 41 within these tiles 80 and 82 are limited to using only data associated with the same tiles of the same images 12 and 15 as the basis for spatial prediction. This means that the spatial prediction of block 39 is limited to using previously encoded portions of the same tiles, but the temporal prediction mode is not limited to relying on information from previously encoded images (e.g., image 12'). Similarly, the spatial prediction mode of block 41 is limited to using only previously encoded data of the same tiles, but is not limited to temporal or inter-layer prediction modes. For illustrative purposes, images 15 and 12 are subdivided into 6 tiles each. The subdivision of tiles can be selected and signaled individually within bitstream 40 for images 12', 12 and 15, 15', respectively. The number of tiles for each image 12 and 15 can be any of 1, 2, 3, 4, 6, etc., where the tile division can be limited to simply regularly dividing the tile face into rows and columns. For completeness, it's important to note that the method of encoding individual tiles is not limited to intra-frame prediction or spatial prediction, but can also include coding parameters across tile boundaries and any predictions selected within the context of entropy coding. That is, the latter can also be limited to data that depends solely on the same tile. Therefore, the decoder can perform the aforementioned operations in parallel, i.e., on a tile-by-tile basis.

Alternatively or additionally, the encoders and decoders of Figures 1 and 2 can use the WPP concept. Refer to Figure 3. WPP substream 100 also represents the spatial partitioning of images 12 and 15 into a WPP substream. Unlike tiles and slices, WPP substreams do not impose restrictions on prediction and context selection across WPP substream 100. WPP substream 100 extends line by line, for example, across LCU (maximum coding unit) 101, i.e., the largest possible block of prediction coding patterns that can be transmitted individually within the bitstream, and only makes a compromise to entropy coding in order to enable parallel processing. In particular, an order 102 is defined within the WPP substreams 100, which exemplarily extends from top to bottom, and for each WPP substream 100 other than the first WPP substream within order 102, the probability estimate of the symbolic alphabet (i.e., entropy probability) is not completely reset, but is adopted or set to be equal to the probability generated after the entropy encoding/decoding of the WPP substream directly preceding it up to its second LCU as shown by line 104. The LCU order, or the decoder order of the substreams, starts on the same side of images 12 and 15 (e.g., the left-hand side indicated by arrow 106 and the leader) for each WPP substream and proceeds along the LCU row direction to the other side. Thus, by adhering to certain encoding delays between the sequences of WPP substreams of the same images 12 and 15, these WPP substreams 100 can be decoded/encoded in parallel, such that the parallel (i.e., simultaneous) decoding/encoding of portions of each image 12, 15 forms a wavefront 108 that moves from left to right across the image in a tile-like manner.

It's important to note that sequences 102 and 104 are still within the LCUs from the top left LCU 101 to the bottom right LCU, defining the raster scan order row by row from top to bottom. Each WPP substream can correspond to one LCU row. Returning simply to the tiles, tiles can also be confined to alignment with LCU boundaries. With respect to the boundary between two slices within a substream, the substream can be divided into one or more slices without being bound to the LCU boundary. However, when transitioning from one slice of a substream to the next, in that case, entropy probability is used. In the case of tiles, with respect to the boundary between two slices within a tile, the entire tile can be summarized as one slice, or a tile can be divided into one or more slices without being bound to the LCU boundary. In the case of tiles, the order changes between LCUs so that before continuing into the next tile according to the tile order, the tiles in the tile order are traversed first according to the raster scan order.

As described so far, image 12 can be divided into tiles or WPP substreams, and similarly, image 15 can also be divided into tiles or WPP substreams. Theoretically, one of images 12 and 15 can be assigned a WPP substream partition/concept, while the other can be assigned a tile partition/concept. Alternatively, a constraint can be imposed on the bitstreams, according to which the concept type (i.e., tile or WPP substream) must be the same across layers. Another instance of a spatial segment includes slices. Slices are used to partition bitstream 40 for transmission purposes. Slices are packaged within NAL units, which are the smallest entities used for transmission. Each slice is individually encodeable/decodeable. That is, any predictions across slice boundaries are prohibited, such as context selection. These are exactly three instances of spatial segments: slices, tiles, and WPP substreams. Furthermore, all three parallelized concepts (tiles, WPP substreams, and slices) can be used in combination; that is, image 12 or image 15 can be divided into tiles, where each tile is divided into multiple WPP substreams. Furthermore, slicing can be used to divide the bitstream into multiple NAL units, for example, (but not limited to) on tile or WPP boundaries. If tiles or WPP substreams are used and slicing is also used to divide images 12 and 15, and the slicing deviates from other WPP/tile divisions, then the spatial segment is defined as the smallest individually decodeable portion of images 12 and 15. Alternatively, constraints can be imposed on the bitstream regarding whether combinations of those concepts can be used within images (12 or 15) and/or at boundaries need to be aligned between the different concepts used.

The foregoing discussed various prediction modes supported by the encoder and decoder, the constraints imposed on the prediction modes, and the context derivation used for entropy coding/decoding to support parallel coding concepts, such as tiles and/or WPP concepts. It was also proposed that the encoder and decoder can operate block-by-block. For example, the aforementioned prediction modes can be selected block-by-block, i.e., with a finer granularity than the image itself. Before continuing with the description of aspects of this application, the relationship between slices, tiles, WPP substreams, and the aforementioned blocks according to one embodiment is explained.

Figure 4 illustrates an image, which can be a layer 0 image (e.g., image 12) or a layer 1 image (e.g., image 15). The image is regularly subdivided into an array of blocks 90. Sometimes, these blocks 90 are referred to as maximum coding blocks (LCBs), maximum coding units (LCUs), coding tree blocks (CTBs), etc. The subdivision of the image into blocks 90 can form a basic or coarsest granularity at which the aforementioned prediction and residual coding can be performed, and this coarsest granularity (i.e., the size of the block 90) can be signaled and set separately for layers 0 and 1 by the encoder. Subdivision can be performed within the data stream using signaling multi-way trees (e.g., quadtrees) to subdivide each block 90 into prediction blocks, residual blocks, and/or coding blocks respectively. In particular, the coding block can be a leaf block of a recursive multi-way tree subdivision of block 90, and some prediction-related decisions can be signaled at the granularity of the coding block (e.g., prediction mode), and the prediction block and residual block can be leaf blocks of separate recursive multi-way tree subdivisions of the coding block. At the granularity of the prediction block, prediction parameters (e.g., motion vectors) will be encoded in the case of intra-time frame prediction and disparity vectors in the case of inter-layer prediction. At the granularity of the residual block, prediction residuals are encoded.

The raster scan encoding/decoding order 92 can be defined within block 90. The encoding/decoding order 92 restricts the availability of adjacent portions for spatial prediction purposes: only portions of the image that, according to the encoding/decoding order 92, precede the current portion (e.g., block 90) or one of its smaller blocks related to the syntax element to be predicted, are available for spatial prediction within the current image. Within each layer, the encoding/decoding order 92 traverses all blocks 90 of the image so that it then continues traversing the blocks of the next image in the corresponding layer according to the image encoding/decoding order, which does not necessarily follow the temporal reproduction order of the images. Within each block 90, the encoding/decoding order 92 is refined into scans within smaller blocks (e.g., encoding blocks).

Relative to the blocks 90 and smaller blocks just outlined, each image is further subdivided into one or more slices along the aforementioned encoding/decoding sequence 92. Slices 94a and 94b, exemplarily shown in Figure 4, correspondingly cover the images without gaps. The boundary or interface 96 between consecutive slices 94a and 94b within an image may or may not be aligned with the boundary of adjacent block 90. More specifically, and as shown on the right-hand side of Figure 4, consecutive slices 94a and 94b within an image may adjoin each other at the boundary of smaller blocks (e.g., encoding blocks), i.e., leaf-like segments subdividing a block 90.

The slices 94a and 94b of the image can form the smallest unit, where portions of the data stream encoded from the image can be packaged into data packets, i.e., NAL units. Further performance characteristics of the slices were described above, namely, the constraints on the slices regarding (e.g.) predictions across slice boundaries and entropy context determination. Slices with such constraints can be called “ordinary” slices. In addition to ordinary slices, there are also “dependent slices”, as described in more detail below.

If the concept of tile division is applied to an image, then the encoding/decoding order 92 defined within the array of blocks 90 can be changed. This is shown in Figure 5, where the image is exemplarily shown as divided into four tiles 82a to 82d. As shown in Figure 5, the tiles themselves are defined as regular subdivisions of the image, in units of blocks 90. That is, each tile 82a to 82d consists of an array of n x m blocks 90, where n is set individually for each row of tiles and m is set individually for each column of tiles. Following the encoding/decoding order 92, the blocks 90 within the first tile are first scanned in raster scan order before proceeding to the next tile 82b, and so on, where tiles 82a to 82d themselves are scanned in raster scan order.

Based on the WPP stream partitioning concept, the image is subdivided into WPP substreams 98a to 98d along the encoding/decoding order 92, with one or more rows of blocks 90 as units. For example, each WPP substream can cover an entire row of blocks 90, as shown in Figure 6.

However, the tile concept and the WPP sub-stream concept can also be mixed. In this case, for example, each WPP sub-stream covers a row of blocks 90 within each tile.

Even image slicing can be used in conjunction with tile slicing and/or WPP substream slicing. Regarding tiles, each of one or more slices into which an image is subdivided can consist of exactly one complete tile, more than one complete tile, or only a sub-part of a tile along the encoding/decoding sequence 92. Slices can also be used to form WPP substreams 98a to 98d. For this purpose, the slice forming the smallest unit for packetization can include, on the one hand, ordinary slices, and on the other hand, relevant slices: while ordinary slices impose the aforementioned restrictions on prediction and entropy context derivation, relevant slices do not. Relevant slices at the boundaries of the image, where the encoding/decoding sequence 92 substantially departs from the point where the row begins, employ the entropy context generated by entropy decoding block 90 directly preceding block 90, and relevant slices starting elsewhere can employ the entropy coding context generated by entropy coding/decoding the slice directly preceding it until its end. Through this measure, each WPP substream 98a to 98d can consist of one or more relevant slices.

That is, the encoding/decoding order 92 defined within block 90 linearly proceeds from the first side (exemplarily the left side in this case) of the corresponding image to the opposite side (exemplarily the right side), and then moves down/to the next line of block 90. Therefore, the available (i.e., already encoded/decoded) portion of the current image is primarily located to the left and top of the currently encoded/decoded portion (e.g., the current block 90). Due to prediction interruptions and entropy context derivation at tile boundaries, tiles of an image can be processed in parallel. Encoding/decoding of tiles of an image can even begin simultaneously. If crossing tile boundaries is also permitted, the limitation stems from the aforementioned loop filtering. Encoding/decoding of the starting WPP substream is then performed in an interleaved manner from top to bottom. Within block 90 (two blocks 90), intra-frame image latency between consecutive WPP substreams is measured.

However, it is advantageous to parallelize the encoding/decoding of images 12 and 15, i.e., at different layers. Clearly, encoding/decoding of image 15, which is related to the layer, must be delayed relative to the base layer's encoding/decoding to guarantee a "spatial correspondence" portion of the base layer that is already available. These ideas remain valid even if no parallelization of encoding/decoding is used independently within any of images 12 and 15. Even when using a single sheet to cover the entire images 12 and 15 separately, without using tiles and without WPP substream processing, the encoding/decoding of images 12 and 15 can be parallelized. Even in the case where tiles or WPP processing are used for any image of a layer, or regardless of whether tiles or WPP processing are used for any image of a layer, the signaling described below (i.e., aspect 6) represents the possibility of such encoding/decoding delays between layers.

Before discussing the concepts presented above in this application, referring again to Figures 1 and 2, it should be noted that the module structures of the encoder and decoder in Figures 1 and 2 are for illustrative purposes only, and the structures may also be different.

In relation to the above description concerning the minimum coding delay between consecutive layers, it should be noted that the decoder can determine the minimum decoding delay based on short-term syntax elements. However, when using long-term syntax elements to signal this inter-layer time delay in advance for a predetermined time period, the decoder can use the provided guarantees to plan for the future and can more easily perform workload allocation within the parallel decoding of bitstream 40.

The first aspect involves limiting inter-layer prediction between views, particularly, for example, parallax-compensated inter-view prediction, supporting lower overall encoding/decoding latency or parallelization capabilities. Details are readily apparent from the following figures. For a simplified explanation, refer to Figure 7.

For example, the encoder can restrict the available domain 301 of the disparity vector of the current block 302 of the relevant view to interlayer prediction at the boundary 300 of the base segment. 303 represents a restriction; for comparison, Figure 7 shows another block 302' of the relevant view without restricting the available domain of its disparity vector. The encoder can signal this behavior, i.e., restriction 303, within the data stream to allow the decoder to utilize it in a low-latency sense. The decoder can operate normally as long as interlayer prediction is relevant to the encoder, but without the need for "unavailable segments," i.e., the decoder can maintain lower interlayer latency. Alternatively, with respect to interlayer prediction at boundary 300, both the encoder and decoder change their operating modes to, for example, further utilize the lower diversity of available states of the interlayer prediction parameters at boundary 300.

Figure 8 illustrates a multi-view encoder 600 configured to encode multiple views 12 and 15 into a data stream 40 using inter-view prediction. In the case of Figure 8, the number of views is exemplaryly chosen to be two, with arrow 602 indicating the inter-view prediction from the first view 12 to the second view 15. Expansion toward more than two views is readily conceivable. This also applies to the implementation described below. The multi-view encoder 600 is configured to modify the inter-view prediction at the spatial segment boundaries 300 of the spatial segments 301 into which the first view is divided.

Regarding the possible implementation details of encoder 600, refer, for example, to the description presented above in Figure 1. That is, encoder 600 can be a picture or video encoder and can operate in a block-by-block manner. In particular, encoder 600 can be a hybrid coding type, configured to subject the first view 12 and the second view 15 to predictive coding, insert prediction parameters into data stream 40, transform the encoded prediction residuals using spectral decomposition into data stream 40, and, at least with respect to the second view 15, switch between different prediction types, including at least spatial and inter-view prediction 602. As mentioned above, the unit by which encoder 600 switches between different prediction types/modes can be called a coding block, the size of which can vary because these coding blocks can represent, for example, leaf blocks of a hierarchical multi-branch tree subdivision of the image of the second view 15 or root blocks of a tree that can be regularly pre-divided into the image of the second view 15. The result of inter-view prediction is the prediction of samples within the corresponding coding block using a disparity vector 604, which represents the displacement of a spatially co-located portion 606 of the image of the first view 12, spatially co-located to the inter-view prediction block 302 of the image of the second view 15, in order to access portion 608, and from which samples within block 302 are predicted by copying a reconstructed version of portion 608 into block 302. However, inter-view prediction 602 is not limited to this type of inter-view prediction of sample values from the second view 15. Specifically, in addition to or alternatively, inter-view prediction supported by encoder 600 can be used for predictive coding prediction parameters themselves: it is envisioned that, in addition to the inter-view prediction modes outlined above, encoder 600 also supports spatial and/or temporal prediction. Spatial prediction of a coding block ends with prediction parameters inserted into data stream 40 for that coding block, just as temporal prediction does. Instead of encoding all these prediction parameters of the coded blocks of the image of the second view 15 separately into the data stream 40, the encoder 600 can use predictive coding and predictive parameters for predictively encoding the coded blocks of the second view 15, based on the prediction parameters or other information available to the encoder 600 from the portion of the data stream 40 encoded from the first view 12. That is, for example, the prediction parameters of a certain coded block 302 of the second view 15, such as motion vectors, can be predicted based on the motion vectors of the corresponding (also temporally predicted) coded blocks of the first view 12. "Correspondence" can take into account the parallax between views 12 and 15. For example, both first and second views 12 and 15 can have associated depth maps, and encoder 600 can be configured to encode texture samples of views 12 and 15 together with associated depth values of the depth maps into data stream 40. Encoder 600 can use depth estimates of coded blocks 302 to determine “corresponding coded blocks” within the first view 12, where the scene content of the first view better matches the scene content of the current coded block 302 of the second view 15. Essentially, encoder 600 can also determine this depth estimate based on the disparity vectors used by coded blocks predicted between neighboring views of view 15, regardless of whether any depth maps are encoded or not.

As described above, the encoder 600 of Figure 8 is configured to alter the inter-view prediction at spatial segment boundaries 300. That is, the encoder 600 changes the manner of inter-view prediction at these spatial segment boundaries 300. The reasons and objectives are further outlined below. In particular, the encoder 600 alters the manner of inter-view prediction in such a way that the predicted entities of the second view 15, for example, the texture sample content of the inter-view predicted coding block 300, or a prediction parameter of such a coding block, depend only exactly on one spatial segment 301 of the first view 12, through the inter-view prediction 602. Its advantages can be readily understood by observing the results of the change in the inter-view prediction of a coding block, the sample values used for inter-view prediction, or the prediction parameters. Without altering or limiting the inter-view prediction 602, encoding the coding block must be postponed until the encoding of two or more spatial segments 301 of the first view 12 involved in the inter-view prediction 602 is completed. Therefore, the encoder 600 must adhere to this inter-view coding delay/offset in all cases, and the encoder 600 encodes views 12 and 15 in a time-overlapping manner, without further reducing the coding delay. The situation is different when the inter-view prediction 602 is changed/modified at the spatial segment boundary 301 as described above. This is because, in this case, once one (only one) spatial segment 301 of the first view 12 is encoded, the encoding block 300 (some entities performing inter-view prediction) under discussion can be encoded. Therefore, the possible encoding delay is reduced.

Therefore, Figure 9 illustrates a multi-view decoder 620 in conjunction with the multi-view encoder of Figure 8. The multi-view decoder 620 of Figure 9 is configured to reconstruct multiple views 12 and 15 from data stream 40 using inter-view prediction 602 from the first view 12 to the second view 15. As described above, the decoder 620 can re-perform the inter-view prediction 602 in the same manner as it should be done by the multi-view encoder 600 of Figure 8: by reading prediction parameters contained within the data stream 40 and applying these prediction parameters, for example, the prediction mode represented by the corresponding coded blocks for the second view 15 (some of which are inter-view prediction coded blocks). As described above, the inter-view prediction 602 can alternatively or additionally relate to the prediction of the prediction parameters themselves, wherein, for such inter-view prediction prediction parameters, data stream 40 may include prediction residuals or indices pointing to a series of predictors, one of which performs inter-view prediction according to 602.

As already described in Figure 8, the encoder can modify the way inter-view predictions are made on boundary 300 to avoid inter-view prediction 602 that combines information from two segments 301. The encoder 600 can achieve this result transparently to the decoder 620. That is, the encoder 600 can impose self-restrictions only on its choices of possible encoding parameter settings so that the decoder 620 applies only the encoding parameters set in this configuration transmitted within data stream 40, inherently avoiding the combination of information from two different segments 301 within inter-view prediction 602.

That is, as long as the decoder 620 applies parallel processing to the decoding of the data stream 40, and is not interested in or unable to apply parallel processing, by decoding views 12 and 15 in parallel, the decoder 620 can simply ignore the signaling of the encoder 600 inserted within the data stream 40, and the aforementioned changes in the inter-view predictions of the signaling. More specifically, according to one embodiment of this application, the encoder of FIG8 transmits as a signal within the data stream the changes in the inter-view predictions at the segment boundary 300 within the data stream 40, i.e., whether there are any changes or no changes at the boundary 300. If signaling is applied, then the decoder 620 can regard the changes in the inter-view prediction 602 at the boundary 300 as a guarantee, i.e., limiting the inter-view prediction 602 at the spatial segment boundary 300 of the spatial segment 301, such that the inter-view prediction 602 does not involve any dependence of any part 302 of the second view 15 on any spatial segment other than the spatial segment where the common location part 306 of the first view 12, which is co-located to the corresponding part 302 of the second view, is located. That is, if a change in the inter-view prediction 602 on boundary 300 is applied as signaling, then decoder 620 can treat this as a guarantee that for any block 302 of the relevant view 15 for which the inter-view prediction 602 is used to predict any of its samples or prediction parameters, this inter-view prediction 602 does not introduce any dependency on any “adjacent spatial segments”. This means the following: for each part/block 302, there is a co-location portion 606 of the first view 12 that is co-located with the corresponding block 302 of the second view 15. For example, “co-location” is intended to mean, for example, that the circumference of a block within view 12 is exactly co-indices with the circumference of block 302. Alternatively, “co-location” is measured not by sample precision, but by the granularity of the blocks into which the image of layer 12 is divided, in order to determine “co-located” blocks, resulting in the selection of that block from the blocks into which the image of layer 12 is divided, i.e.,, for example, selecting a block that contains a co-location to the upper left corner of block 302 or another representative location of block 302. The “co-located portion/block” is denoted as 606. It's important to remember that because views 12 and 15 have different viewing orientations, the co-located portion 606 may not include the same scene content as portion 302. However, in the case of inter-view prediction change signaling, decoder 620 assumes that any portion/block 302 of the second view 15 undergoing inter-view prediction 602 depends solely on the spatial segment 301 within which the co-located portion/block 606 resides. That is, when viewing the images of the first and second views 12 and 15 registered to each other, the inter-view prediction 602 does not cross the segment boundaries of the first view 12, but remains within those segments 301 where the corresponding portion/block 302 of the second view 15 resides. For example, the multi-view encoder 600 appropriately restricts the signaling/selection of the disparity vector 604 for the inter-view prediction portion/block 302 of the second view 15, and/or appropriately encodes/selects the index within the predictor list so as not to index the predictor involved in the inter-view prediction 602 from the information of "adjacent spatial segments 301".

Before proceeding with the description of the encoder and decoder of Figures 8 and 9, which represent various implementations that can be combined or not combined with each other, note the following. It is apparent from the descriptions of Figures 8 and 9 that the encoder 600 can implement its “variations/restrictions” on the inter-view prediction 602 in different ways. In a more relaxed restriction, the encoder 600 restricts the inter-view prediction 602 simply by not combining information from two or more spatial segments. The description of Figure 9 is characterized by a more stringent instance of restriction, according to which the inter-view prediction 602 is even restricted to not crossing spatial segments 302: that is, any part/block 302 of the second view 15 subject to the inter-view prediction 602 obtains its inter-view prediction solely from the information of that spatial segment 301 of the first view 12 where its “co-located block/part 606” is located via the inter-view prediction 602. The encoder performs accordingly. This latter type of restriction represents an alternative to the description of Figure 8 and is even more stringent than the restrictions described above. The decoder 602 can utilize these restrictions according to both alternatives. For example, if a signaling is to be applied, decoder 602 can utilize the constraints of inter-view prediction 602 by reducing/lowering the inter-view decoding offset/delay relative to first view 12 when decoding second view 15. Alternatively or further, when deciding to attempt to decode views 12 and 15 in parallel, decoder 602 can consider a guarantee signaling: if this guarantee signaling is applied, the decoder can wait to attempt inter-view parallel processing and suppress that attempt. For example, in the example shown in Figure 9, where first view 12 is regularly divided into four spatial segments 301, each representing a quarter of the image of first view 12, decoder 620 can begin decoding second view 15 once the first spatial segment 301 of first view 12 is fully decoded. Otherwise, assuming the disparity vector 604 is only horizontal, decoder 620 must wait at least until the two upper spatial segments 301 of first view 12 are fully decoded. A stricter variation/constraint in inter-view prediction along segment boundaries 300 makes it easier to utilize the guarantee.

The aforementioned guarantee signaling can have scope/validity, for example, including only a single image or even a series of images. Therefore, as described later, signaling can be performed within a video parameter set, a sequence parameter set, or even an image parameter set.

To date, embodiments have been presented in Figures 8 and 9, according to which, apart from guaranteeing signaling, the manner in which data stream 40 and the encoders and decoders of Figures 8 and 9 encode/decode the data stream do not change with variations in inter-view prediction 602. Specifically, the manner in which the data stream is decoded/encoded remains the same, regardless of whether self-constraints are applied within inter-view prediction 602. According to an alternative embodiment, however, the encoders and decoders may even change the manner in which they encode/decode data stream 40 to take advantage of the guarantee situation, namely, the constraint of inter-view prediction 602 on the spatial segment boundary 300. For example, for the block/part 302 of inter-view prediction of the second view 15 near the common location of the spatial segment boundary 300 of the first view 12, the domain of possible disparity vectors for trusting signaling in data stream 40 may be constrained. For example, referring again to Figure 7, as described above, Figure 7 shows two exemplary blocks 302' and 302 of the second view 15, one of which (i.e., block 302) is close to the common location of the spatial segment boundary 300 of the first view 12. The co-location position of the spatial segment boundary 300 in the first view 12 is shown as 622 when transferred into the second view 15. As shown in FIG7, the co-located block 306 of block 302 is close to the spatial segment boundary 300, vertically separating the spatial segment 301a including the co-located block 606 and the horizontally adjacent spatial segment 301b. Moving the co-located block/part 606 to the right (i.e., towards the adjacent spatial segment 301b) with a disparity vector that is somewhat too large will cause at least a partial copying of the inter-view prediction block 302 from the sample of this adjacent spatial segment 301b. In this case, the inter-view prediction 602 intersects the spatial segment boundary 300. Therefore, in the "guaranteed case", the encoder 600 does not select such a disparity vector of block 302, thus limiting the encodeable domain of possible disparity vectors of block 302. For example, when using Huffman coding, the Huffman code used to encode the disparity vector of the inter-view prediction block 302 can be changed to take advantage of the limited domain of possible disparity vectors. In the case of using arithmetic coding, for example, another binarization combined with a binary arithmetic scheme can be used to encode disparity vectors, or another probability distribution can be used among the possible disparity vectors. According to this implementation, by reducing the amount of side information to be transmitted within the data stream 40 relative to the disparity vectors of the spatial segment 302 near the co-location of the spatial segment boundary 300, the slight reduction in coding efficiency caused by the inter-view prediction constraints on the spatial segment boundary 300 can be partially compensated.

Therefore, according to the above implementation, depending on whether a guarantee is applied, both the multi-view encoder and the multi-view decoder change the way they decode/encode disparity vectors from the data stream. For example, they change the Huffman code used to decode/encode the disparity vectors, or they change the binarization and/or probability distribution used to arithmetically decode/encode the disparity vectors.

To illustrate more clearly with respect to specific examples how the encoder and decoder in Figures 8 and 9 restrict the domain of possible disparity vectors for trusted orders within data stream 40, refer to Figure 10. Figure 10 again shows the typical behavior of the encoder and decoder for the inter-view prediction block 302: a disparity vector 308 is determined for the current block 302 within the domain of possible disparity instances. Thus, block 302 is the prediction block for disparity compensation prediction. Then, at reference portion 304, the first view 12 is sampled at reference portion 304, which is offset from the colocation portion 306 of the first view 12 from the current portion 302 by the determined disparity vector 308. The restriction on the domain of possible disparity vectors for trusted orders within the data stream is performed as follows: a restriction is made such that reference portion 304 is entirely within the spatial segment 301a in which colocation portion 306 is spatially located. For example, the disparity vector 308 shown in Figure 10 does not satisfy this restriction. Therefore, it is located outside the domain of the possible disparity vectors of block 302, and according to one implementation, it is not signalable within the data stream 40 with respect to block 302. However, according to an alternative implementation, the disparity vector 308 is signalable within the data stream, but under certain conditions, the encoder 600 avoids applying this disparity vector 308 and, for example, chooses to apply another prediction mode of block 302, such as a spatial prediction mode.

Figure 10 also illustrates that, in order to implement the domain limitation of the disparity vector, the half-width 10 of the interpolation filter kernel can be considered. More specifically, when copying the sample content of the disparity-compensated prediction block 302 from the image of the first view 12, in the case of sub-pixel disparity vectors, each sample of block 302 can be obtained from the first view 12 by applying interpolation using an interpolation filter with a certain interpolation filter kernel size. By combining samples located at the center of sample position "x" within filter kernel 311, the sample value shown using "x" in Figure 10 can be obtained. Therefore, in this case, the domain of the possible disparity vector of block 302 can be restricted so that no sample in reference portion 304 or filter kernel 311 covers the adjacent spatial segment 301b, but remains within the current spatial segment 301a. Thus, the domain of the signaling can be restricted or not. According to an alternative implementation, the samples of filter kernel 311 located in adjacent spatial segment 301b can be filled simply according to a certain exception rule to avoid additional restrictions on the domain of the possible disparity vector for sub-pixel disparity vectors. However, the decoder supports alternative filling only when the signaling guarantee is applied.

The latter example clearly demonstrates that, in addition to or instead of changes in the decoded data stream, decoder 620 may or may not, in response to signaling and data streams inserted into the data stream by encoder 600, alter the way it performs interview predictions at spatial segment boundaries 300. For example, as just described, encoders and decoders may, depending on whether a guarantee is applied, differently fill the interpolation filter kernel in the portion extending across spatial segment boundary 300. The same applies to reference portion 306 itself: this also allows at least partial extension into adjacent spatial segment 301b to alternatively fill the corresponding portion using information independent of any information located outside the current spatial segment 301a. In fact, encoders and decoders may, under a guarantee, treat spatial boundary 300 as a picture boundary and fill portions of reference portion 304 and/or interpolation filter kernel 311 by extrapolating from the current spatial segment 301a.

As mentioned above, the inter-view prediction 602 is not limited to the prediction of sample-by-sample content of the inter-view prediction block 302. Specifically, inter-view prediction can also be applied to the prediction of prediction parameters, such as motion parameters related to the prediction of the temporal prediction block 302 of view 15, or spatial prediction parameters related to the prediction of the spatial prediction block 302. To illustrate the possible variations and limitations imposed on this set of inter-view predictions 602 at boundary 300, refer to Figure 11. Figure 11 shows the block 302 of the relevant view 15, whose parameters are predicted using, at least particularly, inter-view prediction. For example, a list of several predictors for the parameters of block 302 can be determined by the inter-view prediction 602. For this purpose, for example, the encoder and decoder perform the following: A reference portion of the first view 12 is selected for the current block 302. The selection or derivation of the reference portion/block 314 is performed among the coding blocks, prediction blocks, etc., divided by the image of the first layer 12. For its derivation, the representative position 318 within the first view 12 can be determined as either the representative position 628 of block 302 or the representative position 630 of an adjacent block 320 adjacent to block 302. For example, the adjacent block 320 could be the block located at the top of block 302. Determining block 320 can involve selecting block 320 from the blocks divided from the image of the second view layer 15, as a block that includes the sample directly above the sample located at the top left corner of block 302. Representative positions 628 and 630 could be samples located at the top left corner or samples located in the middle of the block, etc. Then, the reference position 318 within the first view 12 is the position co-located to 628 or 630. Figure 11 shows the co-location to position 628. The encoder/decoder then estimates the disparity vector 316. For example, this can be based on an estimated depth map of the current scene or using the already decoded disparity vectors that are in the spatial-temporal neighborhood of block 302 or block 320, respectively. The disparity vector 316 thus determined is applied to the representative position 318 so that the head of the vector 316 points to position 632. In the division of the image of the first view 12 into blocks, the reference portion 314 is selected to include the portion at position 632. As described above, the division in selecting the portion/block 314 can be a division of the view 12 into coded blocks, prediction blocks, residual blocks, and/or transform blocks.

According to one implementation, the multi-view encoder checks only whether the reference portion 314 is located within an adjacent spatial segment 301b, i.e., excluding spatial segments of blocks where the common location of reference point 628 is located. If the encoder sends the aforementioned guarantee signaling to the decoder, then the encoder 600 suppresses any application of parameters to the current block 302. That is, the predictor list for the parameters of block 302 may include inter-view predictors that cause the boundary 300 to intersect, but the encoder 600 avoids selecting such predictors and selects the index of block 302 that does not point to an unwanted predictor. If the multi-view encoder and decoder check whether the reference portion 314 is located within an adjacent spatial segment 301b under the guarantee, then the encoder and decoder may use another predictor instead or simply exclude the “boundary intersection” inter-view predictor from the predictor list. For example, the predictor list may also include parameters for spatial and/or temporal prediction and/or one or more default predictors. Conditional checks (i.e., whether the reference portion 314 is or is not part of spatial segment 301a) and conditional substitutions or exclusions are performed only under the guarantee. In the absence of a guarantee, any check on whether reference portion 314 is within spatial segment 301a can be stopped, and the prediction of parameters for block 302 can be performed by applying predictors obtained from the properties of reference portion 314, regardless of whether reference portion 314 is within spatial segment 301a or 301b. Without adding any predictors obtained from the properties of reference portion 314 to the predictor list of the current block 302 or adding replacement predictors, the encoder and decoder 620 perform the corresponding transformations of the usual inter-view prediction based on the reference block 314 located within or outside spatial segment 301a. Through this measure, any predictor index within the determined list of predictors for block 302 points to the same list of predictors within the decoder. The domain of the trust order for the indexes of block 302 can be restricted in response to whether a guarantee is applied. When the guaranteed condition is applied, the encoder performs the check alone, and the multi-view encoder forms a predictor list for block 302, independent of the reference portion 314 located within spatial segment 301a (and even independent of whether the guaranteed condition is applied). However, in the guaranteed condition, the index is restricted so that predictors are not selected from the predictor list, in case a predictor is obtained from the properties of block 314 located outside spatial segment 301a. In this case, decoder 620 can obtain the predictor list for block 302 in the same way, i.e., in the same way in both the guaranteed and unguaranteed conditions, because encoder 600 has noted that inter-view prediction does not require any information from adjacent spatial segments 301b.

Regarding the parameters of block 302 and the properties of reference portion 314, it should be noted that they can be motion vectors, disparity vectors, residual signals (e.g., transform coefficients), and/or depth values.

The concept of inter-view predictive coding described in Figures 8 through 11 can be incorporated into the currently envisioned extension of the HEVC standard, i.e., as described below. The description presented directly below is also interpreted as relating to the possible implementation details of the descriptions presented above in Figures 8 through 11.

As an important point to note, the spatial segment 301 discussed above, which forms a unit on the boundary where the prediction between changing/limiting views is located, does not necessarily need to be formed in a unit that mitigates or supports inter-layer parallel processing. In other words, while the spatial segment discussed above in Figures 8 to 11 can be a tile divided by the base layer 12, other instances are also possible, such as spatial segment 301 forming an instance of the coding tree root block (CTB) of the base layer 12. In the embodiments described below, spatial segment 301 is associated with the definition of a tile, i.e., a spatial segment is a tile or a group of tiles.

As explained later regarding the limitations of ultra-low latency and parallelization in HEVC, inter-layer prediction is constrained by ensuring the partitioning of the base layer image (especially tiles).

HEVC allows the encoded base layer image's CTB to be divided into rectangular regions called tiles using a convenient vertical and horizontal grid. These regions can be processed individually, except for loop filtering. Loop filtering can be turned off at the tile boundaries to make them completely independent.

Breaking the dependency between resolution and prediction at tile boundaries is similar to doing so at image boundaries; however, loop filters can cross tile boundaries if configured accordingly to reduce tile boundary artifacts. Therefore, the processing of individual tiles does not entirely depend on other tiles within the image, or depends to a large extent on the filtering configuration. There are mounting constraints, namely, all CTBs of a tile should belong to the same tile, or all CTBs of a tile should belong to the same tile. As can be seen in Figure 1, tiles cause the CTB scanning order to be related to the tile order; that is, all CTBs belonging to the first (e.g., the top left) tile are traversed before continuing with all CTBs belonging to the second tile (e.g., the top right). The tile structure is defined by the number and size of CTBs forming a grid within each tile row and column within the image. This structure can be changed frame-by-frame or remain constant throughout the encoded video sequence.

Figure 12 shows an example of CTB within an image divided into 9 tiles. The thick black lines indicate tile boundaries, and the numbers indicate the scan order of the CTB, also revealing the tile order.

By decoding all the tiles covering the corresponding image region within the base bitstream, the HEVC extended enhancement layer tiles can be decoded.

The following sections use the concepts from Figures 7 to 11 to describe the constraints, signaling, and encoding/decoding variations that allow for lower inter-layer coding offsets/delays.

Decoding processes involving tile boundary modifications within HEVC can be performed as follows:

a) Motion or parallax vectors should not cross the tiles within the base layer.

If constraints are enabled, the following applies:

If inter-layer prediction (e.g., prediction of sample values, motion vectors, residual data, or other data) uses the base view (layer 12) as a reference image, then the disparity or motion vector should be constrained so that the reference image region belongs to the same tile as the co-located base layer CTU. In a specific implementation, during the decoding process, motion or disparity vectors 308 are pruned so that the reference image region lies within the same tile, and the reference subpixel location is predicted only from information located within the same tile. More specifically, in the current HEVC sample interpolation process, this would restrict motion vectors pointing to subpixel locations to be pruned 3 to 4 pixels away from the tile boundary 300, or in the inter-view residual prediction process for inter-view motion vectors, this would restrict disparity vectors pointing to locations within the same tile. An alternative implementation adjusts the subpixel interpolation filter to handle tile boundaries similar to image boundaries so that motion vectors can point to subpixel locations closer to the tile boundary than the kernel size 310 of the subpixel interpolation filter. The alternative implementation represents bitstream constraints, which prohibit the use of motion or disparity vectors that were pruned in the previously described implementation.

b) When in different tiles, adjacent blocks of the same configuration block in the base layer are not used.

If constraints are enabled, the following applies:

If the base layer is used for prediction of adjacent blocks (e.g., TMVP or adjacent block disparity derivation) and tiles are used, then the following applies: if CTU B belongs to the same tile as the adjacent base layer CTU A, then only predictor candidates derived from CTU B that are different from the adjacent CTU A within the base layer are used. For example, in the current HEVC derivation process, CTU B could be located to the right of the adjacent CTU A. In specific embodiments of the invention, different predictions are used instead of prediction candidates. For example, parallel PUs can be used instead of predictions. In another embodiment of the invention, the use of associated prediction modes within the encoded bitstream is prohibited.

Transferring the HEVC deformation possibilities just outlined to the descriptions of Figures 8 and 11, it is worth noting that, with regard to the predictor replacement in Figure 11, the replacement can be selected as the corresponding attribute of the block of the first layer 12, including the co-location of the reference position 628 of the current block 302 itself.

c) Signaling

In a specific implementation, the following advanced syntax can be used with a VPS or SPS to enable the use of N flags while having the above constraints/restrictions, for example, as shown in Figures 13A and 13B.

Here, PREDTYPE, RESTYPE, and SCAL in inter_layer_PREDTYPE_RESTYPE_SCAL_flag_1 to inter_layer_PREDTYPE_RESTYPE_SCAL_flag_N can be replaced by different values described below:

PREDTYPE indicates the type of forecast to which the constraints/restrictions apply, and may be one of the following or other forecast types not listed:

– For example, temporal_motion_vector_prediction is used to predict temporal motion vectors from adjacent blocks of co-located blocks within the base view;

– For example, disparity_vector_prediction is used to predict disparity vectors from neighboring blocks of a collocated block within the base view;

– For example, depth_map_derivation is used to predict depth values from a base view;

– For example, inter_view_motion_predition is used to predict motion vectors from a base view;

– For example, inter_view_residual_prediction is used to predict residual data from the base view;

– For example, inter_view_sample_prediction is used to predict sample values from a base view.

Alternatively, the signaling may not be explicitly used to restrict/constrain the forecast types to which it applies, and the restriction/constraint may apply to all forecast types or the restriction/constraint may be used only to signal the forecast type groups using one flag per group.

RESTYPE indicates the type of restriction and may be one of the following:

– For example, constraints (representing bitstream constraints, and flags can be included within the VUI);

– For example, restrictions (representing the choice of pruning (a) or different predictors (b)).

–SCAL indicates that the restriction/constraint applies only to layers of the same type:

– For example, same_scal (meaning the restriction only applies when the base layer and the enhancement layer are the same extensible type);

– For example, diff_scal (indicates that the restrictions apply regardless of the extensible type of the base layer and enhancement layer).

In the alternative implementation illustrated in Figure 14, the use of all described limitations can be signaled for ultra-low delay modes in high-level syntax, for example, as ultra_low_delay_decoding_mode_flag in a VPS or SPS.

An ultra_low_delay_decoding_mode_flag of 1 indicates that modified decoding is used at tile boundaries.

The limitations implied by this symbol may also include constraints on tile boundary alignment and limitations on the upsampling filter of the tile boundaries.

That is, referring to Figure 1, the guarantee signaling can be further used to guarantee that, within a predetermined time period (e.g., a time period extended over a series of pictures), the pictures 15 of the second layer are subdivided so that the boundaries 84 between the spatial segments 82 of the second layer pictures cover each boundary 86 of the spatial segment 80 of the first layer (possibly after upsampling if spatial scalability is taken into account). In time intervals shorter than the predetermined time period (e.g., on a per-picture basis) (i.e., within the picture spacing), the decoder still periodically determines that the pictures 12, 15 of the first and second layers are actually subdivided into spatial segments 80 and 82 according to the short syntax elements of the multi-layer video data stream 40, but the alignment indications already help in planning the allocation of parallel processing workloads. For example, the solid line 84 in Figure 1 represents an instance where the tile boundary 84 is completely spatially aligned with the tile boundary 86 of layer 0. However, the above guarantee also allows for tile subdivision of layer 1 to be more refined than that of layer 0, such that the tile subdivision of layer 1 includes further additional tile boundaries that do not spatially cover any tile boundaries 86 of layer 0. In any case, knowledge about tile alignment between layer 1 and layer 0 helps the decoder allocate workload or process available power within segments of space that are processed in parallel simultaneously. Without a long-term syntax element structure, the decoder needs to perform workload allocation in smaller time intervals (i.e., per picture), thus wasting computer power. On the other hand, there is "opportunistic decoding": a decoder with multiple CPU cores can utilize knowledge about the parallelization of layers to decide whether to attempt decoding layers with higher complexity—that is, a higher number of layers, or in other words, further views. By using all the cores of the same decoder, bitstreams exceeding the capabilities of a single core may be decodeable. This information is especially useful if profiles and rank indicators do not include such indications on minimum parallelization.

As described above, in the case of multi-layer video (where the base image 12 also has a different spatial resolution than the associated view image 15), guarantee signaling (exemplarily comparing ultra_low_delay_decoding_mode_flag) can be used to manipulate the upsampling filter 36. If upsampling filtering is performed in layer 0 above the spatial segment boundary 86, then during the combination of upsampling filters, the delay satisfied in the parallel decoding/encoding of the spatial segment 82 of layer 1 relative to the encoding/decoding of the spatial segment 80 of layer 0 increases, thereby making the information of adjacent spatial segments of layer 0 used as prediction reference 38 for inter-layer prediction in block 41 of layer 1 correlated with each other. For example, refer to Figure 15. Images 12 and 15 are displayed in an overlay manner, with the two images marked in size and aligned with each other according to spatial correspondence, i.e., parts of the same part of the scene are overlaid on each other. Images 12 and 15 are exemplarily shown as being divided into 6 and 12 spatial segments, respectively, e.g., tiles. The filter kernel is illustrated as moving over the top-left tile of image 12 to obtain its upsampled version, which serves as the basis for inter-layer prediction of any block within the tiles of image 15, spatially covering the top-left tile. In some intermediate cases, for example at 202, kernel 200 covers the adjacent tiles of image 12. The intermediate sample values of kernel 200 at position 202 of the upsampled version thus depend on two samples from the top-left tile of image 12 and samples from the tile of image 12 to its right. If the upsampled samples of image 12 are used as the basis for inter-layer prediction, then the inter-layer latency of the segments of the parallel processing layer increases. Therefore, constraints can help increase the amount of parallelization across different layers, thus reducing the overall coding latency. Naturally, syntax elements can also be long-term syntax elements that are valid for a range of images. The constraint can be achieved by, for example, filling the covered portion of kernel 200 at covered position 202, having sample values with a central tendency in the non-dashed portion of kernel 200, extrapolating the non-linear portion using linear or other functions into the dashed portion, etc.

The following provides an alternative implementation in a VPS as an example, where the aforementioned limits/constraints are controlled by the `ultra_low_delay_decoding_mode_flag`, but alternatively (when the flag is disabled), each limit/constraint can be enabled individually. For this implementation, refer to Figures 13C and 13D. This implementation can also be included within other non-VCL NAL units (e.g., SPS or PPS). In Figures 13C and 13D,

An ultra_low_delay_decoding_mode_flag value of 1 indicates that du_interleaving_enabled_flag, interlayer_tile_mv_clipping_flag, depth_disparity_tile_mv_clipping_flag, inter_layer_tile_tmvp_restriction_flag, and independent_tile_upsampling_idc are inferred to be equal to 1 and do not exist in VPS, SPS, or PPS.

When using parallelization techniques (e.g., tiles) in hierarchically encoded video sequences, it is advantageous to control the limitations of the encoding tools from a latency perspective, such as inter-view prediction in HEVC extensions, to avoid crossing tile boundaries in a uniform manner.

In one implementation, the value of `independent_tiles_flag` determines the various limitations/constraints controlled by syntax elements, such as `inter_layer_PREDTYPE_RESTYPE_SCAL_flag_x` or `independent_tile_upsampling_idc`. `independent_tiles_flag` can be contained within a VPS, as shown in Figure 13E. Here, an `independent_tiles_flag` equal to 1 specifies that `inter_layer_PREDTYPE_RESTYPE_SCAL_flag_1` through `inter_layer_PREDTYPE_RESTYPE_SCAL_flag_N` and `independent_tile_upsampling_idc` are inferred to be equal to 1 and do not exist within the VPS, SPS, or PPS.

In Figure 13F, an alternative implementation is provided in the VPS as an example, where the aforementioned constraints are controlled by the independent_tiles_flag, but alternatively (when the flag is disabled), each constraint can be enabled individually. This implementation can also be included in other non-VCL NAL units (e.g., SPS or PPS), as shown in Figure 13G.

In summary of the above embodiments described in Figures 8 through 15, decoder 620 may use guarantee signaling within the data stream to optimize inter-layer decoding offsets between different layers/views 12 and 15, or decoder 620 may use guarantees to suppress or allow inter-layer parallel processing attempts by referring to “opportunistic decoding” as described above.

The aspects of this application described below relate to the problem of allowing lower end-to-end latency in multi-layer video coding. It is worth noting that the aspects described below can be combined with those described above, but conversely, implementations relating to the aspects now described can also be implemented without the details described above. In this regard, it should also be noted that the implementations described below are not limited to multi-view coding. The multi-layers mentioned below in relation to the second aspect of this application can refer to different views, but can also represent the same view with different degrees of spatial resolution, SNR accuracy, etc. Possible scalability dimensions (along which the multi-layers discussed below increase the information content passed by the preceding layers) are diverse and include, for example, multiple views, spatial resolution, SNR accuracy, etc., and further possibilities become apparent in conjunction with the discussion of the third and fourth aspects of this application; according to one implementation, these aspects can also be combined with the aspects now described.

The second aspect of this application, now described, relates to the problem of practically achieving low coding latency, i.e., embedding the concept of low latency within the framework of NAL units. As described above, NAL units are composed of tiles. The concepts of tiles and/or WPPs are freely chosen individually for different layers of video data. Thus, each NAL unit having tiles encapsulated within it can be spatially attributed to the region of the picture to which the corresponding tile belongs. Therefore, in order to enable low-latency coding in the case of inter-layer prediction, it is advantageous to interleave NAL units belonging to different layers at the same time, so that the encoder and decoder can begin encoding and transmitting separately and decoding the tiles encapsulated within these NAL units in a manner that allows parallel processing of these pictures belonging to different layers at the same time. However, depending on the application, the encoder may preferably use different coding orders between pictures at different layers, for example, using different GOP structures for different layers, compared to the ability to allow parallel processing within the layer dimension. Therefore, according to the second aspect, the construction of the data stream is described again later in Figure 16.

Figure 16 illustrates multi-layer video data 201, which consists of a series of images 204 for each of the different layers. Each layer can describe a different property of the scene described by the multi-layer video data 201. That is, for example, the meaning of the layer can be selected among: color components, depth map, transparency, and/or viewpoint. Without loss of generality, it is assumed that different layers correspond to different views, where the video data 201 is multi-view video.

In applications requiring low latency, the encoder can determine the signaling long-term high-level syntax elements (compare: setting the du_interleaving_enabled_flag, as described below, to 1). In this case, the data stream generated by the encoder looks the same as the data stream represented in the middle of Figure 16, with circles around it. In this case, the multi-layer video stream 200 is composed of a sequence of NAL units 202, such that the NAL unit 202 belonging to one access unit 206 is associated with a picture at a certain moment in time, and the NAL units 202 of different access units are associated with different moments. Within each access unit 206, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units 208. This means that, as shown above, there are different types of NAL units within the NAL units 202, for example, one side being VCL NAL units and the other side being non-VCL NAL units. More specifically, the NAL units 202 can be of different types, and these types can include:

1) NAL units carry slices, tiles, WPP substreams, etc., meaning the syntax elements involve prediction parameters and/or residual data describing the image content at the image sampling scale/granularity. One or more of this type are possible. VCL NAL units are of this type. This type of NAL unit is not movable;

2) Parameter group NAL units can carry information that does not change frequently, such as long-term encoding settings, some of which have been described above. For example, such NAL units can be distributed throughout the data stream to some extent and repeatedly;

3) The Supplemental Enhancement Information (SEI) NAL unit can carry optional data.

The decoding unit can be composed of the first of the aforementioned NAL units. More precisely, the decoding unit can be composed of one or more VCL NAL units within the access unit and associated non-VCL NAL units. Thus, the decoding unit describes a region of an image, that is, a region encoded as one or more slices is contained therein.

Decoding units 208 associated with different layers of NAL units are interleaved so that, for each decoding unit, the interlayer prediction for encoding the corresponding decoding unit is based on a portion of the image of the layer other than the layer associated with the corresponding decoding unit, which is encoded as a decoding unit preceding the corresponding decoding unit within the corresponding access unit. For example, refer to decoding unit 208a in FIG. 16. Exemplarily, imagine this decoding unit is associated with region 210 of the corresponding image of the associated layer 2 and at a certain moment. A co-located region within the base layer image at the same moment is represented by 212, and needs to slightly exceed this region 212 of the base layer image in order to fully decode decoding unit 208a by utilizing interlayer prediction. For example, slightly exceeding could be the result of parallax compensation prediction. This, in turn, means that decoding unit 208b preceding decoding unit 208a within access unit 206 should completely cover the region required by the interlayer prediction. Referring to the above description, this relates to a delay indication that can be used as a boundary for the interleaving granularity.

However, if the application takes advantage of the freedom to choose the decoding order of images differently between different layers, the encoder can preferably set du_interleaving_enabled_flag to equal 0, which is described as 2 at the bottom of Figure 16, surrounded by a circle. In this case, the multi-layer video data stream has a separate access unit for each image belonging to one or more pairs of values of layer ID and a single transient moment. As shown in Figure 16, according to the (i-1)th decoding order, i.e., at time t(i-1), each layer can be constituted by or not by access units _AU1 , _AU2 (and so on) (compare: time t(i)), where all layers are contained within a single access unit _AU1 . However, interleaving is not allowed in this case. The access unit is set within the data stream 200 after the decoding order index i, i.e., the access unit for each layer at decoding order index i, then the access unit for the images of these layers corresponding to decoding order i+1, and so on. The question is whether the encoding order of the signaling is equal or different between time-series images within the data stream, and whether the encoding order is suitable for different layers. For example, the signal can be placed in one location within the data stream or even redundantly placed in more than one location, so as to be encapsulated into a sheet of NAL units.

Regarding NAL unit types, it should be noted that the ordering rules defined therein allow the decoder to determine the placement of boundaries between consecutive access units, regardless of whether movable packet type NAL units are removed during transmission. For example, movable packet type NAL units may include SEI NAL units, redundant image data NAL units, or other specific NAL unit types. That is, the boundaries between access units do not move but remain, and the ordering rules are still followed within each access unit, but the ordering rules are broken at each boundary between any two access units.

For completeness, Figure 17 shows the case where du_interleaving_flag = 1, for example, allowing packets belonging to different layers (but belonging to the same time t(i–1)) to be distributed within one access unit. The case where du_interleaving_flag = 0 is described as 2, with circles around it, consistent with Figure 16.

However, it should be noted in Figures 16 and 17 that the aforementioned interleaving signaling or interleaving signal can be stopped, resulting in a multi-layer video data stream. According to the case shown as 1 in Figures 16 and 17 (with a circle around it), the multi-layer video data stream must use the access unit definition.

According to one implementation, the decoder can, at its discretion, determine whether the NAL units contained within each access unit are actually interleaved relative to their relevance to the layer of the data stream. To facilitate data stream processing, syntax elements (e.g., `du_interleaving_flag`) can provide the decoder with interleaved or non-interleaved signaling for the NAL units within the access unit that collects all NAL units at a given timestamp, so that the decoder can more easily process the NAL units. For example, whenever interleaved signaling is enabled, the decoder can use more than one picture buffer, as simply illustrated in Figure 18.

Figure 18 illustrates a decoder 700, which can be embodied as described above in Figure 2 and may even conform to the description presented in Figure 9. Exemplarily, the multi-layer video data stream of Figure 17 (Option 1, circled around it) is shown as the input to the decoder 700. To facilitate deinterleaving of NAL units belonging to different layers (but belonging to a common moment) for each access unit AU, the decoder 700 uses two buffers 702 and 704. For each access unit AU, for example, multiplexer 706 forwards NAL units belonging to the first layer access unit AU to buffer 702, and, for example, forwards NAL units belonging to the second layer to buffer 704. The decoding unit 708 then performs decoding. For example, in Figure 18, NAL units belonging to the basic/first layer are shown without shaded lines, while NAL units belonging to the related/second layer are shown with shaded lines. If the aforementioned interleaving signal exists within the data stream, then decoder 700 can respond to this interleaving signaling in the following manner: if the interleaving signaling signals the NAL units to be enabled, i.e., NAL units of different layers are interleaved with each other within an access unit AU, then decoder 700 uses buffers 702 and 704, and multiplexer 706 distributes the NAL units on these buffers, as described above. However, if it does not exist, then decoder 700 uses only one buffer 702 and 704 for all NAL units included by any access unit, for example, buffer 702.

To better understand the implementation of FIG18, referring to FIG18 and FIG19, FIG19 shows an encoder configured to generate a multi-layer video data stream as described above. The encoder of FIG9 is generally represented by reference numeral 720 and encodes (here, exemplary) two layers of input images, which for ease of understanding are represented as layer 12 forming the base layer and layer 1 forming the related layer. As described above, different views can be formed. The general encoding order (along which encoder 720 may encode images of layers 12 and 15) scans the images of these layers generally along their temporal (display time) order, wherein the encoding order 722 may deviate from the display time order of images 12 and 15 on a group-by-group basis. At each transient moment, the encoding order 722 traverses the images of layers 12 and 15 along its dependency, i.e., from layer 12 to layer 15.

Encoder 720 encodes the images of layers 12 and 15 into a data stream 40, in units of the aforementioned NAL units, each NAL unit spatially associated with a portion of the corresponding image. Therefore, the NAL units belonging to a particular image are spatially subdivided or partitioned into the corresponding image as described above, and inter-layer prediction causes portions of the image of layer 15 to depend on portions of the temporally aligned image of layer 12, these portions being substantially co-located within the corresponding portions of the image of layer 15, "substantially" surrounding the parallax shift. In the example of Figure 19, encoder 720 chooses to utilize interleaving probabilities to form access units that collect all NAL units belonging to a given time. In Figure 19, the portions shown in data stream 40 correspond to the portions in the decoder input to Figure 18. That is, in the example of Figure 19, encoder 720 uses inter-layer parallel processing in encoding layers 12 and 15. For time t(i–1), as soon as NAL unit 1 of the image of layer 12 is encoded, encoder 720 begins encoding the image of layer 1. Each NAL unit that has been encoded is output by encoder 720, having an arrival timestamp corresponding to the time when the corresponding NAL unit was output by encoder 720. After encoding the first NAL unit of the image of layer 12 at time t(i–1), encoder 720 continues to encode the content of the image of layer 12 and outputs the second NAL unit of the image of layer 12, having an arrival timestamp after the arrival timestamp of the first NAL unit of the time-aligned image of layer 15. That is, encoder 720 outputs the NAL units of the images of layers 12 and 15 that belong to the same time in an interleaved manner, and through this interleaving, the NAL units of data stream 40 are actually transmitted. Encoder 720 selects the possible use of interleaving and indicates it in data stream 40 by encoder 720 through corresponding interleaving signaling 724. Since encoder 720 can output the first NAL unit of the relevant layer 15 at time t(i–1) earlier than in non-interlaced scenarios, the output of the first NAL unit of layer 15 is delayed until the encoding and output of all NAL units of the time-aligned base image are completed, so the end-to-end delay between the decoder in FIG18 and the encoder in FIG19 can be reduced.

As described above, according to an alternating instance, in the non-interleaved case, i.e., when signaling 724 represents a non-interleaved alternative, the definition of the access unit remains the same; that is, the access unit AU can collect all NAL units belonging to a certain moment. In this case, signaling 724 merely indicates whether NAL units belonging to different layers 12 and 15 are interleaved within each access unit.

As described above, according to signaling 724, the decoder of Figure 18 uses one or two buffers. With interleaving enabled, the encoder 700 distributes NAL units across the two buffers 702 and 704, such that, for example, the NAL units of layer 12 are buffered in buffer 702, while the NAL units of layer 15 are buffered in buffer 704. Buffers 702 and 704 are cleared unit by unit. This is true regardless of whether signaling 724 indicates interleaving or non-interleaving.

If the encoder 720 sets a removal time within each NAL unit, this is preferable so that the decoding unit 708 can take advantage of the possibility of decoding layers 12 and 15 from the data stream 40 using interleaved parallel processing. However, even if the decoder 700 does not apply interlayer parallel processing, the end-to-end latency has already been reduced.

As described above, NAL units can be of different NAL unit types. Each NAL unit can have a NAL unit type index, indicating the type of the corresponding NAL unit in a set of possible types. Within each access unit, the type of the NAL unit of the corresponding access unit can follow an ordering rule among NAL unit types, except between two consecutive access units, where the ordering rule is broken so that the decoder 700 can identify access unit boundaries by investigating this rule. For more information, refer to the H.264 standard.

In Figures 18 and 19, the decoding unit DU is identifiable as several consecutive rings of NAL units belonging to the same layer within an access unit. For example, the NAL units denoted as “3” and “4” in Figure 19 within the access unit AU(i–1) form a DU. Other decoding units in the access unit AU(i–1) each comprise only one NAL unit. The access units AU(i–1) in Figure 9 collectively exemplarily comprise six decoding units DU, which are alternatively arranged within the access unit AU(i–1), i.e., constituted by several rings of NAL units in a layer, which alternately changes between layer 1 and layer 0.

Similar to the first aspect, the second aspect described above can now be incorporated into the HEVC extension, as outlined below.

However, prior to this, for the sake of completeness, a further aspect of the current HEVC is described, which enables parallel processing between images, i.e., WPP processing.

Figure 20 illustrates the current implementation of WPP within HEVC. That is, this description also forms the basis for any alternative implementations of WPP processing in the embodiments described above or below.

At the base layer, wavefront parallel processing allows for parallel processing of Code Tree Block (CTB) rows. Prediction dependencies are not broken above the CTB rows. Regarding entropy coding, WPP alters the CABAC dependency on the top-left CTB within the corresponding upper CTB row, as shown in Figure 20. Once the entropy decoding of the corresponding top-right CTB is complete, entropy coding of the CTB in the following rows can begin.

In the enhancement layer, once the CTB containing the corresponding image region is fully decoded and available, the decoding of the CTB can begin.

In HEVC and its extensions, the following definition of the decoding unit is provided:

Decoding Unit: If SubPicHrdFlag equals 0, then it is an access unit; otherwise, it is a subset of access units, consisting of one or more VCL NAL units within the access unit and associated non-VCL NAL units.

In HEVC, if preferred by an external method, the hypothetical reference decoder (HRD) can optionally operate the CPB and DPB at the decoding unit level (or sub-picture level), and the sub-picture HRD parameters are available.

The HEVC specification[1] is characterized by the concept of a so-called decoding unit as defined below.

3.1 Decoding Unit: If SubPicHrdFlag equals 0, then it is an access unit; otherwise, it is a subset of access units, which consists of one or more VCL NAL units within the access unit and associated non-VCL NAL units.

In a layered encoded video sequence that exists within an HEVC extension for 3D[3], multiple views[2], or spatial scalability[4], where additional representations of the encoded video data (e.g., with higher fidelity, spatial resolution, or different camera views) are encoded according to the lower layer by predictive interlayer/interview coding tools, it is advantageous to interleave related or co-located decoding units of related layers (in terms of picture regions) within the bitstream to minimize end-to-end latency on the encoder and decoder.

In order to allow interleaving of decoding units within the encoded video bitstream, certain constraints on the encoded video bitstream must be signaled and enforced.

The following sections describe in detail how the above interleaving concept can be implemented within HEVC, and explain the reasons.

As far as the current state of the HEVC extension extracted from the initial draft of the MV-HEVC specification [2] is concerned, it emphasizes the use of the definition of an access unit, according to which an access unit contains an encoded picture (with a specific value of nuh_layer_id). An encoded picture is defined below in essentially the same way as a view component in MVC. Whether the access unit should be limited to containing all view components with the same POC value is an open question.

The basic HEVC specification[1] defines it as follows:

3.1 Access Unit: A group of NAL units that are related to each other according to the prescribed classification rules, are consecutive in the decoding order, and contain exactly one encoded image.

Note 1: In addition to VCL NAL units containing encoded images, access units may also contain non-VCL NAL units. Decoding an access unit always produces a decoded image.

It appears that by interpreting each relevant view as a separate encoded image and requiring it to be contained within a separate access unit, the interpretation allows for the definition of an access unit (AU) of only one encoded image within each access unit. This is described as "2" in Figure 17.

In the previous standard, an "encoded image" contained all layers of the view representation of an image at a certain timestamp.

Access units cannot be interleaved. This means that if each view is contained within a different access unit, then the entire image of the base view needs to be received within the DPB before the first decoding unit (DU) of the relevant view can be decoded.

For ultra-low latency operations with associated layers/views, interleaving decoding units can be advantageous.

The example in Figure 21 contains three views, each with three decoding units. The decoding units are received from left to right:

If each view is contained within its own access unit, then the minimum delay of the first decoding unit for decoding view 3 includes fully receiving views 1 and 2.

If views can be sent in an interleaved manner, then as shown in Figure 1 and as explained in Figures 18 and 19, the latency can be minimized.

The NAL units of different layers in the scalable extension of HEVC can be interleaved as follows:

A bitstream interleaving mechanism and decoder are used for layer or view representation. This decoder can be implemented using parallelization techniques, enabling the decoding of related views with very low latency using this bitstream layout. The interleaving of the DU is controlled by flags (e.g., `du_interleaving_enabled_flag`).

To enable low-latency decoding and parallelization in scalable extensions of HEVC, NAL units from different layers of the same AU need to be interleaved. Therefore, a definition can be introduced along the following lines:

Access Unit: A group of NAL units that are related to each other according to the prescribed classification rules, are consecutive in the decoding order, and contain exactly one encoded image;

Encoded layer image component: The encoded representation of the layer image component of all coding tree units containing the layer image component;

Encoded image: The coded representation of an image containing one or more coded layer image components, representing all coded tree units of the image;

Image: An image is a group consisting of one or more layers of image components;

Layer image components: an array of luminance samples in monochrome format or an array of luminance samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2 and 4:4:4 color formats, wherein the encoding representation consists of NAL units of a specific layer among all NAL units within the access unit.

NAL units are interleaved in this way (compare: du_interleaving_enabled_flag == 1), following the dependencies between them, so that each NAL unit can be decoded using only the data received in the previous NAL unit according to the decoding order, that is, the data of NAL units later in the decoding order do not need to be used to decode NAL units.

When applying interleaving (compare: du_interleaving_enabled_flag == 1) and separating the luminance and chrominance components into different color planes, interleaving the corresponding NAL units associated with the color planes is permitted. Each of these corresponding NAL units (associated with a unique value of colour_plane_id) must adhere to the VCL NAL unit order, as described below. Since color planes are expected to have no coded dependencies on each other within the accessed units, they follow the normal order.

The `min_spatial_segment_delay` syntax element can be used to represent constraints on the order of NAL units. This element measures and guarantees the worst-case delay/offset between spatial segments, in CTB units. The syntax element describes the dependency of spatial regions between CTBs or spatial segments of the base and enhancement layers (e.g., CTB rows of tiles, slices, or WPPs). This syntax element is not required for interleaved NAL units or for consecutively encoded NAL units in coding order. Parallel multilayer decoders can use this syntax element to establish parallel decoding of layers.

The following constraints affect the possibility of parallelization on top of layers/views and interleaved decoding units, as mainly described in the first aspect:

1) Prediction of samples and syntactic elements:

Interpolation filters for luminance and chrominance resampling set constraints on the data required in lower layers to generate the upsampled data needed for higher layers. By constraining these filters, decoding dependencies can be reduced, for example, since segments of an image can be upsampled individually. The signaling of specific constraints for tile processing was discussed above in the first aspect.

Motion vector prediction based on a reference index (HLS method) and, more specifically, temporal motion vector prediction (TMVP) sets constraints on the data required in the lower layer to generate the desired resampled image motion fields. The related inventions and signaling have been discussed above in the first aspect.

2) Motion vector:

For SHVC, motion compensation is not used by lower layers; that is, if the lower layer is used as a reference image (HLS method), then the resulting motion vectors must be zero vectors. However, for MV-HEVC 0 or 3D-HEVC 0, disparity vectors can be constraints, but do not have to be zero vectors. That is, motion vectors can be used for inter-view prediction. Therefore, constraints on motion vectors can be applied to ensure that only data received within the previous NAL unit needs to be used for decoding. The related inventions and signaling have been discussed above in the first aspect.

3) Image segmentation with tile boundaries:

If parallel processing and low latency are desired effectively, NAL units from different layers should be interleaved. In this case, image partitioning should be performed within the enhancement layer, in relation to the partitioning within the reference layer.

Regarding the order of VCL NAL units and the correlation of encoded images, the following can be specified.

Each VCL NAL unit is a part of the encoded image.

The order of VCL NAL units (i.e., VCL NAL units of encoded images with the same layer_id_nuh value) within the encoded layer image component is restricted as follows:

– The first VCL NAL unit of the coded layer picture component has a first_slice_segment_in_pic_flag equal to 1;

– Assume sliceSegAddrA and sliceSegAddrB are the slice_segment_address values of any two encoded slice NAL units A and B within the same coding layer picture component. Encoded slice NAL unit A precedes encoded slice NAL unit B if any of the following conditions are true:

–TileId[CtbAddrRsToTs[sliceSegAddrA]] is less than TileId[CtbAddrRsToTs[sliceSegAddrB]];

–TileId[CtbAddrRsToTs[sliceSegAddrA]] is equal to TileId[CtbAddrRsToTs[sliceSegAddrB]], and CtbAddrRsToTs[sliceSegAddrA] is less than CtbAddrRsToTs[sliceSegAddrB].

If the encoded image consists of image components from more than one coding layer, then the VCL NAL units of all image components are subject to the following constraints:

– Assume that within the coded layer picture component layerPicA, which serves as a reference for another layer picture component layerPicB, VCL NAL A is the first VCL NAL unit A. Then, VCL NAL unit A is located before any VCL NAL unit B belonging to layerPicB;

Otherwise (not the first VCL NAL unit), if du_interleaving_enabled_flag equals 0, then it is assumed that VCL NAL A is any VCL NAL unit of the coding layer picture component layerPicA that is used as a reference for another coding layer picture component layerPicB. Then, VCL NAL unit A is located before any VCL NAL unit B belonging to layerPicB;

Otherwise (not the first VCL NAL unit and du_interleaving_enabled_flag equals 0), if ctb_based_delay_enabled_flag equals 1 (i.e., signaling is based on CTB delay regardless of whether tiles or WPP are used within the video sequence), then assume layerPicA is a coded layer picture component used as a reference for another coded layer picture component, layerPicB. Also assume NALUsetA is a series of consecutive thin-segment NAL units belonging to layerPicA and directly following a series of consecutive thin-segment NAL units belonging to layerPicB, and NALUsetB1 and NALUsetB2 are series of consecutive thin-segment NAL units belonging to layerPicB directly following NALUsetA. Assume sliceSegAddrA is the slice_segment_address of the first NAL unit of NALUsetA, and sliceSegAddrB is the slice_segment_address of the first coded thin-segment NAL unit of NALUsetB2. Then, the following condition holds:

–If NALUsetA exists, then NALUsetB2 exists;

–CtbAddrRsToTs[PicWidthInCtbsYA*CtbRowBA(sliceSegAddrB-1)+CtbColBA(sliceSegAddrB-1)+min_spatial_segment_delay] is less than or equal to CtbAddrRsToTs[sliceSegAddrA-1]. See also Figure 23.

Otherwise (not the first VCL NAL unit and du_interleaving_enabled_flag equals 0), if tiles_enabled_flag equals 0 and entropy_coding_sync_enabled_flag equals 0 (i.e., tiles are not used in the video series, nor is WPP used), then layerPicA is a coded layer picture component that is used as a reference to another coded layer picture component, layerPicB. It is also assumed that VCL NAL unit B can be any VCL NAL unit of coded layer picture component layerPicB, and that VCL NAL unit A precedes the VCL NAL unit of layerPicA, the value of slice_segment_address is equal to sliceSegAddrA, and there are (min_spatial_segment_delay-1) VCL NAL units of layerPicA between VCL NAL unit A and VCL NAL unit B. It is also assumed that VCL NAL unit C is the next VCL NAL unit of the coded layer picture component layerPicB following VCL NAL unit B, and the value of slice_segment_address is equal to sliceSegAddrC. It is assumed that PicWidthInCtbsYA is the image width in units of layerPicA's CTB. Then, the following condition holds:

- Always has a min_spatial_segment_delayVCL NAL unit in layerPicA that precedes VCL NAL unit B;

–PicWidthInCtbsYA*CtbRowBA(sliceSegAddrC-1)+CtbColBA(sliceSegAddrC-1) is less than or equal to sliceSegAddrA-1.

– Otherwise (not the first VCL NAL unit, du_interleaving_enabled_flag equals 1, and ctb_based_delay_enabled_flag equals 0), if tiles_enabled_flag equals 0 and entropy_coding_sync_enabled_flag equals 1 (i.e., WPP is used in the video series), then assume sliceSegAddrA is the slice_segment_address of any segment NAL unit A of the coding layer picture component layerPicA directly preceding the thin segment VCL NAL unit B, and slice_segment_address is equal to sliceSegAddrB of the coding layer picture component layerPicB that uses layerPicA as a reference. Also assume PicWidthInCtbsYA is the picture width in CTB units of layerPicA. Then, the following condition holds:

–(CtbRowBA(sliceSegAddrB)–Floor((sliceSegAddrA)/PicWidthInCtbsYA)+1) is equal to or greater than min_spatial_segment_delay.

Otherwise (not the first VCL NAL unit, du_interleaving_enabled_flag equals 1, and ctb_based_delay_enabled_flag equals 0), if tiles_enabled_flag equals 1 and entropy_coding_sync_enabled_flag equals 0 (i.e., tiles are used in the video series), then assume sliceSegAddrA is the slice_segment_address of any segment NAL unit A of the coding layer picture component layerPicA, and thin segment VCL NAL unit B is the following first VCL NAL unit belonging to the coding layer picture component layerPicB that uses layerPicA as a reference, with slice_segment_address equal to sliceSegAddrB. Also assume PicWidthInCtbsYA is the picture width in CTB units of layerPicA. Then, the following condition holds:

–TileId[CtbAddrRsToTs[PicWidthInCtbsYA*CtbRowBA(sliceSegAddrB-1)+CtbColBA(sliceSegAddrB-1)]]-TileId[CtbAddrRsToTs[sliceSegAddrA-1]] should be equal to or greater than min_spatial_segment_delay.

As shown in Figure 24, signaling 724 can be configured within the VPS, where:

When `du_interleaving_enabled_flag` equals 1, it specifies that a frame has a single associated coded picture (i.e., a single associated AU), which consists of all the coded layer picture components used for that frame, and the VCL NAL units corresponding to different layers can be interleaved. When `du_interleaving_enabled_flag` equals 0, a frame can have more than one associated coded picture (i.e., one or more associated AUs), and the VCL NAL units of different coded layer picture components are not interleaved.

To complete the above discussion, in alignment with the embodiment of FIG18, the hypothetical reference decoder associated with decoder 700 is adapted to operate via one or both of buffers 702 and 704 according to the settings of signaling 724, i.e., to switch between these options according to signaling 724.

In the following description, another aspect of this application is described, again in conjunction with aspect 1 and/or aspect 2. A third aspect of this application relates to an extension of scalable signaling for use by a large number (e.g.) of views.

To facilitate understanding of the descriptions presented below, an overview of existing scalable signaling concepts is provided.

Most existing 3D video applications or deployments are characterized by stereoscopic content with or without a corresponding depth map in each of the two camera views, or by multi-view content with a higher number of views (>2) in each of the two camera views with or without a corresponding depth map.

The High Efficiency Video Coding (HEVC) standard [1] and its extensions to 3D and Multi-View Video [2][3] are characterized by scalable signaling on a Network Abstraction Layer (NAL) capable of representing up to 64 different layers, with a 6-bit layer identifier (compare: nuh_layer_id) in the header of each NAL unit, as specified in the syntax table of Figure 25.

Each value of the layer identifier can be converted into a set of scalable identifier variables (e.g., DependencyID, ViewID, etc.), for example, by extending the video parameter group. Depending on the scalability dimension in use, this allows up to 64 dedicated views to be indicated on the network abstraction layer, or 32 dedicated views if the layer identifier is also used to indicate bit depth.

However, there are also applications that require encoding, transmitting, decoding, and displaying a far greater number of views into video bitstreams, for example, in multi-camera arrays with a large number of cameras or in holographic displays requiring a large number of viewpoints, as proposed in [5][6][7]. The following sections describe two inventions that address the aforementioned shortcomings of the extended HEVC high-level syntax.

Simply extending the size nuh_layer_id field within the NAL unit header is not considered a useful solution to this problem. The header is expected to have a fixed length, which is necessary for easily accessible, very simple (low-cost) devices that perform operations on bit streams (e.g., routing and extraction). This means that if far fewer views are used, then extra bits (or bytes) are needed for all cases.

Furthermore, after the first version of the standard was completed, the NAL unit header could no longer be changed.

The following description illustrates the extension mechanisms of the HEVC decoder or intermediate device to extend the capabilities of scalable signaling to meet the above requirements. Signaling activation and data extension can be performed in the HEVC high-level syntax.

In particular, the following describes the signaling that indicates the enabling of the layer identifier extension mechanism (as described in the following sections) within the video bitstream.

In addition to the first and second aspects, a possible implementation of the third concept within the HEVC framework is first described, followed by a generalized implementation. This concept allows for multiple view components with the same existing layer identifier (compare: nuh_layer_id) within the same access unit. An additional identifier extension is used to distinguish these view components. This extension is not encoded within the NAL unit header. Therefore, it is not as easily accessible as within the NAL unit header, but new use cases with more views are still allowed. In particular, through view clustering (see description below), the legacy extraction mechanism can still be used to extract groups of views belonging to each other without any modification.

To extend the existing range of layer identifier values, the present invention describes the following mechanism:

a. Predefined values for existing layer identifiers are used as special values (so-called "escape codes") to indicate the use of alternating derivation processes to determine the actual value (in a particular implementation: using the value of the syntax element nuh_layer_id within the NAL unit header (the maximum value of the layer identifier)).

value));

b. Flags or indexes or bit depth indicators with higher-level syntactic structures (e.g., in sheet header syntax or in video/sequence/picture parameter group extensions, as specified in the following embodiments of the invention) are capable of combining each value of an existing layer identifier value with another syntactic structure.

The activation of the existing mechanism can be achieved as follows.

For a) where no explicit activation signaling is required, i.e., the reserved escape code can always be used for signaling extensions (a ₁ ). However, this reduces the number of possible layers/views that do not use the extension by 1 (the value of the escape code). Therefore, the following switching parameters can be used for both variants (a ₂ ).

The extension mechanism can be enabled or disabled within a bitstream by using one or more syntax elements that persist over the entire bitstream, video sequence, or a portion of a video sequence.

A specific implementation of the present invention, using the variable LayerId representing the existing layer identifier, to enable the extension mechanism is as follows:

Variant I) is shown in Figure 26. Here,

layer_id_ext_flag can use additional LayerId values.

Variant II is shown in Figure 27. Here,

A layer_id_mode_idc value of 1 indicates that the escape code within nuh_layer_id is used to expand the value range of LayerId. A layer_id_mode_idc value of 2 indicates that the value range of LayerId is expanded by the offset value. A layer_id_mode_idc value of 0 indicates that the expansion mechanism is not used for LayerId.

Note: Values may have different assignment patterns.

Variant III is shown in Figure 28. Here,

layer_id_ext_len represents the number of bits used to extend the range of LayerId.

The above syntax source is used as an indicator, using a layer identifier extension mechanism to indicate the layer identifier of the corresponding NAL cell or sheet data.

In the following description, the variable LayerIdExtEnabled is used as a Boolean indicator to enable the extension mechanism. In this specification, variables are used for ease of reference. Instances and implementations of the variable names in this invention may use different names or directly use the corresponding syntax elements. Based on the above, the variable LayerIdExtEnabled is obtained as follows:

For _a1 ), if the predetermined value of the layer identifier source element is used only to enable the layer identifier extension mechanism, then the following applies:

if(nuh_layer_id＝＝predetermined value)

LayerIdExtEnabled = true

else

LayerIdExtEnabled = false

For _a ) and b), if variant I) (i.e., the flag (e.g., layer_id_ext_enable_flag)) is used to enable the layer identifier extension mechanism, then the following applies:

LayerIdExtEnabled=layer_id_ext_enable_flag.

For _a ) and b), if variant II) (i.e., the index (e.g., layer_id_mode_idc)) is used to enable the layer identifier extension mechanism, then the following applies:

if(layer_id_mode_idc==predetermined value)

LayerIdExtEnabled = true

else

LayerIdExtEnabled = false

For _a ) and b), if variant III (i.e., the bit length indicator (e.g., layer_id_ext_len)) is used to enable the layer identifier extension mechanism, then the following applies:

if(layer_id_ext_len>0)

LayerIdExtEnabled = true

else

LayerIdExtEnabled = false

For a ₂ ), if the predefined value is used in conjunction with an enabled syntax element, then the following applies:

LayerIdExtEnabled&=(nuh_layer_id==predetermined value).

Layer identifier extension can be signaled as follows:

If an extension mechanism is enabled (e.g., via signaling, as described in the previous section), then a predefined number of bits (compare: layer_id_ext_len) are used to determine the actual LayerId value. For VCL NAL units, the extra bits may be included within the slice header syntax (e.g., using existing extensions) or within the SEI message, either by position within the video bitstream or by an index associated with the corresponding slice data, which is used to extend the signaling layer of the layer identifier within the NAL unit header.

For non-VCL NAL units (VPS, SPS, PPS, SEI messages), additional identifiers can be added to specific extensions or also added by the associated SEI message.

In further description, a specific syntax element is referred to as `layer_id_ext`, regardless of its position within the bitstream syntax. The name is used as an instance. The following syntax table and semantics provide examples of possible implementations.

Figure 29 illustrates the signaling of layer identifier extension within the sheet header.

Figure 30 shows the signaling for the replacement of the layer identifier extension within the sheet header.

Figure 31 shows an example of signaling used for Video Parameter Group (VPS).

SPS, PPS, and SEI messages have similar extensions. Additional syntax elements can be added to these extensions in a similar manner.

Figure 32 shows the signaling layer identifier within the associated SEI message (e.g., the Layer ID Extended SEI message).

The range of an SEI message can be determined based on its position within the bit stream. In a particular embodiment of the invention, all NAL units are associated with the value of layer_id_ext after the Layer ID Extended SEI message until the reception of a new access unit or the start of a new Layer ID Extended SEI message.

Depending on its position, additional syntax elements can be encoded using fixed (represented here as u(v)) or variable (ue(v)) length codes.

Then, based on the activation of the layer identifier extension mechanism (compare: LayerIdExtEnabled), the layer identifier for a specific NAL cell and/or sheet data is obtained by mathematically combining the layer identifier in the NAL cell header (compare: nuh_layer_id) and the information provided by the layer identifier extension mechanism (compare: layer_id_ext).

A specific implementation obtains the layer identifier, referred to herein as LayerId, by using the existing layer identifier (compare: nuh_layer_id) as the most significant bit and the extended information as the least significant bit:

In case b) nuh_layer_id can represent different values, this signaling scheme allows signaling more different LayerId values through a small range of layer_id_ext values. It also allows specific view clustering, that is, views that are close to each other can use the same nuh_layer_id value to indicate that they belong to each other, as shown in Figure 33.

Figure 33 illustrates the construction of a view cluster, where all NAL units associated with the cluster (i.e., a group of views from physically close cameras) have the same value nuh_layer_id and an equal value layer_id_ext. Alternatively, the syntax element layer_id_ext can be used in another embodiment of the invention to accordingly construct the cluster, and nuh_layer_id can be used to identify views within the cluster.

Another embodiment of the invention obtains the layer identifier, referred to herein as LayerId, by using the existing layer identifier (compare: nuh_layer_id) as the least significant bit and the extended information as the most significant bit:

This signaling scheme allows signaling through the aggregation of specific views, that is, views of cameras that are physically far apart from each other can use the same value nuh_layer_id to indicate that they use the same predictive dependency on the camera view, and have the same value nuh_layer_id (i.e., layer_id_ext in this implementation) within different clusters.

Another implementation uses an additive scheme to extend the range of LayerId (maxNuhLayerId represents the maximum permissible range of layer identifiers (compare: nuh_layer_id)):

This signaling scheme is particularly useful when a) a predefined value of nuh_layer_id is used to enable expansion. For example, the value of maxNuhLayerId can be used as a predefined escape code to allow gapless expansion of the LayerId value range.

In the context of the draft test model of HEVC’s 3D video coding extension as an earlier described draft version [3], a possible implementation is described in the following paragraphs.

In part of the earlier version [3] G.3.5, view components were defined as follows:

View component: The coded representation of a view within a single access unit A view component can include a depth view component and a texture view component.

Based on existing layer identifiers (compare: nuh_layer_id), the mapping of depth and texture view components is limited in the VPS extended syntax. This invention increases the flexibility of mapping an additional range of layer identifier values. An exemplary syntax is shown in Figures 34A and 34B. Shading is used to emphasize the changes to the existing syntax.

If layer identifier extension is used, then VpsMaxLayerId is set to equal vps_max_layer_id; otherwise, it is set to equal vps_max_ext_layer_id.

If layer identifier extension is used, then VpsMaxNumLayers is set to the maximum number of layers that can be encoded using the extension (by a predefined number of bits or according to layer_id_ext_len); otherwise, VpsMaxNumLayers is set to equal vps_max_layers_minus1+1.

vps_max_ext_layer_id is the maximum LayerId value that can be used.

layer_id_in_nalu[i] specifies the value of the LayerId associated with the VCL NAL cell of the i-th layer. For i in the range from 0 to VpsMaxNumLayers–1 (inclusive), if i does not exist, the value of layer_id_in_nalu[i] is inferred to be equal to i.

When i is greater than 0, layer_id_in_nalu[i] is greater than layer_id_in_nalu[i–1].

When splitting_flag equals 1, if the total number of bits in the segment is less than 6, then the MSB of layer_id_in_nuh should be 0.

For i in the range from 0 to vps_max_layers_minus1 (inclusive), the variable LayerIdInVps[layer_id_in_nalu[i]] is set to equal i.

`dimension_id[i][j]` is the identifier of the j-th existing scalability dimension type in the i-th layer. If it does not exist, the value of `dimension_id[i][j]` is inferred to be 0. The number of bits used to represent `dimension_id[i][j]` is `dimension_id_len_minus1[j]+1` bits. When `splitting_flag` is equal to 1, bitstream consistency requires `dimension_id[i][j]` to be equal to `((layer_id_in_nalu[i]&((1<<dimBitOffset[j+1])-1))>>dimBitOffset[j])`.

The following methods are used to obtain the value of the identifier for the smIdx-th scalability dimension type of the i-th layer, ScalabilityId[i][smIdx], the variable ViewId[layer_id_in_nuh[i]] which specifies the view identifier of the i-th layer, and DependencyId[layer_id_in_nalu[i]] which specifies the spatial/SNR scalability identifier of the i-th layer:

In section 2 of the earlier version [3], it is described that the corresponding depth map and texture component of a particular camera can be distinguished from other depth maps and textures by means of its extensibility identifier view order index (compare: ViewIdx) and depth flag (compare: DepthFlag) obtained in the partial NAL unit header semantics of the earlier version [3]:

ViewIdx = layer_id >> 1

DepthFlag = layer_id%2

Therefore, individual view components (i.e., texture and depth map components of a specific camera) must be encapsulated into NAL units with a distinguishable individual value layer_id, i.e., through the variable ViewIdx value, in the early version 0 of G.8, during the decoding process.

The above concepts allow the same value for the layer identifier within the NAL unit header (compare: nuh_layer_id) to be used for different views. Therefore, the derivation of the identifiers ViewIdx and DepthFlag requires the following approach suitable for using the extended view identifiers obtained earlier:

ViewIdx = LayerId >> 1

DepthFlag = LayerId%2

A general implementation of the third aspect is described below with reference to FIG35. This figure shows a decoder 800 configured to decode a multi-layer video signal. The decoder can be represented as described above in FIG2, 9, or 18. That is, examples of a more detailed explanation of the decoder 800 of FIG35 can be obtained using the above aspects and their embodiments, according to a particular embodiment. To illustrate this possible overlap between the above aspects and their embodiments and the embodiment of FIG35, for example, the same reference numerals are used in FIG35 for the multi-layer video signal 40. Regarding what the multi-layers of the multi-layer video signal 40 can be, refer to the statements presented above in the second aspect.

As shown in Figure 35, the multi-layer video signal consists of a series of data packets 804, each data packet including a layer identification syntax element 806, embodied using the nuh_layer_id syntax element in the specific HEVC extension instance described above. The decoder 800 is configured to respond to layer identification extension mechanism signaling within the multi-layer video signal 40, as further described below, whereby the multi-layer video signal may partially involve the layer identification syntax element itself. Layer identification extension mechanism signaling 808 is sensed by the decoder 800, which responds to signal 808 as follows for predetermined data packets within data packets 804, such predetermined data packets being indicated by arrow 810 as entering the decoder 800. As described using switch 812 of the decoder 800, controlled by layer identification extension mechanism signaling 808, for a predetermined data packet 810, signal 808 reads 814 the layer identification extension from the multi-layer data stream 40 and uses this layer identification extension to determine 816 the layer identification index of the current data packet 810. If signaling 808 activates the layer identification extension mechanism, then the layer identification extension read in 814 can be included by the current data packet 810 itself, as shown in 818, or it can be placed elsewhere within data stream 40, but in a manner associatable with the current data packet 810. Therefore, if the layer identification extension mechanism signaling 808 activates the layer identification extension mechanism, then decoder 800 determines the layer identification index of the current data packet 810 according to 814 and 816. However, if the layer identification extension mechanism signaling 808 deactivates the layer identification extension mechanism, then decoder 800 determines the layer identification index of the predetermined data packet 810 simply from the layer identification syntax element 806 of the current data packet 810. In this case, the layer identification extension 818 is not needed (i.e., its presence within signal 40), that is, it does not exist.

According to one implementation, in the sense of each data packet, the layer identification syntax element 806 at least contributes to the layer identification extension mechanism signaling 808: for each data packet (e.g., the current data packet 810), the decoder 800 determines, at least in part, whether the layer identification extension mechanism signaling 808 is activated or deactivated based on the layer identification syntax element 806 of the corresponding data packet 810, which adopts or does not adopt the escape value. For example, the high-level syntax element 822 included by the data stream 40 within a certain parameter group 824 can contribute to the layer identification extension mechanism signaling 808, i.e., whether the layer identification extension mechanism signaling is activated or deactivated, quite macroscopically or on a higher level. In particular, the decoder 800 can be configured to determine whether the layer identification extension mechanism signaling 808 is activated or deactivated for a predetermined data packet 810 primarily based on the high-level syntax element 822: if the high-level syntax element adopts a first state, then the signaling deactivates the layer identification extension mechanism. Whether this is possible, as described above, depends on whether layer_id_ext_flag = 0, layer_id_mode_idc = 0, or layer_id_ext_len = 0. In other words, in the specific syntax examples above, layer_id_ext_flag, layer_id_ext_idc, and layer_id_ext_len represent instances of high-level syntax element 822.

Regarding a given data packet (e.g., packet 810), this means that if the high-level syntax element adopts a state different from the first state and the layer identification syntax element 806 of packet 810 adopts an escape value, then the decoder 800 determines that the layer identification extension mechanism signaling 808 signals to activate the layer identification extension mechanism of packet 810. However, if the high-level syntax element 822 of packet 810 effectively adopts the first state or the layer identification syntax element 806 of packet 810 adopts a value different from the escape value, then the decoder 800 determines that the signaling 808 does not signal to deactivate the layer identification extension mechanism.

Instead of just two possible states, as described in the syntax examples above, the high-level syntax element 822 can include more than one further state that the high-level syntax element 824 may take, in addition to the deactivation state. Depending on these possible further states, determination 816 can vary, as shown by the dashed line 824. For example, in the syntax examples above, the case where layer_id_mode_idc = 2 indicates that determination 816 may cause decoder 800 to concatenate the numbers representing the layer identification syntax element 806 of packet 810 with the numbers representing the layer identification extension to obtain the layer identification index of packet 810. In contrast, the case where layer_id_len ≠ 0 indicates that determination 816 may cause decoder 800 to perform the following: decoder 800 uses the high-level syntax element to determine the length n of the high-level identification extension 818 associated with packet 810, and concatenates the numbers representing the layer identification syntax element 806 of packet 810 with the n numbers representing the layer identification extension 818 of packet 810 to obtain the level identification index of the predetermined packet. Even further, determining 816 may involve adding the level identification extension 818 associated with data packet 810 to a predetermined value, for example, the predetermined value may correspond to the number of maximum representable states exceeding the level identification syntax element 806 (less than the escape value), in order to obtain the level identification index of the predetermined data packet 810.

As described in Figures 34A and 34B using 808', it is also possible to exclude the layer identification syntax element 806 of data packet 810 from contributing to the layer identification extension mechanism signaling 808, so that the entire representable value/state of syntax element 806 remains, and none are retained as escape codes. In this case, signal 808' indicates to decoder 800 whether a layer identification extension 818 exists for each data packet 810, thus instructing layer identification index determination to follow 814 and 816 or 820.

Therefore, the encoder working in conjunction with the decoder in Figure 35 simply forms a data stream. The encoder determines whether to use an extension mechanism based on, for example, the number of layers encoded within the data stream.

The fourth aspect of this application relates to dimension-dependent direct dependency signaling.

In the current HEVC extensions ([2], [3], [4]), a coding layer can use zero or more reference coding layers for predicting data. Each coding layer is identified by a unique nuh_layer_id value, which can be bijectively mapped to a layerIdInVps value. The layerIdInVps values are consecutive, and the bitstream consistency requirement is that A is less than B when a layer with a layerIdInVps equal to A is referenced by a layer with a layerIdInVps.

For each coding layer within the bitstream, a signaling reference coding layer is established within the video parameter group. Therefore, a binary mask is transmitted for each coding layer. For a coding layer with a layerIdinVps value of b, the mask (denoted as direct_dependency_flag[b]) consists of b-1 bits. When a layer with layerIdinVps equal to x is the reference layer for a layer with layerIdinVps equal to b, the x-th bit in the binary mask (denoted as direct_dependency_flag[b][x]) is equal to 1. Otherwise, when a layer with layerIdinVps equal to x is not the reference layer for a layer with layerIdinVps equal to b, the value of direct_dependency_flag[b][x] is equal to 0.

When resolving all direct_dependency_flags, for each coding layer, a list is created that includes the nuh_layer_id values of all reference layers, as specified by direct_dependency_flags.

Furthermore, the signaling information within the VPS allows each layerIdinVps value to be mapped to a location within a T-dimensional scalability space. Each dimension t represents the type of scalability, which can be, for example, an indication of view scalability, spatial scalability, or a depth map.

By signaling one bit for each possible dependency, the current design offers maximum flexibility. However, this flexibility has some drawbacks:

1. For each scalability dimension, a common use case is to use a specific dependency structure.

Moreover, direct interdimensional dependencies are uncommon and may be prohibited.

Figure 36 illustrates an example of a common layer setup. Here, dimension 0 might be a view scalability dimension, utilizing a hierarchical prediction structure. Dimension 1 might be a spatial scalability dimension using an IP structure. Figure 37 shows the direct_dependency_flags involved in the described setup.

The drawback of the current solution is that it does not directly identify this dimension-dependent dependency from the current VPS design, because this requires arithmetically complex analysis of direct_dependency_flags.

2. Even when using only one scalable dimension type, the same structure is often used for a subset of layers. For cases involving only view scalability, such as those involving only view scalability, the view might be mapped to a space spanned by the horizontal and vertical camera positions.

Figure 36 illustrates an example of this scenario, where dimensions 0 and 1 are interpreted as the horizontal and vertical camera position dimensions.

While the conventional practice is to use a single prediction structure for each camera location dimension, the current VPS design cannot take advantage of the resulting redundancy. Furthermore, the current VPS design lacks a direct indication of dimension dependence due to the absence of dependencies.

3. The number of `direct_dependency_flags` is proportional to the square of the number of layers in the bitstream. Therefore, in the worst-case scenario, with 64 layers, approximately 64 * 63 / 2 = 2016 bits are required. Moreover, this results in a significantly increased number of bits when expanding to the maximum number of layers in the bitstream.

Since the dependency of each dimension t in the T-dimensional dependency space of the signaling can be clearly defined, the above-mentioned shortcomings can be solved.

Dimension-dependent direct dependency signaling offers the following advantages:

1. The dependencies for each dependency dimension are directly available within the bitstream and do not require complex analysis of direct_dependency_flags;

2. The number of bits required for signaling dependencies can be reduced.

In one implementation, the dependency space may be (for example) the same as the scalability space described in the current MV and scalability draft [2]. In another implementation, the dependency space may be explicit signaling and may also be (for example) the space traversed by the camera location.

Figure 38 provides an example of dimension-dependent dependency signaling. It can be seen that the dependencies between dimensions can be directly obtained from the binary mask with a reduced number of bits required.

In the following text, it is assumed that each layerIdInVps value is bijectively mapped to a T-dimensional dependency space, with dimensions ₀ , ₁ , ₂ , ..., (T- ₁ ). Therefore, each layer is positioned within the corresponding dimensions ₀ , 1, ₂ , ..., ( _T-1 ) according to the associated vector (d0, _d1 , d2, ..., dT-1)′.

The basic concept is layer-dependent dimension-related signaling. Therefore, for each dimension t ∈ {0, 1, 2 ... (T-1)} and each position d_{_t} within dimension t, a set of reference positions Ref(d_{_t} ) is signaled. This set of reference positions is used to determine the direct dependencies across different layers to the present year, as described below:

When _Ref is an element within Ref(d _t ), the layer with position d _t in dimension t and position d _x in dimension x (x∈{0,1,2 ... (T-1)}\{t}) depends on the layer with position d _{t in dimension t, Ref} and position d _x in dimension x (x∈{0,1,2 ... (T-1)}\{t}).

In another particular implementation, all dependencies are reversed, so the position in Ref(d _t ) represents the position of the layer in dimension _t that depends on the position d t in dimension t.

Regarding signaling and derivation in the dependency space, the signaling described below can be performed, for example, in the VPS, SPS within the SEI message, or elsewhere within the bit stream.

Regarding the number of dimensions and the number of positions within each dimension, note the following: The dependency space is constrained by a specific number of dimensions and a specific number of positions within each dimension.

In one particular implementation, for example as shown in Figure 39, the number of signaling dimensions num_dims and the number of positions num_pos_minus1[t] within dimension t can be explicitly defined.

In another implementation, the value num_dims or the value um_pos_minus1 can be fixed and not signaled within the bit stream.

In another implementation, the value num_dims or the value um_pos_minus1 can be obtained from other syntax elements existing within the bitstream. More specifically, in the current HEVC extended design, the number of dimensions and the number of positions within a dimension can be equal to the number of scalable dimensions and the length of the scalable dimension, respectively.

Therefore, NumScalabilityTypes and dimension_id_len_minus1[t] are as defined in [2]:

num_dims=NumScalabilityTypes

num_pos_minus1[t]=dimension_id_len_minus1[t]

In another implementation, the signaling value num_dims or the value um_pos_minus1 within the bit stream can be used to explicitly indicate whether the signaling is obtained from other syntax elements existing within the bit stream.

In another implementation, the value num_dims can be obtained from other syntax elements existing within the bitstream, and then increased by additional signaling separated by one or more dimensions or by additional signaling dimensions.

Regarding the mapping of layerIdInVps to its location in the dependency space, it's important to note that layers are mapped to the dependency space.

In one particular implementation, for example, the syntax element pos_in_dim[i][t] can be used to explicitly transmit the location of the specified layer, having a layerIdinVps value i within dimension t. This is shown in Figure 40.

In another implementation, the signaling value pos_in_dim[i][t] is not obtained within the bitstream, but directly from the layerIdInVps value i, for example,

Specifically, for the current HEVC extended design, the above may replace the current explicit signaling of the dimension_id[i][t] value.

In another implementation, the value pos_in_dim[i][t] is obtained from other syntax elements within the bitstream. Specifically, in the current HEVC extension design, for example, the value pos_in_dim[i][t] can be obtained from the value of dimension_id[i][t].

pos_in_dim[i][t]=dimension_id[i][t]

In another implementation, the signaling pos_in_dim[i][t] can be explicit or obtained from other syntax elements.

In another implementation, in addition to obtaining the pos_in_dim[i][t] value from other syntax elements existing within the bitstream, the pos_in_dim[i][t] value can also be signaled as to whether it is explicitly signaled.

For signaling and derivation regarding dependencies, use the following.

The use of direct position dependency flags is the subject of the following implementation. In this implementation, the reference position is signaled by, for example, the flag pos_dependency_flag[t][m][n], indicating whether the position in dimension t is included in the reference position group of position m in dimension t, for example, as specified in Figure 41.

In one implementation using a set of reference positions, a variable num_ref_pos[t][m] specifying the number of reference positions in dimension t for position m in dimension t, and a variable ref_pos_set[t][m][j] specifying the j-th reference position in dimension t for position m in dimension t, can then be obtained, for example.

In another implementation, the elements at the set reference positions can be directly signaled, for example, as specified in Figure 42.

In implementations using the direct dependency flag, the direct dependency flag directDependencyFlag[i][j] might be obtained from the reference location group, specifying that layerIdInVps equal to i depends on a layer having layerIdInVps equal to j. For example, it might be done as specified below.

The function posVecToPosIdx(posVector) takes a vector posVector as input and obtains the index posIdx associated with the position posVector in the dependency space, as specified below:

For example, a variable posIdxToLayerIdInVps[idx] can be obtained, which depends on the layerIdinVps value i of the index idx obtained from pos_in_dim[i], as specified below:

for(i=0;i<vps_max_layers_minus1;i++)

posldxToLayerldnVps[posVecToPosldx(pos_in_dim[i])]=i

Obtain the variable directDependencyFlag[i][j] as specified below:

In one implementation, the direct dependency flag `directDependencyFlag[i][j]` may be obtained directly from the `pos_dependency_flag[t][m][n]` flag, which specifies that a layer with `layerIdInVps` equal to `i` depends on a layer with `layerIdInVps` equal to `j`. For example, as specified below:

In one implementation, using a reference layer group, it is possible to obtain a variable NumDirectRefLayers[i] specifying the number of reference layers for layers having a layerIdInVps equal to i, and a variable RefLayerId[i][k] specifying the value of the k-th reference layer, layerIdInVps, for example, as specified below:

In another implementation, the reference layer can be obtained directly from the reference location group without obtaining the directDependencyFlag value, for example, as specified below:

In another implementation, the reference layer may be obtained directly from the pos_dependency_flag variable instead of the ref_pos_set variable.

Therefore, the view discussed above shows the data stream according to the fourth aspect and shows a multi-layer video data stream, which uses inter-layer prediction to encode video data into this multi-layer video data stream with different levels of information content (i.e., the quantity is LayerIdInVps). The levels have a defined order therein. For example, the levels follow sequence 1…vps_max_layers_minus1. For example, refer to Figure 40. Here, the number of layers in the multi-layer video data stream is provided by vps_max_layers_minus1 in 900.

Video data is encoded into multi-layer video data streams such that no layer depends on any subsequent layer in the order of their position in the sequence of events, through inter-layer prediction. That is, using numbers from 1 to vps_max_layers_minus1, layer i depends only on layer j < i.

Inter-layer prediction increases the amount of information per layer, which depends on one or more other layers, where video data is encoded into said one or more other layers. This increase is related to spatial resolution, the number of views, SNR accuracy, or other dimensions.

For example, a multi-tiered video data stream includes a first syntax structure via a VPS tier. In the example above, num_dims can be included by the first syntax structure, as shown at 902 in Figure 39. Therefore, the first syntax structure defines the number M of dependency dimensions 904 and 906. In Figure 36, an example is 2, one horizontally guided and the other vertically guided. In this respect, referring to entry 2 above: the number of dimensions does not necessarily equal the number of different dimension types; in this respect, the tier increases the amount of information. For example, the number of dimensions can be higher, for example, distinguishing between vertical and horizontal view displacements. Figure 36 exemplarily shows M dependency dimensions 904 and 906, spanning dependency space 908.

Furthermore, the first syntactic structure defines a maximum N _{_i} of sorted levels for each dependency dimension i, for example, num_pos_minus1, thereby defining 910 available points within the dependency space 908. In the case of Figure 36, there are 4 x 2 available points 910, which are represented by rectangles in Figure 36. Furthermore, the first syntactic structure defines a bijective mapping 912 (refer to Figure 40), which in the above example is defined by pos_in_dim[i][t] or implicitly. The bijective mapping 40 maps each level up to a corresponding one of the available points 910 in at least a subset within the dependency space 908, i.e., i in Figure 40. pos_in_dim[i][t] is a vector pointing to available points 910 for level i by its component pos_in_dim[i][t], t scanning dimensions 904 and 906. For example, a subset is appropriate if vps_max_layers_minus1 is less than 1. For example, in Figure 36, the hierarchy that is actually used and has the dependency order defined therein can be mapped on fewer than 8 available points.

For each dependency dimension i, for example, by VPS tier, the multi-tier video data stream includes a second syntax structure 914. In the example above, it includes either pos_dependency_flag[t][m][n] or num_ref_pos[t][m] plus ref_pos_set[t][m][j]. For each dependency dimension i, the second syntax structure 914 describes the dependencies within the N _i sorted tiers of dependency dimension i. In the middle 36, dependencies are represented by all horizontal or all vertical arrows between rectangles 910.

In summary, this measure constrains dependencies between available points in the dependency space in a restrictive manner, such that all these dependencies operate in parallel with a corresponding dependency axis, pointing from higher ordinal to lower ordinal levels. For each dependency dimension, dependencies parallel to the corresponding dependency dimension remain unchanged, resisting cyclic shifts along every dependency dimension except the corresponding one. Referring to Figure 36: all horizontal arrows between rectangles in the upward rectangle are repeated in the downward rectangle, and this also applies to the vertical arrows relative to the four vertical columns of the rectangle, where the rectangle corresponds to a variable point and the arrow corresponds to a dependency within it. Through this measure, the second syntactic structure simultaneously constrains dependencies between layers via a bijective mapping.

Network entities (e.g., decoders) or manes (e.g., MMEs) can read out the first and second syntactic structures of the data stream and determine the dependencies between the layers based on the first and second syntactic structures.

The network entity reads the first syntactic structure and obtains from it the number M of dependency dimensions spanning the dependency space and the maximum N_{_i} of the ranking order of each dependency dimension i, thereby obtaining a number of available points within the dependency space. Further, the network entity obtains a bijective mapping from the first syntactic structure. Further, for dependency dimension i, the network entity reads the second syntactic structure and thereby obtains the dependencies within the N_{_i} ranking order of dependency dimension i. Whenever it decides to remove any layer, i.e., a NAL unit belonging to a certain layer, the network entity considers the position of the layer within the dependency space and the dependencies between the available points and the layer, respectively.

In doing so, network entities can select a level; and discard packets of the multi-layer video data stream (e.g., NAL units) that belong (e.g., by nuh_layer_id) to a layer whose dependency between the layers is independent of the selected level.

Although some aspects are described in the context of the device, it is clear that these aspects also represent a description of the corresponding method, wherein blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of corresponding blocks, articles, or features of the corresponding device. Some or all of the method steps may be performed by (or using) hardware devices, such as microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important method steps may be performed by such a device.

Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. Digital memory media can be used to execute the embodiments, such as floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, or FLASH memories, on which electronically readable control signals are stored. These signals cooperate with (or are capable of cooperating with) a programmable computer system to execute corresponding methods. Therefore, the digital memory medium can be computer-readable.

Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

Typically, embodiments of the present invention can be implemented as a computer program product having program code, which, when run on a computer, is effectively used to execute a method. For example, the program code may also be stored on a machine-readable carrier wave.

Other implementations include a computer program for performing one of the methods described herein, the computer program being stored on a machine-readable carrier.

In other words, therefore, one embodiment of the method of the present invention is a computer program having program code, which, when run on a computer, is used to perform a method described herein.

Therefore, a further embodiment of the method of the present invention is a data carrier (or digital storage medium or computer-readable medium) on which a computer program for performing one of the methods described herein is recorded. The data carrier, digital storage medium, or recording medium is typically tangible and/or non-transparent.

Therefore, a further embodiment of the method of the present invention is a data stream or a series of signals representing a computer program for performing a method described herein. For example, the data stream or such series of signals can be configured for transmission via a data communication connection, such as the Internet.

Further embodiments include processing means, such as a computer or programmable logic means, configured or adapted to perform one of the methods described herein.

A further embodiment includes a computer on which a computer program is installed for performing one of the methods described herein.

Further embodiments of the invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing a method described herein to a receiver. For example, the receiver may be a computer, a mobile device, a memory device, etc. For example, the apparatus or system may include a file server for transmitting the computer program to the receiver.

In some embodiments, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

The devices described herein can be implemented using hardware devices, or using computers, or a combination of hardware devices and computers.

The methods described herein can be performed using hardware devices, or using a computer, or a combination of hardware devices and a computer.

The above embodiments merely illustrate the principles of the present invention. It should be understood that modifications and variations of the settings and details described herein will be readily apparent to those skilled in the art. Therefore, the aim is that the scope of the embodiments described and explained herein be limited only by the scope of the impending patent claims, and not by the specific details presented.

Therefore, the above instruction manual specifically describes the following implementation methods:

1. A multi-view decoder configured to reconstruct a plurality of views (12, 15) from a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the multi-view decoder is configured to change the inter-view predictions at spatial segment boundaries (300) of spatial segments (301) into which the first view (12) is divided in response to signaling in the data stream.

2. The multi-view decoder according to embodiment 1, wherein the multi-view decoder is configured to perform a domain restriction on the possible disparity vectors of the trust order in the data stream during the change of the inter-view prediction.

3. The multi-view decoder according to embodiment 1 or 2, wherein the multi-view decoder is configured to determine, based on the data stream, a disparity vector (308) in the domain of possible disparity vectors for the current portion (302) of the second view (15), and sample the first view (12) at a reference portion (304) of the disparity vector (308) determined by offset from the colocation of the first view 12 to the colocation portion (306) of the current portion (302).

4. The multi-view decoder according to embodiment 3, wherein the multi-view decoder is configured to, in changing the inter-view prediction, perform a domain restriction of the possible disparity vectors of the trust order in the data stream, and perform the domain restriction of the possible disparity vectors such that the reference portion (304) is located within the spatial segment (301) in which the co-location portion (306) is located in space.

5. The multi-view decoder according to embodiment 3, wherein the multi-view decoder is configured to, in changing the inter-view prediction, perform a domain restriction of the possible disparity vectors of the trusted order in the data stream, and perform the domain restriction of the possible disparity vectors such that the reference portion (304) is located within the spatial segment in which the co-location portion (306) is located, and when the components of the disparity vectors in the dimension pointing to the boundary (300) have sub-pixel resolution, the reference portion (304) is spaced from the boundary of the spatial segment by a distance greater than or equal to the half-width (310) of the interpolation filter kernel.

6. The multi-view decoder according to any one of the foregoing embodiments, wherein the multi-view decoder is configured to, in changing the inter-view prediction, use alternative data that is independent of the boundary outside the spatial segment to fill the interpolation filter kernel (311) at the portion extending beyond the boundary (300) of the spatial segment, where the co-localization of the first view to the co-localization portion (306) of the current portion (302) of the second view (15) currently being predicted using the inter-view prediction is spatially located in the spatial segment.

7. A multi-view decoder according to any one of the foregoing embodiments, wherein the multi-view decoder is configured to, in the inter-view prediction, derive a reference portion (314) within the first view (12) for the current portion of the second view and according to the signaling in the data stream,

Check whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) of the first view (12) is spatially located to the current portion (302), and apply the predictor to the current portion (302) derived from the attributes of the reference portion (314), or suppress the application of the parameters of the current portion (302) or apply an alternative predictor to the parameters of the current portion (302) depending on whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) is spatially located.

The predictor is applied regardless of whether the reference portion (314) is located within the spatial segment (82) in which the co-location portion is located.

8. The multi-view decoder according to embodiment 7, wherein the multi-view decoder is configured to derive the reference portion (314),

Estimate the disparity vector (316) for the current part (302).

The common location of the first view to the representative position (318) of the current part (302) or the adjacent part (320) of the first view adjacent to the current part (302) is determined, and

The reference portion (314) is determined by applying the disparity vector (316) to the representative position (318).

9. The multi-view decoder according to embodiment 8, wherein the multi-view decoder is configured to estimate a disparity vector for the current portion or a spatially or temporally predicted disparity vector for the current portion based on a depth map transmitted in the data stream.

10. The multi-view decoder according to embodiment 8 or 9, wherein the multi-view decoder is configured to select the reference portion (314) in determining the reference portion (314) by using the disparity vector (316) in the partitioning of the first view (12) into a coding block, a prediction block, a residual block, and/or a transform block.

11. The multi-view decoder according to any one of embodiments 7 to 10, wherein the parameters are motion vectors, disparity vectors, residual signals, and/or depth values.

12. The multi-view decoder according to any one of embodiments 7 to 11, wherein the attribute is a motion vector, a disparity vector, a residual signal, and/or a depth value.

13. A multi-view encoder configured to encode a plurality of views (12, 15) into a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the multi-view encoder is configured to modify the inter-view predictions at the spatial segment boundaries (300) of spatial segments (301) into which the first view (12) is divided.

14. The multi-view encoder according to embodiment 13, wherein the multi-view encoder is configured to perform domain restriction of possible disparity vectors in changing the inter-view prediction.

15. The multi-view encoder according to embodiment 13 or 14, wherein the multi-view encoder is configured to determine (e.g., by optimization) a disparity vector (308) in the domain of possible disparity vectors for the current portion (302) (e.g., a prediction block predicted with disparity compensation) of the second view (15) and transmit it as a signal in the data stream, and sample the first view (12) at a reference portion (304) of the determined disparity vector (308) offset from the colocation of the first view 12 to the colocation portion (306) of the current portion (302).

16. The multi-view encoder according to embodiment 15, wherein the multi-view encoder is configured to perform a domain restriction of the possible disparity vector such that the reference portion (304) is located (e.g., completely) within the spatial segment (301) in which the co-location portion (306) is located in space.

17. The multi-view encoder according to embodiment 15, wherein the multi-view encoder is configured to perform a domain restriction of the possible disparity vector such that the reference portion (304) is located within the spatial segment in which the co-location portion (306) is located, and the reference portion (304) is spaced from the boundary of the spatial segment by a distance greater than or equal to the half-width (310) of the interpolation filter kernel when the components of the disparity vector in the dimension pointing to the boundary (300) have sub-pixel resolution.

18. A multi-view encoder according to any one of embodiments 13 to 17, wherein the multi-view encoder is configured to fill an interpolation filter kernel (311) at a portion extending beyond the boundary (300) of a spatial segment in changing the inter-view prediction, wherein the co-location of the first view to the co-location portion (306) of the current portion (302) of the second view (15) currently being predicted using the inter-view prediction is spatially located in the spatial segment.

19. A multi-view encoder according to any one of embodiments 13 to 18, wherein the multi-view encoder is configured to, in the inter-view prediction, derive a reference portion (314) within the first view (12) for the current portion of the second view, and according to signaling in the data stream,

Check whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) of the first view (12) is spatially located, and apply the predictor to the current portion (302) derived from the attributes of the reference portion (314), or suppress the application of the parameters to the current portion (302) based on whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) is spatially located, or

The predictor is applied regardless of whether the reference portion (314) is located within the spatial segment (301) in which the co-location portion is located.

20. The multi-view encoder according to embodiment 19, wherein the multi-view encoder is configured to, in deriving the reference portion (314),

Estimate the disparity vector (316) for the current part (314),

The common location of the first view to the representative position (318) of the current part (302) or the adjacent part (320) of the first view adjacent to the current part (302) is determined, and

The reference portion (314) is determined by applying the disparity vector (316) to the representative position (318).

21. The multi-view encoder according to embodiment 20, wherein the multi-view encoder is configured to estimate a disparity vector for the current portion or a spatially or temporally predicted disparity vector for the current portion based on a depth map transmitted in the data stream.

22. The multi-view encoder according to any one of embodiments 19 to 21, wherein the parameters are motion vectors, disparity vectors, residual signals and/or depth values.

23. The multi-view encoder according to any one of embodiments 19 to 22, wherein the attribute is a motion vector, a disparity vector, a residual signal, and/or a depth value.

24. The multi-view encoder according to any one of embodiments 13 to 23 is configured to send the change as a signal to the decoder in the data stream so that the decoder relies on the change.

25. A multi-view decoder configured to reconstruct a plurality of views (12, 15) from a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the multi-view decoder is configured to use signaling in the data stream as a guarantee that the inter-view predictions (602) are constrained at the spatial segment boundaries (300) of the spatial segments (301) into which the first view (12) is divided, such that the inter-view predictions do not involve any dependency of any current portion (302) of the second view (15) on any spatial segment other than the spatial segment in which the colocation of the first view to the colocation portion (606) of the corresponding current portion of the second view is located.

26. The multi-view decoder according to embodiment 25 is configured to, in response to the signaling in the data stream, use inter-view parallelism to adjust the inter-view decoding offset or determine to perform an experiment to reconstruct the first and second views.

27. The multi-view decoder according to embodiment 25 or 26, wherein the multi-view decoder is configured to determine, based on the data stream, a disparity vector (308) in the domain of possible disparity vectors for the current portion (302) of the second view (15), and sample the first view (12) at a reference portion (304) of the disparity vector (308) determined by offset from the colocation of the first view 12 to the colocation portion (306) of the current portion (302).

28. A method for reconstructing a plurality of views (12, 15) from a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the method responds to signaling in the data stream to change the inter-view predictions at spatial segment boundaries (300) of spatial segments (301) into which the first view (12) is divided.

29. A method for encoding a plurality of views (12, 15) into a data stream using inter-view prediction from a first view (12) to a second view (15), wherein the method includes changing the spatial segment boundaries (300) of spatial segments (301) into which the first view (12) is divided.

The prediction between views at that location.

30. A method for reconstructing a plurality of views (12, 15) from a data stream using inter-view prediction from a first view (12) to a second view (15), wherein the method includes using signaling in the data stream as a guarantee that the inter-view prediction (602) is constrained at the spatial segment boundary (300) of a spatial segment (301) into which the first view (12) is divided, such that the inter-view prediction does not involve any dependency of any current portion (302) of the second view (15) on a spatial segment other than the spatial segment in which the colocation of the first view to the colocation portion (606) of the corresponding current portion of the second view is located.

31. A computer program having program code, which, when run on a computer, performs the method according to any one of embodiments 27 to 30.

32. A multi-layer video data stream (200) comprising a sequence of NAL units (202), the multi-layer video data stream (200) having images (204) of multiple layers therein encoded using inter-layer predictive coding, each NAL unit (202) having a layer index (nuh_layer_id) representing the layer associated with each NAL unit, the sequence of NAL units being constructed as a sequence of non-interleaved access units (206), wherein the NAL units belonging to an access unit are...

Each unit is associated with an image at a given moment in time, and NAL units in different access units are associated with different moments in time. Within each access unit, for each layer, NAL units associated with each layer are grouped into one or more decoding units (208), and decoding units of NAL units associated with different layers are interleaved such that for each decoding unit (208), the inter-layer prediction used to encode each of the decoding units is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within each of the access units, excluding the layer associated with the respective decoding unit.

33. The multi-layer video data stream (200) according to embodiment 32, wherein the multi-layer video data stream has interleaved signaling, the interleaved signaling having a first possible state and a second possible state, wherein,

If the interleaved signaling adopts the first possible state, then within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units, and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit, the inter-layer prediction used to encode each decoding unit is based on a portion of the image of the layers of the decoding units preceding each decoding unit within each access unit, excluding the layers associated with each decoding unit.

If the interleaving signaling adopts the second possible state, then within each access unit, the NAL unit is arranged to be non-interleaved relative to the layer associated with the NAL unit.

34. A multi-layer video data stream according to embodiment 32 or 33, wherein each NAL unit has a NAL representing the type of the respective NAL unit among a set of possible types.

The unit type index, and within each access unit, the type of the NAL unit of each access unit follows a sorting rule among the NAL unit types, and between each pair of access units, the sorting rule is broken.

35. A multilayer video encoder for generating a multilayer video data stream (200) consisting of sequences of NAL units (202), the multilayer video encoder being configured to generate the multilayer video data stream (200) such that the multilayer video data stream has pictures (204) of a plurality of layers therein encoded using inter-layer predictive coding, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with each NAL unit.

The sequence of NAL units is constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to one access unit is associated with an image at a given time, and NAL units in different access units are associated with different times.

Within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode each of the decoding units is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within each of the access units, excluding the layer associated with the respective decoding unit.

36. A decoder configured to decode a multi-layer video data stream (200) consisting of sequences of NAL units (202), the multi-layer video data stream (200) having pictures (204) of a plurality of layers therein encoded using inter-layer prediction, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with the respective NAL unit.

The sequence of NAL units is constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to one access unit is associated with an image at a given time, and NAL units in different access units are associated with different times.

Within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode each of the decoding units is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within each of the access units, excluding the layer associated with the respective decoding unit.

37. The decoder according to embodiment 36, wherein the decoder is configured to decode images of multiple layers related to the one moment from the multi-layer video data stream in parallel.

38. The decoder according to embodiment 36 or 37, wherein the decoder is configured to distribute the NAL unit to a plurality of buffers according to the layer to which the NAL unit belongs, so as to buffer the multi-layer video data stream in the plurality of buffers.

39. The decoder according to any one of embodiments 36 to 38, wherein the multi-layer video data stream has interleaved signaling, the interleaved signaling having a first possible state and a second possible state, wherein the decoder is configured to respond to the interleaved signaling in that: the decoder knows that

If the interleaved signaling adopts the first possible state, then within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units, and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit, the inter-layer prediction used to encode each decoding unit is based on a portion of the image of the layers of the decoding units preceding each decoding unit within each access unit, excluding the layers associated with each decoding unit.

If the interleaving signaling adopts the second possible state, then within each access unit, the NAL unit is arranged to be non-interleaved relative to the layer associated with the NAL unit.

40. A decoder according to any one of embodiments 36 to 39, wherein the multi-layer video data stream has interleaved signaling, the interleaved signaling having a first possible state and a second possible state, wherein the decoder is configured to, in response to the interleaved signaling, allocate the NAL unit to a plurality of buffers according to the layer to which the NAL unit belongs, to buffer the multi-layer video data stream in the plurality of buffers, and, in the case that the interleaved signaling has the second possible state, buffer the multi-layer video data stream in one of the plurality of buffers, regardless of the individual NAL units.

The layer to which the unit belongs.

41. The decoder according to any one of embodiments 36 to 40, wherein the multi-layer video data stream (200) is arranged such that each NAL unit has a representation of the respective NAL.

The unit is indexed by a type of NAL unit in a set of possible types, and within each access unit, the type of the NAL unit of each access unit follows a sorting rule within the NAL unit type, and between each pair of access units, the sorting rule is broken, wherein the decoder is configured to use the sorting rule to detect access unit boundaries by detecting whether the sorting rule is broken between two directly consecutive NAL units.

42. A method for generating a multi-layer video data stream (200) consisting of a sequence of NAL units (202), the method comprising generating the multi-layer video data stream (200) such that the multi-layer video data stream has pictures (204) of a plurality of layers therein encoded using inter-layer predictive coding, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with the respective NAL unit, the sequence of the NAL units being constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to an access unit is associated with a time... The images are associated with the time of the access unit, and the NAL units in different access units are associated with different time points. Within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode the respective decoding unit is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within the respective access unit, excluding the layer associated with the respective decoding unit.

43. A method for decoding a multi-layer video data stream (200) consisting of a sequence of NAL units (202), the multi-layer video data stream (200) having pictures (204) of a plurality of layers therein encoded using inter-layer prediction, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with each NAL unit, the sequence of NAL units being constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to an access unit is associated with a picture at a given time. Furthermore, the NAL units in different access units are associated with different times, wherein, within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode each of the decoding units is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within each of the access units, excluding the layer associated with the respective decoding unit.

44. A computer program having program code, the computer program being used to perform the method according to any one of embodiments 42 to 43 when running on a computer.

45. A decoder configured to decode a multi-layer video signal consisting of a sequence of data packets, each data packet including a layer recognition syntax element (806), wherein the decoder is configured to respond to layer recognition extension mechanism signaling (808; 808') in the multi-layer video signal to

If the layer identification extension mechanism signaling (808; 808') sends a signal indicating activation of the layer identification extension mechanism, then for a predetermined data packet (810), the layer identification extension (818) is read (814) from the multi-layer data stream and the layer identification extension (818) is used to determine (816) the layer identification index of the predetermined data packet, and

If the layer identification extension mechanism signaling (808; 808') sends a signal to deactivate the layer identification extension mechanism, then for the predetermined data packet (810), the layer identification index of the predetermined data packet is determined (820) from the layer identification syntax element (806) included in the predetermined data packet.

46. The decoder according to embodiment 45, wherein the layer identification syntax element (806) at least contributes to the layer identification extension mechanism signaling (808), wherein the decoder is configured to determine, at least based on the layer identification syntax element included in the predetermined data packet with or without an escape value, whether the layer identification extension mechanism signaling (808) sends an activation or deactivation signal for the layer identification extension mechanism.

47. The decoder according to embodiment 45 or 46, wherein the high-level syntax element (822) at least contributes to the layer identification extension mechanism signaling (808; 808'), and the decoder is configured to determine, based on the high-level syntax element (822), whether to send an activation or deactivation signal for the layer identification extension mechanism signaling for the predetermined data packet (810).

48. The decoder according to embodiment 47, wherein the decoder is configured to, in response to the high-level syntax element employing the first state, determine that the layer identification extension mechanism signaling (808; 808') sends a signal for deactivation of the layer identification extension mechanism.

49. The decoder according to embodiment 48, wherein the layer identification syntax element further contributes to the layer identification extension mechanism signaling (808), and wherein the decoder is configured to determine that the layer identification extension mechanism signaling sends an activation signal for the layer identification extension mechanism for the predetermined data packet if the high-level syntax element adopts a second state different from the first state and the layer identification syntax element of the predetermined data packet adopts an escape value, and to determine that the layer identification extension mechanism signaling sends a deactivation signal for the layer identification extension mechanism if one of the high-level syntax element adopts the first state and the layer identification element adopts a value different from the escape value applies.

50. The decoder according to embodiment 49, wherein the decoder is configured to concatenate a number representing the layer identification syntax element included in the predetermined data packet and a number representing the layer identification extension if the high-level syntax element adopts a third state different from the first and second states, to obtain the level identification index of the predetermined data packet.

51. The decoder according to embodiment 49, wherein the decoder is configured to determine the length n of the level identification extension using the high-level syntax element if the high-level syntax element is in the second state, and to concatenate a number representing the layer identification syntax element included by the predetermined data packet and an n-bit number representing the level identification extension to obtain the level identification index of the predetermined data packet.

52. The decoder according to any one of embodiments 45 to 51, wherein the decoder is configured to:

If the layer identification extension mechanism signaling sends a signal indicating that the layer identification extension mechanism is activated, then the layer identification index of the predetermined data packet is determined (816) by concatenating a number representing the layer identification syntax element included in the predetermined data packet and a number representing the level identification extension, so as to obtain the level identification index of the predetermined data packet.

53. The decoder according to any one of embodiments 45 to 52, wherein the decoder is configured to:

If the layer identification extension mechanism signaling sends a signal indicating that the layer identification extension mechanism is activated, the layer identification index of the predetermined data packet is determined by adding the level identification extension to a predetermined value (e.g., maxNuhLayerId) to obtain the level identification index of the predetermined data packet.

54. A method for decoding a multilayer video signal composed of a sequence of data packets, each data packet including a layer identification syntax element (806), wherein the method responds to layer identification extension mechanism signaling (808; 808') in the multilayer video signal in that: the method includes:

If the layer identification extension mechanism signaling (808; 808') sends a signal indicating activation of the layer identification extension mechanism, then for a predetermined data packet (810), the layer identification extension (818) is read (814) from the multi-layer data stream and the layer identification extension (818) is used to determine (816) the layer identification index of the predetermined data packet, and

If the layer identification extension mechanism signaling (808; 808') sends a signal to deactivate the layer identification extension mechanism, then for the predetermined data packet (810), the layer identification index of the predetermined data packet is determined (820) from the layer identification syntax element (806) included in the predetermined data packet.

55. A computer program having program code, which, when run on a computer, performs the method according to embodiment 54.

56. A multi-layer video data stream that encodes video data into levels of information content using inter-layer prediction, the levels having an order defined therein, and the video data being encoded into the multi-layer video data stream such that, via the inter-layer prediction, a layer is independent of any subsequent layer according to the order, wherein each layer that depends on one or more other layers via the inter-layer prediction increases the information content (e.g., in terms of different dimensional types) when the video data is encoded into the one or more other layers, wherein the multi-layer video data stream comprises:

A first syntactic structure defines the number M of dependency dimensions spanning the dependency space and the maximum number _N _i of sorting ranks for each dependency dimension i, thereby defining a number of available points in the dependency space and a bijective mapping that maps each rank to available points in at least one subset of the available points in the dependency space.

For each dependency dimension i, the second syntactic structure describes the dependency in the N _i ordering rank of dependency dimension i, thereby defining the dependency between the available points in the dependency space. All available points run in parallel with each available point in the dependency axis from higher to lower ordering ranks. For each dependency dimension, the dependencies that run in parallel with each dependency dimension remain unchanged along the cyclic shift of each dependency dimension other than the respective dimensions, thereby simultaneously defining the dependencies between the layers through the bijective mapping.

57. A network entity configured as follows:

Read the first and second syntax structures of the data stream according to embodiment 56, and

The dependencies between the layers are determined based on the first and second syntactic structures.

58. The network entity according to embodiment 56 is configured as follows:

Select one of the stated levels; and

Discard data packets belonging to layers that are independent of each other in a way that is based on the selected level (e.g., via nuh_layer_id) in the multi-layer video data stream.

NAL unit).

59. A method comprising:

Read the first and second syntax structures of the data stream according to embodiment 56, and

The dependencies between the layers are determined based on the first and second syntactic structures.

60. A computer program having program code, which, when run on a computer, performs the method according to embodiment 59.

References

[1]B.Bross et al., "High Efficiency Video Coding(HEVC) textspecification draft 10", JCTVC-L1003, Geneva, CH, 14–23Jan.2013

[2]G.Tech et al., "MV-HEVC Draft Text 3", JCT3V-C1004, Geneva, CH, 17–23Jan.2013

[3]G.Tech et al., "3D-HEVC Test Model 3", JCT3V-C1005, Geneva, CH, 17–23Jan.2013

[4]J.Chen et al., "SHVC Draft Text 1", JCT-VCL1008, Geneva, CH, 17–23Jan.2013

[5]WILBURN,Bennett,et al.High performance imaging using large cameraarrays.ACM Transactions on Graphics,2005,24.Jg.,Nr.3,S.765-776.

[6]WILBURN,Bennett S.,et al.Light field video camera.In:ElectronicImaging 2002.International Society for Optics and Photonics,2001.S.29-36.

[7]HORIMAI,Hideyoshi,et al.Full-color 3D display system with360degree horizontal viewing angle.In:Proc.Int.Symposium of 3D andContents.2010.S.7-10.

Claims (60)

1. A multi-view decoder configured to reconstruct a plurality of views (12, 15) from a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the multi-view decoder is configured to change the inter-view predictions at spatial segment boundaries (300) of spatial segments (301) into which the first view (12) is divided in response to signaling in the data stream. 2. The multi-view decoder of claim 1, wherein the multi-view decoder is configured to perform a domain restriction on the possible disparity vectors of the trusted order in the data stream during the alteration of the inter-view prediction. 3. The multi-view decoder according to claim 1 or 2, wherein the multi-view decoder is configured to determine, based on the data stream, a disparity vector (308) in the domain of possible disparity vectors for the current portion (302) of the second view (15), and sample the first view (12) at a reference portion (304) of the disparity vector (308) determined by offset from the colocation of the first view 12 to the colocation portion (306) of the current portion (302). 4. The multi-view decoder according to claim 3, wherein the multi-view decoder is configured to, in changing the inter-view prediction, perform a domain restriction on the possible disparity vectors of the trust order in the data stream, and perform the domain restriction on the possible disparity vectors such that the reference portion (304) is located within the spatial segment (301) in which the co-location portion (306) is spatially located. 5. The multi-view decoder according to claim 3, wherein the multi-view decoder is configured to, in changing the inter-view prediction, perform a domain restriction on the possible disparity vectors of the trusted order in the data stream, and perform the domain restriction on the possible disparity vectors such that the reference portion (304) is located within the spatial segment in which the co-location portion (306) is located, and the reference portion (304) is spaced from the boundary of the spatial segment by a distance greater than or equal to the half-width (310) of the interpolation filter kernel, provided that the components of the disparity vectors in the dimension pointing to the boundary (300) have sub-pixel resolution. 6. The multi-view decoder according to any one of the preceding claims, wherein the multi-view decoder is configured to, in changing the interview prediction, use alternative data that is independent of the boundary outside the spatial segment to fill the interpolation filter kernel (311) at the portion extending beyond the boundary (300) of the spatial segment, where the colocalization of the first view to the colocalization portion (306) of the current portion (302) of the second view (15) currently being predicted using the interview prediction is spatially located within the spatial segment. 7. The multi-view decoder according to any one of the preceding claims, wherein the multi-view decoder is configured to, in the inter-view prediction, derive a reference portion (314) within the first view (12) for the current portion of the second view and according to the signaling in the data stream, Check whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) of the first view (12) is spatially located to the current portion (302), and apply the predictor to the current portion (302) derived from the attributes of the reference portion (314), or suppress the application of the parameters of the current portion (302) or apply an alternative predictor to the parameters of the current portion (302) depending on whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) is spatially located. The predictor is applied regardless of whether the reference portion (314) is located within the spatial segment (82) in which the co-location portion is located. 8. The multi-view decoder of claim 7, wherein the multi-view decoder is configured to, in deriving the reference portion (314), Estimate the disparity vector (316) for the current part (302). The common location of the first view to the representative position (318) of the current part (302) or the adjacent part (320) of the first view adjacent to the current part (302) is determined, and The reference portion (314) is determined by applying the disparity vector (316) to the representative position (318). 9. The multi-view decoder of claim 8, wherein the multi-view decoder is configured to estimate a disparity vector for the current portion or a spatially or temporally predicted disparity vector for the current portion based on a depth map transmitted in the data stream. 10. The multi-view decoder according to claim 8 or 9, wherein the multi-view decoder is configured to select the reference portion (314) in determining the reference portion (314) by using the disparity vector (316) in the partitioning of the first view (12) into a coding block, a prediction block, a residual block, and/or a transform block. 11. The multi-view decoder according to any one of claims 7 to 10, wherein the parameters are motion vectors, disparity vectors, residual signals, and/or depth values. 12. The multi-view decoder according to any one of claims 7 to 11, wherein the attribute is a motion vector, a disparity vector, a residual signal, and/or a depth value. 13. A multi-view encoder configured to encode a plurality of views (12, 15) into a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the multi-view encoder is configured to modify the inter-view predictions at the spatial segment boundaries (300) of spatial segments (301) into which the first view (12) is divided. 14. The multi-view encoder of claim 13, wherein the multi-view encoder is configured to perform domain constraints on possible disparity vectors in changing the inter-view prediction. 15. The multi-view encoder according to claim 13 or 14, wherein the multi-view encoder is configured to determine (e.g., by optimization) a disparity vector (308) in the domain of possible disparity vectors for the current portion (302) of the second view (15) (e.g., a prediction block predicted with disparity compensation) and transmit it as a signal in the data stream, and sample the first view (12) at a reference portion (304) of the determined disparity vector (308) offset from the colocation of the first view 12 to the colocation portion (306) of the current portion (302). 16. The multi-view encoder of claim 15, wherein the multi-view encoder is configured to perform a domain restriction of the possible disparity vector such that the reference portion (304) is located (e.g., completely) within the spatial segment (301) in which the co-location portion (306) is located in space. 17. The multi-view encoder of claim 15, wherein the multi-view encoder is configured to perform a domain restriction of the possible disparity vector such that the reference portion (304) is located within the spatial segment in which the co-location portion (306) is spatially located, and the reference portion (304) is spaced from the boundary of the spatial segment by a distance greater than or equal to the half-width (310) of the interpolation filter kernel, provided that the components of the disparity vector in the dimension pointing to the boundary (300) have sub-pixel resolution. 18. The multi-view encoder according to any one of claims 13 to 17, wherein the multi-view encoder is configured to fill an interpolation filter kernel (311) at a portion extending beyond the boundary (300) of the spatial segment in changing the inter-view prediction, wherein the co-localization portion (306) of the first view to the current portion (302) of the second view (15) currently being predicted using the inter-view prediction is spatially located in the spatial segment. 19. The multi-view encoder according to any one of claims 13 to 18, wherein the multi-view encoder is configured to, in the inter-view prediction, derive a reference portion (314) within the first view (12) for the current portion of the second view, and according to signaling in the data stream, Check whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) of the first view (12) is spatially located, and apply the predictor to the current portion (302) derived from the attributes of the reference portion (314), or suppress the application of the parameters to the current portion (302) based on whether the reference portion (314) is located within the spatial segment (301) where the co-location portion (306) is spatially located, or The predictor is applied regardless of whether the reference portion (314) is located within the spatial segment (301) in which the co-location portion is located. 20. The multi-view encoder of claim 19, wherein the multi-view encoder is configured to, in deriving the reference portion (314), Estimate the disparity vector (316) for the current part (314), The common location of the first view to the representative position (318) of the current part (302) or the adjacent part (320) of the first view adjacent to the current part (302) is determined, and The reference portion (314) is determined by applying the disparity vector (316) to the representative position (318). 21. The multi-view encoder of claim 20, wherein the multi-view encoder is configured to estimate a disparity vector for the current portion or a spatially or temporally predicted disparity vector for the current portion based on a depth map transmitted in the data stream. 22. The multi-view encoder according to any one of claims 19 to 21, wherein the parameters are motion vectors, disparity vectors, residual signals, and/or depth values. 23. The multi-view encoder according to any one of claims 19 to 22, wherein the attribute is a motion vector, a disparity vector, a residual signal, and/or a depth value. 24. The multi-view encoder according to any one of claims 13 to 23, configured to send the change as a signal to the decoder in the data stream so that the decoder relies on the change. 25. A multi-view decoder configured to reconstruct a plurality of views (12, 15) from a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the multi-view decoder is configured to use signaling in the data stream as a guarantee that the inter-view predictions (602) are constrained at the spatial segment boundaries (300) of the spatial segments (301) into which the first view (12) is divided, such that the inter-view predictions do not involve any dependency of any current portion (302) of the second view (15) on any spatial segment other than the spatial segment in which the colocation of the first view to the colocation portion (606) of the corresponding current portion of the second view is located. 26. The multi-view decoder of claim 25, configured to, in response to the signaling in the data stream, use inter-view parallelism to adjust the inter-view decoding offset or determine to perform an experiment to reconstruct the first and second views. 27. The multi-view decoder according to claim 25 or 26, wherein the multi-view decoder is configured to determine, based on the data stream, a disparity vector (308) in the domain of possible disparity vectors for the current portion (302) of the second view (15), and to sample the first view (12) at a reference portion (304) of the disparity vector (308) determined by offset from the colocation of the first view 12 to the colocation portion (306) of the current portion (302). 28. A method for reconstructing a plurality of views (12, 15) from a data stream using inter-view predictions from a first view (12) to a second view (15), wherein the method responds to signaling in the data stream to change the inter-view predictions at spatial segment boundaries (300) of spatial segments (301) into which the first view (12) is divided. 29. A method for encoding a plurality of views (12, 15) into a data stream using inter-view prediction from a first view (12) to a second view (15), wherein the method includes changing the inter-view prediction at a spatial segment boundary (300) of a spatial segment (301) into which the first view (12) is divided. 30. A method for reconstructing a plurality of views (12, 15) from a data stream using inter-view prediction from a first view (12) to a second view (15), wherein the method includes using signaling in the data stream as a guarantee that the inter-view prediction (602) is constrained at the spatial segment boundary (300) of a spatial segment (301) into which the first view (12) is divided, such that the inter-view prediction does not involve any dependency of any current portion (302) of the second view (15) on a spatial segment other than the spatial segment in which the colocation of the first view to the colocation portion (606) of the corresponding current portion of the second view is located. 31. A computer program having program code, which, when run on a computer, performs the method according to any one of claims 27 to 30. 32. A multi-layer video data stream (200) comprising a sequence of NAL units (202) having pictures (204) of multiple layers therein encoded using inter-layer prediction, each NAL unit (202) having a layer index (nuh_layer_id) representing the layer associated with each NAL unit, the sequence of NAL units being constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to one access unit is associated with a picture at a time, and NAL units in different access units are associated with different times, wherein within each access unit, for each layer, NAL units associated with each layer are grouped into one or more decoding units (208), and decoding units of NAL units associated with different layers are interleaved such that for each decoding unit (208), the inter-layer prediction used to encode each decoding unit is based on a portion of the picture of the layer of the decoding unit preceding the respective decoding unit, excluding the layer associated with the respective decoding unit. 33. The multi-layer video data stream (200) according to claim 32, wherein the multi-layer video data stream has interleaved signaling, the interleaved signaling having a first possible state and a second possible state, wherein, If the interleaved signaling adopts the first possible state, then within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units, and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit, the inter-layer prediction used to encode each decoding unit is based on a portion of the image of the layers of the decoding units preceding each decoding unit within each access unit, excluding the layers associated with each decoding unit. If the interleaving signaling adopts the second possible state, then within each access unit, the NAL unit is arranged to be non-interleaved relative to the layer associated with the NAL unit. 34. The multi-layer video data stream of claim 32 or 33, wherein each NAL unit has a NAL unit type index representing the type of the respective NAL unit among a set of possible types, and within each access unit, the type of the NAL unit of the respective access unit follows a sorting rule among the NAL unit types, and between each pair of access units, the sorting rule is broken. 35. A multilayer video encoder for generating a multilayer video data stream (200) consisting of a sequence of NAL units (202), the multilayer video encoder being configured to generate the multilayer video data stream (200) such that the multilayer video data stream has pictures (204) of a plurality of layers therein encoded using interlayer predictive coding, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with each NAL unit, the sequence of NAL units being constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to an access unit... The NAL units in different access units are associated with a picture at a specific time point, and the NAL units in different access units are associated with different times points. Within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode the respective decoding unit is based on a portion of the picture of the layer of the decoding unit preceding the respective decoding unit within the respective access unit, excluding the layer associated with the respective decoding unit. 36. A decoder configured to decode a multi-layer video data stream (200) consisting of a sequence of NAL units (202), the multi-layer video data stream (200) having pictures (204) of a plurality of layers therein encoded using inter-layer predictive coding, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with the respective NAL unit, the sequence of NAL units being constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to an access unit corresponds to a picture at a given time. The NAL units in different access units are associated with different times, wherein, within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode the respective decoding unit is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within the respective access unit, excluding the layer associated with the respective decoding unit. 37. The decoder of claim 36, wherein the decoder is configured to decode images of multiple layers associated with the one moment from the multi-layer video data stream in parallel. 38. The decoder of claim 36 or 37, wherein the decoder is configured to distribute the NAL unit to a plurality of buffers according to the layer to which the NAL unit belongs, so as to buffer the multi-layer video data stream in the plurality of buffers. 39. The decoder according to any one of claims 36 to 38, wherein the multi-layer video data stream has interleaved signaling, the interleaved signaling having a first possible state and a second possible state, wherein the decoder is configured to respond to the interleaved signaling in that: the decoder knows that If the interleaved signaling adopts the first possible state, then within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units, and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit, the inter-layer prediction used to encode each decoding unit is based on a portion of the image of the layers of the decoding units preceding each decoding unit within each access unit, excluding the layers associated with each decoding unit. If the interleaving signaling adopts the second possible state, then within each access unit, the NAL unit is arranged to be non-interleaved relative to the layer associated with the NAL unit. 40. The decoder according to any one of claims 36 to 39, wherein the multi-layer video data stream has interleaved signaling, the interleaved signaling having a first possible state and a second possible state, wherein the decoder is configured to, in response to the interleaved signaling, allocate the NAL unit to a plurality of buffers according to the layer to which the NAL unit belongs, to buffer the multi-layer video data stream in the plurality of buffers, and in the case that the interleaved signaling has the second possible state, buffer the multi-layer video data stream in one buffer of the plurality of buffers, regardless of the layer to which each NAL unit belongs. 41. The decoder according to any one of claims 36 to 40, wherein the multi-layer video data stream (200) is arranged such that each NAL unit has a NAL unit type index representing the type of the respective NAL unit among a set of possible types, and within each access unit, the type of the NAL unit of the respective access unit follows a sorting rule among NAL unit types, and between each pair of access units, the sorting rule is broken, wherein the decoder is configured to detect access unit boundaries using the sorting rule by detecting whether the sorting rule is broken between two directly consecutive NAL units. 42. A method for generating a multi-layer video data stream (200) consisting of a sequence of NAL units (202), the method comprising generating the multi-layer video data stream (200) such that the multi-layer video data stream has pictures (204) of a plurality of layers therein encoded using inter-layer predictive coding, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with the respective NAL unit, the sequence of the NAL units being constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to an access unit is associated with a time... The images are associated with the time of the access unit, and the NAL units in different access units are associated with different time points. Within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode the respective decoding unit is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within the respective access unit, excluding the layer associated with the respective decoding unit. 43. A method for decoding a multi-layer video data stream (200) consisting of a sequence of NAL units (202), the multi-layer video data stream (200) having pictures (204) of a plurality of layers therein encoded using inter-layer prediction, each NAL unit (202) having a layer index (e.g., nuh_layer_id) representing the layer associated with each NAL unit, the sequence of NAL units being constructed as a sequence of non-interleaved access units (206), wherein a NAL unit belonging to an access unit is associated with a picture at a given time. Furthermore, the NAL units in different access units are associated with different times, wherein, within each access unit, for each layer, at least some of the NAL units associated with each layer are grouped into one or more decoding units (208), and the decoding units of NAL units associated with different layers are interleaved, such that for each decoding unit (208), the inter-layer prediction used to encode each of the decoding units is based on a portion of the image of the layer of the decoding unit preceding the respective decoding unit within each of the access units, excluding the layer associated with the respective decoding unit. 44. A computer program having program code, the computer program being used to perform the method according to any one of claims 42 to 43 when run on a computer. 45. A decoder configured to decode a multi-layer video signal consisting of a sequence of data packets, each data packet including a layer recognition syntax element (806), wherein the decoder is configured to respond to layer recognition extension mechanism signaling (808; 808') in the multi-layer video signal to If the layer identification extension mechanism signaling (808; 808') sends a signal indicating activation of the layer identification extension mechanism, then for a predetermined data packet (810), the layer identification extension (818) is read (814) from the multi-layer data stream and the layer identification extension (818) is used to determine (816) the layer identification index of the predetermined data packet, and If the layer identification extension mechanism signaling (808; 808') sends a signal to deactivate the layer identification extension mechanism, then for the predetermined data packet (810), the layer identification index of the predetermined data packet is determined (820) from the layer identification syntax element (806) included in the predetermined data packet. 46. The decoder of claim 45, wherein the layer identification syntax element (806) at least contributes to the layer identification extension mechanism signaling (808), wherein the decoder is configured to determine, at least based on the layer identification syntax element included in the predetermined data packet with or without an escape value, whether the layer identification extension mechanism signaling (808) sends an activation or deactivation signal for the layer identification extension mechanism. 47. The decoder according to claim 45 or 46, wherein the high-level syntax element (822) at least contributes to the layer identification extension mechanism signaling (808; 808'), and the decoder is configured to determine, based on the high-level syntax element (822), for the predetermined data packet (810) whether to send a signal indicating whether the layer identification extension mechanism is activated or deactivated. 48. The decoder of claim 47, wherein the decoder is configured to, in response to the high-level syntax element employing the first state, determine that the layer identification extension mechanism signaling (808; 808') sends a signal for deactivation of the layer identification extension mechanism. 49. The decoder of claim 48, wherein the layer identification syntax element further contributes to the layer identification extension mechanism signaling (808), and wherein the decoder is configured to determine that the layer identification extension mechanism signaling sends an activation signal for the layer identification extension mechanism for the predetermined data packet if the high-level syntax element adopts a second state different from the first state and the layer identification syntax element of the predetermined data packet adopts an escape value, and to determine that the layer identification extension mechanism signaling sends a deactivation signal for the layer identification extension mechanism if one of the high-level syntax element adopts the first state and the layer identification element adopts a value different from the escape value applies. 50. The decoder of claim 49, wherein the decoder is configured to concatenate a number representing the layer identification syntax element included in the predetermined data packet and a number representing the layer identification extension if the high-level syntax element adopts a third state different from the first and second states, to obtain the level identification index of the predetermined data packet. 51. The decoder of claim 49, wherein the decoder is configured to determine the length n of the level identification extension using the high-level syntax element if the high-level syntax element is in the second state, and to concatenate a number representing the level identification syntax element included in the predetermined data packet and an n-bit number representing the level identification extension to obtain the level identification index of the predetermined data packet. 52. The decoder according to any one of claims 45 to 51, wherein the decoder is configured to: If the layer identification extension mechanism signaling sends a signal indicating that the layer identification extension mechanism is activated, then the layer identification index of the predetermined data packet is determined (816) by concatenating a number representing the layer identification syntax element included in the predetermined data packet and a number representing the level identification extension, so as to obtain the level identification index of the predetermined data packet. 53. The decoder according to any one of claims 45 to 52, wherein the decoder is configured to: If the layer identification extension mechanism signaling sends a signal indicating that the layer identification extension mechanism is activated, the layer identification index of the predetermined data packet is determined by adding the level identification extension to a predetermined value (e.g., maxNuhLayerId) to obtain the level identification index of the predetermined data packet. 54. A method for decoding a multilayer video signal composed of a sequence of data packets, each data packet including a layer identification syntax element (806), wherein the method responds to layer identification extension mechanism signaling (808; 808') in the multilayer video signal in that: the method includes: If the layer identification extension mechanism signaling (808; 808') sends a signal indicating activation of the layer identification extension mechanism, then for a predetermined data packet (810), the layer identification extension (818) is read (814) from the multi-layer data stream and the layer identification extension (818) is used to determine (816) the layer identification index of the predetermined data packet, and If the layer identification extension mechanism signaling (808; 808') sends a signal to deactivate the layer identification extension mechanism, then for the predetermined data packet (810), the layer identification index of the predetermined data packet is determined (820) from the layer identification syntax element (806) included in the predetermined data packet. 55. A computer program having program code, which, when run on a computer, performs the method according to claim 54. 56. A multi-layer video data stream that encodes video data into levels of information content using inter-layer prediction, the levels having an order defined therein, and the video data being encoded into the multi-layer video data stream such that, via the inter-layer prediction, a layer is independent of any subsequent layer according to the order, wherein each layer that depends on one or more other layers via the inter-layer prediction increases the information content (e.g., in terms of different dimensional types) when the video data is encoded into the one or more other layers, wherein the multi-layer video data stream comprises: A first syntactic structure defines the number M of dependency dimensions spanning the dependency space and the maximum number _N _i of sorting ranks for each dependency dimension i, thereby defining a number of available points in the dependency space and a bijective mapping that maps each rank to available points in at least one subset of the available points in the dependency space. For each dependency dimension i, the second syntactic structure describes the dependency in the N _i ordering rank of dependency dimension i, thereby defining the dependency between the available points in the dependency space. All available points run in parallel with each available point in the dependency axis from higher to lower ordering ranks. For each dependency dimension, the dependencies that run in parallel with each dependency dimension remain unchanged along the cyclic shift of each dependency dimension other than the respective dimensions, thereby simultaneously defining the dependencies between the layers through the bijective mapping. 57. A network entity configured as follows: Read the first and second syntax structures of the data stream according to claim 56, and The dependencies between the layers are determined based on the first and second syntactic structures. 58. The network entity according to claim 56, configured as follows: Select one of the stated levels; and Discard packets (e.g., NAL units) belonging to layers that are independent of each other in a way that is dependent on each other in the multi-layer video data stream, which belong to the selected level (e.g., via nuh_layer_id). 59. A method comprising: Read the first and second syntax structures of the data stream according to claim 56, and The dependencies between the layers are determined based on the first and second syntactic structures. 60. A computer program having program code, which, when run on a computer, performs the method according to claim 59.