TWI688263B - Methods and apparatus for deriving composite tracks - Google Patents

Abstract

The techniques described herein relate to methods, apparatus, and computer readable media configured to process (including encoding and decoding) composition layouts. The video data includes a plurality of encoded two-dimensional sub-picture tracks associated with a viewport and a composition operation to compose the tracks. The composition operation comprises data indicative of a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the tracks into a canvas associated with the viewport, and a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. The plurality of two-dimensional tracks are composed into the canvas according to the composition, comprising determining two or more of the composed two-dimensional sub-picture tracks comprise the composition layout, and adjusting the composition based on the composition layout operation to compensate for the composition layout.

Description

Method and device for deriving synthetic track

The techniques described here generally relate to video encoding and decoding, and in particular, to deriving a composite track.

There are different types of 3D content and multi-directional content. For example, panoramic video is a type of video that is captured using a set of cameras, unlike the traditional one-way video captured using only a single camera. For example, cameras can be placed around a specific center point so that each camera captures a portion of the video on the spherical overlay of the scene to capture 360-degree video. Video from multiple cameras can be stitched, possibly rotated, and projected to generate a projected two-dimensional image representing spherical content. For example, isometric projection can be used to map a sphere to a two-dimensional image. This can be further processed, for example, using two-dimensional coding and compression techniques. Finally, using specific transmission mechanisms (eg, thumb drive, digital video disk (DVD), file download, digital broadcasting, and/or online streaming), the encoded and compressed content is stored and transmitted. Such video can be used for virtual reality (VR) and/or 3D video.

On the user side, when the user processes the content, the video decoder decodes the encoded and compressed video and performs back projection to restore the content to the spherical surface. Subsequently, the user can view the rendered content, for example, using a head-mounted viewing device. According to the user's viewport indicating the angle at which the user views the content, the content is usually rendered. The viewport can also include a subdivision that represents the viewing area Amount, which can describe the size and shape of the area viewed by the viewer at a specific angle.

When video processing is not performed in a port-dependent manner, so that the video encoder and/or video decoder do not know what the user is actually going to watch, the entire encoding, transmission, and decoding process will process the entire spherical content. For example, since all spherical content is encoded, transmitted, and decoded, this can allow users to view the content at any particular viewport and/or area.

However, processing all spherical content may be computationally intensive and consume a large bandwidth. For example, for online streaming applications, processing all spherical content places a greater burden on network bandwidth than necessary. Therefore, when bandwidth resources and/or computing resources are limited, it is difficult to maintain the user experience. Some technologies only deal with what the user is watching. For example, if the user is watching the top area (for example, the north pole), there is no need to transmit the bottom portion of the content (for example, the south pole). If the user changes the viewport, the content can be transferred accordingly for the new viewport. For another example, for a free viewpoint TV (FTV) application (for example, it uses multiple cameras to capture video of a scene), the content may be transmitted based on the angle at which the user views the scene. For example, if the user is viewing content from one viewport (eg, camera and/or adjacent camera), there may be no need to transmit content from other viewports.

According to the disclosed subject matter, devices, systems and methods are provided for deriving synthetic orbits.

Some embodiments relate to an encoding method for synthesizing multiple sub-image tracks. The method includes: encoding three-dimensional video data, including encoding into a plurality of two-dimensional sub-image tracks related to a viewport; encoding a synthesis operation for synthesizing a plurality of two-dimensional sub-image tracks of a viewport, wherein the synthesis operation Includes data representing the following: the synthesis of a plurality of 2D sub-image tracks performed to synthesize the plurality of 2D sub-image tracks into a canvas related to the viewport; and the canvas including the canvas Adjust two or more composite layouts created in two or more sub-image tracks to adjust the composite layout operation; and provide encoded 3D video data and composite operations.

In some examples, the composition layout includes a gap between two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas and two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas. At least one of two or more overlaps.

In some examples, the synthesis layout operation of the encoding synthesis operation includes: encoding one or more of the background color, background image, or background video to be used to fill in the plurality of two-dimensional sub-image tracks synthesized on the canvas The gap between two or more.

In some examples, the synthesis layout operation of the encoding synthesis operation includes: encoding the mixed data to be used to mix two or more overlaps among the plurality of two-dimensional sub-image tracks synthesized on the canvas.

In some examples, the synthesis of the encoding synthesis operation includes: selecting a synthesis from the group consisting of: an overlap operation and a track overlap synthesis that specifies an order for overlapping each of the plurality of two-dimensional sub-image tracks on the canvas ; Specify the track grid synthesis used to overlay each of the plurality of two-dimensional sub-image tracks on the canvas; and Specify the combination of the track grids in the plurality of two-dimensional sub-image tracks on the canvas Each performs the overlapping order and matrix track matrix synthesis.

Some embodiments relate to a decoding method for decoding video data. The method includes: receiving (a) a plurality of encoded two-dimensional sub-image tracks related to a viewport and (b) a synthesis operation for synthesizing a plurality of two-dimensional sub-image tracks of a viewport, where the synthesis operation includes Represents data as follows: synthesis of a plurality of two-dimensional sub-image tracks to synthesize a plurality of two-dimensional sub-image tracks into a canvas related to the viewport; and the canvas includes a plurality of two-dimensional sub-pictures synthesized from the canvas Adjust the composition layout operation of the composition when creating two or more composition layouts in the track. The method includes: synthesizing a plurality of two-dimensional sub-image tracks into a canvas based on synthesis, including determining that two or more of the synthesized two-dimensional sub-image tracks include a synthesis layout; and based on the synthesis layout operation, Adjust the composition to make up Compensate the layout.

In some examples, the composition layout includes a gap between two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas and two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas. At least one of two or more overlaps.

In some examples, the synthesis layout operation of the decoding synthesis operation includes: decoding one or a plurality of background colors, background images, or background videos; and synthesizing a plurality of two-dimensional tracks including: filling the plurality of synthesized two on the canvas The gap between two or more in the dimension image track.

In some examples, the synthesis layout operation of the decoding synthesis operation includes: decoding mixed data; and synthesizing the plurality of two-dimensional tracks includes: synthesizing two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas The overlap is mixed.

In some examples, the synthesis of the decoding synthesis operation includes selecting synthesis from a group including: an overlap operation and a track overlap synthesis for ordering overlapping each of the plurality of two-dimensional sub-image tracks on the canvas; Specifies a track grid composition used to overlay each of the plurality of 2D sub-image tracks on the canvas; and specifies each of the plurality of 2D sub-image tracks on the canvas A sequence of overlaps and matrix orbit matrix synthesis.

Some embodiments relate to an apparatus for decoding video data. The device includes a processor that communicates with a memory. The processor is configured to execute a plurality of instructions stored in the memory so that the processor: receives (a) a plurality of encoded two-dimensional sub-images related to the viewport Track and (b) a synthesis operation of synthesizing a plurality of 2D sub-image tracks of the viewport, wherein the synthesis operation includes data representing the following: performing on a plurality of 2D sub-image tracks to convert the plurality of 2D sub-pictures The composition of a canvas like a track composition and a viewport-related composition; and the composition composition operation for adjusting composition when the canvas includes a composition layout created by two or more of a plurality of two-dimensional sub-image tracks synthesized on the canvas. The plurality of instructions causes the processor to synthesize a plurality of two-dimensional sub-image tracks into the canvas according to the synthesis, including determining the synthesized complex Two or more of the two-dimensional sub-image tracks include the composition layout; and the composition is adjusted based on the composition layout operation to compensate the composition layout.

In some examples, the composition layout includes a gap between two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas and two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas. At least one of two or more overlaps.

In some examples, the synthesis layout operation of the decoding synthesis operation includes: decoding one or a plurality of background colors, background images, or background videos; and synthesizing a plurality of two-dimensional tracks including: filling the plurality of synthesized two on the canvas The gap between two or more in the dimension image track.

In some examples, the synthesis layout operation of the decoding synthesis operation includes: decoding mixed data; and synthesizing the plurality of two-dimensional tracks includes: synthesizing two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas The overlap is mixed.

In some examples, the synthesis of the decoding synthesis operation includes selecting synthesis from a group including: an overlap operation and a track overlap synthesis for ordering overlapping each of the plurality of two-dimensional sub-image tracks on the canvas; Specifies track grid synthesis for overlapping each of the plurality of 2D sub-image tracks on the canvas; and specifies each of the plurality of 2D sub-image tracks on the canvas The overlapping order and the matrix of the matrix of the matrix are synthesized.

Some embodiments relate to a device for encoding video data. The device includes a processor that communicates with a memory. The processor is configured to execute a plurality of instructions stored in the memory, so that the processor: encodes three-dimensional video data, including encoding a plurality of two-dimensional sub-pictures related to the viewport Image track; encoding is used to synthesize a plurality of two-dimensional sub-image tracks of a viewport. The synthesis operation includes data representing the following: performing on a plurality of two-dimensional sub-image tracks to perform a plurality of two-dimensional sub-image tracks The composition of the image track composition canvas related to the viewport; and adjustment of the composition layout operation of the composition when the canvas includes a composition layout created by two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas; To And provide encoded 3D video data and synthesis operations.

In some examples, the composition layout includes a gap between two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas and two or more of the plurality of two-dimensional sub-image tracks synthesized on the canvas. At least one of two or more overlaps.

In some examples, the synthesis layout operation of the encoding synthesis operation includes: encoding one or more of the background color, background image, or background video to be used to fill in the plurality of two-dimensional sub-image tracks synthesized on the canvas The gap between two or more.

In some examples, the synthesis layout operation of the encoding synthesis operation includes: encoding the mixed data to be used to mix two or more overlaps among the plurality of two-dimensional sub-image tracks synthesized on the canvas.

In some examples, the synthesis of the encoding synthesis operation includes selecting synthesis from a group including: an overlap operation and a track overlap synthesis that specifies an order for overlapping each of the plurality of two-dimensional sub-image tracks on the canvas; Specifies a track grid composition used to overlay each of the plurality of 2D sub-image tracks on the canvas; and specifies each of the plurality of 2D sub-image tracks on the canvas A sequence of overlaps and matrix orbit matrix synthesis.

Therefore, the characteristics of the disclosed subject matter are outlined in order to better understand the following specific embodiments, and to better understand the contribution to the field. Of course, there are additional features of the disclosed subject matter, which will be described below, and form the subject matter of the scope of the attached patent application. It is understandable that the terms and terms used in this article are for descriptive purposes and should not be regarded as limiting.

100: Video codec configuration

102A~102N: Camera

104: coding equipment

106: Video processor

108: encoder

110: decoding device

112: decoder

114: Renderer

116: Display

200: Process

201: Spherical viewport

202~214, 518, 618: box

300: schematic

302, 304A~304D, 306: track

308: Synthetic Track C

310: synthetic orbit r

312: Meta data track m

400: Form

500, 600, 700: structure

502: Category TrackOverlayComposition

504: ‘tocp’ conversion attribute

506, 606, 706: Version data column

508, 608, 708: sign data column

510, 612, 710: data column output_width

512, 613, 711: data column output_height

514, 614: data column horizontal_offset

516,615: data column vertical_offset

520, 620, 720: background_flag

522, 622, 722: canvas_fill_value

524, 624, 724: image_item_ID

526, 626, 726: video_track_ID

528, 628, 728: blending_flag

530, 630, 730: alpha_blending_mode

532, 632, 732: blending_mode_specific_params

534: Number of input entries num_inputs

602: Category TrackGridComposition

604: ‘tgcp’ conversion attribute

610: parameter rows_minus_one

611: Parameter columns_minus_one

702: Category TrackGridComposition

704: ‘tmcp’ conversion attribute

734:matrix_flag

736: num_inputs data column

738, 750: matrix

740: Data column width

742: Data column height

802: synthetic orbit

804, 806: Clip or sub-image track v _l

808A~808N, 810A~810N: Quality

812, 814: "Alternating"

816: Synthesis

900: Method

902~910: steps

In the drawings, each identical or nearly identical element shown in different drawings is represented by the same reference character. For clarity, not every component is labeled in every drawing. The drawings are not necessarily drawn to scale, but focus on all aspects of the technology and equipment described here.

Figure 1 is an exemplary video codec configuration according to some embodiments.

Figure 2 is a flow of viewport dependent content for virtual reality content according to some examples.

FIG. 3 is an exemplary schematic diagram of using a synthetic track to send a viewport/region of interest (ROI) according to some embodiments.

Figures 4A-4B are exemplary tables of mixed modes according to some examples.

FIG. 5 is an exemplary track overlap composition structure for mixing according to some embodiments.

Figure 6 is an exemplary track grid synthesis structure for mixing according to some embodiments.

FIG. 7A is an exemplary conversion matrix synthesis structure for mixing according to some embodiments.

Figure 7B is an exemplary conversion matrix according to some embodiments.

Figure 8 is an exemplary composite track of a collection of different sub-images and quality tracks according to some embodiments.

Figure 9 is an exemplary computer method for synthesizing a plurality of sub-image tracks according to some embodiments.

Different techniques can be used to derive the synthetic track, including the synthesis track of the file format, such as the ISO Base Media File Format (ISOBMFF).

The existing techniques for deriving synthetic orbits do not provide robust orbit synthesis. For example, in order to perform synthesis, a plurality of sub-image tracks can be synthesized to form a viewport. However, during compositing, the canvas (which may also be referred to as a compositing layout) may include different layouts, for example, gaps and/or overlaps between images from the sub-image track are from the sub-image track Overlap of images. According to an embodiment of the present invention, technical improvements to existing file formats for deriving synthetic tracks have been developed. These techniques may include applying metadata to sub-picture track groups. These techniques may allow sub-picture track groups to be specified in a way that allows metadata to be related to the track group's synthesized content, rather than requiring individual sub-picture track assignments. Metadata can specify how track derivation of synthesized content is performed. In some examples, these techniques may be used, for example, to specify criteria for gap conditions and/or overlap conditions, This includes determining the background for filling and/or merging.

In the following, for a thorough understanding of the disclosed subject matter, a large number of specific details regarding the systems and methods of the disclosed subject matter and the environment in which these systems and methods may operate are provided. Additionally, it is understood that the examples provided below are exemplary, and it is conceivable that there are other systems and methods that fall within the disclosed subject matter.

Figure 1 shows an exemplary video codec configuration 100 according to some embodiments. The cameras 102A-102N are N cameras, and may be any type of camera (for example, a camera including audio recording capability and/or a separate camera and audio recording function). The encoding device 104 includes a video processor 106 and an encoder 108. The video processor 106 processes the video received from the cameras 102A-102N, for example, stitching, projection, and/or mapping. The encoder 108 encodes and/or compresses two-dimensional video data. The decoding device 110 receives the encoded material. Through the broadcast network, through the mobile network (eg, cellular network), and/or through the Internet, the decoding device 110 can receive video as a video product (eg, digital video disc or other computer-readable medium). The decoding device 110 may be, for example, a computer, a handheld device, a part of a head-mounted display, or any device with decoding capabilities. The decoding device 110 includes a decoder 112 that is configured to decode the encoded video. The decoding device 110 also includes a renderer 114 for rendering two-dimensional content back to the sphere. The display 116 displays the rendered content from the renderer 114.

The region of interest (ROI) is somewhat similar to the viewport in concept. For example, the region of interest may represent a region in 3D or 2D encoding of panoramic video. The region of interest may have different shapes (eg, square or circular), which may be designated to be related to 3D or 2D video (eg, based on location, height, etc.). For example, the region of interest may represent a region in an image that can be enlarged, and the corresponding region of interest video may be displayed for the enlarged video content. In some embodiments, the region of interest video has been prepared separately. In these embodiments, the region of interest usually has a separate video track that carries the content of the region of interest. Therefore, encoded video can It can be used to specify the region of interest and how the video of the region of interest relates to the underlying video.

The ROI track or the viewport track, such as a separately encoded ROI track, can be related to the main video. For example, the region of interest may be related to the main video to facilitate zoom-in and zoom-out operations, where the region of interest is used to provide the content of the zoom-in region. For example, MPEG-B, Part 10, titled "Carriage of Timed Metadata Metrics of Media in ISO Base Media File Format," dated June 2, 2016 (w16191), describes the ISO-based media file format (ISO Base The Media File Format (ISOBMFF) file format uses a clocked metadata track (timed metadata track) to signal that the main 2D video track has a 2D region of interest track.

Generally, using spherical content, 3D content can be represented to provide a 360-degree view of the scene (eg, sometimes referred to as panoramic media content). Although multiple views can support the use of a 3D sphere, end users usually only view part of the content on the 3D sphere. The bandwidth required to transmit the entire 3D spherical surface places a heavy burden on the network and may not be sufficient to support spherical content. Therefore, there is a need to make 3D content delivery more efficient. Viewport-based processing can be performed to improve 3D content delivery. The 3D spherical content can be divided into regions/tiles/sub-images, and only regions/slices/sub-images related to the viewing screen (eg viewport) can be sent and transmitted to the end user.

Figure 2 shows a flow 200 of viewport-based content for virtual reality content according to some examples. As shown, in block 202, the spherical viewport 201 (for example, it may include the entire spherical surface) undergoes stitching, projection, and mapping (to generate a projected and mapped area); in block 204, it is encoded ( To generate multiple encoded/transcoded fragments of quality); transmitted in block 206 (in the form of fragments); decoded in block 208 (to generate decoded fragments); in block 210, is Construct (to construct a spherical rendering viewport); and in block 212, is rendered. In block 214, the user interaction may select a viewport, which initiates a plurality of process steps regarding "just in time" as indicated by the dotted arrows.

In process 200, due to current network bandwidth limitations and different adaptability requirements (eg For example, with regard to different qualities, encoders, and protection schemes, the virtual reality content presented using a 3D sphere or any other 3D model is first processed (stitched, projected, and mapped) to a 2D plane (block 202), then, It is encapsulated into a plurality of segment-based (or sub-image-based) and segmented files (in block 204) for transmission and playback. In such a segment-based and segmented file, the spatial segment in the 2D plane (for example, it represents a spatial part, usually in a rectangular shape of the 2D plane content) is usually encapsulated as a collection of variants, for example , With different qualities and bit rates, or with different encoders and protection schemes (for example, different encryption algorithms and encryption methods). In some examples, these variants correspond to representations within the adaptation set in MPEG DASH. In some examples, based on the user's selection of viewports, some of these variants of different segments provide coverage of the selected viewport when put together, these variants of different segments are received by the receiver or transmitted It is given to the receiver (via transmission block 206) and then decoded (in block 208) to construct and render the desired viewport (in block 210 and block 212).

As shown in Figure 2, the window concept is what the end user is watching, which involves the angle and size of the area on the sphere. The viewport can be changed and is therefore not static. For example, when the user moves his head, the system needs to obtain adjacent segments (or sub-images) to cover what the user wants to watch next. However, after performing viewport-based processing, for example, including chopping images and/or encoding different qualities, the technology does not allow the designation or association of metadata (eg, related to background and/or blending) to the entire image Like, or the entire 3D spherical content.

In some embodiments, using synthetic tracks, the viewport or region of interest can be signaled. Using a composite track provides a single track, which can be used to represent the content of the variant track of the segment. As another example, the use of synthetic orbits can help establish a hierarchical structure of orbits to indicate how the orbits are related in a synthetic relationship, for example, when synthetic orbits are derived from self-variant orbits and (synthetic) fragment orbits.

The deduced track can be identified by the track containing sample entries of type'dtrk'. Pushed The imported samples may contain a sorted list of operations that will sequentially execute the sorted list of corresponding images or samples from the sorted list of input tracks. Each operation can be specified or represented by TransformProperty. For example, the list of TransformProperty operations may include identification (idtt'); clear hole ('clap'); rotation ('srot'); dissolve ('dslv'); and/or trim ('2dcc'), etc.

To support the flow of content based on virtual reality viewports, additional TransformProperty entries can be used to derive synthetic tracks from existing tracks. Different types of synthesis tracks can be generated, for example, synthesis of all video tracks ('cmpa'), synthesis of only one track ('cmp1', which can allow switching at the sample layer and sample group layer), synthesis of any track ('cmpn', which can allow switching at the sample layer and the sample group layer), selection of only one track ('sel1', which can be selected for the track layer, and does not include switching at the sample layer), selection of any track ('seln', which can be selected for the track layer, and does not include switching at the sample layer), scaling ('scal') and/or re-adjusting ('resz').

For example, the'cmpa' conversion attribute can specify the reference width and height of the derived sample, and each input image can be placed (e.g., synthesized) at its corresponding specific x,y position and specific size on the derived sample. The ‘cmp1’ conversion attribute can specify the reference width and height of the deduced samples, and one, any, and only one of the input images can be placed at their corresponding positions and sizes on the deduced samples. The'cmpn' transformation attribute can specify the reference width and height of the deduced sample, and one or more input images can be placed at their corresponding positions on the deduced sample and have corresponding sizes. The ‘sel1’ transformation attribute can specify the reference width and height of the deduced sample, and one or more input images can be placed on the deduced sample at its corresponding position and corresponding size. The'sel1' conversion attribute can be similar to selecting a track from the list of input tracks. The'seln' conversion attribute can specify the reference width and height of the deduced samples, and one or more input images from the same subset of the input tracks selected in the entire conversion can be placed in their corresponding positions on the deduced samples And have corresponding dimensions. The'seln' conversion attribute may be similar to selecting n (n>0) tracks from the list of input tracks.

Essentially, the'cmpa' operation,'cmp1' operation,'cmpn' operation,'sel1' operation, and'seln' operation specify a plurality of '2dcc' type data columns, which provide 2D spatial information for use according to their respective definitions And semantic synthesizes the input 2D image samples into the derived 2D image samples. According to some embodiments, with these additional TransformProperty entries, the "fragment" track may use the derived track or synthetic track designated as its variant track by'cmp1' or'sel1'. The track of the entire virtual reality spherical content, when projected onto the 2D plane, can use the deduced track or synthesized track designated by ‘cmpa’ as its “segment” track. The viewport or region of interest track can use the derived track or synthesized track that is designated as "segment" track by "cmpn" or "seln". The technique described in m33971, which was proposed in January 2017 and is named "Deriving Composite Tracks in ISOBMFF" (the entirety of which is incorporated herein by reference), provides mapping of 2D synthetic tracks back to spherical 3D content.

Figure 3 shows an exemplary schematic diagram 300 of using a synthetic track to signal a viewport/region of interest according to some embodiments. Four 2D sub-image (or segment) tracks t1-t4 (ie 302) are encoded for use in 2 different qualities (for example,'h' in HD and's' in SD) and 2 different encryption modes (encryption mode'a' with calculator (Counter, CTR) and cipher block chain (Cipher Block Chaining (CBC) encryption mode'b' common encryption scheme (Common Encryption Scheme, CENC)). Encoding and encryption generate 4 different sets of 4 tracks 304A-304D (commonly referred to as encoded tracks 304) for a total of 16 tracks. The encoded segment 304A corresponds to high-quality ('h') (and therefore,'ha') encoded using the first encryption mode ('a'). The encoded segment 304B corresponds to high-quality ('h') (and therefore,'hb') encoded using the second encryption mode ('b'). The encoded segment 304C corresponds to the low quality ('l') (and therefore,'la') encoded using the first encryption mode ('a'). The encoded segment 304D corresponds to the low quality ('l') (and therefore,'lb') encoded using the second encryption mode ('b').

The synthesized track is generated based on the encoded track 304. The segment is selected from the encoded track 304. These fragments may be selected based on one or more transformation properties (eg, the TransformProperties list as discussed above). For example, according to the operation of this example as shown below, 4 The segment s1-s4 is selected: s1=sel1{cmp1{t1ha,t1la}, cmp1{t1hb,t1lb}}

s2=sel1{cmp1{t2ha,t2la},cmp1{t2hb,t2lb}}

s3=sel1{cmp1{t3ha,t3la},cmp1{t3hb,t3lb}}

s4=sel1{cmp1{t4ha,t4la},cmp1{t4hb,t4lb}}

Referring to the above exemplary operation, synthesis (cmp1) and selection (sel1) are used because the synthesis track comes from a plurality of different tracks encoded using different qualities and encryption modes. With ‘cmp1’, the quality variant is synthesized first, and then with ‘sel1’, the encrypted variant is synthesized. For example, assuming m x n grids of sub-image tracks, where each sub-image has q different qualities and is encrypted in 2 modes, there are m x n x q x 2 tracks. When content is adaptively transferred, only the quality of the underlying connection bandwidth is dynamically selected, and the encryption mode is statically selected. Therefore, as discussed above,'cmp1' is used to select the appropriate quality segment at the sample layer or sample group layer, and'sel1' is used to select the appropriate encrypted segment at the track layer.

The synthesis is performed on the selected track 306 to synthesize the sub-image onto the 2D content track. The synthesis process generates a synthesis track C 308 and a synthesis track r 310. Synthetic track C 308 may include data from all selected tracks 306, and synthetic track r 310 may include data from a selected subset of tracks 306. The synthesis track C 308 may be generated, for example, using cmpa {s1, s2, s3, s4} that synthesizes all the fragments. In some embodiments, if the region of interest or viewport involves or is covered by any one of segment t1, segment t2, segment t3, and segment t4, then'cmpn '(Eg, and/or other conversion properties that may be possible if cropping, readjustment, or rotation is involved), the region of interest or viewport track can be determined. For example, using cmpn{s1, s2, s3, s4}, the region of interest or viewport synthesis track r 310 can be determined. In some embodiments, if the region of interest or viewport involves only segments t1 and t2 or is only covered by segments t1 and t2, use ‘cmpa’ (for example, if trimming, readjustment, or rotation is involved The combination of other conversion attributes that can be used), the region of interest or the viewport track can be determined. For example, the synthetic track r 310 may be generated as cmpa{s1,s2}. The encoder generates a metadata track m 312, for example, a metadata track with a clock.

As shown in Figure 2, the viewport concept is what the end user views, which can be specified based on the angle and/or size of the viewing area on the 3D sphere. Note that the viewport is not static. For example, when the user moves his head, the viewport changes, so that the system needs to acquire other clips (or sub-images), such as adjacent clips, to cover what the user wants to watch next. However, after performing viewport-based processing, including, for example, cutting the image and/or encoding different qualities, the technology does not allow metadata to be specified or related to the entire image, or the entire 3D spherical content. For example, the prior art does not provide specified composition layout operations, including compensating composition layout, such as gaps and overlaps.

The file format, such as the MPEG file format, may include techniques for constructing and/or deriving tracks. For example, the derivation technique can be used to combine segments/sub-images into a larger image, for example, to indicate that the larger image has a specific area. Some techniques can operate in the time domain, for example, to aggregate or extract time periods from other tracks. Image processing operations can also be used to construct and/or derive tracks, and can include, for example, marking, trimming, rotating, and/or dissolving operations. The track construction and/or derivation can be used to take a copy from one track and then connect it with another feature from another track to form a formula. For example, it can be used to play an advertisement and then switch to another advertisement, etc. As another example, it can be used to insert advertisements into the program (for example, as a derivation with a construction track from 2 different tracks-one from the advertising track and one from the TV track).

The file format may include conversion, for example, a conversion matrix (for example, it may be in the track header). Each track head may have a conversion matrix to specify how the pixels of this track itself are converted into a larger image, for example, when placed in a different image/larger image. The transformation matrix can be used for simple operations (eg, pixel doubling, 90° rotation) and/or complex operations (eg, clipping (Shearing, arbitrary rotation).

The file format can also include mixed metadata, for example, alpha (alpha) mixed metadata, which can be used to perform alpha blending of 2 entities. When putting tracks together, alpha mixed metadata can be used for gaps or overlapping parts. In this way, mixed metadata can be used to anticipate overlaps and/or gaps, and can be used to specify how overlaps should be formed and/or how gaps should be filled (eg gaps can be filled with background, video, etc.) )Wait.

In some embodiments, the alpha blending parameters may include synthesis information, mode information, and/or specific parameters. For example, for ISOBMFF, multiple columns of data can be specified for alpha-blended metadata. Relative to the origin of the reference coordinate, it can be specified by the application or the container format of the base track and fragment track containing metadata, and the compositing_top_left_x and compositing_top_left_y data columns can specify the upper left corner of the composite space. The compositing_width column and compositing_height column can specify the width and height of the output image after synthesis. The alpha_blending_mode column can specify the alpha blending mode. Tables with different modes and related algorithms with preset parameters can be specified in a single file, such as ISO/IEC 23001-8 and/or W3C recommendations. As shown in FIGS. 4A-4B, an exemplary table 400 of values of alpha_blending_mode is described. The value of the'layer' parameter in the TrackHeaderBox of each track, which specifies the order of the front to back of the visual track, can be set and used as a relative front and back layer indicator to synthesize 2 tracks. In the table, the term'Source' and the term'Destination' are used interchangeably for the front layer/top layer and rear layer/bottom layer or backdrop, respectively. The data field blending_mode_specific_params can specify optional parameters with a specific mixed mode (for example, it can include alpha channel data in addition to using preset values such as specified in ISO/IEC 23001-8).

Existing file format technologies (e.g., synthesis and grouping) do not provide to correlate global information about the entire 3D content (e.g., 3D content representing virtual reality content) with the sub-image track. For example, the prior art may not allow the metadata to be related to the entire image, for example, the metadata is not allowed to identify a specific region of interest within the entire image. For example, a part of the 3D content may need to be identified for user viewing (for example, if you want to call a remote user at some angle or viewpoint, you need to highlight this part of the 3D content). However, because the viewport-based processing technology may destroy the entire 3D image, it is impossible to mark a part of the content to point to the user. Therefore, the prior art does not provide any derivation of correlating any global information about the entire image with the sub-image track. As another example, the prior art does not provide compositing that specifies overlapping operations and/or background operations (for example, gap filling techniques with colors or images, and/or overlapping techniques).

According to the embodiments of the present invention, the technical improvement of the existing file format is used to derive the synthetic track. These techniques may include applying metadata to sub-picture track groups. These techniques can allow sub-picture track groups to be specified in a way that allows metadata to be related to the track group, rather than requiring each sub-picture track to be specified individually. Metadata can specify how orbit derivation is performed. In some examples, these techniques may be used, for example, to specify criteria for layout (including gaps and overlaps) situations, including determining the background for filling and/or merging.

In some embodiments, using one or more of the following mechanisms, synthetic orbit derivation can be performed. In some embodiments, the image overlay mechanism is used in the image file format. In some embodiments, the image grid mechanism is used in the image file format. In some embodiments, the matrix values (for example, in the header information of the track, for example, ISOBMFF) provide conversion information about the sub-image track, such as spatial synthesis information. Using these techniques, for example, the entire 2D frame track can be designated as a derived synthesis track of its segment or sub-image track, and any region of interest track can be designated as a derived synthesis of its related overlay segment or sub-image track track.

In some examples, for example, according to the MPEG ISOBMFF file format, an entry may be a deduced image entry when it includes a'dimg' entry that refers to one or more other image entries that are input for derivation. The exact operation performed to obtain the reconstructed image is identified by the item_type entry. The image item used as the input of the deduced image item is the output image of the other image item, which may be the coded image item or the deduced image item.

An example of a deduced image type is logo derivation. The item_type value'iden' (i.e., identity conversion) of the derived image entry can be used to use the conversion attribute to derive the image entry. The deduced image entry usually has no entry body (for example, no content), and the reference_count referenced by the'dimg' entry of the'iden' deduced image entry may be equal to 1.

Another example of the deduced image type is image overlap derivation. By overlapping one or more input images in a specific layered order within a larger canvas, an entry with an item_type value of ‘iovl’ can specify the derived image entry. In the SingleItemTypeReferenceBox of the type'dimg' in the ItemReferenceBox, these input images can be listed in the order in which they are layered, for example, the bottommost input image first, and the topmost input image last.

FIG. 5 shows an exemplary track overlap composition 500 for possible gaps and mixing according to some embodiments. As shown in structure 500, the category TrackOverlayComposition 502 extends the composition of all'tocp' conversion attributes 504. When present,'tocp' requires the number of input entries num_inputs 534 to be greater than or equal to 1, and these input entries are all visual tracks. Generally, the'tocp' conversion attribute specifies the deduced track, each of its samples is a larger canvas, in the same hierarchical order as it is listed (for example, the bottommost input image is divided first The layer is the base layer, and each additional layer is sequentially layered until the sample images of one or more input tracks of the topmost input image that is finally layered overlap. As discussed further below, these techniques include integrating background and mixed information within the overlay synthesis structure 500, such as canvas colors, images, and/or video backgrounds. This information can be specified (used), for example, when the input sample image does not cover the entire spherical surface. This can be useful, for example, for 360-degree virtual reality content where virtual reality video does not cover the entire spherical surface. In some embodiments, if this information is used, the background video can be processed as an input video in the case of video overlap. In this non-limiting example, the background video is processed as additional content of the input video.

As further shown in Figure 5, the structure 500 may include different parameters. The structure 500 may include a version data column 506. In this example, although the version number may increase as a new version is created, this version is equal to 0. The version number can be used, for example, so that the reader will not deal with TrackOverlayComposition without an unrecognized version number.

The structure 500 may include a logo data column 508. In this example, the logo data column can be used to indicate the number of bits in the data column. For example, (flags & 1) equal to 0 can specify the length of data column output_width 510, data column output_height 512, data column horizontal_offset 514 and data column vertical_offset 516 to 16 bits. For another example, (flags & 1) equal to 1 can specify that the length of the data column output_width 510, the data column output_height 512, the data column horizontal_offset 514, and the data column vertical_offset 516 are 32 bits. The value of the flag greater than 1 can be retained. The output_width 510 data column and output_height 512 data column can specify the width and height of the reconstructed image where the input image is located, respectively. The image area of the reconstructed image can be inferred as a canvas. num_input 534 can specify the number of input entries for this track derivation operation. Horizontal_offset 514 and vertical_offset 516 can specify the offset from the upper left corner of the canvas to where the input image is. The pixel position with a negative offset value may not be included in the reconstructed image. Horizontal pixel positions greater than or equal to output_width 510 may not be included in the reconstructed image. Vertical pixel positions greater than or equal to output_height 512 may not be included in the reconstructed image.

Box 518 shows a portion of the structure 500 related to background and mixed information, which will be discussed further below. The background_flag 520 can be used to indicate which kinds of background can be used to derive the synthetic track. In this example, a value of 0 means that no background is specified. When (background_flag & 1) is equal to 1, the background is a color background, where the color value is specified by canvas_fill_value 522. canvas_fill_value 522 may represent the pixel value of each channel used when no pixel of the input image is located at a specific pixel position. The fill value may be specified as RGBA (for example, red, green, blue, and A corresponding to the loop counter j equal to 0, 1, 2, and 3, respectively). As indicated by IEC 61966-2-1 The RGB values can be in the sRGB color space. The A value may be a linear opaque value, for example, varying from 0 (fully transparent) to 65535 (fully opaque).

When (background_flag & 2) is equal to 2, the background is the image identified by image_item_ID 524, which is scaled (whenever necessary) to cover the background of the size specified by output_width 510 and output_height 512. image_item_ID 524 may specify the ID of the image entry. When (background_flag & 2) is equal to 3, the background is the video sample image identified by video_track_ID 526, scaled (whenever needed) to cover the background of the size specified by output_width 510 and output_height 512. video_track_ID 526 may specify the ID of the video track.

blending_flag 528 indicates whether blending is involved when the input visual track is overlapped in the derivation of the synthesis track. alpha_blending_mode 530 specifies the alpha blending mode (for example, with possible values for "compositing mode" in the example table in FIG. 4). In some embodiments, tables and/or related algorithms with preset parameters may be specified in a single file, such as ISO/IEC 23001-8 and/or W3C recommendations. The parameter value of the TrackHeaderBox in each track is'layer', which specifies the order of the front to back of the visual track, and can be set and used as a relative front and back layer indicator for synthesizing 2 tracks. In the table in FIGS. 4A-4B, the term'source' and the term'target' may be used interchangeably to be used for the front layer/top layer and the rear layer/bottom layer or background curtain, respectively. blending_mode_specific_params 532 may specify optional parameters with a specific blending mode, in addition to using, for example, the preset values specified in ISO/IEC 23001-8. For example, blending_mode_specific_params 532 may include alpha channel data that can be used for the selected alpha blending mode.

An item with an item_type value of'grid' specifies a deduced image item whose reconstructed image is formed from one or more input images in a specific grid order within a larger canvas. FIG. 6 shows an exemplary track grid synthesis structure 600 for mixing according to some embodiments. Similar to the structure 500 discussed in connection with FIG. 5, the structure 600 can be used to specify overlapping parameters such as background and mixed information. Structure 600 may be used to assemble sub-image grids. For example, you need to assemble 2 adjacent sub-images into one Up. In order to prevent overlapping gap effects, when specifying an image, a guard band can also be specified (for example, more than one guard band can be specified to cover adjacent images). For example, although there may be a grid of sub-images of size mxn when these sub-images are put together, there may still be overlap, and/or it may not cover the entire image. Therefore, the structure 600 may allow the output width and/or height to be specified in a manner larger than the gate size. The structure 600 may also allow specifying columns and rows-minus 1 because it always needs to start at 0.

As shown in structure 600, the category TrackGridComposition 602 extends the synthesis of all'tgcp' conversion attributes 604. When present,'tocp' requires the number of input entries num_input to be greater than or equal to 1, and these input entries are all visual tracks. This transformation attribute specifies the deduced tracks, each of which is a larger canvas, overlapping the sample image of one or more input tracks in a particular grid order. In column-based order, the top column first, from left to right, in the order in which it is listed as the input track, the sample image of the input visual track is inserted. The value of num_input will be equal to column * row. Each sample image can be configured to have the same width and height, ie tile_width and tile_height. The width and height can be specified within each input track.

A tiled input sample image may (or may not) completely "overlay" the deduced sample image grid canvas. For example, the tile_width* row may (or may not) be greater than or equal to output_width 612, and/or the tile_height* column may (or may not) be greater than or equal to output_height 613, which will be discussed further here. As a non-limiting example, by tiling the input sample image into a grid with a column width equal to tile_width (essentially excluding the rightmost row) and a column height equal to tile_height (essentially excluding the lowest end column) without gaps and Overlapping, and then, trimming the right and bottom ends to the output_width and output_height indicated, it has been deduced that the sample image can be formed. When the tiled image does not cover the entire canvas background, the canvas, image, and/or video background can be used. As mentioned above, this can be useful, for example, for 360-degree virtual reality content where virtual reality video does not cover the entire spherical surface.

As further shown in FIG. 6, the structure 600 may include different parameters, which will be further explained below. The structure 600 may include a version information column 606. In this example, although the version number may increase as a new version is created, this version is equal to 0. For example, a version number can be used so that the reader will not handle TrackGridComposition without an unrecognized version number.

The structure 600 may include a sign data column 608. In this example, the logo data column can be used to indicate the number of bits in the data column. For example, as explained above, (flags & 1) equal to 0 can specify that the column output_width 612, column output_height 613, column horizontal_offset 614 and column vertical_offset 615 are 16 bits in length, and/or (flags & 1) Equal to 1 can specify that this length is 32 bits.

Box 618 shows a portion of the structure 600 related to background and mixed information, which will be discussed further below. The background_flag 620 can be used to indicate which kinds of background can be used to derive the composite track. In this example, a value of 0 means that no background is specified. When (background_flag & 1) is equal to 1, the background is a color background, where the color value is specified by canvas_fill_value 622. It should be noted here that canvas_fill_value 622 may represent the pixel value per channel used when no pixel of the input image is located at a specific pixel position. As discussed above, the fill value may be specified as RGBA (eg, red, green, blue, and A corresponding to the loop counter j equal to 0, 1, 2, and 3, respectively).

When (background_flag & 2) is equal to 2, the background is the image identified by image_item_ID 624, scaled (eg, if/whenever needed) to cover the background of the size specified by output_width and output_height. When (background_flag & 2) is equal to 3, the background is the video sample image identified by video_track_ID, which is scaled (whenever necessary) to cover the background of the size specified by output_width and output_height. image_item_ID 624 may specify the ID of the image entry. video_track_ID 626 can specify the ID of the video track.

As discussed above, blending_flag 628 indicates overlapping inputs in the derivation synthesis track Whether mixing is involved when visually orbiting. As discussed above, alpha_blending_mode 630 specifies the alpha blending mode. blending_mode_specific_params 632 may specify optional parameters with a specific blending mode, in addition to using, for example, those of the preset values specified in ISO/IEC 23001-8, and it may include alpha channel data.

The parameters rows_minus_one 610 and parameters columns_minus_one 611 can specify the number of rows of the sample image of the input visual track and the number of sample images of each input visual track. This value can be less than the number of rows (rows or columns). According to the order in which the input visual tracks are listed, the sample images of the input visual tracks can be populated in the top row first, and then the second column etc. As discussed above, output_width 612 and output_height 614 may specify the width and height of the reconstructed image where the input image is located, respectively. The image area of the reconstructed image is called the canvas.

Horizontal_offset 614 and vertical_offset 615 specify the offset from the upper left corner of the canvas to where the first input image is located. The pixel position with a negative offset value may not be included in the reconstructed image. Horizontal pixel positions greater than or equal to output_width 612 may not be included in the reconstructed image. Vertical pixel positions greater than or equal to output_height 613 may not be included in the reconstructed image.

In the structure 600 of the exemplary embodiment, only one of horizontal_offset 614 and vertical_offset 615 needs to be specified (for example, unlike structure 500, which uses two offsets to perform a loop). For structure 600, these techniques do not perform a loop-only one offset needs to be specified because this offset can be used for the entire grid. For example, if starting from the leftmost corner, this offset can be specified for this position and for subsequent processing.

The conversion matrix, for example in ISOBMFF, is specified in MovieHeaderBox'mvhd' and TrackHeaderBox'trkd' for processing the decoded track media for presentation. Using this matrix, the entire movie and/or each track can be converted in this matrix. This may allow simple operations (eg, pixel doubling, correction of 90° rotation) and more complex operations (eg, shear, arbitrary rotation). FIG. 7A shows an exemplary conversion matrix synthesis structure 700 for mixing according to some embodiments. Similar to structure 500 and structure 600 discussed in connection with Figures 5 and 6, respectively, structure 700 can be used to specify overlapping parameters, such as background and mixed information. The structure 700 can be used to specify synthesized matrix values.

As shown in structure 700, the category TrackGridComposition 702 extends the synthesis of all'tmcp' conversion attributes 704. When'tmcp' exists, it requires the number of input entries num_input to be greater than or equal to 1, and these input entries are all visual tracks. This'tmcp' conversion attribute specifies the deduced track, each of its sample images is a larger canvas, overlapping the sample image of one or more input tracks in the same layering order as they are listed, For example, the bottom-most input image is layered first, and then each subsequent image is layered until the top-most input image that is finally layered. The size of the canvas is determined by output_width 710 and output_height 711. As discussed further herein, according to the syntax and semantics of the matrix values in the input track header, the temporal parallel samples of the input track can be spatially arranged on the canvas. For example, the size and/or position of the sample image of the input track can be specified by the width, height, and matrix within TrackHeaderBox'trkd'. As discussed above in connection with Figures 5-6, when the input sample image does not cover the entire canvas background, the canvas/image/video background can be used.

As shown in FIG. 7, the structure 700 may include different parameters, which will be further explained below. The structure 700 may include a version data column 706 and a logo data column 708, for example, similar to the version data column and the logo data column discussed in FIGS. 5-6. In the structure 700, the flag data column 708 may specify the data column output_width 710, the data column output_height 711, the data column width, and the length of the data column height.

As discussed in connection with FIGS. 5-6, block 718 shows a portion of the structure 700 related to background and mixed information. By way of restatement, background_flag 720 can be used to indicate which kinds of background can be used to derive the composition track. In this example, a value of 0 means no background being chosen. When (background_flag & 1) is equal to 1, the background is a color background, where the color value is specified by canvas_fill_value 722 (for example, as described above, it is specified as RGBA). As described above, when (background_flag & 2) is equal to 2, the background is the image identified by image_itom_ID 724, which is scaled (eg, if/whenever needed) to cover the background. As described above, when (background_flag & 2) is equal to 3, the background is the video sample image identified by video_track_ID 726, which is scaled (whenever needed) to cover the background.

As also discussed above, blending_flag 728 indicates whether blending is involved when deriving the input visual track in the synthesized track. As discussed above, alpha_blending_mode 730 specifies the alpha blending mode. blending_mode_specific_params 732 may specify optional parameters with a specific blending mode.

Unlike the exemplary structures 500 and 600 discussed in connection with FIGS. 5-6, respectively, the structure 700 includes a matrix_flag 734. matrix_flag 734 may indicate whether other matrix information is used, for example, the matrix in the track header. For example, matrix_flag 734 may indicate whether this matrix, the width value and height value in the track header of the input visual track are available, and/or will not be used (or will be overwritten). When other such matrix information is not available and/or will not be used (eg, matrix_flag==1), these values can be provided to the input visual track. The num_inputs data field 736 specifies the number of input items for this track derivation operation.

The matrix 738 provides a conversion matrix for video. For example, according to some embodiments, using the matrix 750 shown in FIG. 7B, the point (p,q) can be converted into (p',q'). These values in the matrix 750 are stored in the order {a, b, u, c, d, v, x, y, w}. The matrix 750 is multiplied by (p,q,1) to calculate (m,n,z), where m=ap+cq+x; n=bp+dq+y; and z=up+vq+w. Subsequently, by calculating p'= m/z; q'= n/z, (m, n, z) can be used to calculate (p', q'). In this example in FIG. 7A, (u, v, w) of the matrix 738 is limited to (0, 0, 1), hex (0, 0, 0x40000000). In some embodiments, the values in matrix 750 are stored as 16.16 fixed point values, except for u, v, and w, which are stored as 2.30 fixed point values.

In some embodiments, the coordinates {p,q} are located on the decompressed frame, and {p’,q’} are located at the rendering output. So, for example, the matrix {2,0,0,0,2,0,0,0,1} doubles the pixel size of the image. The coordinates transformed by the matrix may not be normalized in any way, and may represent the actual sample position. Therefore, {x, y} can be considered as a translation vector of the image, for example.

In some embodiments, the origin of the coordinates is located in the upper left corner, and X values are added to the right, and Y values are increased downward. {p,q} and {p',q'} are usually used as the absolute pixel position and relative to the upper left corner of the original image (for example, after scaling to the size determined by the width and height of the track header) The surface has been converted (eg, rendered). Each track can be synthesized using its matrix, as specified for the entire image. Subsequently, according to the matrix at the movie layer in the MovieHeaderBox, the synthesized track can be converted and synthesized. It can be based on whether the resulting image is'cropped' to eliminate pixels, which is not displayed, for example, is'cropped' into a vertical rectangular area within the window. For example, if only one video track is displayed and it has a translation to {20,30}, and the identity matrix is in the MovieHeaderBox, the application may choose not to display the empty "L" shaped area between the image and the original.

The data column width 740 and the data column height 742 may be fixed values, for example, a fixed point 16.16 value. These data columns can specify the visual presentation size of the track. These need not be the same as the pixel size of the image, for example, it is recorded in the sample description. The images in the sequence can be scaled to a size based on the width and height, for example, before any entire conversion of the track represented by the matrix. Therefore, the pixel size of the image can be used as a preset value.

Figure 8 shows an exemplary composite track v 802 for r sub-images and k quality tracks according to some embodiments. The track v 402 is synthesized from other clips or sub-picture tracks v _l 804 to v _r 806. Therefore, the clip/sub-picture tracks 804-806 are the actual tracks in the file format carrying bits. In this example, track derivations called "alternate" 812 and "alternate" 814 are used to select the desired from the included qualities (eg, quality 808A-808N of track 804 and quality 810A-810N of track 806) Quality sub-image, as a representative of this sub-image. An alternate process is performed for each segment track/sub-image track with different qualities, forming mxn sub-image grids and mxn tracks, where r=mxn.

Subsequently, the selected track is put into the composite track 802, which represents the entire image. When performing synthesis 816, this process can utilize the techniques discussed herein to process the background and perform blending. Therefore, using a composition operation with background and mixed information, a composition track 802 can be created as discussed herein (eg, the composition discussed in connection with FIGS. 5-7B). For example, if the device needs to access some parts of the content from the entire image, the synthetic track 802 can be accessed and trained down to view the deduced track, including training down to the selected track to obtain the correct Quality.

Returning to Figure 3, for example, as shown in 304A-304D, each track 302 has two qualities/encryptions. When performing synthesis, the device can select a quality and/or encryption for each track to create a synthesized track. In some examples, the device may put all 4 selections together to obtain the entire image of the synthesized track (eg, synthesized track C 308). Or, if the device only needs some viewport areas, the device can selectively select a subset of sub-images to form a smaller composite track r 310. The compositing operation can additionally utilize background and/or mixed information related to this operation to handle any gaps and/or overlaps.

FIG. 9 shows an exemplary method 900 of synthesizing a plurality of sub-image tracks according to some embodiments. In step 902, the device (for example, the decoding device 110 in FIG. 1) receives a plurality of encoded two-dimensional sub-image tracks related to the viewport. In step 904, the device determines the synthesis operation to be used to synthesize a plurality of two-dimensional sub-image tracks for the viewport. As discussed herein, in some embodiments, the compositing operation includes performing synthesis on the plurality of two-dimensional sub-image tracks to synthesize the plurality of two-dimensional sub-image tracks into a canvas (eg, track overlap synthesis, track grid synthesis, and/or Or conversion matrix synthesis). The composition operation also includes a composition layout operation (eg, background and/or mixed information) to adjust the composition when the canvas includes a specific composition layout (eg, a layout with gaps and/or overlaps). In step 906, According to this synthesis, the device synthesizes a plurality of two-dimensional tracks into a canvas. In step 908, during this synthesis, the device determines that 2 or more synthesized two-dimensional sub-image tracks include a synthesis layout. In step 910, the device adjusts the composition based on the composition layout operation to compensate for the composition layout.

As discussed herein, the composition layout operation may include determining how to handle the composition layout, for example, how to fill the gap. For example, the composite information may include a flag that is used to determine whether to fill the gap with a constant color, whether to use an image of the background or whether to use video track content as the background. As also discussed herein, the composite layout operation may include a flag to determine whether to perform blending. If blending is to be performed, the composite information can include parameters that specify how to perform the blending.

As discussed in this article, these techniques can also include encoding synthetic information. The device (eg, encoding device 104) encodes the three-dimensional video data, including encoding a plurality of two-dimensional sub-image tracks associated with the viewport (eg, as discussed in connection with FIG. 2). The device can encode the synthesis operation for synthesizing a plurality of two-dimensional sub-image tracks for the viewport. The compositing operation may include data representing the following: performing synthesis on a plurality of two-dimensional sub-image tracks to synthesize the plurality of two-dimensional tracks into a canvas related to the viewport, and including on the canvas a plurality of two-dimensional sub-composites composed on the canvas The composition layout operation of the composition is adjusted when the composition layout created by two or more of the image tracks (for example, with or without gaps and/or overlaps). The device can provide encoded 3D video data and synthesis operations so that the encoded 3D video data and synthesis operations can be received through the receiving device (eg, through a wired or wireless connection, or through any computer-readable storage medium).

Technical operations according to the principles described herein may be implemented in any suitable manner. The processing blocks and decision blocks of the above flowchart represent the steps and behaviors in the algorithms involved in performing these different processes. The algorithms derived from these processes can be implemented in software that integrates and directs the operation of one or more single-purpose or multi-purpose processors, and can be dedicated to applications such as digital signal processing (DSP) circuits or applications. Integrated circuit (Application-Specific Integrated Circuit, ASIC) functional equivalent circuit to achieve, or may be in any other suitable way To achieve. It should be understood that the flowcharts contained herein do not describe the syntax or operation of any particular circuit or any particular programming language or type of programming language. Rather, the flowchart shows functional information that can be used by those of ordinary skill in the art to manufacture circuits or implement computer software algorithms to perform processing of specific devices of the type of technology described herein. It should also be understood that, unless otherwise stated herein, the specific order of steps and/or actions described in each flowchart is merely an illustration of an algorithm that can be implemented and can be implemented in the principles described herein The method and the embodiment are changed.

Therefore, in some embodiments, the techniques described herein may be implemented in computer-executable instructions implemented as software, including application software, system software, firmware, intermediary software, embedded code, or any other suitable type of computer Code. By using a large number of suitable programming languages and/or programming tools or scripting tools, such computer executable instructions can be written, compiled into executable machine language code or executed on a framework or virtual machine Intermediate code.

When the techniques described herein are implemented as computer-executable instructions, according to these techniques, these computer-executable instructions can be implemented in any suitable manner, including as several functional facilities, each providing one or more operations to complete the calculation Execution of law operations. However, when the generated entity is integrated and executed by one or more computers, a "functional facility" is a structural element of a computer system that enables one or more computers to perform a specific operating role. Functional facilities can be part of the entire software element. For example, the functional facility may be implemented as a function of processing, or as discrete processing, or as any other suitable processing unit. If the technology described here is implemented in multiple functional facilities, each functional facility can be implemented in its own way; all these functional facilities need not be implemented in the same way. In addition, these functional facilities can be executed in parallel and/or in series, and through the use of messaging protocols or in any other suitable manner, these functional facilities can transfer information to each other by using shared memory on the computer being executed.

In general, functional facilities include performing specific tasks or implementing specific abstract data types Routines, programs, objects, components, data structures, etc. In general, the functions of functional facilities can be combined or distributed as required by the system they operate. In some embodiments, one or more functional facilities that perform the techniques herein can together form a complete software package. In alternative embodiments, these functional facilities may be adapted to interact with other unrelated functional facilities and/or processes to implement software program applications.

This article has described some exemplary functional facilities for performing one or more tasks. However, it should be understood that the described functional facilities and task divisions are merely illustrative of the types of functional facilities that implement the exemplary technologies described herein, and embodiments are not limited to implementation in any particular number, division, or type of functional facility. In some embodiments, all functions may be implemented in a single functional facility. It should also be understood that, in some embodiments, some functional facilities described herein may be implemented together with others or separately (ie, as a single unit or separate unit), or some functional facilities may not be implemented.

In some embodiments, computer-executable instructions that implement the techniques described herein (when implemented as one or more functional facilities or in any other way) are encoded on one or more computer-readable media to function Provide to the media. Computer-readable media include magnetic media such as hard drives, such as compact disks (CDs) or digital versatile disks (Digital Versatile Disks, DVDs), persistent or non-persistent solid-state memory (such as flash memory, magnetic random access) Fetch memory, etc.), or any other suitable storage medium. Such computer readable media can be implemented in any suitable way. As used herein, "computer-readable medium" (also referred to as "computer-readable storage medium") refers to tangible storage media. Tangible storage media are non-transitory and have at least one physical, structural element. In the "computer-readable medium" used herein, at least one physical structural member has at least one physical attribute, in the process of creating a medium with implementation information, in the process of recording information on it, or in coding The medium of any other process, which can be changed in some way. For example, during recording, the magnetization state of a part of the physical structure of the computer-readable medium can be changed.

In addition, some of the techniques described above include storing information (eg, data and/or instructions) in some way for the behavior of these techniques. In some embodiments of these technologies-for example, where the technology is implemented as computer-executable instructions-information can be encoded on a computer-readable storage medium. If the specific structures described herein are an advantageous format for storing the information, these structures can be used to impart the physical structure of the information when encoded on the storage medium. These advantageous structures can then provide functionality to the storage medium by affecting the operation of one or more processors interacting with the information; for example, by increasing the efficiency of computer operations performed by the processors.

In some but not all embodiments where the technology is implemented as computer-executable instructions, these instructions may be executed on one or more suitable computing devices operating on any suitable computer system, or one or more computing devices ( Or one or more processors of one or more computing devices) can be programmed to execute computer-executable instructions. When instructions are stored in a computing device or processor by accessing a computing device or processor, a computing device or processor can be programmed to execute the instructions, for example in data storage (eg, an on-chip cache Or instruction register, computer-readable storage medium accessible through the bus, computer-readable storage medium accessible through one or more networks and accessible by the device/processor, etc.). Functional facilities that include these computer-executable instructions can be integrated with and guide the operation of a single multi-purpose programmable digital computing device, share processing power, and coordinate a system of two or more multi-purpose computing devices that jointly perform the techniques described herein , A single computing device or a coordinated system (same location or geographical distribution) dedicated to the implementation of the technical computing devices described in this paper, one or more field-programmable gate arrays (Field-Programmable Gate Array, FPGA), or any other suitable system.

The computing device may include at least one processor, a network interface card, and a computer-readable storage medium. For example, the computing device may be a desktop or notebook computer, a personal digital assistant (PDA), a smartphone, a server, or any other suitable computing device. The network interface card can be any suitable hardware and/or software to enable the computing device to pass through any suitable The computing network is in wired and/or wireless communication with any other suitable computing device. Computer networks may include wireless access points, switches, routers, gateways and/or other network equipment, as well as any suitable wired and/or wireless communication media or media for exchanging data between two or more computers , Including the Internet. The computer-readable medium may be adapted to store materials to be processed and/or instructions to be executed by the processor. The materials and instructions can be stored on a computer-readable storage medium.

The computing device may also have one or more components and peripheral devices, including input devices and output devices. Among other things, these devices can be used to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visually displaying output, and other sound generating devices for speakers or auditory display output. Examples of input devices that can be used for user interfaces include keyboards and pointing devices, such as mice, touch pads, and digital tablets. For another example, the computing device may receive input information through voice recognition or other audible format.

Embodiments have been described in which these techniques are implemented in circuits and/or computer executable instructions. It should be understood that some embodiments may be in the form of a method in which at least one example is provided. The actions performed as part of the method can be ordered in any suitable way. Therefore, these embodiments may be constructed in a manner different from performing the actions in the order shown, even though they may include performing certain actions at the same time as shown as sequential actions in the illustrated embodiment.

The various aspects of the embodiments described above can be used alone, in combination, or in various arrangements not specifically discussed in the above-described embodiments, and therefore are not limited to the specific details and arrangements of the elements applied to the above description or drawings . For example, the aspects described in one embodiment can be combined with the aspects described in other embodiments in any way.

The use of sequential terms such as "first", "second", and "third" in the scope of the patent application to modify the elements of the scope of the patent application does not mean any priority, precedence, or one claim element in another The order above or the chronological order of the execution methods, but only for marking to distinguish one request item element with the same name from another element with the same name (but For the use of sequential terms), and further distinguish the request element.

In addition, the wording and terminology used herein are for descriptive purposes and should not be considered limiting. As used herein, "including", "comprising", "having", "including", and "involving" means that its variations are used to surround the items listed thereafter and their equivalents, as well as additional items.

As used herein, the term "exemplary" means as an example, instance, or illustration. Therefore, unless otherwise stated, any embodiments, implementations, processes, features, etc. described herein should be understood as an illustrative example, and not as a preferred or advantageous example.

After several aspects of at least one embodiment have been described, it should be understood that various changes, modifications, and improvements will easily occur to those of ordinary skill in the art. Such changes, modifications, and improvements will be part of the present invention and are within the spirit and scope of the principles described herein. Therefore, the above description and drawings are only by way of example.

900‧‧‧Method

902~910‧‧‧Step

Claims (20)

An encoding method for synthesizing a plurality of sub-image tracks. The method includes: encoding three-dimensional video data, including encoding into a plurality of two-dimensional sub-picture tracks related to a viewport; encoding is used to encode the viewport A synthesis operation for synthesizing the two-dimensional sub-image tracks, wherein the synthesizing operation includes data representing the following: performing on the two-dimensional sub-image tracks to synthesize the two-dimensional sub-image tracks with the viewport Composition of the canvas; and adjusting the synthesized composition layout operation when the canvas includes a composition layout created by two or more of the two-dimensional sub-image tracks synthesized on the canvas; and providing the coded The three-dimensional video data and the synthesis operation. The encoding method as described in item 1 of the scope of the patent application, wherein the synthesis layout includes the gap between two or more of the two-dimensional sub-image tracks synthesized on the canvas and the synthesized on the canvas At least one of two or more overlaps in the two-dimensional sub-image tracks. The encoding method as described in item 1 of the patent application scope, wherein the synthesizing layout operation encoding the synthesizing operation includes: encoding one or more of the background color, background image or background video to be used to fill the canvas The gap between two or more of the two-dimensional sub-image tracks synthesized above. The encoding method as described in item 1 of the patent application scope, wherein the synthesis layout operation encoding the synthesis operation includes: encoding the mixed data to be used to synthesize the two-dimensional sub-image tracks on the canvas Two or more overlaps are mixed. The encoding method as described in item 1 of the patent application scope, in which the synthesis operation is encoded The composition includes: selecting the composition from the group consisting of: an overlap operation and a track overlap composition that specifies the order in which each of the two-dimensional sub-image tracks on the canvas is overlapped; Each of the two-dimensional sub-image tracks on the canvas performs overlapping grid order track grid synthesis; and is specified to perform each of the two-dimensional sub-image tracks on the canvas The overlapping order and the matrix of the matrix of the matrix are synthesized. A decoding method for decoding video data. The method includes: receiving (a) a plurality of encoded two-dimensional sub-image tracks related to a view port and (b) the two-dimensional sub-images of the view port A synthesis operation for synthesizing tracks, wherein the synthesis operation includes data representing: synthesis of the two-dimensional sub-image tracks to synthesize the two-dimensional sub-image tracks to the canvas associated with the viewport; and The canvas includes a synthetic layout operation that is adjusted when a synthetic layout created by two or more of the two-dimensional sub-image tracks synthesized on the canvas is adjusted; according to the synthesis, the two-dimensional sub-pictures are adjusted Synthesizing the canvas like a track includes: determining that two or more of the two-dimensional sub-image tracks that have been synthesized include the composition layout; and adjusting the composition based on the composition layout operation to compensate for the composition layout. The decoding method as described in item 6 of the patent application scope, wherein the synthesis layout includes the gap between two or more of the two-dimensional sub-image tracks synthesized on the canvas and the synthesized on the canvas At least one of two or more overlaps in the two-dimensional sub-image tracks. The decoding method as described in item 6 of the patent application scope, wherein the synthesis layout operation for decoding the synthesis operation includes: Decoding one or more of background color, background image or background video; and synthesizing the two-dimensional tracks include: filling two or more of the two-dimensional sub-image tracks synthesized on the canvas The gap between. The decoding method as described in item 6 of the patent application scope, wherein the synthesis layout operation of decoding the synthesis operation includes: decoding mixed data; and synthesizing the two-dimensional tracks includes: synthesizing the two on the canvas Two or more overlaps in the sub-image track are mixed. The decoding method as described in item 6 of the patent application scope, wherein decoding the composition of the composition operation includes: selecting the composition from the group consisting of: specifying an overlap operation and the two-dimensional subgraphs used on the canvas Orbital overlay synthesis in the order in which each of the tracks overlaps; designated track grid synthesis for the mesh order in which each of the two-dimensional sub-image tracks on the canvas is overlapped; and In order to superimpose each of the two-dimensional sub-image tracks on the canvas and synthesize the track matrix of the matrix. An apparatus for decoding video data. The apparatus includes a processor that communicates with a memory. The processor is configured to execute a plurality of instructions stored in the memory so that the processor: receives (a) video port-related A plurality of encoded two-dimensional sub-image tracks and (b) a synthesis operation of synthesizing the two-dimensional sub-image tracks of the viewport, wherein the synthesis operation includes data representing the following: The image track is executed to synthesize the two-dimensional sub-image tracks related to the viewport The composition of the canvas; and adjusting the synthesized composition layout operation when the canvas includes a composition layout created by two or more of the two-dimensional sub-image tracks synthesized on the canvas; according to the composition, the Synthesizing the canvas by the two-dimensional sub-image tracks includes: determining that two or more of the synthesized two-dimensional sub-image tracks include the composition layout; and adjusting the composition based on the composition layout operation, To compensate for the composite layout. The device for decoding video data as described in item 11 of the patent application scope, wherein the synthesis layout includes a gap between two or more of the two-dimensional sub-image tracks synthesized on the canvas and the canvas At least one of two or more overlaps in the two-dimensional sub-image tracks synthesized above. The device for decoding video data as described in item 11 of the patent scope, wherein the composition layout operation for decoding the composition operation includes: decoding one or more of background color, background image, or background video; and The synthesis of the two-dimensional track includes filling a gap between two or more of the two-dimensional sub-image tracks synthesized on the canvas. The device for decoding video data as described in item 11 of the patent application scope, wherein the synthesis layout operation for decoding the synthesis operation includes: decoding mixed data; and synthesizing the two-dimensional tracks includes: synthesizing the canvas Two or more overlaps in the two-dimensional sub-image tracks are mixed. The device for decoding video data as described in item 11 of the patent scope, wherein, Decoding the composition of the composition operation includes selecting the composition from the group consisting of: an overlap operation and a track overlap composition that specifies an order for overlapping each of the two-dimensional sub-image tracks on the canvas; Specifying a track grid synthesis for overlapping each of the two-dimensional sub-image tracks on the canvas; and specifying for use in the two-dimensional sub-image tracks on the canvas Each of them performs an overlapping sequence and matrix orbit matrix synthesis. An apparatus for encoding video data. The apparatus includes a processor that communicates with a memory. The processor is configured to execute a plurality of instructions stored in the memory so that the processor: encodes three-dimensional video data, including encoding and A plurality of two-dimensional sub-image tracks related to the viewport; encoding is used to synthesize the two-dimensional sub-image tracks of the viewport, where the synthesizing operation includes data representing the following: The sub-image track is executed to synthesize the two-dimensional tracks to the canvas related to the viewport; and the canvas includes two or more of the two-dimensional sub-image tracks synthesized from the canvas Adjust the synthesized synthesis layout operation when creating the synthesized layout; and provide the encoded three-dimensional video data and the synthesis operation. The device for encoding video data as described in item 16 of the patent application scope, wherein the composition layout includes a gap between two or more of the two-dimensional sub-image tracks synthesized on the canvas and the canvas At least one of two or more overlaps in the two-dimensional sub-image tracks synthesized above. The device for encoding video data as described in item 16 of the patent application scope, wherein the synthesis layout operation encoding the synthesis operation includes: Encode one or more of the background color, background image, or background video to be used to fill the gap between two or more of the two-dimensional sub-image tracks synthesized on the canvas. The device for encoding video data as described in item 16 of the patent application scope, wherein the synthesis layout operation encoding the synthesis operation includes: encoding mixed data to be used to synthesize the two-dimensional sub-images on the canvas Two or more overlaps in the track are mixed. The device for encoding video data as described in item 16 of the patent application scope, wherein the synthesis encoding the synthesis operation includes: selecting the synthesis from the group consisting of: specifying an overlap operation and the two used to apply the canvas Each of the two-dimensional sub-image tracks performs an overlapping sequence of track overlap synthesis; specifies a track grid synthesis for the overlapping grid order of each of the two-dimensional sub-image tracks on the canvas; And a track matrix composition specifying a sequence and a matrix for overlapping each of the two-dimensional sub-image tracks on the canvas.

Images