HK1228623B - Signaling three-dimensional video information in communication networks - Google Patents

Description

Signaling 3D video information in communication networks

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. Provisional Patent Application No. 61/621,939, filed on April 9, 2012, entitled “ADVANCED WIRELESS COMMUNICATION SYSTEMS AND TECHNIQUES,” and to U.S. Provisional Patent Application No. 61/679,627, filed on August 3, 2012, entitled “ADVANCED WIRELESS COMMUNICATION SYSTEMS AND TECHNIQUES,” the entire disclosures of which are incorporated herein by reference.

Technical Field

Embodiments of the present invention generally relate to the field of communications, and more particularly to signaling three-dimensional video information in a communications network.

Background Art

Three-dimensional (3-D) video offers a high-quality and immersive multimedia experience that has only recently become feasible on consumer electronics and mobile platforms through advances in display technology, signal processing, transmission technology, and circuit design. It is currently being introduced into homes through various channels, including Blu-ray Disc ^™ , cable, and satellite transmission, as well as into mobile networks through 3-D-enabled smartphones. Concepts involving the distribution of such content over wireless networks are under development.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments will be readily understood by the following detailed description taken in conjunction with the accompanying drawings. To facilitate this description, similar reference numerals refer to similar structural elements. The embodiments are illustrated in the figures of the accompanying drawings by way of example and not limitation.

FIG1 schematically illustrates a wireless communication network according to various embodiments.

2a-b illustrate the adaptation of streaming content and/or associated session description and metadata files according to various embodiments.

FIG3 illustrates the setup of a streaming session according to an embodiment.

FIG4 illustrates a frame-compatible packing format according to various embodiments.

FIG5 illustrates a method of signaling 3-D video device capabilities according to various embodiments.

FIG6 illustrates a method of signaling 3-D video content according to various embodiments.

FIG7 schematically depicts an example system in accordance with various embodiments.

DETAILED DESCRIPTION

Illustrative embodiments of the present disclosure include, but are not limited to, methods, systems, computer-readable media, and apparatus for signaling stereoscopic 3D video content capabilities of a client device in a communication network. Some embodiments of the present invention may relate, in this context, to methods, systems, computer-readable media, and apparatus for signaling stereoscopic 3D video content capabilities of a mobile device in a wireless communication network.

Various aspects of the illustrative embodiments will be described using terms commonly employed by those skilled in the art to convey the essence of their work to others skilled in the art. However, alternative embodiments may be practiced using only some of the described aspects, as will be apparent to those skilled in the art. For illustrative purposes, specific numbers, materials, and configurations are set forth to provide a comprehensive understanding of the illustrative embodiments. However, alternative embodiments may be practiced without the specific details, as will be apparent to those skilled in the art. In other instances, well-known features are omitted or simplified so as not to obscure the illustrative embodiments.

Furthermore, various operations will be described as multiple discrete operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations do not need to be performed in the order presented.

The phrase "in some embodiments" is used repeatedly. This phrase generally does not refer to the same embodiment; however, it may refer to the same embodiment. The terms "including," "having," and "comprising" are synonymous unless the context indicates otherwise. The phrase "A and/or B" means (A), (B), or (A and B). The phrases "A/B" and "A or B" mean (A), (B), or (A and B), similar to the phrase "A and/or B." The phrase "at least one of A, B, and C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The phrase "(A)B" means (B) or (A and B), i.e., A is optional.

Although specific embodiments have been illustrated and described herein, those skilled in the art will appreciate that many alternative and/or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the embodiments of the present disclosure. This application is intended to cover any adaptation or variation of the embodiments discussed herein. Therefore, it is expressly intended that the embodiments of the present disclosure be limited only by the claims and their equivalents.

As used herein, a "module" may refer to, be a part of, or include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and/or memory (shared, dedicated, or group) that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

The introduction of the H.264/MPEG-4 Advanced Video Coding (AVC) standard has demonstrated significant improvements in video compression capabilities. Since developing this standard, the joint video group of the ITU-T Video Coding Experts Group (VCEG) and the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Moving Picture Experts Group (MPEG) has also standardized an AVC extension known as Multiview Video Coding (MVC). MVC provides a compact representation of multiple views of a video scene, such as from multiple synchronized video capture devices.

In stereoscopic 3D video applications, two views are displayed. One for the left eye and one for the right eye. Various methods exist for formatting views of stereoscopic 3D video content. In one embodiment, the encoding of stereoscopic paired 3D video can be a special case of MVC, where the left-eye and right-eye views are generated via MVC. Other encoding formats for generating 3D video content are also possible. Various devices may have different capabilities for decoding and rendering these different formats. The embodiments described herein provide various parameters for device capability exchange, which can facilitate the distribution and viewing of 3D video content in communication networks such as wireless networks (e.g., the Evolved Universal Terrestrial Radio Access Network (EUTRAN)).

FIG1 schematically illustrates a network environment 100 according to various embodiments. The network environment 100 includes a user equipment (UE) 104, which may also be referred to as a client terminal or mobile device, wirelessly coupled to a radio access network (RAN) 108. The RAN 108 may include an enhanced node base station (eNB) 112 configured to communicate with the UE 104 via an over-the-air (OTA) interface. The RAN 108 may be part of a 3rd Generation Partnership Project (3GPP) Long Term Evolution (LTE) Advanced network and may be referred to as EUTRAN. In other embodiments, other radio access network technologies may be utilized.

The UE 104 may communicate with a remote media server 116 via the RAN 108. Although the eNB 112 is shown communicating directly with the media server, it will be appreciated that in various embodiments communications may flow through numerous intermediate networking components, such as switches, routers, gateways, etc. For example, in some embodiments, the RAN 108 may be coupled to a core serving network (CSN) that communicatively couples the RAN 108 to a larger network (of which the media server 116 may be considered a part), such as a wide area network.

Although FIG1 depicts the network environment as a wireless communication network, other embodiments may be used in other types of networks, such as a wired network. It will be appreciated that other network environments in which embodiments of the present invention may be employed may include additional, fewer, or different components than those explicitly shown in the example depicted in FIG1 . For example, an embodiment of the present invention employed in a wired network may have a media server 116 and a UE 104 communicating with each other without a RAN 108.

UE 104 and media server 116 may include a number of components configured to facilitate access, storage, transmission, and display of 3D video content. For example, UE 104 may include a content management module 120, a media player 124 having a streaming application 126, and a display 128. Streaming application 126 may include sufficient functionality to receive 3D video content and associated information; decode, disassemble, and otherwise reassemble the 3D video; and render the 3D video on display 128. In various embodiments, streaming application 126 may be referenced in the context of the streaming technology employed. For example, in embodiments where content is streamed via a packet-switched streaming service (PSS), streaming application 126 may be referred to as a PSS application. Content management module 120 may negotiate or otherwise communicate streaming parameters, including, for example, device capability parameters, to enable data to be received in a manner that facilitates operation of media player 124.

The media server 116 may include a content distribution module 132 having a streaming application 134, a content management module 136, and a content store 140. The content distribution module 132 may encode, package, or otherwise assemble 3D video content stored in the content store 140 for transmission to one or more UEs, such as the UE 104. The content management module 136 may negotiate or otherwise communicate streaming parameters (including, for example, device capability parameters) and control the content distribution module 132 in a manner that facilitates the distribution of 3D content.

In some embodiments, one or more of the components shown as part of the media server 116 may be located separately from the media server 116 and communicatively coupled to the media server via a communication link. For example, in some embodiments, the content store 140 may be located remotely from the content distribution module 132 and the content management module 136.

In some embodiments, the content distribution module 132 can distribute the 3D video content to the UE 104 according to a 3GPP streaming standard, in one example, via the eNB 112. For example, the 3D video content can be transmitted according to the PSS standard (e.g., 3GPP TS 26.234 V11.0.0 (March 16, 2012)), the Dynamic Adaptive Streaming over HTTP (DASH) standard (e.g., 3GPP TS 26.247 V.11.0.0 (March 16, 2012)), the Multimedia Broadcast and Multicast Service (MBMS) standard (e.g., TS 26.346 V11.1.0 (June 29, 2012)), and/or the IMS-based PSS and MBMS Service (IMS_PSS_MBMS) standard (e.g., TS 26.237 V.11.0.0 (June 29, 2012)). The streaming application 126 may be configured to receive 3D video content via any of a number of transport protocols, such as Real-time Transport Protocol (RTP), Hypertext Transfer Protocol (HTTP), and the like.

Capability exchange enables a media streaming server, such as media server 116, to provide video content suitable for the particular device in question to a wide range of devices.To facilitate server-side content negotiation for streaming, media server 116 may determine the specific capabilities of UE 104.

Content management module 120 and content management module 136 may negotiate or otherwise communicate parameters for a 3D video content streaming session. This negotiation may occur via session-level signaling via RAN 108. In some embodiments, this session-level signaling may include the transmission of device capability information, including stereoscopic 3D video decoding and rendering capabilities of media player 124. In various embodiments, the device capability information may further include pre-decoder buffer size, initial buffering, decoder capabilities, display characteristics (screen size, resolution, bit depth, etc.), streaming method (Real Time Streaming Protocol (RTSP), HTTP, etc.), adaptation support, Quality of Experience (QoE) support, extended Real-time Transport Protocol (RTCP) reporting support, fast content switching support, supported RTP profiles, Session Description Protocol (SDP) attributes, etc.

During the setup of a streaming session, the content management module 136 can use the device capability information to control the content distribution module 132 in a manner that provides the correct type of multimedia content to the UE 104. For example, the media server 116 can determine which of multiple available variations of the video stream is desirable based on the actual capabilities of the UE 104 to determine the most appropriate stream for the terminal. This can improve the delivery of 3D video content and associated session description and metadata files (e.g., SDP files or media presentation description (MPD) files) to the UE 104.

The content delivery module 132 can access the content in the content storage 140 and adapt the content and/or associated session description and metadata files (e.g., SDP/MPD files) according to the negotiated session parameters before delivering the content/associated files. The content can be decoded by the media player 124 and rendered on the display 128 when delivered to the UE 104.

The adaptation of the content and/or associated session description and metadata files is shown with reference to some specific examples in FIG. 2 a - b , whereas the setup of a streaming session is shown with reference to a specific example in FIG. 3 .

Figure 2a illustrates a DASH-based streaming embodiment with 3D video format adaptation in accordance with some embodiments. Specifically, Figure 2a illustrates an HTTP server 204 communicating with a DASH client 208 and implementing a push-based streaming embodiment, in which streaming control is maintained by the client rather than the server, and the client downloads content from the server through a series of HTTP-based request-response transactions after inspecting the MPD. In DASH-based streaming, the MPD metadata file provides information about the structure and different versions of the media content representations stored in the HTTP server 204 (including different bitrates, frame rates, resolutions, codec types, etc.). Based on this MPD metadata information, which describes the relationships between the segments and how they form the media presentation, the DASH client 208 can request media segments using HTTP GET or partial GET methods. The HTTP server 204 and the DASH client 208 can be similar to and generally interchangeable with the media server 116 and the UE 104, respectively.

In DASH, a set of 3D video formats and corresponding content information can be signaled to the DASH client 208 in an MPD. Depending on the capability profile of the DASH client 208 and the 3D formats it supports, the HTTP server 204 can provide differently formatted content. For example, the HTTP server 204 can exclude 3D formats not supported by the DASH client 208 from the MPD and only include those 3D formats supported by the DASH client 208. In this context, the HTTP server 204 can provide content optimized for different 3D video formats to the DASH client 208. To do so, the HTTP server 204 can exchange signaling using device capabilities from the DASH client 108 that describe the various supported 3D video formats. The DASH client 208 can then request the corresponding version of the 3D video content supported by the DASH client 208. Furthermore, when retrieving the MPD using HTTP, the DASH client 208 can include 3D video codec and format information in the GET request, including any temporary adjustments to the 3D video format based on the profile difference (ProfDiff). In an example, the difference can be configured to temporarily modify one or more MPD parameters for the content rendering session. For example, the difference can be configured to modify the MPD until the content rendering session ends or a subsequent difference (which corresponds to the difference first transmitted) is transmitted to the HTTP server 204. In this way, the HTTP server 204 can distribute the optimized MPD to the DASH client 208.

FIG2 b illustrates an embodiment of RTSP-based streaming with 3D video format adaptation according to some embodiments. In particular, FIG2 b illustrates a server 212 and a client 216 implementing a push streaming approach, where streaming and session control are maintained by the server 212 rather than the client 216. The server 212 and the client 216 can be similar to and generally interchangeable with the media server 116 and the UE 104, respectively.

Examples of push streaming include PSS and IMS_PSS_MBMS services based on RTSP and Session Initiation Protocol (SIP), respectively. In this context, server 212 receives a set of supported 3D video codecs and formats from client 216 and adapts the content based on this information. For example, server 212 selects the most suitable content version from among the stored content versions based on the supported 3D video formats or dynamically transcodes the content, and streams the content to client 216. Session-related metadata carried in SDP can carry 3D video format information for the streamed content.

FIG3 illustrates service discovery with subscription/notification of IMS_PSS_MBMS services, according to some embodiments. Specifically, FIG3 illustrates the interaction between a UE 304, an IP Multimedia (IM) Core Network (CN) subsystem 308, and a Service Discovery Function (SDF) 312. UE 304 may be similar to and generally interchangeable with UE 104. IM CN subsystem 308 and SDF 312 may be part of a core network domain that interfaces with an access network domain (e.g., RAN 108).

In the IMS_PSS_MBMS service, UE 304 may send device capability information (e.g., supported 3D video codecs and formats) to IM CN subsystem 308 in a SIP SUBSCRIBE message during service discovery. IM CN subsystem 308 may then forward this message to SDF 312. SDF 312 determines the correct service discovery information based on, for example, UE 304's capabilities as described in the user profile (personalized service discovery). SDF 312 may then send a SIP 200 OK message to IM CN subsystem 308, which is forwarded to UE 304 to confirm session initiation based on the sent device capability information (which also includes supported 3D video codecs and formats). SDF 132 may then send a SIP NOTIFY message (with the service discovery information) to IM CN subsystem 308, which forwards the SIP NOTIFY message back to UE 304. UE 304 may then respond by sending a SIP 200 OK message to IM CN subsystem 308 , which is then forwarded to SDF 312 .

Such a framework enables optimized service discovery using supported 3D video formats in IMS-based PSS and MBMS user services. Later during the IMS session, UE 304 can also use SIP signaling to indicate updates, including any temporary adjustments to the set of supported 3D video codecs and formats based on ProfDiff (for example, if the current device orientation differs from the default device orientation). This can be done by refreshing the subscription via another SIP SUBSCRIBE message (which includes information about updates to the 3D video format information).

Referring again to FIG. 1 , in some embodiments, the media server 116 may be coupled to a device profile server 144 having profile information for the UE 104. This profile information may include some or all of the device capability information. In such embodiments, the media server 116 may receive identification information from the UE 104 and then retrieve the profile information from the device profile server 144. This may be done as part of session-level signaling.

In some embodiments, UE 104 may supplement the profile information retrieved from device profile server 144 with additional attributes based on ProfDiff signaling or override attributes already defined in its device capability profile. In one example, such temporary adjustments may be triggered by user preferences, such as if the user of a particular session only wants to receive two-dimensional (2D) video (even if the terminal is capable of rendering 3D video).

The streaming application 134 may encode the 3D video content for transmission in the network environment 100 according to a number of different stream types, each of which has an associated frame type. The frame types may include frame packing, simulcast, or 2D plus auxiliary frame types.

Frame packing may include a frame-compatible packing format and a full-resolution per-view (FRPV) packing format. In the frame-compatible packet format, the streaming application 134 may spatially pack the constituent frames of a stereo pair into a single frame and encode the single frame. The output frame generated by the streaming application 126 contains the constituent frames of the stereo pair. The spatial resolution of the original frame of each view and the packed single frame may be the same. In this case, the streaming application 134 may downsample the two constituent frames before the packing operation. The frame-compatible packing format may use vertical interlaced, horizontal interlaced, side-by-side, top-and-bottom, or checkerboard formats as illustrated in Figures 4a-e, respectively, and may be downsampled accordingly.

In some embodiments, the streaming application 134 may indicate the frame packing format used by including one or more frame packing setup supplemental enhancement information (SEI) messages as specified in the H.264/AVC standard into the bitstream. The streaming application 126 may decode the frame, unpack the two component frames of the output frame from the decoder, upsample the frame to recover the encoder-side downsampling process, and render the component frames on the display 128.

The FRPV packaging format can include temporal interleaving. In temporal interleaving, 3D video can be encoded at twice the frame rate of the original video, where each parent image and subsequent image form a stereo pair (left and right views). Rendering of temporally interleaved stereo video can typically be performed at high frame rates, with active (shutter) glasses used to blend the incorrect views in each eye. This relies on accurate synchronization between the glasses and the screen.

In embodiments using the simulcast frame type, the left and right views may be transmitted in separate simulcast streams. The separately transmitted streams may be combined by the streaming application 126 and jointly decoded.

In embodiments using a 2D plus auxiliary frame type, the 2D video content may be sent by the streaming application 134 along with auxiliary information that may be used by the streaming application 126 to render the 3D video on the display 128. The auxiliary information may be, for example, a depth/disparity map that is a 2D map, where each pixel defines the depth/disparity of one or more pixels in the associated 2D video frame.

In some embodiments, other frame types may be used. For example, in some embodiments, the streaming application 134 may be able to encode stereoscopic views into a base view stream and a non-base view stream that can be transmitted in the same or different streams. In some embodiments, this may be referred to as MVC-based stereoscopic video. The non-base view stream may include inter-view prediction frames that provide spatial/temporal prediction information. The base view stream may be sufficient for a single-view (e.g., 2D) decoder to render the base view as 2D video, while the non-base view stream may provide enough information for a 3D decoder, such as the streaming application 126, to render 3D video. If the media server 116 is aware of the UE capabilities, it can omit sending the non-base view stream to devices that do not support 3D video or do not have sufficient bit rate to support 3D video.

In various embodiments, the device capability information transmitted from the content management module 120 and/or the device profile server 144 to the content management module 136 may include a 3D format attribute that includes a list of one or more formats associated with streaming stereoscopic 3D video over a relevant transport protocol (e.g., RTP or HTTP, as supported by the streaming application 126). In some embodiments, the 3D format attribute may be a stream frame packing format for RTP or HTTP, having an integer value of "1" for vertical interleaving, "2" for horizontal interleaving, "3" for side-by-side, "4" for top-and-bottom, "0" for checkerboard, or "5" for temporal interleaving. In some embodiments, the same 3D format attribute may be used to indicate the frame packing formats supported in a specific file or container format. In some embodiments, the 3D format attribute may include a more general value, such as "FP" for frame packing.

In some embodiments, the 3D format attribute may be another stream format with a value of "SC" for simulcast or "2DA" for 2D video plus auxiliary information.

In embodiments where the UE 104 supports more than one format type, it may further indicate one or more preferred format types. This may be done by listing the format types in order of preference, associating a preference indicator with a selected format type, and the like.

In some embodiments, in addition to providing a frame type attribute, content management module 120 and/or device profile server 144 may provide one or more component type attributes. These component type attributes may provide additional details about the specific types of video components that are components of stereoscopic 3D video and that are supported and/or preferred by streaming application 126.

The component type attribute may have a value of "C" for indicating a center-view stream, a value of "CD" for indicating a center-view stream and a depth map, a value of "CP" for indicating a center-view stream and a disparity map, a value of "D" for indicating a depth map, a value of "P" for indicating a disparity map, a value of "L" for indicating a left-view stream, a value of "LD" for indicating a left-view stream and a depth map, a value of "LIL" for indicating a video frame (which includes alternating scan lines from left and right views), a value of "LP" for indicating a left-view stream and a disparity map, a value of "R" for indicating a right-view stream, a value of "Seq" for indicating a frame sequence (e.g., a video stream includes alternating frames from left and right streams - additional signaling (e.g., AVC SEI messages) may be necessary to signal which frames contain left and right views), a value of "SbS" for indicating side-by-side, and a value of "TaB" for indicating top-bottom.

Each format type attribute can be associated with a corresponding set of component type attributes. For example, if the format type is SC, the associated component type can be L or R, indicating left and right views, respectively.

The device capability exchange signaling capability in the PSS specification 3GPP TS 24.234 enables servers to deliver content tailored to the specific device in question to a wide range of devices. To improve the delivery of stereoscopic 3D video content to client terminals, this disclosure describes a new set of attributes that can be included in the PSS vocabulary for device capability exchange signaling. These proposed attributes can describe the 3D decoding and rendering capabilities of a client terminal, including which 3D video frame encapsulation formats the client supports. This can, for example, allow servers and networks to deliver optimized RTSP SDP or DASH MPD to client terminals, as well as perform appropriate transcoding and 3D format conversion to match the transmitted 3D video content to the capabilities of the client device.

Device capability exchange signaling of supported 3D video codecs and formats may be implemented in 3GPP TS 26.234, where three new attributes are included in the PSS vocabulary: (1) for the streaming component, two attributes indicating a list of supported frame packing formats associated with stereoscopic 3D video streaming over RTP and HTTP, respectively, and (2) for the ThreeGPFileFormat component, one attribute indicating a list of supported frame packing formats that may be included in a 3GPP file format (3GP) file associated with stereoscopic 3D video, which is a multimedia container format commonly used for 3GPP-based multimedia services. Details of the attribute definitions are presented below, according to some embodiments.

Property Name: StreamingFramePackingFormatsRTP

Property Definition: List of supported frame packing formats related to stereoscopic 3D video streaming over RTP supported by PSS applications. Frame packing formats within the scope of stereoscopic 3D video include:

Frame compatible packing format: 1 = vertical interlaced, 2 = horizontal interlaced, 3 = side by side, 4 = top and bottom, 0 = checkerboard

Packing format for full resolution per view: 5 = Time Interleaved

Component: Flow

Type: Text (bag)

Legal values: A list of integer values corresponding to supported frame encapsulation formats

Resolution Rules: Attached

Example:

Property name: StreamingFramePackingFormatsHTTP

Property Definition: List of supported frame packing formats related to stereoscopic 3D video streaming over HTTP supported by PSS applications. Frame packing formats in the scope of stereoscopic 3D video include:

Frame compatible packing format: 1 = vertical interlaced, 2 = horizontal interlaced, 3 = side by side, 4 = top and bottom, 0 = checkerboard

Packing format for full resolution per view: 5 = Time Interleaved

Component: Flow

Type: Text (bag)

Legal values: A list of integer values corresponding to supported frame encapsulation formats

Resolution Rules: Attached

Example:

Property Name: ThreeGPFramePackingFormats

Property Definition: A list of supported frame packing formats related to stereoscopic 3D video may be included in a 3GP file and processed by a PSS application.

Component: ThreeGPFileFormat

Type: Text (bag)

Legal values: A list of integer values corresponding to supported frame packing formats. The integer value should be 3 or 4, which correspond to side-by-side and top-and-bottom frame packing formats respectively.

Resolution Rules: Attached

Example:

In some embodiments, a media presentation, such as that described in an MPD, may include attributes and elements common to adaptation sets, representations, and sub-representations. One such element may be a FramePacking element. The FramePacking element may specify frame packing settings for a video media component type. When a FramePacking element is not provided for a video component, frame packing may not be used for the video media component.

The FramePacking element may include a @shcemeIdUri attribute including a Uniform Resource Identifier (URI) for identifying the frame packing configuration scheme employed. In some embodiments, the FramePacking element may further include a @value attribute for providing a value to the descriptor element.

In some embodiments, there may be multiple FramePacking elements. If so, each element may contain sufficient information to select or reject the described representation.

If the scheme or the values of all FramePacking elements are not recognized, the client may ignore the described representation. The client may refuse to adapt the set based on observing the FramePacking elements.

For an adaptation set or representation containing video components conforming to ISO/IEC Information technology — Coding of audiovisual objects — Part 10: Advanced video coding (ISO/IEC 14496-10:2012), the URI of FramePackin@schemeIdUri can be

urn:mpeg:dash:14496:10:frame_pcking_arrangement_type:2011, which may be defined to indicate that the frame packing arrangement as defined by Table D-8 of ISO/IEC 14496-10:2012 ('Definition of frame_packing_arrangement_type') is contained in the FramePacking element. @value may be the 'Value' column as specified in Table D-8 of ISO/IEC 14496-10:2012 and may be interpreted according to the 'Interpretation' column in the same table.

FIG5 illustrates a method 500 for signaling 3D video device capabilities according to some embodiments. Method 500 may be performed by components of a UE (e.g., UE 104). In some embodiments, the UE may include and/or have access to one or more computer-readable media having instructions stored thereon that, when executed, cause the UE or its components to perform method 500.

At 504, the UE may determine device capability information. As described above, this device capability information may include information about the decoding and rendering capabilities of the media player. In some embodiments, a content management module located on the UE or elsewhere may determine this information by directly testing the capabilities by running one or more scripts on the UE. In other embodiments, the content management module may access one or more stored files containing relevant information.

At 508, the UE may provide device capability information, including stereoscopic 3D video decoding and rendering capabilities of the media player at the UE, to the media server 116 or the device profile server 144. As described above, the device capability information may include one or more format type attributes representing a list of frame types supported by the UE's streaming application. In some embodiments, the device capability information may be provided before or after the request at 512.

In some embodiments, some or all of the device capability information may be provided to the media server by another entity (eg, a device profile server).

At 512, the UE may request 3D video content. In some embodiments, the request may be based on an appropriate streaming/transport protocol, such as HTTP, RTP, RTSP, DASH, MBMS, PSS, IMS_PSS_MBMS, etc. The request may be directed to a media server and may include a uniform resource locator (URL) or some other indicator of the requested content or portion thereof. In some embodiments, temporary adjustments to the device capability information (e.g., via ProfDiff signaling) may also be provided along with the request at 508. Accordingly, the UE may supplement the profile information retrieved from the device profile server with additional attributes based on the ProfDiff signaling or override attributes already defined in its device capability profile. In one example, such temporary adjustments may be triggered by user preferences, such as if the user of a particular session only wants to receive two-dimensional (2D) video (even if the terminal is capable of rendering 3D video).

At 516, the UE may receive the requested 3D video content and render the content on the UE's display. Rendering of the content may include various processes, such as, but not limited to, decoding, over-conversion, disassembly, sorting, and the like.

FIG6 illustrates a method 600 for signaling 3D video content according to some embodiments. Method 600 may be performed by components of a media server (e.g., media server 116). In some embodiments, the media server may include and/or have access to one or more computer-readable media having instructions stored thereon that, when executed, cause the media server or components thereof to perform method 600.

The media server may determine device capability information at 604. In some embodiments, the media server may determine the device capability information by receiving information from the UE or a device profile server as part of session level signaling.

At 608, the media server may receive a request for 3D video content. In some embodiments, the request may be in accordance with an appropriate streaming/transport protocol, such as HTTP, RTP, RTSP, DASH, MBMS, PSS, IMS_PSS_MBMS, etc. The request may be from a UE and may include a uniform resource locator (URL) or some other indicator of the requested content or portion thereof. In some embodiments, the request received at 608 may occur concurrently with, before, or after the determination of the device capability information 604. In some embodiments, temporary adjustments to the device capability information (e.g., via ProfDiff signaling) may also be received along with the request at 608. Accordingly, the media server may supplement the profile information retrieved from the device profile server with additional attributes based on the ProfDiff signaling or override attributes already defined in its device capability profile.

At 612 , the media server may generate a session description and/or metadata file to establish a streaming session, such as an SDP file or a media presentation description (MPD), based on device capability information describing stereoscopic 3D video decoding and rendering capabilities of a media player at the UE.

At 616, the media server may encode the 3D video content using a format type indicated in the device capability information as supported by the UE. The 3D video content may then be streamed to the mobile device.

Components described herein, such as UE 104, media server 116, and/or device profile server 144, can be implemented in a system using any suitable hardware and/or software to configure the system as desired. FIG7 illustrates, for one embodiment, an example system 700 that includes one or more processors 704, system control logic 708 coupled to at least one of the processors 704, system memory 712 coupled to the system control logic 708, non-volatile memory (NVM)/storage 716 coupled to the system control logic 708, a network interface 720 coupled to the system control logic 708, and/or an input/output (I/O) device 732 coupled to the system control logic 708.

The processor 704 may include one or more single-core or multi-core processors. The processor 704 may include any combination of general-purpose processors and special-purpose processors (eg, graphics processors, application processors, baseband processors, etc.).

System control logic 708 for one embodiment may include any suitable interface controller for providing any suitable interface to at least one of processors 704 and/or any suitable device or component in communication with system control logic 708 .

System control logic 708 for one embodiment may include one or more memory controllers for providing an interface to system memory 712. System memory 712 may be used to load and store data and/or instructions, such as logic 724. System memory 712 for one embodiment may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

NVM/storage 716 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions, such as logic 724. NVM/storage 716 may include any suitable non-volatile memory, such as flash memory, etc., and/or may include any suitable non-volatile storage devices, such as one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and/or one or more digital versatile disk (DVD) drives, etc.

NVM/storage 716 may include storage resources that are physically part of the device on which system 700 is installed, or that may be accessible by the device but not necessarily part of the device. For example, NVM/storage 716 may be accessible over a network via network interface 720 and/or through input/output (I/O) devices 732.

The logic 724, when executed by at least one of the processors 704, causes the system to perform the operations described herein with respect to the UE 104, the media server 116, and/or the device profile server 144. The logic 724 may additionally/alternatively be disposed in other components of the system, such as in the system control logic 708, and may include any combination of hardware, software, and/or firmware components.

The network interface 720 may include a transceiver 722 for providing a radio interface to the system 700 for communicating over one or more networks and/or with any other suitable device. In various embodiments, the transceiver 722 may be integrated with other components of the system 700. For example, the transceiver 722 may include the processor of the processor 704, the memory of the system memory 712, and the NVM/storage of the NVM/storage 716. The network interface 720 may include any suitable hardware and/or firmware. The network interface 720 may include multiple antennas to provide a multiple-input, multiple-output radio interface. For one embodiment, the network interface 720 may include, for example, a wired network adapter, a wireless network adapter, a telephone modem, and/or a wireless modem.

For one embodiment, at least one of the processors 704 may be packaged together with the logic of one or more controllers of the system control logic 708. For one embodiment, at least one of the processors 704 may be packaged together with the logic of one or more controllers of the system control logic 708 to form a system-in-package (SiP). For one embodiment, at least one of the processors 704 may be integrated on the same chip as the logic of one or more controllers of the system control logic 708. For one embodiment, at least one of the processors 704 may be integrated on the same chip as the logic of one or more controllers of the system control logic 708 to form a system on a chip (SoC).

In various embodiments, I/O device 732 may include a user interface designed to enable a user to interact with system 700, a peripheral component interface designed to enable peripheral components to interact with system 700, and/or a sensor designed to determine environmental conditions and/or location information related to system 700.

In various embodiments, the user interface may include, but is not limited to, a display for rendering 3D video (e.g., an LCD display, a touch screen display, an autostereoscopic display, etc.), a speaker, a microphone, one or more cameras (e.g., a still camera and/or a video camera), a flash (e.g., an LED flash), and a keyboard.

In various embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, a Universal Serial Bus (USB) port, an audio jack, and a power supply interface.

In various embodiments, sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with network interface 720 to communicate with components of a positioning network, such as a Global Positioning System (GPS) satellite.

In various embodiments, the system 700 may be a mobile computing device such as, but not limited to, a laptop computing device, a tablet computing device, a netbook, a smartphone, etc. In various embodiments, the system 700 may have more or fewer components and/or a different architecture.

According to one aspect of the present disclosure, a media server for providing 3D video is provided, the media server comprising: a component for obtaining stream frame encapsulation format attributes from a network entity, the stream frame encapsulation format attributes comprising a list of frame encapsulation formats supported by a client terminal of a wireless communication network and related to streaming of stereoscopic 3D video via a transport protocol supported by a packet switched streaming service (PSS) application on the client terminal, wherein the transport protocol is a real-time transport protocol (RTP) or a hypertext transfer protocol (HTTP); a component for adapting content based on the stream frame encapsulation format attributes; a component for generating a session description or metadata file based on the stream frame encapsulation format attributes to establish a streaming session; and a component for transmitting the adapted content and the generated session description or metadata file to the client terminal.

The list of frame packing formats includes an indication of a vertically interlaced frame compatible packing format, a horizontally interlaced frame compatible packing format, a side-by-side frame compatible packing format, an over-under frame compatible packing format, a checkerboard frame compatible packing format, or a temporally interlaced per-view full-resolution packing format.

The list includes a list of one or more integer values corresponding to one or more supported frame packing formats, respectively.

The list of one or more integer values includes: 1, corresponding to a vertically interlaced frame-compatible packing format; 2, corresponding to a horizontally interlaced frame-compatible packing format; 3, corresponding to a side-by-side frame-compatible packing format; 4, corresponding to an over-under frame-compatible packing format; 0, corresponding to a checkerboard frame-compatible packing format; or 5, corresponding to a time-interlaced full-resolution per-view packing format.

The transmission protocol includes RTP.

The transmission protocol includes HTTP.

The session description or metadata file is a Real Time Streaming Protocol (RTSP) Session Description Protocol (SDP) file or a Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH) Media Presentation Description (MPD) file.

The content is 3D video content and the media server further comprises:

A component transcodes the 3D video content or converts the format of the 3D video content based on the stream frame encapsulation format attributes.

The network entity is a device profile server or a user device including the client terminal.

According to another aspect of the present disclosure, a media server for providing 3D video is provided, the media server comprising: a component for obtaining a frame packing format attribute from a network entity, the frame packing format attribute comprising a list of one or more frame packing formats supported by a user device and related to stereoscopic 3D video that can be included in a Third Generation Partnership Project File Format (3GP) file and processed by a Packet Switched Streaming Service (PSS) application on the user device; and a component for transmitting content to the user device based on the frame packing format attribute.

The list of frame packing formats includes an indication of a side-by-side frame packing format or an over-under frame packing format.

The list includes a list of one or more integer values corresponding to one or more frame packing formats, respectively.

The list of one or more integer values includes 3 corresponding to a side-by-side frame packing format or 4 corresponding to an over-under frame packing format.

According to yet another aspect of the present disclosure, a user equipment (UE) for viewing 3D videos is provided, the UE comprising: a media player for decoding stereoscopic three-dimensional (3D) video content wirelessly received by the UE and rendering it on a display of the UE; and a content management module for: determining device capability information including 3D video codecs and formats supported by the media player; transmitting one or more messages including the device capability information to a media server or a device profile server; and transmitting at least one message to the media server, the at least one message including a request for stereoscopic 3D video content and any temporary adjustments to the device capability information.

The device capability information has a 3D format attribute, which includes the format types supported by the media player circuit.

The format type is a frame packing format type corresponding to a frame-compatible packing format or a per-view full-resolution packing format, a simulcast format type, or a 2D plus auxiliary format type.

The format type is a frame packing format type, which is a frame compatible packing format having a value indicating vertical interlace, horizontal interlace, side by side, top and bottom, or checkerboard.

The 3D video content is wirelessly received by a mobile device via a packet-switched streaming service.

The content management module transmits device capability information in a Session Initiation Protocol (SIP) SUBSCRIBE message to an Internet Protocol Multimedia Core Network Subsystem during service discovery.

After service discovery, the content management module uses SIP signaling to update the set of supported 3D video codecs and formats.

The media player is further configured to receive a stream comprising the requested stereoscopic 3D video content, wherein the media player is further configured to receive the stream according to a Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH) protocol, a Packet Switched Streaming (PSS) protocol, or an Internet Protocol Multimedia Subsystem (IMS)-based PSS and Multimedia Broadcast/Multicast Service (MBMS) protocol.

The media player is further configured to receive a Session Description Protocol (SDP) file or a Media Presentation Description (MPD) metadata file associated with the stream.

The UE further includes an autostereoscopic display configured to render the 3D video content under control of the media player.

According to yet another aspect of the present disclosure, a device profile server for facilitating presentation of 3D video is provided, the device profile server being configured to: obtain a request for 3D capability information relating to a user device; and provide a stream frame packing format attribute comprising a list of frame packing formats supported by the user device associated with streaming of stereoscopic 3D video over a transport protocol supported by a packet-switched streaming service (PSS) application on the user device, wherein the transport protocol is a real-time transport protocol (RTP) or a hypertext transfer protocol (HTTP).

The list of frame packing formats includes an indication of a vertically interlaced frame compatible packing format, a horizontally interlaced frame compatible packing format, a side-by-side frame compatible packing format, an over-under frame compatible packing format, a checkerboard frame compatible packing format, or a temporally interlaced per-view full-resolution packing format.

The transmission protocol includes RTP.

The transmission protocol includes HTTP.

The list includes a list of one or more integer values corresponding to one or more supported frame packing formats, respectively.

The list of one or more integer values includes: 1, corresponding to a vertically interlaced frame-compatible packing format; 2, corresponding to a horizontally interlaced frame-compatible packing format; 3, corresponding to a side-by-side frame-compatible packing format; 4, corresponding to an over-under frame-compatible packing format; 0, corresponding to a checkerboard frame-compatible packing format; or 5, corresponding to a time-interlaced full-resolution per-view packing format.

Although certain embodiments have been illustrated and described herein for purposes of illustration, numerous alternative and/or equivalent embodiments or implementations calculated to achieve the same purpose may be substituted for the embodiments shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptation or variation of the embodiments discussed herein. Therefore, it is expressly stated that the embodiments described herein are limited only by the claims and their equivalents.

Claims (28)

1. An apparatus for use by a media server, the apparatus comprising: A component for obtaining a 3G GP-DASH profile of a user equipment (UE) associated with a wireless communication network, the 3G GP-DASH profile indicating one or more constraints associated with stereoscopic 3D video content supported by the UE. Components used to identify a first media representation that conforms to the obtained 3GP-DASH profile and can be used to transmit to the UE; Components for obtaining a Media Representation Description (MPD), the MPD including information associated with an identified first media representation and information associated with a second media representation that does not conform to the obtained 3GP-DASH profile; A component for modifying the MPD based on the obtained 3GP-DASH profile to exclude information associated with the second media representation that does not conform to the obtained 3GP-DASH profile; and A component used to transmit the modified MPD to the UE. 2. The apparatus of claim 1, wherein the 3GP-DASH profile is a multi-view stereoscopic 3D video profile, used to indicate that the UE supports multi-view stereoscopic 3D video content, which includes temporarily interleaved basic views and non-basic views. 3. The apparatus of claim 1, wherein the 3GP-DASH profile is a frame-encapsulated stereoscopic 3D video profile, used to indicate that the UE supports frame-encapsulated 3D video content, which includes a basic view and non-basic views encapsulated in the same frame. 4. The apparatus of claim 3, further comprising a component for including a frame encapsulation element in the modified MPD to indicate the type of frame encapsulation format for DASH transmission of the media representation. 5. The apparatus of claim 4, wherein the frame encapsulation element indicates that the type of frame encapsulation format used is a vertically interleaved frame compatible encapsulation format, a horizontally interleaved frame compatible encapsulation format, a side-by-side frame compatible encapsulation format, a top-bottom frame compatible encapsulation format, or a checkerboard frame compatible encapsulation format. 6. The apparatus of claim 1, wherein the modified MPD includes one or more attributes associated with an individual DASH representation of the first media representation, wherein the individual DASH representation includes DASH representations associated with different time periods of the first media representation. 7. The apparatus of claim 6, wherein the modified MPD includes attributes associated with a DASH representation that conforms to the first media representation of the 3GP-DASH profile, and excludes attributes associated with one or more representations that do not conform to the first media representation of the 3GP-DASH profile. 8. The apparatus of any one of claims 1-7, further comprising: Components for obtaining a Hypertext Transfer Protocol HTTP GET or partial GET request transmitted by the UE for a DASH representation associated with the first media representation, and A component for sending the DASH representation to the UE in response to the HTTP GET or partial GET request. 9. The apparatus of claim 8, wherein the DASH represents being transmitted to the UE via Multimedia Broadcast and Multicast Service (MBMS). 10. The apparatus of any one of claims 1-7, wherein the 3GP-DASH profile is obtained from a device profile server. 11. An apparatus to be used by a user equipment (UE), the apparatus comprising: The content management module is used for: The identifier of the 3G Partner 3GP-DASH dynamic adaptive stream profile associated with the UE is transmitted via the Long Term Evolution (LTE) wireless communication network. The 3GP-DASH profile is used to indicate one or more constraints associated with stereoscopic 3D video content supported by the UE. The media representation description (MPD) is received, the MPD including information associated with a first media representation conforming to the 3GP-DASH profile, and excluding information associated with one or more other media representations that do not conform to the 3GP-DASH profile; and Transmit a Hypertext Transfer Protocol HTTP GET or partial GET request for the DASH representation associated with the first media representation; and A media player, coupled to the content management module, is used to receive and render the DASH representation. 12. The apparatus of claim 11, wherein the 3GP-DASH profile is a multi-view stereoscopic 3D video profile used to indicate that the UE supports multi-view stereoscopic 3D video content, which includes temporarily interleaved basic views and non-basic views. 13. The apparatus of claim 11, wherein the 3GP-DASH profile is a frame-encapsulated stereoscopic 3D video profile, used to indicate that the UE supports frame-encapsulated 3D video content, which includes a basic view and non-basic views encapsulated in the same frame. 14. The apparatus of claim 13, wherein the MPD includes a frame encapsulation element for indicating the type of frame encapsulation format used for the first media representation. 15. The apparatus of claim 14, wherein the frame encapsulation element indicates that the type of frame encapsulation format used is a vertically interleaved frame compatible encapsulation format, a horizontally interleaved frame compatible encapsulation format, a side-by-side frame compatible encapsulation format, a top-bottom frame compatible encapsulation format, or a checkerboard frame compatible encapsulation format. 16. The apparatus of claim 11, wherein the MPD includes one or more attributes associated with an individual DASH representation of the first media representation, wherein the individual representation includes representations associated with different time periods of the media representation. 17. The apparatus of claim 11, wherein the UE is configured to receive the DASH representation via Multimedia Broadcast and Multicast Service (MBMS). 18. A user equipment comprising the means of any one of claims 11-17, further comprising an autostereoscopic display coupled to the media player for displaying a rendered DASH representation. 19. A media server, comprising: Content management circuitry, used for: Obtain a third-generation partner 3GP-DASH dynamically adaptive streaming profile associated with a user equipment (UE) in a wireless communication network, wherein the 3GP-DASH profile is a multi-view stereoscopic 3D video profile indicating that the UE supports multi-view stereoscopic 3D video content, including temporarily interleaved basic and non-basic views, or the 3GP-DASH profile is a frame-encapsulated stereoscopic 3D video profile indicating that the UE supports frame-encapsulated 3D video content, including basic and non-basic views encapsulated within the same frame; and A Media Representation Description (MPD) is generated based on the obtained 3GP-DASH profile. The MPD includes one or more attributes associated with a first DASH representation that conforms to the 3GP-DASH profile, and excludes attributes associated with a second DASH representation that does not conform to the 3GP-DASH profile. The generated MPD is transmitted to the UE, and A content delivery circuit, coupled to the content management circuit, is used to deliver 3D video content associated with the first DASH representation to the UE. 20. The media server of claim 19, wherein the MPD includes a frame encapsulation element for indicating the type of frame encapsulation format used for the first DASH representation. 21. The media server of claim 20, wherein the frame encapsulation element indicates that the type of frame encapsulation format used is a vertically interlaced frame compatible encapsulation format, a horizontally interlaced frame compatible encapsulation format, a side-by-side frame compatible encapsulation format, a top-bottom frame compatible encapsulation format, or a checkerboard frame compatible encapsulation format. 22. The media server of any one of claims 19-21, wherein the content management circuitry is configured to receive an identifier from the UE, wherein the content management circuitry is configured to obtain the 3GP-DASH profile from the device profile server based on the identifier. 23. An apparatus for use by a user equipment (UE), the apparatus comprising: A component for transmitting, via a Long Term Evolution (LTE) wireless communication network, an identifier of a 3G Partner 3GP-DASH dynamic adaptive stream profile associated with the UE, the 3GP-DASH profile indicating one or more constraints associated with stereoscopic 3D video content supported by the UE. A component for receiving a Media Representation Description (MPD), the MPD including one or more attributes associated with a first individual DASH representation that conforms to the 3GP-DASH profile and excluding one or more attributes associated with a second individual DASH representation that does not conform to the 3GP-DASH profile. Components for transmitting a Hypertext Transfer Protocol HTTP GET or partial GET request for the first individual DASH representation of the media representation; Components for obtaining the DASH representation of the first individual; and Used to render the component represented by the first individual DASH obtained. 24. The apparatus of claim 23, wherein the 3GP-DASH profile is a multi-view stereoscopic 3D video profile used to indicate that the UE supports multi-view stereoscopic 3D video content, which includes temporarily interleaved basic views and non-basic views. 25. The apparatus of claim 23, wherein the 3GP-DASH profile is a frame-encapsulated stereoscopic 3D video profile, used to indicate that the UE supports frame-encapsulated 3D video content, which includes a basic view and non-basic views encapsulated in the same frame. 26. The apparatus of claim 25, wherein the MPD includes a frame encapsulation element for indicating the type of frame encapsulation format used for the media representation. 27. The apparatus of claim 26, wherein the frame encapsulation element indicates that the type of frame encapsulation format used is a vertically interleaved frame compatible encapsulation format, a horizontally interleaved frame compatible encapsulation format, a side-by-side frame compatible encapsulation format, a top-bottom frame compatible encapsulation format, or a checkerboard frame compatible encapsulation format. 28. The apparatus of claim 23, wherein the MPD excludes information associated with one or more other media representations that do not conform to the 3GP-DASH profile.