HK1229978A1 - Systems and methods for automatically generating top level index files - Google Patents

Description

System and method for automatically generating top level index files

The present application is a divisional application of an invention patent application with the application number of 201180074087.1, the application date of 2011, 12 and 31, and the invention name of "a system and a method for automatically generating a top-level index file".

Technical Field

The present invention relates generally to streaming media and more particularly to automatically generating a top level index file for use in adaptive bit rate streaming.

Background

The term streaming media describes the playback of media on a playback device, where the media is stored on a server and continuously transmitted to the playback device over a network during playback. Typically, the playback device stores a sufficient amount of media in the buffer at any given time during playback to prevent playback interruption due to the playback device completing playback of all buffered media before receiving the next portion of media. Adaptive bit rate streaming or adaptive streaming involves detecting the current streaming conditions (e.g., the user's network bandwidth and video decoding capabilities) in real-time and adjusting the quality of the streaming media accordingly. Typically, the source media is encoded as separate streams with different maximum bit rates, and the playback device or client switches between streaming the different encodings depending on the available resources. When a playback device starts adaptive bitrate streaming, the playback device typically starts by requesting portions of the media from an initial set of streams. When the playback device downloads the requested media, the playback device may measure the available bandwidth. In case the available bandwidth increases or decreases, the playback device may switch to a higher or lower bit rate stream.

Adaptive streaming schemes typically utilize the hypertext transfer protocol (HTTP) published by Internet Engineering Task Force and World wide web Consortium as RFC 2616 to stream media between a server and a playback device. HTTP is a stateless protocol that enables playback devices to request byte ranges within a file. HTTP is described as stateless because the server does not need to record information about the state of the playback apparatus requesting the information or about the byte range requested by the playback apparatus in order to respond to a request received from the playback apparatus.

In adaptive streaming systems, source media is typically stored on a media server as a top-level index file that points to multiple alternative streams containing actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming resolutions typically use different indexes and media containers. The Synchronized Multimedia Integration Language (SMIL), developed by World Wide WebConsortium, is used to create indexes in several adaptive Streaming schemes including IIS Smooth Streaming (Smooth Streaming), developed by Microsoft Corporation of redmond, washington, and Flash Dynamic Streaming (Flash Dynamic Streaming), developed by Adobe Systems Incorporated of san jose, california. HTTP Adaptive Bitrate Streaming (HTTP Adaptive Bitrate Streaming) developed by Apple Computer Incorporated of cupertino, california, implements an index file by using an extended M3U playlist file (.m3U8) that is a text file containing a list of URIs that typically identify media container files. The most commonly used media container formats are the MP4 container format specified in MPEG-4 part 14 (i.e., ISO/IEC 14496-14) and the MPEG Transport Stream (TS) container specified in MPEG-2 part 1 (i.e., ISO/IEC standard 13818-1). The MP4 container format is used in IIS smooth streaming and Flash dynamic streaming. TS containers are used in HTTP adaptive bit rate streaming.

Disclosure of Invention

Systems and methods according to embodiments of the present invention automatically generate a top level index file for use in adaptive bitrate streaming. One embodiment of the method of the present invention comprises: receiving, at a playback server, a request from a playback device, wherein the request identifies a piece of content; retrieving, by using the playback server, a list of assets associated with the identified content segment; filtering the asset list using at least one predetermined criterion by using the playback server; generating a top level index file describing each asset in the filtered asset list by using a playback server; and transmitting the top level index file to the playback apparatus by using the playback server.

Further additional embodiments of the method of the present invention include: receiving, at a playback server, a request from a playback device, wherein the request identifies a piece of content; retrieving, by using the playback server, a list of assets associated with the identified content segment; filtering the asset list using at least one predetermined criterion by using the playback server; generating a top level index file describing each asset in the filtered asset list by using a playback server; transmitting the top level index file to the playback apparatus by using the playback server; retrieving cryptographic information by using a playback server, wherein the cryptographic information is used to access a protected stream described in a top-level index file; and providing the password information to the playback apparatus by using the playback server.

A further embodiment of the invention includes a database storing a description of assets associated with a particular content segment and a processor configured using a playback management application. In addition, the playback management application configures the processor to: receiving a request from a playback device identifying a piece of content; retrieving a list of assets associated with the identified content segment; filtering the asset list by using at least one predetermined criterion; generating a top level index file describing each asset in the filtered list of assets; and transmitting the top level index file to the playback device.

Another further embodiment includes a machine-readable medium containing processor instructions, wherein execution of the instructions by a processor causes the processor to perform a process comprising: receiving a request from a playback device identifying a piece of content; retrieving a list of assets associated with the identified content segment; filtering the asset list by using at least one predetermined criterion; generating a top level index file describing each asset in the filtered list of assets; and transmitting the top level index file to the playback device.

Yet another embodiment of the method of the present invention comprises: requesting a top level index file from a playback server by using a playback device, wherein the request identifies a piece of content and includes information describing capabilities of the playback device; receiving a top level index file from a playback server by using a playback device, wherein the top level index file describes a plurality of streams accessible to the playback device and identifies locations of the streams; selecting an initial stream by using a playback device; and retrieving, using the playback device, at least a portion of the initial stream from the location identified in the top level index file.

A further embodiment of the invention includes a memory containing information describing capabilities of a playback device and a processor configured by a client application. In addition, the client application configures the processor to: requesting a top level index file from a playback server, wherein the request identifies a piece of content and includes information describing capabilities of a playback device; receiving a top level index file from a playback server, wherein the top level index file describes a plurality of streams accessible to a playback device and identifies locations of the streams; selecting an initial stream; and retrieving at least a portion of the initial stream from the location identified in the top-level index file.

Another further embodiment includes a machine-readable medium containing processor instructions, wherein execution of the instructions by a processor causes the processor to perform a process comprising: requesting a top level index file from a playback server, wherein the request identifies a piece of content and includes information describing capabilities of a playback device; receiving a top level index file from a playback server, wherein the top level index file describes a plurality of streams accessible to a playback device and identifies locations of the streams; selecting an initial stream; and retrieving at least a portion of the initial stream from the location identified in the top-level index file.

Drawings

Fig. 1 is a network schematic diagram of a streaming system including a playback server according to an embodiment of the present invention.

Fig. 2 is a system diagram of a playback apparatus according to an embodiment of the present invention.

Fig. 3 is a flow diagram illustrating a process for obtaining a top level index file and cryptographic information from a playback server to stream content from a content distribution network according to an embodiment of the present invention.

FIG. 4 is a flow diagram illustrating a process for automatically generating a top level index file according to an embodiment of the present invention.

FIG. 5 is a flowchart illustrating a process for generating a list of streams contained in an automatically generated top-level index file according to an embodiment of the present invention.

Fig. 6 is a flowchart illustrating a process for generating an SMIL file containing a top level index file according to an embodiment of the present invention.

Fig. 7 is a flowchart illustrating a process for providing password information to a playback apparatus according to an embodiment of the present invention.

Fig. 8 is a flowchart illustrating a process for processing a play event received from a playback apparatus according to an embodiment of the present invention.

Fig. 9 is a timing diagram illustrating communication between a playback device and a playback server and a content distribution network during a process of adaptive bitrate streaming using an automatically generated top-level index file according to an embodiment of the present invention.

Detailed Description

Turning now to the drawings, systems and methods for automatically generating a top-level index file for use in adaptive bitrate streaming are shown, according to embodiments of the present invention. The top level index file is a file that describes the location and content of a container file containing a stream of media (e.g., audio, video, metadata, and subtitles) that may be used by a playback device to stream and playback the content. In adaptive bitrate streaming systems, the top level index file typically references alternative streams between which playback devices can switch. In many embodiments, the top level index file is generated in response to a request from a playback device. In several embodiments, the process of generating the top level index file involves determining all asset or container files containing streams associated with a particular content segment, and then filtering the assets based on one or more predetermined criteria. In several embodiments, the assets are filtered based on predetermined criteria including, but not limited to, business rules defined by the content owner, device capabilities, and/or information associated with the user account. In many embodiments, the generated top level index file is a SMIL file. In other embodiments, any of a variety of file formats may be used to generate a top level index file according to embodiments of the present invention.

In several embodiments, the playback server automatically generates the top level index file in response to a request from the playback device, and the playback server may also provide cryptographic information to the playback device that enables playback of the protected content stream. In several embodiments, the playback device provides the play event report to the playback server, and the playback server can track the playback position of the stream being played back by the playback device. The playback position may be used to enable a user to switch between viewing content via different playback devices. The user may start playback on a first device and then request playback on a second device. When the playback server receives a playback request from the second playback device, the playback server may pause playback on the first playback device and provide instructions to the second playback device to begin playback at the last playback position reported by the first playback device. The playback apparatus, the playback server, and the process for automatically generating the top-level index file in response to a request from the playback apparatus according to the embodiments of the present invention are further discussed below.

Stream transmission system structure

An adaptive bitrate streaming system according to many embodiments of the present invention includes a playback server that automatically generates a top level index file in response to a request from a playback device. A streaming system according to an embodiment of the invention is shown in fig. 1. The streaming system 10 includes several playback devices 12 configured to request streaming of content from remote servers within a Content Delivery Network (CDN)14 via a network 16, such as the internet. To stream content, the playback device obtains a top level index file that is automatically generated by the playback server 18 using the database 20 of available assets (i.e., container files containing streams of content associated with particular titles) and a set of predetermined filters or criteria.

To perform adaptive bit rate streaming, the playback device 12 selects content from the different alternative streams described in the top level index file. The alternative streams are streams of the same media content that are encoded in different ways. In many cases, the alternative streams encode media content (such as, but not limited to, video) at different maximum bitrates. In several embodiments, the alternate streams are encoded at different resolutions and/or at different frame rates. In many embodiments, the top level index file generated by playback server 18 describes at least the maximum bit rate for each of the alternate streams. The playback device may select one or more streams for conventional streaming or may switch between alternative streams to perform adaptive bit rate streaming.

In several embodiments, the playback device 12 is able to purchase content for streaming via one or more online content stores 22. In many cases, the playback device purchases content via an online content store and receives an electronic token or ticket that the playback device can provide to the playback server 18 to gain access to the purchased content. When the playback server 18 receives the ticket, the playback server automatically generates a top level index file for the playback device using the information stored in the database 20. The playback server is typically a conventional server system configured using a playback management application and in a manner such that the playback server can access the data 20. In several embodiments, the playback device 12 provides information to the playback server 18 regarding its playback capabilities, and the server automatically generates the top level index file by filtering assets associated with the purchased content segments based on device capabilities. Such capabilities include, but are not limited to, playback capability, display aspect ratio, and/or expected maximum network connection data rate. In several embodiments, the capabilities of a particular playback device 12 are communicated to the playback server 18 in the form of a product identifier or product ID. The playback server 18 may maintain a database of product IDs and associated device capabilities and may utilize the product IDs provided by the playback devices to retrieve the associated device capabilities and apply filters to the list of available assets when generating the top level index file. A system and method for assigning and associating Product IDs with information about the capabilities of the device to which the Product ID is assigned is described in U.S. provisional patent application serial No. 61/581,598 entitled "Systems and methods for Identifying Electronic Products Using a Playback device with a Product Identifier" by Grab et al, filed 12, 29/2011. The disclosure of U.S. provisional patent application serial No. 61/581,598 is hereby incorporated by reference in its entirety. Other filters that may be applied by the playback server, according to embodiments of the present invention, include, but are not limited to, filters based on the location (i.e., country/region) of the playback device and the language associated with a particular user. Various filters that may be applied to generate the top-level index file according to embodiments of the present invention will be discussed further below.

When playback server 18 receives the play request, the playback server retrieves the cryptographic information needed to play back the protected content stream. The playback device may then request portions of the container file containing the content stream from one or more servers within content distribution network 14 and may use the cryptographic information to access the protected content. A system and method for digital rights management of Protected Streams used in adaptive bit rate streaming is described in U.S. patent application serial No. 13/340,632 entitled "Systems and Methods for Playing Back alternate Streams of Protected Content Using Common Cryptographic Information" filed on 29.12.2011. The disclosure of U.S. patent application serial No. 13/340,632 is incorporated herein in its entirety by this reference.

During playback, the playback device 12 may provide a play event report to the playback server 18. The play event report may include information including, but not limited to, the current stream from which the playback device is requesting data and the current playback position of the playback device. In many embodiments, the playback server 18 responds to the play event report by instructing the playback device to continue playback or to stop playback. In several embodiments, the playback server enforces concurrency restrictions on the playback device. In the context of an adaptive bitrate streaming system, the term "concurrency limit" describes the number of playback devices that can simultaneously playback content associated with a particular user account. When the concurrency limit is exceeded, the playback server 18 may send an instruction to the playback device 12 instructing it to stop playback. In several embodiments, the playback position information provided in the play event report may be used to enable a user to switch from playing back content on a first device to playing back content on a second device, and the second device will automatically resume content playback at the last playback position of the first playback device. In addition, a user pausing playback on a first playback device may resume playback on the same playback device or a second playback device at some later time.

Although a particular streaming system is shown in fig. 1, a playback server that automatically generates a top level index file may be incorporated into any of a variety of streaming system configurations according to embodiments of the present invention. Playback devices that may be used to perform network bandwidth measurements during a streaming session in accordance with embodiments of the present invention are discussed further below.

Playback apparatus

In the embodiment shown in fig. 1, the playback apparatus includes a personal computer, a CE player, and a mobile phone. In other embodiments, the playback device may include consumer electronics devices such as DVD players, Blue-ray players, televisions, set-top boxes, video game consoles, tablets, and other devices capable of connecting to a server and playing back encoded media. A basic structure of a playback apparatus according to an embodiment of the present invention is shown in fig. 2. The playback device 30 includes a processor 32 in communication with non-volatile memory 34 and volatile memory 36. In the illustrated embodiment, the non-volatile memory includes a client application 38 that configures the processor and playback device to stream content from a remote server. The non-volatile memory 34 also includes device cryptographic data 39 that can be used to access the encrypted content and a product ID 40 that can be used to identify the capabilities of the playback device.

During operation, the client application 38 may configure the playback device to purchase content from a content store and request the top level index file 42 from the playback server. As described above, the request for the top level index file may include the playback device's product ID 40, and the product ID 40 may be used to filter the assets described in the top level index file. Upon receiving the top level index file 42 from the playback server, the client application 38 configures the playback device to store the top level index file 42 in memory 36 and requests content cryptographic information 44 to play back the stream requested from the playback server (or a separate DRM server) in response to user instructions to play back the content. The content cryptographic information 44 received from the playback server may also be stored in the memory 36. Client application 38 may then configure playback device 30 to playback the content by requesting portions of the container file containing the content from a server within the content distribution network. The particular container file selected by the client application 38 may be determined based on information available to the client application within the top-level index file 42. When the playback device receives the requested portion of the container file, the client application 38 may access the protected content within the requested portion of the container file by using the content cryptographic information 44. The client application may continue to request and playback content from different container files depending on prevailing streaming conditions until a user instruction to pause playback is received or until playback is completed.

Although a particular playback mechanism structure is shown in fig. 2, any of a variety of structures, including structures in which a client application resides on a disk or some other form of storage device and is loaded into volatile memory at runtime, may be used to implement a playback mechanism for use in an adaptive bitrate streaming system according to embodiments of the present invention. Also, any of various system architectures including, but not limited to, the system architecture shown in fig. 1 may be used to perform adaptive bitrate streaming by using a top-level index file that is automatically generated in response to a request from a playback apparatus according to an embodiment of the present invention.

Content playback

A process for playing back content by using a top-level index file generated in response to a request from a playback apparatus according to an embodiment of the present invention is illustrated in fig. 3. The process 50 includes the purchase of content (52). The purchase may be performed by using the playback device or may be performed by using another device. In many embodiments, the purchase is made via an online content store that provides electronic tickets or tokens that can be used to gain access to the content. To access content, a playback device requests generation of a top-level index file customized for a particular playback device from a playback server. In many embodiments, the top level index file is customized in the sense that the playback server filters all available assets to produce a top level index file that describes the assets that are available to and/or best suited for playback by the playback device. Once the customized top level index file is retrieved (54), the playback device obtains (56) cryptographic information that enables playback of the protected stream identified within the top level index file. In many embodiments, the playback device waits until a playback request is received from the user and then password information is requested from the playback server.

When the playback device has the top-level index file and any necessary cryptographic information, the playback device may begin streaming (58) the content by requesting portions of one or more container files containing encoded media. Any of a variety of processes may be used to select the initial stream, including (but not limited to): selecting the lowest bit rate Stream, Performing Bandwidth probing in a manner similar to that disclosed in U.S. patent application serial No. 13/251,061 entitled "Systems and Methods for determining available Bandwidth and performance Stream Selection When Using high performance Transfer Protocol," filed on van der Schaar et al, 30/9.2011, or estimating the available Bandwidth by Using another technique. The disclosure of U.S. patent application serial No. 13/251,061 is hereby incorporated by reference in its entirety.

Once playback begins (60), adaptive bit rate streaming may be performed. During playback, the play event report may be provided to the playback server periodically or in response to various events including, but not limited to, user instructions received by the playback device to provide information regarding the playback state of the playback device. When the play event report is provided to the playback server, the playback server may provide a play event instruction to the playback device. In the illustrated embodiment, the play event instructions that the playback device may receive (62) include a "play" instruction to continue playback and a "stop" instruction to stop playback. As discussed further below, the ability of the playback server to stop playback on the playback devices enables the playback devices to enforce concurrency requirements on multiple playback devices associated with a user. The playback server may also utilize a combination of play event reports and event playback instructions to enable the second playback device to resume playback of the content at the last playback position reported by the first playback device. In this way, the user can seamlessly move between playback devices without having to find a location to continue playback.

Although a specific process is illustrated in fig. 3, any of various processes may be used to request a customized top-level index file and playback content according to embodiments of the present invention. The playback server and process for automatically generating index files according to embodiments of the present invention are discussed further below.

Automatic generation of top level indices

A playback server according to embodiments of the present invention maintains a database of assets associated with particular content segments and is configurable by a playback management application to retrieve and filter information about the assets to automatically generate a top level index file for a particular playback device. A database maintained by the playback server may list the names of the files containing the assets and the content distribution network in which the assets are located. When the asset is a video stream, the database may describe characteristics of the video stream including, but not limited to, the language associated with the video stream, the maximum bit rate at which the video stream is encoded, the frame rate, resolution, and/or sample aspect ratio of the video stream. When the asset is an audio stream, the database may describe characteristics of the audio stream including, but not limited to, the language of the audio stream, the encoding of the audio stream, and the bandwidth requirements of the audio stream. When the asset is a subtitle stream, the database may describe characteristics of the subtitle stream including, but not limited to, the language of the subtitle stream, the encoding of the subtitle stream, and the bandwidth requirements of the subtitle stream. In addition to maintaining information about the asset, the playback device may also maintain information about the playback capabilities of the playback device. In several embodiments, information regarding playback capabilities of different playback devices is indexed using product IDs. In many embodiments, the playback device also maintains information regarding rules governing the use of different assets based on criteria including, but not limited to, the type of playback device, the playback capabilities of the playback device, the location (e.g., country/region) of the playback device, and/or the language of the request. Although particular pieces of information are described above as being associated in a database maintained by a playback server, any of a variety of pieces of information may be maintained in any of a variety of data structures to enable generation of a top level index file by a playback server in accordance with embodiments of the present invention. As discussed further below, the information maintained by the playback server may be filtered in real-time in response to a request from the playback device for the top level index file to produce a top level index file that includes only links to assets that the playback device is allowed to play back.

A process for generating a top level index file by using information about assets associated with a particular content segment maintained by a playback server according to an embodiment of the present invention is illustrated in fig. 4. In the illustrated embodiment, the process 70 begins when the playback server receives (72) a request from a particular playback device for a top level index file for a particular piece of content. As discussed above, the request may include information describing the playback device, including, but not limited to, the playback capabilities of the playback device, the user account to which the playback device is registered, and information indicating the geographic location of the playback device (e.g., the IP address of the playback device). In many embodiments, the capabilities of the playback device are identified using a product ID associated with a particular playback capability in a database accessible to the playback server.

The playback server retrieves (74) the asset associated with the requested content segment. The playback server filters (76) the assets based on one or more filters associated with capabilities of the playback device, preferences of the user, and requirements of the content owner. Thus, different top-level index files may be generated with respect to the same content depending on factors including, but not limited to, differences in playback capabilities between devices, differences in geographic location, and/or differences in language preferences associated with playback devices. The process for retrieving and filtering assets associated with a particular content segment according to embodiments of the present invention will be discussed further below. After filtering, the assets that remain may be used to generate (78) a top level index file, which may be provided (80) to a playback device. In several embodiments, the top-level index file is securely forwarded using Secure communication techniques including, but not limited to, Secure Socket Layer (SSL) communication or an equivalent Secure communication protocol.

Although a specific process for generating a top level index file according to an embodiment of the present invention is illustrated in fig. 4, any of various processes involving collecting information about assets associated with a specific content segment and generating an index file that indexes assets to be made available to playback apparatuses through an adaptive bitrate streaming system may be utilized according to an embodiment of the present invention. The following discusses a particular process for filtering assets included in a top level index file according to an embodiment of the present invention.

Filtering assets for inclusion in a top-level index file

Typically, more assets or streams are created than are needed for adaptive bit rate streaming of content by a playback device. In many cases, video streams are created for different categories of playback devices. For example, a lower bit rate stream may be created for a device that will stream content over a cellular data network, and a higher bit rate stream may be created for a device that will stream content over a home network connected to the internet via a high speed internet connection. In addition, different video streams may be created with different aspect ratios, and different audio streams may be created for different languages. In several embodiments, the content store provides premium high-resolution content, and playback devices that have purchased only low-resolution content are not allowed to access the high-resolution content. In many embodiments, a particular flow may not be accessible in certain geographic areas. When the top level index file is created in real-time in response to a request from a particular playback device, the playback server that created the top level index file may first collect information about assets associated with a particular content segment or title. The playback server may then apply one or more filters to the list of available assets to generate a list of assets that satisfy criteria including, but not limited to, criteria regarding the capabilities of the playback device, the preferences of the user, and/or the requirements of the content owner.

A process for generating a list of assets to which a playback device is authorized to access in accordance with an embodiment of the present invention is shown in fig. 5. Process 90 includes retrieving video (92), audio (94), and/or subtitle or sub-text (96) assets. In many embodiments, processing may also include retrieving other types of assets including, but not limited to, metadata assets and/or trick play stream (trick play) assets. When the assets associated with the content segments have been retrieved, the playback server may filter (98) the assets to exclude assets that cannot be played back by the playback device, that are not allowed to be played back, or that are not desired to be played back.

The information about the assets associated with the content collected by the playback server may include information about assets that are capable of being played back by both the playback device and the content owner. However, the information may also include information about assets that the playback device is not allowed to play back or assets that the user does not wish to play back. Assets that may not allow playback may include, but are not limited to, premium content assets that have not yet been purchased (e.g., high definition video streams) or assets that do not allow playback within a particular geographic location or region, assets that playback devices cannot play back include, but are not limited to, assets that playback devices cannot safely play back (e.g., playback devices have unencrypted digital output and content providers are reluctant to allow high definition content to be streamed to such devices) or streams encoded in an unsupported format or using a profile and/or buffer size that is not supported by the playback device. Assets that a user may not wish to play back include, but are not limited to, content in a language unrelated to the particular user, video assets having an aspect ratio different from the display aspect ratio of the playback device (where the video assets are associated with content having the same aspect ratio as the display aspect ratio of the device), content encoded at a resolution exceeding the resolution of the display of the playback device, and/or content encoded at a maximum bit rate exceeding the maximum data rate of the network connection of the playback device. Although specific filters are discussed above, the criteria used by the playback server to filter (98) the initial list of assets to generate a list for the assets included in the top level index file are limited only by the requirements of the particular adaptive bitrate streaming system.

Once the filtering is applied to the asset list, the filtered asset list may be returned (100) and used to generate the top level index file. In the illustrated embodiment, the filtered list of assets corresponds to content streams that the playback server allows the playback device to access.

Although a particular process is shown in FIG. 5, any of a variety of processes may be used to retrieve and filter assets to be included in the top level index file according to the requirements of a particular application, in accordance with embodiments of the present invention.

Generating top-level index SMIL files

In several embodiments, the top level index file generated by the playback server in response to a request from the playback device is a SMIL file, which is an XML file that includes a list of URIs that describe each of the streams and a container file that contains the streams. The URI may include information such as the "system bit rate" of the stream contained within the stream and information about the location of a particular data segment within the container file.

The basic structure of a SMIL file involves providing XML declarations and SMIL elements. The SMIL element defines a stream or asset available to a playback device and includes a HEAD element that is normally left empty and a BODY element that typically contains only PAR (parallel) elements. The PAR element describes streams that can be played simultaneously (i.e., includes media that can be presented simultaneously).

The SMIL specification defines several sub-elements for PAR elements that can be used to specify streams that can be used in adaptive bit rate streaming. The VIDEO, AUDIO, and textstart elements may be used to define a particular VIDEO, AUDIO, or subtitle stream. The VIDEO, AUDIO, and TEXTSTAREAM elements may be collectively referred to as a media object. The basic attributes of a media object are the SRC attribute specifying the full path or URI to the container file containing the associated stream, and the XML: LANG attribute containing the 3-letter language code. Additional information about the media object may be specified using the PARAM element. The PARAM element is a standard way within the SMIL format to provide a generic name-value pair. In several embodiments of the present invention, specific PARAM elements are defined for use during adaptive bit rate streaming.

In many embodiments, a "header-request" PARAM element is defined that specifies the size of the header of the container file containing the stream. The value of the "header-request" PARAM element typically specifies the number of bytes between the start of the file and the start of the encoded media within the file. In many embodiments, the header contains information about the manner in which the media was encoded, and, to enable configuration of a decoder for playback of the encoded media, the playback device retrieves the header prior to playback of the encoded media. Examples of "header-request" PARAM elements are as follows:

<param

name＝″header-request″

value＝″1026″

valuetype＝″data″/>

in several embodiments, a "MIME" PARAM element is defined that specifies the MIME type of the stream. The "mime" PARAM elements that identify a stream as an H.264 stream (i.e., a stream encoded according to the MPEG-4 Advanced Video Codec) standard) are as follows:

<param

name＝″mime″

value＝″V_MPEG4/ISO/AVC″

valuetype＝″data″/>

the MIME type of a stream can be specified by using the "MIME" PARAM element according to the encoding of a particular stream (e.g., AAC audio or UTF-8 text stream).

When the media object is a VIDEO element, additional attributes may be defined within the SMIL file format specification, including a systembitray attribute specifying the bit rate of the stream in the container file identified by the VIDEO element and width and height attributes specifying the size of the encoded VIDEO in pixels. Additional attributes may also be defined using the PARAM element. In several embodiments, a "VBV" PARAM element is defined that specifies the VBV buffer size of the video stream in bytes. The Video Buffer Verifier (VBV) is a theoretical model of an MPEG video buffer used to ensure that an encoded video stream can be correctly buffered and played back at a decoder device. An example of a "VBV" PARAM element specifying a VBV size of 1000 bytes is as follows:

<param

name＝″vbv″

value＝″1000″

valuetype＝″data″/>

examples of VIDEO elements that include the attributes discussed above are as follows:

an adaptive bitrate streaming system according to embodiments of the present invention can support trick-play streams that can be used to provide a smooth visual search through the source content encoded for adaptive bitrate streaming. The trick-play stream may be encoded to appear to accelerate visual searches through the source media when played back, whereas in practice the trick-play stream is merely a single track encoding the source media at a lower frame rate. In many embodiments of the system, the VIDEO element referencing the trick play stream is indicated by the systembabsoprofile attribute of the VIDEO element. In other embodiments, any of a variety of techniques may be used to indicate within the top level index file that a particular stream is a trick play stream. Examples of trick play stream VIDEO elements according to embodiments of the present invention are as follows:

in several embodiments of the present invention, a "reservedBandwidth" PARAM element may be defined for the AUDIO element. The "reservedBandth" PARAM element specifies the bit rate of the audio stream in Kbps. Examples of the AUDIO element specified according to the embodiment of the present invention are as follows:

in several embodiments, a "reservedBandwidth" PARAM element is also defined for the TEXTSTREAM element. An example of a TEXTSTREAM element that includes a "reservedBandwidth" PARAM element according to an embodiment of the present invention is as follows:

in other embodiments, any of a variety of mechanisms may be used to specify information about the VIDEO, AUDIO, and SUBTITLE elements depending on the particular application.

The SWITCH element is a mechanism defined in the SMIL file format specification that can be used to define adaptive or alternative streams. An example of the manner in which the SWITCH element can be used to specify alternative video streams at different bit rates is as follows:

the SWITCH element specifies the URLs of the three alternative video streams. The file name indicates a different bit rate for each of the streams. As discussed further below, the SMIL file format specification provides a mechanism by which additional information about a stream and the container file in which it is contained may be specified within a top-level indexing SMIL file in embodiments according to the invention.

In many embodiments of the invention, EXCL (exclusive) elements are used to define alternative tracks that do not adapt to streaming conditions during playback. For example, the EXCL element may be used to define an alternative audio track or an alternative subtitle track. An example of the manner in which the EXCL elements may be used to specify alternative English and French audio streams is as follows:

an example of a top-level index SMIL file defining the attributes and parameters of two alternative video levels, audio streams and subtitle streams according to an embodiment of the present invention is as follows:

when the playback device receives the top-level indexed SMIL file, the playback device may parse the SMIL file to identify available streams. The playback device may then select the initial stream and may use the SMIL file to identify the portion of the container file to download to obtain information about the encoding of the particular stream and/or to obtain an index to the encoded media within the container file.

A process for building a top-level index SMIL file according to an embodiment of the present invention is shown in fig. 6. Process 110 involves generating (112) a header for the SMIL file and generating XML strings for video (114), audio (116), subtitles/sub-text (118), and/or any other asset (e.g., trick play stream, metadata stream) that is in the list of assets that the playback server is allowing the playback device to access. At this point, a final SMIL file is generated (120).

an assetID is assigned to the audio and text streams to uniquely identify the streams. This value is passed up to the server, so the server knows the exact audio, subtitle stream being played.

<param

name＝″assetID″

value＝″123456″

valuetype＝″data″/>

Although a top-level index SMIL file is described above, any of a variety of top-level index file formats may be used to automatically create a top-level index file for a particular playback device according to a particular application, according to embodiments of the present invention. The use of a top level index file to enable playback of media encoded using adaptive bitrate streaming in accordance with embodiments of the present invention is discussed further below.

Protected stream

Some or all of the assets associated with a particular piece of content may be encrypted or otherwise technically protected. A Digital Rights Management (DRM) system enables cryptographic information to be transferred to a playback device so that the playback device can access the protected stream in the clear (i.e., in unencrypted form). A DRM system that may be used in an adaptive bitrate streaming system according to an embodiment of the invention is described in U.S. patent application serial No. 13/340,632. In many embodiments, the playback server acts as a DRM server. In several embodiments, the playback server is configured to obtain the cryptographic information from the playback server.

A process for providing cryptographic information to a playback device according to an embodiment of the present invention is shown in fig. 7. The process 130 includes checking (132) the concurrency rules to confirm that the number of devices on which the user is currently playing back content and/or has requested rights to play back content is less than a predetermined concurrency limit. Assuming the concurrency rules are satisfied, the process retrieves (134) cryptographic information that enables access to a protected stream associated with the particular piece of content being accessed by the playback device. The playback server may generate the cryptographic information or obtain the cryptographic information from the DRM server. The cryptographic information is then provided (136) to the playback device.

In many embodiments, the playback device includes cryptographic information stored locally on the playback device. The device password information is typically issued to the playback device when the playback device first registers with the playback server and/or DRM server. In several embodiments, the content password information provided to the playback device is encrypted so that it can be accessed using the device password information of the playback device. In other embodiments, any of a variety of different techniques may be used to securely provide cryptographic information for accessing the protected stream to the playback device.

Although a specific process for providing password information to a playback apparatus is shown in fig. 7, any of various processes may be used according to an embodiment of the present invention. In addition, the playback device may obtain the cryptographic information from the DRM server in a process that is completely independent of the playback server and/or the automatic generation of the top-level index file. Playback may begin once the playback device has the cryptographic information necessary to play back the protected content stream. During playback, the playback device may provide periodic play event reports to the playback server. The generation and processing of play event reports according to embodiments of the present invention are discussed further below.

Playing event reports

Playback devices according to embodiments of the present invention may generate play event reports periodically or in response to particular events such as, but not limited to, receiving user instructions (e.g., play, stop, pause commands). The information that may be included in the event report typically includes the stream being accessed by the playback device and the current playback position in one or more of the streams. In several embodiments, the event report includes, but is not limited to, time, audio, and/or caption assetID, and/or bandwidth measurements. The playback position may be used to resume playback of the content. The server may utilize the play event to perform a wide variety of functions including, but not limited to, enabling playback to transition from one device to another. When a user begins playing content on a first device and then begins playing the same content on a second device, the playback server may provide the second playback device with the current playback position of the most recent play event report from the first device. The second playback device may then use the playback position information to resume playback of the content. In this manner, the user may seamlessly transition from viewing content on the first playback device to viewing content on the second playback device.

A process for processing a play event report according to an embodiment of the present invention is shown in fig. 8. The process 150 includes receiving (152) a play event report and determining an appropriate response. In embodiments where concurrency limits exist, the response to the play event report may be to instruct the playback device to "stop" content playback in response to a request for playback content on another device that exceeds the predetermined concurrency limit. In the case where the content is rented, the playback may exceed the rental period, and the playback server may instruct the playback device to "stop" the playback. The playback server may also instruct the playback device to request content from the alternate stream or switch to accessing content on a different content distribution network, which may involve transferring a new top-level index file. In several embodiments, the playback server may instruct the playback device to display an error message or string. However, in general, the response is to instruct the playback apparatus to continue playing the content. The particular conditions under which the playback server will pause playback or allow playback to continue are determined based on the requirements of the particular application. The response is sent (156) to the playback device, which manages playback accordingly.

Although a specific process for processing the broadcast event report is illustrated in fig. 8, any of various processes for processing the broadcast event report according to the requirements of a specific application may be used according to an embodiment of the present invention.

Communication during adaptive streaming

The above describes a process for automatically generating a top-level index file in response to a request from a playback apparatus to access content. A timing diagram illustrating the communication between the playback device, the playback server and the content distribution network during adaptive bitrate streaming of content according to an embodiment of the invention is shown in fig. 9. The timing diagram shows the playback device 12 requesting an index file from the playback server 18. The playback server generates a top-level index file to respond to the request by using a process similar to that outlined above. The playback server 18 provides the top level index file to the playback device 12, and the playback device 12 then sends a playback request to the playback server. The playback server checks the concurrency rules and retrieves the cryptographic information that enables playback of the protected stream. The cryptographic information is provided to the playback device 12. The playback device may then use the top level index file to request content from the content distribution network 14. In many embodiments, the playback device utilizes adaptive bitrate streaming to obtain content from the content distribution network server(s). The playback device 12 may utilize the cryptographic information received from the playback server 18 to access the protected content stream and playback the content in the clear. During playback, the playback device 12 sends a playback event report to the playback server. The playback server 18 processes the play event report and will send a response to the playback device 12, which may cause the playback device to perform any of a variety of functions including, but not limited to, pausing playback of the content.

Although a particular order of communication is shown in fig. 9, according to embodiments of the present invention, the playback device, the playback server, and the content distribution network may communicate in any of a variety of ways depending on the particular application.

Although the present invention has been described in certain specific aspects, many additional modifications and variations will be apparent to those of ordinary skill in the art. It is therefore to be understood that the invention may be practiced otherwise than as specifically described, including as various modifications in implementation, such as use of encoders and decoders which support features beyond those specified in the particular standard to which they are compliant, without departing from the scope and spirit of the invention. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims (29)

1. A method of generating a top level index file, comprising:

receiving, at a playback server system, a request from a playback device, wherein the request (i) identifies a content segment and (ii) includes a product identifier;

retrieving, using a playback server system, (i) a list of assets associated with the identified content segment and (ii) at least one device capability based on the product identifier, wherein each asset is a different stream associated with the content segment;

filtering the asset list using the at least one device capability using a playback server system, wherein the playback server system maintains a database of product identifiers and associated device capabilities;

generating a top level index file describing each asset in the filtered list of assets by using the playback server system; and

the top level index file is sent to the playback device using the playback server system, where the top level index file is used by the playback device to determine which assets to request for playback on the device.

2. The method of claim 1, further comprising filtering the asset list based on at least one of: a geographic location of the playback device, a language associated with the playback device, one or more user preferences, and one or more requirements of the content owner.

3. The method of claim 1, wherein the at least one device capability is at least one of: display aspect ratio, expected maximum network connection data rate, device output, supported formats, device buffer size, device resolution, device region, and device language.

4. The method of claim 1, wherein the playback server system maintains a database of assets associated with particular content segments.

5. The method of claim 1, wherein the top level index file describes at least a bit rate of each asset in the filtered list of assets and identifies a location of the asset in the filtered list of assets.

6. The method of claim 5, wherein the top level index file is a SMIL file.

7. The method of claim 6, further comprising generating an XML string comprising SWITCH elements describing alternative streams for adaptive bit rate streaming.

8. The method of claim 6, further comprising generating an XML string comprising EXCL elements describing alternative streams for regular streaming.

9. The method of claim 6, further comprising generating an XML string that includes the URI for each asset, wherein the URI references the container file and the XML string for each asset includes an element that defines a size of a header of the container file.

10. The method of claim 6, wherein the XML string includes an encoded element that identifies the asset.

11. The method of claim 6, wherein the XML string of the video asset comprises at least one element selected from the group consisting of:

an element describing a maximum bit rate of the video;

elements describing the width and height of the video; and

an element describing the video buffer verifier size of the video.

12. The method of claim 1, wherein each asset is a different alternative stream associated with a content segment, and each alternative stream encodes the content segment at a different maximum bitrate.

13. A playback server system comprising:

a database storing a description of assets associated with a particular content segment;

a database storing a plurality of product identifiers and associated device capabilities;

a processor configured by using a playback management application;

wherein the playback management application configures the processor to:

receiving a request from a playback device, wherein the request (i) identifies a content segment and (ii) includes a product identifier;

retrieving (i) a list of assets associated with the identified content segment and (ii) at least one device capability based on the product identifier, wherein each asset is a different stream associated with the content segment;

filtering a list of assets using the at least one device capability;

generating a top level index file describing each asset in the filtered list of assets; and

the top level index file is sent to the playback device, where the top level index file is used by the playback device to determine which assets to request for playback on the device.

14. The playback server system of claim 13, wherein the playback management application further configures the processor to filter the asset list based on at least one of: a geographic location of the playback device, a language associated with the playback device, one or more user preferences, and one or more requirements of the content owner.

15. The playback server system of claim 13, wherein the at least one device capability is at least one of: display aspect ratio, expected maximum network connection data rate, device output, supported formats, device buffer size, device resolution, device region, and device language.

16. The playback server system of claim 13, wherein the top level index file describes at least a bit rate of each asset in the filtered list of assets and identifies a location of the asset in the filtered list of assets.

17. The playback server system of claim 16, wherein the top level index file is a SMIL file that is an XML file that includes a list of URIs that describe each of the different streams associated with the content segments and a container file that contains the streams.

18. The playback server system of claim 13, wherein each asset is a different alternative stream associated with a content segment, and each alternative stream encodes the content segment at a different maximum bitrate.

19. A method for authorizing playback of content, comprising:

receiving, at a playback server, a request for content from a playback device, wherein the request includes a product identifier;

identifying a plurality of device capabilities based on the product identifier, the plurality of device capabilities including a device type and a device software version indicating a version number of an adaptive streaming software component implemented on the device;

retrieving a list of assets associated with the identified content segment and filtering the list of assets based on the plurality of device capabilities, wherein each asset is a different stream associated with the content segment;

generating a top level index file describing each asset in the filtered list of assets; and

the top level index file is sent to the playback device.

20. The method of claim 19, wherein the content request from the playback device further comprises information about a web browser on the playback device being used to request the content.

21. The method of claim 19, wherein the content request from the playback device further comprises information related to an operating system of the playback device.

22. The method of claim 19, wherein the request is made over a network via hypertext transfer protocol (HTTP).

23. The method of claim 19, wherein the request including information includes (i) information describing a type of playback device and (ii) information related to network bandwidth.

24. The method of claim 19, further comprising filtering the asset list based on a version of a web browser on a playback device requesting the content.

25. The method of claim 19, wherein the top level index file is a SMIL file.

26. A playback apparatus comprising:

a memory containing information identifying capabilities of the playback device; and

a processor configured by a client application;

wherein the client application configuration processor performs the following operations:

requesting a top level index file from a playback server, wherein the request identifies a content segment and includes a software version indicating a version number of an adaptive streaming software component implemented on a device;

receiving a top level index file from the playback server, wherein the top level index file identifies locations of a plurality of different alternative streams associated with the identified content segment and accessible by the playback device;

selecting an initial stream from the plurality of different alternative streams; and

at least a portion of the initial stream is retrieved from a location identified in the top-level index file.

27. The playback device of claim 26, wherein the top level index file describes at least a bitrate for each of the plurality of different alternative streams associated with the identified content segment and accessible by the playback device.

28. The playback device of claim 26, wherein at least one of the capabilities of the playback device is selected from the group consisting of: display aspect ratio, expected maximum network connection data rate, device output, supported formats, device buffer size, device resolution, device region, and device language.

29. A method for streaming a content segment at a playback device, the method comprising:

requesting a top level index file from a playback server via a hypertext transfer protocol (HTTP) over a network by using a playback device, wherein the request identifies a piece of content and includes (i) information describing a type of the playback device and (ii) information related to network bandwidth;

receiving, by using the playback device, a top level index file from the playback server, wherein the top level index file comprises an index to a plurality of different alternative streams associated with the identified content segment, wherein each alternative stream encodes the content segment at a different bitrate, wherein the top level index file provides at least a Uniform Resource Identifier (URI) and a maximum bitrate for each alternative stream associated with and accessible by the content segment based on a type of the playback device and information related to network bandwidth;

selecting an initial stream from the plurality of different alternative streams provided in the top level index file by using a playback apparatus; and

requesting, using a playback device, at least a portion of the initial stream from a URI location identified in the top level index file using an HTTP byte range request corresponding to a container file containing encoded video for the portion of the initial stream; and

a requested portion of the initial stream is received for playback by a playback device.