WO2013091010A1

WO2013091010A1 - Media output methods and devices

Info

Publication number: WO2013091010A1
Application number: PCT/AU2012/001578
Authority: WO
Inventors: Davin MCCALL; Matthew Wesley KELLER
Original assignee: SAMSARA TECHNOLOGY Pty Ltd
Current assignee: SAMSARA TECHNOLOGY Pty Ltd
Priority date: 2011-12-21
Filing date: 2012-12-21
Publication date: 2013-06-27
Anticipated expiration: 2014-06-21

Abstract

A method of outputting a media data sample at a media output device, comprising: receiving from a media capture device a media data sample and a timestamp associated with the received media data sample; associating the received media data sample with an adjusted timestamp; and outputting the received media data sample according to the adjusted timestamp associated with the received media data sample. A method of outputting a media data sample at a media output device, comprising: receiving a media data sample from a media capture device; upon the size of the received media data sample being less than a predetermined media data sample size, generating a padded media data sample comprising padding data and the received media data sample; and outputting the padded media data sample.

Description

MEDIA OUTPUT METHODS AND DEVICES

Related Application

This application claims the benefit of the priority of Australian provisional application no. 201 1905362, the content of which as filed is incorporated by reference in its entirety.

Field

The invention relates to methods and device for outputting a media data sample. Background

Systems for broadcasting live events (such as live music concerts or live sports games) are known. In such systems, a media capture device (that is, a video and/or audio capture device such as a video camera, microphone etc) is used to capture the live event and the video and/or audio of the live event captured by the device is transmitted to a media output device while the live event is occurring. The transmission can be an analogue transmission or a digital transmission. Examples of media output devices that can receive digital transmissions include digital televisions, computers, smart phones etc.

With digital transmissions, video and/or audio of the live event captured by the media capture device is encoded into a digital stream comprising a plurality of data samples before the digital stream is transmitted to the media output device. At the media output device, the received data samples are decoded to output or play the video and/or audio of the live event captured by the media capture device.

The H.264 video encoding and compression standard describes a video encoding and compression technique where each video data sample of a digital video stream is associated with a timestamp. The timestamp indicates the time at which the video data sample should be output or played at the media output device relative to other video data samples of the digital video stream. Thus, the output or playback rate of the digital video stream is determined by timestamps associated with the video data samples of the digital video stream.

In most systems, video data samples and timestamps of a received digital video stream are stored in a buffer before the video data samples are output or played at the media output device. The video data samples are output or played only when the buffer reaches a certain size. Thus, there is usually a delay or a time lag between the video output by the media output device and the actual live event.

There is a need for an improved or alternative technique for outputting a media data sample. Summary of the Invention

In a first aspect, the invention provides a method of outputting a media data sample at a media output device, comprising:

receiving from a media capture device a media data sample and a timestamp

associated with the received media data sample;

associating the received media data sample with an adjusted timestamp; and outputting the received media data sample according to the adjusted timestamp associated with the received media data sample.

In a second aspect, the invention provides a method of outputting a media data sample at a media output device, comprising:

receiving a media data sample from a media capture device;

upon the size of the received media data sample being less than a predetermined media data sample size, generating a padded media data sample comprising padding data and the received media data sample; and

outputting the padded media data sample.

In a third aspect, the invention provides a media output device for outputting a media data sample, the media output device comprising:

a media receiver arranged to receive from a media capture device a media data sample and a timestamp associated with the received media data sample; a timestamp adjuster arranged to associate the received media data sample with an adjusted timestamp; and

a media player arranged to output the received media data sample according to the adjusted timestamp associated with the received media data sample.

In a fourth aspect, the invention provides a media output device for outputting a media data sample, the media output device comprising:

a media receiver arranged to receive a media data sample from a media capture device;

a padded media generator arranged to, upon the size of the received media data sample being less than a predetermined media data sample size, generate a padded media data sample comprising padding data and the received media data sample; and

a media player arranged to output the padded media data sample.

Brief Description of the Drawings

An embodiment of the invention will now be described by way of non-limiting examples in conjunction with the following drawings, in which:

Figure 1 is a block diagram of a live event streaming system comprising a plurality of media capture devices and a plurality of media output devices; Figure 2 is a functional block diagram showing the components of a media output device;

Figure 3 is a functional block diagram showing the components of a media modifier of the media output device;

Figure 4 is a diagram of a media stream and a modified media stream;

Figure 5 is a functional block diagram showing the components of a media player of the media output device;

Figure 6 is a flowchart of a method of an embodiment;

Figures 7 A and 7B are flowcharts showing how a timestamp adjustment is determined by the timestamp adjustment generator;

Figure 8 is a flowchart showing how a padded media data sample is generated by a padded media data generator;

Figure 9 is a flowchart showing how a received or padded media data sample is output by the media player; and

Figure 10 is a diagram of a padded video data sample.

Detailed Description

Referring to the drawings, there is shown an example of a media output device 30 that can be used in a live event streaming system 300 for streaming a live event such as a live music concert or a live sports game.

Figure 1 is a diagram of the live event streaming system 300. The system 300 is arranged to provide users with a plurality of video streams of a live event. Persons skilled in the art will appreciate that in other examples, the live event system 300 may be arranged to provide other forms of media streams including an audio stream of a live event, or a video and audio stream of a live event. The system 300 comprises a plurality of media capture devices 10. Each media capture device 10 is arranged to generate a video stream of the live event. The media capture devices 10 are spread around the live event venue to capture the live event from different geographic positions. Each of the media capture devices 10 is a video camera. Persons skilled in the art will appreciate that media capture devices other than IP video cameras may be used in other examples, and that such other media capture devices include devices that are capable of generating audio streams of the live event and devices that are capable of generating combined video and audio streams of the live event. Examples of such media capture devices include microphones, Internet Protocol (IP) web cameras etc. The video streams generated by the media capture devices 10 are encoded and compressed according to the H.264 video encoding and compression standard (see "Information technology— Coding of audio-visual objects— Part 10: Advanced Video Coding" ISO/IEC 14496-10:2005 published by International Organization for Standardization (ISO)). Persons skilled in the art will appreciate that the media capture devices 10 of other examples may generate media streams that are encoded and/or compressed using other encoding and/or compression techniques, or that are not encoded and/or compressed.

The media capture devices 10 are connected to a media server 50 which in turn is connected to a network 20. The media server 50 is arranged to receive and aggregate the video streams captured by the media capture devices 10. In this example, the video streams are transmitted from the media capture devices 10 to the media server 50 using the Real-time Transport Protocol (RTP) and the Real-time Streaming Protocol (RTSP) protocols. However, it is envisaged that in other examples, the video streams may be transmitted using other transport protocols. In this example, the media server 50 is arranged to multicast the video streams via the network 20. Persons skilled in the art will appreciate that in other examples, the video streams may be distributed in other ways, for example, by broadcasting or uni-casting the video streams from the media server 50 or directly from the media capture devices 10.

The system 300 also comprises a plurality of media output devices 30. Each of the media output devices 30 is arranged to output one or more of the video streams of the live event distributed from the media server 50 via the network 20. Examples of media output devices 30 include a smart phone, a tablet computer, a laptop computer, a MP3 player etc. Persons skilled in the art will appreciate that a media output device may be any device that is able to output or play a video stream of the live event.

Data is communicated between the media capture devices 10 and the media data output devices 30 via the network 20. As indicated above, each one of the media output devices 30 can receive one or more of the video streams captured by the media capture devices 10 via a multicast of the video stream or video streams from the media server 50 via the network 20. Persons skilled in the art will appreciate that the network 20 may comprise a number of devices (for example, wireless access points, modems, routers etc) to receive the video streams of the live event captured by the media capture devices 10 and to forward the video streams of the live event to the media output devices 30.

In this example, data is transmitted from the media server 50 to the media output devices 30 via one of the IEEE 802.1 1 standards. However, other communication protocols (such as Bluetooth, UMTS etc) may be used in other examples to transfer data from the media server 50 to the media output devices 30. Persons skilled in the art will also appreciate that the media capture devices 10 and the media output devices 30 may be connected to the network via different transmission mediums such as optical fibre, radio etc.

Figure 2 is a block diagram showing the functional components of a media output device 30 in the form of a smart phone arranged to implement a software application arranged to output a video stream derived from the video stream received from the network 20. In this example, the software application is implemented via Application Programming Interfaces (APIs) provided by the operating system (such as Apple iOS or Google Android) running on a processor in the smart phone. However, persons skilled in the art will appreciate that in other examples, the software application could be implemented in an alternative manner, for example, as a dedicated circuit.

As indicated above, the media output device 30 is arranged to receive a multicast of a video stream captured by a media capture device 10 that is formatted according to the H.264 video encoding and compression standard. The video stream comprises a plurality of video data samples and a plurality of timestamps corresponding to respective ones of the video samples. In Figure 2, the video data samples is denoted by VS^ TS^ VS₂, TS₂; VS₃, TS₃... where VS^ VS₂ and VS₃ represent the first, second and third video data samples respectively, and TS^ TS₂ and TS₃ represent timestamps that are respectively associated with the first, second and third video data samples.

The software application comprises a number of modules. One of the modules is a media modifier 80. The media modifier 80 is arranged to generate a modified video stream based on the received video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃... In use, the media modifier 80 receives the video stream VS^ TSi ; VS₂, TS₂; VS₃, TS₃... and outputs a modified video stream derived from the video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃... in response to receiving the video stream. In this example, the modified video stream is output as a MPEG Transport Stream (MPEG-TS) comprising video data samples formatted according to the H.264 video encoding and compression standard.

Another one of the modules is a media player 90. The media player 90 is arranged to receive the modified video stream from the media modifier 80, and to output the modified video stream. In addition, the media player 90 is arranged to output a notification when the media player 90 is outputting the received video stream. In use, the media player 90 outputs the modified video stream in response to receiving the modified video stream from the media modifier 80. The media player 90 outputs the modified video stream by displaying the H.264 video data samples of the modified video stream on a display. In this example, the display is part of the smart phone 30. However, it is envisaged that the display may be a separate device from the smart phone 30.

The media output device 30 also comprises a user interface 70. The user interface 70 is arranged to control the media player 90 to receive the modified video stream derived from the received video stream VS^ TSi ; VS₂, TS₂; VS₃, TS₃... instead of the received video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃... In addition, the user interface 70 is arranged to receive a notification from the media player 90, and to forward or relay the notification to the media modifier 80.

Figure 3 is a block diagram showing the functional components of the media modifier 80 of the media output device 30. As indicated above, the media modifier 80 is arranged to generate a modified video stream based on the received video stream VSi , TS^ VS₂, TS₂; VS₃, TS₃... received by the media output device 30. A number of sub-modules are used to generate the modified video stream. The sub-modules include a media receiver 830, a padded media generator 860, and a timestamp adjuster 890. The media receiver 830 is arranged to receive the video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃... In addition, the media receiver 830 is arranged to output the received video data samples VS^ VS₂, VS₃ ... to the padded media generator 860, and to output the received timestamps TS^ TS₂, TS₃... to the timestamp adjuster 890.

The padded media generator 860 is arranged to receive the video data samples of the video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃... from the media receiver 830, and to generate a padded video data sample for any video data sample that is less than a predetermined video data sample size. In use, the padded media generator 860 generates the padded video sample in response to receiving a video data sample from the media receiver 830. A media size determiner 868 of the padded media generator 860 is arranged to determine the size of a video data sample. In use, when the padded media generator 860 receives a video data sample, the media size determiner 868 determines the size of the received video data sample by counting the number of bytes in the received video data sample. A media size comparator 864 of the padded media generator 860 is arranged to determine whether or not the size of the received video data sample is less than a predetermined video data sample size. In this example, the predetermined video data sample size is the largest possible video data sample size according to the H.264 standard. However, persons skilled in the art will appreciate that in other examples, the predetermined video data sample size may be less than or greater than the largest possible video data sample size according to the H.264 standard. In use, the media size comparator 864 determines whether or not the size of the received video data sample is less than the predetermined video data sample size by comparing the size of the received video data sample and a predetermined video data sample size. The padded media generator 860 generates a padded media data sample based on the determination by the media size comparator 864. If the size of a received video data sample is less than the predetermined video data sample size, the padded media generator 860 generates a padded video data sample by adding padding data to the video data sample so that the padded video data sample is of the predetermined video data sample size. In Figure 3, the padded video data samples are denoted by VS^ and VS₂*. In this example, the padding data comprises an NAL padding packet as defined in the H.264 video encoding and compression standard. However, it is envisaged that in other examples, padding data such as an MPEG-TS "null packet" (as defined by the MPEG standard) may be used.

As indicated above, the media modifier 80 receives a notification from the user interface 70 in response to the media player 90 outputting a video stream. The timestamp adjuster 890 is arranged to receive the timestamps TS^ TS₂, TS₃... , and to output an adjusted timestamp for each one of the received timestamps TS^ TS₂, TS₃... based on whether or not the media modifier 80 has received a notification. In use, the timestamp adjuster 890 outputs an adjusted timestamp in response to receiving a timestamp. The timestamp adjuster 890 comprises an adjusted timestamp generator 898 and a timestamp adjustment generator 894.

The timestamp adjustment generator 894 is arranged to determine a timestamp adjustment. The determination of the timestamp adjustment is based on whether or not the media modifier 80 has received a notification. If the media modifier 80 has not received a notification, the timestamp adjustment generator 894 generates a timestamp adjustment based on a received timestamp (5_baSed on received timestamp)- Otherwise the timestamp adjustment generator 894 retrieves or calculates a target timestamp adjustment (5_target).

If the media modifier 80 has not received a notification, a timestamp adjustment based on a received timestamp (5_based on received timestamp) is generated by the timestamp adjustment generator 894 when the timestamp adjuster 890 receives a timestamp from the media receiver 830. When the timestamp adjuster 890 receives the timestamp, the timestamp adjustment generator 894 determines whether or not the received timestamp is associated with the first video data sample of the video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃... If the timestamp adjustment generator 894 determines that the received timestamp is associated with the first video data sample of the video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃..., the timestamp adjustment generator 894 retrieves an initial timestamp adjustment from memory (δ₀), and outputs a timestamp adjustment that is equivalent to the initial timestamp adjustment. Thus, if it is determined that the received timestamp is associated with the first video data sample of the video stream, the timestamp adjustment is equal to the initial timestamp adjustment. That is, δ = δ₀. If the timestamp adjustment generator 894 determines that the received timestamp is not associated with the first video data sample, the timestamp adjustment generator 894 retrieves from memory the previous timestamp adjustment (δ_ρΓβνί₀υ₅) output by the timestamp adjustment generator 894, and generates a timestamp adjustment that is equal to the result of adding the predetermined timestamp adjustment increment (6_increment) to the previous timestamp adjustment (6_previous) generated by the timestamp adjustment generator 894. Thus, if it is determined that the received timestamp is not associated with the first video data sample of the video stream, the timestamp adjustment is equal to the sum of the predetermined timestamp adjustment increment and the previous timestamp adjustment. That is, δ = δ_ρΓβνί ⁺ 5_increment-

In this example, the predetermined timestamp adjustment increment is equal to the initial timestamp adjustment. However, persons skilled in the art will appreciate that the predetermined timestamp adjustment increment can be different to the initial timestamp adjustment.

As indicated above, if the media modifier 80 has received a notification, the timestamp adjustment generator 894 generates or retrieves a target timestamp adjustment (Starget)- First, the timestamp adjustment generator 894 attempts to retrieve a target timestamp adjustment (5_target)- That is, the timestamp adjustment generator 894 checks whether or not a target timestamp adjustment (5_target) has previously been generated for the current video stream. If so, the timestamp adjustment is equal to the target timestamp adjustment (5_target) in memory. Otherwise, the timestamp adjustment generator 894 generates a target timestamp adjustment (5_target) based on the amount of time the media player 90 takes to output a video data sample. In this example, the timestamp adjustment generator 894 calculates the target timestamp adjustment (5t_arget) by subtracting the time at which the media player 90 receives the video data sample from the time at which the media player 90 outputs the video data sample, and normalizing the result based on the clock rate of the received timestamps.

The adjusted timestamp generator 898 is arranged to generate an adjusted timestamp for each one of the timestamps of the video stream VS^ TS^ VS₂, TS₂; VS₃, TS₃... The generation of the adjusted timestamp is based on the received timestamp and the timestamp adjustment generated by the timestamp adjustment generator 894. In use, the adjusted timestamp generator 898 generates the adjusted timestamp by subtracting the received timestamp by the timestamp adjustment. In Figure 3, the adjusted timestamps are denoted by TS^, TS₂* and TS₃*.

The media modifier 80 is arranged to generate a modified video stream that includes the timestamps generated by the time adjuster 890, and the padded video data samples generated by the padded media data sample generator 860. The media modifier 80 generates the modified video stream by (i) associating an adjusted timestamp with a padded video data sample, or (ii) associating an adjusted timestamp with a received video data sample, (i) is carried out by the media modifier 80 when the timestamp adjuster 890 generates an adjusted timestamp generated and the padded media generator 860 generates a padded video data sample, and (ii) is carried out by the media modifier 80 when the media modifier 80 receives an adjusted timestamp generated by the timestamp adjuster 890 but the padded media generator 860 does not produce a padded video data sample. The media modifier 80 associates an adjusted timestamp with a received video data sample by removing the timestamp associated with the video data sample, and replacing the removed timestamp with the adjusted timestamp. The media modifier 80 associates an adjusted timestamp with a padded video data sample by removing the received video data sample and the timestamp associated with the received video data sample, and replacing the removed video data sample and timestamp with the padded video data sample and adjusted timestamp.

Figure 4 is a diagram of a video stream received by the media modifier 80 and a modified video stream that is generated by the media modifier 80 based on the received video stream. VS^ VS₂ and VS₃ represents respectively the first, second and third video data sample of the received video stream. TS₁ represents the timestamp that is associated with VS₁. TS₂ represents the timestamp that is associated with VS₂. TS₃ represents the timestamp that is associated with VS₃. The modified video stream comprises two padded video data samples VS^ and VS₂*, and a received video data sample VS₃. As illustrated in Figure 3, the size of each one of the video data samples VS^, VS₂ ^* and VS₃ is equal to the predetermined video data sample size (that is, the largest possible video data sample size according to the H.264 standard). The timestamp associated with VS^ of the modified video stream is received timestamp TS^ = TS₁ - δ where δ is equal to the initial timestamp adjustment. The timestamp associated with VS₂* of the modified video stream is adjusted timestamp TS₂* = TS₂ - 2δ. The timestamp associated with VS₃ of the modified video stream is adjusted timestamp TS₃* = TS₃ - 3δ.

Figure 5 is a block diagram showing the functional components of the media player 90. The media player 90 is arranged to receive the modified video stream from the media modifier 80, and to output the modified video stream by playing the video data samples of the modified video stream according to the adjusted timestamps of the modified video stream.

The media player 90 comprises a buffer 940 arranged to store the video data samples and timestamps of the modified video stream. In this example, the media player 90 plays the video data samples stored in the buffer 940 when the buffer reaches a predetermined buffer size, and the media player 90 comprises a buffer monitor 980 which is arranged to generate a notification that the media player 90 is playing the video data samples in the buffer 940 when the buffer 940 reaches the predetermined buffer size. However, it is envisaged that in alternative example, whether or not the video data samples stored in the buffer 940 is played by the media player 90 may depend on a condition other than the amount of video data samples stored in the buffer (that is, the buffer size). For example, in an alternative example, whether or not the video data samples stored in the buffer 940 is played by the media player 90 may depend on the difference in the timestamp values of the first and last (most recently added) samples stored in the buffer 940, in that it can be recognized the buffer contains samples spanning a duration which is greater than a specific threshold. In such alternative examples, it is envisaged that the buffer monitor 980 would be arranged to generate a notification that the media player 90 is playing the video data samples in the buffer 940 only upon a determination that the condition has occurred. The user interface 70 is arranged to receive the notification from the buffer monitor 980, and to forward the notification to the media modifier 80 in response to receiving the notification from the buffer monitor 980.

Figure 6 is a flowchart of a method of an embodiment. At step 1 10, the media output device 30 receives a video stream comprising a video data sample and a timestamp associated with the video data sample. At step 120, the timestamp adjustment generator 890 of the timestamp adjuster 890 generates a timestamp adjustment. Details of step 120 are provided in Figures 7A and 7B. At step 130, the adjusted timestamp generator 898 of the timestamp adjuster 890 generates an adjusted timestamp based on the received timestamp and the timestamp adjustment generated by the timestamp adjustment generator 894. At step 150, a padded video data sample is generated by the padded media data generator 860 if required. Details of step 150 are provided in Figure 8.

At step 160, if the padded media data generator 860 generates a padded video data sample, the media modifier 80 associates the adjusted timestamp with the padded video data sample. Otherwise, the media modifier 80 associates the adjusted timestamp with the video data sample received by the media output device 30. The media modifier 80 then outputs the received or padded video data sample and the adjusted timestamp associated with the received or padded video data sample to the media player 90.

The media player 90 is controlled by the user interface 70 to receive the modified video stream from the media modifier 80. At step 180, the media player 90 stores the received or padded video data sample and the adjusted timestamp associated with the received or padded video data sample in the buffer 940. The steps 1 10, 120, 130, 150, 160 and 180 are repeated for subsequent video data samples and timestamps of the video stream received by the media output device 30.

At step 190, the media player 90 outputs the video data samples of the buffer 940 according to the timestamps associated with the respective ones of the video data samples of the buffer 940 at step 190. Thus, at step 190, the received or padded video data sample is output by the media player 90 according to the adjusted timestamp associated with the received or padded video data sample. Details of step 190 are provided in Figure 9.

Figure 7A is a flowchart showing how a timestamp adjustment is generated by the timestamp adjustment generator 894 at step 120. At step 121 , the timestamp adjustment generator 894 determines whether or not a notification has been received from the user interface 70. If the timestamp adjustment generator 894 determines that a received or padded video data samples has not been received from the user interface 70, the timestamp adjustment generator 894 generates a timestamp adjustment based on the received timestamp (5_based on received timestamp) at step 122, and the timestamp adjustment generator 894 outputs a timestamp adjustment that is equal to the timestamp adjustment based on the received timestamp (5_based on received timestamp) at step 128. Details of step 122 are provided in Figure 7B. If the timestamp adjustment generator 894 determines that a received or padded video data samples has been received from the user interface 70, the timestamp adjustment generator 894 retrieves or generates a target timestamp adjustment (5_target) at step 123. Specifically, the timestamp adjustment generator 894 retrieves a target timestamp adjustment (5_target) from memory if the target timestamp adjustment (5_target) was previously generated by the timestamp adjustment generator 894 with respect to a previous video data sample of the received video stream. Otherwise, the timestamp adjustment generator 894 generates a target timestamp adjustment (δ_ί3Γ9Θ by subtracting the time at which the media player 90 received the video data sample from the time at which the media player 90 output the video data sample, and normalizing the result based on the clock rate of the received timestamps. At step 1 29, the timestamp adjustment generator 894 subsequently outputs a timestamp adjustment that is equal to the target timestamp adjustment (5_target)-

Figure 7B is a flowchart showing how a timestamp adjustment based on a received timestamp is generated by the timestamp adjustment generator 894 at step 1 22. At step 201 , the timestamp adjustment generator 894 determines whether or not the received timestamp is a timestamp that is associated with the first video data sample of the received video stream. If the timestamp adjustment generator 894 determines that the received timestamp is associated with the first video data sample of the received video stream, the timestamp adjustment generator 894 retrieves an initial timestamp adjustment (6₀) from memory at step 204, and outputs a timestamp adjustment that is equal to the initial timestamp adjustment (δ₀) at step 208. Otherwise, if the timestamp adjustment generator 894 determines that the received timestamp is not associated with the first video data sample of the received video stream, the timestamp adjustment generator 894 retrieves a timestamp adjustment increment (5_increment) from memory at step 202, and a previous timestamp adjustment (5_preVi₀us) generated by the timestamp adjustment generator 894 from memory at step 202. Then, the timestamp adjustment generator 894 adds the timestamp adjustment increment (5_increm8nt) and the previous timestamp adjustment (5_previous), and outputs a timestamp adjustment that is equal to the sum of timestamp adjustment increment increment) and the previous timestamp adjustment (6_preVi₀us) at step 209.

Figure 8 is a flowchart showing how a padded media data sample is generated by a padded media data generator 860 at step 1 50. At step 1 53, the media size determiner 864 of the padded media data generator 860 determines the length of the video data sample (L) received by the media output device 30. At step 1 56, the media size comparator 868 of the padded media generator 860 determines whether or not the length of the received video data sample (L) is less than a predetermined video data sample size (L*). If the media size comparator 868 determines that the length of the received video data sample (L) is not less than a predetermined video data sample size (L*), the padded media data generator 860 does not generate a padded video data sample. Otherwise, if the media size comparator 868 determines that the length of the received video data sample (L) is less than a predetermined video data sample size (L*), the padded media data generator 860 generates a padded video data sample having a length L* by adding padding data. As indicated above, the padding data comprises a padding NAL packet. As indicated above, at step 1 60, if the padded media data generator 860 generates a padded video data sample, the padded video data sample is associated with an adjusted timestamp. Otherwise, the adjusted timestamp is associated with the video data sample received by the media output device 30. The received or padded video data sample and the adjusted timestamp are then transmitted from the media modifier 80 to the media player 90 and stored in the buffer 940 of the media player 90.

Figure 9 is a flowchart showing how the received or padded video data sample stored in the buffer 940 is output by the media player 90 of this example. At step 193, the buffer monitor 980 determines the number of video data samples in the buffer 940 (N_bUffer)- If the buffer monitor 980 determines that the number of video data samples in the buffer 940 (N_bu_ffer) is equal to a predetermined number of video data samples, the media player 90 outputs the video data samples stored in the buffer 940 according to the timestamps associated with the respective video data samples. In addition, the media player 90 outputs a notification to the user interface 70 which in turn relays the notification to the media modifier 80. If the buffer monitor 193 determines that the number of video data samples in the buffer 193 (N_buffer) is not equal to a predetermined number of video data samples, the media player 90 waits for the next video data sample and timestamp.

In the example above, the media modifier 80 comprises both a timestamp adjuster 890 and a padded media generator 860. However, it is envisaged that in an alternative example, the media modifier 80 may comprise only the timestamp adjuster 890 or only the padded media generator 860. That is, an alternative example of the live event streaming system 300 may be arranged to adjust the timestamps of the video data samples of a received video stream but not to pad the video data samples of the received video stream, or to pad the video data samples of a received video stream but not to adjust the timestamps of the video data samples of the received video stream. Example

In this example, the live event streaming system 300 comprises a media server 50 connected to plurality of IP cameras 10. Each one of the IP cameras is arranged to generate a video stream encoded and compressed according to the H.264 standard, and to transmit the video stream to the media server 50. The media server 50 is arranged to distribute the video streams using IP multicasting. The live event streaming system 300 also comprises a number of media output devices including one in the form of a smart phone 30. The smart phone 30 is configured to receive the video stream from the media server 50 via a wireless local area network 20. In addition, the smart phone 30 implements a number of software modules to modify the received video stream, and to play the modified video stream. The modules include a media modifier in the form of an internal HTTP server 80, a user interface in the form of a GUI 70, and a media decoder/player 90 provided by the operating system running on the smart phone 30. Upon the smart phone 30 receiving a video stream, the GUI 70 is operated by a user to initiate the media decoder/player 90, and requests the media decoder/player 90 to play the modified video stream via a URL referring to a virtual MPEG-TS file to be served by the internal HTTP server 80. The modified video stream is generated by the internal HTTP server 80 by generating an adjusted timestamp for each one of the video data samples of the received video stream, generating a padded video data sample for any one of the video data samples of the received video stream that is less than a predetermined video data sample size, and associating each adjusted timestamp with a received or padded video data sample.

Each padded video data sample is generated by padding or appending the received video data sample with a H.264 NAL packet such that the size of the padded video data sample is equal to the predetermined video data sample size. Persons skilled in the art will appreciate that the addition of the H.264 NAL packet increases the size (in bytes) of the video sample, but that the addition does not affect the semantics of the video data sample. The structure of a padded video data sample is illustrated in Figure 10. In this example, the size of the video data sample received by the smart phone 30 is 15,000 bytes, and the predetermined video data sample size is 20,000 bytes. The video data packet received by the smart phone 30 consists of two NAL packets separated by a marker sequence. The marker sequence is a 4-byte packet defined in the H.264 video compression standard that is used to separate the two NAL packets. The padded video data sample comprises the received video data packet and padding data in the form of a marker sequence and a padding NAL packet comprising 4996 bytes.

As indicated above, an adjusted timestamp is generated by the internal HTTP server 80 for each one of the video data samples of the received video stream. Each adjusted timestamp is generated by generating a timestamp adjustment, and generating the adjusted timestamp based on the timestamp adjustment and the received timestamp. With respect to the first video data sample of the received video stream, the timestamp adjustment is generated by retrieving an initial timestamp adjustment from memory (δ₀), and the adjusted timestamp is generated by subtracting the received timestamp with the timestamp adjustment. In this example, the timestamp s₀ = 50000 associated with the first received video data sample is s₀ = 50000, and the initial timestamp adjustment is δ₀ = 0. Thus, the adjusted timestamp for the first video data packet of the received video stream is So* = So - δ₀ = 50000.

With respect to each one of the second and subsequent video data samples of the received video stream, the timestamp adjustment is generated by summing the previous timestamp adjustment with a timestamp adjustment increment until the internal HTTP server 80 receives a notification from the GUI 70, and the adjusted timestamp is generated by subtracting the timestamp adjustment from the received timestamp. In this example, the timestamps associated with the second and third video data samples are s₁ = 53000 and s₂

= 56000. The timestamp increment 5increment = 500. Thus, the timestamp adjustment generated with respect to the second and third video data samples are δ₁ = 500 and 52 = 1000 respectively, and the adjusted timestamps generated with respect to the second and third video data samples are Si * = 52500 and s₂* = 55000 respectively.

Each one of the adjusted timestamps is associated with a received or padded video data sample. Thus, the modified video stream comprises a plurality of received or padded video data samples, and a plurality of adjusted timestamps associated with respective ones of the received or padded video data samples. As indicated above, the modified video stream is transmitted from the media server 50 via the wireless local area network to the smart phone 30. At the smart phone 30, the received or padded video data samples and adjusted timestamps are received and stored by the media player 90 in a buffer 940. When the buffer 940 reaches a predetermined size, the media player 90 plays the video data samples in the buffer 940 according to the adjusted timestamps associated with the video data samples, and a notification is output by the media player 90 to the GUI 70. The notification is then forwarded by the GUI 70 to the internal HTTP server 80. In this example, the first video data sample of the video stream is received by the smart phone 30 at time t_a = 100ms, and

the media player 90 plays the video data samples in the buffer 940 at time t_b =

182ms. The difference between t_a and t_b, 82ms, is the length of time between video data first being delivered to the media player 90 and the commencement of video playback. Assuming that network latency is negligible, this measurement indicates how far the playback of the video stream lags behind the live event. Upon receiving the notification from the GUI 70, a target timestamp adjustment (5_target), designed to bring the playback as close as feasible to the live event, is generated by the internal HTTP server 80 according to the formula below: target = (t_b - t_a) / 1000 X CLK - m where CLK is the clock rate of the timestamps, and m is a predetermined margin of allowance. In this example, m = 500, CLK = 90 kHz and thus the target timestamp adjustment 5_target = 7380 - m = 6880. Below is a table listing the timestamps associated with the fourth to seventeenth video data samples received by the smart phone 30, and the timestamp adjustments and adjusted timestamps generated by the internal HTTP server 80: Video data sample Received Timestamp Adjusted timestamp sequence no timestamp adjustment

3 S₃ = 59000 δ₃ = 1500 S₃ ^* = 57500

4 S₄ = 62000 δ₄ = 2000 S₄ ^* = 60000

5 S₅ = 65000 δ₅ = 2500 S₅ ^* = 62500

6 S₆ = 68000 δ₆ = 3000 S₆ ^* = 65000

7 S₇ = 71000 δ₇ = 3500 S₇ ^* = 67500

8 S₈ = 74000 δ₈ = 4000 S₈ ^* = 70000

9 S₉ = 77000 δ₉ = 4500 S₉ ^* = 72500

10 Sio = 80000 διο = 5000 S₁₀ ^* = 75000

11 Sii = 83000 δι ι = 5500 Sii^* = 77500

12 Si2 = 86000 δΐ2 = 6000 S₁₂ ^* = 80000

13 Sis = 89000 δΐ3 = 6500 S₁₃ ^* = 82500

14 Si4 = 92000 δ_Μ = 6880 S₁₄ ^* = 85120

15 Sis = 95000 δΐ5 = 6880 S₁₅ ^* = 88120

16 Si6 = 98000 δΐ6 = 6880 S₁₆ ^* = 91120

It will be understood to persons skilled in the art of the invention that many modifications may be made without departing from the spirit and scope of the invention, in particular it will be apparent that certain features of embodiments of the invention can be employed to form further embodiments.

It is to be understood that references to prior art made herein does not constitute an admission that the prior art formed or forms a part of the common general knowledge in the art in Australia or in any other country.

In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word "comprise" or variations such as "comprises" or "comprising" is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention.

Claims

CLAIMS:

1 . A method of outputting a media data sample at a media output device, comprising: receiving from a media capture device a media data sample and a timestamp associated with the received media data sample;

2. A method as claimed in claim 1 , wherein associating the received media data sample with an adjusted timestamp comprises removing the received timestamp, and replacing the removed timestamp with the adjusted timestamp.

3. A method as claimed in claim 1 or claim 2, wherein receiving the media sample and the timestamp comprises receiving from the media capture device a media stream comprising a plurality of media data samples and a plurality of timestamps associated with respective ones of the plurality of media data samples.

4. A method as claimed in any one of claims 1 to 3, comprising:

generating a timestamp adjustment based on the received timestamp; and generating an adjusted timestamp based on the received timestamp and the timestamp adjustment.

5. A method as claimed in claim 4, wherein generating the adjusted timestamp comprises subtracting the timestamp adjustment from the received timestamp.

6. A method as claimed in claim 4 or claim 5, wherein generating the timestamp adjustment comprises retrieving an initial timestamp adjustment from memory.

7. A method as claimed in claim 4 or claim 5, wherein generating the timestamp adjustment comprises:

retrieving a previous timestamp adjustment from memory; and

generating the timestamp adjustment based on the previous timestamp adjustment.

8. A method as claimed in claim 7, wherein generating the timestamp adjustment comprises adding a predetermined timestamp adjustment to the previous timestamp adjustment.

9. A method as claimed in claim 8, further comprising:

receiving a notification that the received media data sample has been output, wherein generating the timestamp adjustment comprises generating a target timestamp adjustment upon receiving the notification that the received media data sample has been output.

10. A method as claimed in claim 9, wherein the target timestamp adjustment is generated based on the time at which the media data sample is received and the time at which the media data sample is output.

1 1 . A method as claimed in any one of claims 1 to 10, comprising storing the received media data sample and the adjusted timestamp associated with the received media data sample in a buffer before outputting the received data sample.

12. A method as claimed in claim 1 1 , comprising determining whether or not the number of media data samples in the buffer is equal to a predetermined number of media data samples.

13. A method as claimed in claim 1 1 , wherein the received media data sample is output upon the number of media data samples in the buffer being equal to the predetermined number of media data samples.

14. A method as claimed in any one of claims 1 to 13, comprising:

upon the size of the received media data sample being less than a predetermined media data sample size, generating a padded media data sample comprising padding data and the received media data sample,

wherein outputting the received media data sample comprises outputting the padded media data sample according to the adjusted timestamp associated with the received media data sample.

15. A method as claimed in claim 14, comprising:

determining the size of the received media data sample; and

determining whether or not the size of the received media data sample is less than the predetermined media data sample size.

16. A method as claimed in any one of claims 1 to 15, wherein the media stream is received from the media capture device via a multicast transmission of the media stream.

17. A method as claimed in any one of claims 1 to 15, wherein the media stream is received from the media capture device via a broadcast transmission of the media stream.

18. A method as claimed in any one of claims 1 to 15, wherein the media stream is received from the media capture device via a unicast transmission of the media stream.

19. A method as claimed in any one of claims 1 to 18, wherein the media stream is a video stream.

20. A method as claimed in claim 19, wherein the video stream is a H.264 video stream.

21 . A method as claimed in any one of claims 1 to 18, wherein the media stream is an audio stream.

22. A method as claimed in any one of claims 1 to 18, wherein the media stream is a video and audio stream.

23. A method of outputting a media data sample at a media output device, comprising: receiving a media data sample from a media capture device;

upon the size of the received media data sample being less than a predetermined media data sample size, generating a padded media data sample comprising padding data and the received media data sample; and outputting the padded media data sample.

24. A method as claimed in claim 23, wherein receiving the media data sample comprises receiving from the media capture device a media stream comprising a plurality of media data samples.

25. A method as claimed in claim 23 or claim 24, comprising:

determining the size of the received media data sample; and

26. A method as claimed in any one of claims 23 to 25, wherein the size of the padded media data sample is equal to the predetermined media data sample size.

27. A method as claimed in any one of claims 23 to 26, comprising storing the padded media data sample in a buffer before outputting the padded media data sample.

28. A method as claimed in claim 27, comprising determining whether or not the number of media data samples in the buffer is equal to a predetermined number of media data samples.

29. A method as claimed in claim 28, wherein the padded media data sample is output when the number of media data samples in the buffer is equal to the predetermined number of media data samples.

30. A method as claimed in any one of claims 23 to 29, wherein the media stream is received from the media capture device via a multicast transmission of the media stream.

31 . A method as claimed in any one of claims 23 to 29, wherein the media stream is received from the media capture device via a broadcast transmission of the media stream.

32. A method as claimed in any one of claims 23 to 29, wherein the media stream is received from the media capture device via a unicast transmission of the media stream.

33. A method as claimed in any one of claims 23 to 32, wherein the media stream is a video stream.

34. A method as claimed in claim 33, wherein the video stream is a H.264 video stream.

35. A method as claimed in any one of claims 23 to 34, wherein the padding data comprises a 4-byte marker sequence and a padding NAL packet.

36. A method as claimed in any one of claims 23 to 32, wherein the media stream is an audio stream.

37. A method as claimed in any one of claims 23 to 32, wherein the media stream is a video and audio stream.

38. A media output device for outputting a media data sample, the media output device comprising:

39. A device as claimed in claim 38, wherein the timestamp adjuster associates the received media data sample with the adjusted timestamp by removing the received timestamp, and replacing the removed timestamp with the adjusted timestamp.

40. A device as claimed in claim 38 or claim 39, wherein the media receiver is arranged to receive from the media capture device a media stream comprising a plurality of media data samples and a plurality of timestamps associated with respective ones of the plurality of media data samples.

41 . A device as claimed in any one of claims 38 to 40, wherein the timestamp adjuster comprises:

a timestamp adjustment generator arranged to generate a timestamp adjustment based on the received timestamp; and

an adjusted timestamp generator arranged to generate the adjusted timestamp based on the received timestamp and the timestamp adjustment.

42. A device as claimed in claim 41 , wherein the adjusted timestamp generator generates the adjusted timestamp by subtracting the timestamp adjustment from the received timestamp.

43. A device as claimed in claim 41 or claim 42, wherein the timestamp adjustment generator generates the timestamp adjustment by retrieving an initial timestamp adjustment from memory.

44. A device as claimed in claim 41 or claim 42, wherein the timestamp adjustment generator generates the timestamp adjustment by:

retrieving a previous timestamp adjustment from memory; and

generating the timestamp adjustment based on the previous timestamp adjustment.

45. A device as claimed in claim 44, wherein the timestamp adjustment generator generates the timestamp adjustment by adding a predetermined timestamp adjustment to the previous timestamp adjustment.

46. A device as claimed in claim 45, wherein the timestamp adjustment generator is arranged to receive a notification that the received media data sample has been output by the media player, and to generate the timestamp adjustment by generating a target timestamp adjustment upon receiving the notification that the received media data sample has been output by the media player.

47. A device as claimed in claim 46, wherein the timestamp adjustment generator generates the timestamp adjustment based on the time at which the media data sample is received by the media receiver and the time at which the media data sample is output by the media player.

48. A device as claimed in any one of claims 38 to 47, comprising a buffer for storing the received media data sample and the adjusted timestamp associated with the received media data sample.

49. A device as claimed in claim 48, comprising a buffer monitor arranged to determine whether or not the number of media data samples in the buffer is equal to a predetermined number of media data samples.

50. A device as claimed in claim 49, wherein the media player outputs the received media data sample upon a determination by the buffer monitor that the number of media data samples in the buffer is equal to the predetermined number of media data samples.

51 . A device as claimed in any one of claims 38 to 50, comprising:

a padded media generator arranged to, upon the size of the received media data sample being less than a predetermined media data sample size, generate a padded media data sample comprising padding data and the received media data sample,

wherein the media player is arranged to output the padded media data sample according to the adjusted timestamp associated with the received media data sample.

52. A device as claimed in claim 51 , wherein the padded media generator comprises: a media size determiner arranged to determine the size of the received media data sample; and

a media size comparator arranged to determine whether or not the size of the received media data sample is less than the predetermined media data sample size.

53. A device as claimed in any one of claims 38 to 52, wherein the media stream is received from the media capture device via a multicast transmission of the media stream.

54. A device as claimed in any one of claims 38 to 52, wherein the media stream is received from the media capture device via a broadcast transmission of the media stream.

55. A device as claimed in any one of claims 38 to 52, wherein the media stream is received from the media capture device via a unicast transmission of the media stream.

56. A device as claimed in any one of claims 38 to 55, wherein the media stream is a video stream.

57. A device as claimed in claim 56, wherein the video stream is a H.264 video stream.

58. A device as claimed in any one of claims 38 to 55, wherein the media stream is an audio stream.

59. A device as claimed in any one of claims 38 to 55, wherein the media stream is a video and audio stream.

60. A media output device for outputting a media data sample, the media output device comprising: a media receiver arranged to receive a media data sample from a media capture device;

a media player arranged to output the padded media data sample.

61 . A method as claimed in claim 60, wherein the media receiver is arranged to receive from the media capture device a media stream comprising a plurality of media data samples.

62. A device as claimed in claim 60 or claim 61 , wherein the padded media generator comprises:

a media size determiner arranged to determine the size of the received media data sample; and

63. A device as claimed in any one of claims 60 to 62, wherein the size of the padded media data sample is equal to the predetermined media data sample size.

64. A device as claimed in any one of claims 60 to 63, comprising a buffer for storing the padded media data sample.

65. A device as claimed in claim 64, comprising a buffer monitor arranged to determine whether or not the number of media data samples in the buffer is equal to a predetermined number of media data samples.

66. A device as claimed in claim 65, wherein the media player outputs the received media data sample upon a determination by the buffer monitor that the number of media data samples in the buffer is equal to the predetermined number of media data samples.

67. A device as claimed in any one of claims 60 to 66, wherein the media stream is received from the media capture device via a multicast transmission of the media stream.

68. A device as claimed in any one of claims 60 to 66, wherein the media stream is received from the media capture device via a broadcast transmission of the media stream.

69. A device as claimed in any one of claims 60 to 66, wherein the media stream is received from the media capture device via a unicast transmission of the media stream.

70. A device as claimed in any one of claims 60 to 69, wherein the media stream is a video stream.

71 . A device as claimed in claim 70, wherein the video stream is a H.264 video stream.

72. A device as claimed in any one of claims 60 to 71 , wherein the padding data comprises a 4-byte marker sequence and a padding NAL packet.

73. A device as claimed in any one of claims 60 to 69, wherein the media stream is an audio stream.

74. A device as claimed in any one of claims 60 to 69, wherein the media stream is a video and audio stream.