HK40080095B - Method for processing events in media stream, apparatus, storage medium and computer equipment - Google Patents

Description

Methods, apparatus, storage media and computer equipment for processing events in media streams

References merged

This application claims priority to U.S. Provisional Application No. 63/176,748, filed April 19, 2019, and U.S. Application No. 17/495,299, filed October 6, 2021, the entire contents of which are incorporated herein by reference.

Technical Field

This application relates to Internet transmission technology, and more particularly to a method, apparatus, storage medium, and computer device for processing events in a media stream.

Background Technology

The Moving Picture Experts Group (MPEG) Hypertext Transfer Protocol (DASH) provides a standard for streaming media content over IP networks. In MPEG DASH, media presentation descriptions (MPDs) and in-band events are used to deliver timeline-related events to the client. DASH provides a mechanism for media segments to carry in-band events. Similarly, the Common Media Application Format (CMAF) provides a mechanism for carrying in-band events in CMAF blocks. A popular DASH event is the in-band MPD validity expiration event. Other events include application events such as the SCTE-35 ("Digital Program Insertion Prompt Message for Cable") event.

Streaming media players, including DASH players, use Media Source Extensions (MSEs), which allow browsers or user agents to process media segments. However, the current MSE specification does not support parsing and processing in-band events embedded in DASH or CMAF media segments.

Summary of the Invention

This application provides a method, apparatus, storage medium, and computer device for processing events in a media stream, enabling browsers or user agents using MSE to natively process DASH and CMAF in-band events and send them to the corresponding applications.

This application provides a method for processing events in a media stream, including:

Obtain media data;

Generate at least one event segment from the media data;

The at least one event segment is attached to a first event processing buffer, the at least one event segment including an event start time offset for each of the at least one event segments, the event start time offset being determined based on the time when the event segment is attached to the first event processing buffer;

The at least one event fragment is appended to a second event processing buffer, the at least one event fragment further comprising event dispatch information for each event fragment in the at least one event fragment; and,

The at least one event fragment is dispatched based on the event dispatch information in the first event processing buffer and the second event processing buffer.

This application also provides an apparatus for processing events in a media stream, comprising:

The first acquisition module is used to acquire media data;

The first generation module is used to generate at least one event fragment from the media data;

A first attachment module is configured to attach the at least one event segment to a first event processing buffer, the at least one event segment including an event start time offset for each of the at least one event segments, the event start time offset being determined based on the time when the event segment is attached to the first event processing buffer;

A second append module is configured to append the at least one event fragment to a second event processing buffer, wherein the at least one event fragment further includes event dispatch information for each event fragment in the at least one event fragment; and,

A first dispatch module is configured to dispatch the at least one event fragment based on the event dispatch information in the first event processing buffer and the second event processing buffer.

This application also provides a computer device, including a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above method.

This application also provides a non-transitory computer-readable medium storing instructions that, when performed by a computer for video decoding, cause the computer to perform the above-described method.

This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the described method.

As can be seen from the above technical solutions, the method provided by the embodiments of the present invention uses the event start time offset of the event segment, which avoids the need for the application to compare a common time base when determining the relative start time of the event segment, thereby increasing the application processing efficiency and reducing network overhead (such as bandwidth requirements), server computing overhead and application computing overhead.

Attached Figure Description

Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, wherein:

Figure 1 shows a simplified schematic diagram of a communication system according to an embodiment of this application;

Figure 2 shows a schematic diagram of component placement in a flow environment according to an embodiment of this application;

Figure 3 shows a simplified block diagram of a DASH processing model according to an embodiment of this application;

Figure 4 shows a simplified block diagram of a DASH processing model according to another embodiment of this application;

Figure 5 shows a simplified block diagram of a media buffer and an event buffer according to an embodiment of this application;

Figure 6 shows a simplified block diagram of a media buffer and an event buffer according to another embodiment of this application;

Figure 7 shows a simplified block diagram of a media buffer and an event buffer according to yet another embodiment of this application;

Figure 8A shows a simplified schematic diagram of the process for processing DASH and CAMF in-band events according to an embodiment of this application;

Figure 8B shows a simplified schematic diagram of the process for processing DASH and CAMF in-band events according to another embodiment of this application;

Figure 9 shows a simplified schematic diagram of the process for processing DASH and CAMF in-band events according to another embodiment of this application;

Figure 10 shows a schematic diagram of a computer device according to an embodiment.

Detailed Implementation

The features proposed in this application, discussed below, can be used individually or in combination in any order. Furthermore, embodiments of this application can be implemented by processing circuitry (e.g., at least one processor, or at least one integrated circuit). In one example, at least one processor executes a program stored on a non-volatile computer-readable medium.

Figure 1 is a simplified block diagram of a communication system 1000 according to an embodiment disclosed in this application. The communication system 1000 includes a first terminal 102 and a second terminal 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 can encode video data at its local location for transmission to the other terminal 102 via the network 105. The second terminal 102 can receive the encoded video data from the other terminal via the network 105, decode the encoded video data to recover the video data, and display video images based on the recovered video data. Unidirectional data transmission is common in applications such as media services.

Figure 1 illustrates another pair of terminals 101 and 104 to support bidirectional transmission of encoded video data, for example, during a video conference. For bidirectional data transmission, each terminal 101 and/or 104 can encode locally acquired video data for transmission to the other terminal via network 105. Each terminal 101 and/or 104 can also receive encoded video data transmitted by the other terminal, decode the encoded video data to recover the video data, and display video images on an accessible display device based on the recovered video data.

In Figure 1, terminals 101, 102, 103, and/or 104 may be servers, personal computers, and smartphones, but the principles disclosed in this application are not limited to these. The embodiments disclosed in this application are applicable to laptop computers, tablet computers, media players, and/or dedicated video conferencing equipment. Network 105 refers to any number of networks transmitting encoded video data between terminals 101, 102, 103, and/or 104, including, for example, wired and/or wireless communication networks. Communication network 105 may exchange data in circuit-switched and/or packet-switched channels. This network may include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of this application, unless explained below, the architecture and topology of network 105 may be irrelevant to the operation of this application.

As an example, Figure 2 illustrates the placement of the video encoder and video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

The streaming system may include an acquisition subsystem 203, which may include a video source 201 such as a digital camera, which creates, for example, an uncompressed video sample stream 213. The video sample stream 213 is emphasized as a high-data-volume video image stream compared to an encoded video stream and may be processed by an encoder 202 coupled to the video source 201. The encoder 202 may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video stream 204 is emphasized as a lower-data-volume stream compared to the sample stream and may be stored on a streaming server 205 for future use. At least one streaming client 212 and 207 can access the streaming server 205 to retrieve encoded video streams 208 and further 206, both of which are copies of the encoded video stream 204. Client 212 includes a video decoder 211. Video decoder 211 decodes the incoming copy of the encoded video stream 208 and produces an output video sample stream 210 that can be displayed on display 209 or another presentation device. In some streaming systems, the encoded video streams 204, 206, and 208 may be encoded according to certain video encoding/compression standards. Examples of these standards are as noted above and are further described herein.

A 5G Media Streaming (5GMS) system is a collection of application functions, application servers, and interfaces interconnected with a 5G Media Streaming architecture that supports downlink media streaming services, uplink media streaming services, or both. 5GMS application providers may include parties that interact with and provide 5GMS-aware applications that interact with the functions of the 5GMS system. 5GMS-aware applications may refer to applications in a User Equipment (UE) provided by a 5GMS application provider, containing the service logic of 5GMS application services, and interacting with other 5GMS clients and network functions via interfaces and application programming interfaces (APIs) defined in the 5GMS architecture. A 5GMS client may refer to a UE function, including a 5GMS downlink (5GMSd) client, a 5GMS uplink (5GMSu) client, or both.

A 5GMSd client can refer to a UE function that includes at least a 5G media streaming player and a media session processor for downlink streaming, and can be accessed through a well-defined interface/API. A 5GMSu client can refer to an initiator of a 5GMSu service, where 5GMSu is accessed through a well-defined interface/API. A 5GMSu streaming media player can refer to a UE function that enables uplink delivery of streaming media content to the application server (AS) function of a 5GMS application provider, and that can interact not only with 5GMSu-aware applications used for media capture and subsequent streaming, but also with a media session processor used for media session control.

Dynamic policies refer to dynamic policies and charging control (PCC) rules used for uplink or downlink application flows during a media session. An egest session refers to an uplink media flow session from the 5GMS AS to the 5GMS Application Provider. An ingest session refers to a session that uploads media content to the 5GMSd AS. A policy template refers to a set of (semi-static) policy or control function (PCF)/network open function (NEF) API parameters, where NEF API parameters are specific to the 5GMS Application Provider and the resulting PCC rules. A policy template ID identifies the desired policy template used by the 5GMSd Application Function (AF) to select the appropriate PCF/NEF API for the 5G system, enabling the PCF to compile the desired PCC rules. A media player entry refers to a document or an indicator pointing to a document used to define media presentation (e.g., a Media Presentation Description (MPD) for DASH or a Uniform Resource Locator (URL) for a video clip file). A Media Streamer Entry is an indicator (e.g., in the form of a URL) that defines the entry point for an uplink media streaming session. A Rendering Entry refers to a document or an indicator pointing to a document, used to define application rendering, such as an HTML5 document.

A provisioning session refers to a data structure provided by the 5GMSd application provider at the interface (M1d), which configures 5GMSd features associated with a set of 5GMSd-aware applications. A 5GMSd media player refers to a UE function that enables the replay and presentation of media based on media playback items and provides basic controls to 5GMSd-aware applications, such as play, pause, search, and stop. Server access information refers to a set of parameters and addresses (including 5GMSd AF and 5GMSd AS addresses) required to activate a streaming session. Service and content discovery refers to the functions and procedures provided by the 5GMSd application provider to 5GMSd-aware applications, enabling end users to discover available streaming services and content settings and select specific services or content items for access. A service announcement is a procedure executed between a 5GMS sensing application and a 5GMS application provider, enabling the 5GMS sensing application to obtain 5GMS service access information directly or through reference information.

Third-party media players refer to a subset of applications that use APIs to exercise selected 5GMSd features to play back media content. Third-party uplink streaming media players refer to a subset of applications that use APIs to exercise selected 5GMSu features to capture and stream media content.

Figure 3 illustrates a sample DASH processing model 300, such as a sample client architecture for handling DASH and CMAF events. In DASH processing model 300, client requests for media segments can be based on addresses described in manifest parser 303. Manifest parser 303 also describes metadata tracks from which the client can access segments, parse, and send them to application 301.

The manifest parser 303 includes MPD events or in-band events. The in-band event and 'moof' parser 306 can parse MPD event fragments or in-band event fragments and append the event fragments to the event and metadata buffer 330. The in-band event and 'moof' parser 306 can also extract media fragments and append them to the media buffer 340. The event and metadata buffer 330 can send event and metadata information to the event and metadata synchronizer and dispatcher 335. The event and metadata synchronizer and dispatcher 335 can dispatch specific events to the DASH player control, selection, and heuristic logic 302, and dispatch application-related event and metadata tracks to the application 301.

According to some embodiments, the MSE may include a media buffer 340 and a media decoder 345. The MSE 320 is one or more logical buffers for media segments, which can be identified and ordered based on the presentation time of the media segments. Each media segment can be added to or appended to the media buffer 340 based on a timestamp offset, and the timestamp offset can be used to order the media segments in the media buffer 340.

As long as a media segment exists in the media buffer 340, the event and metadata buffer 330 maintains the corresponding event segment and metadata. According to Figure 3, the MSE 320 only includes the media buffer 340 and the media decoder 345. The event and metadata buffer 330, as well as the event and metadata synchronizer and dispatcher 335, are not native to the MSE 320, preventing the MSE 320 from natively processing events and sending them to the application.

Figure 4 illustrates a sample DASH processing model 400, such as a sample client architecture for handling DASH and CMAF events. In DASH processing model 400, client requests for media segments can be based on the addresses described in Listing 403. Listing 403 also describes a metadata track from which the client can access segments, parse, and send them to application 401.

Listing 403 includes MPD events or in-band events, and the in-band event and 'moof' parser 406 can parse MPD event fragments or in-band event fragments and append the event fragments to the event purge buffer 430. Based on media data or media streams and the information in Listing 403, the in-band event and 'moof' parser 406 can extract media fragments and append them to the media buffer 440. The event purge buffer 430 can send event and metadata information to the event synchronizer and dispatcher 435. The event synchronizer and dispatcher 435 can dispatch specific events to the DASH player control, selection, and heuristic logic 402, and dispatch application-related event and metadata tracks to the application 401.

According to an exemplary embodiment, in FIG4, the MSE is extended to include: a media buffer 340, a media decoder 345, an event clearing buffer 430, an event synchronizer and dispatcher 435, and an already-dispatched event table 450. The already-dispatched event table 450 can record events that have been dispatched to application 401. The extension of the event clearing buffer 430 and the event synchronizer and dispatcher 435 enables the MSE to locally process in-band events and create the already-dispatched event table 450, enabling the recording and detection of dispatched event messages.

In an exemplary embodiment, the MSE 420 or its components may dispatch events based on an event-specific offset of an event segment. Examples include an event start time offset. The MSE 420 or its components may determine the event start time offset of an event segment by referencing an event presentation time, and use the event start time offset to sort and dispatch events in the event clear buffer 430 or the event synchronizer and dispatcher 435. In an exemplary embodiment, the event start time offset and the timestamp offset may be equivalent, wherein the event start time offset refers to an event segment in the event clear buffer 430 or the event synchronizer and dispatcher 435, and the timestamp offset refers to a media segment in the media buffer 440.

In an exemplary embodiment, MSE 420 can process event segments and media segment clearing and rewriting for associated events and media. If the media segment associated with the event is retained in media buffer 440, MSE 420 can retain the event in event clearing buffer 430 or event synchronizer and dispatcher 435. MSE 420 can manage the timing and duration of event segments based on the timing and duration of the associated media segments.

In an exemplary embodiment, application 401 may set a timestamp offset or an event timestamp offset. Application 401 may also set a scheme URI/value and a dispatch mode for the scheme URI/value. MSE 420 may dispatch events based on the scheme URI/value, the event ID, the event association ID indicating the media segment associated with the event segment, the presentation time of the event segment, the start time offset of the event, and the event duration.

Figure 5 illustrates an example media and event buffer group 500 for processing media segments and event segments. Each media stream 510 may include media segments (S0, S1, S2, S3) and event segments (E0, E1, E2). Media segments may be appended to a media source buffer or a media buffer 520.

The in-band event and 'moof' parser 406 can parse the media stream and generate media segments and event segments. Each media segment can be appended to a media source buffer or media buffer 520 based on a timestamp offset. Each media segment appended to the media source buffer or media buffer 520 can have a unique event association ID. Using the event association ID of the corresponding media segment, the generated event segments (E0, E1, E2) can be associated with a specific media segment (S0, S1, S2, S3). In an exemplary embodiment, multiple event segments can be associated with a single media segment.

In an exemplary embodiment, event segments (E0, E1, E2) can be appended to the event clearing buffer 530 by aligning the event start time and event duration of the event segments in the event clearing buffer 530 with the start time and duration of the associated media segments (S0, S1, S2, S3) in the media buffer 520. As an example, the rendering time or event start time offset of one or more event segments and their durations in the event clearing buffer 530 can be aligned with the rendering time and duration of the associated media segments in the media buffer 620. In some exemplary embodiments, if more than one event segment is associated with the same media segment, the event start time offset of each event segment can be adjusted based on the rendering time of the associated media segment.

In some exemplary embodiments, an event segment can be appended to an event dispatch buffer 540 by copying the event start time and duration of the corresponding event segment into the event clearing buffer 530. In some embodiments, the order, rendering time, and duration of the event segments can be equivalent in both the event clearing buffer 530 and the event dispatch buffer 540. As an example, if the application has set the scheme to "on_receive", the rendering time or event start time offset of the event segment and the duration in the event dispatch buffer 540 can be aligned with the rendering time and duration of the associated media segment; that is, the event dispatch buffer 540 can be a copy of the event clearing buffer 530.

Figure 6 illustrates an example group of media and event buffers for processing media segments and event segments. Each media stream 610 may include media segments (S0, S1, S2, S3) and event segments (E0, E1, E2). Media segments may be appended to a media source buffer or a media buffer 620.

The in-band event and 'moof' parser 406 can parse the media stream and generate media segments and event segments. Each media segment can be appended to a media source buffer or media buffer 620 based on a timestamp offset. Each media segment appended to the media source buffer or media buffer 620 can have a unique event association ID. Using the event association ID of the corresponding media segment, the generated event segments (E0, E1, E2) can be associated with a specific media segment (S0, S1, S2, S3). In an exemplary embodiment, multiple event segments can be associated with a single media segment.

In an exemplary embodiment, event segments (E0, E1, E2) can be appended to the event clearing buffer 630 by aligning the event start time and event duration of the event segments in the event clearing buffer 630 with the start time and duration of the associated media segments (S0, S1, S2, S3) in the media buffer 620. As an example, the rendering time or event start time offset of one or more event segments and their durations in the event clearing buffer 630 can be aligned with the rendering time and duration of the associated media segments in the media buffer 620. In some exemplary embodiments, if more than one event segment is associated with the same media segment, the event start time offset of each event segment should be adjusted based on the rendering time of the associated media segment.

In an exemplary embodiment, an event fragment can be appended to the event dispatch buffer 640 by adding an event fragment, such that the event start time offset of the event fragment is the rendering time of the event in the event dispatch buffer 640, and the duration of the event fragment is the duration of the event in the event dispatch buffer 640. As an example, if the application has already set this scheme to "on_start", the rendering time of the event fragment and the duration in the event dispatch buffer 640 can reflect the event start time offset and duration of the event fragment. In some embodiments, the event dispatch buffer 640 can reflect the event start time and duration of the event fragment; that is, event fragments can be appended such that the event fragment in the event dispatch buffer 640 extends from the beginning of the event fragment to the end of the event fragment. Therefore, in some embodiments, the event clearing buffer 630 and the event dispatch buffer 640 may not be equivalent.

Figure 7 illustrates an example media and event buffer group 700 for processing media segments and event segments. Each media stream 710 may include media segments (S0, S1, S2, S3) and event segments (E0, E1, E2). Media segments may be appended to a media source buffer or a media buffer 720.

In an exemplary embodiment, event fragments (E0, E1, E2) in the event clearing buffer 730 and the event dispatch buffer 740 may be removed, cleared, or rewritten. In some embodiments, event fragments may be partially removed, cleared, or rewritten. To remove, clear, or rewrite event fragments, event fragments may be removed from the event clearing buffer 730 and the event dispatch buffer 740. As an example, once a media fragment is rewritten in the media buffer 720, the corresponding event fragments associated with the rewritten media fragment in the event clearing buffer 730 and the event dispatch buffer 740 may also be rewritten.

In some embodiments, to partially remove or rewrite event segments, event segments can be divided into smaller sections in the event clearing buffer 730 and the event dispatch buffer 740. Only the sections of the event segments that overlap with the new event segment can be removed, cleared, or rewritten, thereby adding sections of the new event segment. As an example, if the new event segment is smaller than the current event segment in the event clearing buffer 730 or the event dispatch buffer 740, one or more current event segments in the event clearing buffer 730 can be divided into smaller sections to match the new event segment. The sections of the event segments in the event clearing buffer 730 that overlap with the new, smaller event segment can be removed, cleared, or rewritten.

In some embodiments, only event segments overlapping with the new event segment in the event clearing buffer 730 may be removed, cleared, or rewritten. In some embodiments, only event segments overlapping with the new event segment in the event dispatch buffer 740 may be removed, cleared, or rewritten. The event clearing buffer 730 may be rewritten, and then the event dispatch buffer 740 may be rewritten, such that the event clearing buffer 730 and the event dispatch buffer 740 may be equivalent. As an example, if the application has already set the scheme to "on_receive", then event segments or individual segments of event segments overlapping with the new event segment in both the event clearing buffer 730 and the event dispatch buffer 740 may be rewritten.

In some exemplary embodiments, the removal, clearing, or rewriting of an event segment requires determining that no other event segments associated with the same media segment exist in the event clearing buffer 730. In some exemplary embodiments, if no other event segments associated with the same media segment exist in the event clearing buffer 730, the corresponding event in the event dispatch buffer 740 can be removed, cleared, or rewritten. As an example, if more than one event segment overlaps with a new event segment in the event clearing buffer 730, rewriting the overlapping event segments requires dispatching all event segments associated with the same media segment before rewriting the overlapping event segments in the event clearing buffer 730. In some embodiments, if no event segments associated with the same media segment exist in the event clearing buffer 730, the overlapping event segments can be rewritten in the event cleanup buffer 730. As an example, if the application has set this scheme to "on_start", rewriting the overlapping events in the event clearing buffer 730 requires dispatching all event segments associated with the same media segment before rewriting the overlapping event segments in the event clearing buffer 730. In some embodiments, once an event fragment is removed, cleared, or rewritten in the event clearing buffer 730, the corresponding event fragment can be removed, cleared, or rewritten in the event dispatch buffer 740. As an example, if the application has set the scheme to "on_start", then after one or more overlapping event fragments in the event clearing buffer 730 have been rewritten, the overlapping segment to be rewritten in the event dispatch buffer 740 can be rewritten.

Figure 8A illustrates an exemplary flowchart of process 800 for processing DASH and CMAF in-band events, wherein, according to an exemplary embodiment, event fragments may be appended to an event clearing buffer and an event dispatch buffer. At 810, media data is obtained. The media data can be any type of media data. As an example, the media data can be audio data, video data, a document, or any combination thereof. The obtained media data may also contain event data.

At point 815, at least one event fragment can be generated from the acquired media data. In some embodiments, the MSE or a parser component of the MSE can generate event fragments from the acquired media data. Event fragments may include MPDs and in-band events to deliver media timeline-related events to a client or application. As an example, an in-band event and 'moof' parser (306 or 406) can generate event fragments from the acquired media data. In some embodiments, the parser can generate both a media fragment and an event fragment corresponding to the media fragment. As an example, an in-band event and 'moof' parser (306 or 406) can generate media fragments and associated event fragments based on media data. Event fragments can be associated with an event association ID, where the event association ID can be a unique ID associated with the media fragment. In some embodiments, multiple event fragments can be associated with the same media fragment. Event fragments associated with the same media fragment can have an event association ID.

At 820, the event start time offset of the event segment is determined. In some embodiments, MSE or its components determine the event start time offset for each event segment to be attached to the event clear buffer or the event dispatch buffer. In some embodiments, the event start time offset of the event segment may be determined based on the time the event segment is attached to the event clear buffer. In some embodiments, the event start time offset of the event segment may be determined based on the time the event segment is attached to the event dispatch buffer. In some embodiments, the event start time offset of the event segment may be determined based on the rendering time of the event segment or media segment in the MSE API. The start time offset of the event segment indicates the start time of the event from the dispatch time. Therefore, when the event segment is sent to the application, the application can determine the relative start of the event from the dispatch time, rather than comparing a common time base.

The traditional approach uses the event start time in the MSE (Multi-Security Environment) and processes events in the application. However, this traditional approach requires maintaining a common time reference.

According to this embodiment, an alternative time synchronization method is provided, which uses the event start time offset of the event segment determined based on the time in the MSE API. This avoids the need for the application to compare a common time base when determining the relative start time of the event segment, thereby increasing the application processing efficiency and reducing bandwidth requirements.

At 825, event segments can be appended to the event clearing buffer based on their event start time offset. The event clearing buffer can be used to determine the order of event segments, their location, and dispatch timing. In some embodiments, appending event segments to the clearing buffer can include aligning the render time and duration of the event segment with the corresponding render time and duration of the associated media segment. As an example, event segments associated with media segments can be placed in the event clearing buffer such that each event segment and its associated media segment have the same render time and duration.

At 830, an event fragment can be appended to the event dispatch buffer based on the event fragment's dispatch information. In some embodiments, the dispatch information may include information indicating at least one of the following: event initialization, event appending, event cleanup, event duration, event rewriting, or attributes set by the application. These attributes, or attributes set by the application based on a set of attributes, include scheme URI/value, id, event start time offset of the event fragment, start rendering increment of the event fragment, and event fragment duration.

In some embodiments, attaching event fragments to an event dispatch buffer may include repeatedly attaching event fragments to an event clear buffer, or duplicating this attachment operation. Therefore, the order, render time, and duration of event fragments can be the same in both the event clear buffer and the event dispatch buffer. As an example, if the application has set this scheme to "on_receive," the render time or event start time offset of one or more event fragments and their duration in the event dispatch buffer can be aligned with the render time and duration of the associated media fragment; that is, the event dispatch buffer can be a copy of the event clear buffer.

In some embodiments, appending an event fragment to an event dispatch buffer can be achieved by adding the event fragment in such a way that the event start time offset of the event fragment is the event rendering time in the event dispatch buffer, and the duration of the event fragment is the event duration in the event dispatch buffer. As an example, if the application has already set this scheme to "on_start", the rendering time and duration of the event fragment in the event dispatch buffer can reflect the event start time offset and duration of the event fragment. In some embodiments, the event dispatch buffer can reflect the event start time and duration of the event fragment; that is, event fragments can be appended such that the event fragment in the buffer extends from the beginning of the event fragment to the end of the event fragment. Therefore, in some embodiments, the event clearing buffer and the event dispatch buffer may not be equivalent.

At point 835, event fragments can be dispatched to the application based on the dispatch information for the corresponding event fragment. The dispatch information may include information about the following: event fragment initialization, event association ID, event start time offset, event fragment duration, application scheme URI/value, and event clearing, event attaching, and event rewriting attributes.

Figure 8B shows an exemplary flowchart illustrating a process 850 for processing DASH and CMAF in-band events, wherein, according to an exemplary embodiment, event fragments can be dispatched from an event clear buffer and an event dispatch buffer.

At 860, an event fragment in the event dispatch buffer for a specific presentation time is identified. In some embodiments, MSE or its components may identify event fragments in the event dispatch buffer for a specific presentation time. In some embodiments, the presentation time may be determined based on the MSE API event presentation time. In other embodiments, the presentation time may be determined based on the presentation time of the media fragment. Multiple event fragments may be associated with an event association ID; that is, multiple event fragments may be associated with the same media fragment.

At 865, it is determined whether the event fragment identified in the event dispatch buffer for a specific rendering time is included in the dispatch event table. In some embodiments, the MSE or its components may determine whether the event fragment is included in the dispatch event table. The dispatch event table may store records of event fragments or event messages that have been dispatched to the application. Therefore, if an event fragment is included in the dispatch event table, the event fragment may have already been dispatched to the application. If the event fragment in the event dispatch buffer at the rendering time is not included in the dispatch event table, the event fragment may be dispatched to the application at 870 because the event fragment may not have been dispatched to the application yet. As an example, if the event ID of the event fragment in the event dispatch buffer at the rendering time does not exist in the dispatch event table, the event fragment is dispatched to the application. As another example, if the event ID of the event fragment in the event dispatch buffer at the rendering time is included in the dispatch event table, the event fragment may have already been dispatched to the application.

At 875, event fragments dispatched to the application can be added to the dispatch event table. In some embodiments, event fragments that are dispatched but not included in the dispatch event table are added to the dispatch event table. As an example, if an event fragment is dispatched to the application, the event ID of that event fragment can be added to the dispatch event table to maintain a record of the event fragment being dispatched to the application.

Figure 9 illustrates an exemplary flowchart of process 900 for processing DASH and CMAF in-band events, wherein, according to an exemplary embodiment, event fragments can be removed, cleared, or rewritten from the event clearing buffer and the event dispatch buffer. At 910, a new event fragment from the new media data is generated. In some embodiments, the MSE or its components can generate one or more new event fragments from the new media data. In some exemplary embodiments, a new event fragment can be generated for a new media fragment appended to the media buffer. As an example, when a new media fragment is parsed and appended to the media buffer, a new event fragment corresponding to the new media fragment can be appended to the event clearing buffer and the event dispatch buffer.

At 915, event segments overlapping with new event segments in the event clear buffer can be split to match the event duration, event start time offset, or rendering time of the new event segment. If the new segment is smaller than the event segments in the event clear buffer, the overlapping event segments in the event clear buffer can be segmented to match the event duration of the new event segment. As an example, if a new event segment overlapping with an event segment in the event clear buffer has a shorter event duration, the overlapping event segment in the event clear buffer can be segmented into two or more segments corresponding to the shorter event duration of the new event segment. In some embodiments, overlapping event segment segments in the event clear buffer are removed, cleared, or overwritten based on the duration of the new event segment.

At 920, event segments overlapping with new event segments in the event dispatch buffer can be split to match the event duration, event start time offset, or rendering time of the new event segment. If the new segment is smaller or shorter than the event segments in the event dispatch buffer, the overlapping event segments in the event dispatch buffer can be split to match the event duration, event start time offset, or rendering time of the new event segment. As an example, if a new event segment overlapping with an event segment in the event dispatch buffer has a shorter event duration, the overlapping event segment in the event dispatch buffer can be split into two or more segments corresponding to the shorter event duration of the new event segment. In some embodiments, overlapping event segment segments in the event dispatch buffer are removed, cleared, or rewritten based on the duration of the new event segment.

At 925, it is determined whether event fragments from the event clearing buffer and the event dispatch buffer can be removed, cleared, or overwritten. Whether event fragments from the event clearing buffer or the event dispatch buffer can be removed, cleared, or overwritten depends on whether all event fragments associated with the media fragment being overwritten have been dispatched. If an event fragment from the event clearing buffer that overlaps with the new event fragment is not associated with the same media fragment, then at 930, the event fragment from the event clearing buffer that overlaps with the new event fragment can be removed from the event dispatch buffer. At 935, the event fragment from the event clearing buffer that overlaps with the new event fragment can be removed from the event clearing buffer. As an example, if the application has set this scheme to "on_start", and if an event fragment from the event clearing buffer that overlaps with the new event fragment is not associated with the same media fragment, then the event fragment overlapping with the new event fragment can be removed from the event clearing buffer, and also from the event dispatch buffer, because there may be other events associated with the same media fragment that have already been dispatched.

In some embodiments, the step of determining whether overlapping event fragments in the event clear buffer are not associated with the same media fragment may be performed optionally. As an example, if the application has already set the scheme to "on_receive", the step of determining whether overlapping event fragments in the event clear buffer are not associated with the same media fragment may not be performed.

At 930, an event fragment overlapping with the new event fragment can be removed from the event clearing buffer. At 935, an event fragment overlapping with the new event fragment can be removed from the event dispatch buffer. At 940, according to embodiments of this disclosure, the new event fragment can be appended to both the event clearing buffer and the event dispatch buffer.

While Figures 8A, 8B, and 9 illustrate example blocks of processors 800, 850, and 900, in some implementations, processors 800, 850, and 900 may include more blocks, fewer blocks, different blocks, or blocks with different arrangements compared to those depicted in Figures 8A, 8B, and 9. In embodiments, any blocks of processors 800, 850, and 900 may be merged or arranged in any number or order as needed. In embodiments, at least two blocks of processors 800, 850, and 900 may execute in parallel.

The above-described technologies can be implemented as computer software using computer-readable instructions and physically stored in one or at least two computer-readable media. For example, Figure 10 illustrates a computer device 1000 adapted to implement certain embodiments of the disclosed subject matter.

The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by one or at least two computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through decoding, microcode, etc.

The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

The components shown in Figure 10 for the computer device 1000 are exemplary in nature and are not intended to limit the scope or functionality of the computer software used to implement the embodiments of this application. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of the computer device 1000.

Computer device 1000 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or at least two human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

The human-machine interface input device may include one or at least two of the following (only one is shown): keyboard 1001, mouse 1002, touchpad 1003, touch screen 1010, joystick 1005, microphone 1006, scanner 1008, and camera 1007.

Computer device 1000 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or at least two human users through, for example, tactile output, sound, light, and smell/taste. These HMI output devices may include tactile output devices (e.g., tactile feedback via touchscreen 1010 or joystick 1005, but may also include tactile feedback devices not used as input devices), audio output devices (e.g., speakers 1009, headphones), visual output devices (e.g., screen 1010 including cathode ray tube screens, liquid crystal screens, plasma screens, organic light-emitting diode screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which may output two-dimensional or higher-dimensional visual outputs through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes), and printers.

Computer device 1000 may also include human-accessible storage devices and related media, such as optical media including high-density read-only/rewritable optical discs (CD/DVD ROM/RW) 1020 or similar media 1011, thumb drives 1022, removable hard disk drives or solid-state drives 1023, conventional magnetic media such as magnetic tapes and floppy disks, dedicated devices based on ROM/ASIC/PLD such as security software protectors, etc.

Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

Computer device 1000 may also include interfaces 1099 providing access to one or more communication networks 1098. For example, network 1098 may be wireless, wired, or optical. Network 1098 may also be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Network 1098 also includes Ethernet, wireless LAN, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial broadcast TV), vehicular and industrial networks (including CANBus), etc. Some networks 1098 typically require external network interface adapters for connection to certain general-purpose data ports or peripheral buses (1050 and 1051) (e.g., the USB port of computer device 1000); other systems are typically integrated into the core of computer device 1000 via connection to system buses as described below (e.g., an Ethernet interface integrated into a PC computer device or a cellular network interface integrated into a smartphone computer device). By using any of these networks 1098, computer device 1000 can communicate with other entities. This communication can be unidirectional, used only for receiving (e.g., wireless television), unidirectional, used only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as via a local area or wide area digital network to other computer devices. Each of the aforementioned networks and network interfaces can use certain protocols and protocol stacks.

The aforementioned human-computer interface device, human-accessible storage device, and network interface can be connected to the core 1040 of the computer device 1000.

The core 1040 may include one or at least two central processing units (CPUs) 1041, a graphics processing unit (GPU) 1042, a dedicated programmable processing unit in the form of a field-programmable gate array (FPGA) 1043, a task-specific hardware accelerator 1044, etc. These devices, along with read-only memory (ROM) 1045, random access memory 1046, internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives, etc.) 1047, etc., can be connected via a system bus 1048. In some computer devices, the system bus 1048 can be accessed with one or at least two physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus 1048 or connected via a peripheral bus 1049. Peripheral bus architectures include external controller interfaces (PCI), universal serial buses (USB), etc.

The CPU 1041, GPU 1042, FPGA 1043, and accelerator 1044 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 1045 or RAM 1046. Transient data can also be stored in RAM 1046, while permanent data can be stored, for example, in internal mass storage 1047. Fast storage and retrieval of any memory device can be achieved through the use of a cache memory, which can be closely associated with one or more of the CPU 1041, GPU 1042, mass storage 1047, ROM 1045, RAM 1046, etc.

The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this application, or they may be media and code well-known and usable by those skilled in the art of computer software.

By way of example and not limitation, the computer device 1000 having the illustrated structure, particularly the core 1040, can serve as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to perform the function of executing software contained in one or at least two tangible computer-readable media. Such computer-readable media may be a medium associated with the aforementioned user-accessible mass storage, and a specific memory of the non-volatile core 1040, such as the core's internal mass storage 1047 or ROM 1045. Software implementing various embodiments of this application can be stored in such a device and executed by the core 1040. Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core 1040, particularly the processor therein (including a CPU, GPU, FPGA, etc.), to perform a specific process or a specific portion of a specific process described herein, including defining data structures stored in RAM 1046 and modifying such data structures according to a software-defined process. Alternatively or as an alternative, the computer device may provide logic hardwired or otherwise included in circuitry (e.g., accelerator 1044) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing the execution of software (such as an integrated circuit (IC)), circuitry containing the execution logic, or both. This application includes any suitable combination of hardware and software.

While at least two exemplary embodiments have been described in this application, various modifications, arrangements, and equivalent substitutions of the embodiments are within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are thus within its spirit and scope.

Claims (12)

1. A method for processing events in a media stream, characterized in that it includes: Obtain media data; Generate at least one event segment from the media data; The at least one event segment is attached to a first event processing buffer, the at least one event segment including an event start time offset for each of the at least one event segments, the event start time offset being determined based on the time when the event segment is attached to the first event processing buffer; The at least one event fragment is appended to a second event processing buffer, the at least one event fragment further comprising event dispatch information for each event fragment in the at least one event fragment; and, Based on the event dispatch information in the first event processing buffer and the second event processing buffer, the at least one event fragment is dispatched; The method further includes: Generate at least one new event fragment; Based on the event duration of the at least one new event fragment, split the at least one event fragment in the first event processing buffer and the second event processing buffer; Based on the event duration of the at least one new event fragment, rewrite the at least one event fragment in the first event handling buffer and the second event handling buffer; Wherein, rewriting the at least one event segment in the first event processing buffer and the second event processing buffer based on the event duration of the at least one new event segment includes: Determine that at least one event segment in the first event processing buffer is not associated with the same media segment; Delete the at least one event fragment from the second event handling buffer; Delete at least one event fragment from the first event handling buffer; The at least one new event fragment is appended to the first event handling buffer and the second event handling buffer. 2. The method according to claim 1, wherein appending the at least one event fragment to the first event processing buffer comprises: Align the rendering time and duration of each event segment in the at least one event segment with the rendering time and duration of at least one associated media segment in the media buffer. 3. The method according to claim 1, wherein appending the at least one event fragment to the second event processing buffer comprises: The additional operations of the at least one event fragment are copied to the first event processing buffer. 4. The method according to claim 1, wherein appending the at least one event fragment to the second event processing buffer comprises: Based on the event dispatch information of each of the at least one event fragments, the at least one event fragment is appended to the second event processing buffer. 5. The method according to claim 1, wherein dispatching the at least one event fragment based on the event dispatch information in the first event processing buffer and the second event processing buffer comprises: Determine whether the at least one event fragment is included in the dispatch event table; When it is determined that at least one event fragment is not included in the dispatch event table, the at least one event fragment is dispatched to the application. 6. An apparatus for processing events in a media stream, characterized in that it comprises: The first acquisition module is used to acquire media data; The first generation module is used to generate at least one event fragment from the media data; A first attachment module is configured to attach the at least one event segment to a first event processing buffer, the at least one event segment including an event start time offset for each of the at least one event segments, the event start time offset being determined based on the time when the event segment is attached to the first event processing buffer; A second append module is configured to append the at least one event fragment to a second event processing buffer, wherein the at least one event fragment further includes event dispatch information for each event fragment in the at least one event fragment; and, A first dispatch module is configured to dispatch the at least one event fragment based on the event dispatch information in the first event processing buffer and the second event processing buffer. The second generation module is used to generate at least one new event fragment; The first splitting module is used to split the at least one event segment in the first event processing buffer and the second event processing buffer based on the event duration of the at least one new event segment; A first rewrite module is configured to rewrite at least one event segment in the first event processing buffer and the second event processing buffer based on the event duration of the at least one new event segment. The first rewriting module includes: The second determining module is used to determine that the at least one event segment in the first event processing buffer is not associated with the same media segment; A first deletion module is used to delete at least one event fragment from the second event processing buffer; The second deletion module is used to delete at least one event fragment from the first event processing buffer; A third additional module is used to attach the at least one new event fragment to the first event processing buffer and the second event processing buffer. 7. The apparatus of claim 6, wherein the first additional module is configured to align the rendering time and duration of each event segment in the at least one event segment with the rendering time and duration of at least one associated media segment in the media buffer. 8. The apparatus according to claim 6, wherein the second additional module is configured to copy the additional operation of the at least one event fragment to the first event processing buffer. 9. The apparatus according to claim 6, wherein the second additional module is configured to attach the at least one event segment to the second event processing buffer based on the event dispatch information of each event segment in the at least one event segment. 10. The apparatus according to claim 6, wherein the first dispensing module comprises: A first determining module is used to determine whether the at least one event fragment is included in the dispatch event table; The second dispatch module is used to dispatch the at least one event fragment to the application when it is determined that the at least one event fragment is not included in the dispatch event table. 11. A non-transitory computer-readable medium, characterized in that it stores a program thereon that, when executed by at least one processor, implements the method as described in any one of claims 1-5. 12. A computer device, characterized in that it comprises a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method as described in any one of claims 1-5.