CN113382300B - Audio and video playing method and device - Google Patents

Abstract

The application discloses an audio and video playing method and device, wherein the method comprises the following steps: in the process of audio and video playing, a playing terminal carries out rendering synchronous control based on a display timestamp of a data frame to be rendered and playing elapsed time of data of a corresponding type, and carries out synchronous adjustment control on a special clock of the playing terminal based on source end time so as to match the source end time; the play elapsed time is obtained based on the dedicated clock. According to the method and the device, rendering synchronous control is performed through the special clock based on the playing terminal, synchronous adjustment control is performed on the special clock periodically based on the time of the source end, and the playing terminal can smoothly play audio and video.

Description

A method and device for playing audio and video

technical field

The present invention relates to the field of multimedia technologies, in particular to an audio and video playing method and device.

Background technique

As more and more TV manufacturers support the next-generation TV ATSC3.0 standard, 4K/8K high-resolution high-frame-rate programs will become the mainstream of future TV.

In the process of implementing the present invention, the inventor found that when the ATSC3.0 standard is adopted, the existing audio and video playback method used by the user terminal will have the problem that the audio and video cannot be played smoothly. The reason why the inventor finds that this problem exists through serious research and analysis is as follows:

In the existing audio and video playback method, after the broadcast operator transmits the audio and video signal to the user terminal, the user terminal needs to restore a clock that is consistent with the frequency of the broadcast operator. The initial value is the first program reference received by the user terminal. Time, and based on this clock to process data in the decoding link. In existing television standards (such as the MPEG-2 international standard specification "ISO/IEC 13818"), it is clearly stipulated that the clock frequency of the audio and video source is specifically 27 MHz. In this way, the user terminal can recover a clock that is completely consistent with the frequency of the audio and video source terminal based on standard regulations. However, in other systems such as the ATSC3.0 standard, the specific value of the clock frequency of the audio and video source is not determined. In this way, the user terminal will not be able to recover a clock that is completely consistent with the frequency of the audio and video source, resulting in the failure of the user terminal. There is a difference between the reference clock and the encoding clock of the audio and video source. This difference, after several hours or even longer time of continuous accumulation, will eventually lead to data starvation in the user terminal (that is, because the reference clock is faster than the encoding clock of the audio and video source, the playback progress is faster than the source end The speed at which data is generated, which leads to the need to wait for the arrival of audio and video data during playback) or data accumulation (that is, because the reference clock is slower than the encoding clock at the source end, the playback progress is slower than the speed at which data is generated at the source end, resulting in Audio and video data accumulate on the user terminal side), resulting in audio and video cannot be played smoothly.

Contents of the invention

In view of this, the main purpose of the present invention is to provide a method and device for playing audio and video, which can enable smooth playback of audio and video.

In order to achieve the above object, the technical scheme proposed in the embodiment of the present invention is:

A method for playing audio and video, comprising:

During audio and video playback, the playback terminal performs rendering synchronization control based on the display timestamp of the data frame to be rendered and the playback elapsed time of the corresponding type of data, and performs synchronous adjustment control on the dedicated clock of the playback terminal based on the source time, to match the source time; the playback elapsed time is obtained based on the dedicated clock.

Preferably, said rendering synchronization control includes:

When the data frame to be rendered arrives at the renderer, if Frame _pts ≤ STC _elapsed ≤ Frame _pts + D _max is satisfied, rendering is performed based on the current data frame to be rendered; where, STC _elapsed is the current elapsed playback time, and Frame _pts is The display timestamp of the current data frame to be rendered, D _max is the preset maximum allowed delay display time; 0≤D _max ≤Frame _duration ; Frame _duration is the display duration of a single data frame;

If STC _elapsed <Frame _pts is satisfied, render based on the current data frame to be rendered after the waiting time △t, △t=Frame _pts -STC _elapsed ;

If STC _elapsed >Frame _pts +D _max is satisfied, the current data frame to be rendered is discarded.

Preferably, the acquisition of the playing elapsed time includes:

When the data frame to be rendered arrives at the renderer, obtain the current counting time of the dedicated clock; wherein, the counting time of the dedicated clock is a monotonically increasing value;

Calculate the difference between the counted time and the first initial time to obtain the elapsed playback time; the first initial time is the counted time acquired when the first data frame played by the audio and video arrives at the renderer.

Preferably, the synchronous adjustment control of the dedicated clock includes:

When the preset synchronization adjustment time arrives, the playback terminal acquires the current source time SrcTime _curr and the current counting time STC _curr of the dedicated clock, wherein the counting time STC _curr and the source time SrcTime _curr Both are monotonically increasing values;

Calculate the difference between the counting time STC _curr and the second initial time STC _init to obtain the first relative time STC _diff ; calculate the difference between the source time SrcTime _curr and the third initial time SrcTime _init to obtain the second relative time SrcTime _diff ; wherein, the second initial time STC _init is the counting time obtained when the first synchronization adjustment moment arrives; the third initial time SrcTime _init is the count time obtained when the first synchronization adjustment moment arrives source time;

Calculate the difference between the first relative time STC _diff and the second relative time SrcTime _diff to obtain the time error Time _error of the dedicated clock relative to the audio and video source clock;

If the time error Time _error is greater than a preset maximum error threshold, fine-tune the step size according to the preset first frequency to reduce the output frequency of the dedicated clock; the maximum error threshold is greater than zero;

If the time error Time _error is smaller than a preset minimum error threshold, fine-tune the step according to a preset second frequency to increase the output frequency of the dedicated clock; the minimum error threshold is smaller than zero.

Preferably, the acquisition of the current counting time of the dedicated clock includes:

Obtain the current clock count value W _{ctr_curr} from the dedicated clock;

If the current last clock count value W _{ctr_pre} is a preset initial value, then update the last clock count value W _{ctr_pre} to the current clock count value W _{ctr_curr} ;

If W _{ctr_curr} <W _{ctr_pre} is satisfied, add one to the current number of revolutions; the initial value of the number of revolutions is 0;

According to W _{ctr_accu} =W _{ctr_curr} +N×W _{ctr_max} , calculate clock accumulative count W _{ctr_accu} ; wherein, N is the current number of revolutions; updating the last clock count value W _{ctr_pre} is the current clock count value W _{ctr_curr} ; W _{ctr_max} is the The maximum clock count value of the dedicated clock;

The clock cumulative count W _{ctr_accu} is divided by the current clock frequency of the dedicated clock to obtain the current counting time of the dedicated clock.

The embodiment of the present invention discloses an audio and video playback device, including: a dedicated clock, a rendering control unit, and a clock synchronization unit; wherein,

The rendering control unit is used to perform synchronous rendering control based on the display time stamp of the data frame to be rendered and the playback elapsed time of the corresponding type of data during the audio and video playback process, and the playback elapsed time is obtained based on a dedicated clock of the playback terminal ;

The clock synchronization unit is configured to perform synchronous adjustment control on the dedicated clock based on the source time during the audio and video playing process, so as to match the source time.

Preferably, the rendering control unit is specifically configured to perform the rendering synchronization control, including:

When the data frame to be rendered arrives at the renderer, if Frame _pts ≤ STC _elapsed ≤ Frame _pts + D _max is satisfied, rendering is performed based on the current data frame to be rendered; where, STC _elapsed is the current elapsed playback time, and Frame _pts is The display timestamp of the current data frame to be rendered, D _max is the preset maximum allowed delay display time; 0≤D _max ≤Frame _duration ; Frame _duration is the display duration of a single data frame;

If STC _elapsed <Frame _pts is satisfied, render based on the current data frame to be rendered after the waiting time △t, △t=Frame _pts -STC _elapsed ;

If STC _elapsed >Frame _pts +D _max is satisfied, the current data frame to be rendered is discarded.

Preferably, the rendering control unit is specifically configured to acquire the elapsed playback time, including:

When the data frame to be rendered arrives at the renderer, obtain the current counting time of the dedicated clock; wherein, the counting time of the dedicated clock is a monotonically increasing value;

Calculate the difference between the counted time and the first initial time to obtain the elapsed playback time; the first initial time is the counted time acquired when the first data frame played by the audio and video arrives at the renderer.

Preferably, the clock synchronization unit is specifically configured to perform synchronous adjustment control on the dedicated clock, including:

When the preset synchronization adjustment time arrives, obtain the current source time SrcTime _curr and the current counting time STC _curr of the dedicated clock, wherein both the counting time STC _curr and the source time SrcTime _curr are monotonically increasing value;

Calculate the difference between the counting time STC _curr and the second initial time STC _init to obtain the first relative time STC _diff ; calculate the difference between the source time SrcTime _curr and the third initial time SrcTime _init to obtain the second relative time SrcTime _diff ; wherein, the second initial time STC _init is the counting time obtained when the first synchronization adjustment moment arrives; the third initial time SrcTime _init is the count time obtained when the first synchronization adjustment moment arrives source time;

Calculate the difference between the first relative time STC _diff and the second relative time SrcTime _diff to obtain the time error Time _error of the dedicated clock relative to the audio and video source clock;

If the time error Time _error is greater than a preset maximum error threshold, fine-tune the step size according to the preset first frequency to reduce the output frequency of the dedicated clock; the maximum error threshold is greater than zero;

If the time error Time _error is smaller than a preset minimum error threshold, fine-tune the step according to a preset second frequency to increase the output frequency of the dedicated clock; the minimum error threshold is smaller than zero.

Preferably, the device further includes a dedicated timing unit;

The timing unit is configured to determine the current counting time of the dedicated clock according to the time acquisition instruction of the rendering control unit or the clock synchronization unit, and feed it back to the corresponding unit; wherein, determine the current counting time of the dedicated clock Time includes:

Obtain the current clock count value W _{ctr_curr} from the dedicated clock;

If the current last clock count value W _{ctr_pre} is a preset initial value, then update the last clock count value W _{ctr_pre} to the current clock count value W _{ctr_curr} ;

If W _{ctr_curr} <W _{ctr_pre} is satisfied, add one to the current number of revolutions; the initial value of the number of revolutions is 0;

According to W _{ctr_accu} =W _{ctr_curr} +N×W _{ctr_max} , calculate clock accumulative count W _{ctr_accu} ; wherein, N is the current number of revolutions; updating the last clock count value W _{ctr_pre} is the current clock count value W _{ctr_curr} ; W _{ctr_max} is the The maximum clock count value of the dedicated clock;

The clock cumulative count W _{ctr_accu} is divided by the current clock frequency of the dedicated clock to obtain the current counting time of the dedicated clock.

The embodiment of the invention also discloses an audio and video playback device, including a processor and a memory;

An application program that can be executed by the processor is stored in the memory, and is used to make the processor execute the above-mentioned audio and video playing method.

The embodiment of the present invention also discloses a computer-readable storage medium, in which computer-readable instructions are stored, and the computer-readable instructions are used to execute the above-mentioned audio and video playing method.

It can be seen from the above technical solutions that the audio and video playback scheme proposed in the embodiment of the present invention, in the audio and video playback process, the dedicated clock of the playback terminal is synchronously adjusted and controlled based on the source time, which can ensure the playback terminal's time during the audio and video playback process. The dedicated clock matches the encoding clock at the source, and in the rendering process, based on the display timestamp of the data frame to be rendered and the playback elapsed time based on the dedicated clock of the playback terminal, rendering synchronization control is performed. In this way, on the one hand, a dedicated clock that matches the source time can be used to achieve smooth playback. On the other hand, by synchronizing in the rendering process, it can meet the audio and video playback needs with high real-time requirements such as live broadcast. In addition, the synchronous adjustment and control of the dedicated clock of the playback terminal based on the source time can realize the decoupling between the source clock and the playback clock, and the rendering processing link has lower requirements for clock accuracy than the encoding clock accuracy requirements of the source. , allowing for tolerance errors in audio and video rendering. In this way, the clock frequency of the playback end does not need to be as high as the source clock frequency (such as reaching 1GHz), and the reduction of the clock oscillation frequency also means the reduction of power consumption, which can reduce the playback Implementation cost and power consumption of the end-specific clock.

Description of drawings

FIG. 1 is a schematic flowchart of an audio and video playback method according to an embodiment of the present invention.

Fig. 2 is a schematic flow chart of a method for obtaining counting time by a dedicated clock according to an embodiment of the present invention;

3 is a schematic flow chart of a synchronous adjustment control method for a playback terminal dedicated clock according to an embodiment of the present invention;

FIG. 4 is a structural diagram of an audio and video playback device according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Considering that many programs played by broadcasting operators have strong real-time performance, and the conventional method of synchronizing with buffering in the decoding stage, on the one hand, the audio and video playback time will be prolonged, and the real-time performance of audio and video playback cannot be guaranteed. On the one hand, the smoothness of audio and video cannot be guaranteed. For this reason, this application will perform rendering control based on the dedicated clock of the user terminal in the rendering link of audio and video playback, and periodically adjust the dedicated clock based on the source time of the source during the playback process. In this way, the receiving The speed at which the audio and video are played on the end is the same as the speed at which the source generates audio and video, so that the audio and video can be played smoothly.

Fig. 1 is the schematic flow chart of audio and video playing method of the embodiment of the present invention, as shown in Fig. 1, this embodiment mainly comprises:

Step 101. During audio and video playback, the playback terminal performs rendering synchronization control based on the display timestamp of the data frame to be rendered and the playback elapsed time of the corresponding type of data. The playback elapsed time is obtained based on the dedicated clock of the playback terminal.

In this step, after the playback terminal starts the audio and video playback process, it will perform synchronous rendering control on each data frame that enters the rendering link after demultiplexing and audio and video decoder processing, so as to achieve a smooth playback effect.

Here, compared with the traditional cache synchronization method used in the decoding link, the synchronization control of the rendering link can satisfy the real-time performance of audio and video playback on the one hand, and achieve smooth audio and video playback on the other hand.

The playback elapsed time is used to represent: in the audio and video playback process, after the first audio frame or video arrives at the renderer of the playback terminal for rendering processing, the playback duration of the corresponding type of data recorded based on the time of the dedicated clock.

Preferably, in a specific implementation manner, the following method may be used to obtain the playback elapsed time:

When the data frame to be rendered arrives at the renderer, obtain the current counting time of the dedicated clock; wherein, the counting time of the dedicated clock is a monotonically increasing value; calculate the difference between the counting time and the first initial time, Obtain the playback elapsed time.

The first initial time is the count time obtained when the first data frame of the same type as the data frame to be rendered arrives at the renderer during the audio and video playback. Thus, the initial value of the playback elapsed time is zero. Since the counting time of the dedicated clock is an incremental time value, the playback elapsed time is a time incremented from zero.

Preferably, in one embodiment, in order to accurately obtain the counting time of the dedicated clock, as shown in Figure 2, the following method can be used to obtain the counting time:

Step a1. Obtain the current clock count value W _{ctr_curr} from the dedicated clock of the playback terminal.

It should be noted here that the dedicated clock in this step generates pulses and counts through hardware, and the corresponding counting time can be obtained by dividing the counted value by the frequency value. Clock hardware typically uses a limited number of register bits to store clock counts. Assuming that the number of registers is M and the clock frequency is Clock _Freq , then the maximum time that can be expressed is (2 ^M -1)/Clock _Freq , and when the maximum value is reached, it will turn around and start from 0 again. For example, a register with 33 bits and 90KHz is For example, the maximum value is (2 ³³ -1)/90000, about 95,443 seconds, 26.5 hours. In order to make the counting time of the special clock increase monotonically, the above-mentioned rotation situation needs to be dealt with when counting the time, that is, the number of rotations needs to be considered when counting the time.

Step a2. If the current previous clock count value W _{ctr_pre} is a preset initial value, update the previous clock count value W _{ctr_pre} to the current clock count value W _{ctr_curr} .

This step is used to update the previous clock count value W _{ctr_pre} to the current clock count value W _{ctr_curr} when the count time is acquired for the first time.

Specifically, the initial value of W _{ctr_pre} can be set by those skilled in the art according to actual needs, for example, it can be zero, but it is not limited thereto.

Step a3. If W _{ctr_curr} <W _{ctr_pre} is satisfied, add one to the current number of revolutions.

The initial value of the number of revolutions is 0.

Here, if W _{ctr_curr} < W _{ctr_pre} , it means that a counting rotation has occurred currently, so it is necessary to add one to the number of rotations.

Step a4 , according to W _{ctr_accu} =W _{ctr_curr} +N×W _{ctr_max} , calculate clock accumulation count W _{ctr_accu} .

Wherein, N is the current number of revolutions; the updated previous clock count value W _{ctr_pre} is the current clock count value W _{ctr_curr} ; W _{ctr_max} is the maximum clock count value of the dedicated clock.

Here, when calculating the accumulated clock count W _{ctr_accu} , the current number of revolutions is taken into account, so it can be guaranteed that the counting time of the dedicated clock is monotonically increasing.

Step a5: Divide the clock cumulative count W _{ctr_accu} by the current clock frequency of the dedicated clock to obtain the current counting time of the dedicated clock.

The presentation time stamp (Presentation Time Stamp, PTS) of the data frame can specifically adopt the existing calculation method, that is, calculate the presentation time stamp Frame _pts of the data frame according to Frame _pts = (n-1) × Frame _duration , n is the number of the audio data frame or the video data frame, and the number is the number of the same type of data frame, which means the nth audio data frame or the nth video data frame. Taking video frame data as an example, the display timestamp of the first video frame is 0, and subsequent video frames are accumulated through Frame _duration . For example, for a 60fps video, the Frame _duration is 16.667ms, the Frame _pts of the first video frame is 0ms, the Frame _pts of the second video frame is 16.667ms, and the Nth frame Frame_pts is (N-1)×16.667ms, therefore, Frame _pts is a time value incrementing from 0. In this way, by comparing the display time stamp Frame _pts with the playback elapsed time that also increments from zero, it can be known whether the time when the data frame to be rendered arrives at the renderer matches the playback speed.

Specifically, in step 101, the display time stamp of the data frame to be rendered is compared with the elapsed playback time of the corresponding type of data, and rendering synchronization control is performed according to the comparison result.

In order to obtain a better smooth playback effect, in an implementation manner, the following method may be used for rendering synchronization control:

When the data frame to be rendered arrives at the renderer, if Frame _pts ≤ STC _elapsed ≤ Frame _pts + D _max is satisfied, rendering is performed based on the current data frame to be rendered; where, STC _elapsed is the current elapsed playback time, and Frame _pts is The display timestamp of the current data frame to be rendered, D _max is the preset maximum allowed delay display time; 0≤D _max ≤Frame _duration ; Frame _duration is the display duration of a single data frame;

If STC _elapsed <Frame _pts is satisfied, render based on the current data frame to be rendered after the waiting time △t, △t=Frame _pts -STC _elapsed ;

If STC _elapsed >Frame _pts +D _max is satisfied, the current data frame to be rendered is discarded.

Using the method above, the smooth playback of audio frame data and the smooth playback of video frame data can be realized respectively. Among them, if Frame _pts ≤ STC _elapsed ≤ Frame _pts +D _max is satisfied, it means that the current moment meets the requirements of smooth playback and is suitable for rendering the data frame to be rendered. Therefore, rendering will be performed based on the current data frame to be rendered immediately.

The D _max satisfies: 0 ≤ D _max ≤ Frame _duration . Specifically, those skilled in the art can set an appropriate value of D _max according to the actual smoothing requirements. Preferably, in order to meet the smooth display of the screen while reducing the impact on the data frame For discarding, D _max =Frame _duration can be set.

If STC _elapsed < Frame _pts is satisfied, it means that the current data frame to be rendered reaches the renderer earlier than the speed of smooth playback, so it is necessary to wait for a certain interval (ie Frame _pts -STC _elapsed ) before rendering based on the current data frame to be rendered. In order to achieve the effect of smooth playback.

If STC _elapsed >Frame _pts +D _max is satisfied, it means that the current data frame to be rendered arrives at the renderer later than the speed of smooth playback. In this way, the current data frame to be rendered has missed the playback moment corresponding to the smooth playback requirement. Therefore, The current data frame to be rendered needs to be discarded.

In the above method, Frame _pts ≤ STC _elapsed ≤ Frame _pts + D _max is currently suitable for immediate rendering, so that there is a certain tolerable error between the dedicated clock of the playback terminal and the source clock, and there is no need to communicate with the source The accuracy of the encoding clock is exactly the same, and the accuracy reaches milliseconds and above, that is, the clock frequency is greater than or equal to 1KHz, and smooth audio and video playback can be realized, which can reduce the accuracy requirements of the dedicated clock, thereby reducing the power consumption and production cost of the dedicated clock. The applicability of the method is improved.

Step 102. During the audio and video playback process, the playback terminal performs synchronous adjustment control on the dedicated clock based on the source time to match the source time.

In this step, in order to ensure the accuracy of rendering synchronization control based on the dedicated clock of the playback terminal in the above step 101, during the audio and video playback process, the dedicated clock is synchronously adjusted and controlled based on the source time to match the source time, That is, the dedicated clock matches the encoding clock of the audio and video source.

Here, since the dedicated clock of the playback terminal is a reference clock for synchronization in the rendering process, and each data frame has a certain display duration, the playback time of the data frame has a certain tolerable floating range. The precision requirements of the dedicated clock do not need to be as high-precision as the encoding clock at the source. It is only necessary to ensure that the time error between the dedicated clock and the source clock does not affect the viewing experience of the human eye. Therefore, by using playback in the rendering link The dedicated clock of the terminal is adjusted synchronously, and the dedicated clock is adjusted and controlled synchronously during the playback process, so that the dedicated clock matches the source clock, which can realize the decoupling of the playback terminal and the source clock, thereby reducing the cost of the playback terminal. Implementation cost of a dedicated clock.

Preferably, in order to enable the dedicated clock to be better synchronized with the source clock, in one embodiment, as shown in Figure 3, the following method can be used to periodically adjust and control the dedicated clock of the playback terminal:

Step 1021. When the preset synchronization adjustment time arrives, the playback terminal acquires the current source time SrcTime _curr and the current count time STC _curr of the dedicated clock.

Wherein, both the counting time STC _curr and the source time SrcTime _curr are monotonically increasing values, so that the relative times calculated in subsequent steps based on these two parameters are also increasing values.

In this step, when each synchronization adjustment time arrives, it is necessary to obtain the current source time SrcTime _curr and the current counting time of the dedicated clock at the same time, so that the current dedicated clock can be further determined based on the two currently obtained times in subsequent steps. The time error of the clock relative to the clock of the audio and video source end, and then determine whether the dedicated clock needs to be adjusted synchronously according to the time error, so that the time of the dedicated clock matches the time of the source end.

In practical applications, those skilled in the art can set the synchronization adjustment time according to the actual timing adjustment strategy, for example, the synchronization adjustment time can be determined according to the preset synchronization adjustment cycle. For the synchronization adjustment period, if the setting is too long, the timeliness of the synchronization adjustment cannot be satisfied; if the setting is too short, too much control overhead will be generated. Specifically, the error between the actual dedicated clock and the source clock can be considered by those skilled in the art The time accumulation rate, according to the strategy of minimizing the control overhead while satisfying the timeliness of synchronization adjustment, sets an appropriate cycle length, such as 10s, but not limited to this.

Here, the current counting time of the dedicated clock can be obtained by the same method as that in step 101 above, which will not be repeated here.

It should be noted that the source time will be sent to the playback terminal at regular intervals by the source, so that the playback terminal can obtain the source time by parsing the source signal.

Step 1022: Calculate the difference between the counting time STC _curr and the second initial time STC _init to obtain the first relative time STC _diff ; calculate the difference between the source time SrcTime _curr and the third initial time SrcTime _init to obtain the first relative time STC diff Two relative time SrcTime _diff .

Wherein, the second initial time STC _init is the counting time obtained when the first synchronization adjustment moment arrives; the third initial time SrcTime _init is the source time obtained when the first synchronization adjustment moment arrives .

In this step, the first relative time STC _diff represents the time interval between the currently obtained counting time of the dedicated clock and the counting time obtained when the first synchronization adjustment moment arrives, and the second relative time SrcTime _diff represents the currently obtained source The time interval between the time and the source time obtained when the first synchronization adjustment moment arrives. The difference between the first relative time and the second relative time may reflect whether the dedicated clock matches the source clock.

Step 1023, calculate the difference between the first relative time STC _diff and the second relative time SrcTime _diff , and obtain the time _error Time error of the dedicated clock relative to the audio and video source clock.

Here, if the dedicated clock matches the source clock, the difference between the first relative time and the second relative time (that is, the time error Time _error ) will be small; otherwise, the difference between the two will be large.

Step 1024, if the time error Time _error is greater than the preset maximum error threshold, fine-tune the step size according to the preset first frequency to reduce the output frequency of the dedicated clock; the maximum error threshold is greater than zero;

If the time error Time _error is smaller than a preset minimum error threshold, fine-tune the step according to a preset second frequency to increase the output frequency of the dedicated clock; the minimum error threshold is smaller than zero.

Here, if the time error Time _error is greater than the preset maximum error threshold, and the maximum error threshold is greater than zero, it means that the dedicated clock is faster than the source clock. At this time, the output frequency of the dedicated clock needs to be reduced to slow down the timing of the dedicated clock. , so that the dedicated clock matches the source clock. If the time error Time _error is less than the preset minimum error threshold, and the minimum error threshold is less than zero, it means that the dedicated clock is slower than the source clock. At this time, the output frequency of the dedicated clock needs to be increased to speed up the timing of the dedicated clock. , so that the dedicated clock matches the source clock. If the time error Time _error is between the minimum error threshold and the maximum error threshold, it means that the current dedicated clock matches the source clock, and the frequency of the dedicated clock does not need to be adjusted.

Specifically, the first frequency fine-tuning step is used to control the magnitude of the reduction of the output frequency of the dedicated clock each time, and the second frequency fine-tuning step is used to control the output frequency of the dedicated clock each time The magnitude of the increase, specifically, those skilled in the art can set the values of the first frequency fine-tuning step size and the second frequency fine-tuning step size according to the actual adjusted convergence speed requirements.

As for the minimum error threshold and the maximum error threshold, those skilled in the art can set them according to the matching degree requirements of the dedicated clock and the source clock in practical applications. Preferably, the minimum error threshold and the maximum error threshold have the same absolute value, but not limited thereto.

It can be seen from the above method embodiments that the above audio and video playback method performs synchronous adjustment and control on the dedicated clock of the playback terminal based on the source end time during the audio and video playback process, so as to ensure that the dedicated clock of the playback terminal matches the encoding clock of the source end , and in the rendering process, based on the display time stamp of the data frame to be rendered and the playback elapsed time of the corresponding type of data obtained based on the dedicated clock of the playback terminal, the rendering synchronization control is performed. In this way, on the one hand, a dedicated clock that matches the source time can be used to achieve smooth playback. On the other hand, by synchronizing in the rendering process, it can meet the audio and video playback needs with high real-time requirements such as live broadcast. In addition, the synchronous adjustment and control of the dedicated clock of the playback terminal based on the source time can realize the decoupling between the source clock and the playback clock, and the rendering processing link has lower requirements for clock accuracy than the encoding clock accuracy requirements of the source. , tolerable errors in audio and video rendering are allowed. In this way, the clock frequency of the playback end does not need to be as high as the source clock, thereby reducing the implementation cost and power consumption of the dedicated clock at the playback end.

Corresponding to the embodiment of the above audio and video playback method, the embodiment of the present invention also discloses an audio and video playback device, as shown in FIG. 4 , the device at least includes: a rendering control unit 401, a dedicated clock 402 and a clock synchronization unit 403; in,

The rendering control unit 401 is configured to perform synchronous rendering control based on the display timestamp of the data frame to be rendered and the playback elapsed time of the corresponding type of data during the audio and video playback process, and the playback elapsed time is based on the dedicated clock of the playback terminal get.

The clock synchronization unit 403 is configured to perform synchronous adjustment control on the dedicated clock based on the source time during the audio and video playback, so as to match the source time.

In one implementation manner, the rendering control unit 401 is specifically configured to perform the rendering synchronization control, including:

When the data frame to be rendered arrives at the renderer, if Frame _pts ≤ STC _elapsed ≤ Frame _pts + D _max is satisfied, rendering is performed based on the current data frame to be rendered; where, STC _elapsed is the current elapsed playback time, and Frame _pts is The display timestamp of the current data frame to be rendered, D _max is the preset maximum allowed delay display time; 0≤D _max ≤Frame _duration ; Frame _duration is the display duration of a single data frame;

If STC _elapsed <Frame _pts is satisfied, render based on the current data frame to be rendered after the waiting time △t, △t=Frame _pts -STC _elapsed ;

If STC _elapsed >Frame _pts +D _max is satisfied, the current data frame to be rendered is discarded.

In one implementation manner, the rendering control unit 401 is specifically configured to obtain the elapsed playback time, including:

When the data frame to be rendered arrives at the renderer, obtain the current counting time of the dedicated clock; wherein, the counting time of the dedicated clock is a monotonically increasing value;

Calculate the difference between the counted time and the first initial time to obtain the elapsed playback time; the first initial time is the counted time acquired when the first data frame played by the audio and video arrives at the renderer.

In one implementation manner, the clock synchronization unit 402 is specifically configured to perform synchronous adjustment control on the dedicated clock, including:

When the preset synchronization adjustment time arrives, obtain the current source time SrcTime _curr and the current counting time STC _curr of the dedicated clock, wherein both the counting time STC _curr and the source time SrcTime _curr are monotonically increasing value;

Calculate the difference between the counting time STC _curr and the second initial time STC _init to obtain the first relative time STC _diff ; calculate the difference between the source time SrcTime _curr and the third initial time SrcTime _init to obtain the second relative time SrcTime _diff ; wherein, the second initial time STC _init is the counting time obtained when the first synchronization adjustment moment arrives; the third initial time SrcTime _init is the count time obtained when the first synchronization adjustment moment arrives source time;

Calculate the difference between the first relative time STC _diff and the second relative time SrcTime _diff to obtain the time error Time _error of the dedicated clock relative to the audio and video source clock;

If the time error Time _error is greater than a preset maximum error threshold, fine-tune the step size according to the preset first frequency to reduce the output frequency of the dedicated clock; the maximum error threshold is greater than zero;

If the time error Time _error is smaller than a preset minimum error threshold, fine-tune the step according to a preset second frequency to increase the output frequency of the dedicated clock; the minimum error threshold is smaller than zero.

In one embodiment, the device further includes a dedicated timing unit 404;

The timing unit 404 is configured to determine the current counting time of the dedicated clock according to the time acquisition instruction of the rendering control unit or the clock synchronization unit, and feed it back to the corresponding unit; wherein, determining the current counting time of the dedicated clock Counting times include:

Obtain the current clock count value W _{ctr_curr} from the dedicated clock;

If the current last clock count value W _{ctr_pre} is a preset initial value, then update the last clock count value W _{ctr_pre} to the current clock count value W _{ctr_curr} ;

If W _{ctr_curr} <W _{ctr_pre} is satisfied, add one to the current number of revolutions; the initial value of the number of revolutions is 0;

According to W _{ctr_accu} =W _{ctr_curr} +N×W _{ctr_max} , calculate clock accumulative count W _{ctr_accu} ; wherein, N is the current number of revolutions; updating the last clock count value W _{ctr_pre} is the current clock count value W _{ctr_curr} ; W _{ctr_max} is the The maximum clock count value of the dedicated clock;

The clock cumulative count W _{ctr_accu} is divided by the current clock frequency of the dedicated clock to obtain the current counting time of the dedicated clock.

Corresponding to the above audio and video playback method embodiments, the embodiment of the present invention also discloses an audio and video playback device, including a processor and a memory;

An application program that can be executed by the processor is stored in the memory, and is used to make the processor execute the above-mentioned audio and video playing method.

Wherein, the memory may specifically be implemented as various storage media such as electrically erasable programmable read-only memory (EEPROM), flash memory (Flash memory), and programmable program read-only memory (PROM). A processor may be implemented to include one or more central processing units or one or more field programmable gate arrays integrating one or more central processing unit cores. Specifically, the central processing unit or central processing unit core may be implemented as a CPU or an MCU.

It should be noted that not all steps and modules in the above-mentioned processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only to facilitate the description of the functional division adopted. In actual implementation, one module can be divided into multiple modules, and the functions of multiple modules can also be realized by the same module. These modules can be located in the same device. , or on a different device.

The hardware modules in the various embodiments may be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuits or logic devices (such as special-purpose processors, such as FPGAs or ASICs) to perform specific operations. Hardware modules may also include programmable logic devices or circuits (eg, including general-purpose processors or other programmable processors) temporarily configured by software to perform particular operations. As for implementing the hardware module in a mechanical way, using a dedicated permanent circuit, or using a temporarily configured circuit (such as configured by software) to realize the hardware module, it can be decided according to cost and time considerations.

The embodiment of the present invention also discloses a computer-readable storage medium, in which computer-readable instructions are stored, and the computer-readable instructions are used to execute the above-mentioned audio and video playing method.

Specifically, a system or device equipped with a storage medium may be provided, on which the software program code for realizing the functions of any implementation manner in the above-mentioned embodiments is stored, and the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium. In addition, an operating system or the like operated on a computer may also complete part or all of the actual operations through instructions based on program codes. It is also possible to write the program code read from the storage medium into the memory set in the expansion board inserted into the computer or into the memory set in the expansion unit connected to the computer, and then based on the instructions of the program code, the memory installed in the The expansion board or the CPU on the expansion unit executes part or all of the actual operations, so as to realize the functions of any one of the above-mentioned implementation manners.

Embodiments of storage media for providing program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), Tape, non-volatile memory card, and ROM. Alternatively, the program code can be downloaded from a server computer or cloud by a communication network.

In this article, "schematic" means "serving as an example, example or illustration", and any illustration or implementation described as "schematic" should not be interpreted as a more preferred or more advantageous Technical solutions. In order to make the drawings concise, the figures in each figure only schematically show the relevant parts of the present invention, and do not represent the actual structure of the product. In addition, to make the drawings concise and easy to understand, in some drawings, only one of the components having the same structure or function is schematically shown, or only one of them is marked. Herein, "one" does not mean limiting the number of relevant parts of the present invention to "only one", and "one" does not mean excluding the case that the number of relevant parts of the present invention is "more than one". In this article, "upper", "lower", "front", "rear", "left", "right", "inner", "outer", etc. are only used to indicate the relative positional relationship between related parts, and The absolute positions of these relative parts are not limited.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (8)

1. An audio/video playing method is characterized by comprising the following steps:

in the audio and video playing process, the playing terminal carries out rendering synchronous control based on the display timestamp of the data frame to be rendered and the playing elapsed time of the data of the corresponding type, and carries out synchronous adjustment control on a special clock of the playing terminal based on the source end time so as to match the source end time; the play elapsed time is obtained based on the dedicated clock; the special clock generates pulses through hardware and counts the pulses, and the count value is divided by the frequency value to obtain corresponding counting time;

wherein the acquiring of the play elapsed time includes:

when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased;

calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time; the first initial time is the counting time obtained when the first data frame of the audio and video playing reaches a renderer;

the step of performing synchronous adjustment control on the dedicated clock of the play terminal includes:

when the preset synchronous adjustment time arrives, the broadcastThe playing terminal obtains the current source terminal time SrcTime _curr And the current count time STC of the special clock _curr Wherein the count time STC _curr And the source time SrcTime _curr Are all monotonically increasing values;

calculating the count time STC _curr And a second initial time STC _init To obtain a first relative time STC _diff (ii) a Calculating the source time SrcTime _curr And a third initial time SrcTime _init To obtain a second relative time SrcTime _diff (ii) a Wherein the second initial time STC _init The count time obtained when the first synchronization adjustment time arrives; the third initial time SrcTime _init The source end time obtained when the first synchronous adjustment moment arrives is obtained;

calculating the first relative time STC _diff And a second relative time SrcTime _diff Obtaining the Time error Time of the special clock relative to the audio/video source end clock _error ；

If the Time error Time _error If the error is larger than the preset maximum error threshold, the step length is finely adjusted according to a preset first frequency, and the output frequency of the special clock is reduced; the maximum error threshold is greater than zero;

if the Time error Time _error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.

2. The method according to claim 1, wherein the performing rendering synchronization control includes:

when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied _pts ≤STC _elapsed ≤Frame _pts +D _max Rendering is carried out based on the current data frame to be rendered; wherein, STC _elapsed Frame for the current elapsed time of the playback _pts Display time stamp for the current data frame to be rendered, D _max Displaying the time for a preset maximum allowable delay; d is not less than 0 _max ≤Frame _duration ；Frame _duration A display duration for a single data frame;

if STC is satisfied _elapsed <Frame _pts Then rendering is performed after waiting time Δ t based on the current data Frame to be rendered, Δ t = Frame _pts -STC _elapsed ；

If STC is satisfied _elapsed >Frame _pts +D _max Then the current data frame to be rendered is discarded.

3. The method of claim 1, wherein obtaining the current count time of the dedicated clock comprises:

obtaining a current clock count value W from the dedicated clock _{ctr_curr} ；

If the current last clock count value W _{ctr_pre} If the value is the preset initial value, the last clock count value W is updated _{ctr_pre} Is the current clock count value W _{ctr_curr} ；

If W is satisfied _{ctr_curr} ＜W _{ctr_pre} Adding one to the current revolution times; the initial value of the revolution number is 0;

according to W _{ctr_accu} ＝W _{ctr_curr} +N×W _{ctr_max} Calculating the cumulative count W of the clock _{ctr_accu} (ii) a Wherein N is the current revolution number; updating the last clock count value W _{ctr_pre} Is the current clock count value W _{ctr_curr} ；W _{ctr_max} A maximum clock count value for the dedicated clock;

cumulatively counting the clock by W _{ctr_accu} And dividing the current clock frequency of the special clock to obtain the current counting time of the special clock.

4. An audio-video playback device, comprising: the system comprises a special clock, a rendering control unit and a clock synchronization unit; wherein,

the rendering control unit is used for performing rendering synchronization control based on a display timestamp of a data frame to be rendered and the playing elapsed time of corresponding type data in the audio and video playing process, wherein the playing elapsed time is obtained based on a special clock of a playing terminal; the special clock generates pulses through hardware and counts the pulses, and the count value is divided by the frequency value to obtain corresponding counting time;

the clock synchronization unit is used for carrying out synchronous adjustment control on the special clock based on source end time in the audio and video playing process so as to match the source end time;

the rendering control unit is specifically configured to acquire the play elapsed time, and includes:

when the data frame to be rendered reaches a renderer, acquiring the current counting time of the special clock; wherein the counting time of the special clock is monotonically increased;

calculating the difference value between the counting time and the first initial time to obtain the playing elapsed time; the first initial time is the counting time obtained when the first data frame of the audio and video playing reaches a renderer;

the clock synchronization unit is specifically configured to perform synchronization adjustment control on a dedicated clock of the playback terminal, and includes:

when the preset synchronous adjustment moment is reached, the current source end time SrcTime is acquired _curr And the current count time STC of the special clock _curr Wherein the count time STC _curr And the source time SrcTime _curr Are all monotonically increasing values;

calculating the count time STC _curr And a second initial time STC _init To obtain a first relative time STC _diff (ii) a Calculating the source time SrcTime _curr And a third initial time SrcTime _init To obtain a second relative time SrcTime _diff (ii) a Wherein the second initial time STC _init The count time obtained when the first synchronization adjustment time arrives; the third initial time SrcTime _init The source end time obtained when the first synchronous adjustment moment arrives is obtained;

calculating the first relative time STC _diff And a second relative time SrcTime _diff Obtaining the Time error Time of the special clock relative to the audio/video source end clock _error ；

If the Time error Time _error If the error is larger than the preset maximum error threshold, the step length is finely adjusted according to a preset first frequency, and the output frequency of the special clock is reduced; the maximum error threshold is greater than zero;

if the Time error Time _error If the frequency is smaller than the preset minimum error threshold, the output frequency of the special clock is increased according to a preset second frequency fine tuning step length; the minimum error threshold is less than zero.

5. The apparatus according to claim 4, wherein the rendering control unit, in particular configured to perform the rendering synchronization control, comprises:

when the data Frame to be rendered arrives at the renderer, if the Frame is satisfied _pts ≤STC _elapsed ≤Frame _pts +D _max Rendering is carried out based on the current data frame to be rendered; wherein, STC _elapsed Frame for the current elapsed time of the playback _pts Display time stamp for the current data frame to be rendered, D _max Displaying the time for a preset maximum allowable delay; d is not less than 0 _max ≤Frame _duration ；Frame _duration A display duration for a single data frame;

if STC is satisfied _elapsed <Frame _pts Rendering based on the current data Frame to be rendered after waiting time Δ t, Δ t = Frame _pts -STC _elapsed ；

If STC is satisfied _elapsed >Frame _pts +D _max And discarding the current data frame to be rendered.

6. The apparatus of claim 4, further comprising a dedicated timing unit;

the timing unit is used for determining the current counting time of the special clock according to the time acquisition instruction of the rendering control unit or the clock synchronization unit and feeding back the current counting time to the corresponding unit; wherein determining the current count time of the dedicated clock comprises:

obtaining a current clock count value W from the dedicated clock _{ctr_curr} ；

If the current last clock count value W _{ctr_pre} If the value is the preset initial value, the last clock count value W is updated _{ctr_pre} Counting value W for current clock _{ctr_curr} ；

If W is satisfied _{ctr_curr} ＜W _{ctr_pre} Adding one to the current revolution times; the initial value of the revolution number is 0;

according to W _{ctr_accu} ＝W _{ctr_curr} +N×W _{ctr_max} Calculating the cumulative count W of the clock _{ctr_accu} (ii) a Wherein N is the current revolution number; updating the last clock count value W _{ctr_pre} Counting value W for current clock _{ctr_curr} ；W _{ctr_max} A maximum clock count value for the dedicated clock;

accumulating the clock by a count W _{ctr_accu} And dividing the current clock frequency of the special clock to obtain the current counting time of the special clock.

7. An audio and video playing device is characterized by comprising a processor and a memory;

the memory stores an application program executable by the processor, and the application program is used for causing the processor to execute the audio and video playing method according to any one of claims 1 to 3.

8. A computer-readable storage medium having stored therein computer-readable instructions for executing the audio-video playback method according to any one of claims 1 to 3.

Images