TW201442496A - System and method for synchronizing video data and audio data - Google Patents

Abstract

The present invention provides a system and method for synchronizing video data and audio Data. The system is configured for: storing decoded video data of a video packet in a first buffer; outputting the decoded video data from the first buffer to a display device when a time stamp of the video packet meets a preset condition; storing decoded audio data of an audio packet in a second buffer at the same time of decoding the video packet, and transmitting a time stamp of the audio packet to the video packet; transmitting the decoded audio data from the second buffer to a specified queue, reading the decoded audio data from the specified queue after each preset time interval, and transmitting the decoded audio data to a third buffer; reading the decoded audio data from the third buffer sequentially, and outputting the decoded audio data to the display device. The invention can synchronize video data and audio data.

Description

Video and audio synchronization system and method

The present invention relates to an encoding system and method, and more particularly to an audio and video synchronization system and method.

Generally speaking, the synchronization of the image (video packet) and the sound (audio packet) is performed by the time stamp (such as multimedia time, MM Time) carried in the data segment, and the image and the sound refer to the same MM. Time, the audio packet is responsible for updating the MM Time, and the program decides whether to present the picture immediately, render it later, or discard the expired picture according to the MM Time of the image.

Many network-based multimedia audio and video applications (such as video conferencing, network video telephony, remote desktop audio and video playback) in order to reduce the bandwidth loss, the image part will use compression technology such as H.264, but due to inherent limitations The compression end (encode for transmitting bitstream) must temporarily store the slice data in the buffer so that it cannot be immediately output to the decoder (decode for receiving bitstream) through the network, and the decoding end In decoding, it is also necessary to wait until the image data has not been referenced or used, and the frame is output from the buffer, resulting in a situation in which the picture and the sound are not synchronized.

If the audio packet is still used to update the MM Time, then the screen has a high probability that it will be considered as expired and dropped (frame drop), and the more serious the frame drop situation will cause the video playback to be like a slide effect.

In view of the above, it is necessary to provide an audio-visual synchronization system and method, which can use a queue as a buffer of audio packets, and synchronize the multimedia time of the video packet and the audio packet to achieve the purpose of video and audio synchronization. .

An audio-visual synchronization system is applied to an electronic device, the system comprising: a video decoding module, configured to decode a received video packet, and store the decoded video data into a first buffer; the video decoding module The video output module is configured to sequentially read the decoded video data from the first buffer, and the video output module is configured to output the read video data to the electronic device when the time stamp of the video packet meets the preset requirement. On the display device, the audio decoding module is configured to decode the received audio packet while decoding the video, store the decoded audio data in the second buffer, and transmit the time stamp of the audio packet to the video. a packet; a generating module, configured to read the decoded audio data from the second buffer, move the decoded audio data to a specified queue, and generate a consumption module at a preset time; the consumption module , for reading the decoded audio data from the specified queue, transmitting the decoded audio data to a third buffer; and an audio output module for buffering from the third buffer Sequentially reading the decoded audio data, the read audio data output to the display device.

An audio-video synchronization method is applied to an electronic device, the method comprising: video decoding step 1 for decoding a received video packet, and storing the decoded video data in a first buffer; video decoding step two, from The decoded video data is sequentially read in the first buffer; the video output step is, when the time stamp of the video packet meets the preset requirement, the read video data is output to the display device of the electronic device; the audio decoding step, While performing video decoding, the received audio packet is decoded, the decoded audio data is stored in the second buffer, and the time stamp of the audio packet is transmitted to the video packet; and the generating step is performed from the second buffer. Reading the decoded audio data, moving the decoded audio data to a specified queue, and generating a consumption step every preset time; in the consumption step, reading the decoded audio data from the specified queue, The decoded audio data is transmitted to the third buffer; the audio output step reads the decoded audio resources sequentially from the third buffer , Reads the audio data output to the display device.

Compared with the prior art, the video synchronization system and method can use a queue as a buffer of audio packets, and synchronize the multimedia time of the video packet and the audio packet to achieve video and audio synchronization. For the purpose, and without changing the code of the server side (ie, the compression side).

2. . . Electronic device

20. . . display screen

twenty two. . . input device

twenty three. . . Storage

twenty four. . . Video synchronization system

25. . . processor

240. . . Video decoding module

241. . . Audio decoding module

242. . . Generation module

243. . . Consumer module

244. . . Video output module

245. . . Audio output module

1 is a schematic diagram of an operating environment of a video and audio synchronization system of the present invention.

2 is a functional block diagram of the video and audio synchronization system of the present invention.

3 is a flow chart of a method for synchronizing video and audio according to the present invention.

FIG. 4 is a schematic diagram showing another manner of description of FIG. 3.

Referring to FIG. 1 , it is a schematic diagram of an operating environment of the audio-visual synchronization system of the present invention. The AV synchronization system 24 operates in the electronic device 2. The electronic device 2 further includes an input device 22, a memory 23, and a processor 25 connected through a data bus. The electronic device 2 may be a computer, a mobile phone, a PDA (Personal Digital Assistant), or the like.

The storage 23 is configured to store data such as code and video of the AV synchronization system 24. The input device 22 is used to input various materials set by the user, such as a keyboard, a mouse, and the like. In a special embodiment, the electronic device 2 may include a display device 20 connected to the data bus, the display device 20 is configured to display the image and the like, and the display device 20 may be a liquid crystal display screen of the computer. The touch screen of the mobile phone, etc.

In this embodiment, the video synchronization system 24 may be divided into one or more modules, the one or more modules being stored in the storage 23 and configured to be processed by one or more The present invention (this embodiment is a processor 25) is executed to complete the present invention. For example, as shown in FIG. 2, the video synchronization system 24 is divided into a video decoding module 240, an audio decoding module 241, a generation module 242, a consumption module 243, a video output module 244, and an audio output module 245. . The module referred to in the present invention is a program segment for performing a specific function, and is more suitable for describing the execution process of the software in the electronic device 2 than the program. The specific functions of each module will be described below with reference to FIGS. 3 and 4.

Referring to FIG. 3, it is a flowchart of the video and audio synchronization method of the present invention.

In the following description, the video decoding steps S10-S13 are performed in synchronization with the audio decoding steps S20-S23. When the user plays a movie or uses the audio/video software on the virtual machine, the server side establishes a video stream channel (Video Stream Channel) and an audio stream channel (Audio Stream) with the user end (such as the electronic device 2). Channel), used to transmit video packets (ie video packets) and audio packets (sound packets). The electronic device 2 will continue to receive the video packet and the audio packet via the two channels.

In step S10, the video decoding module 240 receives a video packet from the server through the video stream channel.

In step S11, the video decoding module 240 decodes the video packet, and stores the decoded video data (ie, raw data) into the first buffer, as shown in the frame buffer in FIG. In this embodiment, the video decoding module 240 decodes the video packet by using a corresponding decoding algorithm according to the encoding algorithm of the video packet. For example, if the video packet is encoded by the H.264 technology, the video decoding module 240 decodes the video packet by using an H.264 decoder.

In other embodiments, the video decoding module 240 further performs color gamut conversion on the decoded video data according to the operating system type of the electronic device 2. For example, suppose the operating system of the client (such as the electronic device 2) is Windows, the color gamut displayed on Windows is RGBA (or RGB32, RGB series), but the image encoding (such as H.264) is adopted on the server side. The YUV color gamut (such as YUV420, YUV440, YUV444), so the video decoding module 240 initially decodes the frame (frame) is the YUV color gamut, and then the video decoding module 240 will convert the decoded video data into The RGB color gamut enables the decoded video material to be displayed on the user end in an optimal manner.

In step S12, the video decoding module 240 sequentially reads the decoded video data from the first buffer, for example, reads a frame of the video image.

In step S13, the video decoding module 240 determines whether the time stamp of the video packet meets the preset requirement. In this embodiment, the time stamp is taken as an example of a multimedia time (MM Time), and the multimedia time MM Time of the video packet is obtained from the audio packet.

If the MM Time of the video packet is consistent with the current time of the electronic device 2 (eg, equal), the video decoding module 240 determines that the time stamp of the video packet meets the preset requirement, and performs step S24, and the video output module 244 reads The video material is output to the display device 20. The current time of the electronic device is the current time recorded by the operating system of the electronic device.

If the MM Time of the video packet does not match the current time of the electronic device 2, the video decoding module 240 determines that the time stamp of the video packet does not meet the preset requirement, and the process returns to step S12, and the video decoding module 240 reads the next frame. Image screen.

In step S20, the audio decoding module 241 receives the audio packet from the server through the audio stream channel while receiving the video packet and decoding the video decoding module 240.

In step S21, the audio decoding module 241 decodes the audio packet, and stores the decoded audio data (ie, raw data) into the second buffer. See PCM (Pulse Code Modulation) in FIG. Buffer. At the same time, the audio decoding module 241 transmits the time stamp of the audio packet (such as MM Time) to the video packet, and the video packet is synchronized with reference to the MM Time of the audio packet (refer to step S13).

In this embodiment, the audio decoding module 241 decodes the audio packet by using a corresponding decoding algorithm according to the encoding algorithm of the audio packet. For example, if the audio packet is encoded by PCM coding technology, the audio decoding module 241 decodes the audio packet by using a PCM decoder.

In step S22, the generating module 242 reads the decoded audio data from the second buffer, and moves the decoded audio data to a designated queue. Referring to the PCM Ring in FIG. In the present embodiment, the generation module 242 is a thread, such as a Producer thread.

In step S23, the generating module 242 generates a consumption module 243 every preset time. Then, the consumption module 243 reads the decoded audio data from the specified queue, and transmits the decoded audio data to the third buffer, as shown in the Wave Ring in FIG. In this embodiment, the consumption module 243 is a thread, such as a Consumer thread, and the consumption module 243 ends itself after transmitting the decoded audio data to the third buffer.

In this embodiment, the preset time is a time difference between the first decoded audio data and the first decoded video data. That is to say, in the present invention, the decoded audio data is not immediately transmitted to the third buffer for output, but the decoded audio data is first stored in a designated queue, and the video packet is decoded first. After the picture is displayed, the generation module 242 begins to generate a consumption module 243 to consume the data in the specified queue, thereby synchronizing the sound and the picture.

In step S24, the audio output module 245 sequentially reads the decoded audio data from the third buffer, and outputs the read audio data to the display device 20.

The present invention can be applied to a remote desktop, a video conference, and a network video call. For example, the remote desktop application can take the following steps:

(1) Install the client program in the electronic device 2 and connect to the remote desktop.

(2) Select the audio/video playback software on the remote desktop or the application with multimedia playback function, wherein the image portion is encoded by H.264.

(3) The user program synchronizes playback of image data and sound data.

It should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, and the present invention is not limited thereto. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that Modifications or equivalents are made without departing from the spirit and scope of the invention.

2. . . Electronic device

20. . . display screen

twenty two. . . input device

twenty three. . . Storage

twenty four. . . Video synchronization system

25. . . processor

Claims (10)

An audio-visual synchronization system is applied to an electronic device, the system comprising:
a video decoding module, configured to decode the received video packet, and store the decoded video data into the first buffer;
The video decoding module is further configured to sequentially read the decoded video data from the first buffer.
The video output module is configured to output the read video data to the display device of the electronic device when the time stamp of the video packet meets the preset requirement;
The audio decoding module is configured to decode the received audio packet while the video decoding is performed, store the decoded audio data into the second buffer, and transmit the time stamp of the audio packet to the video packet;
a generating module, configured to read the decoded audio data from the second buffer, move the decoded audio data to a specified queue, and generate a consumption module every preset time;
a consumption module, configured to read the decoded audio data from the specified queue, and transmit the decoded audio data to a third buffer; and an audio output module for sequentially reading from the third buffer The decoded audio data is output to the display device. The video synchronization system of claim 1, wherein the video packet is received from a server through a video stream channel, and the audio packet is received from a server through an audio stream channel. The video synchronization system of claim 1, wherein the video decoding module is further configured to perform color gamut conversion on the decoded video data according to an operating system type of the electronic device. The video synchronization system of claim 1, wherein the video decoding module is further configured to: if the time stamp of the video packet is consistent with the current time of the electronic device, determine that the time stamp of the video packet matches Preset requirements. The video synchronization system of claim 1, wherein the preset time is a time difference between the first decoded audio data and the first decoded video data. An audio-video synchronization method is applied to an electronic device, and the method includes:
Video decoding step 1 is configured to decode the received video packet, and store the decoded video data into the first buffer;
Video decoding step 2, sequentially reading the decoded video data from the first buffer;
In the video output step, when the time stamp of the video packet meets the preset requirement, the read video data is output to the display device of the electronic device;
In the audio decoding step, when the video decoding is performed, the received audio packet is decoded, the decoded audio data is stored in the second buffer, and the time stamp of the audio packet is transmitted to the video packet;
a generating step of reading the decoded audio data from the second buffer, moving the decoded audio data to the specified queue, and generating a consumption step every preset time;
In the consumption step, the decoded audio data is read from the specified queue, and the decoded audio data is transmitted to the third buffer; and the audio output step sequentially reads the decoded audio data from the third buffer. The read audio data is output to the display device. The video synchronization method of claim 6, wherein the video packet is received from a server through a video stream channel, and the audio packet is received from a server through an audio stream channel. The video synchronization method of claim 6, wherein the video decoding step 1 further includes:
The decoded video data is subjected to color gamut conversion according to the type of the operating system of the electronic device. The video synchronization method of claim 6, wherein the video decoding step 2 further includes:
If the time stamp of the video packet matches the current time of the electronic device, it is determined that the time stamp of the video packet meets the preset requirement. The video synchronization method of claim 6, wherein the preset time is a time difference between the first decoded audio data and the first decoded video data.