CN106817614A - Audio frequency and video frame losing device and method - Google Patents

Abstract

The invention discloses an audio and video frame loss device, which comprises: an acquisition module, used to acquire the time difference between the audio frame time and the video frame time at the current playback moment in real time or regularly; a judging module, used if the If the absolute value of the time difference is greater than or equal to the preset duration, then it is judged whether the video contains a B frame, and the preset duration is a preset multiple of the display duration of the unit video frame; the determination module is used to determine if the video contains B frame, then according to the first frame rate of the currently playing video source file and the second frame rate of the current video playback, determine the time interval of discarding the B frame in the video; the first discarding module is used to discard B according to the time interval frame, so that the frame rate of the video source file reaches the frame rate of the current video playback. The invention also discloses an audio and video frame dropping method. The present invention can adjust the output synchronization of audio and video by discarding B frames that have no influence on video playback.

Description

Audio and video frame dropping device and method

technical field

The invention relates to the field of video playback, in particular to an audio and video frame dropping device and method.

Background technique

In the era of mobile Internet, with the promotion and popularization of smart terminals, users are constantly pursuing high-quality audio-visual experience. The picture quality requirements of multimedia videos are getting higher and higher.

When developing video players for smart terminals, it is usually necessary to integrate and test video sources of different file types and encoding types, as well as video sources of the same encoding type, different levels, and different configurations, so as to meet the diversity of user audio and video sources. Due to the constraints of hardware performance, it is difficult to achieve the FPS (Frames Per Second, video playback frame rate) required by the source file during video playback, so frame skipping will occur during playback, making the audio and video out of sync.

Contents of the invention

The main purpose of the present invention is to propose an audio and video frame dropping device and method, aiming to solve the technical problem in the prior art that frame skipping occurs during playback, making the audio and video out of sync.

In order to achieve the above object, the present invention provides an audio and video frame dropping device, said device comprising:

The obtaining module is used to obtain the time difference between the audio frame time and the video frame time of the current playback moment in real time or regularly;

A judging module, configured to judge whether the video contains a B frame if the absolute value of the time difference is greater than or equal to a preset duration, and the preset duration is a preset multiple of the display duration of a unit video frame;

Determining module, for if comprising B frame in the video, then according to the first frame rate of current playing video source file and the second frame rate of currently performing video playback, determine the time interval of discarding the B frame in the video;

The first discarding module is configured to discard the B frame according to the time interval, so that the frame rate of the video source file reaches the frame rate of the current video playback.

Optionally, the determination module includes:

a determining unit, configured to determine a third frame rate supported by the terminal according to the performance of the terminal chip;

a judging unit, configured to judge whether the second frame rate is lower than the third frame rate;

an adjustment unit, configured to add a preset increment to the third frame rate to update the third frame rate when the second frame rate is greater than or equal to the third frame rate;

The judging unit is further configured to judge whether the sum of the second frame rate and a preset value is greater than the third frame rate when the second frame rate is lower than the third frame rate;

A first calculation unit, configured to calculate an average value of the second frame rate and the third frame rate when the sum of the second frame rate and a preset value is less than or equal to the third frame rate, and calculating the time interval according to the average value and the first frame rate;

The adjustment unit is further configured to subtract the preset increment from the third frame rate when the sum of the second frame rate and a preset value is greater than the third frame rate, so as to update the The third frame rate.

Optionally, the device also includes:

The playback module is used to play the next I frame if there is no B frame in the video.

Optionally, the device also includes:

The comparison module is used to compare the current system playback time with the actual playback time in real time;

The second discarding module is used for discarding the B frame in the video source file when the time difference between the current system playing time and the actual playing time is greater than the first preset time;

The playing module is also used for playing the next I frame when the time difference between the current system playing time and the actual playing time is greater than the second preset time, wherein the first preset time is less than the second preset time .

Optionally, the second discarding module includes:

The second calculation unit is used to calculate the number of B frames that need to be discarded by the second frame rate of the currently played video and the first frame rate of the currently played video source file;

The discarding unit is configured to discard corresponding B frames in the video source file according to the calculated number of B frames to be discarded.

In addition, in order to achieve the above object, the present invention also provides an audio and video frame loss method, the method includes the following steps:

Obtain the time difference between the audio frame time and the video frame time at the current playback moment in real time or regularly;

If the absolute value of the time difference is greater than or equal to a preset duration, it is judged whether a B frame is included in the video, and the preset duration is a preset multiple of the display duration of a unit video frame;

If so, then determine the time interval of discarding the B frame in the video according to the first frame rate of the currently played video source file and the second frame rate of the current video playback;

B frames are discarded according to the time interval, so that the frame rate of the video source file reaches the current frame rate of video playback.

Optionally, the step of determining the time interval of discarding B frames in the video according to the first frame rate of the currently playing video source file and the second frame rate of the current video playback includes:

determining a third frame rate supported by the terminal according to the performance of the terminal chip;

judging whether the second frame rate is less than the third frame rate;

When the second frame rate is greater than or equal to the third frame rate, add a preset increment to the third frame rate to update the third frame rate, and continue to perform the judging the second frame rate The step of whether the frame rate is less than the third frame rate;

When the second frame rate is less than the third frame rate, judging whether the sum of the second frame rate and a preset value is greater than the third frame rate;

When the sum of the second frame rate and the preset value is less than or equal to the third frame rate, calculate the average value of the second frame rate and the third frame rate, and based on the average value and the calculating the time interval at the first frame rate;

When the sum of the second frame rate and the preset value is greater than the third frame rate, subtract the preset increment from the third frame rate to update the third frame rate, and continue to execute The step of judging whether the second frame rate is lower than the third frame rate.

Optionally, after the step of judging whether there are B frames in the video, it also includes:

If there is no B frame in the video, play the next I frame.

Optionally, the method also includes:

Compare the current system playback time with the actual playback time in real time;

When the time difference between the current system playback time and the actual playback time is greater than the first preset time, the B frame in the video source file is discarded;

When the time difference between the current system play time and the actual play time is greater than a second preset time, then play the next I frame, wherein the first preset time is less than the second preset time.

Optionally, the step of discarding the B frame in the video source file includes:

Calculate the number of B frames that need to be discarded by the second frame rate of the currently playing video and the first frame rate of the currently playing video source file;

The corresponding B frames are discarded in the video source file according to the calculated number of B frames to be discarded.

The audio and video frame loss device and method proposed by the present invention obtain the time difference between the audio frame time and the video frame time at the current playback moment in real time or regularly; when the absolute value of the time difference is greater than or equal to the preset duration, it is judged Whether the video contains B frames, wherein the preset duration is a preset multiple of the display duration of the unit video frame; when the video contains B frames, according to the first frame rate of the currently played video source file and the current The second frame rate for video playback determines the time interval for discarding B frames in the video, and discards B frames according to the time interval. By discarding B frames according to the time interval, the frame rate of the video source file reaches the current video. The playback frame rate, so as to realize the output synchronization of audio and video.

Description of drawings

FIG. 1 is a schematic diagram of an optional hardware structure of a mobile terminal implementing various embodiments of the present invention;

FIG. 2 is a schematic diagram of a wireless communication device of a mobile terminal as shown in FIG. 1;

3 is a schematic diagram of functional modules of the first embodiment of the audio and video frame dropping device of the present invention;

FIG. 4 is a schematic diagram of a detailed functional module of the determination module in the third embodiment of the audio and video frame loss device of the present invention;

5 is a schematic diagram of functional modules of the fourth embodiment of the audio and video frame dropping device of the present invention;

Fig. 6 is a schematic diagram of the refined functional modules of the second discarding module in the fifth embodiment of the audio and video frame dropping device of the present invention;

Fig. 7 is a schematic flow chart of the first embodiment of the audio and video frame loss method of the present invention;

Fig. 8 is the details of the step of determining the time interval for discarding B frames in the video according to the first frame rate of the currently playing video source file and the second frame rate of the current video playing in the third embodiment of the audio and video frame dropping method of the present invention Schematic diagram of chemical process;

Fig. 9 is a schematic flow chart of the fourth embodiment of the audio and video frame loss method of the present invention;

FIG. 10 is a schematic diagram of a detailed flowchart of the step of discarding B frames in the video source file in the fifth embodiment of the method for dropping audio and video frames according to the present invention.

The realization of the purpose of the present invention, functional characteristics and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

detailed description

It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

A mobile terminal implementing various embodiments of the present invention will now be described with reference to the accompanying drawings. In the following description, use of suffixes such as 'module', 'part' or 'unit' for denoting elements is only for facilitating description of the present invention and has no specific meaning by itself. Therefore, "module" and "component" may be mixedly used.

Mobile terminals may be implemented in various forms. For example, terminals described in the present invention may include devices such as mobile phones, smart phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), navigation devices, etc. mobile terminals and fixed terminals such as digital TVs, desktop computers, etc. In the following, it is assumed that the terminal is a mobile terminal. However, those skilled in the art will understand that the configuration according to the embodiments of the present invention can also be applied to stationary type terminals, in addition to elements specifically used for mobile purposes.

FIG. 1 is a schematic diagram of a hardware structure of a mobile terminal implementing various embodiments of the present invention.

The mobile terminal 100 may include a wireless communication unit 110, an A/V (audio/video) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, an interface unit 170, a controller 180, and a power supply unit 190. wait. FIG. 1 shows a mobile terminal having various components, but it is understood that implementing all shown components is not a requirement. Greater or fewer components may alternatively be implemented. Elements of the mobile terminal will be described in detail below.

The wireless communication unit 110 generally includes one or more components that allow radio communication between the mobile terminal 100 and a wireless communication device or network. For example, the wireless communication unit may include at least one of the mobile communication module 112 and the wireless Internet module 113 .

The mobile communication module 112 transmits and/or receives radio signals to at least one of a base station (eg, access point, Node B, etc.), an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data transmitted and/or received according to text and/or multimedia messages.

The wireless Internet module 113 supports wireless Internet access of the mobile terminal. The module can be coupled to the terminal internally or externally. The wireless Internet access technologies involved in this module may include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless Broadband), Wimax (Global Microwave Interconnection Access), HSDPA (High Speed Downlink Packet Access), etc. .

The user input unit 130 may generate key input data according to commands input by the user to control various operations of the mobile terminal. The user input unit 130 allows a user to input various types of information, and may include a keyboard, domes, touch pad (e.g., a touch-sensitive component that detects changes in resistance, pressure, capacitance, etc. due to being touched), a scroll wheel , joystick, etc. In particular, when a touch pad is superimposed on the display unit 151 in a layer form, a touch screen may be formed.

The interface unit 170 serves as an interface through which at least one external device can be connected with the mobile terminal 100 . For example, an external device may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, audio input/output (I/O) ports, video I/O ports, headphone ports, and more. The identification module may store various information for authenticating a user using the mobile terminal 100 and may include a User Identity Module (UIM), a Subscriber Identity Module (SIM), a Universal Subscriber Identity Module (USIM) and the like. In addition, a device having an identification module (hereinafter referred to as "identification device") may take the form of a smart card, and thus, the identification device may be connected to the mobile terminal 100 via a port or other connection means. The interface unit 170 may be used to receive input (eg, data information, power, etc.) transfer data between.

In addition, when the mobile terminal 100 is connected with an external cradle, the interface unit 170 may serve as a path through which power is supplied from the cradle to the mobile terminal 100 or may be used as a path allowing various command signals input from the cradle to be transmitted to the mobile terminal 100 therethrough. The path to the terminal. Various command signals or power input from the base may be used as a signal for identifying whether the mobile terminal is accurately mounted on the base. The output unit 150 is configured to provide output signals (eg, audio signals, video signals, alarm signals, vibration signals, etc.) in a visual, audio and/or tactile manner. The output unit 150 may include a display unit 151, an audio output module 152, and the like.

The display unit 151 can display information processed in the mobile terminal 100 . For example, when the mobile terminal 100 is in a phone call mode, the display unit 151 may display a user interface (UI) or a graphical user interface (GUI) related to calls or other communications (eg, text messaging, multimedia file downloading, etc.). When the mobile terminal 100 is in a video call mode or an image capture mode, the display unit 151 may display captured images and/or received images, a UI or GUI showing videos or images and related functions, and the like.

Meanwhile, when the display unit 151 and the touch pad are superimposed on each other in a layer to form a touch screen, the display unit 151 may serve as an input device and an output device. The display unit 151 may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), an organic light emitting diode (OLED) display, a flexible display, a three-dimensional (3D) display, and the like. Some of these displays can be configured to be transparent to allow a user to view from the outside, which can be called a transparent display, and a typical transparent display can be, for example, a TOLED (Transparent Organic Light Emitting Diode) display or the like. Depending on the particular desired implementation, the mobile terminal 100 may include two or more display units (or other display means), for example, the mobile terminal may include an external display unit (not shown) and an internal display unit (not shown). . The touch screen can be used to detect touch input pressure as well as touch input position and touch input area.

The audio output module 152 may convert audio data received by the wireless communication unit 110 or stored in the memory 160 when the mobile terminal is in a call signal receiving mode, a call mode, a recording mode, a voice recognition mode, a broadcast receiving mode, etc. audio signal and output as sound. Also, the audio output module 152 may provide audio output related to a specific function performed by the mobile terminal 100 (eg, call signal reception sound, message reception sound, etc.). The audio output module 152 may include a pickup, a buzzer, and the like.

The memory 160 may store software programs and the like for processing and controlling operations executed by the controller 180, or may temporarily store data that has been output or will be output (eg, phonebook, messages, still images, videos, etc.). Also, the memory 160 may store data on various patterns of vibration and audio signals output when a touch is applied to the touch screen.

Memory 160 may include at least one type of storage medium including flash memory, hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), random access memory (RAM), static random access memory ( SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. Also, the mobile terminal 100 may cooperate with a web storage device performing a storage function of the memory 160 through a network connection.

The controller 180 generally controls the overall operations of the mobile terminal. For example, the controller 180 performs control and processing related to voice calls, data communications, video calls, and the like. In addition, the controller 180 may include a multimedia module 181 for reproducing (or playing back) multimedia data, and the multimedia module 181 may be constructed within the controller 180 or may be constructed separately from the controller 180 . The controller 180 may perform a pattern recognition process to recognize a handwriting input or a picture drawing input performed on the touch screen as a character or an image.

The power supply unit 190 receives external power or internal power and supplies appropriate power required to operate various elements and components under the control of the controller 180 .

Various implementations described herein can be implemented on a computer readable medium using, for example, computer software, hardware, or any combination thereof. For hardware implementation, the embodiments described herein can be implemented by using Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays ( FPGA), processors, controllers, microcontrollers, microprocessors, electronic units designed to perform the functions described herein, in some cases, such implementations may be implemented in the controller 180 implemented in. For software implementation, an embodiment such as a procedure or a function may be implemented with a separate software module that allows at least one function or operation to be performed. The software codes may be implemented by a software application (or program) written in any suitable programming language, which may be stored in memory 160 and executed by controller 180 .

So far, the mobile terminal has been described in terms of its functions. Hereinafter, for the sake of brevity, a slide-type mobile terminal among various types of mobile terminals such as folder-type, bar-type, swing-type, slide-type mobile terminals, etc. will be described as an example. Therefore, the present invention can be applied to any type of mobile terminal and is not limited to a slide type mobile terminal.

The mobile terminal 100 as shown in FIG. 1 may be configured to operate using communication means, such as wired and wireless, and satellite-based communication means, which transmit data via frames or packets.

A communication device in which a mobile terminal according to the present invention can operate will now be described with reference to FIG. 2 .

Such communication devices may use different air interfaces and/or physical layers. Air interfaces used by communication devices include, for example, Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), and Universal Mobile Telecommunications Set (UMTS) (in particular, Long Term Evolution (LTE) ), Global System for Mobile Communications (GSM), etc. As a non-limiting example, the following description refers to a CDMA communication device, but such teachings are equally applicable to other types of devices.

Referring to FIG. 2 , a CDMA wireless communication device may include a plurality of mobile terminals 100 , a plurality of base stations (BS) 270 , a base station controller (BSC) 275 , and a mobile switching center (MSC) 280 . MSC 280 is configured to interface with Public Switched Telephone Network (PSTN) 290 . MSC 280 is also configured to interface with BSC 275 which may be coupled to BS 270 via a backhaul line. The backhaul may be constructed according to any of several known interfaces including, for example, E1/T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL or xDSL. It will be appreciated that an arrangement as shown in FIG. 2 may include multiple BSCs 275 .

Each BS 270 may serve one or more sectors (or areas), each sector covered by a multi-directional antenna or an antenna pointing in a particular direction radially away from the BS 270 . Alternatively, each sector may be covered by two or more antennas for diversity reception. Each BS 270 may be configured to support multiple frequency allocations, with each frequency allocation having a specific frequency spectrum (eg, 1.25 MHz, 5 MHz, etc.).

The intersection of partition and frequency allocation may be referred to as a CDMA channel. BS 270 may also be referred to as a Base Transceiver Subset (BTS) or other equivalent terminology. In such cases, the term "base station" may be used to refer generally to a single BSC 275 and at least one BS 270. A base station may also be referred to as a "cell site." Alternatively, sectors of a particular BS 270 may be referred to as multiple cell sites.

As shown in FIG. 2, a broadcast transmitter (BT) 295 transmits a broadcast signal to the mobile terminal 100 operating within the device. The broadcast receiving module 111 as shown in FIG. 1 is provided at the mobile terminal 100 to receive broadcast signals transmitted by the BT295. In FIG. 2, several Global Positioning Device (GPS) satellites 300 are shown. The satellite 300 helps locate at least one of the plurality of mobile terminals 100 .

In FIG. 2, a plurality of satellites 300 are depicted, but it is understood that any number of satellites may be utilized to obtain useful positioning information. The location information module 115 as shown in FIG. 1 is generally configured to cooperate with the satellite 300 to obtain desired positioning information. Instead of or in addition to GPS tracking techniques, other techniques that can track the location of the mobile terminal may be used. Additionally, at least one GPS satellite 300 may optionally or additionally handle satellite DMB transmissions.

As a typical operation of the wireless communication device, the BS 270 receives reverse link signals from various mobile terminals 100 . Mobile terminal 100 typically engages in calls, messaging, and other types of communications. Each reverse link signal received by a particular BS 270 is processed within the particular BS 270. The obtained data is forwarded to the relevant BSC275. The BSC provides call resource allocation and mobility management functions including coordination of soft handover procedures between BS270. BSC 275 also routes received data to MSC 280 , which provides additional routing services for interfacing with PSTN 290 . Similarly, PSTN 290 interfaces with MSC 280 , MSC interfaces with BSC 275 , and BSC 275 controls BS 270 to send forward link signals to mobile terminal 100 accordingly.

Based on the above-mentioned hardware structure of the mobile terminal and the structure of the communication device, various embodiments of the device and method of the present invention are proposed.

The invention provides an audio and video frame dropping device.

Referring to FIG. 3 , FIG. 3 is a schematic diagram of functional modules of a first embodiment of an audio and video frame dropping device according to the present invention.

In this embodiment, the audio and video frame loss device includes:

The acquiring module 10 is configured to acquire the time difference between the audio frame time and the video frame time at the current playback moment in real time or periodically.

In this embodiment, when the audio and video playback of the video played by the mobile terminal is not synchronized, and the decoding time of the video image is longer than the display time of the image, the output of the audio and video needs to be adjusted. First trigger the instruction to start the audio and video frame loss synchronization mechanism, and the mobile terminal obtains the audio frame time and the video frame time at the current playback moment in real time or regularly according to the instruction, and calculates the difference between the audio frame time and the video frame time at the current playback moment. time difference between. In the video coding sequence, there are mainly three coding frames: I frame, P frame, and B frame. Wherein: the I frame is an Intra-coded picture (intra-coded picture frame), which does not refer to other picture frames, and only uses the information of this frame for coding. The P frame is a Predictive-coded Picture (predictive coded image frame), which uses the previous I frame or P frame to perform inter-frame predictive coding in a motion prediction manner. B frame is Bidirectionally predicted picture (bidirectional predictive coding image frame), which provides the highest compression ratio. It requires both the previous image frame (I frame or P frame) and the subsequent image frame (P frame), which is performed by motion prediction. Inter-frame bidirectional predictive coding, under normal circumstances, the B frame is only used to record the difference between the current frame and the preceding and following frames. Therefore, during video playback, when the audio and video are out of sync, the corresponding B frame in the video is generally discarded.

The judging module 20 is configured to judge whether the video contains B frames if the absolute value of the time difference is greater than or equal to a preset duration, the preset duration being a preset multiple of the display duration of a unit video frame.

In this embodiment, firstly, the absolute value of the time difference between the audio frame time and the video frame time is compared with a preset duration, and the preset duration is a preset multiple of the display duration of a unit video frame. If the absolute value of the time difference is greater than or equal to the preset duration, that is, when the audio and video frames differ by a preset multiple of continuous video frame display duration, the preset multiple can be 2 times or a multiple of other values. If you feel that the audio and video are out of sync, it is judged that the audio and video are out of sync. At this time, it is necessary to adjust the output of the audio and video, so it is necessary to determine whether there is a B frame in the video that does not affect the video playback, so as to discard the B frame.

The determination module 30 is configured to determine the time interval for discarding B frames in the video according to the first frame rate of the currently played video source file and the second frame rate of the current video playback if the video contains B frames.

Specifically, if there are B frames in the video, the time interval for discarding the B frames in the video is determined according to the first frame rate of the currently played video source file and the second frame rate of the current video playback. Specifically, the step of determining the time interval of discarding the B frames in the video may include:

Step 1, determining a third frame rate supported by the terminal according to the performance of the terminal chip;

Step 2, judging whether the second frame rate is lower than the third frame rate;

Step 3, when the second frame rate is greater than or equal to the third frame rate, increase a preset increment to the third frame rate to update the third frame rate, and continue to perform step 2;

Step 4, when the second frame rate is lower than the third frame rate, judging whether the sum of the second frame rate and a preset value is greater than the third frame rate;

Step 5, when the sum of the second frame rate and the preset value is less than or equal to the third frame rate, calculate the average value of the second frame rate and the third frame rate, and calculate the average value based on the average value and the first frame rate to calculate the time interval;

Step 6, when the sum of the second frame rate and a preset value is greater than the third frame rate, subtracting the preset increment from the third frame rate to update the third frame rate, And proceed to Step 1.

The first discarding module 40 is configured to discard B frames according to the time interval, so that the frame rate of the video source file reaches the current frame rate of video playback.

After the time interval is calculated, B frames are discarded according to the time interval, so that the frame rate of the video source file reaches the current frame rate of video playback.

The audio and video frame loss device proposed by the present invention obtains the time difference between the audio frame time and the video frame time at the current playback moment in real time or regularly; when the absolute value of the time difference is greater than or equal to the preset duration, the video is judged Whether contains B frame, wherein, the preset duration is a preset multiple of the display duration of the unit video frame; when the video contains B frame, according to the first frame rate of the currently playing video source file and the current video The second frame rate of playback determines the time interval for discarding B frames in the video, and discards B frames according to the time interval. By discarding B frames according to the time interval, the frame rate of the video source file reaches the current video playback time interval. Frame rate, so as to realize the output synchronization of audio and video.

Further, referring to FIG. 4 , based on the first embodiment of the audio and video frame dropping device of the present invention, a second embodiment of the audio and video frame dropping device of the present invention is proposed.

In this embodiment, the determination module 30 includes:

A determining unit 31, configured to determine a third frame rate supported by the terminal according to the performance of the terminal chip;

A judging unit 32, configured to judge whether the second frame rate is lower than the third frame rate;

An adjustment unit 33, configured to add a preset increment to the third frame rate to update the third frame rate when the second frame rate is greater than or equal to the third frame rate;

The judging unit 32 is further configured to judge whether the sum of the second frame rate and a preset value is greater than the third frame rate when the second frame rate is lower than the third frame rate;

The first calculation unit 34 is configured to calculate the average value of the second frame rate and the third frame rate when the sum of the second frame rate and a preset value is less than or equal to the third frame rate, and calculating the time interval according to the average value and the first frame rate;

The adjusting unit 33 is further configured to subtract the preset increment from the third frame rate when the sum of the second frame rate and a preset value is greater than the third frame rate, so as to update the the third frame rate.

In this embodiment, the specific steps of calculating the time interval of the B frame in the discarded video may be:

Step 1. Determine the third frame rate supported by the terminal according to the performance of the terminal chip. Specifically, the third frame rate is a frame rate directly determined according to the performance of the terminal chip. The frame rates supported by different terminal chips are generally The following are not the same, and will not be repeated here;

Step 2, judging whether the second frame rate is lower than the third frame rate;

Step 3, when the second frame rate is greater than or equal to the third frame rate, increase the preset increment to the third frame rate to update the third frame rate, and continue to perform step 2; specifically Preferably, the preset increment may be 1, and a value other than 1 may also be used in a specific implementation process.

Step 4, when the second frame rate is lower than the third frame rate, judge whether the sum of the second frame rate and a preset value is greater than the third frame rate; specifically, the preset value can be is 5, and values other than 5 can also be used in the specific implementation process.

Step 5, when the sum of the second frame rate and the preset value is less than or equal to the third frame rate, calculate the average value of the second frame rate and the third frame rate, and calculate the average value based on the average value and the first frame rate to calculate the time interval; specifically, the method for calculating the time interval according to the average value and the first frame rate is: combine the first frame rate with the average values to get the difference, and then divide 1000 by the difference to get the time interval.

Step 6, when the sum of the second frame rate and a preset value is greater than the third frame rate, subtracting the preset increment from the third frame rate to update the third frame rate, And proceed to Step 1.

The audio and video frame loss device proposed in this embodiment calculates the time interval for discarding B frames in the video through the first frame rate, the second frame rate, and the second frame rate, so that in the subsequent playback process, according to the time interval Precisely and real-time discarding of B frames in the video, so as to achieve precise and real-time control of discarding B frames that have no effect on video playback, so as to adjust the audio and video synchronization output.

Further, based on the first embodiment of the audio and video frame dropping device of the present invention, a third embodiment of the audio and video frame dropping device of the present invention is proposed.

In this embodiment, the device also includes:

The playback module 50 is used to play the next I frame if there is no B frame in the video.

In this embodiment, if there is no B frame in the video, the next I frame needs to be played to adjust the output synchronization of audio and video.

Further, referring to FIG. 5 , based on the first embodiment of the audio and video frame dropping device of the present invention, a fourth embodiment of the audio and video frame dropping device of the present invention is proposed.

In this embodiment, the device also includes:

Contrast module 60, is used for comparing current system playing time and actual playing time in real time;

The second discarding module 70 is used for discarding the B frame in the video source file when the time difference between the current system play time and the actual play time is greater than the first preset time;

The playing module 50 is also used for playing the next I frame when the time difference between the current system playing time and the actual playing time is greater than the second preset time, wherein the first preset time is less than the second preset time time.

In this embodiment, when the buffering time of the currently playing video is too long, the audio and video will also be out of sync, so it is also necessary to discard the B frame in the video, or play the next I frame to achieve synchronous audio and video output . Specifically, first compare the current system playback time and the actual playback time in real time; when the time difference between the current system playback time and the actual playback time is greater than the first preset time, then discard the B frame in the video source file; When the time difference of the actual playing time is greater than the second preset time, the next I frame is played, wherein the first preset time is less than the second preset time. The first preset time may be 100ms, and the second preset time may be 500ms, and the first preset time and the second preset time may also be set to other values during specific implementation.

The audio and video frame loss device proposed in this embodiment compares the current system playback time with the actual playback time in real time; when the time difference between the current system playback time and the actual playback time is greater than the first preset time, the B frame in the video source file is discarded ; when the time difference between the current system play time and the actual play time is greater than the second preset time, then play the next I frame, wherein the first preset time is less than the second preset time, thereby realizing the sound in the video Video synchronization output.

Further, referring to FIG. 6 , based on the fourth embodiment of the audio and video frame dropping device of the present invention, a fifth embodiment of the audio and video frame dropping device of the present invention is proposed.

In this embodiment, the second discarding module 70 may include:

The second calculation unit 71 is used to calculate the number of B frames that need to be discarded by the second frame rate of the currently played video and the first frame rate of the currently played video source file;

The discarding unit 72 is configured to discard corresponding B frames in the video source file according to the calculated number of B frames to be discarded.

In this embodiment, the number of B frames that need to be discarded can be calculated through the second frame rate of the currently playing video and the first frame rate of the currently playing video source file, and then the number of B frames that need to be discarded is calculated in the video source file The corresponding B frame is discarded. Calculating the number of B frames to be discarded based on the second frame rate of the currently played video and the first frame rate of the currently played video source file is an existing general technique, and will not be repeated here.

The audio and video frame loss device proposed in this embodiment first calculates the number of B frames that need to be discarded by the second frame rate of the current playing video and the first frame rate of the current playing video source file; then according to the calculated B frames that need to be discarded Quantity The corresponding B frames are discarded in the video source file to avoid discarding too many B frames, resulting in discontinuous and paused playback.

In order to better understand the audio and video frame loss device provided by the present invention, here is a brief description of the position of the audio and video frame loss device of the present invention in the video playback system, in the video playback system application layer, including the video player application Programs and media players include media player services in the native framework layer. The specific multimedia playback framework is as follows: output the video data source file to the video file parsing module for parsing, then call it through the demultiplexer, then output the B frame that needs to be discarded, and finally output it after decoding through the audio and video decoder. The frame dropping device for checking audio and video synchronization provided by the present invention can analyze video files, discard B frames in the video, and then realize synchronous output of audio and video.

The present invention further provides an audio and video frame dropping method.

Referring to FIG. 7 , FIG. 7 is a schematic flowchart of a first embodiment of audio and video frame loss in the present invention.

In this embodiment, the method includes:

Step S10, acquiring the time difference between the audio frame time and the video frame time at the current playback moment in real time or periodically.

In this embodiment, when the audio and video playback of the video played by the mobile terminal is not synchronized, and the decoding time of the video image is longer than the display time of the image, the output of the audio and video needs to be adjusted. First trigger the instruction to start the audio and video frame loss synchronization mechanism, and the mobile terminal obtains the audio frame time and the video frame time at the current playback moment in real time or regularly according to the instruction, and calculates the difference between the audio frame time and the video frame time at the current playback moment. time difference between. In the video coding sequence, there are mainly three coding frames: I frame, P frame, and B frame. Wherein: the I frame is an Intra-coded picture (intra-coded picture frame), which does not refer to other picture frames, and only uses the information of this frame for coding. The P frame is a Predictive-coded Picture (predictive coded image frame), which uses the previous I frame or P frame to perform inter-frame predictive coding in a motion prediction manner. B frame is Bidirectionally predicted picture (bidirectional predictive coding image frame), which provides the highest compression ratio. It requires both the previous image frame (I frame or P frame) and the subsequent image frame (P frame), which is performed by motion prediction. Inter-frame bidirectional predictive coding, under normal circumstances, the B frame is only used to record the difference between the current frame and the preceding and following frames. Therefore, during video playback, when the audio and video are out of sync, the corresponding B frame in the video is generally discarded.

Step S20, if the absolute value of the time difference is greater than or equal to a preset duration, it is judged whether the video contains a B frame, and the preset duration is a preset multiple of the display duration of a unit video frame.

In this embodiment, firstly, the absolute value of the time difference between the audio frame time and the video frame time is compared with a preset duration, and the preset duration is a preset multiple of the display duration of a unit video frame. If the absolute value of the time difference is greater than or equal to the preset duration, that is, when the audio and video frames differ by a preset multiple of continuous video frame display duration, the preset multiple can be 2 times or a multiple of other values. If you feel that the audio and video are out of sync, it is judged that the audio and video are out of sync. At this time, it is necessary to adjust the output of the audio and video, so it is necessary to determine whether there is a B frame in the video that does not affect the video playback, so as to discard the B frame.

Step S30, if the video contains B frames, then determine the time interval for discarding B frames in the video according to the first frame rate of the currently played video source file and the second frame rate of the currently played video.

Specifically, if there are B frames in the video, the time interval for discarding the B frames in the video is determined according to the first frame rate of the currently played video source file and the second frame rate of the current video playback. Specifically, the step of determining the time interval of discarding the B frames in the video may include:

Step 1, determining a third frame rate supported by the terminal according to the performance of the terminal chip;

Step 2, judging whether the second frame rate is lower than the third frame rate;

Step 3, when the second frame rate is greater than or equal to the third frame rate, increase a preset increment to the third frame rate to update the third frame rate, and continue to perform step 2;

Step 4, when the second frame rate is lower than the third frame rate, judging whether the sum of the second frame rate and a preset value is greater than the third frame rate;

Step 5, when the sum of the second frame rate and the preset value is less than or equal to the third frame rate, calculate the average value of the second frame rate and the third frame rate, and calculate the average value based on the average value and the first frame rate to calculate the time interval;

Step 6, when the sum of the second frame rate and a preset value is greater than the third frame rate, subtracting the preset increment from the third frame rate to update the third frame rate, And proceed to Step 1.

Step S40, discarding the B frames according to the time interval, so that the frame rate of the video source file reaches the current frame rate of video playback.

After the time interval is calculated, B frames are discarded according to the time interval, so that the frame rate of the video source file reaches the current frame rate of video playback.

The audio and video frame loss method proposed by the present invention obtains the time difference between the audio frame time and the video frame time at the current playback moment in real time or regularly; when the absolute value of the time difference is greater than or equal to the preset duration, the video is judged Whether contains B frame, wherein, the preset duration is a preset multiple of the display duration of the unit video frame; when the video contains B frame, according to the first frame rate of the currently playing video source file and the current video The second frame rate of playback determines the time interval for discarding B frames in the video, and discards B frames according to the time interval. By discarding B frames according to the time interval, the frame rate of the video source file reaches the current video playback time interval. Frame rate, so as to realize the output synchronization of audio and video.

Further, referring to FIG. 8 , based on the first embodiment of the audio and video frame loss method of the present invention, a second embodiment of the audio and video frame loss method of the present invention is proposed.

In this embodiment, the step S30 includes:

Step S31, determining a third frame rate supported by the terminal according to the performance of the terminal chip;

Step S32, judging whether the second frame rate is lower than the third frame rate;

Step S33, when the second frame rate is greater than or equal to the third frame rate, add a preset increment to the third frame rate to update the third frame rate, and continue to perform the determination. The step of whether the second frame rate is less than the third frame rate;

Step S34, when the second frame rate is lower than the third frame rate, judging whether the sum of the second frame rate and a preset value is greater than the third frame rate;

Step S35, when the sum of the second frame rate and the preset value is less than or equal to the third frame rate, calculate the average value of the second frame rate and the third frame rate, and calculate the average value based on the average value and the first frame rate to calculate the time interval;

Step S36, when the sum of the second frame rate and a preset value is greater than the third frame rate, subtracting the preset increment from the third frame rate to update the third frame rate, And continue to execute the step of judging whether the second frame rate is lower than the third frame rate.

In this embodiment, the specific steps of calculating the time interval of the B frame in the discarded video may be:

Step 1. Determine the third frame rate supported by the terminal according to the performance of the terminal chip. Specifically, the third frame rate is a frame rate directly determined according to the performance of the terminal chip. The frame rates supported by different terminal chips are generally The following are not the same, and will not be repeated here;

Step 2, judging whether the second frame rate is lower than the third frame rate;

Step 3, when the second frame rate is greater than or equal to the third frame rate, increase the preset increment to the third frame rate to update the third frame rate, and continue to perform step 2; specifically Preferably, the preset increment may be 1, and a value other than 1 may also be used in a specific implementation process.

Step 4, when the second frame rate is lower than the third frame rate, judge whether the sum of the second frame rate and a preset value is greater than the third frame rate; specifically, the preset value can be is 5, and values other than 5 can also be used in the specific implementation process.

Step 5, when the sum of the second frame rate and the preset value is less than or equal to the third frame rate, calculate the average value of the second frame rate and the third frame rate, and calculate the average value based on the average value and the first frame rate to calculate the time interval; specifically, the method for calculating the time interval according to the average value and the first frame rate is: combine the first frame rate with the average values to get the difference, and then divide 1000 by the difference to get the time interval.

Step 6, when the sum of the second frame rate and a preset value is greater than the third frame rate, subtracting the preset increment from the third frame rate to update the third frame rate, And proceed to Step 1.

The audio and video frame loss method proposed in this embodiment calculates the time interval for discarding B frames in the video through the first frame rate, the second frame rate, and the second frame rate, so that in the subsequent playback process, according to the time interval Precisely and real-time discarding of B frames in the video, so as to achieve precise and real-time control of discarding B frames that have no effect on video playback, so as to adjust the audio and video synchronization output.

Further, based on the first embodiment of the audio and video frame loss method of the present invention, a third embodiment of the audio and video frame loss method of the present invention is proposed.

In this embodiment, the method also includes:

If there is no B frame in the video, play the next I frame.

In this embodiment, if there is no B frame in the video, the next I frame needs to be played to adjust the output synchronization of audio and video.

Further, referring to FIG. 9 , based on the third embodiment of the audio and video frame loss method of the present invention, a fourth embodiment of the audio and video frame loss method of the present invention is proposed.

In this embodiment, the method also includes:

S50, compare the current system playback time with the actual playback time in real time;

S60, when the time difference between the current system playback time and the actual playback time is greater than the first preset time, then discard the B frame in the video source file;

S70. When the time difference between the current system playback time and the actual playback time is greater than a second preset time, play the next I frame, wherein the first preset time is shorter than the second preset time.

In this embodiment, when the buffering time of the currently playing video is too long, the audio and video will also be out of sync, so it is also necessary to discard the B frame in the video, or play the next I frame to achieve synchronous audio and video output . Specifically, first compare the current system playback time and the actual playback time in real time; when the time difference between the current system playback time and the actual playback time is greater than the first preset time, then discard the B frame in the video source file; When the time difference of the actual playing time is greater than the second preset time, the next I frame is played, wherein the first preset time is less than the second preset time. The first preset time may be 100ms, and the second preset time may be 500ms, and the first preset time and the second preset time may also be set to other values during specific implementation.

The audio and video frame loss method proposed in this embodiment compares the current system playback time with the actual playback time in real time; when the time difference between the current system playback time and the actual playback time is greater than the first preset time, the B frame in the video source file is discarded ; when the time difference between the current system play time and the actual play time is greater than the second preset time, then play the next I frame, wherein the first preset time is less than the second preset time, thereby realizing the sound in the video Video synchronization output.

Further, referring to FIG. 10 , based on any one of the first to fourth embodiments of the audio and video frame loss method of the present invention, a fifth embodiment of the audio and video frame loss method of the present invention is proposed.

In this embodiment, the step S60 includes:

Step S61, calculating the number of B frames to be discarded by the second frame rate of the currently played video and the first frame rate of the currently played video source file;

Step S62, discarding corresponding B frames in the video source file according to the calculated number of B frames to be discarded.

In this embodiment, the number of B frames that need to be discarded can be calculated through the second frame rate of the currently playing video and the first frame rate of the currently playing video source file, and then the number of B frames that need to be discarded is calculated in the video source file The corresponding B frame is discarded. Calculating the number of B frames to be discarded based on the second frame rate of the currently played video and the first frame rate of the currently played video source file is an existing general technique, and will not be repeated here.

The audio and video frame loss method proposed in this embodiment first calculates the number of B frames that need to be discarded by the second frame rate of the current playing video and the first frame rate of the current playing video source file; then according to the calculated B frames that need to be discarded Quantity The corresponding B frames are discarded in the video source file to avoid discarding too many B frames, resulting in discontinuous and paused playback.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to enable a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process conversion made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (10)

1. An audio and video frame loss device, characterized in that the device comprises: The obtaining module is used to obtain the time difference between the audio frame time and the video frame time of the current playback moment in real time or regularly; A judging module, configured to judge whether the video contains a B frame if the absolute value of the time difference is greater than or equal to a preset duration, and the preset duration is a preset multiple of the display duration of a unit video frame; Determining module, for if comprising B frame in the video, then according to the first frame rate of current playing video source file and the second frame rate of currently performing video playback, determine the time interval of discarding the B frame in the video; The first discarding module is configured to discard the B frame according to the time interval, so that the frame rate of the video source file reaches the frame rate of the current video playback. 2. audio and video frame loss device as claimed in claim 1, is characterized in that, described determining module comprises: a determining unit, configured to determine a third frame rate supported by the terminal according to the performance of the terminal chip; a judging unit, configured to judge whether the second frame rate is lower than the third frame rate; an adjustment unit, configured to add a preset increment to the third frame rate to update the third frame rate when the second frame rate is greater than or equal to the third frame rate; The judging unit is further configured to judge whether the sum of the second frame rate and a preset value is greater than the third frame rate when the second frame rate is lower than the third frame rate; A first calculation unit, configured to calculate an average value of the second frame rate and the third frame rate when the sum of the second frame rate and a preset value is less than or equal to the third frame rate, and calculating the time interval according to the average value and the first frame rate; The adjustment unit is further configured to subtract the preset increment from the third frame rate when the sum of the second frame rate and a preset value is greater than the third frame rate, so as to update the The third frame rate. 3. audio and video frame dropping device as claimed in claim 1, is characterized in that, described device also comprises: The playback module is used to play the next I frame if there is no B frame in the video. 4. audio and video frame dropping device as claimed in claim 1, is characterized in that, described device also comprises: The comparison module is used to compare the current system playback time with the actual playback time in real time; The second discarding module is used for discarding the B frame in the video source file when the time difference between the current system playing time and the actual playing time is greater than the first preset time; The playing module is also used for playing the next I frame when the time difference between the current system playing time and the actual playing time is greater than the second preset time, wherein the first preset time is less than the second preset time . 5. audio and video frame loss device as claimed in claim 4, is characterized in that, described second discarding module comprises: The second calculation unit is used to calculate the number of B frames that need to be discarded by the second frame rate of the currently played video and the first frame rate of the currently played video source file; The discarding unit is configured to discard corresponding B frames in the video source file according to the calculated number of B frames to be discarded. 6. A method for audio and video frame loss, characterized in that the method comprises the following steps: Obtain the time difference between the audio frame time and the video frame time at the current playback moment in real time or regularly; If the absolute value of the time difference is greater than or equal to a preset duration, it is judged whether a B frame is included in the video, and the preset duration is a preset multiple of the display duration of a unit video frame; If so, then determine the time interval of discarding the B frame in the video according to the first frame rate of the currently played video source file and the second frame rate of the current video playback; B frames are discarded according to the time interval, so that the frame rate of the video source file reaches the current frame rate of video playback. 7. audio and video frame loss method as claimed in claim 6, is characterized in that, described according to the first frame rate of current play video source file and the second frame rate that carries out video playback at present to determine the B frame in the discarded video The time interval steps include: determining a third frame rate supported by the terminal according to the performance of the terminal chip; judging whether the second frame rate is less than the third frame rate; When the second frame rate is greater than or equal to the third frame rate, add a preset increment to the third frame rate to update the third frame rate, and continue to perform the judging the second frame rate The step of whether the frame rate is less than the third frame rate; When the second frame rate is less than the third frame rate, judging whether the sum of the second frame rate and a preset value is greater than the third frame rate; When the sum of the second frame rate and the preset value is less than or equal to the third frame rate, calculate the average value of the second frame rate and the third frame rate, and based on the average value and the calculating the time interval at the first frame rate; When the sum of the second frame rate and the preset value is greater than the third frame rate, subtract the preset increment from the third frame rate to update the third frame rate, and continue to execute The step of judging whether the second frame rate is lower than the third frame rate. 8. audio and video frame dropping method as claimed in claim 7, is characterized in that, also comprises after the step of B frame in described judging video: If there is no B frame in the video, play the next I frame. 9. The audio and video frame loss method according to any one of claims 6-8, wherein the method further comprises: Compare the current system playback time with the actual playback time in real time; When the time difference between the current system play time and the actual play time is greater than the first preset time, then discard the B frame in the video source file; When the time difference between the current system play time and the actual play time is greater than a second preset time, then play the next I frame, wherein the first preset time is less than the second preset time. 10. audio and video frame loss method as claimed in claim 9, is characterized in that, the step of described discarding the B frame in described video source file comprises: Calculate the number of B frames that need to be discarded by the second frame rate of the currently playing video and the first frame rate of the currently playing video source file; The corresponding B frames are discarded in the video source file according to the calculated number of B frames to be discarded.