WO2025007738A1 - Audio-picture synchronization detection method and apparatus, and device and storage medium - Google Patents

Abstract

Provided in the present disclosure are an audio-picture synchronization detection method and apparatus, and a device and a storage medium. The method comprises: acquiring a first video to be subjection to detection, wherein the first video is a video obtained by means of transcoding a second video with audios and pictures synchronized (S110); extracting a first key frame in the first video and second key frames in the second video, and aligning the first key frame with the second key frames, so as to determine a target second key frame which is aligned with the first key frame (S120); extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame (S130); and matching the first audio data with the second audio data, so as to determine an audio-picture synchronization detection result for the first video (S140). The present disclosure makes it possible to perform automatic audio-picture synchronization detection on a transcoded video, thereby improving the efficiency for audio-picture synchronization detection and ensuring the accuracy of audio-picture synchronization detection.

Description

Audio and video synchronization detection method, device, equipment and storage medium

This application claims priority to Chinese Patent Application No. 202310813430.4 filed on July 4, 2023, the entire text of which is incorporated herein by reference as a part of this application.

Technical Field

The embodiments of the present disclosure relate to a method, apparatus, device and storage medium for detecting audio and video synchronization.

Background Art

With the rapid development of computer technology, it is often necessary to transcode videos. However, in the process of video transcoding, the audio and video are often out of sync, that is, the audio and video images cannot be accurately consistent. For example, the video screen shows that someone is talking but there is no corresponding sound, which greatly reduces the user's viewing experience. At present, it is usually judged by manually watching the video whether there is a situation where the audio and video are out of sync due to transcoding. It can be seen that this manual detection method is time-consuming and labor-intensive, and reduces the detection efficiency of audio and video synchronization.

Summary of the invention

The present disclosure provides a method, device, equipment and storage medium for detecting audio and video synchronization, so as to automatically detect audio and video synchronization of a transcoded video, thereby improving the detection efficiency of audio and video synchronization and ensuring the accuracy of audio and video synchronization detection.

In a first aspect, an embodiment of the present disclosure provides a method for detecting audio and video synchronization, including:

Acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

Extracting a first key frame in the first video and a second key frame in the second video, aligning the first key frame and the second key frame, and determining a target second key frame aligned with the first key frame;

Extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

The first audio data is matched with the second audio data to determine an audio-visual synchronization detection result of the first video.

In a second aspect, the present disclosure also provides an audio-video synchronization detection device, including:

A first video acquisition module is used to acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

A key frame alignment module, used to extract a first key frame in the first video and a second key frame in the second video, align the first key frame with the second key frame, and determine a target second key frame aligned with the first key frame;

An audio data extraction module, used to extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

The audio and video synchronization detection module is used to match the first audio data with the second audio data to determine the audio and video synchronization detection result of the first video.

In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the audio and video synchronization detection method as described in any one of the embodiments of the present disclosure.

In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the audio-visual synchronization detection method as described in any one of the embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

FIG1 is a flow chart of a method for detecting synchronization of audio and video provided by an embodiment of the present disclosure;

FIG2 is a waveform matching diagram of first audio data and second audio data involved in an embodiment of the present disclosure;

FIG3 is a flow chart of another method for detecting synchronization between audio and video provided by an embodiment of the present disclosure;

FIG4 is an example of audio data extraction according to an embodiment of the present disclosure;

FIG5 is a schematic diagram of the structure of a device for detecting synchronization between audio and video provided by an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. In addition, the method embodiments may include additional steps and/or omit the steps shown. The scope of the present disclosure is not limited in this respect.

The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

Figure 1 is a flow chart of a method for detecting audio and video synchronization provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case of performing audio and video synchronization detection on a transcoded video. The method can be executed by an audio and video synchronization detection device, which can be implemented in the form of software and/or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

As shown in FIG1 , the audio-video synchronization detection method specifically includes the following steps:

S110: Acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video.

The first video refers to the video to be detected after transcoding. The second video refers to the video before transcoding. The audio and video images in the second video are synchronized, so the second video can be used as a reference video to detect the synchronization of audio and video of the first video.

Specifically, users can create and publish videos in the client. After receiving the video uploaded by the client, the server needs to transcode the uploaded video, such as adjusting the video resolution and bit rate, so as to improve the picture quality, and send the transcoded video to other clients for playback. In the above application scenario, the video uploaded to the server can be used as the second video with audio and video synchronization, and the transcoded video can be used as the first video to be detected. By using the second video before transcoding as a reference video for audio and video synchronization, it is possible to accurately detect whether the first video has audio and video asynchronization caused by video transcoding.

S120: extract a first key frame in the first video and a second key frame in the second video, align the first key frame and the second key frame, and determine a target second key frame that is aligned with the first key frame.

Among them, a key frame may refer to a video frame that plays a decisive role in the video content, such as a video frame with scene switching, a video frame with motion changes, a video frame where a key action is located, etc. A first key frame may refer to a key frame in a first video. The number of first key frames may be one or more. A second key frame may refer to a key frame in a second video. The number of second key frames may also be one or more. The number of first key frames may be the same as or different from the number of second key frames. Alignment refers to the operation of aligning key frames with the same picture. The target second key frame refers to a second key frame with the same picture as the first key frame.

Specifically, key frames of the first video and the second video are detected, and all first key frames in the first video and all second key frames in the second video are extracted. The first key frames and the second key frames can be aligned by detecting the image similarity between the first key frames and the second key frames. For example, for each first key frame, the image similarity between the first key frame and each second key frame can be determined, and the second key frame with the highest image similarity and greater than or equal to a preset similarity threshold can be used as the target second key frame to be aligned with the first key frame, thereby obtaining a target second key frame having the same picture as the first key frame.

Among them, image similarity can be measured using any indicator that can measure the similarity between two images. For example, image similarity is characterized by the structural similarity SSIM (Structural Similarity) indicator. The value range of the SSIM indicator is 0 to 1, and the larger the SSIM indicator, the higher the image similarity.

It should be noted that if there is no target second key frame identical to a first key frame among all second key frames, it indicates that the first key frame cannot be aligned. In this case, the first key frame can be ignored and only the aligned first key frame can be used for subsequent audio and video synchronization detection.

S130: Extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame.

Specifically, for each first key frame, the first audio data corresponding to the preset audio duration can be extracted at the first key frame position in the first video. At the target second key frame position in the second video, the second audio data corresponding to the preset audio duration is also extracted, so that the extracted first audio data and the second audio data have the same audio duration. For example, the extracted first audio data is audio data with a preset audio duration using the playback timestamp of the first key frame as a preset reference moment. The extracted second audio data is audio data with a preset audio duration using the playback timestamp of the target second key frame as a preset reference moment. The preset reference moment may refer to the start moment, center moment or end moment of the audio data.

It should be noted that the picture content of the aligned first key frame and the target second video frame is the same, but their corresponding playback timestamps may be different. When the extracted first audio data and the second audio data have the same audio duration, the first audio data and the second audio data need to be extracted at the same position of the key frame to ensure the accuracy of subsequent audio and video synchronization detection. For example, the first audio data with a preset audio duration starting at the playback timestamp of the first key frame is extracted from the first video. The second audio data with a preset audio duration starting at the playback timestamp of the target second key frame is extracted from the second video.

S140: Match the first audio data with the second audio data to determine an audio-video synchronization detection result of the first video.

Specifically, FIG2 shows a waveform matching diagram of the first audio data and the second audio data. As shown in FIG2, the first audio data and the second audio data have the same audio duration. In view of this, the matching result between the first audio data and the second audio data can be determined based on the similarity between the two audio waveforms in FIG2. For example, the covariance between the first audio data and the second audio data can be used as the audio similarity between the first audio data and the second audio data, and the audio similarity can be used as the covariance between the first audio data and the second audio data. When the similarity is greater than or equal to the preset similarity threshold, it is determined that the first audio data and the second audio data match successfully. When the audio similarity is less than the preset similarity threshold, it is determined that the first audio data and the second audio data fail to match. If the match is successful, it indicates that the audio and video images at the position of the first key frame are synchronized. If the match fails, it indicates that the audio and video images at the position of the first key frame are not synchronized, and there is a situation where the audio and video are not synchronized due to transcoding. According to the matching results corresponding to each first key frame in the first video, the audio and video synchronization detection result of the first video can be determined. For example, if the matching results corresponding to all the first key frames in the first video are successful matches, the audio and video synchronization detection result of the first video is determined to be audio and video synchronization. If the matching result corresponding to at least one first key frame in the first video is a match failure, it is determined that the audio and video synchronization detection result of the first video is audio and video synchronization, and there is an audio and video synchronization situation caused by transcoding in the first video.

The technical solution of the disclosed embodiment detects the transcoded first video by taking the second video with synchronized audio and video before transcoding as a reference, and matches the first audio data corresponding to the first key frame after key frame alignment with the second audio data corresponding to the target second key frame, so as to accurately determine whether the first video has audio and video asynchronism caused by transcoding, thereby realizing automatic detection of audio and video synchronization and ensuring the accuracy of audio and video synchronization detection.

As an optional embodiment, "aligning the first key frame and the second key frame to determine a target second key frame aligned with the first key frame" in S120 may include: obtaining an optional second key frame sequence for the first key frame; obtaining the current second key frame in sequence according to the order of the second key frame sequence, and determining the image similarity between the first key frame and the current second key frame; if the image similarity is greater than or equal to a preset similarity threshold, determining the current second video frame as the target second key frame aligned with the first key frame.

Among them, image similarity can be characterized by any indicator that can measure the similarity between two images. For example, image similarity is characterized by the structural similarity SSIM (Structural Similarity) indicator. The value range of the SSIM indicator is 0 to 1, and the larger the SSIM indicator, the higher the image similarity.

Specifically, all the extracted first key frames can be sorted in the order of video playback to obtain a first key frame sequence, such as {A1, A2, ..., An}. Similarly, all the extracted second key frames can be sorted in the order of video playback to obtain a second key frame sequence, such as {B1, B2, ..., Bm}. The target second key frame to be aligned with each first key frame can be determined from the second key frame sequence in sequence according to the order of the first key frame sequence.

For example, the optional second key frame sequence of the first first key frame A1 is {B1, B2, ..., Bm}. At this time, according to the order of the second key frame sequence, the second key frame B1 can be first used as the current second key frame, and it is detected whether the image similarity between the first key frame A1 and the second key frame B1 is greater than or equal to the preset similarity threshold; if so, it indicates that the first key frame A1 and the second key frame B1 are similar in picture. At this time, the second key frame B1 can be directly used as the target second key frame to be aligned with the first key frame A1; if not, the next second key frame B2 is used as the current second key frame to re-detect whether the image similarity between the first key frame A1 and the second key frame B2 is greater than or equal to the preset similarity threshold, until the second key frame matching the first key frame A1 is determined, thereby completing the alignment of the first key frame A1. Assuming that the target second key frame aligned with the first key frame A1 is B2, when aligning the second first key frame A2, since the target second key frame aligned with the first key frame A2 appears in the key frame after the second key frame B2, the second key frame after the second key frame B2 can be used as the optional second key frame sequence of the first key frame A2, that is, {B3, B4, ..., Bm}, and based on the same detection process as above, the target second key frame aligned with the first key frame A2 is determined from the second key frame sequence {B3, B4, ..., Bm}. Similarly, the execution is carried out sequentially, and the target second key frame aligned with each first key frame can be obtained more quickly, thereby improving the key frame alignment speed and further improving the detection efficiency of audio and video synchronization.

FIG3 is a flow chart of another method for detecting audio-visual synchronization provided by an embodiment of the present disclosure. Based on the above disclosed embodiments, the audio duration of the extracted first audio data is greater than the audio duration of the second audio data, and on this basis, the step of "matching the first audio data with the second audio data to determine the audio-visual synchronization detection result of the first video" is optimized. Explanations of terms that are the same or corresponding to the above disclosed embodiments are not repeated here.

As shown in FIG3 , the audio-video synchronization detection method specifically includes the following steps:

S310: Obtain a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video.

S320: extract a first key frame in the first video and a second key frame in the second video, align the first key frame and the second key frame, and determine a target second key frame that is aligned with the first key frame.

S330: Extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame.

The first audio data corresponds to an audio duration of the first preset duration, and the second audio data corresponds to an audio duration of the first preset duration. The audio duration is the second preset duration. The first preset duration is greater than the second preset duration. For example, the first preset duration is 500ms and the second preset duration is 20ms.

Specifically, for each first key frame, the first audio data corresponding to the first preset duration can be extracted at the first key frame position in the first video. The second audio data corresponding to the second preset duration can be extracted at the target second key frame position in the second video, so that the audio duration of the extracted first audio data is greater than the audio duration of the second audio data, so that the second audio data can be matched in the first audio data later.

Exemplarily, the extraction of the first audio data needs to be performed within the left and right range of the first key frame position so as to perform accurate audio matching later. For example, the first audio data can be audio data with a first preset duration centered at the playback timestamp of the first key frame. FIG4 shows an example of first audio data extraction. Referring to FIG4, first audio data with a first preset duration of 500ms centered at the playback timestamp of each first key frame is extracted from the audio of the first video.

Exemplarily, there may be a variety of ways to extract the second audio data used as a reference, and the data may be extracted to the left, to the left, or in the left and right range of the target second key frame position. For example, the second audio data may be audio data having a second preset duration with the playback timestamp of the target second key frame as a preset reference moment. The preset reference moment may be one of the following: a start moment, a center moment, and an end moment. FIG4 also shows an example of second audio data extraction. Referring to FIG4, second audio data having a second preset duration of 20 ms and starting at the playback timestamp of each target second key frame is extracted from the audio of the second video.

S340: Determine, from the first audio data, target audio data having a second preset duration and matching the second audio data.

The target audio data may refer to the audio data in the first audio data that successfully matches the second audio data. In other words, the target audio data and the second audio data may be approximately the same data.

Specifically, since the duration of the first audio data is greater than that of the second audio data, the first audio data can be traversed to extract multiple audio data to be selected with the second preset duration, for example, one audio data to be selected with the second preset duration is extracted every 1 ms. The audio similarity between each audio data to be selected and the second audio data can be determined, and the audio data to be selected with the greatest audio similarity is used as the target audio data to match the second audio data.

Exemplarily, S340 may include: based on a preset sliding duration and a second preset duration, The audio data is slid to obtain current audio data with a second preset duration corresponding to the current sliding; the audio similarity between the current audio data and the second audio data is determined; and based on the audio similarity corresponding to the current audio data, the target audio data matching the second audio data is determined.

Specifically, based on the preset sliding duration, the first audio data can be slid successively to obtain the current audio data with the second preset duration. For example, when the preset sliding duration is 1ms, referring to FIG. 4, the current audio data obtained by the first sliding is the audio data from the 0th ms to the 20th ms in the first audio data, and the current audio data obtained by the second sliding is the audio data from the 1st ms to the 21st ms in the first audio data, and so on. After each sliding, the covariance between the current audio data obtained by the sliding and the second audio data can be determined, and the covariance can be used as the audio similarity corresponding to the current audio data. If the audio similarity is greater than or equal to the preset similarity threshold, it can be determined that the current audio data matches the second audio data, and the current audio data is used as the target audio data. If the audio similarity is less than the preset similarity threshold, it can be determined that the current audio data does not match the second audio data. At this time, it is necessary to update the current audio data by sliding, and detect whether the current audio data obtained by the next sliding matches the second audio data, until the matching target audio data is determined, the sliding is stopped. The above-mentioned sliding matching method does not need to traverse all audio data with the second preset duration in the first audio data. The target audio data matching the second audio data can be obtained more quickly while ensuring the accuracy of synchronization detection, thereby further improving the efficiency of audio and video synchronization detection.

S350: Taking the second audio data and the target second key frame as a reference for audio-visual synchronization, determine an audio-visual offset between the target audio data and the first key frame.

The audio-video offset may refer to the time difference between the sound of the target audio data and the picture of the first key frame.

Specifically, since the second video is synchronized in audio and video, and thus the second audio data and the target second key frame are synchronized in audio and video, the first time difference between the second audio data and the target second key frame can be used as a reference time difference for audio and video synchronization, and the target time difference between the target audio data and the first key frame is detected. If the target time difference is equal to the reference time difference, it indicates that the target audio data and the first key frame are synchronized in audio and video, and the audio and video offset at this time is 0. If the target time difference is not equal to the reference time difference, the specific audio and video offset can be determined based on the difference between the target time difference and the reference time difference.

Exemplarily, S350 may include: determining a reference time difference for audio-visual synchronization based on the start timestamp of the second audio data and the play timestamp of the target second key frame; determining a reference time difference for audio-visual synchronization based on the start timestamp of the target audio data; The target time difference corresponding to the first key frame is determined based on the timestamp and the playback timestamp of the first key frame; and the audio and video offset corresponding to the first key frame is determined based on the target time difference and the reference time difference.

Specifically, the start timestamp of the second audio data can be subtracted from the playback timestamp of the target second key frame, and the obtained time difference value can be determined as the reference time difference for audio and video synchronization. For the extraction method of the second audio data in Figure 4, the start timestamp of the second audio data is the playback timestamp of the target second key frame, so that the reference time difference for audio and video synchronization is 0. The start timestamp of the target audio data is subtracted from the playback timestamp of the first key frame, and the obtained time difference value is determined as the target time difference corresponding to the first key frame. The target time difference is subtracted from the reference time difference, and the obtained time difference value is determined as the audio and video offset corresponding to the first key frame. Referring to Figure 4, the reference parallax is 0, and the target time difference is the audio and video offset. If the audio and video offset is greater than 0, it indicates that the audio is ahead of the video picture. If the audio and video offset is less than 0, it indicates that the audio lags behind the video picture.

S360: Determine an audio and video synchronization detection result of the first video based on the audio and video offset corresponding to the first key frame.

Specifically, based on the audio and video offset corresponding to each first key frame in the first video, the audio and video synchronization detection result of the first video as a whole can be determined.

Exemplarily, S360 may include: if the audio and video offsets corresponding to each first video frame in the first video are within a preset allowable range, then determining that the audio and video synchronization detection result of the first video is audio and video synchronization; if there is at least one first video frame in the first video whose audio and video offsets correspond to are not within the preset allowable range, then determining that the audio and video synchronization detection result of the first video is audio and video asynchrony.

The preset allowable range may be a time difference range of the allowed audio and video offset that is pre-set based on the detection standard. For example, the preset allowable range is: [-185, 90], which indicates that, with respect to the video screen, an audio lag of 185ms or an audio lead of 90ms is acceptable and is considered to be synchronized with the audio and video.

Specifically, detect whether the audio and video offset corresponding to each first video frame in the first video is within the preset allowable range. If the audio and video offsets corresponding to all first video frames are within the preset allowable range, determine that the audio and video synchronization detection result of the first video is audio and video synchronization, that is, the audio and video of the transcoded first video is also synchronized, and the video transcoding does not cause the audio and video to be out of sync. If there is at least one first video frame whose audio and video offset corresponding to the first video frame is not within the preset allowable range, determine that the audio and video synchronization detection result of the first video is audio and video out of sync, that is, audio and video out of sync caused by transcoding occurs at the position of the first video frame that is not within the preset allowable range. At this time, the audio and video offset corresponding to the first video frame that is not within the preset allowable range can be output to remind the user that the audio and video are out of sync. The specific location and degree of asynchrony between audio and video allow users to process the problem quickly.

The technical solution of the disclosed embodiment extracts the first audio data with a longer audio duration, extracts the target audio data matching the second audio data again from the first audio data, and uses the second audio data and the target second key frame as the reference for audio and video synchronization, determines the audio and video offset between the target audio data and the first key frame, and based on the audio and video offset corresponding to the first key frame, can more accurately determine the audio and video synchronization detection result of the first video, thereby further improving the accuracy of the audio and video synchronization detection.

FIG5 is a schematic diagram of the structure of an audio-video synchronization detection device provided by an embodiment of the present disclosure. As shown in FIG5 , the device specifically includes: a first video acquisition module 510 , a key frame alignment module 520 , an audio data extraction module 530 and an audio-video synchronization detection module 540 .

Among them, the first video acquisition module 510 is used to acquire the first video to be detected, where the first video is a video obtained by transcoding the second video with synchronized audio and video; the key frame alignment module 520 is used to extract the first key frame in the first video and the second key frame in the second video, and align the first key frame and the second key frame to determine the target second key frame aligned with the first key frame; the audio data extraction module 530 is used to extract the first audio data corresponding to the first key frame and the second audio data corresponding to the target second key frame; the audio and video synchronization detection module 540 is used to match the first audio data with the second audio data to determine the audio and video synchronization detection result of the first video.

The technical solution provided by the embodiment of the present disclosure detects the transcoded first video by taking the second video with synchronized audio and video before transcoding as a reference, and matches the first audio data corresponding to the first key frame after key frame alignment with the second audio data corresponding to the target second key frame, so as to accurately determine whether the first video has audio and video asynchronism caused by transcoding, thereby realizing automatic detection of audio and video synchronization and ensuring the accuracy of audio and video synchronization detection.

Based on the above technical solution, the key frame alignment module 520 is specifically used for:

Obtain an optional second key frame sequence of the first key frame; obtain the current second key frame in sequence according to the order of the second key frame sequence, and determine the image similarity between the first key frame and the current second key frame; if the image similarity is greater than or equal to a preset similarity threshold, determine the current second video frame as the target second key frame aligned with the first key frame.

On the basis of the above technical solutions, the audio duration corresponding to the first audio data is a first preset duration, and the audio duration corresponding to the second audio data is a second preset duration, wherein the first The first preset time length is greater than the second preset time length.

Based on the above technical solutions, the first audio data is audio data with a first preset duration and a playback timestamp of the first key frame as the center moment;

The second audio data is audio data having a second preset duration with the playback timestamp of the target second key frame as a preset reference time; wherein the preset reference time is one of the following:

The beginning moment, the center moment, and the end moment.

Based on the above technical solutions, the audio and video synchronization detection module 540 includes:

a target audio data determining unit, configured to determine, from the first audio data, target audio data having the second preset duration and matching the second audio data;

an audio-visual offset determination unit, configured to determine an audio-visual offset between the target audio data and the first key frame by taking the second audio data and the target second key frame as an audio-visual synchronization reference;

The audio and video synchronization detection unit is used to determine the audio and video synchronization detection result of the first video based on the audio and video offset corresponding to the first key frame.

On the basis of the above technical solutions, the target audio data determination unit is specifically used to:

Based on the preset sliding duration and the second preset duration, the first audio data is slid to obtain the current audio data with the second preset duration corresponding to the sliding; the audio similarity between the current audio data and the second audio data is determined; based on the audio similarity corresponding to the current audio data, the target audio data matching the second audio data is determined.

On the basis of the above technical solutions, the audio and video offset determination unit is specifically used to:

Based on the start timestamp of the second audio data and the playback timestamp of the target second key frame, determine the reference time difference for audio and video synchronization; based on the start timestamp of the target audio data and the playback timestamp of the first key frame, determine the target time difference corresponding to the first key frame; based on the target time difference and the reference time difference, determine the audio and video offset corresponding to the first key frame.

On the basis of the above technical solutions, the audio and video synchronization detection unit is specifically used for:

If the audio and video offsets corresponding to each first video frame in the first video are within the preset allowable range, the audio and video synchronization detection result of the first video is determined to be audio and video synchronization; if there is at least one first video frame in the first video whose audio and video offsets are not within the preset allowable range, the audio and video synchronization detection result of the first video is determined to be audio and video out of synchronization.

The audio-video synchronization detection device provided in the embodiments of the present disclosure can execute the audio-video synchronization detection device provided in any embodiment of the present disclosure. The provided audio and video synchronization detection method has the corresponding functional modules and beneficial effects for executing the audio and video synchronization detection method.

It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

FIG6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to FIG6 below, a schematic diagram of the structure of an electronic device (e.g., a terminal device or server in FIG6 ) 500 suitable for implementing an embodiment of the present disclosure is shown. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG6 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG6 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit/output (I/O) interface 505 is also connected to the bus 504.

Typically, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 6 shows an electronic device 500 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer readable medium, the computer program including a computer program for executing the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above functions defined in the method of the embodiment of the present disclosure are performed.

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

The electronic device provided by the embodiment of the present disclosure and the audio and video synchronization detection method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the audio and video synchronization detection method provided by the above embodiment is implemented.

It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server may utilize any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol). The network may communicate using any protocol and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internetwork (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any network currently known or developed in the future.

The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video; extracts a first key frame in the first video and a second key frame in the second video, and aligns the first key frame and the second key frame to determine a target second key frame aligned with the first key frame; extracts first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame; matches the first audio data with the second audio data to determine an audio and video synchronization detection result of the first video.

Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession may actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that It is noted that each block in the block diagram and/or flowchart, and combinations of blocks in the block diagram and/or flowchart, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, Example 1 provides a method for detecting audio and video synchronization, including:

Acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

Extracting a first key frame in the first video and a second key frame in the second video, aligning the first key frame and the second key frame, and determining a target second key frame aligned with the first key frame;

Extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

The first audio data is matched with the second audio data to determine the first video The audio and video synchronization detection results.

According to one or more embodiments of the present disclosure, Example 2 provides a method for detecting audio and video synchronization, further comprising:

Optionally, aligning the first key frame and the second key frame to determine a target second key frame aligned with the first key frame includes:

Obtain a second key frame sequence that is optional for the first key frame;

Obtaining the current second key frame in sequence according to the order of the second key frame sequence, and determining the image similarity between the first key frame and the current second key frame;

If the image similarity is greater than or equal to a preset similarity threshold, the current second video frame is determined as a target second key frame aligned with the first key frame.

According to one or more embodiments of the present disclosure, Example 3 provides a method for detecting audio-visual synchronization, further comprising:

Optionally, the audio duration corresponding to the first audio data is a first preset duration, and the audio duration corresponding to the second audio data is a second preset duration, wherein the first preset duration is greater than the second preset duration.

According to one or more embodiments of the present disclosure, Example 4 provides a method for detecting audio and video synchronization, further comprising:

Optionally, the first audio data is audio data with a first preset duration and a playback timestamp of the first key frame as the center moment;

The second audio data is audio data having a second preset duration with the playback timestamp of the target second key frame as a preset reference time; wherein the preset reference time is one of the following:

The beginning moment, the center moment, and the end moment.

According to one or more embodiments of the present disclosure, Example 5 provides a method for detecting audio and video synchronization, further comprising:

Optionally, matching the first audio data with the second audio data to determine an audio-video synchronization detection result of the first video includes:

Determine, from the first audio data, target audio data having the second preset duration and matching the second audio data;

Taking the second audio data and the target second key frame as an audio-visual synchronization reference, determining an audio-visual offset between the target audio data and the first key frame;

Based on the audio and video offset corresponding to the first key frame, an audio and video synchronization detection result of the first video is determined.

According to one or more embodiments of the present disclosure, Example 6 provides a method for detecting audio and video synchronization, further comprising:

Optionally, the determining, from the first audio data, target audio data having the second preset duration and matching the second audio data includes:

Based on the preset sliding duration and the second preset duration, sliding the first audio data, and acquiring the current audio data with the second preset duration corresponding to the sliding;

Determining audio similarity between the current audio data and the second audio data;

Based on the audio similarity corresponding to the current audio data, target audio data matching the second audio data is determined.

According to one or more embodiments of the present disclosure, Example 7 provides a method for detecting audio-visual synchronization, further comprising:

Optionally, the determining the audio and video offset between the target audio data and the first key frame by taking the second audio data and the target second key frame as an audio and video synchronization reference includes:

Determine a reference time difference for audio and video synchronization based on a start timestamp of the second audio data and a play timestamp of the target second key frame;

Determining a target time difference corresponding to the first key frame based on a start timestamp of the target audio data and a play timestamp of the first key frame;

Based on the target time difference and the reference time difference, an audio and video offset corresponding to the first key frame is determined.

According to one or more embodiments of the present disclosure, Example 8 provides a method for detecting audio and video synchronization, further comprising:

Optionally, determining the audio and video synchronization detection result of the first video based on the audio and video offset corresponding to the first key frame includes:

If the audio and video offset corresponding to each first video frame in the first video is within a preset allowable range, determining that the audio and video synchronization detection result of the first video is audio and video synchronization;

If the audio and video offset corresponding to at least one first video frame in the first video is not within a preset allowable range, it is determined that the audio and video synchronization detection result of the first video is that the audio and video are out of synchronization.

According to one or more embodiments of the present disclosure, Example 9 provides a device for detecting audio-visual synchronization. include:

A first video acquisition module is used to acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video;

A key frame alignment module, used to extract a first key frame in the first video and a second key frame in the second video, align the first key frame with the second key frame, and determine a target second key frame aligned with the first key frame;

An audio data extraction module, used to extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame;

The audio and video synchronization detection module is used to match the first audio data with the second audio data to determine the audio and video synchronization detection result of the first video.

The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Although the subject matter has been described in language specific to structural features and/or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims (11)

A method for detecting audio and video synchronization, comprising: Acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video; Extracting a first key frame in the first video and a second key frame in the second video, aligning the first key frame and the second key frame, and determining a target second key frame aligned with the first key frame; Extracting first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame; The first audio data is matched with the second audio data to determine an audio-visual synchronization detection result of the first video. According to the audio-visual synchronization detection method of claim 1, wherein aligning the first key frame and the second key frame and determining the target second key frame aligned with the first key frame comprises: Obtain a second key frame sequence that is optional for the first key frame; Obtaining the current second key frame in sequence according to the order of the second key frame sequence, and determining the image similarity between the first key frame and the current second key frame; If the image similarity is greater than or equal to a preset similarity threshold, the current second video frame is determined as a target second key frame aligned with the first key frame. The audio-visual synchronization detection method according to claim 1, wherein the audio duration corresponding to the first audio data is a first preset duration, and the audio duration corresponding to the second audio data is a second preset duration, wherein the first preset duration is greater than the second preset duration. The audio-visual synchronization detection method according to claim 3, wherein the first audio data is audio data with a first preset duration and a playback timestamp of the first key frame as the center moment; The second audio data is audio data having a second preset duration with the playback timestamp of the target second key frame as a preset reference time; wherein the preset reference time is one of the following: The beginning moment, the center moment, and the end moment. The method for detecting audio-visual synchronization according to claim 3, wherein the first audio The method of matching the first video data with the second audio data to determine the audio and video synchronization detection result of the first video includes: Determine, from the first audio data, target audio data having the second preset duration and matching the second audio data; Taking the second audio data and the target second key frame as an audio-visual synchronization reference, determining an audio-visual offset between the target audio data and the first key frame; Based on the audio and video offset corresponding to the first key frame, an audio and video synchronization detection result of the first video is determined. The method for detecting audio-visual synchronization according to claim 5, wherein the step of determining the target audio data having the second preset duration and matching the second audio data from the first audio data comprises: Based on the preset sliding duration and the second preset duration, sliding the first audio data, and acquiring the current audio data with the second preset duration corresponding to the sliding; Determining audio similarity between the current audio data and the second audio data; Based on the audio similarity corresponding to the current audio data, target audio data matching the second audio data is determined. According to the audio-visual synchronization detection method of claim 5 or 6, wherein the step of determining the audio-visual offset between the target audio data and the first key frame using the second audio data and the target second key frame as an audio-visual synchronization reference comprises: Determine a reference time difference for audio and video synchronization based on a start timestamp of the second audio data and a play timestamp of the target second key frame; Determining a target time difference corresponding to the first key frame based on a start timestamp of the target audio data and a play timestamp of the first key frame; Based on the target time difference and the reference time difference, an audio and video offset corresponding to the first key frame is determined. According to any one of claims 5 to 7, the method for detecting audio and video synchronization, wherein the step of determining the audio and video synchronization detection result of the first video based on the audio and video offset corresponding to the first key frame comprises: If the audio and video offset corresponding to each first video frame in the first video is within a preset allowable range, determining that the audio and video synchronization detection result of the first video is audio and video synchronization; If the audio and video offset corresponding to at least one first video frame in the first video is not within a preset allowable range, it is determined that the audio and video synchronization detection result of the first video is that the audio and video are out of synchronization. A device for detecting synchronization between audio and video, comprising: A first video acquisition module is used to acquire a first video to be detected, where the first video is a video obtained by transcoding a second video with synchronized audio and video; A key frame alignment module, used to extract a first key frame in the first video and a second key frame in the second video, align the first key frame with the second key frame, and determine a target second key frame aligned with the first key frame; An audio data extraction module, used to extract first audio data corresponding to the first key frame and second audio data corresponding to the target second key frame; The audio and video synchronization detection module is used to match the first audio data with the second audio data to determine the audio and video synchronization detection result of the first video. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the audio-visual synchronization detection method as described in any one of claims 1-8. A storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute the audio-video synchronization detection method as described in any one of claims 1-8 when executed by a computer processor.