HK40067617B - Method and apparatus for recognizing video segment, device and storage medium - Google Patents

Description

Methods, devices, equipment, and storage media for video clip recognition

Technical Field

This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for identifying video clips.

Background Technology

With the development of computer technology, video content has increased dramatically, and more and more users are watching videos online. Videos include TV series, which typically have opening and ending credits. To facilitate viewing, video platforms offer the function of skipping the opening and ending credits. The basis for skipping these credits is determining their exact positions within the TV series.

In related technologies, the positions of the opening and closing credits of a TV series are determined by manual annotation, that is, a person watches the TV series and then marks the positions of the opening and closing credits.

However, manual annotation requires a lot of time and manpower, resulting in low efficiency in determining the positions of the opening and closing credits of TV series.

Summary of the Invention

This application provides a method, apparatus, device, and storage medium for identifying video clips, which can improve the efficiency of determining the positions of the opening and closing credits of a TV series. The technical solution is as follows:

On the one hand, a method for identifying video clips is provided, the method comprising:

Based on the video frame features of the first video and the video frame features of at least one second video, multiple video frame pairs are determined. The video frame pairs include a first video frame and a second video frame whose similarity meets the similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video.

Based on the occurrence time difference of the plurality of video frame pairs, the first video frame in the plurality of video frame pairs is fused to obtain at least one candidate video segment in the first video. The occurrence time difference refers to the difference between the occurrence times of two video frames in the video frame pair in the video.

Based on the at least one candidate video segment and the target time range, at least one target video segment in the first video is determined, wherein the target video segment is within the target time range of the first video.

On the one hand, a video clip recognition device is provided, the device comprising:

A video frame pair determination module is used to determine multiple video frame pairs based on video frame features of a first video and video frame features of at least one second video. The video frame pairs include a first video frame and a second video frame whose similarity meets a similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video.

The fusion module is used to fuse the first video frame in the plurality of video frame pairs based on the occurrence time difference of the plurality of video frame pairs to obtain at least one candidate video segment in the first video. The occurrence time difference refers to the difference between the occurrence times of two video frames in the video frame pair in the video.

A target video segment determination module is used to determine at least one target video segment in the first video based on the at least one candidate video segment and a target time range, wherein the target video segment is within the target time range of the first video.

In one possible implementation, the fusion module is configured to divide the plurality of video frame pairs into a plurality of video frame groups based on the occurrence time difference of the plurality of video frame pairs, wherein video frame pairs in the same video frame group correspond to the same occurrence time difference; for any video frame group in the plurality of video frame groups, the first video frame of the video frame pair in the video frame group is fused into a candidate video segment according to the occurrence time of the first video frame of the video frame pair in the first video.

In one possible implementation, the fusion module is used to divide video frame pairs with the same occurrence time difference into an initial video frame group; and to fuse the multiple initial video frame groups based on the occurrence time difference corresponding to the multiple initial video frame groups to obtain the multiple video frame groups.

In one possible implementation, the fusion module is used to sort the plurality of initial video frame groups according to a target order to obtain a plurality of candidate video frame groups; if the matching time difference between any two adjacent candidate video frame groups in the plurality of candidate video frame groups meets the matching time difference condition, the two adjacent candidate video frame groups are fused into one video frame group, wherein the matching time difference refers to the difference between the occurrence time differences of the corresponding two adjacent candidate video frame groups.

In one possible implementation, the two adjacent candidate video frame groups include a first candidate video frame group and a second candidate video frame group. The fusion module is used to add video frame pairs from the first candidate video frame group to the second candidate video frame group to obtain the video frame group when the matching time difference between the occurrence time difference corresponding to the first candidate video frame group and the occurrence time difference corresponding to the second candidate video frame group is less than or equal to a matching difference threshold.

In one possible implementation, the fusion module is configured to add video frame pairs from the first candidate video frame group to the second candidate video frame group; based on the occurrence time difference value corresponding to the second candidate video frame group, a reference second video frame is used to replace the target second video frame to obtain the video frame group, wherein the target second video frame is the second video frame newly added to the second candidate video frame group, the reference second video frame is the second video frame in the second video whose occurrence time difference with the target first video frame is the occurrence time difference value corresponding to the second candidate video frame group, and the target first video frame is the first video frame in the video frame pair to which the target second video frame belongs.

In one possible implementation, the fusion module is configured to compare the occurrence times of the first video frames of any two adjacent video frame pairs in the video frame group within the first video; if the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video meets the occurrence time condition, add the two adjacent video frame pairs to a temporary frame list; if the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video does not meet the occurrence time condition, fuse the video frame pairs in the temporary frame list into a reference video segment; and determine the at least one candidate video segment based on multiple reference video segments.

In one possible implementation, the plurality of reference video segments includes a first overlapping video segment and/or a second overlapping video segment. The first overlapping video segment refers to a reference video segment belonging to the first reference video segment among the plurality of reference video segments, and the second overlapping video segment refers to a reference video segment that partially overlaps with the second reference video segment among the plurality of reference video segments. The fusion module is configured to perform at least one of the following:

If the plurality of reference video segments include the first overlapping video segment, the first overlapping video segment is deleted to obtain the at least one candidate video segment;

If the plurality of reference video segments include the second overlapping video segment, the overlapping portion between the second overlapping video segment and the second reference segment is deleted to obtain the at least one candidate video segment.

In one possible implementation, the fusion module is further configured to: compare the duration of a third type of reference video segment with a target duration, wherein the third type of reference video segment refers to the second overlapping video segment with the overlapping portion deleted; retain the third type of reference video segment if the duration of the third type of reference video segment is greater than or equal to the target duration; and delete the third type of reference video segment if the duration of the third type of reference video segment is less than the target duration.

In one possible implementation, the target video segment determination module is configured to determine at least one target candidate video segment based on the at least one candidate video segment, wherein the number of times the target candidate video segment appears in the at least one candidate video segment meets a frequency condition;

If the occurrence time of any of the target candidate video segments in the first video falls within the target time range, the target candidate video segment is determined as the target video segment in the first video.

In one possible implementation, the target video segment determination module is configured to determine at least one reference candidate video segment based on the at least one candidate video segment; determine the number of times each of the reference candidate video segments appears in the at least one reference candidate video segment; and determine the reference candidate video segments whose number of appearances meets the appearance count condition as target candidate video segments.

In one possible implementation, the at least one candidate video segment includes a third overlapping video segment and/or a fourth overlapping video segment. The third overlapping video segment refers to a candidate video segment belonging to a first candidate video segment among the at least one candidate video segments, and the fourth overlapping video segment refers to a candidate video segment that partially overlaps with a second candidate video segment among the at least one candidate video segments. The target video segment determination module is configured to perform at least one of the following:

If the at least one candidate video segment includes the third overlapping video segment, the third overlapping video segment is deleted to obtain the at least one reference candidate video segment;

If the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment meets the overlap condition, the occurrence count of the fourth overlapping video segment is determined; based on the occurrence count of the fourth overlapping video segment, the at least one reference candidate video segment is determined.

If the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment does not meet the overlap condition, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment.

If the at least one candidate video segment includes the fourth overlapping video segment, and the duration of the fourth overlapping video segment is less than that of the second candidate video segment, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment.

In one possible implementation, the target video segment determination module is configured to perform any of the following:

If the number of occurrences of the fourth overlapping video segment is greater than or equal to the first occurrence threshold, the fourth overlapping video segment is fused with the second candidate video segment to obtain the at least one reference candidate video segment.

If the number of occurrences of the fourth overlapping video segment is less than the first occurrence threshold, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment.

In one possible implementation, the device further includes:

The feature extraction module is used to extract features from multiple target video frames of the target video to be identified, and to obtain the video frame features of the multiple target video frames.

The target video segment determination module is further configured to determine at least one target video segment of the target video based on the video frame features of the plurality of target video frames, the video frame features of the first video frame, and the video frame features of the at least one second video.

On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the video segment recognition method.

On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the method for recognizing the video segment.

On the one hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned method for recognizing video segments.

The technical solution provided in this application determines video frame pairs containing similar video frames based on the similarity between video frame features. The first video frame in the video frame pair is fused based on the time difference of its occurrence to obtain at least one candidate video segment. Finally, a target video segment within a target time range is determined from the at least one candidate video segment. The process of determining the target segment requires no manual intervention; it can be automatically performed by computer equipment directly based on the first video and at least one second video, resulting in high efficiency.

Attached Figure Description

To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

Figure 1 is a schematic diagram of the implementation environment of a video segment recognition method provided in an embodiment of this application;

Figure 2 is a flowchart of a video segment recognition method provided in an embodiment of this application;

Figure 3 is a flowchart of a video segment recognition method provided in an embodiment of this application;

Figure 4 is a flowchart of a method for extracting video frame features according to an embodiment of this application;

Figure 5 is a schematic diagram of a first sub-segment and a second sub-segment provided in an embodiment of this application;

Figure 6 is a schematic diagram of a first sub-segment with different overlapping methods provided in an embodiment of this application;

Figure 7 is a schematic diagram of candidate video segment fusion provided in an embodiment of this application;

Figure 8 is a flowchart of a video segment recognition method provided in an embodiment of this application;

Figure 9 is a flowchart of a fragment mining system provided in an embodiment of this application;

Figure 10 is a flowchart of a method for obtaining the opening and closing credits of a TV series according to an embodiment of this application;

Figure 11 is a schematic diagram of a storage method for a fragment database provided in an embodiment of this application;

Figure 12 is a flowchart of a method for obtaining the opening and closing credits of a TV series according to an embodiment of this application;

Figure 13 is a flowchart of a method for identifying infringing videos provided in an embodiment of this application;

Figure 14 is a flowchart of a video segment recognition method provided in an embodiment of this application;

Figure 15 is a schematic diagram of a video segment recognition device provided in an embodiment of this application;

Figure 16 is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

Figure 17 is a schematic diagram of the structure of a server provided in an embodiment of this application.

Detailed Implementation

To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating/interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning/deep learning.

Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

Hamming distance: used to measure the distance between binary features. It is achieved by counting the number of feature bits with different values. For example, the Hamming distance between (1000) and (0011) is 3.

It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Figure 1 is a schematic diagram of the implementation environment of a video segment recognition method provided in an embodiment of this application. Referring to Figure 1, the implementation environment may include a terminal 110 and a server 140.

Terminal 110 is connected to server 140 via a wireless or wired network. Optionally, terminal 110 may be a vehicle-mounted terminal, smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or smart TV, but is not limited to these. Terminal 110 has an application installed and running that supports video clip recognition.

Server 140 is a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. Server 140 provides background services for the applications running on terminal 110.

The embodiments of this application do not limit the number of terminals 110 and servers 140.

After introducing the implementation environment of the embodiments of this application, the application scenarios of the embodiments of this application will be introduced below in conjunction with the above implementation environment. In the following description, the terminal is the terminal 110 in the above implementation environment, and the server is the server 140 in the above implementation environment.

The video segment recognition method provided in this application embodiment can be applied to scenarios of recognizing the opening and closing credits of videos, such as recognizing the opening and closing credits of TV series, documentaries, or short video collections.

Taking the video segment recognition method provided in this application embodiment as an example in the scenario of recognizing the opening and closing credits of a TV series, the technician selects the TV series for which the opening and closing credits need to be recognized via a terminal. The TV series includes multiple videos, each video being an episode of the TV series. When the TV series is selected via the terminal, the server can use the technical solution provided in this application embodiment to process the multiple videos in the TV series to obtain the opening and closing credits of the multiple videos. During the processing of the multiple videos, the server determines multiple video frame pairs based on the video frame features of the first video and the video frame features of at least one second video. Each video frame pair includes a first video frame and a second video frame whose similarity meets the similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video. That is, each video frame pair includes one video frame from the first video and one video frame from the second video, and both the first video and the at least one second video frame belong to the multiple videos. Based on the time difference between the occurrences of multiple video frame pairs, the server fuses the first video frames from each pair to obtain at least one candidate video segment from the first video. The time difference refers to the difference in the occurrence times of two video frames within a video frame pair, specifically the difference between the occurrence times of the first and second video frames in the first video. Based on at least one candidate video segment and a target time range, the server determines at least one target video segment from the first video. Since this is applied to the scenario of identifying the opening and closing credits of a TV series, the target time period is the time period containing the opening or closing credits, and the determined target video segment is the opening or closing credits of the first video.

It should be noted that the above description is based on the application of the video segment recognition method provided in the embodiments of this application to the scenario of recognizing the opening and closing credits of a TV series. The implementation process of the other application scenarios described above belongs to the same inventive concept as the above description, and the implementation process will not be repeated.

In addition, the video segment recognition method provided in this application embodiment can be applied not only to recognizing the opening and closing scenes of TV series, documentary series, and short video collections, but also to recognizing the opening and closing scenes of other types of videos. This application embodiment does not limit this application.

After introducing the implementation environment and application scenarios of the embodiments of this application, the video segment recognition method provided by the embodiments of this application will be described below. Referring to Figure 2, the technical solution provided by the embodiments of this application can be executed by a terminal or a server, or by both a terminal and a server. In this embodiment of the application, the execution subject is the server as an example for description, and the method includes:

201. The server determines multiple video frame pairs based on the video frame features of a first video and the video frame features of at least one second video. The video frame pairs include a first video frame and a second video frame whose similarity meets the similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video.

In this setup, the first video and at least one second video belong to the same video set; for example, the first video and the second video may be different episodes of the same TV series. Video frame features are the embedding features of the video frames, such as deep hash features. The similarity between the first video frame and the second video frame is determined using the video frame features of both the first and second video frames. Each video frame pair includes one first video frame and one second video frame, and the similarity between the first and second video frames in the pair meets a similarity condition, meaning that the first and second video frames in the pair are highly similar.

202. Based on the occurrence time difference of the multiple video frame pairs, the server fuses the first video frame in the multiple video frame pairs to obtain at least one candidate video segment in the first video. The occurrence time difference refers to the difference between the occurrence times of the two video frames in the video frame pair in the video.

In this context, the first video frame in a video frame pair is one that has a high similarity to the second video frame. The candidate video segment is obtained by fusing the first video frames from multiple video frame pairs; therefore, the candidate video segment is a segment in the first video that overlaps with at least one second video frame. The occurrence time difference reflects the discrepancy in the timing of the first and second video frames within the first and second videos, respectively.

203. Based on the at least one candidate video segment and the target time range, the server determines at least one target video segment in the first video, wherein the target video segment is within the target time range of the first video.

The target time range refers to the time range in the video. The target time range is set by technicians according to the actual situation, and this application embodiment does not limit it.

The technical solution provided in this application determines video frame pairs containing similar video frames based on the similarity between video frame features. The first video frame in the video frame pair is fused based on the time difference of its occurrence to obtain at least one candidate video segment. Finally, a target video segment within a target time range is determined from the at least one candidate video segment. The process of determining the target segment requires no manual intervention; it can be automatically performed by computer equipment directly based on the first video and at least one second video, resulting in high efficiency.

Steps 201-203 above are a brief introduction to the video segment recognition method provided in the embodiments of this application. The following will provide a more detailed description of the video segment recognition method provided in the embodiments of this application, using some examples. Referring to Figure 3, the technical solution provided in the embodiments of this application can be executed by a terminal or a server, or jointly by a terminal and a server. In this embodiment, the execution subject is taken as a server as an example, and the method includes:

301. The server performs feature extraction on the first video and at least one second video to obtain video frame features of the first video and video frame features of at least one second video.

In one possible implementation, the server inputs a first video and at least one second video into a feature extraction model, and performs feature extraction on the first video and the at least one second video through the feature extraction model to obtain the video frame features of the first video and the video frame features of the at least one second video.

The process of the server extracting features from the first video and at least one second video using a feature extraction model is the process of extracting features from the first video frame of the first video and the second video frame of the second video. In this case, the feature extraction model is an image feature extraction model.

In this implementation, feature extraction is performed on the first video and the at least one second video using a feature extraction model to obtain the video frame features of the first video and the video frame features of the at least one second video, thereby enabling an abstract representation of the first video and the at least one second video and improving the efficiency of subsequent operations.

To illustrate the above implementation methods, three examples are provided below.

Example 1: The server inputs the first video and the at least one second video into a feature extraction model. The feature extraction model performs convolution and pooling on multiple first video frames and multiple second video frames to obtain the video frame features of the multiple first video frames and the video frame features of the multiple second video frames. The multiple first video frames are video frames of the first video, and the multiple second video frames are video frames of at least one second video.

The following describes a method for feature extraction from a first video by the server. The server inputs multiple first video frames into a feature extraction model. The convolutional layer of this model convolves the multiple first video frames to obtain feature maps. The server then uses the pooling layer of the feature extraction model to perform either max pooling or average pooling on the feature maps of the multiple first video frames to obtain the video frame features. In some embodiments, the server represents the first video frame as a matrix and the video frame features as vectors. During the convolution process, the convolution kernel slides across the first video frame.

In some embodiments, the feature extraction model is a feature extractor based on Convolutional Neural Networks (CNNs), such as a ResNet-101 neural network (Residual Network 101) pre-trained on the large-scale open-source dataset ImageNet. The structure of the ResNet-101 neural network is shown in Table 1. The output of the pooling layer of the ResNet-101 neural network is the video frame feature, where 101 refers to the number of layers in the model, and the video frame feature is a 1×2048 vector.

Table 1

In this context, Layer name refers to the name of each layer in the ResNet-101 feature extraction model, Output size refers to the size of the output feature map, max pool refers to max pooling, stride refers to the stride, blocks refers to layers (a layer may contain multiple convolutional kernels), Conv refers to convolutional layers, Pool refers to pooling layers, Class refers to classification layers, and full connection refers to fully connected layers. In the process of extracting video frame features, the Class layer is not used.

It should be noted that the above description uses ResNet-101 as an example for feature extraction model. In other possible implementations, the feature extraction model can also be other structures, and this application embodiment does not limit this.

Furthermore, the above feature extraction process is based on convolution, and the resulting video frame features are used to express the image texture features of the video frame. Such video frame features are also called the low-level features of the video frame. In other possible implementations, this feature extraction model can also extract the semantic features of the video frame. The resulting video frame features can reflect the semantics of the video frame. The following describes the method by which the server extracts the semantic features of the video frame using this feature extraction model.

Example 2: The server inputs the first video and the at least one second video into a feature extraction model. The feature extraction model encodes multiple first video frames and multiple second video frames using an attention mechanism, obtaining video frame features for the multiple first video frames and multiple second video frames. Here, the multiple first video frames are video frames of the first video, and the multiple second video frames are video frames of at least one second video. The video frame features obtained through this feature extraction model are also the semantic features of the corresponding video frames. In this implementation, the feature extraction model is a semantic feature encoder, such as a Transformer encoder.

The following describes the method of feature extraction from multiple first videos by the server. The server inputs multiple first video frames from the first video into a feature extraction model. This model performs embedding encoding on the multiple first video frames, obtaining multiple embedding vectors. Each embedding vector corresponds to one first video frame, representing the position and content of the first video frame within the first video. The server inputs these embedding vectors into the feature extraction model and performs linear transformations using three linear transformation matrices, obtaining a query vector, a key vector, and a value vector corresponding to each first video frame. Based on the query and key vectors corresponding to the multiple first video frames, the server obtains the attention weights for each first video frame using the feature extraction model. Finally, based on the attention weights and value vectors of each first video frame, the server obtains the attention encoding vector for each first video frame. This attention encoding vector is essentially the video frame feature of the first video frame.

For example, the server uses a feature extraction model to multiply each embedding vector by three linear transformation matrices to obtain the query vector, key vector, and value vector corresponding to each first video frame. For the first video frame among multiple first video frames, the server uses the feature extraction model to determine multiple attention weights between the first video frame and the key vectors of the multiple first video frames, based on the query vector of the first video frame. For the first video frame among multiple first video frames, the server uses the feature extraction model to perform a weighted sum of the attention weights between the multiple first video frames and the value vectors of the multiple first video frames to obtain the attention encoding vector of the first video frame, which is the video frame feature of the first video frame.

Examples 1 and 2 above illustrate the extraction of low-level features and semantic features of video frames by the feature extraction model. In other possible implementations, the server can also obtain low-level features and semantic features of video frames simultaneously through the feature extraction model, as illustrated in Example 3 below.

Example 3: The server inputs the first video and at least one second video into a feature extraction model. The model performs convolution and pooling on multiple first and second video frames to obtain the low-level features of the first and second video frames. The first video frames are video frames of the first video, and the second video frames are video frames of at least one second video. The server then encodes the multiple first and second video frames using an attention mechanism based on the feature extraction model to obtain the semantic features of the first and second video frames. The server fuses the low-level and semantic features of each first video frame to obtain the video frame features of each first video frame. Similarly, the server fuses the low-level and semantic features of each second video frame to obtain the video frame features of each second video frame.

For example, the feature extraction model includes a first sub-model and a second sub-model. The first sub-model is used to extract the low-level features of video frames, and the second sub-model is used to extract the semantic features of video frames. After the server inputs the first video and the at least one second video into the feature extraction model, it obtains the low-level features of the multiple first video frames and the multiple low-level features of the multiple second video frames through the first sub-model, and obtains the semantic features of the multiple first video frames and the multiple second video frames through the second sub-model. When the server fuses the low-level features and semantic features of each video frame, it can use a weighted summation method. The weight of the weighted summation can be set by the technician according to the actual situation, such as 0.5. This application embodiment does not limit this. The method by which the server obtains the low-level features and semantic features of video frames through the first sub-model and the second sub-model is the same as in Examples 1 and 2 above, and the implementation process will not be described in detail here.

It should be noted that the above description uses the feature extraction model to extract the low-level features and semantic features of video frames as an example. With the development of science and technology, servers can also use feature extraction models with other structures to obtain video frame features. This application does not limit this.

In some embodiments, the first video and at least one second video belong to the same video set. The first video is the video of the target video segment to be determined, and the at least one second video is all videos in the video set except for the first video. Alternatively, the at least one second video is a video extracted from the video set, with the first video masked during extraction. When the at least one second video is a video extracted from the video set, the server randomly extracts a target number of second videos from the video set. During the extraction process, the first video is masked; that is, the extracted target number of second videos does not include the first video. The number of target videos is set by a technician according to actual conditions, and this application embodiment does not limit this. The server groups the first video and the at least one second video into at least one video pair, and each video pair includes one of the first video and the at least one second video.

For example, if the video set includes 46 videos, for each first video i, the server randomly selects 10 second videos r from the remaining videos in the video set, and forms 10 video pairs with the first video i and the 10 second videos r respectively. In subsequent processing, the processing is carried out in units of video pairs, where 10 is the target number of videos.

In some embodiments, before performing feature extraction on the first video and the at least one second video, the server performs frame extraction on the first video and the at least one second video to obtain multiple first video frames of the first video and multiple second video frames of each second video. By extracting frames from the video, the computational load of the subsequent feature extraction process can be reduced, thereby improving the efficiency of feature extraction.

Taking the first video as an example, the server extracts frames from the first video at target intervals to obtain multiple first video frames. The target interval refers to the target playback duration of the first video, such as 1 second, or it refers to a target number of frame intervals, such as 25 frames. When the target interval refers to the target playback duration of the first video, the server extracts one frame from the first video every target playback duration. If the first video is 6 seconds long and the target playback duration is 1 second, the server extracts 6 first video frames from the first video. When the target time interval refers to the target number of frame intervals, the server extracts frames from the first video every target number of video frames to obtain multiple first video frames. If the first video includes 100 video frames and the target number is 10, the server extracts 10 first video frames from the first video. For example, referring to Figure 4, the server extracts frames from the first video 400 at target intervals to obtain multiple first video frames 401. The server inputs multiple first video frames 401 of the first video into the feature extraction model 402, and outputs video frame features 403 of the multiple first video frames 401 through the feature extraction model 402.

It should be noted that step 301 above is an optional step. It can be executed by the server in advance or by the server when executing the technical solution provided in the embodiments of this application. The embodiments of this application do not limit this step.

302. The server determines multiple video frame pairs based on the video frame features of the first video and the video frame features of at least one second video. The video frame pairs include a first video frame and a second video frame whose similarity meets the similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video.

In one possible implementation, the server determines the similarity between video frame features of a plurality of first video frames and video frame features of a plurality of second video frames. The server identifies first and second video frames whose similarity meets a target condition as a video frame pair, and each video frame pair includes one first video frame and one second video frame.

The similarity between video frame features is determined by Euclidean distance or cosine similarity, but this application does not limit the specific method used.

In this implementation, the server can determine multiple video frame pairs based on the similarity between the first video frame and the second video frame. Since the video frames in the video frame pair are video frames with high similarity from different videos, similar video segments can be quickly determined based on the video frame pair, thereby finally determining the target video segment, which is highly efficient.

When the similarity is determined by Euclidean distance, the server determines the Euclidean distance between the video frame features of multiple first video frames and the video frame features of multiple second video frames. The server identifies first and second video frames whose Euclidean distance is less than or equal to a distance threshold as a video frame pair. The distance threshold is set by a technician based on actual conditions, and this embodiment does not limit its setting. When the distance threshold is 0.5, if the Euclidean distance between the video frame features of any first video frame and the video frame features of any second video frame is less than or equal to 0.5, the server identifies the first and second video frames as a video frame pair.

When the similarity is cosine similarity, the server determines the cosine similarity between the video frame features of multiple first video frames and the video frame features of multiple second video frames. The server identifies first and second video frames with a cosine similarity greater than or equal to a similarity threshold as a video frame pair. When the similarity threshold is 0.8, if the cosine similarity between the video frame features of any first video frame and the video frame features of any second video frame is greater than or equal to 0.8, the server identifies that first and second video frame as a video frame pair.

In some embodiments, when the server forms at least one video pair with a first video and at least one second video, the server determines the similarity between the video frame features of the first video and the video frame features of the second video in the video pair on a per-video-pair basis, thereby determining multiple video frame pairs under the video pair. For example, for video pair (i, r), the server determines the similarity between the video frame features of the first video i and the video frame features of the second video r. The server determines the first video frame and the second video frame whose similarity meets the target condition as a video frame pair. That is, for each video frame j in the first video i, the Euclidean distance between the first video frame j and the video frame features of each second video frame in the second video r is determined. The server takes the second video frame with an Euclidean distance less than _t0 as a similar frame of the first video frame j, and the first video frame j and the similar frame form a video frame pair. The server stores the obtained similar frames of the first video frame j in a first list, which is also called a similar frame list (sim-id-list). In some embodiments, the server stores the frame identifier in the first list, and the frame identifier is used to indicate the video to which the frame belongs and the position of the frame in the video. For example, for the first video frame j=1, the similar frame list sim-id-list is [1, 2, 3], which means that the video frames corresponding to the 1st, 2nd, and 3rd seconds of the second video r are similar frames, where j=1 means the video frame corresponding to the 1st second in the first video.

Optionally, after step 302, if the number of determined video frame pairs is 0, the server determines that the target video segment does not exist in the first video.

303. The server determines the time difference between the occurrence of multiple video frame pairs.

In one possible implementation, the server subtracts the occurrence time of the first video frame in the first video from the occurrence time of the second video frame in the second video to obtain the occurrence time difference of the multiple video frame pairs. In some embodiments, the server stores the occurrence time difference of the multiple video frame pairs in a second list, also known as the occurrence time difference list (diff-time-list), from which the corresponding occurrence time difference can be directly retrieved during subsequent processing. For example, for the first video frame j=1, if the similar frame list sim-id-list is [1, 2, 3], then the corresponding occurrence time difference list diff-time-list is [0, 1, 2].

304. Based on the occurrence time difference of the multiple video frame pairs, the server divides the multiple video frame pairs into multiple video frame groups. The video frame pairs in the same video frame group correspond to the same occurrence time difference, which refers to the difference between the occurrence times of the two video frames in the video frame pair in the video.

In one possible implementation, the server groups video frame pairs with the same occurrence time difference into an initial video frame group. Based on the occurrence time difference corresponding to multiple initial video frame groups, the server merges these multiple initial video frame groups to obtain the multiple video frame groups.

The initial video frame group includes multiple pairs of video frames with the same occurrence time difference. Different initial video frame groups correspond to different occurrence time differences. The occurrence time difference of the initial video frame group refers to the occurrence time difference of the video frame pairs in the initial video frame group.

In this implementation, video frames in a pair of video frames with the same time difference may constitute a complete video segment. By aggregating video frame pairs into video frame groups, it is easier to identify similar video segments later.

For example, the server sorts the initial video frame groups according to the target order to obtain multiple candidate video frame groups. If the matching time difference between any two adjacent candidate video frame groups meets the matching time difference condition, the server merges the two adjacent candidate video frame groups into one video frame group. The matching time difference refers to the difference between the occurrence time differences of the corresponding two adjacent candidate video frame groups.

To provide a clearer explanation of the technical processes described in the examples above, the examples will be further explained in two parts below.

Part 1: The server sorts the multiple initial video frame groups according to the target order to obtain multiple candidate video frame groups.

In one possible implementation, the server sorts the multiple initial video frame groups in ascending order of their corresponding time differences to obtain multiple candidate video frame groups. In this case, the target order refers to the order of their time differences from largest to smallest. In some embodiments, within any initial video frame group, the server sorts the frames according to the order in which the first video frame appears in the first video frame pair.

In this implementation, the server sorts the multiple initial video frame groups in descending order. Among the multiple candidate video frame groups obtained, the time difference between any two candidate video frame groups is relatively close, which facilitates the subsequent fusion process.

For example, consider multiple initial video frame groups: [3, 5], [11, 12], [2, 4], [4, 6], [6, 9], [7, 10], [10, 11]. Each set of parentheses represents a video frame pair [i, r]. The first number in the parentheses is the identifier of the first video frame i, and the second number is the identifier of the second video frame r, which represents the appearance time of the video frame in the video. For the video frame pair [3, 5], the appearance time difference is 5 - 3 = 2, and for the video frame pair [6, 9], the appearance time difference is 9 - 6 = 3. The server sorts these initial video frame groups in ascending order of their respective appearance time differences, resulting in multiple candidate video frame groups: [10, 11], [11, 12], [2, 4], [3, 5], [4, 6], [6, 9], [7, 10].

In one possible implementation, the server sorts the multiple initial video frame groups in ascending order of their corresponding time differences to obtain multiple candidate video frame groups. In this case, the target order refers to the order of their time differences from smallest to largest. In some embodiments, within any initial video frame group, the server sorts the frames according to the order in which the first video frame appears in the first video frame pair.

In this implementation, the server sorts the multiple initial video frame groups in ascending order. Among the multiple candidate video frame groups obtained, the time difference between any two candidate video frame groups is relatively close, which facilitates the subsequent fusion process.

In some embodiments, when a first list is used to store video frame pairs and a second list is used to store occurrence differences, the server generates a third list based on the first and second lists. This third list is used to store video frame pairs and occurrence differences. The third list can store multiple initial video frame groups. For example, the third list can be in the form of a third list (match-dt-list): {d: {count, start-id, match-id-list}, ...}, where d is the occurrence time difference, d: {count, start-id, match-id-list} represents the initial video frame group with occurrence time difference d, count is the number of video frame pairs in the initial video frame group, start-id is the minimum identifier of the first video frame, and match-id-list is the video frame pair.

Part Two: If the matching time difference between any two adjacent candidate video frame groups in the multiple candidate video frame groups meets the matching time difference condition, the server merges the two adjacent candidate video frame groups into one video frame group.

In one possible implementation, the two adjacent candidate video frame groups include a first candidate video frame group and a second candidate video frame group. When the matching time difference between the occurrence time difference corresponding to the first candidate video frame group and the occurrence time difference corresponding to the second candidate video frame group is less than or equal to the matching difference threshold, the server adds the video frame pair in the first candidate video frame group to the second candidate video frame group to obtain the video frame group.

The process of fusing multiple candidate video frame groups into multiple video frame groups involves multiple iterative processes. After fusing the first and second candidate video frame groups into one video frame group, the server can further determine the matching time difference between the newly fused video frame group and the next candidate video frame group. If the matching time difference meets the matching time difference condition, the newly fused video frame group is fused again with the next candidate video frame group. This fusion process is based on the same inventive concept as the fusion of the first and second candidate video frame groups, and its implementation will not be elaborated further. Of course, if the matching time difference does not meet the matching time difference condition, the server further determines the matching time difference between the next candidate video frame group and the next two candidate video frame groups, and performs further processing based on the matching time difference. The matching difference threshold can be set by a technician according to the actual situation, and this application embodiment does not limit it.

In this implementation, by fusing candidate video frame groups based on the time difference of occurrence, the number of candidate video frame groups can be reduced, thereby reducing the amount of computation in subsequent processing and improving computational efficiency.

For example, the server determines the matching time difference between the first candidate video frame group and the second candidate video frame group. If the matching time difference is less than or equal to a matching difference threshold, the server replaces the target second video frame with a reference second video frame based on the occurrence time difference corresponding to the second candidate video frame group, thus obtaining the video frame group. The target second video frame is the second video frame newly added to the second candidate video frame group, and the reference second video frame is the second video frame in the second video frame whose occurrence time difference with the target first video frame is the occurrence time difference corresponding to the second candidate video frame group. The target first video frame is the first video frame in the video frame pair to which the target second video frame belongs.

In this implementation, after adding video frame pairs from the first candidate video frame group to the second candidate video frame group, the server can also adjust the newly added video frame pairs to the second candidate video frame group according to the occurrence time difference of the second candidate video frame group, so that the occurrence time difference of the adjusted video frame pairs is the same as that of the second candidate video frame group, thus maintaining the consistency between the occurrence difference of the video frame pairs and the occurrence difference of the video frame group.

To illustrate this more clearly, let's take the following example: the first candidate video frame group has an occurrence time difference of 3, including two video frame pairs [6, 9] and [7, 10]; the second candidate video frame group has an occurrence time difference of 2, including three video frame pairs [2, 4], [3, 5], and [4, 6]; and the matching difference threshold is 3. Since the matching time difference between the first and second candidate video frame groups is 1, the server determines that this matching time difference is less than the matching difference threshold and therefore needs to merge the first and second candidate video frame groups. The server adds two video frame pairs [6, 9] and [7, 10] from the first candidate video frame group to the second candidate video frame group, making the second candidate video frame group [2, 4], [3, 5], [4, 6], [6, 9], [7, 10]. Since the occurrence time difference of the corresponding second candidate video frame group is 2, the server adjusts the second video frame in the two video frame pairs [6, 9] and [7, 10] added to the second candidate video frame group based on this occurrence time difference of 2, resulting in two new video frame pairs [6, 8] and [7, 9]. After adjusting the second video frame newly added to the second candidate video frame group, the second candidate video frame group becomes [2, 4], [3, 5], [4, 6], [6, 8], [7, 9], with the occurrence time difference of each video frame pair being 2.

It should be noted that the above description is based on the example of the server adding video frame pairs from the first candidate video frame group to the second candidate video frame group. In other possible implementations, the server can also add video frame pairs from the second candidate video frame group to the first candidate video frame group.

In some embodiments, the server determines whether to add video frame pairs from the first candidate video frame group to the second candidate video frame group or vice versa, based on the number of video frame pairs in the first candidate video frame group and the second candidate video frame group. For example, if the number of video frame pairs in the first candidate video frame group is greater than the number of video frame pairs in the second candidate video frame group, the server adds the video frame pairs from the second candidate video frame group to the first candidate video frame group. If the number of video frame pairs in the second candidate video frame group is greater than the number of video frame pairs in the first candidate video frame group, the server adds the video frame pairs from the first candidate video frame group to the second candidate video frame group. If the number of video frame pairs in the second candidate video frame group is equal to the number of video frame pairs in the first candidate video frame group, the server adds the video frame pairs from the first candidate video frame group to the second candidate video frame group. Alternatively, if the number of video frame pairs in the second candidate video frame group is equal to the number of video frame pairs in the first candidate video frame group, the server adds the video frame pairs from the second candidate video frame group to the first candidate video frame group.

In this scenario, the server can determine how to merge candidate video frame groups based on the number of video frame pairs in each group, adding candidate video frame groups with fewer video frames to video frame groups with more video frames, thereby reducing computational load and improving efficiency.

305. For any video frame group among the multiple video frame groups, the server merges the first video frames of the video frame pair in the video frame group into a candidate video segment according to the appearance time of the first video frame in the first video.

In one possible implementation, the server compares the occurrence times of the first video frames of any two adjacent video frame pairs in the video frame group within the first video. If the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video meets the occurrence time condition, the server adds the two adjacent video frame pairs to a temporary frame list. If the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video does not meet the occurrence time condition, the server merges the video frame pairs in the temporary frame list into a reference video segment. The server determines the at least one candidate video segment based on multiple reference video segments.

The temporary frame list is used to store video frame pairs whose occurrence time differences meet the occurrence time condition. In some embodiments, the occurrence time difference meeting the occurrence time condition means that the occurrence time difference is less than or equal to an occurrence time difference threshold. The occurrence time difference threshold is set by technicians according to actual conditions, such as 8 seconds. This embodiment of the application does not limit this.

To provide a clearer explanation of the above embodiments, the following description will be divided into four parts.

Part 1: The server compares the occurrence times of the first video frame in any two adjacent video frame pairs in the video frame group within the first video.

In some embodiments, the server uses the occurrence time of a first video frame in the first video as the identifier of the first video frame, and the occurrence time of a second video frame in the second video as the identifier of the second video frame. In this case, when the server compares the occurrence times of the first video frames of any two adjacent video frame pairs in the first video, it only needs to compare the identifiers of these two first video frames. For example, the video frame group includes video frame pairs [2, 4], [3, 5], [4, 6], [6, 8], [7, 9], and the server compares the occurrence times of the first video frames of the video frame pairs in the first video in the first video. In the first comparison process, the server compares the occurrence times of the first video frame 2 of the first video frame pair [2, 4] with the first video frame 3 of the second video frame pair [3, 5] in the first video in the first video.

Part Two: If the difference between the occurrence times of the first video frame in the first video of the two adjacent video frame pairs meets the occurrence time condition, the server adds the two adjacent video frame pairs to the temporary frame list.

In one possible implementation, if the difference between the occurrence times of the first video frames in the first video of two adjacent video frame pairs is less than or equal to an occurrence time difference threshold, the server adds the two adjacent video frame pairs to a temporary frame list. For example, taking the video frame group [2,4], [3,5], [4,6], [6,8], [7,9] as an example, for video frame pairs [2,4] and [3,5], when the occurrence time difference threshold is 3, since the occurrence time difference of the first video frames in [2,4] and [3,5] in the first video is 3-2=1, the server adds the two video frame pairs to a temporary frame list (Tmplist), Tmplist=[[2,4], [3,5]].

The server determines that adding video frame pairs to the temporary frame list involves multiple iterations. In any iteration, the server compares the time difference between the first video frame of the current video frame pair and the first video frame of the previous video frame pair in the first video. Here, the current video frame pair refers to the video frame pair currently being processed, and the previous video frame pair refers to the video frame pair processed in the previous iteration. For example, after adding video frame pairs [2, 4] and [3, 5] to the temporary frame list, the server further determines the relationship between the time difference between the first video frames of video frame pairs [3, 5] and [4, 6] in the first video and the time difference threshold. Since the time difference between the first video frames of [3, 5] and [4, 6] in the first video is 4-3=1, the server adds video frame pair [4, 6] to the temporary frame list (Tmplist), where Tmplist = [[2, 4], [3, 5], [4, 6]]. Through multiple iterations, a temporary frame list Tmplist = [[2, 4], [3, 5], [4, 6], [6, 8], [7, 9] is obtained.

Part Three: If the difference between the occurrence times of the first video frame in the first video of two adjacent video frame pairs does not meet the occurrence time condition, the server merges the video frame pairs in the temporary frame list into a reference video segment.

The reference video segment includes a first sub-segment and a second sub-segment. The first sub-segment is composed of the first video frame in the video frame pair, and the second sub-segment is composed of the second video frame in the video frame pair.

In one possible implementation, if the difference in the occurrence time between the first video frames of two adjacent video frame pairs in the first video is greater than a threshold for occurrence time difference, the server merges the first video frames in the temporary frame list into a first sub-segment and the second video frames in the temporary frame list into a second sub-segment. The first and second sub-segments together constitute the reference video segment. Since the first and second video frames in the video frame pair are highly similar, the first and second sub-segments are also highly similar segments. For example, referring to Figure 5, which illustrates the form of first sub-segment 501 and second sub-segment 502, the first video frame at the beginning of first sub-segment 501 and the first video frame at the beginning of second sub-segment 502 constitute one video frame pair, and the first video frame at the end of first sub-segment 501 and the first video frame at the end of second sub-segment 502 constitute another video frame pair. In some embodiments, the first and second sub-segments in a reference video segment are also referred to as matching segments.

For example, if the two adjacent video frame pairs are [9, 11] and [2, 4], and the difference between the appearance times of the first video frame in the first video is 9-2=7, then the server will merge the first video frame in the temporary frame list into a reference video segment. For example, if the temporary frame list Tmplist = [[2,4],[3,5],[4,6],[6,8],[7,9]], then the server merges the first video frames [2,], [3,], [4,], [6,], [7,] in the temporary frame list into the first sub-segment (2,7), and merges the second video frames [,4], [,5], [,6], [,8], [,9] in the temporary frame list into the second sub-segment (4,9). The first sub-segment (2,7) and the second sub-segment (4,9) constitute the reference video segment (2,7,4,9), where the format of the reference video segment is (src-startTime, src-endTime, ref-startTime, ...). The `ref-endTime` parameter is used to define the timeline of the first sub-segment. Here, `src-startTime` refers to the beginning of the first sub-segment (the first video frame with the smallest sequence number in the temporary frame list), and `src-endTime` refers to the end of the first sub-segment (the first video frame with the largest sequence number in the temporary frame list). Similarly, `ref-startTime` refers to the beginning of the second sub-segment (the second video frame with the smallest sequence number in the temporary frame list), and `ref-endTime` refers to the end of the second sub-segment (the second video frame with the largest sequence number in the temporary frame list). The sequence number identifies the video frame's position within the video; a smaller sequence number indicates an earlier position, and vice versa. In some embodiments, the server stores reference video segments in a `match-duration-list`. Because determining the video frame pair involves traversing all video frames of both the first and second videos, a single video frame may resemble multiple video frames, resulting in overlapping timestamps between two reference video segments in the `match-duration-list`.

In some embodiments, the reference video segment may also carry information such as the occurrence time difference of the first sub-segment, the duration of the first sub-segment, and the number of video frames included in the first sub-segment, so that the server can call it.

In addition to the methods provided in Part III above, this application provides another method for triggering the fusion of video frame pairs in a temporary frame list into a reference video segment.

In one possible implementation, when the currently processed video frame pair is the last video frame pair in the video frame group, the server adds the video frame pair to a temporary frame list and merges the video frame pairs in the temporary frame list into a reference video frame segment. For example, if the video frame group includes five video frame pairs [2, 4], [3, 5], [4, 6], [6, 8], and [7, 9], when the server processes video frame pair [7, 9], since [7, 9] is the last video frame pair in the video frame group, the server adds video frame pair [7, 9] to the temporary frame list and merges the video frame pairs in the temporary frame list into a reference video segment. The merging process is described in the previous implementation and will not be repeated here.

Part Four: The server determines at least one candidate video segment based on multiple reference video segments.

The plurality of reference video segments includes a first overlapping video segment and/or a second overlapping video segment. The first overlapping video segment refers to a reference video segment that belongs to the first reference video segment among the plurality of reference video segments, and the second overlapping video segment refers to a reference video segment that partially overlaps with the second reference video segment among the plurality of reference video segments.

Wherein, the first overlapping video segment belongs to the first reference video segment, meaning that the content of the first overlapping video segment is completely contained by the first reference video segment, or the first reference video segment completely contains the first overlapping video segment.

To provide a clearer explanation of the content in Part Four above, the method by which the server determines the first overlapping video segment from multiple reference video segments will be described below.

In one possible implementation, the server determines the first overlapping video segment from the plurality of reference video segments based on the occurrence time of the first sub-segment in the first video.

The first sub-segment is a video segment composed of the first video frame, and its occurrence time includes the start and end times of the first sub-segment in the first video.

For example, for reference video segment _A1 and reference video segment _B1 among the multiple reference video segments, the server compares the occurrence time of the first sub-segment of reference video segment _A1 in the first video with the occurrence time of the first sub-segment of reference video segment _B1 in the first video. If the occurrence time of the first sub-segment of reference video segment _B1 in the first video is a subset of the occurrence time of the first sub-segment of reference video segment _A1 in the first video, then reference video segment _B1 is determined to be the first overlapping video segment. For example, referring to Figure 6, the multiple reference video segments include reference video segment _A1 and reference video segment _B1 . The server compares the occurrence time of the first sub-segment _m1 of reference video segment _A1 in the first video with the occurrence time of the first sub-segment _n1 of reference video segment _B1 in the first video. If the start time of the first sub-segment _n1 is after the start time of the first sub-segment _m1 and the end time of the first sub-segment _n1 is before the end time of the first sub-segment _m1 , the server determines the reference video segment _B1 as the first overlapping video segment, and the reference video segment _A1 is the aforementioned first reference video segment.

After explaining the method by which the server determines the first overlapping video segment from multiple reference video segments, the method by which the server determines the second overlapping video segment from multiple reference video segments will now be explained.

In one possible implementation, the server determines the second overlapping video segment from the plurality of reference video segments based on the occurrence time of the first sub-segment in the first video.

For example, for reference video segment _A2 and reference video segment _B2 among the multiple reference video segments, the server compares the occurrence time of the first sub-segment of reference video segment _A2 in the first video with the occurrence time of the first sub-segment of reference video segment _B2 in the first video. If there is an intersection between the occurrence time of the first sub-segment of reference video segment B in the first video and the occurrence time of the first sub-segment of reference video segment A in the first video, the reference video segment with shorter duration between reference video segments A and B is identified as the second overlapping video segment. For example, referring to Figure 6, the multiple reference video segments include reference video segment _A2 and reference video segment _B2 . The server compares the occurrence time of the first sub-segment _m2 of reference video segment _A2 in the first video with the occurrence time of the first sub-segment _n2 of reference video segment _B2 in the first video. If the start time of the first sub-segment _n2 is after the start time of the first sub-segment _m2 and before its end time, and the end time of the first sub-segment _n2 is after the first sub-segment _m2 , or if the start time of the first sub-segment _n2 is before the first sub-segment _m2 and the end time of the first sub-segment _n2 is before the end time of the first sub-segment _m2 and after its start time, and the duration of the reference video segment _B2 is less than that of the reference video segment _A2 , then the server determines the reference video segment _B2 as the second overlapping video segment, and the reference video segment _A2 is the aforementioned second reference video segment.

After introducing the method by which the server determines the first and second overlapping video segments, the steps provided in Part IV above will be explained below.

In one possible implementation, if the plurality of reference video segments include the first overlapping video segment, the server deletes the first overlapping video segment to obtain the at least one candidate video segment.

In this implementation, the server can remove duplicate first overlapping video segments from multiple reference video segments to reduce the number of candidate video segments obtained, reduce the amount of computation, and improve computational efficiency.

In one possible implementation, if the plurality of reference video segments include the second overlapping video segment, the server deletes the overlapping portion between the second overlapping video segment and the second reference segment to obtain the at least one candidate video segment.

Based on the above implementation method, the server may optionally also perform the following steps:

In some embodiments, after deleting the overlapping portion between the second overlapping video segment and the second reference segment, the server compares the duration of a third type of reference video segment with a target duration. The third type of reference video segment refers to the second overlapping video segment after deleting the overlapping portion. If the duration of the third type of reference video segment is greater than or equal to the target duration, the server retains the third type of reference video segment. If the duration of the third type of reference video segment is less than the target duration, the server deletes the third type of reference video segment.

The target duration is set by technicians according to the actual situation, and this application embodiment does not limit it. When the server retains the third type of reference video segment, that is, the third type of reference video segment replaces the original second overlapping video segment.

The above implementation method is illustrated below with two examples:

Example 1: For reference video segment _A2 and reference video segment _B2 among multiple reference video segments, if the first sub-segment _m2 of reference video segment _A2 and the first sub-segment _n2 of reference video segment _B2 partially overlap, and the start time of the first sub-segment _m2 is earlier than that of the first sub-segment _n2 , the server moves the start time of the first sub-segment _n2 to the end time of the first sub-segment _m2 , resulting in sub-segment _l1 . Sub-segment _l1 is the first sub-segment of the third type of reference video segment. If the duration of sub-segment _l1 is less than or equal to the target duration, the server deletes sub-segment _l1 and the third type of reference video segment to which sub-segment _l1 belongs. If the duration of sub-segment _l1 is greater than the target duration, the server retains sub-segment _l1 and the third type of reference video segment to which sub-segment _l1 belongs.

Example 2: For reference video segment _A2 and reference video segment _B2 among the multiple reference video segments, if the first sub-segment _m2 of reference video segment _A2 and the first sub-segment _n2 of reference video segment _B2 partially overlap, and the start time of the first sub-segment _n2 is earlier than that of the first sub-segment _m2 , the server moves the end time of the first sub-segment _n2 to the start time of the first sub-segment _m2 , resulting in sub-segment _l2 . This sub-segment _l2 is the first sub-segment of the third type of reference video segment. If the duration of sub-segment _l2 is less than or equal to the target duration, the server deletes sub-segment _l2 and the third type of reference video segment to which sub-segment _l2 belongs. If the duration of sub-segment _l2 is greater than the target duration, the server retains sub-segment _l2 and the third type of reference video segment to which sub-segment _l2 belongs.

306. Based on the at least one candidate video segment, the server determines the at least one target candidate video segment, wherein the number of times the target candidate video segment appears in the at least one candidate video segment meets the frequency condition.

In one possible implementation, the server determines at least one reference candidate video segment based on the at least one candidate video segment. The server determines the number of times each reference candidate video segment appears in the at least one reference candidate video segment. The server determines the reference candidate video segments whose appearance count meets the appearance count condition as target candidate video segments.

The number of times a reference candidate video segment appears in the at least one reference candidate video segment refers to the number of that reference candidate video segment among the at least one reference candidate video segments. For example, if the at least one reference candidate video segment is 1, 2, 3, 1, 4, 5, then the number of times reference candidate video segment 1 appears is 2.

To illustrate the above embodiments, the following description will be divided into three parts.

Part 1: Based on the at least one candidate video segment, the server determines at least one reference candidate video segment.

The at least one candidate video segment includes a third overlapping video segment and/or a fourth overlapping video segment. The third overlapping video segment refers to a candidate video segment that belongs to the first candidate video segment among the at least one candidate video segments, and the fourth overlapping video segment refers to a candidate video segment that partially overlaps with the second candidate video segment among the at least one candidate video segments.

To provide a clearer explanation of the content in the first part above, the following describes the method by which the server determines a third overlapping video segment from at least one candidate video segment.

In one possible implementation, the server determines a third overlapping video segment from the at least one candidate video segment based on the occurrence time of the first sub-segment in the first video.

The candidate video segments include a first sub-segment and a second sub-segment. The first sub-segment is composed of the first video frame in the video frame pair, and the second sub-segment is composed of the second video frame in the video frame pair.

For example, if the at least one candidate video segment is two candidate video segments, for candidate video segment _C1 and candidate video segment _D1 in the at least one candidate video segment, the server compares the occurrence time of the first sub-segment of candidate video segment _C1 in the first video with the occurrence time of the first sub-segment of candidate video segment _D1 in the first video. If the occurrence time of the first sub-segment of candidate video segment _D1 in the first video is a subset of the occurrence time of the first sub-segment of candidate video segment _C1 in the first video, then candidate video segment _D1 is determined to be the third overlapping video segment.

For example, the at least one candidate video segment may be two candidate video segments, including candidate video segment _C1 and candidate video segment _D1 . The server compares the appearance time of the first sub-segment _o1 of candidate video segment _C1 in the first video with the appearance time of the first sub-segment _p1 of candidate video segment _D1 in the first video. If the start time of the first sub-segment _p1 is after the first sub-segment _o1 and the end time of the first sub-segment _p1 is before the first sub-segment _o1 , the server determines candidate video segment _D1 as the third overlapping video segment, and candidate video segment _C1 is the aforementioned first candidate video segment.

After describing the method by which the server determines the third overlapping video segment from at least one candidate video segment, the method by which the server determines the fourth overlapping video segment from at least one candidate video segment will now be described.

In one possible implementation, the server determines a fourth overlapping video segment from the at least one candidate video segment based on the occurrence time of the first sub-segment in the first video.

For example, the at least one candidate video segment may be two candidate video segments. For candidate video segment _C2 and candidate video segment _D2 , the server compares the occurrence time of the first sub-segment of candidate video segment _C2 in the first video with the occurrence time of the first sub-segment of candidate video segment _D2 in the first video. If there is an intersection between the occurrence time of the first sub-segment of candidate video segment _D2 and the occurrence time of the first sub-segment of candidate video segment _C2 in the first video, the candidate video segment with shorter duration between candidate video segment _C2 and candidate video segment _D2 is determined as the fourth overlapping video segment.

For example, the at least one candidate video segment may be two candidate video segments, including candidate video segment _C2 and candidate video segment _D2 . The server compares the appearance time of the first sub-segment _o2 of candidate video segment _C2 in the first video with the appearance time of the first sub-segment _p2 of candidate video segment _D2 in the first video. If the start time of the first sub-segment _p2 is after the start time of the first sub-segment _o2 and before its end time, and the end time of the first sub-segment _p2 is after the first sub-segment _o2 , or if the start time of the first sub-segment _p2 is before the first sub-segment _o2 and the end time of the first sub-segment _p2 is before _its start time, and the duration of candidate video segment _D2 is less than that of candidate video segment _C2 , the server determines candidate video segment _D2 as the fourth overlapping video segment, and candidate video segment _C2 is the aforementioned second candidate video segment.

After introducing the method by which the server determines the third and fourth overlapping video segments, the steps provided in the first part above will be explained below.

In one possible implementation, if the at least one candidate video segment includes the third overlapping video segment, the server deletes the third overlapping video segment to obtain the at least one reference candidate video segment. In some embodiments, before deleting the third overlapping video segment, the server adds the occurrence count of the third overlapping video segment to the first candidate video segment. Since the third overlapping video segment is completely contained by the first candidate video segment, adding the occurrence count of the third overlapping video segment to the first candidate video segment can increase the weight of the first candidate video segment in subsequent processing.

In this implementation, the server can remove duplicate third overlapping video segments from at least one candidate video segment, thereby reducing the number of reference candidate video segments obtained, reducing the amount of computation, and improving computational efficiency.

The following is a specific example to illustrate this.

If the first sub-segment _o1 of candidate video segment _D1 is a subset of the first sub-segment _p1 of candidate video segment _C1 , and the duration of the first sub-segment _o1 is greater than 0.5 * the duration of the first sub-segment _p1 , then the server deletes the first sub-segment _o1 and the candidate video segment _D1 , and adds the occurrence count of the candidate video segment _D1 to the candidate video segment _C1 .

Based on the above implementation, optionally, before the server adds the occurrence count of the third overlapping video segment to the first candidate video segment, it can also determine the duration of the third overlapping video segment and the duration of the first candidate video segment, and determine whether to add the occurrence count of the third overlapping video segment to the first candidate video segment based on the duration of the third overlapping video segment and the duration of the first candidate video segment.

For example, the server determines the duration of the third overlapping video segment and the duration of the first candidate video segment. The server determines a first ratio between the duration of the third overlapping video segment and the duration of the first candidate video segment. If the first ratio is greater than or equal to a ratio threshold, the server adds the occurrence count of the third overlapping video segment to the first candidate video segment; if the first ratio is less than the ratio threshold, the server does not add the occurrence count of the third overlapping video segment to the first candidate video segment. The ratio threshold is set by a technician according to the actual situation, for example, it is set to 0.5. This application embodiment does not limit this.

In one possible implementation, if the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment meets the overlap condition, the server determines the number of occurrences of the fourth overlapping video segment. Based on the number of occurrences of the fourth overlapping video segment, the server determines the at least one reference candidate video segment.

The overlap ratio refers to the ratio between the duration of the overlapping video segments and the duration of the video segment being compared. For example, considering a fourth overlapping video segment and a second candidate video segment (the second candidate segment being compared), determining the overlap ratio between them involves dividing the duration of the overlapping video segment by the duration of the second candidate video segment. A successful overlap condition is defined as an overlap ratio greater than or equal to an overlap ratio threshold.

The following describes, through two implementation methods, the method by which the server determines the at least one reference candidate video segment based on the occurrence frequency of the fourth overlapping video segment in the above implementation.

Implementation Method 1: If the number of occurrences of the fourth overlapping video segment is greater than or equal to a first occurrence threshold, the server merges the fourth overlapping video segment with the second candidate video segment to obtain the at least one reference candidate video segment. In some embodiments, before merging the fourth overlapping video segment with the second candidate video segment, the server adds the occurrence count of the fourth overlapping video segment to the second candidate video segment.

The first occurrence threshold is set by technicians according to the actual situation, such as 3, but this application embodiment does not limit this. An occurrence count greater than or equal to the first occurrence threshold indicates that the fourth overlapping video segment cannot be ignored and requires further processing to improve the accuracy of obtaining the target video segment.

The method by which the server merges the fourth overlapping video segment with the second candidate video segment in the above embodiment will be described below.

In some embodiments, taking the example where the duration of the fourth overlapping video segment is shorter than that of the second candidate video segment, the server will delete the duplicate portion between the fourth overlapping video segment and the second candidate video segment, and add the remaining portion to the second candidate video segment to obtain a candidate video segment. For example, referring to Figure 7, the duration of the fourth overlapping video segment 701 is shorter than that of the second candidate video segment 702, and the duration of the fourth overlapping video segment 704 is shorter than that of the second candidate video segment 705. If the end time of the fourth overlapping video segment 701 is later than that of the second candidate video segment 702, the server will merge the fourth overlapping video segment 701 with the second candidate video segment 702 to obtain a candidate video segment 703. If the start time of the fourth overlapping video segment 704 is earlier than that of the second candidate video segment 705, the server will merge the fourth overlapping video segment 704 with the second candidate video segment 705 to obtain a candidate video segment 706.

Implementation Method 2: If the number of occurrences of the fourth overlapping video segment is less than the first occurrence threshold, the server deletes the fourth overlapping video segment and obtains the at least one reference candidate video segment. The server then adds the occurrence count of the fourth overlapping video segment to the second candidate video segment.

If the number of occurrences is less than the first occurrence threshold, it means that the fourth overlapping video segment can be ignored, and the server can delete the fourth overlapping video segment.

In one possible implementation, if the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment does not meet the overlap condition, the server deletes the fourth overlapping video segment to obtain the at least one reference candidate video segment. In some embodiments, before deleting the fourth overlapping video segment, the server adds the occurrence count of the fourth overlapping video segment to the second candidate video segment.

In one possible implementation, if the at least one candidate video segment includes the fourth overlapping video segment, and the duration of the fourth overlapping video segment is less than that of the second candidate video segment, the server deletes the fourth overlapping video segment to obtain the at least one reference candidate video segment. In some embodiments, before deleting the fourth overlapping video segment, the server adds the occurrence count of the fourth overlapping video segment to the second candidate video segment.

In some embodiments, at least one reference candidate video segment is stored in a match list by the server for retrieval.

Part Two: The server determines the number of times a reference candidate video segment appears in at least one reference candidate video segment.

Through the processing described in the first part, the server determines at least one reference candidate video segment based on at least one candidate video segment. This determination process involves merging and deleting occurrence counts, and the server then re-determines the occurrence count of the at least one reference candidate video segment. In some embodiments, the server can store the occurrence count of the at least one reference candidate video segment in a count list for later retrieval.

For example, when determining the target video segment in the first video, the server uses three second videos for mining. For ease of explanation, the first video is named i, and the three second videos are named vid1, vid2, and vid3, respectively. After the above steps, the server determines two candidate video segments [(2, 7, 4, 9), (10, 11, 11, 12)] based on the first video i and the second video vid1, one candidate video segment [(2, 7, 4, 9)] based on the first video i and the second video vid2, and one candidate video segment [(2, 7, 4, 10)] based on the first video i and the second video vid3. The server counts these four candidate video segments and determines that the candidate video segment (2, 7, 4, 9) appears twice, (2, 7, 4, 10) appears once, and (10, 11, 11, 12) appears once. After fusing these four candidate video segments using the method described in Part 1 above, two reference candidate video segments are obtained [(2, 7, 4, 9), (10, 11, 11, 12)]. The reference candidate video segment (2, 7, 4, 9) appears 3 times, and the reference candidate video segment (10, 11, 11, 12) appears 1 time. When stored as a count list, the count list is [3, 1].

Part Three: The server identifies the reference candidate video segments whose occurrence count meets the occurrence count condition as target candidate video segments.

In one possible implementation, the server identifies reference candidate video segments that appear more than or equal to a second occurrence threshold as target candidate video segments.

The second occurrence threshold is positively correlated with the number of the at least one reference candidate video segment; that is, the more the number of the at least one reference candidate video segment, the larger the second occurrence threshold; and the fewer the number of the at least one reference candidate video segment, the smaller the second occurrence threshold. In some embodiments, the second occurrence threshold is the product of a target ratio and the number of the at least one reference candidate video segment, where the target ratio is a positive number less than 1.

For example, given two candidate video segments [(2, 7, 4, 9), (10, 11, 11, 12)], where candidate video segment (2, 7, 4, 9) appears 3 times and candidate video segment (10, 11, 11, 12) appears 1 time, and the second occurrence threshold is 3, the server will delete candidate video segment (10, 11, 11, 12) and ultimately retain candidate video segment (2, 7, 4, 9) with an occurrence count of 3. When storing the data in a match-list and a count-list, match-list = (2, 7, 4, 9); count-list = [3].

307. If the appearance time of any target candidate video segment in the first video is within the target time range, the server determines the target candidate video segment as the target video segment in the first video.

The target time range is set by technicians based on actual conditions. For example, when the technical solution provided in this application is applied to the scenario of identifying the intro and outro of a video, the target time range is the possible time range of the intro and outro of the video. In this case, the target time range includes a first time range and a second time range. The first time range is the range in which the intro may exist, and the second time range is the range in which the outro may exist. For example, the first 1/5 of the video time can be set as the intro time, which is the first time range, and the last 1/5 of the video time can be set as the outro time, which is the second time range. For a 10-minute video, the intro can be set to appear only in the first 2 minutes, and the outro in the last 2 minutes. The 1/5 is set by technicians based on actual conditions and can be adjusted accordingly for different types of videos. For example, 1/5 can be used for a children's animation of about 15 minutes, and 1/8 can be used for a 45-minute long TV series.

It should be noted that steps 301-307 above are illustrated using the example of the server determining the target video segment in the first video. When the first video and the at least one second video belong to the same video set, the server can use the same method as steps 301-307 above to determine the target video segments of other videos in the video set. Other videos refer to videos other than the first video.

The technical solutions provided in the embodiments of this application will be described below with reference to Figure 8.

Referring to Figure 8, in this embodiment, the server matches video frame features based on similarity to obtain multiple video frame pairs. The server divides these multiple video frame pairs into multiple initial video frame groups based on their occurrence time differences. The server merges these multiple initial video frame groups into multiple candidate video frame groups based on their occurrence time differences. The server merges these multiple candidate video frame groups into multiple video frame groups. Based on these multiple video frame groups, the server outputs the target video segment of the first video.

In some embodiments, steps 301-307 described above can be implemented by a segment mining system. When the technical solution provided in this application is applied to the scenario of identifying video intros and outros, the segment mining system is also an intro/outro mining system. Referring to Figure 9, the video segment mining system provides the following functions: Extracting video frame features from multiple videos. For each video, forming a video pair with other videos in the multiple videos. Matching multiple video pairs to obtain multiple video frame pairs. Fusing multiple video frame pairs to obtain multiple video frame groups. Determining the position of the target video segment in the video based on the multiple video frame groups. Obtaining the target video segment based on its position in the video. When the technical solution provided in this application is applied to the scenario of identifying video intros and outros, the target video segment is also the intro or outro of the video.

Referring to Figure 10, when the technical solution provided in this application is applied to the scenario of identifying the opening and closing credits of a TV series, the TV series is obtained, which includes multiple videos. These multiple videos are input into a segment mining system, which then outputs the opening and closing credits of the multiple videos. In some embodiments, the segment mining system can output the timestamps of the opening and closing credits of the multiple videos.

308. The server stores the target video segment from the first video in the segment database.

In one possible implementation, the server extracts features from a target video segment of the first video to obtain the video frame features of the target video segment. The server stores these video frame features in a segment database. In some embodiments, the server associates the video frame features of the target video segment with the first video; for example, the server sets the identifier of the video frame features of the target video segment as the identifier of the first video. If the first video belongs to a video set, the server associates the identifier of the first video with the identifier of the video set to facilitate subsequent querying.

The process of extracting features from the target video segment to obtain the video frame features of the target video segment belongs to the same inventive concept as step 301 above. The implementation process is described in step 301 above and will not be repeated here.

For example, if the target video segment is (2, 7), the server obtains the target video segment corresponding to seconds 2-7 from the first video, and extracts multiple reference video frames from the target video segment. The server performs feature extraction on these multiple reference video frames to obtain the video frame features of these multiple reference video frames. The server stores the video frame features of these multiple reference video frames in a segment database. The server associates the video frame features of these multiple reference video frames with the identifier Vid1 of the first video, and associates the identifier Vid1 of the first video with the identifier Cid1 of the video set to which the first video belongs. Figure 11 shows a storage format of a segment database. Referring to Figure 11, in database 1100, em1-emN are video frame features, vid1-vidK are identifiers of different videos, and N and K are both positive integers.

After the server stores the target video segment from the first video in the segment database, it can also use this segment database to retrieve video segments, as follows:

In one possible implementation, the server extracts features from multiple target video frames of the target video to be identified, obtaining video frame features of the multiple target video frames. Based on the video frame features of the multiple target video frames, the video frame features of the first video frame, and the video frame features of the at least one second video, the server determines at least one target video segment of the target video.

The process by which the server extracts features from multiple target video frames of the target video to obtain the video frame features of these multiple target video frames belongs to the same inventive concept as step 301 above, and the implementation process is described in step 301 above, and will not be repeated here. The process by which the server determines at least one target video segment of the target video based on the video frame features of the multiple target video frames, the video frame features of the first video frame, and the video frame features of the at least one second video belongs to the same inventive concept as steps 302-307 above, and the implementation process is described in steps 302-307 above, and will not be repeated here. In some embodiments, video segment retrieval in the segment database is implemented by a video retrieval system. In some embodiments, the video frame features of the first video frame and the video frame features of the at least one second video are stored in the segment database.

The video clip recognition method described above can be applied to scenarios involving the identification of video clip intros and outros, as well as to the identification of infringing videos. These two application scenarios will be introduced below.

This video clip retrieval method is applied to the scenario of retrieving the beginning and end of video clips. The target video to be identified is input into the video retrieval system, which extracts features from the target video to obtain the video frame features of multiple target video frames. Based on these video frame features, the video retrieval system matches the clips in a clip database to obtain the target video clip, which is either the beginning or the end of the target video.

Taking the identification of the opening and closing credits of newly updated videos in a TV series as an example, suppose the TV series has already released 10 episodes. The opening and closing credits of these 10 episodes are obtained through steps 301-307 above, and stored in the segment database through step 308. When the TV series releases its 11th episode, this 11th episode is used as the target video. The target video is input into the video retrieval system, which extracts features from the target video to obtain the video frame features of multiple target video frames. The video retrieval system then matches these video frame features against the segment database to obtain the target video segment, which is either the opening or closing credits of the target video. By associating the video frame features with the video identifier and the video set identifier in the segment database, matching can be performed within a limited range based on the video set identifier, thereby improving the efficiency of identifying the target video segment. The video set is, in this case, the TV series itself.

The following will provide further explanation with reference to Figure 12.

The TV series to be identified for title sequence and ending sequence recognition is determined. Multiple videos from the TV series are acquired. These multiple videos are input into a segment mining system 1201, which outputs the title and ending sequences of the videos. The title and ending sequences of these videos are stored in a segment database 1202. If the target video for the TV series is updated, the target video is input into a video retrieval system 1203, which uses the target video to search the segment database 1202 to obtain the title and ending sequences of the target video. The technical solution provided in this application uses a method of searching for video segments within the same video set that occur within the same video set. That is, for the same video set, identical video segments are found through retrieval and temporal positioning, and these are used as the mined title and ending sequences. Cross-duplicate deduplication refers to finding duplicate video segments within a video set through mutual retrieval. The purpose of video deduplication retrieval is to retrieve video segments that are identical to those in the database for the first video.

It should be noted that a video may have multiple opening or closing credits that meet the above requirements, which is normal. For a TV series with an opening theme song + episode highlights + the same product placement + main episode, the opening theme song and product placement can be matched in multiple videos, but the highlights will not be matched because they are different in each episode, so there will be 2 opening credits.

In the application of this video clip retrieval method to identify infringing videos, the target video to be identified is input into the video retrieval system. The video retrieval system extracts features from the target video to obtain video frame features of multiple target video frames, where the target video is the video to be identified for infringement. Through the video retrieval system, the video frame features of the multiple target video frames are matched in a clip database to obtain the target video clip within the target video, which is either the opening or closing credits of the target video. The target video clip is then deleted from the target video. Infringement identification is performed based on the target video after deleting the target video clip. The purpose of infringement identification is to determine whether the content of the target video after deleting the target video clip is the same as that of the specified video. Infringement identification is implemented by an infringement identification system that can deduplicate the query video in the infringement protection video database. If duplicates are found, it indicates infringement. However, since only the main content needs to be protected, the opening and closing credits of regular films and television dramas are not within the scope of infringement deduplication. The technical solution provided in this application embodiment can achieve the identification of opening and closing credits of films and television dramas.

The following will provide further explanation with reference to Figure 13.

The TV series to be identified for infringement identification is determined. Multiple videos from the TV series are obtained and stored in the infringement protection video database 1301. These videos are then input into a segment mining system 1302, which outputs the intro and outro of each video. The intro and outro of these videos are stored in the segment database 1303. If infringement identification of the target video is required, the target video is input into a video retrieval system 1304, which searches the segment database 1303 using the target video to obtain its intro and outro. The intro and outro of the target video are then deleted, and the infringement identification system 1305 outputs the infringement result for the target video, which includes whether it infringes or not.

In some embodiments, after querying the segment database based on the target video in the above manner, if multiple target video segments of the target video are obtained, the server determines the longest target video segment among the multiple target video segments as the final target video segment. When the technical solution provided in this application embodiment is applied to identify the beginning and end of a video, the target video segment is also the beginning and end of the target video. The process is shown in Figure 14.

In addition, the video retrieval system and the segment mining system can simultaneously provide external interfaces, allowing for both retrieval and mining to be performed simultaneously, with users specifying the specific functions they wish to use. Alternatively, only one identification interface can be provided, with the backend determining whether to perform a retrieval or mining based on whether the corresponding TV series' opening and closing credits already exist in the database. The backend then triggers the specific functions to be used, which include both retrieval and mining.

All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

The technical solution provided in this application determines video frame pairs containing similar video frames based on the similarity between video frame features. The first video frame in the video frame pair is fused based on the time difference of its occurrence to obtain at least one candidate video segment. Finally, a target video segment within a target time range is determined from the at least one candidate video segment. The process of determining the target segment requires no manual intervention; it can be automatically performed by computer equipment directly based on the first video and at least one second video, resulting in high efficiency.

Based on the above video segment matching algorithm design, a similar video segment matching method based on video frame features is implemented. This method supports similar video segments with variable length (reflected in the matching logic, when merging video frame pairs with the same occurrence time difference, the merged frames do not need to be consecutive) and positional changes (reflected in the matching logic, when the occurrence time difference is 0, the position remains unchanged; when the occurrence time difference is greater than 0, the position can change). This method is time-efficient and has excellent performance.

Figure 15 is a schematic diagram of a video segment recognition device provided in an embodiment of this application. Referring to Figure 15, the device includes: a video frame pair determination module 1501, a fusion module 1502, and a target video segment determination module 1503.

The video frame pair determination module 1501 is used to determine multiple video frame pairs based on the video frame features of a first video and the video frame features of at least one second video. The video frame pair includes a first video frame and a second video frame whose similarity meets the similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video.

The fusion module 1502 is used to fuse the first video frame in the plurality of video frame pairs based on the occurrence time difference of the plurality of video frame pairs to obtain at least one candidate video segment in the first video. The occurrence time difference refers to the difference between the occurrence times of two video frames in the video frame pair in the video.

The target video segment determination module 1503 is used to determine at least one target video segment in the first video based on the at least one candidate video segment and the target time range, wherein the target video segment is within the target time range of the first video.

In one possible implementation, the fusion module 1502 is used to divide the plurality of video frame pairs into a plurality of video frame groups based on the occurrence time difference of the plurality of video frame pairs, wherein video frame pairs in the same video frame group correspond to the same occurrence time difference. For any video frame group in the plurality of video frame groups, the first video frames of the video frame pairs in the video frame group are fused into a candidate video segment according to the occurrence time of the first video frame of the video frame pair in the first video.

In one possible implementation, the fusion module 1502 is used to divide video frame pairs with the same occurrence time difference into an initial video frame group. Based on the occurrence time difference corresponding to multiple initial video frame groups, the multiple initial video frame groups are fused to obtain the multiple video frame groups.

In one possible implementation, the fusion module 1502 is used to sort the multiple initial video frame groups according to a target order to obtain multiple candidate video frame groups. If the matching time difference between any two adjacent candidate video frame groups meets a matching time difference condition, the two adjacent candidate video frame groups are fused into one video frame group. The matching time difference refers to the difference between the occurrence time differences of the corresponding two adjacent candidate video frame groups.

In one possible implementation, the two adjacent candidate video frame groups include a first candidate video frame group and a second candidate video frame group. The fusion module 1502 is used to add the video frame pairs in the first candidate video frame group to the second candidate video frame group to obtain the video frame group when the matching time difference between the occurrence time difference corresponding to the first candidate video frame group and the occurrence time difference corresponding to the second candidate video frame group is less than or equal to the matching difference threshold.

In one possible implementation, the fusion module 1502 is used to add video frame pairs from the first candidate video frame group to the second candidate video frame group. Based on the occurrence time difference corresponding to the second candidate video frame group, a reference second video frame is used to replace the target second video frame to obtain the video frame group. The target second video frame is the second video frame newly added to the second candidate video frame group, and the reference second video frame is the second video frame in the second video frame whose occurrence time difference with the target first video frame is the occurrence time difference corresponding to the second candidate video frame group. The target first video frame is the first video frame in the video frame pair to which the target second video frame belongs.

In one possible implementation, the fusion module 1502 is used to compare the occurrence times of the first video frames of any two adjacent video frame pairs in the video frame group within the first video. If the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video meets the occurrence time condition, the two adjacent video frame pairs are added to a temporary frame list. If the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video does not meet the occurrence time condition, the video frame pairs in the temporary frame list are fused into a reference video segment. Based on multiple reference video segments, at least one candidate video segment is determined.

In one possible implementation, the plurality of reference video segments includes a first overlapping video segment and/or a second overlapping video segment. The first overlapping video segment refers to a reference video segment belonging to the first reference video segment among the plurality of reference video segments, and the second overlapping video segment refers to a reference video segment that partially overlaps with the second reference video segment among the plurality of reference video segments. The fusion module 1502 is configured to perform at least one of the following:

If the multiple reference video segments include the first overlapping video segment, the first overlapping video segment is deleted to obtain the at least one candidate video segment.

If the plurality of reference video segments include the second overlapping video segment, the overlapping portion between the second overlapping video segment and the second reference segment is deleted to obtain the at least one candidate video segment.

In one possible implementation, the fusion module 1502 is further configured to: compare the duration of a third type of reference video segment with a target duration, wherein the third type of reference video segment refers to the second overlapping video segment after deleting overlapping portions. If the duration of the third type of reference video segment is greater than or equal to the target duration, the third type of reference video segment is retained. If the duration of the third type of reference video segment is less than the target duration, the third type of reference video segment is deleted.

In one possible implementation, the target video segment determination module 1503 is used to determine at least one target candidate video segment based on the at least one candidate video segment, wherein the number of times the target candidate video segment appears in the at least one candidate video segment meets a frequency condition.

If the appearance time of any target candidate video segment in the first video falls within the target time range, the target candidate video segment is determined as the target video segment in the first video.

In one possible implementation, the target video segment determination module 1503 is configured to determine at least one reference candidate video segment based on the at least one candidate video segment. It determines the frequency of each reference candidate video segment within the at least one reference candidate video segment. Reference candidate video segments whose frequency of occurrence meets the occurrence frequency condition are determined as target candidate video segments.

In one possible implementation, the at least one candidate video segment includes a third overlapping video segment and/or a fourth overlapping video segment. The third overlapping video segment refers to a candidate video segment that belongs to the first candidate video segment among the at least one candidate video segments, and the fourth overlapping video segment refers to a candidate video segment that partially overlaps with the second candidate video segment among the at least one candidate video segments. The target video segment determination module 1503 is configured to perform at least one of the following:

If the at least one candidate video segment includes the third overlapping video segment, the third overlapping video segment is deleted to obtain the at least one reference candidate video segment.

If at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment meets the overlap condition, the occurrence count of the fourth overlapping video segment is determined. Based on the occurrence count of the fourth overlapping video segment, the at least one reference candidate video segment is determined.

If the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment does not meet the overlap condition, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment.

If the at least one candidate video segment includes the fourth overlapping video segment, and the duration of the fourth overlapping video segment is less than that of the second candidate video segment, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment.

In one possible implementation, the target video segment determination module 1503 is configured to perform any of the following:

If the number of occurrences of the fourth overlapping video segment is greater than or equal to the first occurrence threshold, the fourth overlapping video segment is fused with the second candidate video segment to obtain the at least one reference candidate video segment.

If the number of occurrences of the fourth overlapping video segment is less than the first occurrence threshold, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment.

In one possible implementation, the device further includes:

The feature extraction module is used to extract features from multiple target video frames of the target video to be identified, thereby obtaining the video frame features of the multiple target video frames.

The target video segment determination module 1503 is further configured to determine at least one target video segment of the target video based on the video frame features of the plurality of target video frames, the video frame features of the first video frame, and the video frame features of the at least one second video.

It should be noted that the video segment recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video segment recognition device and the video segment recognition method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

The technical solution provided in this application determines video frame pairs containing similar video frames based on the similarity between video frame features. The first video frame in the video frame pair is fused based on the time difference of its occurrence to obtain at least one candidate video segment. Finally, a target video segment within a target time range is determined from the at least one candidate video segment. The process of determining the target segment requires no manual intervention; it can be automatically performed by computer equipment directly based on the first video and at least one second video, resulting in high efficiency.

This application provides a computer device for performing the above-described method. This computer device can be implemented as a terminal or a server. The structure of the terminal will be described below:

Figure 16 is a schematic diagram of the structure of a terminal provided in an embodiment of this application.

Typically, terminal 1600 includes one or more processors 1601 and one or more memories 1602.

Processor 1601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

The memory 1602 may include one or more computer-readable storage media, which may be non-transitory. The memory 1602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1602 are used to store at least one computer program, which is executed by the processor 1601 to implement the video segment recognition method provided in the method embodiments of this application.

In some embodiments, the terminal 1600 may also optionally include a peripheral device interface 1603 and at least one peripheral device. The processor 1601, memory 1602, and peripheral device interface 1603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1604, a display screen 1605, a camera assembly 1606, an audio circuit 1607, and a power supply 1608.

Peripheral interface 1603 can be used to connect at least one I/O (Input/Output) related peripheral device to processor 1601 and memory 1602. In some embodiments, processor 1601, memory 1602 and peripheral interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1601, memory 1602 and peripheral interface 1603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The radio frequency (RF) circuit 1604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc.

Display screen 1605 is used to display a user interface (UI). This UI may include graphics, text, icons, video, and any combination thereof. When display screen 1605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1601 for processing. In this case, display screen 1605 can also be used to provide virtual buttons and/or a virtual keyboard, also known as soft buttons and/or a soft keyboard.

The camera assembly 1606 is used to capture images or videos. Optionally, the camera assembly 1606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal.

The audio circuit 1607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1601 for processing, or input to the radio frequency circuit 1604 to realize voice communication.

Power supply 1608 is used to supply power to the various components in terminal 1600. Power supply 1608 can be AC power, DC power, a disposable battery, or a rechargeable battery.

In some embodiments, the terminal 1600 further includes one or more sensors 1609. The one or more sensors 1609 include, but are not limited to: an accelerometer 1610, a gyroscope 1611, a pressure sensor 1612, an optical sensor 1613, and a proximity sensor 1614.

Accelerometer 1610 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established with terminal 1600.

The gyroscope sensor 1611 can detect the orientation and rotation angle of the terminal 1600. The gyroscope sensor 1611 can work in conjunction with the accelerometer sensor 1610 to collect the user's 3D movements on the terminal 1600.

The pressure sensor 1612 can be installed on the side bezel of the terminal 1600 and/or on the lower layer of the display screen 1605. When the pressure sensor 1612 is installed on the side bezel of the terminal 1600, it can detect the user's grip signal on the terminal 1600, and the processor 1601 can perform left/right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1612. When the pressure sensor 1612 is installed on the lower layer of the display screen 1605, the processor 1601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1605.

An optical sensor 1613 is used to collect ambient light intensity. In one embodiment, a processor 1601 can control the display brightness of a display screen 1605 based on the ambient light intensity collected by the optical sensor 1613.

The proximity sensor 1614 is used to detect the distance between the user and the front of the terminal 1600.

Those skilled in the art will understand that the structure shown in FIG16 does not constitute a limitation on the terminal 1600, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

The aforementioned computer equipment can also be implemented as a server. The structure of a server is described below:

Figure 17 is a schematic diagram of a server structure provided in an embodiment of this application. The server 1700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1701 and one or more memories 1702. The one or more memories 1702 store at least one computer program, which is loaded and executed by the one or more processors 1701 to implement the methods provided in the various method embodiments described above. Of course, the server 1700 may also have wired or wireless network interfaces, a keyboard, and input/output interfaces for input and output. The server 1700 may also include other components for implementing device functions, which will not be elaborated upon here.

In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to implement the video segment recognition method described in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method for recognizing video segments.

In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims (23)

1. A method for identifying video segments, characterized in that the method comprises: Based on the video frame features of the first video and the video frame features of at least one second video, multiple video frame pairs are determined. The video frame pairs include a first video frame and a second video frame whose similarity meets the similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video. Video frame pairs with the same occurrence time difference are divided into an initial video frame group, where the occurrence time difference refers to the difference between the occurrence times of the two video frames in the video frame pair. Multiple initial video frame groups are sorted according to the target order to obtain multiple candidate video frame groups; If the matching time difference between any two adjacent candidate video frame groups in the plurality of candidate video frame groups meets the matching time difference condition, the two adjacent candidate video frame groups are merged into a target video frame group. The matching time difference refers to the difference between the occurrence time differences of the corresponding two adjacent candidate video frame groups. For any one of the multiple target video frame groups, the first video frame of the video frame pair in the target video frame group is merged into a candidate video segment according to the occurrence time of the first video frame of the video frame pair in the first video. Based on at least one candidate video segment, at least one target candidate video segment is determined, wherein the number of times the target candidate video segment appears in the at least one candidate video segment meets a frequency condition; If the appearance time of any of the target candidate video segments in the first video is within the target time range, the target candidate video segment is determined as the target video segment in the first video, and the target video segment is the beginning or the end of the video. 2. The method according to claim 1, wherein the two adjacent candidate video frame groups include a first candidate video frame group and a second candidate video frame group, and the step of fusing the two adjacent candidate video frame groups into a target video frame group includes: If the matching time difference between the occurrence time difference corresponding to the first candidate video frame group and the occurrence time difference corresponding to the second candidate video frame group is less than or equal to the matching difference threshold, the video frame pairs in the first candidate video frame group are added to the second candidate video frame group to obtain the target video frame group. 3. The method according to claim 2, wherein adding the video frame pairs from the first candidate video frame group to the second candidate video frame group to obtain the target video frame group comprises: Add the video frame pairs from the first candidate video frame group to the second candidate video frame group; Based on the occurrence time difference value corresponding to the second candidate video frame group, the target second video frame is replaced by a reference second video frame to obtain the target video frame group. The target second video frame is the second video frame newly added to the second candidate video frame group. The reference second video frame is the second video frame in the second video whose occurrence time difference with the target first video frame is the occurrence time difference value corresponding to the second candidate video frame group. The target first video frame is the first video frame in the video frame pair to which the target second video frame belongs. 4. The method according to claim 1, wherein fusing the first video frames of the video frame pairs in the target video frame group into a candidate video segment according to the occurrence time of the first video frame of the video frame pair in the first video includes: Compare the occurrence times of the first video frame in the first video of any two adjacent video frame pairs in the target video frame group. If the difference between the occurrence times of the first video frame of the two adjacent video frame pairs in the first video meets the occurrence time condition, the two adjacent video frame pairs are added to the temporary frame list. If the difference between the occurrence times of the first video frame in the first video of two adjacent video frame pairs does not meet the occurrence time condition, the video frame pairs in the temporary frame list are merged into a reference video segment. Based on multiple reference video segments, at least one candidate video segment is determined. 5. The method according to claim 4, wherein the plurality of reference video segments includes a first overlapping video segment and/or a second overlapping video segment, the first overlapping video segment being a reference video segment belonging to a first reference video segment among the plurality of reference video segments, the second overlapping video segment being a reference video segment partially overlapping with a second reference video segment among the plurality of reference video segments, and determining the at least one candidate video segment based on the plurality of reference video segments includes at least one of the following: If the plurality of reference video segments include the first overlapping video segment, the first overlapping video segment is deleted to obtain the at least one candidate video segment; If the plurality of reference video segments include the second overlapping video segment, the overlapping portion between the second overlapping video segment and the second reference video segment is deleted to obtain the at least one candidate video segment. 6. The method according to claim 5, characterized in that, after deleting the overlapping portion between the second overlapping video segment and the second reference video segment when the plurality of reference video segments include the second overlapping video segment, the method further includes: Compare the duration of the third type of reference video segment with the target duration, wherein the third type of reference video segment refers to the second overlapping video segment after deleting the overlapping part; If the duration of the third type of reference video segment is greater than or equal to the target duration, the third type of reference video segment shall be retained. If the duration of the third type of reference video segment is less than the target duration, the third type of reference video segment shall be deleted. 7. The method according to claim 1, wherein determining at least one target candidate video segment based on at least one candidate video segment comprises: Based on the at least one candidate video segment, at least one reference candidate video segment is determined; Determine the number of times each of the reference candidate video segments appears in the at least one reference candidate video segment; Reference candidate video segments whose occurrence frequency meets the stated frequency condition are identified as target candidate video segments. 8. The method according to claim 7, wherein the at least one candidate video segment includes a third overlapping video segment and/or a fourth overlapping video segment, the third overlapping video segment being a candidate video segment belonging to a first candidate video segment among the at least one candidate video segments, the fourth overlapping video segment being a candidate video segment partially overlapping with a second candidate video segment among the at least one candidate video segments, and determining at least one reference candidate video segment based on the at least one candidate video segment includes at least one of the following: If the at least one candidate video segment includes the third overlapping video segment, the third overlapping video segment is deleted to obtain the at least one reference candidate video segment; If the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment meets the overlap condition, the occurrence count of the fourth overlapping video segment is determined; based on the occurrence count of the fourth overlapping video segment, the at least one reference candidate video segment is determined. If the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment does not meet the overlap condition, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment. If the at least one candidate video segment includes the fourth overlapping video segment, and the duration of the fourth overlapping video segment is less than that of the second candidate video segment, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment. 9. The method according to claim 8, wherein determining the at least one reference candidate video segment based on the occurrence count of the fourth overlapping video segment includes any one of the following: If the number of occurrences of the fourth overlapping video segment is greater than or equal to the first occurrence threshold, the fourth overlapping video segment is fused with the second candidate video segment to obtain the at least one reference candidate video segment. If the number of occurrences of the fourth overlapping video segment is less than the first occurrence threshold, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment. 10. The method according to claim 1, wherein the method further comprises: Feature extraction is performed on multiple target video frames of the target video to be identified to obtain the video frame features of the multiple target video frames; Based on the video frame features of the plurality of target video frames, the video frame features of the first video frame, and the video frame features of the at least one second video, at least one target video segment of the target video is determined. 11. A video clip recognition device, characterized in that the device comprises: A video frame pair determination module is used to determine multiple video frame pairs based on video frame features of a first video and video frame features of at least one second video. The video frame pairs include a first video frame and a second video frame whose similarity meets a similarity condition. The first video frame belongs to the first video, and the second video frame belongs to the at least one second video. The fusion module is used to divide video frame pairs with the same occurrence time difference into an initial video frame group, wherein the occurrence time difference refers to the difference between the occurrence times of the two video frames in the video frame pair in the video. The fusion module is further configured to sort multiple initial video frame groups according to the target order to obtain multiple candidate video frame groups; if the matching time difference between any two adjacent candidate video frame groups meets the matching time difference condition, the two adjacent candidate video frame groups are fused into a target video frame group, wherein the matching time difference refers to the difference between the occurrence time differences of the corresponding two adjacent candidate video frame groups. The fusion module is further configured to, for any target video frame group among the plurality of target video frame groups, fuse the first video frame of the video frame pair in the target video frame group into a candidate video segment according to the occurrence time of the first video frame of the video frame pair in the first video. The target video segment determination module is used to determine at least one target candidate video segment based on at least one candidate video segment, wherein the number of times the target candidate video segment appears in the at least one candidate video segment meets a frequency condition; The target video segment determination module is further configured to determine the target candidate video segment as the target video segment in the first video when the appearance time of any target candidate video segment in the first video is within a target time range, wherein the target video segment is the beginning or the end of a video. 12. The apparatus according to claim 11, wherein the two adjacent candidate video frame groups include a first candidate video frame group and a second candidate video frame group, and the fusion module is used to add video frame pairs in the first candidate video frame group to the second candidate video frame group to obtain the target video frame group when the matching time difference between the occurrence time difference corresponding to the first candidate video frame group and the occurrence time difference corresponding to the second candidate video frame group is less than or equal to a matching difference threshold. 13. The apparatus according to claim 12, wherein the fusion module is configured to add video frame pairs from the first candidate video frame group to the second candidate video frame group; based on the occurrence time difference corresponding to the second candidate video frame group, replace the target second video frame with a reference second video frame to obtain the target video frame group, wherein the target second video frame is a second video frame newly added to the second candidate video frame group, the reference second video frame is a second video frame in the second video whose occurrence time difference with the target first video frame is the occurrence time difference corresponding to the second candidate video frame group, and the target first video frame is the first video frame in the video frame pair to which the target second video frame belongs. 14. The apparatus according to claim 11, wherein the fusion module is configured to: compare the occurrence times of the first video frames of any two adjacent video frame pairs in the target video frame group in the first video; add the two adjacent video frame pairs to a temporary frame list if the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video meets the occurrence time condition; fuse the video frame pairs in the temporary frame list into a reference video segment if the difference between the occurrence times of the first video frames of the two adjacent video frame pairs in the first video does not meet the occurrence time condition; and determine the at least one candidate video segment based on multiple reference video segments. 15. The apparatus according to claim 14, wherein the plurality of reference video segments includes a first overlapping video segment and/or a second overlapping video segment, the first overlapping video segment being a reference video segment belonging to a first reference video segment among the plurality of reference video segments, and the second overlapping video segment being a reference video segment partially overlapping with a second reference video segment among the plurality of reference video segments, the fusion module being configured to perform at least one of the following: If the plurality of reference video segments include the first overlapping video segment, the first overlapping video segment is deleted to obtain the at least one candidate video segment; If the plurality of reference video segments include the second overlapping video segment, the overlapping portion between the second overlapping video segment and the second reference video segment is deleted to obtain the at least one candidate video segment. 16. The apparatus according to claim 15, wherein the fusion module is further configured to compare the duration of a third type of reference video segment with a target duration, wherein the third type of reference video segment refers to the second overlapping video segment with the overlapping portion deleted; retain the third type of reference video segment if the duration of the third type of reference video segment is greater than or equal to the target duration; and delete the third type of reference video segment if the duration of the third type of reference video segment is less than the target duration. 17. The apparatus according to claim 11, wherein the target video segment determination module is configured to determine at least one reference candidate video segment based on the at least one candidate video segment; determine the number of times each of the reference candidate video segments appears in the at least one reference candidate video segment; and determine the reference candidate video segments whose number of appearances meets the number condition as target candidate video segments. 18. The apparatus according to claim 17, wherein the at least one candidate video segment includes a third overlapping video segment and/or a fourth overlapping video segment, the third overlapping video segment being a candidate video segment belonging to a first candidate video segment among the at least one candidate video segments, and the fourth overlapping video segment being a candidate video segment partially overlapping with a second candidate video segment among the at least one candidate video segments, and the target video segment determining module is configured to perform at least one of the following: If the at least one candidate video segment includes the third overlapping video segment, the third overlapping video segment is deleted to obtain the at least one reference candidate video segment; If the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment meets the overlap condition, the occurrence count of the fourth overlapping video segment is determined; based on the occurrence count of the fourth overlapping video segment, the at least one reference candidate video segment is determined. If the at least one candidate video segment includes the fourth overlapping video segment, and the overlap between the fourth overlapping video segment and the second candidate video segment does not meet the overlap condition, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment. If the at least one candidate video segment includes the fourth overlapping video segment, and the duration of the fourth overlapping video segment is less than that of the second candidate video segment, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment. 19. The apparatus according to claim 18, wherein the target video segment determination module is configured to perform any of the following: If the number of occurrences of the fourth overlapping video segment is greater than or equal to the first occurrence threshold, the fourth overlapping video segment is fused with the second candidate video segment to obtain the at least one reference candidate video segment. If the number of occurrences of the fourth overlapping video segment is less than the first occurrence threshold, the fourth overlapping video segment is deleted to obtain the at least one reference candidate video segment. 20. The apparatus according to claim 11, wherein the apparatus further comprises: The feature extraction module is used to extract features from multiple target video frames of the target video to be identified, and to obtain the video frame features of the multiple target video frames. The target video segment determination module is further configured to determine at least one target video segment of the target video based on the video frame features of the plurality of target video frames, the video frame features of the first video frame, and the video frame features of the at least one second video. 21. A computer device, characterized in that the computer device comprises one or more processors and one or more memories, wherein the one or more memories store at least one computer program, the computer program being loaded and executed by the one or more processors to implement the video segment recognition method as described in any one of claims 1 to 10. 22. A computer-readable storage medium, characterized in that the computer-readable storage medium stores at least one computer program, the computer program being loaded and executed by a processor to implement the video segment recognition method as described in any one of claims 1 to 10. 23. A computer program product comprising a computer program, characterized in that, when executed by a processor, the computer program implements the video segment recognition method according to any one of claims 1 to 10.