TWI881909B - Video similarity judgment method and video recommendation method based on video key frames - Google Patents

Abstract

The present invention provides a video similarity determination method and a video recommendation method. First, a first video and a second video are received. A first key scene contained in the first video is identified, and a second key scene contained in the second video is identified, wherein the first key scene of the first video has a first time proportion, and the second key scene of the second video has a second time proportion. A first key frame in the first key scene of the first video is extracted, and a second key frame in the second key scene of the second video is extracted. Using image recognition means, a similarity between the first key frame and the second key frame is determined. Based on the product value of the first time proportion, the second time proportion and the similarity, a recommendation value of the first video or the second video is calculated.

Description

Video similarity judgment method and video recommendation method based on video key frames

The present invention relates to a video similarity determination method and a video recommendation method, and in particular to a video similarity determination method and a video recommendation method based on video key frame analysis. More specifically, the method of the present invention is applied to video recommendation, and by capturing one or more key frames in a video for image analysis, other videos related to the target video are recommended.

Common video recommendation mechanisms mainly generate recommendations by analyzing pre-defined video content categories and tags and users' viewing or purchasing habits. However, these recommendations have blind spots and cannot detect that some videos actually have similar content but completely different categories or tags.

The present invention proposes a recommendation mechanism, which finds videos with relevant audio and video content through image analysis and recommends them accordingly. The present invention uses video keyframes to train an image analysis model and uses the analysis results to recommend videos.

The present invention aims to provide a method for determining video similarity based on video key frames. The method is executed by a processor and comprises: receiving a first video and a second video; identifying a first key scene contained in the first video and a second key scene contained in the second video, wherein the first key scene of the first video has a first time proportion, and the second key scene of the second video has a second time proportion. extracting a first key frame from a first key scene of the first video and extracting a second key frame from a second key scene of the second video; determining a similarity between the first key frame and the second key frame by image recognition means; and determining a similarity between a first scene of the first video and a second scene of the second video based on a product value of the first time proportion, the second time proportion and the similarity.

Another object of the present invention is a method for recommending videos based on key frames of videos, the method being executed by a processor, the method comprising: receiving a first video and a second video, wherein the first video is a video selected by a user to watch on a streaming video platform, and the second video is one of a set of candidate videos to be recommended; identifying a first key scene contained in the first video, and identifying a second key scene contained in the second video, wherein the first key scene of the first video has a first time proportion, and the second key scene of the second video has a first time proportion. The key scene has a second time proportion; a first key frame is extracted from the first key scene of the first video, and a second key frame is extracted from the second key scene of the second video; a similarity between the first key frame and the second key frame is determined by image recognition means; based on the product of the first time proportion, the second time proportion and the similarity, it is determined that the second scene of the second video has a recommendation value; and based on the recommendation value, the second video is presented on the streaming video platform where the user watches the first video.

In a specific embodiment, the first key frame and the second key frame are complete image files.

In a specific embodiment, the first time proportion is a play length of the first scene divided by the total play length of the first video, and the second time proportion is a play length of the second scene divided by the total play length of the second video.

In a specific embodiment, the data description of the first scene and the second scene is composed of at least a scene name, a key frame name, a time code description and a time ratio.

The method of the present invention is applied to video recommendation. By capturing one or more key frames in a video and performing image analysis, other videos related to the target video are recommended to enhance the user experience and stickiness of the video platform.

The present invention will be more fully described below with reference to the drawings, and specific exemplary embodiments are shown by way of illustration. However, the claimed subject matter may be embodied in many different forms, and thus the construction of the claimed subject matter covered or claimed is not limited to any exemplary embodiment disclosed in this specification; the exemplary embodiments are merely illustrative. Likewise, the present invention is to provide a reasonably broad scope for the claimed or covered subject matter. In addition, the drawings and illustrations in the present invention are generally not drawn to scale and are not intended to correspond to actual relative sizes.

For the purpose of consistency and ease of understanding, the same features are marked by reference numerals in the exemplary drawings (although in some examples they are not so marked). However, the features in different embodiments may differ in other aspects and should not be narrowly limited to the features shown in the drawings. The terms "first" and "second" in the specification of the present invention and the above drawings are used to distinguish different objects, rather than to describe a specific order.

The "video" mentioned in the present invention specifically refers to the video provided by the streaming service platform and technology, such as a movie trailer, but is not limited thereto. The first figure illustrates the composition of a video, which includes multiple scenes, such as scene 1 to scene 10 with different play lengths. Each scene is composed of multiple frames. The "frame" mentioned in the present invention represents an image. For example, scene 4 is composed of frame 4-1 to frame 4-7. The sum of the different play lengths of all scenes constitutes a total play length of the video.

In one embodiment of the present invention, the composition of a video is divided into one or more key scenes and one or more non-key scenes. The "key scene" described in the present invention represents a scene containing a key frame (Keyframe) or other custom scenes. Since a key scene is composed of multiple frames, in order to compress a video, it is known that one of the multiple frames in a key scene can be designated as a key frame, and other frames can be compressed based on the key frame, so that other frames only retain image change information and omit repeated information, thereby achieving the effect of image compression. The detailed technical means are already well known to technicians in this field, so the details are not repeated here. The playback length corresponding to each key scene can be described by a time code, which represents the time from which the key scene is played to which minute in the video. For example, the time code can be described as "00:00-00:03". In one embodiment, each key frame is represented by a named image file, such as "001.jpg". In another embodiment, the key frame is not limited to the first frame in the key scene. In other embodiments, there can be more than one key frame in a key scene, in which case one should be selected as the key frame of the key scene.

Generally speaking, trailers provided by film distributors can be regarded as a type of film containing a large number of key scenes (compared to a complete film). The method of the present invention extracts the key scene portion of a film and analyzes the relationship between the key frames and the film, thereby establishing key scene data for each trailer and forming a key scene database for multiple trailers.

The second figure shows the composition of a video. The total playback length of the video is 10 seconds, and the video is composed of key scenes 1 to 3 and a non-key scene. Key scene 2 is composed of five frames, the first frame is set as the key frame, and the rest are compressed frames 1 to 4.

Since a key frame is a complete image, it can be used to compare with other images to obtain a similarity. The time code of a key scene can be used to analyze the proportion of the key scene in a complete video, which is called "time proportion" in the present invention.

The third figure illustrates the key scenes 1 to 3 in the video A and their defined key frame names, time codes and time proportions. In this example, the total playback length of the video A is 10 seconds. According to the description of the time code, key scene 1 is played from the 0th second to the 3rd second, key scene 2 is played from the 4th second to the 6th second, and key scene 3 is played from the 6th second to the 10th second. As for the 3rd second to the 4th second, it is not a key scene. Key scene 1 contains a key frame named "001.jpg". Similarly, key scene 2 and key scene 3 contain key frames "002.jpg" and "003.jpg" respectively. The time proportion of key scene 1 is the playback length of the scene divided by the total playback length of video A, that is, 3 seconds divided by 10 seconds, which equals 0.3. Similarly, the time proportion of key scene 2 is 0.2, and the time proportion of key scene 3 is 0.4. Accordingly, the key scene data of video A (including key scenes, key frame images, time codes and time proportions) is established.

FIG. 4 schematically shows the video recommendation system (2) of the present invention, which includes a scene recognition module (20), a scene weight analysis module (22), a similarity analysis module (24), and a recommendation module (26). Although not shown, the video recommendation system (2) can be connected to a streaming video system and establish key scene data for all videos provided by the streaming video system. The video recommendation system (2) is mainly implemented by one or more processors, memories, power supplies, and network interfaces. The video recommendation system (2) can access one or more videos from a video database (3), analyze the videos, and generate corresponding key scene data. The video database (3) can be a part of an existing streaming video system or be dedicated to the video recommendation system (2).

In response to a user selecting to watch video A in the streaming video system, the video recommendation system (2) reads key scene data about video A or about video A from the video database (3). At the same time, the video recommendation system (2) also reads other candidate videos to be recommended, such as video B to video H, from the video database (3). These candidate videos have not been selected by the user to watch in the streaming video system. The video recommendation system (2) analyzes video B to video H to form their respective key scene data.

The scene recognition module (20) is configured to analyze and identify one or more scenes and their properties contained in a video. In one case, the description of the video file includes a custom scene, and the scene in the video can be identified by defining a name. In another case, the description of the video file does not include a custom scene, and the scene in the video can be defined by using deep learning and image recognition techniques. After analyzing the video, the scene recognition module (20) forms key scene data about the video, such as the scene number/name, key frame file name, and scene time code in the third figure.

The scene proportion analysis module (22) is configured to calculate the proportion of a scene in the entire video based on the time code of the scene, and generate the time proportion of each scene.

The similarity analysis module (24) is configured to perform image recognition based on a frame in one scene (such as taken from video A) and another frame in another scene (such as taken from video B) to determine the degree of similarity between the two frames. This can be achieved by adopting known image recognition, which is not an improved part of the present invention and is not described in detail here. The degree of similarity can be represented by a numerical range, such as 0.0 to 1.0, where a value of 1.0 is highly similar and a value of 0.0 is low similarity. In one embodiment, the image recognition can be implemented based on the VGG (Visual Geometry Group) model. In another embodiment, the similarity can be determined based on the content of two different frames. For example, if the same brand or similar objects can be identified from two different frames, then the two are judged to be highly similar.

The recommendation module (26) is configured to determine a recommendation value for a video (such as videos B to H) and to sort and recommend the video to a user of a streaming video platform based on the recommendation value. More specifically, the recommendation module (26) determines a recommendation value for a scene in the video (such as videos B to H). In one embodiment of the present invention, the "recommendation value" is a time proportion of a scene in a candidate video multiplied by a time proportion of a scene in a video selected for viewing multiplied by a similarity, wherein the "similarity" is the similarity between a key frame of a scene in the candidate video and a key frame of a scene in the video selected for viewing, which is expressed as a numerical value. The "recommendation value" represents a specific scene in a candidate video, relative to a specific scene in a video selected for viewing, and the degree of similarity between the two scenes. In other words, the recommendation value of a scene is determined by the relationship between the scene and another scene.

The fifth figure illustrates the correlation between the key scene 1 of the movie A and other candidate movie scenes. As shown in the third figure, the movie A has a key scene 1, the key frame name represented by it is "001.jpg", the time code describing the playback length is "00:00-00:03", and the time weight is "0.3". It is known that the time weight of the key scene 2 in the movie B is "0.1", then the recommendation value of the key scene 2 of the movie B = 0.1 (the time weight of the key scene 2 of the movie B) * 0.3 (the time weight of the key scene 1 of the movie A) * 1 (the similarity between the key frame of the key frame of the key scene 2 of the movie B and the key frame "001.jpg" of the key scene 1 of the movie A) = 0.03. It is known that the time proportion of key scene 2 in video C is "0.1", then the recommendation value of key scene 2 in video C = 0.1 (time proportion of key scene 2 in video C) * 0.3 (time proportion of key scene 1 in video A) * 0.9 (similarity between the key frame of key scene 2 in video C and the key frame "001.jpg" of key scene 1 in video A) = 0.27. It is known that the time proportion of key scene 2 in video F is "0.1", then the recommendation value of key scene 2 in video F = 0.1 (time proportion of key scene 2 in video F) * 0.3 (time proportion of key scene 1 in video A) * 0.6 (similarity between the key frame of key scene 2 in video F and the key frame "001.jpg" of key scene 1 in video A) = 0.018. Therefore, for key scene 1 in video A, key scene 2 in video B has the highest recommendation value, followed by key scene 2 in video C, and then key scene 2 in video F. In order to simplify the explanation, this example only lists the top three for easy understanding.

The sixth figure illustrates the correlation between the key scene 2 of video A and other candidate video scenes. As shown in the third figure, video A has a key scene 2, the key frame name of which is "002.jpg", the time code describing the playback length is "00:04-00:06", and the time weight is "0.2". It is known that the time weight of the key scene 5 in video B is "0.2", so the recommendation value of the key scene 5 of video B = 0.2 (the time weight of the key scene 5 of video B) * 0.2 (the time weight of the key scene 2 of video A) * 1 (the similarity between the key frame of the key scene 5 of video B and the key frame "002.jpg" of the key scene 2 of video A) = 0.04. It is known that the time proportion of key scene 5 in movie D is "0.2", then the recommendation value of key scene 5 in movie D = 0.2 (time proportion of key scene 5 in movie D) * 0.2 (time proportion of key scene 2 in movie A) * 0.8 (similarity between the key frame of key scene 5 in movie D and the key frame "002.jpg" of key scene 2 in movie A) = 0.032. It is known that the time proportion of key scene 5 in movie G is "0.2", then the recommendation value of key scene 5 in movie G = 0.2 (time proportion of key scene 5 in movie G) * 0.2 (time proportion of key scene 2 in movie A) * 0.6 (similarity between the key frame of key scene 5 in movie G and the key frame "002.jpg" of key scene 2 in movie A) = 0.024. Therefore, for key scene 2 in movie A, key scene 5 in movie B has the highest recommendation value, followed by key scene 5 in movie D, and then key scene 5 in movie G. In order to simplify the explanation, this example only lists the top three for easy understanding.

FIG7 illustrates the correlation between the key scene 3 of movie A and other candidate movie scenes. As shown in FIG3, movie A has a key scene 3, the key frame name represented by it is "003.jpg", the time code describing the playback length is "00:06-00:10", and the time weight is "0.4". It is known that the time weight of the key scene 8 in movie C is "0.3", so the recommendation value of the key scene 8 in movie C = 0.3 (the time weight of the key scene 8 in movie C) * 0.4 (the time weight of the key scene 3 in movie A) * 1 (the similarity between the key frame of the key scene 8 in movie C and the key frame "003.jpg" of the key scene 3 in movie A) = 0.18. It is known that the time proportion of key scene 8 in movie E is "0.3", then the recommendation value of key scene 8 in movie E = 0.3 (time proportion of key scene 8 in movie E) * 0.4 (time proportion of key scene 3 in movie A) * 0.8 (similarity between the key frame of key scene 8 in movie E and the key frame "003.jpg" of key scene 3 in movie A) = 0.096. It is known that the time proportion of key scene 8 in movie H is "0.3", then the recommendation value of key scene 8 in movie H = 0.3 (time proportion of key scene 8 in movie H) * 0.4 (time proportion of key scene 3 in movie A) * 0.7 (similarity between the key frame of key scene 8 in movie H and the key frame "003.jpg" of key scene 3 in movie A) = 0.084. Therefore, with respect to key scene 3 in movie A, key scene 8 in movie C has the highest recommendation value, followed by key scene 8 in movie E, and then key scene 8 in movie H. In order to simplify the explanation, this example only lists the top three for easy understanding.

Figure 8 lists the total recommendation values of other films. According to the recommendation values analyzed in Figures 5, 6, and 7, the top three recommendations for key scene 1 of film A in the candidate films are key scene 2 of film B, key scene 2 of film C, and key scene 2 of film F, respectively; the top three recommendations for key scene 2 of film A in the candidate films are key scene 5 of film B, key scene 5 of film D, and key scene 5 of film G, respectively; the top three recommendations for key scene 3 of film A in the candidate films are key scene 8 of film C, key scene 8 of film E, and key scene 8 of film G, respectively. If only the top three recommendations are considered for each scene, the other candidate film recommendations of film A are the ranking of the total recommendation values of the candidate films. In the above example, video C is the first recommendation (recommendation value = 0.12 + 0.027 = 0.147, the highest), video E is the second recommendation (recommendation value = 0.096, the second highest), and video H is the third recommendation (recommendation value = 0.084, the third highest). The recommendation module (26) determines the recommendation ranking of the candidate videos based on the sum of the recommendation values.

FIG. 9 shows the video recommendation method of the present invention.

In step S900, a video is received, key scenes and a key frame in each key scene are analyzed and extracted, a description of the length of each key scene is obtained and converted into a time proportion representing each scene, and scene data specific to each video is formed and stored in the video database for use in recommending videos.

In step S902, similar scenes are determined using key frames of different videos. For example, a key frame of key scene 1 of video A and a key frame of key scene 2 of video B are image recognized, and the similarity between the two is calculated. Therefore, if the key frame of key scene 1 of video A and the key frame of key scene 2 of video B are both images containing cars, the two scenes are relatively similar; conversely, if one of the key frames is an image containing a car and the other key frame is an image containing a boat, the two scenes are relatively dissimilar. The description of similar scenes can use a numerical range, such as from low correlation of 0.0 to high correlation of 1.0.

In step S904, based on the time proportion of each key scene in each video and the similarity between key scenes in different videos, a recommendation value (i.e., the total recommendation value shown in FIG. 8 ) of the candidate video relative to the selected video is calculated.

In step S906, based on the recommendation values of all the candidate videos, a video recommendation ranking is generated, which indicates several partial videos that are preferentially presented to a user, so that the user can obtain viewing options of the recommended partial candidate videos after choosing to watch a video.

However, it should be understood that the specific embodiments of the present invention are only for illustrative purposes, and various changes can be made without departing from the scope and spirit of the patent application of the present invention, and all should be included in the patent scope of the present invention. Therefore, the specific embodiments described in this specification are not used to limit the present invention, and the true scope and spirit of the present invention are disclosed in the following patent application scope.

2: Recommendation system 20: Scene recognition module 22: Scene weight analysis module 24: Similarity analysis module 26: Recommendation module S900 to S906: Steps

The present invention may be further understood with reference to the following drawings and descriptions. Non-limiting and non-exhaustive examples are described with reference to the following drawings. The components in the drawings are not necessarily to actual size; the emphasis is on illustrating the structure and principle.

The first figure shows the composition of a video.

The second figure shows the composition of a video, including key scenes and non-key scenes.

The third figure illustrates key scenes 1 to 3 in video A.

FIG. 4 illustrates the video recommendation system of the present invention.

FIG. 5 illustrates the correlation between the key scene 1 of video A and other candidate video scenes.

FIG. 6 illustrates the correlation between the key scene 2 of video A and other candidate video scenes.

FIG. 7 illustrates the correlation between the key scene 3 of video A and other candidate video scenes.

The eighth figure lists the total recommendation values of other videos.

FIG. 9 shows the video recommendation method of the present invention.

S900 to S906: Steps

Claims (8)

A method for determining video similarity based on video key frames, the method being executed by a processor, the method comprising: receiving a first video and a second video; identifying a first key scene contained in the first video, and identifying a second key scene contained in the second video, the first key scene of the first video having a first time proportion, and the second key scene of the second video having a second time proportion; extracting a first key frame from the first key scene of the first video, and extracting a second key frame from the second key scene of the second video; using image recognition means to determine a similarity between the first key frame and the second key frame; and Based on the product of the first time proportion, the second time proportion and the similarity, the similarity between the first scene of the first video and the second scene of the second video is determined. The method as described in claim 1, wherein the first key frame and the second key frame are complete image files. A method as described in claim 1, wherein the first time ratio is a playback length of the first scene divided by a total playback length of the first video, and the second time ratio is a playback length of the second scene divided by a total playback length of the second video. The method as described in claim 1, wherein the data description of the first scene and the second scene is composed of at least a scene name, a key frame name, a time code description and a time ratio. A video recommendation method based on video key frames, the method is executed by a processor, and the method includes: Receiving a first video and a second video, wherein the first video is a video selected by a user to watch on a streaming video platform, and the second video is one of a set of candidate videos to be recommended; Identifying a first key scene contained in the first video, and identifying a second key scene contained in the second video, the first key scene of the first video has a first time proportion, and the second key scene of the second video has a second time proportion; Extracting a first key frame in the first key scene of the first video, and extracting a second key frame in the second key scene of the second video; Using image recognition means, determining a similarity between the first key frame and the second key frame; Based on the product of the first time proportion, the second time proportion and the similarity, determine that the second scene of the second video has a recommendation value; and Based on the recommendation value, present the second video on the streaming video platform. The method as described in claim 5, wherein the first key frame and the second key frame are complete image files. A method as described in claim 5, wherein the first time ratio is a playback length of the first scene divided by a total playback length of the first video, and the second time ratio is a playback length of the second scene divided by a total playback length of the second video. The method as described in claim 5, wherein the data description of the first scene and the second scene is composed of at least a scene name, a key frame name, a time code description and a time ratio.

Images