CN115665508A - Method, device, electronic device and storage medium for video abstract generation - Google Patents

Abstract

The present application provides a video summary generation method, device, electronic equipment and storage medium, the method may include the following steps: determining the characteristics of the video clip, the video clip is obtained by segmenting the video; determining the video image frame in the video clip Features, the number of video image frames in the video clip is multiple; determine the features of the attention information, and the attention information is used to represent the attention to the video in different dimensions; use the features of the video clips, the features of the video image frames in the video clips and the attention features of the information to generate a video summarization of the video. According to the embodiment of the present application, video summaries are adaptively generated for content with different user preferences by focusing on information. The same set of models is realized to meet the individual needs of different users.

Description

Method, device, electronic device and storage medium for video abstract generation

technical field

The present application relates to the technical field of video processing, and in particular to a method and device for generating a video abstract, electronic equipment, and a storage medium.

Background technique

As a method of concentrating videos, video summarization is often used in scenarios such as video preview, video clipping or video segment search. Existing video summary generation methods, one is to generate video summaries about people or cars for specific categories of targets that are concerned with specific scenes, such as traffic scenes; the other is general scenes, by collecting a large amount of data from various scenes and Annotation is performed to learn scene-appropriate video summarization. Existing video summarization methods have disadvantages such as single scene or large amount of labeled data. Apparently, the existing method for generating video summaries will cause different users to see the same video summaries, which will bring less freshness and poor experience to users.

Contents of the invention

Embodiments of the present application provide a method, device, electronic device, and storage medium for generating video summaries, so as to generate video summaries from different angles according to attention information.

In the first aspect, the embodiment of the present application provides a method for generating a video summary, which may include the following steps:

Determining the characteristics of the video clip, the video clip is obtained by segmenting the video;

Determine the characteristics of the video image frame in the video clip, the number of video image frames in the video clip is multiple;

Determine the characteristics of the attention information, and the attention information is used to represent the attention to the video in different dimensions;

A video summarization of a video is generated using the features of the video clip, the features of the video image frames in the video clip, and the features of the information of interest.

In a second aspect, the embodiment of the present application provides a method for generating a video summary, which may include the following steps:

Send the received attention information to the video to the generator of the video summary;

The generation terminal receiving the video summary responds to the video summary of the video generated by the attention information; the video summary of the video is generated by the generation terminal of the video summary using the characteristics of the video segment, the characteristics of the video image frame in the video segment and the characteristics of the attention information; the video segment is obtained by segmenting the video;

Displays the video summary of the video in the video preview window.

In a third aspect, the embodiment of the present application provides an apparatus for generating a video summary, which may include:

A video feature determination module is used to determine the feature of the video segment, and the video segment is obtained by segmenting the video;

Image characteristic determination module, is used for determining the feature of video image frame in video segment, the quantity of video image frame in each video segment is a plurality of;

A concerned information determination module is used to determine the characteristics of the concerned information, and the concerned information is used to represent the attention to the video in different dimensions;

The video summary generating module is used to generate a video summary of the video by using the features of the video clip, the features of the video image frame in the video clip and the features of the attention information.

In a fourth aspect, the embodiment of the present application provides an apparatus for generating a video summary, which may include:

The attention information sending module is used to send the received attention information to the video to the generation end of the video summary;

The video summary acquisition module is used to receive the video summary of the video generated by the generation end of the video summary in response to the attention information; the video summary of the video is that the generation terminal of the video summary utilizes the characteristics of the video clip, the characteristics of the video image frame in the video clip and the attention information generated by features; the video segment is obtained by segmenting the video;

The video summary display module is used to display the video summary of the video in the video preview window.

In a fifth aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory, and the processor implements the method described in any one of the above when executing the computer program.

In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method described in any one of the foregoing is implemented.

Compared with the prior art, the present application has the following advantages:

According to the embodiment of the present application, video summaries are adaptively generated based on content of different user preferences by focusing on information. It realizes that different users of the same model can generate summary videos that meet user preferences, so as to meet the individual needs of different users. At the same time, in the process of video summarization, multi-dimensional information such as video segment features, video image frame features, and attention information features can be referred to. Comprehensive consideration of multi-dimensional information can obtain higher-quality video summarization results.

The above description is only an overview of the technical solution of the present application. In order to understand the technical means of the present application more clearly, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, Specific embodiments of the present application are enumerated below.

Description of drawings

In the drawings, unless otherwise specified, the same reference numerals designate the same or similar parts or elements throughout the several drawings. The drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some implementations according to the application, and should not be considered as limiting the scope of the application.

FIG. 1 is a schematic diagram of a scene of a method for generating a video abstract provided by the present application;

FIG. 2 is one of the flow charts of the method for generating a video abstract according to an embodiment of the present application;

Fig. 3 is one of the flow charts of the method of determining the video clip according to an embodiment of the present application;

Fig. 4 is the second flow chart of the method for determining the video segment in an embodiment of the present application;

FIG. 5 is one of the flow charts of the specific process of generating a video abstract according to an embodiment of the present application;

FIG. 6 is the second flow chart of the specific process of generating a video abstract according to an embodiment of the present application;

FIG. 7 is the third flowchart of the specific process of generating a video abstract according to an embodiment of the present application;

FIG. 8 is one of the flow charts of the method of generating attention information according to an embodiment of the present application;

FIG. 9 is the second flow chart of the method of generating attention information in an embodiment of the present application;

FIG. 10 is the second flow chart of the method for generating a video abstract according to an embodiment of the present application;

Fig. 11 is the third flowchart of the method for generating a video abstract according to an embodiment of the present application;

FIG. 12 is a fourth flowchart of a method for generating a video abstract according to an embodiment of the present application;

FIG. 13 is one of the structural block diagrams of a video abstract generation device according to an embodiment of the present application;

Fig. 14 is the second structural block diagram of a video abstract generating device according to an embodiment of the present application; and

FIG. 15 is a block diagram of an electronic device used to implement an embodiment of the present application.

Detailed ways

In the following, only some exemplary embodiments are briefly described. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present application. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not restrictive.

In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies may be optionally combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them belong to the protection scope of the embodiments of the present application.

First, the nouns involved in this application are explained.

Chinese multimodal pre-training model (M6, Multi-Modality to Multi-Modality MultitaskMega-Transformer): This model is based on the translation model (Transformer) and pre-trained through multiple tasks. Pre-training makes the model capable of both unimodal and multimodal understanding and generation. The Chinese multimodal pre-training model can be applied to a series of downstream applications, for example, it can include description generation applied to objects, visual question answering, Chinese poetry generation, etc.

Video Feature Extraction Model (Video Transformer): Based on the self-attention mechanism, feature extraction of input video clips can be realized. The output of the video feature extraction model is the feature of the video clip.

Video shot-segmentation algorithm (Video Shot-Segmentation): It relies on pixel changes, histogram changes, edge changes, etc. between consecutive frames of the video to segment the video. Each video segment may correspond to a camera switching process in the video.

Video Scene Segmentation Algorithm (Video Scene-Segmentation): As a sub-task in video understanding, its main goal is to divide the long video into several video segments based on the scene content as the clue to segment the video. A video clip consists of consecutive multiple camera clips.

Video Captioning Algorithm (Video Captioning): The video captioning algorithm applies deep learning to the fields of computer vision and natural language processing. Specifically, given a video, the video captioning algorithm outputs text describing the video.

Fig. 1 is a schematic diagram of an exemplary application scenario for implementing the method of the embodiment of the present application. In the process of generating video summarization, different users often have different requirements. For example, for the same video, the first user's preference is the actor's clothing collocation, and the second user's preference is the food segment in the video. In order to solve this problem of different user preferences, the model is usually adjusted by relabeling the training samples to fine-tune the model parameters to obtain a new model, so as to use different models to meet the differentiated needs of users. However, the disadvantages of relabeling training samples are the increase in human labeling costs and model training time costs.

For this, in the current application scenario, the target video for which the video summary is to be generated can be split into multiple video segments according to specified rules. In addition, user's preference information may be received. The user's preference information may be received text information, audio information, or video information, etc. The user's preference information is used to represent the user's preference for the target video in different dimensions. For example, the clothing collocation and food clips shown in Figure 1 can represent the preferences of 2 different users. Multiple video clips of the target video are matched with the preference information, and the matching may include feature extraction of the video clips, features of the preference information, similarity comparison between the features of the video clips and the features of the preference information, and the like. Finally, the target video segment related to user preference can be determined. Based on the determined target video clips, after processes such as combination of target video clips, image quality adjustment, or video clip duration adjustment, a video summary that matches the user's preference information can be obtained. For example, in FIG. 1 , corresponding to the preference of the first user, a video summary related to clothing collocation content may be generated. Corresponding to the preferences of the second user, video summaries related to food may be generated. This can solve the disadvantages of relabeling training data for training fine-tuning brought about by different scenarios and different user preferences, and greatly reduce the disadvantages of resource consumption caused by multiple trainings for different scenarios and different users. , which can use a set of models to realize adaptive generation of video summaries based on preference information.

The embodiment of the present application provides a method for generating a video abstract, as shown in FIG. 2 , which is a flowchart of a method for generating a video abstract in an embodiment of the present application, which may include:

Step S201: Determine the features of the video segment, which is obtained by segmenting the video.

The execution subject of this embodiment of the application may be a cloud or a client. The video segment can be obtained by segmenting the target video to be generated with a video summary by using a specified rule. Wherein, the designated rule may be a time rule. For example, each video segment may be 1 minute or 30 seconds, etc. Alternatively, the specified rules may be shot rules. For example, the switching of each shot of the target video may correspond to a video clip. Alternatively, the specified rule may also be a scenario rule. For example, each scene may correspond to a video clip. A scene may include at least one shot. Where a scene includes multiple shots, the content captured by the multiple shots may be similar. For example, multiple shots are shot at one shooting location, or multiple shots are all shots of one actor, and so on.

The features of video clips can be performed using pre-trained video feature extraction models. Input the video clip into the pre-trained video feature extraction model, and the corresponding features of the video clip can be obtained. This feature can be represented in the form of a vector.

Step S202: Determine the features of the video image frames in the video clip, where the number of video image frames in the video clip is multiple.

Each video segment may include multiple video image frames. The number of video image frames is proportional to the duration of the video segment. The features of a video image frame may include features such as pixel dimensions and text dimensions.

The features of the pixel dimension can be determined by using the pre-trained Chinese multimodal pre-training model. In the Chinese multimodal pre-training model, an image encoder and a text encoder are included. Using an image encoder, the features of a video image frame can be determined directly from the pixel dimension. The characteristics of the text dimension can be determined using the video description generation algorithm and the Chinese multimodal pre-trained model. The text description of the specified video image frame in the video segment can be obtained by using the video description generation algorithm. Text encoders can be used to obtain text dimension features based on text representations.

The specified video image frame may be all video image frames of the video segment, or may be image frames obtained at a certain time interval or at a certain number of frames. For example, one video image frame may be acquired every 0.1 second. For another example, an odd number or an even number of video image frames may be used. For another example, it is also possible to acquire a video image frame every 5 frames and the like.

Step S203: Determine the characteristics of the attention information, which is used to represent the attention to the video in different dimensions.

The attention information may be input by different users or preference information obtained by collecting data from different users, and is used to represent attention to videos in different dimensions. Taking a football game video between team A and team B as an example, user A is a fan of team A, and user B is a fan of player C of team B. Then user A's attention information may be the wonderful performance of team A or the mistakes of team B. The attention information of user B may be the wonderful performance of player C. Taking the video of a gourmet restaurant as an example, user A’s attention information can be an introduction to the types of dishes. For example, the restaurant’s signature dish is roasted pork. How to choose meat and how to marinate meat is the focus of user A’s attention. The attention information of the user B may be the address and environment of the store. For example, it can be the business district where the store is located, whether the store is an online celebrity store, whether it is a store on the XX list, etc.

The features of the attention information can be performed using the pre-trained Chinese multimodal pre-training model. When the received preference information is speech data, the speech data can be converted into text information first, and then the text encoder included in the Chinese multimodal pre-training model can be used to determine the characteristics of the concerned information. In the case that the received preference information is text information, the text encoder contained in the Chinese multimodal pre-training model can be directly used to determine the characteristics of the attention information.

Step S204: Using the features of the video segment, the features of the video image frame in the video segment, and the features of the attention information to generate a video summary of the video.

Starting from two dimensions of each video image frame and the video segment corresponding to each video image frame, one or more video image frames that are highly correlated with the attention information are determined. The so-called high correlation degree may be a video image frame whose correlation degree is not lower than a corresponding threshold. A pre-trained scoring model can be used. The input of the model is the feature of the video clip, the feature of the video image frame in the video clip and the feature of the attention information, and the output is the score of each video image frame.

If multiple video image frames are screened out according to the scores, it may be further determined whether each video image frame corresponds to the same video segment. If corresponding to the same video segment, the determined unique video segment may be used to generate the video summary. If corresponding to multiple video clips, the multiple video clips may be used to generate the video summary. For example, there are 3 video image frames corresponding to the first video segment, and 7 image frames corresponding to the second video segment. Since the number of video image frames of the first video segment is relatively small, only the second video segment may be used to generate the video summary. Alternatively, the video abstract may be generated by directly splicing the first video segment and the second video segment. Alternatively, the video summary may also be generated by using a partially continuous first video segment including 3 video image frames and a partially continuous second video segment including 7 image frames. For example, the partially continuous first video segment containing 3 video image frames may be a video segment composed of a plurality of forward adjacent video image frames and a plurality of backward adjacent video image frames of each of the three video image frames .

Through the above process, the self-adaptive generation of video summaries for content with different user preferences is realized by focusing on information. The same set of models is realized to meet the needs of different users. At the same time, in the process of matching video clips, you can refer to the features of video clips, video image frame features, and features of attention information. Multi-dimensional feature association matching can obtain higher-quality video summary results.

As shown in Figure 3, in a possible implementation, the method of determining the video segment involved in step S201 may include:

Step S301: Divide the video into multiple shot segments according to the scene switching of the video.

During video capture, there may be multiple capture shots. The final video can be obtained by editing, splicing, and other operations of the content captured by multiple shooting shots. For example, in a rival scene between two actors, the first shot may be a shot of actor A, and the second shot may be a shot of actor B. In the video, there may be switching back and forth between multiple lens images. Based on this, according to the camera switching situation of the video, each switching of the camera lens can be used as a segmentation instruction of the camera segment to realize the segmentation of the video. Slicing a video into multiple shot segments according to the shot switching situation can be realized by using a video shot cutting algorithm.

Step S302: Take the shot segment as the determined video segment.

According to the camera switching situation, the video can be divided into multiple camera clips. Based on this, each shot segment can be used as a determined video segment.

As shown in Figure 4, in a possible implementation, the method of determining the video segment involved in step S201 may also include:

Step S401: According to the similarity of the contents of the shot clips, merge the shot clips to obtain at least one content clip.

Similar Shots can be multiple Shots in the same scene. The same scene may include the same shooting location, the same or similar shooting style, or the same or similar shooting content. Taking the gourmet restaurant exploration program as an example, the exterior shots before entering the gourmet restaurant can be classified as the first scene. Shots shot in the dining area of the gourmet restaurant can be classified as the second scene. Shots shot in the back kitchen of the gourmet restaurant can be classified as the third scene. Taking film and television dramas with different themes as an example, the scenes of literary dramas can be classified as the first scene, and the scenes of martial arts can be classified as the second scene. Each content segment may consist of at least one shot segment.

The content clip may be obtained by merging multiple shot clips after the shot clip is determined. In addition, the content segment can also be implemented by using a video scene cutting algorithm.

Step S402: Use the content segment as the determined video segment; or, use the content segment and the shot segment as the determined video segment at the same time.

In one case, the content segment may be used as the determined video segment. The beneficial effect is that the amount of data can be effectively reduced. In another case, the content clip and the shot clip can be used as the determined video clip at the same time. The beneficial effect is that the features of the video can be marked from more dimensions.

As shown in FIG. 5, in a possible implementation manner, the use of the features of the video clip, the features of the video image frame in the video clip and the features of the attention information involved in step S204 to generate a video summary of the video may include:

Step S501: Determine at least one key frame in the video image frame by using the first degree of association between the feature of the video segment and the feature of the information of interest, and the second degree of association between the feature of the video image frame and the feature of the information of interest.

The characteristics of the attention information can be used as reference information. Therefore, based on the multi-dimensional features such as the features of the video image frame and the features of the video clips, the features of the attention information are matched, and a key frame with a high degree of matching with the attention information is determined. The so-called high degree of matching with features of the information of interest may be video image frames whose matching degree is not lower than a corresponding threshold.

For example, the first correlation degree between the features of the video clip and the features of the attention information may be determined first. Secondly, a second degree of association between the feature of the video image frame and the feature of the information of interest may be determined. Finally, the score of each video image frame can be determined based on the first degree of association and the second degree of association. The video image frame with the highest score can be used as a key frame. It can also be sorted by the score from high to low, and select a specified number of video image frames as key frames.

Step S502: Generate a video summary of the video according to the video segment corresponding to the key frame.

The video summary can be obtained after corresponding processing is performed on the video segment corresponding to the key frame according to the duration requirement of the video summary. For example, according to the duration requirement, the video segment corresponding to the key frame may be subjected to frame extraction processing, compression processing, etc., to obtain a video summary meeting the duration requirement.

As shown in FIG. 6, in a possible implementation manner, generating a video summary of the video according to the video segment corresponding to the key frame involved in step S502 may include:

Step S601: Obtain the duration of the video segment corresponding to the key frame.

It may first be judged whether the video segments corresponding to each key frame are the same. In the case that there are at least two video clips, the duration of each video clip needs to be obtained separately.

Step S602: According to the determined score of the key frame and the duration of the video segment corresponding to the key frame, filter the video segment corresponding to the key frame to obtain a screening result.

In the case that there are at least two video clips, at least one video clip can be screened out by taking the score of the key frame in each video clip and the duration of the video clip as reference information at the same time. For example, the knapsack algorithm can be used to select a video segment with a suitable duration as the screening result according to the score of the key frame.

Step S603: Generate a video summary of the video by using the filtering result.

After the screening results are obtained, video summaries can be generated by direct splicing. Or, it is also possible to firstly adjust the duration of the video clips in the screening result, and use the duration-adjusted video clips to generate a video summary. Duration adjustments can be made based on keyframe ratings. For example, video clips with relatively high-scoring keyframes can be adjusted relatively little. The video segment where the key frame with a relatively low score is located can be adjusted relatively more.

As shown in FIG. 7 , in a possible implementation manner, when the duration corresponding to the screening result exceeds the corresponding threshold, the generation of the video summary of the video according to the video segment corresponding to the key frame involved in step S502 may include :

Step S701: Based on the features of the video image frames in the video segment corresponding to the key frame, filter the video image frames in the video segment corresponding to the key frame.

The duration requirement of the video summary may be determined first. If the duration requirement of the video summary is 1 minute, and there are multiple video clips in the filtering result, and the total duration of the multiple video clips exceeds 1 minute, the video image frames in the multiple video clips can be filtered. For example, video image frames in multiple video clips may be compared for similarity based on previously determined features of video image frames. When the difference between the video image frames participating in the comparison is not greater than the difference threshold, it can be considered that the video image frames participating in the comparison are similar. Further, the video image frame is filtered according to the comparison result. For example, if there are 10 consecutive video image frames that are all similar, then the feature mean value of the 10 video image frames may be calculated. Then compare the 10 video image frames with the feature mean value respectively, and keep the video image frame with the largest difference from the feature mean value, or keep the video image frame with the difference exceeding the corresponding threshold. In addition, it is also possible to directly select the first frame, the last frame and the middle frame as the reserved video image frames.

Step S702: Using the filtered video image frames to generate a video summary of the video.

Since the purpose of the filtering process is to perform time compression, the number of video image frames retained after filtering meets the requirements. That is, the duration of the video abstract obtained by combining the filtered video image frames can meet the duration requirement of the video abstract.

As shown in FIG. 8, in a possible implementation manner, the generating manner of paying attention to information may include:

Step S801: Determine the instruction content and reference content from the obtained video summary generation instruction; the instruction content is generated by using at least one of voice data and action data; the reference content includes at least one of text, video or image.

The acquired video summary generation instruction may be information input by the user or may be information collected by the execution subject. Exemplarily, the acquired or collected information may be that the user is holding a (text) image, or a (handholding) video playback device that is playing a video, etc., and the user points to a certain part of the image and says "generated with this character Video clips with similar status", the user points to or circles a certain text in the image and says "I need such content", points to a certain screen and says "I want a video clip like this", etc. In the above scenario, the collected finger movements, circled actions, and user voice can all be used as instruction content, while the image, text or video being played by the user can be used as reference content.

Based on this, the acquired video summary generation instruction can be parsed. For the user's voice data and action data, it can be directly determined as the instruction content. For the item held by the user, the content pointed to, the content circled, etc., it may be determined as the reference content. The above-mentioned analysis can be determined by using a semantic model, and the instruction content and reference content can be determined by performing semantic recognition on the collected content.

After the instruction content and reference content are determined, the user can also be identified based on the instruction content. For example, the user is identified by information such as image or sound, so as to set an identification for the user. During the subsequent presentation of the video abstract, the user may be identified based on the user's video abstract display instruction. If the recognition result is a user who has output attention information, the video summary generated by the attention information corresponding to the user may be directly displayed.

Step S802: Use the instruction content to determine the attention information in the reference content.

The content of the instruction can be identified to determine the intent of the instruction. For example, in the case that the content of the instruction is voice data, voice recognition technology may be used to determine the intent of the instruction. When the content of the instruction is action data, the action can be identified to determine the intent of the instruction. Wherein, the action may include the action of pointing, the action of circling in the foregoing example, and the like. In this case, the position pointed by the finger, or the content corresponding to the position selected by the circle can be used as the selected reference content. For example, a person in the image is selected, or a building is selected, or a piece of text is circled. The selected people, buildings, or circled text in the image can be used as attention information. Furthermore, the features of the attention information can be determined through an image encoder, a text encoder, and the like.

As shown in FIG. 9, in a possible implementation manner, when the instruction content includes voice data and action data, the use of the instruction content involved in step S802 to determine the attention information in the reference content may include:

Step S901: Determine the first moment when voice data appears and the second moment when motion data appears.

When the instruction content is determined, the time when the instruction content appears can also be recorded. By analyzing the content of the instruction, the voice data and the motion data can be distinguished. The parsing principle can be realized by adopting the existing technology, and will not be repeated here. For example, if the user circles the target while talking in front of the video being played, the first moment when the voice data appears and the second moment when the action data appears can be determined correspondingly. In the current implementation manner, the voice data and action data may be contents representing instruction meanings.

Step S902: Using the first time point and the second time point, correlate the voice data with the action data, and obtain a correlation result.

The time difference threshold can be set in advance. If the time difference between the first moment and the second moment is not greater than the time difference threshold, it can be determined that the voice data and the action data occur simultaneously. That is, simultaneous occurrence may be that voice data and motion data overlap in the time dimension. Based on this, simultaneous voice data and action data can be associated to obtain an association result. The expression form of the association result can be: {start time t ₁ , end time t ₂ , voice data, action data}, or the expression form of the association result can be: {start time t ₁ , end time t ₂ , voice data}, {start time t ₁ , end time t ₃ , action data}. In the current example, the first time period from time _t1 to time _t2 overlaps with the second time period from time _t1 to time _t3 . It is not difficult to understand that for the instruction content that does not overlap in time, it can be recorded sequentially based on time sequence.

Step S903: Using the association result, determine the scope of the reference content; the scope includes at least one of the duration of the video, the number of pages of the image, and the effective content of the image.

As in the foregoing example, if the start time of the association result is t ₁ , then the reference content at time t ₁ may be recorded. When the reference content is a video, the video frame at time _t1 can be recorded, so that the time _t1 can be used as the starting time of the video. In order to continue the picture, it is also possible to use 1 second before time _t1 or 2 seconds before time _t1 as the start time of the video.

During the first period from time _t1 to time _t2 , or within the second period from time _t1 to time _t3 , the instruction content may be analyzed. For example, voice content includes things like "I need a composition like this part" or "I want a great performance from this player". The action content includes actions such as circle selection and fingering in the playing video. Based on this, the time tags of the voice data and action data can be used to add the same time tag to the content displayed in the corresponding time period of the reference content. The content displayed in the corresponding time period may be a video segment of the corresponding time length, a single or multiple images, and the like. Alternatively, the content displayed in the corresponding time period may also be the content pointed to by the voice content and/or the content pointed to by the action content within the time period. If the voice content includes "I need the same composition as this part" and "I want this player's wonderful performance", then the composition method of the video or image is valid content, and the player in the image is valid content.

Step S904: Use the scope of the reference content to determine the attention information in the reference content.

In the current implementation manner, the expression form of the concerned information may be text. After determining the scope of the reference content, if the reference content is a video or an image, the textual expression of the video or image can be obtained by using a video description generation algorithm. If the reference content is text, the text expression can be used directly.

As shown in Figure 10, in a possible implementation, when the execution subject is the client, it may also include:

Step S1001: Associating video summaries with videos.

According to different attention information, a video summary corresponding to the attention information can be obtained. As before, the attention information can be clothing collocations and food clips. In this regard, for the same target video, the first video summary related to clothing matching and the second video summary related to food can be obtained. For the two video summaries, both may be associated with the target video, to represent that the two video summaries are obtained from the target video.

Step S1002: In the case of receiving the video abstract display instruction, the video preview window of the video displays the video abstract.

The video summary display instruction may be in the form of speech or action. For example, the voice form could be "Play the video summary of XX video". The form of the action may be the video of the summary to be played selected through the action. Further, it is also possible to identify the user who issued the video summary display instruction. For example, the user who issued the video summary display instruction may be determined by means of image recognition or voice recognition. Furthermore, it can be judged whether there is already concerned information matching the user. If it exists, a video abstract corresponding to the attention information may be displayed in a video preview window of the video. On the contrary, if it does not exist, you can select a more popular video abstract for display.

The method for generating the above-mentioned video summary can be a local application program (APP) of the user terminal, or a functional module in an APP, or a service provided by the cloud. The information is uploaded to the cloud, and the results fed back by the cloud, such as video summaries of videos, are received. In addition, the cloud can obtain video-related content from the video database.

Exemplarily, several distributed computing nodes may be deployed on the cloud, and each computing node has processing resources such as computing and storage. In the cloud, multiple computing nodes can be organized to provide one or several services in the card identification method; or, multiple computing nodes can be organized to provide one or several of the video summary generation methods Serve. Exemplarily, the service may include segmenting the video to obtain video clips; determining the features of the video clips; determining the features of the video image frames in the video clips; determining the features of the concerned information; Features and features that focus on information, generate video summaries for videos, etc. Of course, a computing node can also provide one or more services. The way the cloud provides the service may be to provide a service interface externally, and the user invokes the service interface to use the corresponding service.

For the solutions provided by the embodiments of the present invention, the cloud may provide a service interface for information identification services, which is called a target service interface. When the user needs to view the video summary, the user device invokes the target service interface to trigger a request to the cloud for invoking the target service interface, and the request carries attention information. The cloud determines the computing node that responds to the request, and uses the processing resources in the computing node to execute the steps provided in the embodiments of the present application.

An embodiment of the present application provides a method for generating a video abstract, as shown in FIG. 11 , which is a flowchart of a method for generating a video abstract in an embodiment of the present application, which may include:

Step S1101: Send the received attention information on the video to the generator of the video summary.

The execution subject of this embodiment of the application may be a client. The attention information may be the video preference information sent by the user. The scenarios for sending attention information may include video preview scenarios, video editing scenarios, video clip search scenarios, and the like. The form of the preference information may be text information, audio information or video information, etc. The user's preference information is used to represent the user's preference for the target video in different dimensions. Exemplarily, the attention information may be that the user is holding a (text) image, or a (handheld) video playback device that is playing a video, etc., and the user points to a certain part of the image and says, "Generate a video similar to this character's state." Clip", the user points to or circles a certain text in the image and says "I need such content", points to a certain screen and says "I want a video clip like this", etc.

Step S1102: Receive the video summary of the video generated by the generating end of the video summary in response to the attention information; the video video summary is generated by the generation terminal of the video summary using the characteristics of the video clip, the characteristics of the video image frame in the video clip and the characteristics of the attention information ; The video segment is obtained by segmenting the video.

The generating end of the video summary is used to generate a video summary of the video based on attention information on the video. During the generation of the video summary, the features of the video clip and the features of the video image frames in the video clip can also be referred to at the same time.

The video segment can be obtained by segmenting the video to be generated by using specified rules. The features of video clips can be performed using pre-trained video feature extraction models. Input the video clip into the pre-trained video feature extraction model, and the corresponding features of the video clip can be obtained. This feature can be represented in the form of a vector.

Each video segment may include multiple video image frames. The number of video image frames is proportional to the duration of the video segment. The features of a video image frame may include features such as pixel dimensions and text dimensions.

The features of the pixel dimension can be determined by using the pre-trained Chinese multimodal pre-training model. In the Chinese multimodal pre-training model, an image encoder and a text encoder are included. Using an image encoder, the features of a video image frame can be determined directly from the pixel dimension. The characteristics of the text dimension can be determined using the video description generation algorithm and the Chinese multimodal pre-trained model. The text description of the specified video image frame in the video segment can be obtained by using the video description generation algorithm. Text encoders can be used to obtain text dimension features based on text representations.

The features of the attention information can be performed using the pre-trained Chinese multimodal pre-training model. In the case that the received attention information is speech data, the speech data can be converted into text information first, and then the text encoder included in the Chinese multimodal pre-training model can be used to determine the characteristics of the attention information. In the case that the received attention information is text information, the text encoder included in the Chinese multimodal pre-training model can be directly used to determine the characteristics of the attention information. When the received attention information is action information, the action information can be converted into text information based on the action recognition technology, and then the text encoder included in the Chinese multimodal pre-training model can be used to determine the characteristics of the attention information.

Using a pre-trained scoring model, the input of the model is the features of the video clip, the features of the video image frames in the video clip and the features of the attention information, and the output is the score of each video image frame. According to the score of the video image frame and the duration of the video segment of the video image frame, the target video segment can be determined by using the knapsack algorithm. That is, the generator of the video summary responds to the video summary of the video generated by the concerned information. The generator of the video summary sends the video summary to the client. For the functions and specific workflow of the generation end of the video summary, refer to the corresponding description in the above method, and have corresponding beneficial effects, and will not be repeated here.

Step S1103: displaying the video summary of the video in the video preview window.

After receiving the video summary sent by the video summary generator in response to the attention information, the client can display the video summary in the video preview window.

By paying attention to information, the video summarization is adaptively generated for different user preferences. The same set of models is realized to meet the needs of different users.

An embodiment of the present application provides a method for generating a video abstract, as shown in FIG. 12 , which is a flowchart of a method for generating a video abstract in an embodiment of the present application, which may include:

其中j可以用于表示图像帧的序号。Use the image feature extraction model to determine the pixel dimension features of the image frame in the target video

Where j can be used to represent the sequence number of the image frame.

Using a video description generation algorithm, determine the textual representation T of the image frame in the target video.

像素维度特征

和文字表示T的特征

可以共同作为图像特征。Use the text feature extraction model to obtain the features of the text representation T of the image frame

Pixel Dimension Features

and literals denote the characteristics of T

can be used together as image features.

The attention information may be user preference information. After converting the attention information into text, use the text feature extraction model to focus on the characteristics of the information

其中i可以用于表示镜头片段的序号。Using the video shot cutting algorithm, the target video can be divided into multiple shot segments, and the duration of the shot segment can be expressed as T _shot . Using the video feature extraction model, the features of the shot clips can be obtained

Where i can be used to indicate the serial number of the shot clip.

其中q可以用于表示场景片段的序号。Using the video scene cutting algorithm, the target video can be divided into multiple scene segments, and the scene segments can include at least one shot segment. The duration of a scene segment can be expressed as T _sence . Using the video feature extraction model, the features of scene clips can be obtained

Among them, q can be used to indicate the sequence number of the scene segment.

文字表示T的特征

关注信息的特征

镜头片段的特征

场景片段的特征

输入预先训练好的评分模型，可以得到第j帧图像帧的评分Sj。预先训练好的评分模型可以是基于翻译模型(Transformer)结构的神经网络模型。该神经网络模型可以包括输入层、编码层和输出层。输入层可以是各项特征。编码层可以采用翻译模型结构的编码器。翻译模型结构具有自注意力机制(self-attention)，自注意力机制可以直接建模任意两个特征之间的相互影响。输出层可以基于编码层的结果，可以对每个图像帧生成一个分数。基于分数进行排序，即可得到第j帧图像帧的评分Sj。对于评分较高的一帧图像或者指定数量的多帧图像，可以用于视频摘要的生成。The pixel dimension feature

The text indicates the characteristics of T

Focus on the characteristics of the information

Characteristics of Shots

Characteristics of Scene Clips

Input the pre-trained scoring model to get the score Sj of the jth image frame. The pre-trained scoring model may be a neural network model based on a translation model (Transformer) structure. The neural network model may include an input layer, an encoding layer and an output layer. The input layer can be various features. The encoding layer can adopt the encoder of the translation model structure. The translation model structure has a self-attention mechanism, which can directly model the interaction between any two features. The output layer can be based on the results of the encoding layer, which can generate a score for each image frame. Sorting based on the scores can obtain the score Sj of the jth image frame. For a frame of image with a higher score or a specified number of multiple frames of images, it can be used to generate video summaries.

Using the knapsack algorithm, based on the score Sj of the image frame and the duration T _shot of the shot segment, a shot segment suitable as a video summary can be obtained. Finally, a video summary is generated based on the duration requirement of the video summary.

Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application further provides an apparatus for generating a video summary. As shown in Figure 13, it is a structural block diagram of a device for generating a video summary according to an embodiment of the present application. The device for generating a video summary may include:

The video feature determination module 1301 is configured to determine the feature of the video segment, and the video segment is obtained by segmenting the video.

The image feature determining module 1302 is configured to determine features of video image frames in the video segment, and the number of video image frames in each video segment is multiple.

The attention information determination module 1303 is configured to determine the characteristics of the attention information, and the attention information is used to represent the attention to the video in different dimensions.

The video summary generation module 1304 is configured to generate a video summary of the video by using the features of the video clip, the features of the video image frame in the video clip and the features of the attention information.

In a possible implementation, the video feature determination module 1301 may include:

The cutting sub-module is used to cut the video into multiple shot segments according to the camera switching situation of the video.

The video segment determination sub-module is used to use the shot segment as the determined video segment.

In a possible implementation, the video feature determination module 1301 may also include:

A content segment determining submodule, configured to merge the shot segments according to the similarity of the contents of the shot segments to obtain at least one content segment;

The video segment determining submodule is also used to use the content segment as the determined video segment; or,

The content segment and the lens segment are simultaneously used as determined video segments.

In a possible implementation, the video summary generating module 1304 may include:

The key frame determination submodule is used to determine at least one key frame in the video image frame by using the first degree of association between the feature of the video segment and the feature of the information of interest, and the second degree of association between the feature of the video image frame and the feature of the information of interest frame;

The video summary generation execution sub-module is configured to generate a video summary of the video according to the video segment corresponding to the key frame.

In a possible implementation, the execution submodule of video summary generation may include:

The duration acquisition unit is used to acquire the duration of the video segment corresponding to the key frame;

A filtering result determination unit is used to filter the video clips corresponding to the key frames according to the determined scoring of the key frames and the duration of the video clips corresponding to the key frames to obtain a screening result;

The video summary generation unit is used to generate a video summary of the video by using the filtering result.

In a possible implementation, when the duration of the video segment corresponding to the key frame exceeds the corresponding threshold, the execution submodule of video summary generation may include:

A filter unit, configured to filter the video image frames in the video segment corresponding to the key frame based on the features of the video image frame in the video segment corresponding to the key frame;

The video summary generation unit is used to generate a video summary of the video using the filtered video image frame.

In a possible implementation manner, the concerned information determining module 1303 may include:

The content acquisition sub-module is used to determine the instruction content and reference content from the obtained video summary generation instruction; the instruction content is generated by using at least one of voice data and action data; the reference content includes text, video or image at least one;

The attention information determination execution submodule is used to determine the attention information in the reference content by using the instruction content.

In a possible implementation, when the instruction content includes voice data and action data, the attention information determination execution submodule may include:

A time determination unit is used to determine the first moment when the voice data appears and the second moment when the action data appears;

An associating unit, configured to use the first moment and the second moment to associate the speech data with the action data to obtain an association result;

A range determination unit, configured to determine the range of the reference content by using the association result; the range includes at least one of the duration of the video, the number of pages of the image, and the effective content of the image;

The attention information determining unit is configured to determine the attention information in the reference content by using the scope of the reference content.

In a possible implementation manner, a display module is also included, and the display module may include:

an associating subunit for associating the video abstract with the video;

The display execution subunit is configured to display the video summary in the video preview window of the video when a video summary display instruction is received.

Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application further provides an apparatus for generating a video summary. As shown in FIG. 14, it is a structural block diagram of a device for generating a video summary according to an embodiment of the present application. The device for generating a video summary may include:

The attention information sending module 1401 is used to send the received attention information to the video to the generation end of the video summary;

Video summary acquisition module 1402, for receiving the video summary of the video generated by the generating end of the video summary in response to the attention information; The features of the information are generated; the video clip is obtained by segmenting the video;

The video abstract display module 1403 is configured to display the video abstract of the video in the video preview window.

The functions of each module in each device in the embodiment of the present application can refer to the corresponding description in the above method, and have corresponding beneficial effects, and will not be repeated here.

FIG. 15 is a block diagram of an electronic device used to implement an embodiment of the present application. As shown in FIG. 15 , the electronic device includes: a memory 1510 and a processor 1520 , and the memory 1510 stores computer programs that can run on the processor 1520 . The processor 1520 implements the methods in the foregoing embodiments when executing the computer program. The number of memory 1510 and processor 1520 may be one or more.

This electronic device also includes:

The communication interface 1530 is used to communicate with external devices for interactive data transmission.

If the memory 1510, the processor 1520, and the communication interface 1530 are implemented independently, the memory 1510, the processor 1520, and the communication interface 1530 may be connected to each other through a bus to complete mutual communication. The bus may be an Industry Standard Architecture (Industry Standard Architecture, ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 15 , but it does not mean that there is only one bus or one type of bus.

Optionally, in a specific implementation, if the memory 1510, the processor 1520, and the communication interface 1530 are integrated on one chip, the memory 1510, the processor 1520, and the communication interface 1530 may communicate with each other through the internal interface.

The embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and implements the method provided in the embodiment of the present application when the program is executed by a processor.

The embodiment of the present application also provides a chip, the chip includes a processor, configured to call and execute instructions stored in the memory from the memory, so that the communication device installed with the chip executes the method provided in the embodiment of the present application.

The embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor, and a memory, the input interface, the output interface, the processor, and the memory are connected through an internal connection path, and the processor is used to execute the code in the memory , when the code is executed, the processor is used to execute the method provided by the embodiment of the application.

It should be understood that the above-mentioned processor may be a central processing unit (Central Processing Unit, CPU), and may also be other general-purpose processors, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field Programmable Gate Array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or any conventional processor or the like. It should be noted that the processor may be a processor supporting Advanced RISC Machines (ARM) architecture.

Further, optionally, the foregoing memory may include a read-only memory and a random access memory. The memory can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the non-volatile memory can include read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically programmable Erase Programmable Read-Only Memory (Electrically EPROM, EEPROM) or Flash. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of RAM are available. For example, Static Random Access Memory (Static RAM, SRAM), Dynamic Random Access Memory (Dynamic Random Access Memory, DRAM), Synchronous Dynamic Random Access Memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM) , DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced SDRAM, ESDRAM), Synchronous Link Dynamic Random Access Memory (Sync link DRAM, SLDRAM) and Direct Memory Bus Random Access Memory (Direct RambusRAM, DR RAM).

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the present application are produced in whole or in part. A computer can be a general purpose computer, special purpose computer, computer network, or other programmable device. Computer instructions may be stored in, or transmitted from, one computer-readable storage medium to another computer-readable storage medium.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present application. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "plurality" means two or more, unless otherwise specifically defined.

Any process or method described in a flowchart or otherwise described herein may be understood as representing a module, segment, or code comprising one or more executable instructions for implementing a specific logical function or step of the process part. Also, the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order as the functions involved are involved.

The logic and/or steps described in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium , for use with an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute instructions), or in conjunction with such an instruction execution system, device or equipment.

It should be understood that each part of the present application may be realized by hardware, software, firmware or a combination thereof. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the method in the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or its combination.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the above-mentioned integrated modules are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above is only an exemplary embodiment of the application, but the scope of protection of the application is not limited thereto, and any skilled person familiar with the technical field can easily think of its various changes within the technical scope of the application Or replacement, all of these should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (14)

1. A method for video summary generation, comprising:

determining the characteristics of a video clip, wherein the video clip is obtained by segmenting a video;

determining the characteristics of video image frames in the video segment, wherein the number of the video image frames in the video segment is multiple;

determining the characteristics of attention information, wherein the attention information is used for representing attention conditions of the video in different dimensions;

and generating a video abstract of the video by using the characteristics of the video clip, the characteristics of the video image frame in the video clip and the characteristics of the attention information.

2. The method of claim 1, wherein determining the video segment comprises:

according to the shot switching condition of the video, cutting the video into a plurality of shot segments;

and taking the shot segment as the determined video segment.

3. The method of claim 2, wherein determining the video segment further comprises:

according to the similarity condition of the contents of the shot segments, merging the shot segments to obtain at least one content segment;

taking the content segment as the determined video segment; or the like, or, alternatively,

and simultaneously taking the content segment and the shot segment as the determined video segment.

4. The method of claim 1, wherein the generating a video summary of the video using the features of the video segments, the features of the video image frames, and the features of the information of interest comprises:

determining at least one key frame in the video image frame by utilizing a first correlation degree of the characteristics of the video segment and the characteristics of the attention information and a second correlation degree of the characteristics of the video image frame and the characteristics of the attention information;

and generating a video abstract of the video according to the video segment corresponding to the key frame.

5. The method according to claim 4, wherein the generating the video summary of the video according to the video segment corresponding to the key frame comprises:

acquiring the duration of a video clip corresponding to the key frame;

screening the video clips corresponding to the key frames according to the determined scores of the key frames and the duration of the video clips corresponding to the key frames to obtain a screening result;

and generating a video abstract of the video by using the screening result.

6. The method according to claim 4 or 5, wherein the generating the video summary of the video according to the video segment corresponding to the key frame when the duration of the video segment corresponding to the key frame exceeds the corresponding threshold value comprises:

based on the characteristics of the video image frames in the video clips corresponding to the key frames, filtering the video image frames in the video clips corresponding to the key frames;

and generating a video abstract of the video by using the video image frame after the filtering processing.

7. The method according to claim 1, wherein the manner of generating the attention information includes:

determining instruction content and reference content from the acquired video abstract generating instruction; the instruction content is generated by at least one of voice data and motion data; the reference content comprises at least one of characters, videos or images;

and determining attention information in the reference content by using the instruction content.

8. The method according to claim 7, wherein in a case where the instruction content includes voice data and motion data, the determining, with the instruction content, attention information in the reference content includes:

determining a first time instant at which the speech data occurs and a second time instant at which the motion data occurs;

associating the voice data with the action data by using the first time and the second time to obtain an association result;

determining a range of the reference content using the correlation result; the range comprises at least one of video duration, image page number and effective image content;

and determining attention information in the reference content by using the range of the reference content.

9. The method of any of claims 1 to 8, further comprising:

associating the video summary with the video;

and under the condition that a video abstract display instruction is received, displaying the video abstract in a video preview window of the video.

10. A method for video summary generation, comprising:

sending the received attention information to the video to a video abstract generating end;

receiving the video abstract of the video generated by the generation end of the video abstract responding to the attention information; the video abstract of the video is generated by a generation end of the video abstract by utilizing the characteristics of a video segment, the characteristics of a video image frame in the video segment and the characteristics of the attention information; the video clip is obtained by segmenting the video;

and displaying the video abstract of the video in a video preview window.

11. An apparatus for video summary generation, comprising:

the video characteristic determining module is used for determining the characteristics of a video clip, wherein the video clip is obtained by segmenting a video;

the image characteristic determining module is used for determining the characteristics of video image frames in the video segments, and the number of the video image frames in each video segment is multiple;

the attention information determining module is used for determining the characteristics of attention information, wherein the attention information is used for representing attention conditions of the video in different dimensions;

and the video abstract generating module is used for generating a video abstract of the video by utilizing the characteristics of the video clip, the characteristics of the video image frame in the video clip and the characteristics of the attention information.

12. An apparatus for video summary generation, comprising:

the attention information sending module is used for sending the received attention information of the video to a video abstract generating end;

the video abstract acquisition module is used for receiving a video abstract of the video generated by the generation end of the video abstract responding to the attention information; the video abstract of the video is generated by a generation end of the video abstract by utilizing the characteristics of a video segment, the characteristics of a video image frame in the video segment and the characteristics of the attention information; the video clip is obtained by segmenting the video;

and the video abstract display module is used for displaying the video abstract of the video in a video preview window.

13. An electronic device comprising a memory, a processor and a computer program stored on the memory, the processor implementing the method of any one of claims 1-10 when executing the computer program.

14. A computer-readable storage medium, having stored therein a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

Images