CN115396728B - Method, device, electronic device and medium for determining video playback speed - Google Patents

Abstract

The present application discloses a method, device, electronic device and medium for determining video playback speed, which belongs to the field of image technology. The method includes: obtaining first video content feature information corresponding to N video clips in a first video, where N is a positive integer; respectively splicing the user behavior feature information of the target user with the first video content feature information corresponding to the N video clips to obtain the video feature information corresponding to each of the N video clips; determining the first playback speed corresponding to each video clip based on the video feature information corresponding to each video clip; and playing the first video based on the first playback speed.

Description

Method, device, electronic device and medium for determining video playback speed

Technical Field

The present application belongs to the field of artificial intelligence technology, and specifically relates to a method, device, electronic device and medium for determining a video playback speed.

Background Art

Nowadays, there are more and more videos on the Internet, and the video content is also richer. Since users have different interests in different video clips when watching videos, various players on the market currently have the function of multiple-speed playback to meet users' various needs such as fast browsing or slow appreciation.

However, the speed playback function of various current players requires users to actively trigger and adjust the video playback speed during the viewing process. This makes the overall video playback speed adjustment process too cumbersome, requiring users to constantly adjust the video playback speed on their own, thereby reducing the user's video viewing experience.

Summary of the invention

The purpose of the embodiments of the present application is to provide a method, device, electronic device and medium for determining a video playback speed, which can enable the electronic device to adaptively adjust the playback speed of the current video clip while the user is watching the video.

In a first aspect, an embodiment of the present application provides a method for determining a video playback speed, the method comprising: obtaining first video content feature information corresponding to N video clips in a first video, where N is a positive integer; respectively splicing user behavior feature information of a target user with the first video content feature information corresponding to the above N video clips to obtain video feature information corresponding to each of the above N video clips; determining a first playback speed corresponding to each of the above video clips based on the video feature information corresponding to the above each video clip; and playing the first video based on the first playback speed.

In the second aspect, an embodiment of the present application provides a device for determining a video playback speed, and the device for determining a video playback speed includes: an acquisition module and a processing module: the acquisition module is used to acquire N video clips in a first video, N being a positive integer; the acquisition module is also used to acquire video feature information corresponding to each of the above N video clips; the video feature information corresponding to any video clip includes: user behavior feature information of a target user and video content feature information of any video clip; the processing module is used to determine a first playback speed corresponding to each video clip based on the video feature information corresponding to each video clip; the processing module is also used to apply the first playback speed corresponding to each video clip to each corresponding video clip.

In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

In an embodiment of the present application, the first video content feature information corresponding to N video clips in the first video is obtained, where N is a positive integer; and the user behavior feature information of the target user is respectively spliced with the first video content feature information corresponding to the above N video clips to obtain the video feature information corresponding to each of the above N video clips; based on the video feature information corresponding to each of the above video clips, the first playback speed corresponding to each of the above video clips is determined; and then the first video is played based on the first playback speed. In this way, by integrating the user behavior feature information of the target user into the video content feature information of each video clip, the electronic device can set the video playback speed corresponding to each clip in a personalized manner according to the user behavior feature information and the video content feature information, thereby eliminating the need for the user to manually adjust the video playback speed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for determining a video playback speed according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a model of a method for determining a video playback speed provided in an embodiment of the present application;

FIG3 is a second schematic diagram of a model of a method for determining a video playback speed provided by an embodiment of the present application;

FIG4 is a third model schematic diagram of a method for determining a video playback speed provided in an embodiment of the present application;

FIG5 is a second flow chart of a method for determining a video playback speed according to an embodiment of the present application;

FIG6 is a schematic diagram of the structure of a device for determining a video playback speed provided by an embodiment of the present application;

FIG7 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

FIG. 8 is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated with each other are in an "or" relationship.

In conjunction with the accompanying drawings, the following detailed description is given of the method, device, electronic device and readable storage medium for determining the video playback speed provided by the embodiments of the present application through specific embodiments and their application scenarios.

The method for determining the video playback speed provided in the embodiment of the present application can be applied to the video watching scenario.

Nowadays, there are more and more videos on the Internet, and the video content is also richer. Since users have different interests in different video clips when watching videos, various players on the market currently have the function of multiple-speed playback to meet users' various needs such as fast browsing or slow viewing. However, the multiple-speed playback function of various current players requires users to actively trigger and adjust the video playback speed during the viewing process. As a result, the overall video playback speed adjustment process is too cumbersome, requiring users to constantly adjust the video playback speed on their own, thereby reducing the user's experience of watching videos.

For example, the speed playback function of various current players has two adjustment methods. One is that the user actively clicks to set a fixed value (for example, 0.5x/0.75x/1.0x/1.5x/2.0x), and plays the current video clip at the speed corresponding to the fixed value. The other is that the user long presses the screen to trigger a fixed value (for example, 3.0x) during the long press of the screen, and plays the current video clip at the speed corresponding to the fixed value.

It can be seen that the current scheme for determining video speed playback is too dependent on active user input. Both methods have obvious defects. Each time, only a fixed playback speed can be set through user input, and it is impossible to automatically adjust in time for different clips in the video.

In summary, how to solve the problem that the user can determine the playback speed of the current video clip without manual triggering during the viewing process is a technical problem that needs to be solved urgently in this application.

In the method for determining the video playback speed provided in the embodiment of the present application, the first video content feature information corresponding to the N video clips in the first video is obtained, where N is a positive integer; and the user behavior feature information of the target user is respectively spliced with the first video content feature information corresponding to the above N video clips to obtain the video feature information corresponding to each of the above N video clips; based on the video feature information corresponding to each of the above video clips, the first playback speed corresponding to each of the above video clips is determined; and then the first video is played based on the first playback speed. In this way, by integrating the user behavior feature information of the target user into the video content feature information of each video clip, the electronic device can set the video playback speed corresponding to each clip in a personalized manner according to the user behavior feature information and the video content feature information, thereby eliminating the need for the user to manually adjust the video playback speed.

The execution subject of the method for determining the video playback speed provided in the embodiment of the present application may be a device for determining the video playback speed, and the device for determining the video playback speed may be an electronic device or a functional module in the electronic device.

The embodiment of the present application provides a method for determining a video playback speed. FIG1 shows a flowchart of a method for determining a video playback speed provided by the embodiment of the present application. The method can be applied to electronic devices. As shown in FIG1, the method for determining a video playback speed provided by the embodiment of the present application can include the following steps 201 to 204.

Step 201: Obtain first video content feature information corresponding to N video clips in a first video.

Wherein, N is a positive integer.

In the embodiment of the present application, the first video may be a video in any application in the electronic device, or a video locally stored in the electronic device.

In an embodiment of the present application, the electronic device may regularly or randomly split the first video into N video segments according to a predetermined number of segments.

In an embodiment of the present application, the electronic device may input the acquired first video into a video segmentation model for segmentation processing, thereby obtaining N video clips.

In the embodiment of the present application, the above-mentioned video segmentation model is a trained video segmentation model.

Exemplarily, after the electronic device inputs the first video with a duration of T into the video segmentation model, it can sample and obtain N _f =T/I frame images according to the set video sampling interval I, and synthesize the N _f frame images according to the rule that the number of frames per segment is M, to obtain N _s =N _f /M video segments.

It should be noted that each frame of the video clip finally obtained is an RGB image of size H×W×3 obtained by Resize and Crop processing.

Optionally, in the embodiment of the present application, the process of the above step 201 of "obtaining first video content feature information corresponding to N video clips in the first video" includes the following steps 201a to 201d:

Step 201a: for one frame of the N video segments, divide the video image in the video segment to obtain X image blocks.

Step 201b: Obtain image feature information corresponding to the X image blocks.

Step 201c: input the image feature information corresponding to the X image blocks into a video content feature extraction model for feature extraction to obtain second video content feature information corresponding to the X image blocks.

Step 201d: performing feature fusion on the second video content feature information corresponding to the X image blocks to obtain the first video content feature information.

Exemplarily, one image block corresponds to one first key video content feature information.

Exemplarily, as shown in Figure 2, the video content feature extraction model M _video can be composed of a basic structure repeated N times, wherein the basic structure is composed of a multi-head attention module (Multi-Head Attention), a residual and normalization module (Add&Norm), and a feedforward network module (Feed Forward), wherein the Multi-Head Attention is composed of 12 self-attention modules (Self-Attention).

Exemplarily, the electronic device may input the N video clips into a video content feature extraction model M _video , extract the second video content feature information of each video clip from the video content information of the N video clips through the video content feature extraction model M _video , and perform feature fusion on the second video content feature information to finally obtain the first video content feature information.

Exemplarily, the electronic device may first divide each frame of an image in one of the N video clips into X image blocks of a preset size, and straighten the X image blocks into vectors of the same size and length, thereby obtaining image feature information corresponding to the X image blocks. Then, the vector is linearly transformed through a fully connected layer to obtain a vector F _video-in of a length L _video- in corresponding to the video content information in the above video clip.

Optionally, in the embodiment of the present application, the process of the above step 201c of "inputting the image feature information corresponding to the above X image blocks into a video content feature extraction model for feature extraction to obtain the second video content feature information corresponding to the X image blocks" includes the following steps 201c1 to 201c3:

Step 201c1: Based on the multi-head attention module, perform video content feature extraction on the image feature information corresponding to the X image blocks to obtain X first key video content feature information corresponding to the X image blocks.

Step 201c2: Based on the residual and standardization module, calculate the mean and standard deviation corresponding to the X first key video content feature information, and based on the mean and standard deviation, obtain X third video content feature information corresponding to the X first key video content feature information.

Step 201c3: Based on the forward feedback module, all feature information in each of the X pieces of third video content feature information are merged to obtain X pieces of second video content feature information.

Specifically, first, the image information feature information corresponding to the above X image blocks is input into Multi-Head Attention, and the 12 Self-Attentions contained therein are used using the formula The weights are calculated based on the correlation between the features, thereby extracting different first key video content feature information. Among them, Q (query), K (key), V (value) are all vectors obtained by remapping the input image feature information through an FC layer, and d _k represents the length of the vector.

Next, these different key video content feature information are input into the above Add&Norm, the mean and standard deviation corresponding to the first key video content feature information are calculated, and the first key video content feature information is calculated based on the mean and standard deviation to obtain the third video content feature information. Among them, the Add operation is equivalent to Short Cut, that is, the previous features can be reused, and the optimization problem of gradient disappearance during training can be alleviated, thereby improving the effect of the model; Norm is the layer normalization (LayerNor) operation of the feature, which can normalize the features to ensure the consistency of feature distribution.

Then, the third video content feature information is input into Feed Forward, passes through an FC layer and a RelU activation function, and then passes through an FC layer, and all feature information in the X third video content feature information are fused respectively, so as to obtain more fully described X second video content feature information.

Example 2, combined with Example 1, divide each RGB image frame in the above N video clips into P×P blocks, and obtain P×P×3 blocks. Then, each block is straightened into a vector of length P×P×3. A fully connected layer is used for linear transformation to obtain a vector F video _{-in of length L video -in} . In this way, the electronic device inputs the obtained vector F _video-in into the video content feature extraction model M _video , and after extracting the feature information of the video content, the video content feature information F _video-out of length L _video-out can be obtained. Wherein, L _video-out = L _user-out .

Step 202: respectively concatenate the user behavior feature information of the target user with the first video content feature information corresponding to the N video clips to obtain video feature information corresponding to each of the N video clips.

In an embodiment of the present application, the electronic device concatenates the corresponding positions of the video content feature information F _video-out and the user behavior feature information F _user-out corresponding to each of the above video clips through a Concatenate function, and finally obtains the video feature information corresponding to each of the above N video clips.

In the embodiment of the present application, the video feature information corresponding to any video clip includes: user behavior feature information of the target user and video content feature information of the any video clip. Wherein, the any video clip is any video clip among the N video clips.

In the embodiment of the present application, the above-mentioned user behavior feature information and video content feature information can both be represented in the form of feature vectors.

In the embodiment of the present application, the user behavior feature information of the target user includes user basic feature F _user-base information and learnable user attribute feature F _user-laern information.

In the embodiment of the present application, the above-mentioned user basic feature F _user-base information may be feature information that can reflect the user's gender, age, favorite film and television genres, and other information.

In a possible embodiment, the learnable user attribute feature F _user-laern information may be updated according to user behavior parameters.

In a possible embodiment, for the user basic feature F _user-base , the electronic device may encode the user attribute data (UserData) after acquiring the user attribute data, and then calculate a user basic feature F _user-base of length L user _-base based on the encoded user attribute data and the first formula.

Exemplarily, the above user attribute data may be shown in Table 1 below.

Table 1

Exemplarily, when encoding user attribute data, the electronic device usually encodes the collected user attribute data into a feature vector of each user attribute.

Exemplarily, when the electronic device converts the user attribute data into the user attribute feature vector, it is implemented by the following forwarding method. Specifically, the conversion method is: the optional values of each attribute are shown in Table 1, the feature vector length of each attribute is the number of its optional values, if the i-th value of attribute a is selected, then Va _,i = 1, otherwise Va _,i = 0.

Exemplarily, the electronic device encodes the user attribute data and represents it with a vector V, and normalizes each attribute using the first formula to obtain a data distribution suitable for neural network input. All normalized vectors are encoded again to obtain a user basic feature F _user-base with a length of L _user-base .

Exemplarily, the first formula is:

Among them, V _norm is the normalized attribute feature;

V _mean is the attribute mean obtained from the statistics of training data;

V _std is the standard deviation obtained from the training data.

In an embodiment of the present application, user behavior feature information is extracted by a user behavior feature extraction model M _user based on the above-mentioned user basic feature F _user-base information and learnable user attribute feature F _user-laern information.

In a possible embodiment, the electronic device concatenates the user basic feature F _user-base and the learnable user attribute feature F _user-laern using Concatenate to obtain F _user-in .

Exemplarily, after the electronic device inputs the above F _user-in into the user behavior feature extraction model M _user , the user behavior feature extraction model M _user may extract user behavior feature information from F _user-in to obtain user behavior feature information F _{user-out having a length of L user- out} .

Example 1: As shown in Figure 3, after the electronic device inputs F _user-in into the user behavior feature extraction model M _user , it will pass through a linear transformation layer (Fully Connected, FC) to change the dimension of F _user-in , and then pass through a regularization layer (BatchNorm) and an activation function layer (such as a Relu activation function) to convert the input into a vector F _user-hidden of length L _user-hidden ; then, it passes through the FC layer, BatchNorm regularization, and ReLU activation function again, and the final user behavior feature information F user _{-out of} size L _user-out is output.

Step 203: Determine a first playback speed corresponding to each video segment based on the video feature information corresponding to each video segment.

Step 204: Play the first video based on the first playback speed.

In the embodiment of the present application, the first playback speed corresponding to each of the above-mentioned video clips can be applied to the corresponding video clips respectively.

In an embodiment of the present application, the electronic device applies the determined first playback speed to each video clip, so that when playing the video, the video can automatically adjust the current playback speed according to the first playback speed.

Example 3: for the Ni _- th video segment S _Ni , assuming that the original number of frames is N, if the first playback speed determined is V, N/V frames are uniformly sampled from the original N frames to achieve a playback speed of V.

In the method for determining the video playback speed provided in the embodiment of the present application, the first video content feature information corresponding to the N video clips in the first video is obtained, where N is a positive integer; and the user behavior feature information of the target user is respectively spliced with the first video content feature information corresponding to the above N video clips to obtain the video feature information corresponding to each of the above N video clips; based on the video feature information corresponding to each of the above video clips, the first playback speed corresponding to each of the above video clips is determined; and then the first video is played based on the first playback speed. In this way, by integrating the user behavior feature information of the target user into the video content feature information of each video clip, the electronic device can set the video playback speed corresponding to each clip in a personalized manner according to the user behavior feature information and the video content feature information, thereby eliminating the need for the user to manually adjust the video playback speed.

Optionally, in the embodiment of the present application, the process of the above step 203 of "determining the first playback speed corresponding to each video clip based on the video feature information corresponding to each video clip" specifically includes the following steps 203a and 203b:

Step 203a: input the video feature information corresponding to each of the above video clips into a video playback speed estimation model, and output a playback speed prediction parameter corresponding to each of the above video clips.

Exemplarily, M _speed in the above video playback speed estimation model includes attention mechanism modules SEModules.

Optionally, in the embodiment of the present application, the process of step 203a includes the following steps 203a1 and 203a2:

Step 203a1: for one frame of the video clip among the above-mentioned N video clips, input the video feature information corresponding to the video clip into the attention mechanism module to obtain the key video feature information corresponding to the video clip.

Step 203a2: Obtain the playback speed information that is mapped to the key video feature information, and obtain the playback speed prediction parameter corresponding to the video clip based on the playback multiple information and the key video feature information.

Exemplarily, as shown in FIG4 , for the i-th video clip, the electronic device inputs the video feature information F _speed-in corresponding to the i-th video clip into the video playback speed estimation model M _speed , firstly performs key feature extraction through SEModules to obtain key video feature information, and obtains F _speed-se through the output of the attention module SEModules, then obtains F speed _-hidden through the FC layer, BatchNorm regularization, and ReLU activation function, and then obtains F _speed-out through the FC layer, and finally outputs the playback speed prediction parameter P _i corresponding to the i-th video clip through the Softmax function. i=1, 2..., N.

Further exemplarily, first, the corresponding user behavior feature information in the i-th video segment and the first video content feature information are concatenated using a Concatenate function operation to obtain the video feature information F _speed-in with a vector shape of (BatchSize, L _video-out +L _user-out ).

Next, the video feature information F _speed-in is input into SEModules, the user behavior feature information and the first video content feature information contained in F _speed-in are further fused, and different weights are assigned to the features of each channel in the module to indicate their importance, thereby extracting the key video feature information corresponding to the i-th video clip. Specifically, let the input of SEModule be F _speed-in , and the shape be (Batch Size, L _video-out +L _user-out ). F _speed-in passes through 2 FC layers and 1 Sigmoid layer to generate a feature weight coefficient W with a value range of 0-1 and a shape of (Batch Size, L _video-out +L _user-out ). Finally, F _speed-in and W are multiplied to obtain the weighted F _speed-in , i.e., the key video feature information F _speed-se , and its shape is still (Batch Size, L _video-out +L _user-out ).

Then, the key video feature information F _speed-se is input into the FC layer, and the BatchNorm regularization and ReLU activation function map the key video feature information to the dimension of the playback speed, and the abstract feature F _speed-hidden for estimating the playback speed is obtained. Then, the playback speed information F _speed-out with which the above key video feature information has a mapping relationship is obtained through the FC layer and output to the playback speed estimation module. Finally, the output F _speed-out is passed through the Softma function to obtain the prediction parameters of the playback speed of the i-th segment.

It should be noted that the above-mentioned i-th video clip is any video clip among the above-mentioned N video clips. In other words, each of the above-mentioned N video clips can be implemented according to the estimation process of the video playback speed estimation model for the above-mentioned i-th video clip, which will not be repeated here.

Step 203b: Determine a first playback speed corresponding to each video segment from X preset playback speeds based on the playback speed prediction parameters corresponding to each video segment.

Wherein, X is a positive integer.

Exemplarily, the X preset playback speeds may be playback speeds pre-stored in various video playback applications, for example, 0.5x/0.75x/1.0x/1.5x/2.0x.

It should be noted that the preset playback speeds in different applications may be the same or different. For example, if the first video is a video in a first application, the X preset playback speeds are the preset playback speeds corresponding to the first application.

Exemplarily, the playback speed prediction parameter corresponding to each video segment is used to indicate at least one preset playback speed. Thus, the electronic device can select a preset playback speed from the preset playback speeds indicated by the playback speed prediction parameter corresponding to a video segment as the first playback speed of the video segment.

Exemplarily, the above X preset playback multiples can also be set by the user. Specifically, the minimum playback speed C _min and the maximum playback speed C _max are set, and the number of selectable playback speeds is X, then the playback speed of the i-th level can be expressed as The set of selectable playback speeds is represented by C = {C _i }.

Optionally, in the embodiment of the present application, the playback speed prediction parameter corresponding to any video segment among the above-mentioned N video segments includes: X probability values.

Exemplarily, X probability values correspond to X preset playback speeds.

Exemplarily, any probability value among the X probability values is used to represent the probability that the playback speed of any video segment is a preset playback speed corresponding to any probability value.

Optionally, in the embodiment of the present application, the process of step 203b of "determining the first playback speed corresponding to each video clip from X preset playback speeds based on the playback speed prediction parameter corresponding to each video clip" specifically includes the following steps 203b1:

Step 203b1: Determine the preset playback speed corresponding to the maximum probability value among the X probability values corresponding to any one of the video clips as the first playback speed corresponding to any one of the video clips.

Exemplarily, the playback speed prediction parameter may be a vector of length X.

Exemplarily, for the i-th video clip, the electronic device may use the preset playback speed corresponding to the maximum element in the playback speed prediction parameter vector corresponding to the i-th video clip (i.e., the element with the largest probability value in the vector) as the corresponding first playback speed of the i-th video clip.

Exemplarily, the electronic device may use the second formula to select, from the X preset playback speeds, a preset playback speed corresponding to the maximum probability value among the X probability values corresponding to any one of the video clips.

Exemplarily, the second formula is:

Wherein, V is the first playback speed mentioned above;

_{Ci is} the i-th level preset playback speed corresponding to the above set C;

P _j is the vector corresponding to the position of the largest element.

In this way, based on the extraction of video content features, the playback speed corresponding to each video in the video is calculated, and the multiple is adaptively applied to the current video, so that the user does not need to manually adjust the playback speed during the video viewing process.

Optionally, in the embodiment of the present application, after the above step 204 of "applying the first playback speed corresponding to each of the above video segments to the corresponding video segments respectively", the method for determining the video playback speed provided by the present application further includes the following steps 301 and 302:

Step 301: after detecting a first input of a user adjusting a video playback speed of a first video clip, use a playback speed prediction parameter corresponding to the first video clip and an actual video playback speed corresponding to the first input as a first training sample.

Step 302: Use the first training sample to train the video playback speed estimation model to obtain the trained video playback speed estimation model.

Exemplarily, the first video clip is one of the N video clips.

Exemplarily, the first input may include: a touch input of a user to a display screen, or a voice command input by a user, or a specific gesture input by a user, which may be determined according to actual usage requirements and is not limited in the embodiments of the present invention.

Exemplarily, the first input is an input for adjusting the playback speed of the current video.

Exemplarily, the first input may include: a first input to the video interface when the user is not satisfied with the video playback speed of the first video clip.

Exemplarily, the electronic device may process the first training sample based on the third formula, and train the video playback speed estimation model using the processed training sample to obtain the trained video playback speed estimation model.

Exemplarily, the third formula above may be a cross entropy loss function.

Specifically, the third formula is: L=-logP _i [V _i ].

Wherein, Pi _is the playback speed prediction parameter corresponding to the first video clip;

Vi _is the actual video playback speed corresponding to the first input.

Optionally, in the embodiment of the present application, the method for determining the video playback speed provided by the present application further includes the following steps 401:

Step 401: When it is detected that the behavior of the target user adjusting the video playback speed within a predetermined time period satisfies a first condition, the behavior characteristic information of the target user is updated based on the first information of the target user.

Exemplarily, the first information includes: behavior parameters corresponding to the target user's behavior of adjusting the video playback speed within the predetermined time period, and video information of the video adjusted by the target user.

Exemplarily, the above-mentioned predetermined time period may be set by the user himself or customized by the electronic device.

Exemplarily, the first condition includes at least one of the following:

The number of times that the user actively adjusts the video playback speed of the first video within the predetermined time period reaches a first threshold (for example, the number of times that the user adjusts the video playback speed of a certain video segment within the predetermined time period reaches threshold A);

The adjustment duration of the video playback speed of the first video by the user within the predetermined time period reaches a second threshold.

Exemplarily, in the process of detecting the user's actively triggered speed playback behavior and the content of the corresponding video clip in the process of watching the video, the learnable user attribute feature F _user-laern information is continuously updated as the latest learnable user attribute feature F _user-laern information in the subsequent video watching process. In the process of updating the learnable user attribute feature F _user-laern information, only F _user-laern is used as a learnable parameter, and the remaining model parameters and F _user-base are fixed, and the cross entropy loss function is still used for training. In the process of the user watching the video, when the first condition is met within the predetermined time, that is, when the user actively adjusts the video playback speed to a predetermined threshold, the learnable user attribute feature F _user-laern information is updated.

In this way, through continuous training, it can be ensured that when the playback speed recommended by the model provided by the electronic device no longer meets the user's needs, training can be triggered in time and updated to the latest model, thereby greatly reducing the user's behavior of actively triggering the adjustment of the video playback speed when watching the video.

The image recognition method provided in this application is exemplified by an embodiment as follows.

In this embodiment, a personalized and adaptive video playback speed adjustment function is designed based on user behavior feature information and video content feature information, and the user does not need to manually adjust the playback multiple during video viewing. The constructed user portrait can help understand the type of video clips that the user is interested in, and the understanding of the video content can help locate the content that the user is interested in. Combining the above two, the user's expected video playback rate can be adaptively estimated and adjusted in time. Specifically, as shown in Figure 5, the following steps S1 to S6 are included:

Step S1: The electronic device obtains the user portrait and original video content information.

Step S2: The electronic device encodes the user portrait into a vector form through an encoder, and extracts the user behavior feature information vector based on the user behavior feature extraction model; at the same time, the original video obtained in step S1 is segmented to obtain N video clips.

Step S3: The electronic device inputs each video clip in step S2 into the video content feature extraction model, analyzes and encodes it, and obtains the user behavior feature information vector corresponding to each video.

Step S4: The electronic device inputs the user behavior feature information vector in step S2 and the user behavior feature information vector in step S3 into a playback speed estimation model to obtain a playback speed corresponding to each video clip.

Step S5: the electronic device applies the playback speed corresponding to each video segment to each video segment.

Step S6: The electronic device updates the user portrait in step S1, that is, updates the user behavior feature information, based on the user's active interaction in adjusting the video playback speed.

In this way, the user behavior feature extraction model and the video content feature extraction model are used to obtain abstract representations of users and video content respectively, and the relationship between the abstract representation and the video playback speed is established through the playback speed estimation model, thereby realizing the video speed playback function without user interaction.

It should be noted that the method for determining the video playback speed provided in the embodiment of the present application can be performed by a device for determining the video playback speed, or an electronic device, or a functional module or entity in the electronic device. In the embodiment of the present application, the method for determining the video playback speed performed by the device for determining the video playback speed is taken as an example to illustrate the device for determining the video playback speed provided in the embodiment of the present application.

Fig. 6 shows a possible structural diagram of a device for determining the video playback speed involved in an embodiment of the present application. As shown in Fig. 6, the device 700 for determining the video playback speed may include: an acquisition module 701, a processing module 702 and a playback module 703: the acquisition module 701 is used to acquire the first video content feature information corresponding to N video clips in the first video, where N is a positive integer; the processing module 702 is used to respectively splice the user behavior feature information of the target user with the first video content feature information corresponding to the above N video clips to obtain the video feature information corresponding to each video clip in the N video clips; the processing module 702 is used to determine the first playback speed corresponding to each video clip based on the video feature information corresponding to each video clip; the playback module 703 is used to play the above first video based on the first playback speed.

Optionally, in an embodiment of the present application, the processing module 702 is specifically used to divide the video image in one of the N video segments into X image blocks; the acquisition module 701 is also used to obtain image feature information corresponding to the X image blocks; the processing module 702 is specifically used to input the image feature information corresponding to the X image blocks into a video content feature extraction model for feature extraction to obtain second video content feature information corresponding to the X image blocks; the processing module 702 is specifically used to perform feature fusion on the second video content feature information corresponding to the X image blocks to obtain the first video content feature information.

Optionally, in an embodiment of the present application, the processing module 702 is specifically used to: after inputting the image feature information corresponding to the X image blocks into the video content feature extraction model, based on the multi-head attention module, perform video content feature extraction on the image feature information corresponding to the X image blocks to obtain X first key video content feature information corresponding to the X image blocks; based on the residual and normalization module, calculate the mean and standard deviation values corresponding to the X first key video content feature information, and based on the mean and standard deviation values, obtain X third video content feature information corresponding to the X first key video content feature information; based on the forward feedback module, respectively fuse all feature information in each of the X third video content feature information to obtain X second video content feature information; wherein, one image block corresponds to one first key video content feature information.

Optionally, in an embodiment of the present application, the processing module 702 is specifically used to: input the video feature information corresponding to each of the video clips into a video playback speed estimation model, and output a playback speed prediction parameter corresponding to each video clip; based on the playback speed prediction parameter corresponding to each video clip, determine the first playback speed corresponding to each of the video clips from X preset playback speeds; wherein X is a positive integer.

Optionally, in an embodiment of the present application, the processing module 702 is specifically used to: for one frame of the N video clips, input the video feature information corresponding to the video clip into the attention mechanism module to obtain the key video feature information corresponding to the video clip; obtain the playback speed information that has a mapping relationship with the key video feature information, and based on the playback multiple information and the key video feature information, obtain the playback speed prediction parameter corresponding to the video clip.

In the apparatus for determining the video playback speed provided in the embodiment of the present application, the first video content feature information corresponding to the N video clips in the first video is obtained, where N is a positive integer; and the user behavior feature information of the target user is respectively spliced with the first video content feature information corresponding to the above N video clips to obtain the video feature information corresponding to each of the above N video clips; based on the video feature information corresponding to each of the above video clips, the first playback speed corresponding to each of the above video clips is determined; and then the first video is played based on the first playback speed. In this way, by integrating the user behavior feature information of the target user into the video content feature information of each video clip, the electronic device can set the video playback speed corresponding to each clip in a personalized manner according to the user behavior feature information and the video content feature information, thereby eliminating the need for the user to manually adjust the video playback speed.

The device for determining the video playback speed in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

The device for determining the video playback speed in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

The apparatus for determining the video playback speed provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 6, and will not be described again here to avoid repetition.

Optionally, as shown in Figure 7, an embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, and the memory 802 stores a program or instruction that can be executed on the processor 801. When the program or instruction is executed by the processor 801, the various steps of the above-mentioned method for determining the video playback speed embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

FIG8 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

Those skilled in the art can understand that the electronic device 100 can also include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 110 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The electronic device structure shown in FIG8 does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than shown in the figure, or combine certain components, or arrange components differently, which will not be described in detail here.

Among them, the processor 110 is used to obtain the first video content feature information corresponding to N video clips in the first video, where N is a positive integer; the processor 110 is used to respectively splice the user behavior feature information of the target user with the first video content feature information corresponding to the above-mentioned N video clips to obtain the video feature information corresponding to each video clip in the N video clips; the processor 110 is used to determine the first playback speed corresponding to each video clip based on the video feature information corresponding to each video clip; the processor 110 is used to play the above-mentioned first video based on the first playback speed.

Optionally, in an embodiment of the present application, the processor 110 is specifically used to divide a video image in one of the N video segments into X image blocks; the processor 110 is also used to obtain image feature information corresponding to the X image blocks; the processor 110 is specifically used to input the image feature information corresponding to the X image blocks into a video content feature extraction model for feature extraction to obtain second video content feature information corresponding to the X image blocks; the processor 110 is specifically used to perform feature fusion on the second video content feature information corresponding to the X image blocks to obtain the first video content feature information.

Optionally, in an embodiment of the present application, the processor 110 is specifically configured to: after inputting the image feature information corresponding to the X image blocks into a video content feature extraction model, based on a multi-head attention module, perform video content feature extraction on the image feature information corresponding to the X image blocks to obtain X first key video content feature information corresponding to the X image blocks; based on a residual and normalization module, calculate the mean and standard deviation values corresponding to the X first key video content feature information, and based on the mean and standard deviation values, obtain X third video content feature information corresponding to the X first key video content feature information; based on a forward feedback module, respectively fuse all feature information in each of the X third video content feature information to obtain X second video content feature information; wherein one image block corresponds to one first key video content feature information.

Optionally, in an embodiment of the present application, the processor 110 is specifically used to: input the video feature information corresponding to each of the video clips into a video playback speed estimation model, and output a playback speed prediction parameter corresponding to each of the video clips; based on the playback speed prediction parameters corresponding to each of the video clips, determine the first playback speed corresponding to each of the video clips from X preset playback speeds; where X is a positive integer.

Optionally, in an embodiment of the present application, the processor 110 is specifically used to: for one frame of the N video clips, input the video feature information corresponding to the video clip into the attention mechanism module to obtain key video feature information corresponding to the video clip; obtain playback speed information that is mapped to the key video feature information, and based on the playback multiple information and the key video feature information, obtain the playback speed prediction parameter corresponding to the video clip.

In the electronic device for determining the video playback speed provided in the embodiment of the present application, the electronic device obtains the first video content feature information corresponding to N video clips in the first video, where N is a positive integer; and respectively splices the user behavior feature information of the target user with the first video content feature information corresponding to the above N video clips to obtain the video feature information corresponding to each of the above N video clips; based on the video feature information corresponding to each of the above video clips, determines the first playback speed corresponding to each of the above video clips; and then plays the first video based on the first playback speed. In this way, by integrating the user behavior feature information of the target user into the video content feature information of each video clip, the electronic device can set the video playback speed corresponding to each clip in a personalized manner according to the user behavior feature information and the video content feature information, thereby eliminating the need for the user to manually adjust the video playback speed.

It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned method for determining the video playback speed is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method for determining the video playback speed embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method for determining the video playback speed, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims (8)

1. A method for determining a video playback speed, characterized in that the method comprises: Obtaining first video content feature information corresponding to N video segments in the first video, wherein the N video segments are: N video segments synthesized according to a preset number of segment frames from video frames collected at fixed intervals of the first video, where N is a positive integer; The user behavior feature information of the target user is respectively spliced with the first video content feature information corresponding to the N video clips to obtain video feature information corresponding to each of the N video clips; Determining a first playback speed corresponding to each video clip based on the video feature information corresponding to each video clip; Play the first video based on the first playback speed; The obtaining first video content feature information corresponding to N video segments in the first video includes: For one of the N video segments, dividing each frame of video image in the video segment to obtain X image blocks; Obtaining image feature information corresponding to the X image blocks; Inputting the image feature information corresponding to the X image blocks into a video content feature extraction model for feature extraction to obtain second video content feature information corresponding to the X image blocks; Performing feature fusion on the second video content feature information corresponding to the X image blocks to obtain the first video content feature information; The video content feature extraction model includes a multi-head attention module, a residual and normalization module, and a forward feedback module. The image feature information corresponding to the X image blocks is input into the video content feature extraction model for feature extraction to obtain the second video content feature information corresponding to the X image blocks, including: After inputting the image feature information corresponding to the X image blocks into the video content feature extraction model, based on the multi-head attention module, performing video content feature extraction on the image feature information corresponding to the X image blocks to obtain X first key video content feature information corresponding to the X image blocks; Based on the residual and standardization module, calculate the mean and standard deviation values corresponding to the X first key video content feature information, and based on the mean and standard deviation values, obtain X third video content feature information corresponding to the X first key video content feature information; Based on the forward feedback module, all feature information of each third video content feature information in the X third video content feature information are respectively merged to obtain X second video content feature information; Among them, one image block corresponds to one first key video content feature information. 2. The method according to claim 1, characterized in that the step of determining the first playback speed corresponding to each video segment based on the video feature information corresponding to each video segment comprises: Inputting the video feature information corresponding to each video clip into a video playback speed estimation model, and outputting a playback speed prediction parameter corresponding to each video clip; Based on the playback speed prediction parameters corresponding to each video segment, determine the first playback speed corresponding to each video segment from Y preset playback speeds; Wherein, Y is a positive integer. 3. The method according to claim 2, characterized in that the step of inputting the video feature information corresponding to each video segment into a video playback speed estimation model and outputting the playback speed prediction parameter corresponding to each video segment comprises: For one of the N video clips, inputting video feature information corresponding to the video clip into an attention mechanism module to obtain key video feature information corresponding to the video clip; The playback speed information that is mapped to the key video feature information is obtained, and based on the playback speed information and the key video feature information, a playback speed prediction parameter corresponding to the video segment is obtained. 4. A device for determining a video playback speed, characterized in that the device for determining a video playback speed comprises: an acquisition module, a processing module and a playback module: The acquisition module is used to acquire first video content feature information corresponding to N video segments in the first video, where the N video segments are: N video segments synthesized according to a preset number of segment frames from video frames collected at fixed intervals of the first video, where N is a positive integer; The processing module is used to respectively splice the user behavior feature information of the target user with the first video content feature information corresponding to the N video clips acquired by the acquisition module to obtain video feature information corresponding to each of the N video clips; The processing module is used to determine the first playback speed corresponding to each video segment based on the video feature information corresponding to each video segment; The playback module is configured to play the first video based on the first playback speed; The processing module is specifically configured to divide each frame of a video image in one of the N video segments into X image blocks; The acquisition module is further used to acquire image feature information corresponding to the X image blocks; The processing module is specifically configured to input the image feature information corresponding to the X image blocks into a video content feature extraction model for feature extraction, so as to obtain second video content feature information corresponding to the X image blocks; The processing module is specifically used to perform feature fusion on the second video content feature information corresponding to the X image blocks to obtain the first video content feature information; The video content feature extraction model includes a multi-head attention module, a residual and normalization module, and a forward feedback module; The processing module is specifically used for: After inputting the image feature information corresponding to the X image blocks into the video content feature extraction model, based on the multi-head attention module, performing video content feature extraction on the image feature information corresponding to the X image blocks to obtain X first key video content feature information corresponding to the X image blocks; Based on the residual and standardization module, calculate the mean and standard deviation values corresponding to the X first key video content feature information, and based on the mean and standard deviation values, obtain X third video content feature information corresponding to the X first key video content feature information; Based on the forward feedback module, all feature information of each third video content feature information in the X third video content feature information are respectively merged to obtain X second video content feature information; Among them, one image block corresponds to one first key video content feature information. 5. The device according to claim 4, characterized in that The processing module is specifically used for: Inputting the video feature information corresponding to each video clip into a video playback speed estimation model, and outputting a playback speed prediction parameter corresponding to each video clip; Based on the playback speed prediction parameters corresponding to each video segment, determine the first playback speed corresponding to each video segment from Y preset playback speeds; Wherein, Y is a positive integer. 6. The device according to claim 5, characterized in that The processing module is specifically used for: For one of the N video clips, inputting video feature information corresponding to the video clip into an attention mechanism module to obtain key video feature information corresponding to the video clip; The playback speed information that is mapped to the key video feature information is obtained, and based on the playback speed information and the key video feature information, a playback speed prediction parameter corresponding to the video segment is obtained. 7. An electronic device, characterized in that it comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the method for determining the video playback speed as described in any one of claims 1 to 3 are implemented. 8. A readable storage medium, characterized in that a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the method for determining the video playback speed as described in any one of claims 1 to 3 are implemented.