CN118075409A - Data fusion method and device for multi-level user terminal - Google Patents

Abstract

The present invention provides a data fusion method and device for a multi-level user terminal, wherein the method comprises: obtaining a video to be processed, and performing a modification mark based on a processing type according to the video to be processed; based on the corresponding relationship between the processing type and the processing authority, distributing the video to be processed to each user terminal, and each user terminal performs a modification operation based on the processing type on the video frame of the video to be processed based on different processing authorities, and obtains each processed video corresponding to each user terminal; based on each processed video, determining the modification operations included in each video frame with the same frame sequence, and sorting the modification operations based on a preset sorting rule; using the video frame corresponding to the modification operation indicating the highest sorting as a base image, and performing modification operations corresponding to other modification marks in a descending order based on the base layer, and obtaining a fused video frame; merging the fused video frames according to the frame sequence, and obtaining a fused video. The present invention at least improves work efficiency.

Description

Data fusion method and device for multi-level user terminals

Technical Field

The present invention relates to data processing technology, and in particular to a data fusion method and device for multi-level user terminals.

Background technique

Video processing refers to the process of editing, cropping, merging, adding special effects, etc. to the original video materials in order to produce a video work that meets the requirements. In video processing, editors can select and extract specific clips from the original materials as needed, remove irrelevant content, adjust the order, add transition effects, etc. to achieve the purpose of storytelling. At the same time, video processing can also involve color correction, audio processing, and adding post-production elements such as subtitles, special effects, and music. Through video processing, fragmented video materials can be organically combined to create smooth and expressive visual works such as movies, TV shows, and advertisements.

The inventors discovered during their research that when processing the same video, it often includes multiple modification operations on the video frames, such as image specification modification, original content modification, additional content modification, etc., and multiple modification operations are often performed simultaneously on the same user end, which will result in slower video processing speed and the user may not be proficient in a certain modification operation, further reducing the corresponding work efficiency.

Summary of the invention

Based on the above problems, the present invention is proposed to provide a multi-level user terminal data fusion method and device that overcomes the above problems or at least partially solves the above problems.

According to one aspect of the present invention, a method for data fusion of a multi-level user terminal is provided, comprising the following steps:

Acquire a video to be processed, and perform a modification mark based on a processing type according to the video frame content of each video frame of the video to be processed;

Based on a one-to-one correspondence between different processing types and different processing permissions, the video to be processed is distributed to corresponding user terminals, and each user terminal performs a modification operation based on the corresponding processing type on the video frame of the video to be processed based on the different processing permissions it has, so as to obtain processed videos corresponding to each user terminal;

Determine all modification operations included in each video frame with the same frame sequence based on each processed video, and sort all modification operations based on a preset sorting rule;

The video frame corresponding to the modification operation with the highest indication ranking is used as the base image, and based on the base layer, modification operations corresponding to other modification marks are performed one by one in a descending order to obtain a fused video frame;

All fused video frames are merged in frame order to obtain a fused video.

Optionally, in the method according to the present invention, performing modification marking based on processing type according to the video frame content of each video frame of the video to be processed includes:

Splitting the video to be processed frame by frame to obtain each video frame;

Sending each video frame to a management end, the management end determining each processing type corresponding to each video frame and each modification content corresponding to each processing type based on the video frame content of each video frame, and establishing an initial modification marking layer corresponding to the processing type, wherein the initial modification marking layer includes a type area and a modification area;

Adding the processing type to the type area and adding the modified content to the modification area to obtain a current modification marking layer, and establishing a transparent pasting layer on the video frame corresponding to the current modification marking layer;

The current modification marking layer is pasted on the transparent pasting layer to complete the modification marking based on the processing type.

Optionally, in the method according to the present invention, the method further comprises:

generating a color area in the current modification marking layer;

Retrieving a color comparison table, wherein the color comparison table includes different processing types and marking colors corresponding to the different processing types;

Traversing the color comparison table, determining the marking color corresponding to the processing type, and filling the color area with the marking color;

Establishing transparent edge enclosing areas at the edges of each video frame respectively;

When the number of currently modified marking layers included in the same video frame is one, the edge enclosed area of the video frame is filled with the corresponding marking color;

When the number of current modification marking layers included in the same video frame is greater than one, the edge enclosing area of the video frame is divided into sub-areas equal in number to the number of current modification marking layers, and each sub-area is filled with a corresponding different marking color.

Optionally, in the method according to the present invention, each group of the user terminals includes the same number of processing sub-terminals;

The video to be processed is distributed to the corresponding user terminals, and then the following steps are further included:

Determine the processing quantity of the current modification indication layer corresponding to each user terminal in the to-be-processed video, and determine the processing proportion corresponding to each user terminal based on the processing quantity corresponding to each user terminal;

Based on the processing proportion corresponding to each user terminal, the number of corresponding processing sub-terminals for each user terminal is increased or decreased.

Optionally, in the method according to the present invention, each of the user terminals performs modification operations based on the corresponding processing types on the video frames of the video to be processed based on the different processing permissions it possesses, including:

In response to the processing signal sent by each user terminal after receiving the video to be processed, calling each operation plug-in corresponding to the different processing permissions possessed by each user terminal, and sending each operation plug-in to each user terminal;

Each user terminal respectively responds to the use of the corresponding operation plug-in, and performs a modification operation based on the corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed;

In response to the user terminal completing the modification operation, an operation area is generated in the current modification indication layer in the video frame corresponding to the modification operation, and an operation description corresponding to the modification operation is added to the operation area.

Optionally, in the method according to the present invention, the modification operation at least includes image specification modification;

Performing a modification operation based on a corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed includes:

Determine the modification content based on the current modification marking layer, and determine the preset multiple and the modification reference point based on the modification content;

Modifying the video frame by a preset multiple based on the modification reference point within the original frame of the video frame; and/or

The modification operation at least includes original content modification;

Performing a modification operation based on a corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed includes:

Determine the modification content based on the current modification marking layer, and determine the modification range and the modified image based on the modification content;

Determine a target area within the original frame of the video frame based on the modification range, and crop the target area to obtain a blank area;

Filling the blank area with the modified image; and/or

The modification operation at least includes additional content modification;

Performing a modification operation based on a corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed includes:

Determine the modified content based on the current modified marking layer, and determine the additional content and the additional area based on the modified content;

A transparent additional layer is established based on the additional area within the original frame of the video frame, and the additional content is added to the transparent additional layer.

Optionally, in the method according to the present invention, the order of image specification modification, original content modification, and additional content modification is used as a preset ordering rule;

Determine all modification operations included in each video frame with the same frame sequence based on each processed video, and sort all modification operations based on a preset sorting rule, including:

Obtaining each video frame in the same frame sequence in each processed video, and determining all modification operations included in each video frame corresponding to each frame sequence;

Sort all modification operations according to the preset sorting rules.

Optionally, in the method according to the present invention, the video frame corresponding to the modification operation with the highest indication ranking is used as the base image, and other modification operations are performed one by one in a descending order based on the base layer to obtain a fused video frame, including:

When the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and original content modification, the video frames corresponding to the image specification modification are used as base images;

Determine a modified reference point and a preset multiple based on the image specification modification, and establish a base coordinate system centered on the modified reference point based on the base image;

Determine a modification range based on the original content modification, establish a video frame coordinate system centered on the modification reference point based on the video frame corresponding to the original content modification, and obtain pixel coordinate points corresponding to pixel points at the edge of the modification range in the video frame coordinate system;

Modifying each of the pixel coordinate points based on the preset multiple, and determining whether each of the pixel coordinate points after modification is located in the base coordinate system;

When at least a portion of each pixel coordinate point after modification is located in the base coordinate system, determining a partial area in the base image occupied by at least a portion of each pixel coordinate point located in the base coordinate system;

Modify and determine a modified image based on the original content, and determine a portion of the image corresponding to at least a portion of the coordinate points of each pixel point located in the base coordinate system;

The partial area is cropped and the partial image is filled to obtain a fused video frame.

Optionally, in the method according to the present invention, the video frame corresponding to the modification operation with the highest indication ranking is used as the base image, and other modification operations are performed one by one in a descending order based on the base layer to obtain a fused video frame, including:

When the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and additional content modification, the video frames corresponding to the image specification modification are used as base images;

modifying and determining additional content and an additional area based on the additional content;

Establishing a transparent additional layer based on the additional area within the base frame of the base image, and adding the additional content to the transparent additional layer;

Obtaining first pixel values corresponding to edge pixel points located at the edge of the additional content;

Acquire second pixel values respectively corresponding to base pixel points respectively adjacent to edge pixel points in the base image;

Calculating an absolute difference between the first pixel value and the second pixel value;

When the absolute difference is less than a preset threshold, the edge pixel point corresponding to the first pixel value is configured as a third pixel value.

According to another aspect of the present invention, a data fusion device for a multi-level user terminal is provided, comprising:

a marking module configured to obtain a video to be processed, and perform a modification mark based on a processing type according to a video frame content of each video frame of the video to be processed;

The modification module is configured to distribute the to-be-processed video to the corresponding user terminals based on a one-to-one correspondence between different processing types and different processing permissions, and each user terminal performs a modification operation based on the corresponding processing type on the video frame of the to-be-processed video based on its different processing permissions, so as to obtain processed videos corresponding to each user terminal;

A sorting module is configured to determine all modification operations included in each video frame with the same frame sequence based on each processed video, and sort all the modification operations based on a preset sorting rule;

The video frame fusion module is configured to use the video frame corresponding to the modification operation with the highest indication ranking as a base image, and perform other modification operations one by one in a descending order based on the base layer to obtain a fused video frame;

The video fusion module is configured to merge all fused video frames in frame order to obtain a fused video.

According to the solution of the present invention, firstly, modification marks based on the processing type can be made for each video frame of the to-be-processed video that needs to be processed. Then, the to-be-processed video is distributed to user terminals with different processing permissions, so that each user terminal can perform modification operations of the corresponding processing types on the to-be-processed video based on its different processing permissions, thereby obtaining each processed video corresponding to each user terminal, and then the modification operations corresponding to each video frame with the same frame sequence in each processed video can be sorted based on a preset sorting rule, and the video frame corresponding to the modification operation indicating the highest sorting is used as the base image, and only the modification operations of other modification marks are performed on the base image, thereby obtaining a fused video frame; finally, all the fused video frames are merged to obtain the corresponding fused video, thereby increasing the speed of video processing and improving the corresponding work efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 shows a flow chart of a data fusion method for a multi-level user terminal according to an embodiment of the present invention;

FIG2 shows a structural block diagram of a data fusion device for a multi-level user terminal according to another embodiment of the present invention.

Detailed ways

The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

Video editing refers to the process of editing, cropping, merging, adding special effects, etc. to the original video materials in order to produce a video work that meets the requirements. In video editing, editors can select and extract specific clips from the original materials as needed, remove irrelevant content, adjust the order, add transition effects, etc., to achieve the purpose of storytelling. At the same time, video editing can also involve color correction, audio processing, and adding post-production elements such as subtitles, special effects, and music. Through video editing, fragmented video materials can be organically combined to create smooth and expressive visual works, such as movies, TV shows, and advertisements.

The inventors found in their research that when video editing is performed on the same video, it often includes multiple modification operations on the video frames, such as image specification modification, original content modification, additional content modification, etc., and multiple modification operations are often performed simultaneously on the same user end, which will result in a slower video editing speed and the user may not be good at a certain modification operation, further reducing the corresponding work efficiency.

In order to solve the above problems in the prior art, the inventor proposes the solution of the present invention. An embodiment of the present invention provides a data fusion method for a multi-level user terminal, which can be executed in a computing device.

FIG1 shows a flow chart of a multi-level user-side data fusion method according to an embodiment of the present invention. The method is suitable for execution in a computing device, wherein the computing device can be understood as a computer with data processing function in the prior art.

As shown in FIG. 1 , a multi-level user-side data fusion method in this embodiment starts with step S101. In step S101, the following contents are included: obtaining a video to be processed, and performing a modification mark based on the processing type according to the video frame content of each video frame of the video to be processed.

For example, in this embodiment, the video to be processed can be understood as a video to be edited. After obtaining the video to be processed, the server can perform modification marking based on the processing type for the video frame content corresponding to each video frame in the video to be processed, so that in the subsequent processing process, the corresponding processing type can be performed based on the modification mark.

Furthermore, in this embodiment, the above-mentioned “performing a modification mark based on the processing type according to the video frame content of each video frame of the video to be processed” may also include the following steps:

Splitting the video to be processed frame by frame to obtain each video frame;

Sending each video frame to a management end, the management end determining each processing type corresponding to each video frame and each modification content corresponding to each processing type based on the video frame content of each video frame, and establishing an initial modification marking layer corresponding to the processing type, wherein the initial modification marking layer includes a type area and a modification area;

Adding the processing type to the type area and adding the modified content to the modification area to obtain a current modification marking layer, and establishing a transparent pasting layer on the video frame corresponding to the current modification marking layer;

The current modification marking layer is pasted on the transparent pasting layer to complete the modification marking based on the processing type.

For example, in this embodiment, in order to implement modification marking for each video frame to be processed, the following method may be used:

First, the video to be processed needs to be split frame by frame to obtain each video frame;

Next, each video frame is sent to the management end, and the management end determines the processing type corresponding to each video frame and the modification content corresponding to each processing type according to the video frame content of each video frame, wherein the processing type can be understood as, for example, processing based on image specifications, processing based on original content, processing based on additional content, etc., and the modification content can be understood as the area and corresponding content that need to be modified in the video frame. Each video frame may correspond to one or more processing types, and accordingly, each processing type may correspond to one or more modification contents. For example, the video frame content of a certain video frame includes a kitten and a puppy, and the modification operation to be performed on the video frame is: deleting the kitten and adding a subtitle with the content of "Puppy Picture", then the processing type corresponding to the video frame is processing based on the original content and processing based on the additional content, and the modification content involved is the kitten and the puppy;

After determining the processing type and modification content corresponding to the video frame, the server will also establish an initial modification marking layer corresponding to the processing type, wherein the initial modification marking layer includes a type area and a modification area. Since the same video frame may include multiple processing types, the server will also correspondingly establish initial modification marking layers corresponding to the multiple processing types;

Next, based on the established initial modification marking layer, the processing type can be added to the type area in the initial modification marking layer, and the modification content can be added to the modification area in the initial modification marking layer, so as to obtain the corresponding current modification marking layer, and establish a transparent pasting layer on the surface of the video frame corresponding to the current modification marking layer;

Finally, the current modification marking layer is pasted on the transparent pasting layer to complete the modification marking based on the processing type of a video frame, and the modification marking based on the processing type of other video frames can also be processed simultaneously or sequentially according to the above method.

It should be noted that the transparent sticky layer can be understood as a transparent layer of the same size as the video frame. Since it is transparent throughout, establishing a transparent sticky layer on the surface of the video frame will not interfere with the video frame content of the video frame. In addition, through the setting of the transparent sticky layer, it can be ensured that there is no direct contact between the current modification marking layer and the video frame, and it can also prevent the content in the current modification marking layer from interfering with the video frame content of the video frame, thereby improving the protection of the video frame content.

Furthermore, in this embodiment, the above-mentioned multi-level user terminal data fusion method further includes the following steps:

Generate a color area in the current modification marking layer;

Retrieving a color comparison table, wherein the color comparison table includes different processing types and marking colors corresponding to the different processing types;

Traversing the color comparison table, determining the marking color corresponding to the processing type, and filling the color area with the marking color;

Establishing transparent edge enclosing areas at the edges of each video frame respectively;

When the number of currently modified marking layers included in the same video frame is one, the edge enclosed area of the video frame is filled with the corresponding marking color;

When the number of current modification marking layers included in the same video frame is greater than one, the edge enclosing area of the video frame is divided into sub-areas equal in number to the number of current modification marking layers, and each sub-area is filled with a corresponding different marking color.

For example, in this embodiment, in order to improve the identification of the processing type in the current modification marking layer, a color area can be generated in the current modification marking layer, and a color comparison table can be called, wherein the color comparison table includes different processing types and marking colors corresponding to different processing types. By traversing the color comparison table, the marking color corresponding to the processing type in the current modification marking layer is determined, and the color area is filled based on the marking color, thereby marking the type of the corresponding processing type by color based on the current modification marking layer; further, the processing type in the current modification marking layer can also be marked based on the video frame, for example, a transparent edge enclosing area can be established at the edge of each video frame, and the edge enclosing area can be filled with color based on the marking color, thereby achieving the corresponding processing type can be marked based on the video frame and the current modification marking layer, thereby improving the corresponding display effect.

It should be noted that, since the same video frame may include one or more different modification types, when the number of current modification marking layers included in the same video frame is one, the edge enclosing area of the video frame can be filled with the corresponding marking color; when the number of current modification marking layers included in the same video frame is greater than one, it is necessary to first divide the edge enclosing area of the video frame into sub-areas equal to the number of current modification marking layers, and fill each sub-area with a corresponding different marking color, so as to ensure that each modification type will be marked based on the video frame based on the corresponding marking color, thereby improving the corresponding marking. For example, when a certain video frame includes 3 current modification marking layers, and the corresponding marking colors are red, green and blue, then it is necessary to divide the edge enclosing area of the video frame into 3 sub-areas, and fill the 3 sub-areas with red, green and blue respectively.

In step S102, the following contents are included: based on the one-to-one correspondence between different processing types and different processing permissions, the video to be processed is distributed to the corresponding user terminals, and each user terminal performs a modification operation based on the corresponding processing type on the video frame of the video to be processed based on the different processing permissions it possesses, to obtain processed videos corresponding to each user terminal.

For example, in this embodiment, after completing the modification marking of each frame of the video to be processed, the video to be processed can be distributed to the corresponding user terminal based on the one-to-one correspondence between different processing types and different processing permissions. It should be noted that the user terminal can be understood as a terminal device with a data processing function, and each different user terminal has different processing permissions, and different processing permissions correspond to different processing types. For example, when the processing types include processing based on image specifications, processing based on original content, and processing based on additional content, the video processing can be sent to three user terminals corresponding to the first permission, the second permission, and the third permission for processing, wherein the first permission corresponds to the processing type based on image specifications, the second permission corresponds to the processing type based on original content, and the third permission corresponds to the processing type based on additional content.

After each user terminal receives the same video to be processed, it can perform modification operations based on the corresponding processing type on the video frame of the video to be processed based on its corresponding processing authority. For example, when a video frame in a certain frame sequence in the video to be processed includes two current modification marking layers, and the two current modification marking layers are respectively the two processing types of processing based on image specifications and processing based on original content, then the user terminal with the first authority and the user terminal with the second authority can perform corresponding modification operations on the video frame, while the user terminal with the third authority cannot perform any modification operations on the video frame. Through the above allocation method, different user terminals can modify the video frame based on different processing types. Modification operations can be performed, thereby improving the corresponding work efficiency; and corresponding permissions can be set for the corresponding user terminals according to the operation proficiency of each user. For example, user A is good at modification operations corresponding to processing based on image specifications, user B is good at modification operations corresponding to processing based on original content, and user C is good at modification operations corresponding to processing based on additional content. At this time, the first permission can be set for the user terminal corresponding to user A, the second permission can be set for the user terminal corresponding to user B, and the third permission can be set for the user terminal corresponding to user C, thereby further improving the corresponding work efficiency and the work enthusiasm of users accordingly.

When all the client terminals have respectively completed the modification operation on the to-be-processed video, the server can obtain the corresponding processed videos based on each client terminal, thereby facilitating further processing.

Furthermore, in this embodiment, each group of the user terminals includes the same number of processing sub-terminals, and the above-mentioned "distributing the to-be-processed video to the corresponding user terminals" may further include the following steps:

Determine the processing quantity of the current modification indication layer corresponding to each user terminal in the to-be-processed video, and determine the processing proportion corresponding to each user terminal based on the processing quantity corresponding to each user terminal;

Based on the processing proportion corresponding to each user terminal, the number of corresponding processing sub-terminals for each user terminal is increased or decreased.

For example, in this embodiment, in order to further improve work efficiency, each user terminal can be divided into groups, that is, each user terminal is an independent group, and each group of each user terminal includes a preset same number of processing sub-terminals. Here, the processing sub-terminal, like the user terminal, can also be understood as a terminal with data processing functions such as a mobile phone, a computer, a head display, etc., and each user terminal corresponds to a user group, and a processing sub-terminal corresponds to a user. By increasing the corresponding number of users, each group of user terminals can perform a common modification operation on the same video to be processed, thereby greatly reducing the working time and improving the corresponding processing effect.

It should be noted that, since the number of processing of the current modification indication layer corresponding to each user terminal in the same video to be processed is different, the processing time of each user terminal for the same video to be processed may also be different. For example, there are 10 video frames corresponding to the processing type based on image specifications, 20 video frames corresponding to the processing type based on original content, and 30 video frames corresponding to the processing type based on additional content. Assuming that the time required for modification operations based on different processing types is the same, then if three groups of user terminals with the same number of processing sub-terminals perform modification operations on the video to be processed at the same time, the corresponding processing time ratio should be 1:2:3. In order to ensure that each user terminal completes the modification operation on the same video to be processed at the same time or approximately at the same time, and to ensure that the workload of each user terminal is the same, it is necessary to adjust the number of processing sub-terminals of each user terminal according to the different modification numbers. The adjustment method includes: determining the processing proportion corresponding to each user terminal based on the modification number corresponding to each user terminal, and then increasing or decreasing the number of processing sub-terminals of each user terminal based on the processing proportion, so as to ensure that the workload corresponding to each user terminal is the same, improve work efficiency and reduce the overall processing time.

Furthermore, in this embodiment, the above-mentioned “each user terminal performs a modification operation based on a corresponding processing type on the video frame of the to-be-processed video based on the different processing permissions it possesses” may also include the following steps:

In response to the processing signal sent by each user terminal after receiving the video to be processed, calling each operation plug-in corresponding to the different processing permissions possessed by each user terminal, and sending each operation plug-in to each user terminal;

Each user terminal respectively responds to the use of the corresponding operation plug-in, and performs a modification operation based on the corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed;

In response to the user terminal completing the modification operation, an operation area is generated in the current modification indication layer in the video frame corresponding to the modification operation, and an operation description corresponding to the modification operation is added to the operation area.

For example, in this embodiment, after each user terminal receives the video to be processed and makes corresponding processing preparations, each user terminal can send a processing signal to the server. After receiving the processing signal, the server station retrieves the corresponding operation plug-ins according to the different processing permissions of each user terminal, and sends each operation plug-in to the user terminal respectively. For example, the operation plug-in corresponding to the first permission is an image specification processing plug-in, the operation plug-in corresponding to the second permission is an original content processing plug-in, and the operation plug-in corresponding to the third permission is an additional content processing plug-in. Each user terminal can obtain the corresponding processing type according to the current modification indication layer in each video frame, and use the corresponding operation plug-in to perform the modification operation of the corresponding processing type; and after completing the modification operation, a corresponding operation area will be generated in the corresponding current modification indication layer, and an operation description of the modification operation performed by the user terminal will be added in the operation area. That is, the modification operation of the user terminal is recorded, which is convenient for subsequent query of historical data, so that the modification operation completed by each user terminal can be understood.

Here, the operation instructions can be a further description of the modification operation. For example, when a user terminal uses the original content processing plug-in to modify a video frame, the original image in the video frame is replaced with another image. At this time, an operation instruction based on the complete replacement process can be generated.

Furthermore, in this embodiment, the modification operation at least includes image specification modification, and the above-mentioned "performing a modification operation based on the corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed" may also include the following steps:

Determine the modification content based on the current modification marking layer, and determine the preset multiple and the modification reference point based on the modification content;

The video frame is modified by a preset multiple based on the modification reference point within the original frame of the video frame.

For example, in this embodiment, when the user terminal's modification operation on the video frame is to modify the image specifications, first, the user terminal can determine the modification content based on the current modification marking layer in the video frame, wherein the modification content may include a preset multiple (for example, 0.5 or 2) and a modification reference point; then, the user terminal can find the location of the modification reference point in the video frame, and modify the video frame by a preset multiple based on the modification reference point, thereby completing the image specification modification of the video frame; it should be noted that when the image specification is modified, the size of the video frame remains unchanged, but the video frame content of the video frame will be modified accordingly by a preset multiple, that is, when the video frame content of the video frame is modified, the size of the video frame content will be larger than the size of the video frame, and at this time, the video frame content will be cropped accordingly based on the modification reference point, thereby ensuring that the size of the cropped video frame content is consistent with the size of the video frame.

Similarly, the modification operation at least includes modification of the original content. The above-mentioned “performing a modification operation based on the corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed” may also include the following steps:

Determine the modification content based on the current modification marking layer, and determine the modification range and the modified image based on the modification content;

Determine a target area within the original frame of the video frame based on the modification range, and crop the target area to obtain a blank area;

The blank area is filled with the modified image.

For example, in this embodiment, when the user terminal's modification operation on the video frame is to modify the original content, first, the user terminal can determine the modification content based on the current modification marking layer in the video, wherein the modification content may include a modification range and a modified image, and the modification range may be understood as the range involved in modifying the video frame, and the modified image may be understood as the image content involved in modifying the video frame; then, the user terminal can determine the target area based on the modification range within the original frame of the video frame, and crop the target area to obtain a blank area; finally, the user terminal can fill the modified image into the blank area, thereby completing the modification of the original content of the video frame; it should be noted that when modifying the original content, the size of the modified image should be consistent with the size of the modification range, so as to ensure the integrity of the content of the modified video frame.

Similarly, the modification operation at least includes additional content modification, and the above-mentioned "performing a modification operation based on the corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed" may also include the following steps:

Determine the modified content based on the current modified marking layer, and determine the additional content and the additional area based on the modified content;

A transparent additional layer is established based on the additional area within the original frame of the video frame, and the additional content is added to the transparent additional layer.

For example, in this embodiment, when the user terminal's modification operation on the video frame is additional content modification, first, the user terminal can determine the modification content based on the current modification indication layer in the video, where the modification content may include additional content and additional area; then, the user terminal can find the area corresponding to the additional area in the original frame of the video frame and establish a transparent additional layer in the area; finally, the user terminal can add the additional content to the transparent additional layer, thereby completing the modification operation based on the additional content modification.

In step S103, the following contents are included: based on each processed video, all modification operations included in each video frame with the same frame sequence are determined, and all modification operations are sorted based on a preset sorting rule.

It can be seen from the above content that each user terminal with different processing permissions will respectively obtain the same video to be processed and perform corresponding modification operations, thereby obtaining each processed video corresponding to each user terminal, that is, each processed video includes the same number of video frames. Therefore, all modification operations included in each video frame in the same frame sequence can be determined based on each processed video, and all modification operations can be sorted based on a preset sorting rule; for example, the processed videos include three, the modification operation included in the video frame with a frame sequence of one in the first processed video is image specification modification, the modification operation included in the video frame with a frame sequence of one in the second processed video is original content modification, and the modification operation included in the video frame with a frame sequence of one in the third processed video is additional content modification. At this time, the three modification operations can be sorted accordingly based on the preset sorting rule.

Furthermore, in this embodiment, the order of image specification modification, original content modification, and additional content modification is used as a preset ordering rule, and the above-mentioned "determining all modification operations included in each video frame with the same frame sequence based on each processed video, and ordering all modification operations based on the preset ordering rule" may also include the following steps:

Obtaining each video frame in the same frame sequence in each processed video, and determining all modification operations included in each video frame corresponding to each frame sequence;

Sort all modification operations according to the preset sorting rules.

For example, in this embodiment, when there are three processed videos, and the modification operation included in the video frame with a frame order of one in the first processed video is image specification modification, the modification operation included in the video frame with a frame order of one in the second processed video is original content modification, and the modification operation included in the video frame with a frame order of one in the third processed video is additional content modification, then the three modification operations can be sorted accordingly based on the preset sorting rules, that is, the image specification modification is in the first place, the original content modification is in the second place, and the additional content modification is in the third place.

In step S104, the following contents are included: the video frame corresponding to the modification operation with the highest indication ranking is used as the base image, and other modification operations are performed one by one in a descending order based on the base layer to obtain a fused video frame.

For example, in the present embodiment, the influence of different modification operations on the video frame content of the video frame is different. For example, for image specification modification, since it is necessary to modify the image specifications of the video frame content, the influence of image specification modification is the greatest; for original content modification, since it is necessary to modify the original content of the video frame content, the influence of original content modification is smaller; and for additional content modification, since it is mainly to add additional content to the video frame content, the influence of additional content modification is the smallest; based on the above judgment on the influence of different modification operations, different modification operations can be sorted according to corresponding processing priorities, and the video frame corresponding to the highest-ranked modification operation is used as the base image, and other modification operations are performed one by one according to the sorting based on the base image, so as to obtain the corresponding fused video frame. Compared with the fusion method of fusing multiple video frames in the prior art, the corresponding image processing amount is reduced, and the corresponding processing efficiency and processing effect are improved.

Furthermore, in this embodiment, the above-mentioned “using the video frame corresponding to the modification operation indicating the highest ranking as the base image, and performing other modification operations one by one in a descending order based on the base layer to obtain a fused video frame” may also include the following steps:

When the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and original content modification, the video frames corresponding to the image specification modification are used as base images;

Determine a modified reference point and a preset multiple based on the image specification modification, and establish a base coordinate system centered on the modified reference point based on the base image;

Determine a modification range based on the original content modification, establish a video frame coordinate system centered on the modification reference point based on the video frame corresponding to the original content modification, and obtain pixel coordinate points corresponding to pixel points at the edge of the modification range in the video frame coordinate system;

Modifying each of the pixel coordinate points based on the preset multiple, and determining whether each of the pixel coordinate points after modification is located in the base coordinate system;

When at least a portion of each pixel coordinate point after modification is located in the base coordinate system, determining a partial area in the base image occupied by at least a portion of each pixel coordinate point located in the base coordinate system;

Modify and determine a modified image based on the original content, and determine a portion of the image corresponding to at least a portion of the coordinate points of each pixel point located in the base coordinate system;

The partial area is cropped and the partial image is filled to obtain a fused video frame.

For example, in this embodiment, it can be known from the above content that in the preset sorting rules, the image specification modification is ranked the highest, that is, its influence on the video frame content is the greatest. Therefore, when the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and original content modification, the video frame corresponding to the image specification modification can be used as the base image, and the corresponding video frames are fused. The above-mentioned process of fusing the video frames can be as follows:

First, the modification reference point and the preset multiple can be determined based on the image specification modification, and a base coordinate system centered on the modification reference point can be established based on the base image. It should be noted that, as can be seen from the above content, when performing a modification operation based on the image specification modification, the modification content can be determined according to the current modification marking layer corresponding to the video frame, and the modification content includes the preset multiple and the modification reference point. The preset multiple can be understood as the multiple of enlarging or reducing the image, and the modification reference point can be understood as the corresponding preset multiple adjustment based on the point. Here, the preset multiple and the modification reference point can be set manually.

Next, the modification range is determined based on the modification of the original content, and a video frame coordinate system centered on the modification reference point is established based on the video frame corresponding to the modification of the original content, and each pixel point located at the edge of the modification range is obtained, and each pixel coordinate point corresponding to each pixel point in the video frame coordinate system is determined. It should be noted that, from the above content, when performing a modification operation based on the modification of the original content, the modification content can be determined according to the current modification marking layer corresponding to the video frame, and the modification content correspondingly includes the modification range and the modified image;

Next, each pixel coordinate point is modified based on a preset multiple, and it is determined whether each pixel coordinate point after modification is located in the base coordinate system. For example, when a pixel coordinate point is (1, 1) and the preset multiple is 2, the pixel coordinate point after modification is (2, 2).

Then, if at least part of the modified coordinate points of each pixel point is located in the base coordinate system, a partial area occupied by at least part of the coordinate points of each pixel point located in the base coordinate system in the base image is determined;

Then, a modified image may be determined based on the video frame corresponding to the modification of the original content, and a partial image corresponding to the coordinate system of at least a portion of each pixel point located in the base coordinate system may be determined in the modified image;

Finally, some areas are cropped in the base image and some images are filled to obtain the corresponding fused video frame.

Similarly, similar to the above solution, in this embodiment, the above-mentioned “using the video frame corresponding to the modification operation indicating the highest ranking as the base image, and performing other modification operations in a descending order based on the base layer to obtain a fused video frame” may also include the following steps:

When the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and additional content modification, the video frames corresponding to the image specification modification are used as base images;

modifying and determining additional content and an additional area based on the additional content;

Establishing a transparent additional layer based on the additional area within the base frame of the base image, and adding the additional content to the transparent additional layer;

Obtaining first pixel values corresponding to edge pixel points located at the edge of the additional content;

Acquire second pixel values respectively corresponding to base pixel points respectively adjacent to edge pixel points in the base image;

Calculating an absolute difference between the first pixel value and the second pixel value;

When the absolute difference is less than a preset threshold, the edge pixel point corresponding to the first pixel value is configured as a third pixel value.

For example, in this embodiment, it can be known from the above content that in the preset sorting rules, the image specification modification is ranked the highest, that is, its influence on the video frame content is the greatest. Therefore, when the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and additional content modification, the video frame corresponding to the image specification modification can be used as the base image, and the corresponding video frames are fused. The above-mentioned process of fusing the video frames can be as follows:

First, additional content and additional area may be determined based on the additional content modification;

Next, a transparent additional layer is established based on the additional area within the base frame of the base image, and the additional content is further added to the transparent additional layer;

Next, first pixel values corresponding to edge pixel points located at the edge of the additional content are obtained, and second pixel values corresponding to base pixel points adjacent to the edge pixel points in the base image are obtained;

Finally, the first pixel value and the second pixel value are difference-calculated to obtain the corresponding absolute difference, and when the absolute difference is less than a preset threshold, the edge pixel point corresponding to the first pixel value is configured as the third pixel value.

Here, the absolute difference can be understood as the absolute value of the difference. For example, when the difference is -1, the absolute difference is 1. It can be explained that, since in the above technical solution, the additional content is added to the base image, if the edge pixel points of the additional content are close to the values of the adjacent base pixel points, it means that the colors corresponding to the two are close, and it may not be possible to make a clear visual distinction between the two. Therefore, the corresponding preset threshold can be set in advance, and the absolute difference obtained by performing the difference calculation between the first pixel value of the corresponding edge pixel point and the second pixel value of the corresponding base pixel point is compared with the preset threshold. When it is less than the preset threshold, it means that the corresponding colors of the two are close. At this time, the corresponding edge pixel point needs to be set to the third pixel value to ensure that the colors between the two are clearly distinguishable, thereby improving the corresponding video display effect.

It should be noted that the above-mentioned preset threshold can be set as needed, for example, it can be set to between 90-140, so as to ensure that the two have obvious color distinction.

In S105, the following contents are included: all fused video frames are merged in frame order to obtain a fused video.

For example, in this embodiment, after all fused video frames are acquired according to the aforementioned technical solution, all fused video frames may be merged in frame order to obtain a corresponding fused video.

According to the scheme of this embodiment, firstly, modification marks based on the processing type can be made for each video frame of the to-be-processed video that needs to be processed. Then, the to-be-processed video is distributed to the user terminals with different processing permissions, so that each user terminal can perform modification operations of the corresponding processing types on the to-be-processed video based on its different processing permissions, thereby obtaining each processed video corresponding to each user terminal, and then the modification operations corresponding to each video frame with the same frame sequence in each processed video can be sorted based on the preset sorting rules, and the video frame corresponding to the modification operation with the highest order is used as the base image, and only the modification operations of other modification marks are used on the base image to obtain a fused video frame; finally, all the fused video frames are merged to obtain the corresponding fused video, thereby increasing the speed of video processing and improving the corresponding work efficiency.

Another embodiment of the present invention provides a data fusion device for a multi-level user terminal, and FIG2 is a corresponding device block diagram thereof. The system includes:

a marking module configured to obtain a video to be processed, and perform a modification mark based on a processing type according to a video frame content of each video frame of the video to be processed;

The modification module is configured to distribute the to-be-processed video to the corresponding user terminals based on a one-to-one correspondence between different processing types and different processing permissions, and each user terminal performs a modification operation based on the corresponding processing type on the video frame of the to-be-processed video based on its different processing permissions, so as to obtain processed videos corresponding to each user terminal;

A sorting module is configured to determine all modification operations included in each video frame with the same frame sequence based on each processed video, and sort all the modification operations based on a preset sorting rule;

The video frame fusion module is configured to use the video frame corresponding to the modification operation with the highest indication ranking as a base image, and perform modification operations corresponding to other modification marks in a descending order based on the base layer to obtain a fused video frame;

The video fusion module is configured to merge all fused video frames in frame order to obtain a fused video.

In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages can be utilized to implement the content of the present invention described herein, and the above description of specific languages is for the purpose of disclosing the preferred embodiment of the present invention.

In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and/or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative and not restrictive with respect to the scope of the present invention, and the scope of the present invention is defined by the appended claims.

Claims (10)

1. A multi-level user terminal data fusion method, characterized in that it includes the following steps: Acquire a video to be processed, and perform a modification mark based on a processing type according to the video frame content of each video frame of the video to be processed; Based on a one-to-one correspondence between different processing types and different processing permissions, the video to be processed is distributed to corresponding user terminals, and each user terminal performs a modification operation based on the corresponding processing type on the video frame of the video to be processed based on the different processing permissions it has, so as to obtain processed videos corresponding to each user terminal; Determine all modification operations included in each video frame with the same frame sequence based on each processed video, and sort all modification operations based on a preset sorting rule; The video frame corresponding to the modification operation with the highest indication ranking is used as a base image, and other modification operations are performed one by one in a decreasing order based on the base layer to obtain a fused video frame; All fused video frames are merged in frame order to obtain a fused video. 2. The multi-level user terminal data fusion method according to claim 1, characterized in that: Performing a modification mark based on a processing type according to the video frame content of each video frame of the video to be processed includes: Splitting the video to be processed frame by frame to obtain each video frame; Sending each video frame to a management end, the management end determining each processing type corresponding to each video frame and each modification content corresponding to each processing type based on the video frame content of each video frame, and establishing an initial modification marking layer corresponding to the processing type, wherein the initial modification marking layer includes a type area and a modification area; Adding the processing type to the type area and adding the modified content to the modification area to obtain a current modification marking layer, and establishing a transparent pasting layer on the video frame corresponding to the current modification marking layer; The current modification marking layer is pasted on the transparent pasting layer to complete the modification marking based on the processing type. 3. The multi-level user terminal data fusion method according to claim 2, characterized in that: The method further comprises: Generate a color area in the current modification marking layer; Retrieving a color comparison table, wherein the color comparison table includes different processing types and marking colors corresponding to the different processing types; Traversing the color comparison table, determining the marking color corresponding to the processing type, and filling the color area with the marking color; Establishing transparent edge enclosing areas at the edges of each video frame respectively; When the number of currently modified marking layers included in the same video frame is one, the edge enclosed area of the video frame is filled with the corresponding marking color; When the number of current modification marking layers included in the same video frame is greater than one, the edge enclosing area of the video frame is divided into sub-areas equal in number to the number of current modification marking layers, and each sub-area is filled with a corresponding different marking color. 4. The multi-level user terminal data fusion method according to claim 3, characterized in that: Each group of the user terminals includes the same number of processing sub-terminals; The video to be processed is distributed to the corresponding user terminals, and then the following steps are further included: Determine the processing quantity of the current modification indication layer corresponding to each user terminal in the to-be-processed video, and determine the processing proportion corresponding to each user terminal based on the processing quantity corresponding to each user terminal; Based on the processing proportion corresponding to each user terminal, the number of corresponding processing sub-terminals for each user terminal is increased or decreased. 5. The multi-level user terminal data fusion method according to claim 3 is characterized in that: The user terminals respectively perform modification operations based on the corresponding processing types on the video frames of the video to be processed based on the different processing permissions they have, including: In response to the processing signal sent by each user terminal after receiving the video to be processed, calling each operation plug-in corresponding to the different processing permissions possessed by each user terminal, and sending each operation plug-in to each user terminal; Each user terminal respectively responds to the use of the corresponding operation plug-in, and performs a modification operation based on the corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed; In response to the user terminal completing the modification operation, an operation area is generated in the current modification indication layer in the video frame corresponding to the modification operation, and an operation description corresponding to the modification operation is added to the operation area. 6. The multi-level user terminal data fusion method according to claim 5, characterized in that: The modification operation at least includes image specification modification; Performing a modification operation based on a corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed includes: Determine the modification content based on the current modification marking layer, and determine the preset multiple and the modification reference point based on the modification content; Modifying the video frame by a preset multiple based on the modification reference point within the original frame of the video frame; and/or The modification operation at least includes original content modification; Performing a modification operation based on a corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed includes: Determine the modification content based on the current modification marking layer, and determine the modification range and the modified image based on the modification content; Determine a target area within the original frame of the video frame based on the modification range, and crop the target area to obtain a blank area; Filling the blank area with the modified image; and/or The modification operation at least includes additional content modification; Performing a modification operation based on a corresponding processing type on the video frame according to the current modification indication layer in the video frame of the video to be processed includes: Determine the modified content based on the current modified marking layer, and determine the additional content and the additional area based on the modified content; A transparent additional layer is established based on the additional area within the original frame of the video frame, and the additional content is added to the transparent additional layer. 7. The multi-level user terminal data fusion method according to claim 6, characterized in that: The preset sorting rules are image specification modification, original content modification, and additional content modification; Determine all modification operations included in each video frame with the same frame sequence based on each processed video, and sort all modification operations based on a preset sorting rule, including: Obtaining each video frame in the same frame sequence in each processed video, and determining all modification operations included in each video frame corresponding to each frame sequence; Sort all modification operations according to the preset sorting rules. 8. The multi-level user terminal data fusion method according to claim 7, characterized in that: The video frame corresponding to the modification operation with the highest indication ranking is used as the base image, and other modification operations are performed one by one in a descending order based on the base layer to obtain a fused video frame, including: When the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and original content modification, the video frames corresponding to the image specification modification are used as base images; Determine a modified reference point and a preset multiple based on the image specification modification, and establish a base coordinate system centered on the modified reference point based on the base image; Determine a modification range based on the original content modification, establish a video frame coordinate system centered on the modification reference point based on the video frame corresponding to the original content modification, and obtain pixel coordinate points corresponding to pixel points at the edge of the modification range in the video frame coordinate system; Modifying each of the pixel coordinate points based on the preset multiple, and determining whether each of the pixel coordinate points after modification is located in the base coordinate system; When at least a portion of each pixel coordinate point after modification is located in the base coordinate system, determining a partial area in the base image occupied by at least a portion of each pixel coordinate point located in the base coordinate system; Modify and determine a modified image based on the original content, and determine a portion of the image corresponding to at least a portion of the coordinate points of each pixel point located in the base coordinate system; The partial area is cropped and the partial image is filled to obtain a fused video frame. 9. The multi-level user terminal data fusion method according to claim 7, characterized in that: The video frame corresponding to the modification operation with the highest indication ranking is used as the base image, and other modification operations are performed one by one in a descending order based on the base layer to obtain a fused video frame, including: When the modification operations corresponding to the video frames in the same frame sequence include at least image specification modification and additional content modification, the video frames corresponding to the image specification modification are used as base images; modifying and determining additional content and an additional area based on the additional content; Establishing a transparent additional layer based on the additional area within the base frame of the base image, and adding the additional content to the transparent additional layer; Obtaining first pixel values corresponding to edge pixel points located at the edge of the additional content; Acquire second pixel values respectively corresponding to base pixel points respectively adjacent to edge pixel points in the base image; Calculating an absolute difference between the first pixel value and the second pixel value; When the absolute difference is less than a preset threshold, the edge pixel point corresponding to the first pixel value is configured as a third pixel value. 10. A data fusion device for a multi-level user terminal, characterized by comprising: A marking module is configured to obtain a video to be processed, and perform a modification mark based on a processing type according to the video frame content of each video frame of the video to be processed; The modification module is configured to distribute the to-be-processed video to the corresponding user terminals based on a one-to-one correspondence between different processing types and different processing permissions, and each user terminal performs a modification operation based on the corresponding processing type on the video frame of the to-be-processed video based on its different processing permissions, so as to obtain processed videos corresponding to each user terminal; A sorting module is configured to determine all modification operations included in each video frame with the same frame sequence based on each processed video, and sort all the modification operations based on a preset sorting rule; The video frame fusion module is configured to use the video frame corresponding to the modification operation with the highest indication ranking as a base image, and perform other modification operations one by one in a descending order based on the base layer to obtain a fused video frame; The video fusion module is configured to merge all fused video frames in frame order to obtain a fused video.