CN110602424A - Video processing method and electronic equipment - Google Patents

Abstract

The embodiment of the invention discloses a video processing method and electronic equipment. Wherein, the video processing method includes: acquiring first image data and first audio data of the first video data; performing focus processing on pixels of at least one subject object in the first image data through a preset object separation network to obtain at least A second image data; through the preset voice separation network, focus processing on the audio data matched with at least one subject object in the first audio data to obtain at least one second audio data; for the second image data and the second audio The data is encoded and compressed to obtain the second video data. Using the embodiment of the present invention can realize focusing processing on the image and audio of each subject object.

Description

Video processing method and electronic device

technical field

Embodiments of the present invention relate to the technical field of artificial intelligence, and in particular, to a video processing method and electronic equipment.

Background technique

At present, image and audio separation technologies have been widely used; after image separation or audio separation, focusing processing can be performed on the separated image or audio to realize focusing on the image or audio.

However, it only focuses on the image or audio without considering that the image and audio corresponding to each main object in a video may be different, and does not focus on the image and audio corresponding to each main object separately.

Contents of the invention

Embodiments of the present invention provide a video processing method and an electronic device to solve the problem that the image and audio corresponding to each subject object cannot be separately focused.

In order to solve the problems of the technologies described above, the present invention is achieved in that:

In a first aspect, an embodiment of the present invention provides a video processing method, the video processing method comprising:

Acquiring first image data and first audio data of the first video data;

Focusing on pixels of at least one subject in the first image data through a preset object separation network to obtain at least one second image data;

Focusing on the audio data matched by at least one subject object in the first audio data through a preset voice separation network to obtain at least one second audio data;

Encoding and compressing the second image data and the second audio data is performed to obtain second video data.

In a second aspect, an embodiment of the present invention provides an electronic device, and the electronic device includes:

An acquisition module, configured to acquire the first image data and the first audio data of the first video data;

The first focusing module is configured to focus on pixels of at least one subject object in the first image data through a preset object separation network to obtain at least one second image data;

The second focusing module is configured to focus on the audio data matched by at least one subject object in the first audio data through a preset voice separation network, to obtain at least one second audio data;

The encoding module is used to encode and compress the second image data and the second audio data to obtain the second video data.

In the third aspect, the embodiment of the present invention provides an electronic device, which is characterized in that it includes a processor, a memory, and a computer program stored in the memory and operable on the processor, and the computer program is controlled by the The steps for implementing the video processing method described in the first aspect when the processor executes.

In a fourth aspect, an embodiment of the present invention further provides an electronic device, which is characterized in that it includes:

a touch screen, wherein the touch screen includes a touch-sensitive surface and a display screen;

one or more processors;

one or more memories;

one or more sensors;

and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs comprising instructions, which when executed by the electronic device , causing the electronic device to execute the steps of the video processing method described in the first aspect.

In the fifth aspect, the embodiment of the present invention also provides a computer non-transitory storage medium, the computer non-transitory storage medium stores a computer program, and it is characterized in that, when the computer program is executed by a computing device, the computer program implements the first aspect. The steps of the video processing method described above.

In a sixth aspect, an embodiment of the present invention further provides a computer program product, which is characterized in that, when the computer program product is run on a computer, the computer is made to execute the video processing method described in the first aspect.

In an embodiment of the present invention, at least one second image data is obtained by performing focusing processing on pixels of at least one subject object in the first image data of the electronic device through a preset object separation network; and through a preset voice separation network performing focus processing on the audio data matched by at least one main object in the first audio data of the electronic device to obtain at least one second audio data, and then realizing the focus processing on the image data and audio data of each main object.

Description of drawings

FIG. 1 is a flowchart of a video processing method provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of multi-person focusing provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of video processing provided by an embodiment of the present invention;

FIG. 4 is another schematic diagram of video processing provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of an electronic device provided by an embodiment of the present invention;

Fig. 6 is a schematic diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

FIG. 1 is a flowchart of a video processing method provided by an embodiment of the present invention. As shown in Figure 1, the video processing method may include:

Step 101: acquiring image data and audio data of the first video data;

Step 102: Focusing on pixels of at least one subject in the first image data through a preset object separation network to obtain at least one second image data;

Step 103: Focusing on the audio data matched by at least one subject object in the first audio data through a preset voice separation network to obtain at least one second audio data;

Step 104: Perform encoding and compression processing on the second image data and the second audio data to obtain second video data.

In an embodiment of the present invention, at least one second image data is obtained by performing focusing processing on pixels of at least one subject object in the first image data of the electronic device through a preset object separation network; and through a preset voice separation network performing focus processing on the audio data matched by at least one main object in the first audio data of the electronic device to obtain at least one second audio data; furthermore, the focus processing on the image data and audio data of each main object can be realized.

In the embodiment of the present invention, the first image data described in step 101 is the image data in the first video data; the first audio data described in step 101 is the audio data in the first video data; wherein, the first image The data can be acquired from the first audio data through a camera or other image acquisition equipment; the first audio data can be acquired from the first audio data through a microphone or other audio acquisition equipment.

In the embodiment of the present invention, the focusing processing on the pixels of at least one subject object in step 102 includes:

identifying pixels of non-subject objects from each frame of image data based on pixels of at least one subject object;

performing Gaussian filter processing on pixels of non-subject objects based on predetermined Gaussian filter processing coefficients; or

Based on a predetermined grayscale processing coefficient, grayscale processing is performed on the pixels of the non-subject object.

In the embodiment of the present invention, according to at least one pixel of the main object, the pixels of the non-main object are identified from each frame of image, and the pixels of the non-main object are blurred or the pixels of the main object are color-retained, and then Realize the focus processing on the pixels of the main object; among them, the focus processing on the pixels refers to concentrating on displaying the pixels of the main object, and weakening the display of the pixels of the non-main object; among them, the blurring process refers to applying Gaussian to the pixels of the non-main object Filtering for fuzzy processing: Ib'=GaussBlur(Ib, alpha); color retention is to retain the original color for the pixels of the main object, and grayscale processing for the pixels of non-main objects: Ib'=Gray(Ib, alpha) . Wherein, Ib is a pixel of a non-subject object, Ib' is the second image data, and alpha is an adjustment parameter.

Among them, the alpha parameter can be a fixed value or a variable value; for example, when the first video data is recorded video data, the alpha parameter is a fixed value, and the alpha parameter is generally set in advance before recording. After the recording is completed, the value of the alpha parameter cannot be changed; when the first video data is the video data in the recording process, the value of the alpha parameter can be changed according to the user's settings, so that the second video data can better meet the user's needs , improve user experience.

In the embodiment of the present invention, after identifying the pixels of the non-subject object from the first image data, the video processing method further includes: adjusting a predetermined Gaussian filter processing coefficient or a predetermined Gamma processing coefficients, specifically including:

Obtain the image brightness of the non-subject object;

According to the brightness of the image, a predetermined Gaussian filter processing coefficient or a predetermined grayscale processing coefficient is adjusted.

It should be noted that if the brightness value of the image of the non-subject object is very high, that is, when the brightness is very large, the predetermined Gaussian filter processing coefficient or the predetermined gray scale processing coefficient can be increased to darken the image brightness of the non-subject object, Further, the focus processing on the pixels of the subject object is realized.

In the embodiment of the present invention, the focusing processing on the audio data of at least one subject object in the first audio data in step 103 includes:

identifying audio data of a non-subject object from the first audio data based on audio data of at least one subject object;

Based on a preset attenuation coefficient, perform attenuation processing on the audio data of the non-main object.

In the embodiment of the present invention, according to the audio data of at least one main object, the audio data of the non-main object is identified from each frame of audio, and the audio data of the non-main object is suppressed and attenuated, and then the main object is realized. The focus processing of audio data, the focus processing of audio data refers to the attenuation processing of the audio data of the non-subject object, and then the audio data of the main object is concentratedly displayed; wherein, the suppression of attenuation processing: Ab'=beta*Ab, where beta is Attenuation coefficient between 0 and 1, beta = 0 if attenuation is completely suppressed.

Wherein, the beta parameter can be a fixed value or a value that can be changed; for example, when the first video data is recorded video data, the beta parameter is a fixed value, and generally the beta parameter is pre-set before recording. After recording, the value of the beta parameter cannot be changed; when the first video data is the video data in the recording process, the value of the beta parameter can be changed according to the user's settings, so that the second video data can better meet the user's needs , improve user experience.

In the embodiment of the present invention, the focusing processing on the audio data of at least one subject object described in step 103 includes:

Replace the audio data of the main object with the preset audio data.

In the embodiment of the present invention, the method of focusing on the audio data of the main object is not limited to suppressing attenuation, replacing by preset audio data, etc., for example, virtual sound, as long as the audio data of the main object can be compared with the non-main object The highlighted audio data can be included in the scope of protection of the embodiments of the present invention, and will not be repeated here.

In an embodiment of the present invention, after acquiring the first image data and the first audio data of the first video data, the video processing method further includes:

determining a target pixel for each subject object based on a user selection input of at least one subject object;

determining target audio data matching each subject object;

A mapping relationship between the target pixels of each main object and the target audio data matching each main object is established and stored in the second video data.

Wherein, the target pixels are all pixels of the main object, and the target audio data are all audio data of the main object.

In the process of separating the main object, based on the user's selection input of at least one main object, the pixels of a plurality of main objects are segmented from the first image data, for example, I0, I1, ..., In; and determined and The audio data matched by each subject object, such as A0, A1, ..., An; wherein, I0 and A0 correspond to the same subject object, I1 and A1 correspond to the same subject object, ..., In and An correspond to the same subject object; And establishing a mapping relationship between pixels of the same subject object and audio data, and storing the mapping relationship in the second video data. Wherein, the audio data matching the main object refers to the audio data generated by the main object or the audio data corresponding to the main object.

After the mapping relationship is established, different importance coefficients (ie, weight values) can also be set for different main objects, for example, different first weight values are set for pixels of different main objects, and different first weight values are set for different main objects. Different second weight values are set for the data. After the weight value is set, the mapping relationship is stored in the second audio data. The aforementioned setting of different weight values for multiple main objects can further enrich the user's visual effect and improve the user experience during subsequent playback.

In an embodiment of the present invention, determining the target audio data matched with each subject object includes:

From at least one pre-stored audio data, screening target audio data matching the audio characteristics of each subject object;

Alternatively, the audio data selected by the user is determined as target audio data matching each subject object.

It should be noted that for the first type of target audio data that matches the subject object, it is mainly identified by electronic equipment, that is, the audio data of each subject object is stored in advance and screened, because each subject object Audio characteristics (for example, voiceprint) are different, can carry out voiceprint recognition separately, and screen out the target audio data that matches the audio characteristic of each subject object; A user's selection is determined as target audio data matched with each subject object.

In the embodiment of the present invention, by matching the audio data of the main object, the pixels of the same main object are associated with the audio data, which facilitates subsequent processing of the pixel and audio data of the main object at the same time, so that it can be further played during subsequent playback. Enrich user visual effects and improve user experience.

In this embodiment of the present invention, the selection input includes at least one of the following items: single click, double click, long press, and the like.

In an embodiment of the present invention, after performing focus processing on pixels of at least one subject object in the first image data through a preset object separation network, and obtaining at least one second image data, the video processing method further includes:

When at least two of the at least one main object are selected, based on the first weight value, perform Gaussian filtering or grayscale processing on the target pixels of the at least two main objects;

Playing the second image data processed by Gaussian filtering or grayscale processing;

Wherein, the preset Gaussian filter processing coefficients or the preset gray scale processing coefficients of each of the main objects in at least one main object correspond to different first weight values; and/or,

In the embodiment of the present invention, after the at least one second audio data is obtained, the video processing method further includes:

In the case where at least two of the at least one main object are selected, based on the second weight value, performing mixing processing on the target audio data of the at least two main objects;

Play the second audio data after audio mixing;

Wherein, the preset attenuation coefficients of each of the at least one main object correspond to different second weight values.

In the embodiment of the present invention, the user directly clicks to select a certain subject object. After the subject object is identified, the non-subject objects other than the pixel Ix of the subject object (I0, I1, ..., a pixel of a subject object in In) The pixels of will be blurred or decolorized, and at the same time, the audio data of non-subject objects other than the audio data Ax (A0, A1, . When the user selects multiple main objects, different importance coefficients can be set for multiple main objects, such as c0, c1, c2, ..., cn; for the pixels of the main object, different main objects can be controlled based on the importance coefficient For the audio data of the subject object, weighted mixing can be performed based on the importance coefficient (A=c0*A0+c1*A1+c2*A2+...+cn*An).

It should be noted that for the video data in the recording process, different importance coefficients can be set for different execution subject objects during the recording process, so as to facilitate real-time processing of the second video data for the subject object selected by the user, and the pixel For example, when the user selects the pixel Ix of the main object 1, then for pixels other than Ix (including the pixels of the non-main object corresponding to the main object I, the pixels of other main objects other than I, and other main bodies other than I The pixels of the non-subject object corresponding to the object) are processed by Gaussian filtering or grayscale processing and played.

For the recorded video data, it is necessary to set different importance coefficients for different execution subject objects before the recording is completed, and store them in the second video data after setting, so as to facilitate the subsequent playback of the second video data , when the user selects the pixel Ix of the main object I, Gaussian filter processing or grayscale processing is performed on pixels other than Ix in the second video data and played.

The embodiments of the present invention can introduce importance parameters for different main objects, so as to facilitate different levels of processing for different main objects, thereby enriching the user's visual experience and enhancing the fun of the video.

In addition, the mixing processing of audio data may be performed simultaneously with or separately from the Gaussian filter processing or gradation processing of image data.

In an embodiment of the present invention, when at least two main objects in at least one main object are selected, the video processing method includes:

The 1st step: collect the first image data and the first audio data in the first video data respectively;

The second step: apply INet to each frame of image I in the first image data to perform subject object segmentation, and segment pixels I0, I1, ..., In of a plurality of subject objects; and to each frame of audio in the first audio data Apply ANet to segment, and segment audio data A0, A1,..., An that match multiple subject objects;

Step III: Establish the mapping relationship between the audio data of the main object and the pixels of the main object (Ix<->Ay);

Select on the separated audio waveform graph, judge the main object belonging to through the audio data, and then click to select the main object on the screen of the electronic device, and then establish the mapping relationship between the pixels of the same main object and the audio data.

The fourth step: save the mapping relationship (Ix<->Ay) in the recorded video data or the video data in the recording process;

Step Ⅴ: After analyzing the mapping relationship at the video playback end, the user can directly click on an object (for example, a portrait) on the playback interface, and the pixels of non-subject objects other than the portrait 1x will be blurred or decolorized, and at the same time Audio data of non-subject objects other than the corresponding audio Ax will be suppressed and attenuated.

In Figure 2, if I0 is selected, the pixels of I1 and I2 will be blurred or decolorized, the audio data A1 of I1 and the audio data A2 of I2 will be suppressed and attenuated; if I0 and I1 are selected, the The importance coefficients c1, c2, and c3 control the pixels of I0, I1, and I2 to be blurred or decolorized, and weighted mixing is performed according to the importance coefficients c1, c2, and c3, and the audio data obtained after suppressing attenuation is A=c1*A1 +c2*A2+c3*A3.

It should be noted that the user can also select multiple subjects. At this time, the importance coefficients (c0, c1, c2, ..., cn) are introduced. For visual pixels, the coefficients are used to control the blurring degree or color fading degree of different subjects , for audio data, the importance coefficients can be used for weighted mixing (A=c0*A0+c1*A1+c2*A2+...+cn*An).

Step VI: the second image data I' and the second audio data A' are recompressed and encoded as new video data.

The embodiment of the present invention introduces interaction on the basis of single-portrait focusing, realizes multi-portrait focusing, and increases scalability and operation fun.

It should be noted that, for the recorded video data, the first weight value corresponding to the pixel of each main object and the second weight value corresponding to the audio data of each main object can be preset before recording the video data ; In the process of subsequent playback, control the blurring of the pixels of the non-main object or the color retention of the pixels of the main object according to the first weight value, and control the suppression and attenuation processing of the audio data of the main object according to the second weight value, and The processed video is stored; in subsequent playback, the stored processed video data is played. For the video data in the recording process, the first weight value corresponding to the pixel of each main object and the second weight value corresponding to the audio data of each main object can be set during the recording process. processing for real-time playback.

In the embodiment of the present invention, the subject-object separation network includes: Mask R-CNN, and/or, the speech separation network includes: Long Short-Term Memory (Long Short-Term Memory, LSTM, LSTM).

In the embodiment of the present invention, the first video data is video data in the process of recording or recorded video data.

The video processing method will be described in detail below for the video data in the recording process or the recorded video data.

In an embodiment of the present invention, when the first video data is video data in the recording process (as shown in FIG. 3 ), the video processing method includes:

The first step: using a camera and a microphone to respectively collect the first image data and the first audio data of the video data in the recording process;

Second step: apply INet to each frame image (that is, the i-th frame) I in the first image data for image segmentation, assuming that the pixel of the main object that is segmented is In, and the background pixel is Ib (that is, the pixel of the non-main object );

Step 3: performing blurring on the separated pixels of the non-main object or color-reserving processing on the pixels of the main object to obtain the second image data.

Blurring is to use Gaussian filter to blur the pixels of the non-subject object: Ib'=GaussBlur(Ib, alpha), color retention means to retain the original color of the pixels of the subject object, and the grayscale of the pixel of the non-subject object Bl: Ib'=Gray(Ib, alpha). Among them, alpha is an adjustment parameter. For the video data in the recording process, the blur or color retention parameters can be adjusted in real time to change the final effect. The second image data is denoted by I'.

The fourth step: apply ANet to each frame of audio (that is, the i-th frame) A in the first audio data to separate the audio data corresponding to the subject object matching, assuming that An is the audio data matched by the subject object, and Ab is the background audio data ( That is, the audio data of the non-subject object), the original overall audio data is the superposition of the two A=An+Ab;

The fifth step: suppressing and attenuating the audio data of the non-subject object to obtain the second audio data;

where Ab'=beta*Ab, where beta is an attenuation coefficient between 0 and 1, and beta=0 if complete inhibition. The beta parameter can also be adjusted in real time in the video data during the recording process. The overall audio data after suppression processing is A'=An+Ab'=An+beta*Ab.

The sixth step: encoding and compressing the second image data P' and the second audio data A' respectively;

The seventh step: transmit the focused compressed video and compressed audio real-time streams through the network, and the real-time streams are the video data in the recording process, such as live video data.

The invention effectively combines image segmentation and audio data separation, and realizes two-dimensional focusing of image and audio in video data during recording.

In an embodiment of the present invention, in the case where the first video data is recorded video data (as shown in FIG. 4 ), the video processing method includes:

Step 1: Decoding and separating the recorded video data to obtain first image data and first audio data respectively;

The 2nd step: apply INet to every frame image (that is, the i frame) I in the first image data and carry out image segmentation, assume that the pixel of the subject object that is divided is In, and the background pixel is Ib;

Step 3: Perform blurring or color retention processing on the separated pixels of non-subject objects to obtain second image data;

Blurring is to use Gaussian filter to blur the pixels of the non-subject object: Ib'=GaussBlur(Ib, alpha), color retention means to retain the original color of the pixels of the subject object, and the grayscale of the pixel of the non-subject object Bl: Ib'=Gray(Ib, alpha). Unlike the video data in the recording process, alpha can only be preset once, and once the recorded video data is generated, it cannot be modified. The second image data is represented by I';

Step 4: ANet is applied to each frame of audio (that is, the i-th frame) A in the first audio data to segment, assuming that An is the audio data matched by the main object, and Ab is the background audio data (that is, the audio data of the non-main object ), the original overall audio data is the superposition A=An+Ab of the two;

Step 5: suppress and attenuate the audio data of the non-subject object to obtain the second video data;

Inhibition decay is Ab'=beta*Ab, where beta is an attenuation coefficient between 0 and 1, and beta=0 if complete inhibition. The overall audio data after suppressing attenuation processing is A'=An+Ab'=An+beta*Ab;

Step 6: Recompressing and encoding the second image data I' and the second audio data A' as second video data, which can be shared as a short video on the Internet.

The invention effectively combines AI image segmentation and AI audio data separation, and realizes two-dimensional focus processing of images and audio in recorded video data.

In the embodiment of the present invention, the subject object is not limited to characters, animals, cartoon characters, cartoon animals and the like.

Fig. 5 is a schematic diagram of an electronic device provided by an embodiment of the present invention. As shown in Figure 5, the electronic device 50 includes:

An acquisition module 501, configured to acquire first image data and first audio data in the first video data;

The first focusing module 502 is configured to perform focusing processing on pixels of at least one subject object in the first image data through a preset object separation network to obtain at least one second image data;

The second focusing module 503 is configured to focus on the audio data matched by at least one subject object in the first audio data through a preset voice separation network, to obtain at least one second audio data;

The encoding module 504 is configured to perform encoding and compression processing on the second image data and the second audio data to obtain second video data.

In an embodiment of the present invention, at least one second image data is obtained by performing focusing processing on pixels of at least one subject object in the first image data of the electronic device through a preset object separation network; and through a preset voice separation network performing focus processing on the audio data matched by at least one main object in the first audio data of the electronic device to obtain at least one second audio data, and then realizing the focus processing on the image data and audio data of each main object.

Optionally, the first focusing module 502 is also used for:

identifying pixels of non-subject objects from the first image data based on pixels of at least one subject object;

Based on the preset Gaussian filter processing coefficient, Gaussian filter processing is performed on the pixels of the non-main object, or, based on the preset gray scale processing coefficient, grayscale processing is performed on the pixels of the non-main object.

In the embodiment of the present invention, by performing Gaussian filter processing or grayscale processing on the pixels of the non-main object, the focus processing on the pixels of the main object is further realized.

Optionally, the electronic device also includes:

The obtaining module is also used to obtain the image brightness of the non-subject object;

The adjustment module is configured to adjust a preset Gaussian filter processing coefficient or a preset gray scale processing coefficient according to image brightness.

The embodiment of the present invention can more flexibly adjust the preset Gaussian filter processing coefficient or the preset grayscale processing coefficient based on the image brightness of the pixel of the non-subject object more flexibly, and then dynamically adjust the focusing effect.

Optionally, the second focusing module 503 is also used for:

identifying audio data of a non-subject object from the first audio data based on audio data of at least one subject object;

Based on a preset attenuation coefficient, perform attenuation processing on the audio data of the non-main object.

In the embodiment of the present invention, by performing attenuation processing on the audio data of the non-main object, the focus processing on the audio data of the main object is realized.

Optionally, the second focusing module 503 is also used for:

Replace the audio data of the main object with the preset audio data.

In the embodiment of the present invention, the method of focusing on the audio data of the main object is not limited to suppressing attenuation, replacing by preset audio data, etc., for example, virtual sound, as long as the audio data of the main object can be compared with the non-main object The highlighted audio data can be included in the scope of protection of the embodiments of the present invention, and will not be repeated here.

Optionally, the electronic device also includes:

A determining module, configured to determine the target pixel of each subject object based on the user's selection input of at least one subject object;

A determination module is also used to determine the target audio data matched with each subject object;

Establishing a module for establishing a mapping relationship between the target pixels of each subject object and the target audio data matched with each subject object

A storage module, configured to store the mapping relationship into the second video data.

In the embodiment of the present invention, by establishing a mapping relationship between the pixels of each main object and the audio data, it is possible to perform focus processing on the pixels and audio data of each main object.

Optional, identify modules, also used for:

From at least one pre-stored audio data, screening target audio data matching the audio characteristics of each subject object;

Alternatively, the audio data selected by the user is determined as target audio data matching each subject object.

In the embodiment of the present invention, by matching the audio data of the main object, the pixels of each main object can be associated with the audio data, which facilitates subsequent processing of the pixels and audio data of the main object at the same time, so that during subsequent playback , which can further enrich user visual effects and improve user experience.

Optionally, the electronic device also includes:

A processing module, configured to perform Gaussian filtering or grayscale processing on the target pixels of the at least two subject objects based on the first weight value when at least two subject objects in the at least one subject object are selected;

A playback module, configured to play the second image data processed by Gaussian filtering or grayscale processing;

Wherein, the preset Gaussian filter processing coefficients or the preset gray scale processing coefficients of each of the at least one main object correspond to different first weight values.

The embodiments of the present invention can introduce importance parameters for different main objects, so as to perform different degrees of Gaussian filtering or grayscale processing on the pixels of different main objects, thereby enriching the user's visual experience and enhancing the interest of the video.

Optionally, the electronic device further includes: a processing module, configured to mix the target audio data of at least two main objects based on the second weight value when at least two main objects in at least one main object are selected. Audio processing; a playback module, configured to play the second audio data after audio mixing; wherein, the preset attenuation coefficient of each of the at least one main object corresponds to a different second weight value.

The embodiments of the present invention can introduce importance parameters for different main objects, so as to attenuate the audio data matched by different main objects to different degrees, thereby enriching the user's visual experience and enhancing the fun of the video.

Optionally, the first video data is video data during recording or recorded video data.

Optionally, the subject-object separation network includes: Mask R-CNN, and/or, the speech separation network includes: long short-term memory network LSTM.

The electronic device provided by the embodiment of the present invention can implement various processes implemented by the electronic device in the method embodiment in FIG. 1 , and details are not repeated here to avoid repetition.

In an embodiment of the present invention, at least one second image data is obtained by performing focusing processing on pixels of at least one subject object in the first image data of the electronic device through a preset object separation network; and through a preset voice separation network performing focus processing on the audio data matched by at least one subject object in the first audio data of the electronic device to obtain at least one second audio data, and then realizing the focus processing on the image and audio in the first video data.

6 is a schematic diagram of the hardware structure of an electronic device implementing various embodiments of the present invention, the electronic device 100 includes but not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, and a display unit 106 , a user input unit 107 , an interface unit 108 , a memory 109 , a processor 110 , and a power supply 111 and other components. Those skilled in the art can understand that the structure of the electronic device shown in Figure 6 does not constitute a limitation on the electronic device, and the electronic device may include more or less components than shown in the illustration, or combine certain components, or different components layout. In the embodiment of the present invention, the electronic device includes, but is not limited to, an electronic device, a tablet computer, a notebook computer, a palmtop computer, a vehicle terminal, a wearable device, and a pedometer.

The processor 110 is configured to perform focus processing on pixels of at least one subject object in the first image data through a preset object separation network to obtain at least one second image data;

performing focus processing on the audio data matching at least one subject object in the first audio data through a preset voice separation network, to obtain at least one second audio data;

Encoding and compressing the second image data and the second audio data to obtain second video data.

In an embodiment of the present invention, at least one second image data is obtained by performing focusing processing on pixels of at least one subject object in the first image data of the electronic device through a preset object separation network; and through a preset voice separation network performing focus processing on the audio data matched by at least one main object in the first audio data of the electronic device to obtain at least one second audio data, and then realizing the focus processing on the image data and audio data of each main object.

It should be understood that, in the embodiment of the present invention, the radio frequency unit 101 can be used for receiving and sending signals during sending and receiving information or during a call. Specifically, after receiving the downlink data from the base station, the processor 110 processes it; Uplink data is sent to the base station. Generally, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency unit 101 can also communicate with the network and other devices through a wireless communication system.

The electronic device provides users with wireless broadband Internet access through the network module 102, such as helping users send and receive emails, browse web pages, and access streaming media.

The audio output unit 103 may convert audio data received by the radio frequency unit 101 or the network module 102 or stored in the memory 109 into an audio signal and output as sound. Also, the audio output unit 103 may also provide audio output related to a specific function performed by the electronic device 100 (eg, call signal reception sound, message reception sound, etc.). The audio output unit 103 includes a speaker, a buzzer, a receiver, and the like.

The input unit 104 is used to receive audio or video signals. The input unit 104 may include a graphics processing unit (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processing unit 1041 is used for still pictures or video images obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The data is processed. The processed image frames may be displayed on the display unit 106 . The image frames processed by the graphics processor 1041 may be stored in the memory 109 (or other storage media) or sent via the radio frequency unit 101 or the network module 102 . The microphone 1042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format that can be sent to a mobile communication base station via the radio frequency unit 101 for output in the case of a phone call mode.

The electronic device 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1061 and the / or backlighting. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when it is still, and can be used to identify the posture of electronic equipment (such as horizontal and vertical screen switching, related games) , magnetometer posture calibration), vibration recognition-related functions (such as pedometer, knocking), etc.; the sensor 105 can also include fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, Infrared sensors, etc., will not be repeated here.

The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061 , and the display panel 1061 may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD) or an organic light-emitting diode (Organic Light-Emitting Diode, OLED).

The user input unit 107 can be used to receive input numbers or character information, and generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 107 includes a touch panel 1071 and other input devices 1072 . The touch panel 1071, also referred to as a touch screen, can collect touch operations of the user on or near it (for example, the user uses any suitable object or accessory such as a finger or a stylus on the touch panel 1071 or near the touch panel 1071). operate). The touch panel 1071 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and sends it to the For the processor 110, receive the command sent by the processor 110 and execute it. In addition, the touch panel 1071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071 , the user input unit 107 may also include other input devices 1072 . Specifically, other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.

Further, the touch panel 1071 can be covered on the display panel 1061, and when the touch panel 1071 detects a touch operation on or near it, it will be sent to the processor 110 to determine the type of the touch event, and then the processor 110 can The type of event provides a corresponding visual output on the display panel 1061 . Although in FIG. 6, the touch panel 1071 and the display panel 1061 are used as two independent components to realize the input and output functions of the electronic device, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated. The implementation of the input and output functions of the electronic device is not specifically limited here.

The interface unit 108 is an interface for connecting an external device to the electronic device 100 . For example, an external device may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, audio input/output (I/O) ports, video I/O ports, headphone ports, and more. The interface unit 108 can be used to receive input from an external device (for example, data information, power, etc.) transfer data between devices.

The memory 109 can be used to store software programs as well as various data. The memory 109 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.) etc.; Data created by the use of electronic devices (such as audio data, phonebook, etc.), etc. In addition, the memory 109 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

The processor 110 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device, by running or executing software programs and/or modules stored in the memory 109, and calling data stored in the memory 109 , to perform various functions of the electronic equipment and process data, so as to monitor the electronic equipment as a whole. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem The processor mainly handles wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 110 .

The electronic device 100 can also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. and other functions.

In addition, the electronic device 100 includes some functional modules not shown, which will not be repeated here.

Preferably, the embodiment of the present invention also provides an electronic device, including a processor 110, a memory 109, a computer program stored on the memory 109 and operable on the processor 110, when the computer program is executed by the processor 110 Each process of the foregoing video processing method embodiment can achieve the same technical effect. To avoid repetition, details are not repeated here.

The embodiment of the present invention also provides an electronic device, including:

a touch screen, wherein the touch screen includes a touch-sensitive surface and a display screen;

one or more processors 110;

one or more memories 109;

one or more sensors;

and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs comprising instructions, which when executed by the electronic device , so that the electronic device executes the various processes for realizing the above video processing method embodiment, and can achieve the same technical effect, to avoid repetition, no more details are given here.

The embodiment of the present invention also provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, each process of the above-mentioned video processing method embodiment is realized, and the same technology can be achieved. Effect, in order to avoid repetition, will not repeat them here. Wherein, the computer-readable storage medium is, for example, a read-only memory (Read-Only Memory, ROM for short), a random access memory (Random Access Memory, RAM for short), a magnetic disk or an optical disk, and the like.

An embodiment of the present invention also provides a computer non-transitory storage medium, the computer non-transitory storage medium stores a computer program, and when the computer program is executed by a computing device, each process of the above video processing method embodiment is implemented, and can To achieve the same technical effect, in order to avoid repetition, no more details are given here.

The embodiment of the present invention also provides a computer program product, when the computer program product is run on a computer, the computer can realize each process of the above video processing method embodiment when executed, and can achieve the same technical effect, for To avoid repetition, I won't go into details here.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal (which may be an electronic device, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present invention.

Embodiments of the present invention have been described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementations, and the above-mentioned specific implementations are only illustrative, rather than restrictive, and those of ordinary skill in the art will Under the enlightenment of the present invention, without departing from the gist of the present invention and the protection scope of the claims, many forms can also be made, all of which belong to the protection of the present invention.

Claims (14)

1. A video processing method, characterized in that, comprising: Acquiring first image data and first audio data of the first video data; performing focus processing on pixels of at least one subject object in the first image data through a preset object separation network to obtain at least one second image data; performing focus processing on the audio data matching at least one subject object in the first audio data through a preset voice separation network, to obtain at least one second audio data; Encoding and compressing the second image data and the second audio data to obtain second video data. 2. The method according to claim 1, wherein the focusing processing on pixels of at least one subject object in the first image data comprises: identifying pixels of a non-subject object from the first image data based on pixels of the at least one subject object; Based on a preset Gaussian filter processing coefficient, perform Gaussian filter processing on the pixels of the non-main object, or perform grayscale processing on the pixels of the non-main object based on a preset grayscale processing coefficient. 3. The method according to claim 2, wherein after identifying the pixels of the non-subject object from the first image data, further comprising: Acquiring the image brightness of the non-subject object; According to the brightness of the image, the preset Gaussian filter processing coefficient or the preset grayscale processing coefficient is adjusted. 4. The method according to claim 1, wherein the focusing processing on the audio data of at least one subject object in the first audio data comprises: identifying audio data of a non-subject object from the first audio data based on the audio data of the at least one subject object; Based on a preset attenuation coefficient, attenuation processing is performed on the audio data of the non-subject object. 5. The method according to claim 1, wherein the focusing processing on the audio data of the at least one subject object comprises: The audio data of the main object is replaced with preset audio data. 6. The method according to claim 1, characterized in that, after the acquisition of the first image data and the first audio data of the first video data, the method further comprises: determining a target pixel for each subject object based on a user selection input of the at least one subject object; determining target audio data matching each subject object; A mapping relationship between the target pixels of each main object and the target audio data matching each main object is established and stored in the second video data. 7. The method according to claim 6, wherein said determining the target audio data matched with each subject object comprises: From at least one pre-stored audio data, screening target audio data matching the audio characteristics of each subject object; Alternatively, the audio data selected by the user is determined as target audio data matching each subject object. 8. The method according to claim 2, wherein the pixel of at least one main object in the first image data is subjected to focusing processing through a preset object separation network to obtain at least one second image After the data, the method also includes: When at least two of the at least one subject object are selected, perform Gaussian filter processing or grayscale processing on the target pixels of the at least two subject objects based on the first weight value; Playing the second image data processed by Gaussian filtering or grayscale processing; Wherein, the preset Gaussian filter processing coefficients or the preset grayscale processing coefficients of each of the at least one main object correspond to different first weight values. 9. The method according to claim 4, characterized in that, through the preset voice separation network, the audio data matched with at least one subject object in the first audio data is subjected to focus processing to obtain at least one second audio data. After the second audio data, the method also includes: In the case where at least two of the at least one main object are selected, performing sound mixing processing on the target audio data of the at least two main objects based on a second weight value; Play the second audio data that has been mixed; Wherein, the preset attenuation coefficients of each of the at least one main object correspond to different second weight values. 10. The method according to claim 1, wherein the first video data is video data in a recording process or recorded video data. 11. The method according to claim 1, wherein the subject object separation network comprises: Mask R-CNN. 12. The method according to claim 1, wherein the speech separation network comprises: a long short-term memory network (LSTM). 13. An electronic device, characterized in that it comprises: An acquisition module, configured to acquire first image data and first audio data of the first video data; A first focusing module, configured to focus on pixels of at least one subject object in the first image data through a preset object separation network to obtain at least one second image data; The second focusing module is configured to focus on the audio data matched by at least one subject object in the first audio data through a preset voice separation network, to obtain at least one second audio data; The encoding module is configured to perform encoding and compression processing on the second image data and the second audio data to obtain second video data. 14. An electronic device, characterized by comprising a processor, a memory, and a computer program stored on the memory and operable on the processor, when the computer program is executed by the processor, the The steps of the video processing method described in any one of claims 1 to 12.