CN109068166B - Video synthesis method, device, equipment and storage medium - Google Patents

Abstract

The application relates to a video synthesis method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring configuration information of a preset picture layout, wherein the configuration information comprises the display size and the display position of each area in the display container; loading M preset local video pictures into M areas according to the configuration information, and loading N shooting video pictures acquired in real time into N areas, wherein L is a positive integer larger than 1, M and N are both positive integers, and the sum of M and N is equal to L; and rendering the pictures of the M areas and the pictures of the N areas to generate a first playing file. The method has the advantages that the multiple real-time shot videos and the multiple local videos are rendered to generate the video composite picture, a video manufacturer can realize special effect editing of multi-picture display in the same display container without professional skills, the multiple pictures are displayed in the same display container, a video viewer can view the multiple videos at the same time, and the information pickup time is saved.

Description

A video synthesis method, device, equipment and storage medium

technical field

The present application relates to the field of video processing, and in particular, to a video synthesis method, apparatus, device, and storage medium.

Background technique

With the continuous development of the Internet era, people's dissemination of information is no longer limited to text and pictures. Video dissemination is also one of the mainstream dissemination methods today. There are various types of video dissemination, which can be distinguished by the length of time. Video types, for example, short videos of 10 seconds and 15 seconds; long videos of 1 minute to 5 minutes; and live online videos, etc.

At present, the dissemination of videos can be disseminated through video websites, social applications and other platforms. For social applications, most of the videos are recorded and generated by users. Users display or share the content they want to express through videos. Direct and accurate.

However, with the increasing number of videos, the content displayed in the video needs to be expressed accurately and completely by increasing the time, and it takes more time for video producers to process the videos one by one (such as editing, special effects, etc.); Video viewers may need to spend more time watching the complete content of each video, which causes users to spend more time on video processing or video viewing.

SUMMARY OF THE INVENTION

In order to overcome the problems existing in the related art, the present application provides a video synthesis method, apparatus, device and storage medium.

According to a first aspect of the embodiments of the present application, a video synthesis method is provided, including:

Obtaining configuration information of a preset screen layout, wherein the configuration information includes the display size and display position of each area in the display container;

Load M preset local video images into M areas according to the configuration information, and load N shooting video images collected in real time into N areas, where L is a positive integer greater than 1, The M and the N are both positive integers, and the sum of the M and the N is equal to the L;

A first play file is generated after rendering the pictures of the M regions and the pictures of the N regions.

Optionally, before acquiring the configuration information of the preset screen layout, wherein the configuration information includes the display size and display position of each area in the display container, it further includes:

obtain the resolution of the display container;

calculating the aspect ratio of the display container according to the resolution of the display container;

When the aspect ratio is greater than a preset first threshold, confirm that the preset first layout format is the configuration information of the screen layout, wherein the first layout format is to make the L areas in the The display containers are stacked one after the other in the width direction.

Optionally, after calculating the aspect ratio of the display container according to the resolution of the display container, the method further includes:

When the aspect ratio is less than or equal to the preset first threshold, confirm that the preset second layout format is the configuration information of the screen layout, wherein the second layout format is to make the L The regions are stacked one after another in the height direction of the display container.

Optionally, when both the M and the N are 1, the local video picture is loaded in the first area, the shot video picture is loaded in the second area, and the configuration of obtaining a preset picture layout is performed in the Before the information, the configuration information includes the display size and display position of each area in the display container and also includes:

obtaining the resolution of the local video picture and the captured video picture;

According to the resolution of the local video picture and the captured video picture, calculate the width ratio or height ratio of the local video picture and the captured video picture;

When the width ratio or the height ratio is smaller than the preset second threshold, determine that the preset third layout format is the configuration information of the screen layout, wherein the third layout format is to make the first area embedded within the second area.

Optionally, in the loading of M preset local video images into M areas according to the configuration information, and loading N shooting video images collected in real time into N areas, wherein the L is a positive integer greater than 1, the M and the N are both positive integers, and the sum of the M and the N is equal to the L, and also includes:

obtaining the local video picture resolution;

scaling the local video image in equal proportions according to the display size of the M areas in the display container, so that the local video image is completely displayed in the M areas;

The resolution of the captured video picture is determined according to the display size of the N regions in the display container. Optionally, in the generating of the first playback file after rendering the pictures of the M regions and the pictures of the N regions, it further includes:

Decoding the pictures of the M regions and the pictures of the N regions, and generating L pieces of texture information respectively Decoding the pictures of the M regions and the pictures of the N regions, and generating L pieces of texture information respectively ;

According to the configuration information of the preset screen layout, respectively calculate the texture coordinates and vertex coordinates of the L pieces of texture information;

The first play file is generated according to the texture coordinates and the vertex coordinates.

Optionally, in the loading of M preset local video images into M areas according to the configuration information, and loading N shooting video images collected in real time into N areas, wherein the L is a positive integer greater than 1, the M and the N are both positive integers, and after the sum of the M and the N is equal to the L, it also includes:

Obtain the preset special effect screen for shooting video;

Loading the special effect pictures of the shooting video into the N regions.

Optionally, generating a first playback file after rendering the pictures of the M regions and the pictures of the N regions, further comprising:

The images of the M areas and the images of the N areas are superimposed and rendered to obtain the rendered first playback file.

According to a second aspect of the embodiments of the present application, a device for video synthesis is provided, which is used for picture synthesis of L regions in the same display container, including:

an obtaining unit, configured to obtain configuration information of a preset screen layout, wherein the configuration information includes the display size and display position of each area in the display container;

The processing unit is configured to load the M preset local video images into the M areas according to the configuration information, and load the N shooting video images collected in real time into the N areas, wherein the L is a positive integer greater than 1, the M and the N are both positive integers, and the sum of the M and the N is equal to the L;

The execution unit is configured to generate a first play file after rendering the pictures of the M regions and the pictures of the N regions.

Optionally, the video synthesis apparatus further includes:

a first acquiring subunit, configured to acquire the resolution of the display container;

a first processing subunit configured to calculate the aspect ratio of the display container according to the resolution of the display container;

The first execution subunit is configured to confirm that the preset first layout format is the configuration information of the screen layout when the aspect ratio is greater than a preset first threshold, wherein the first layout format is The L regions are sequentially stacked in the width direction of the display container.

Optionally, the video synthesis apparatus further includes:

The second execution subunit is configured to confirm that the preset second layout format is the configuration information of the screen layout when the aspect ratio is less than or equal to the preset first threshold, wherein the first The second layout format is that the L regions are stacked in sequence in the height direction of the display container.

Optionally, the video synthesis apparatus further includes:

a second obtaining subunit, configured to obtain the resolution of the local video picture and the captured video picture;

a second processing subunit, configured to calculate a width ratio or a height ratio of the local video frame and the captured video frame according to the resolution of the local video frame and the captured video frame;

The third execution subunit is configured to, when the width ratio or the height ratio is smaller than a preset second threshold, determine that the preset third layout format is the configuration information of the screen layout, wherein the third layout The format is such that the first area is nested within the second area.

Optionally, the video synthesis apparatus further includes:

a third obtaining subunit, configured to obtain the local video picture resolution;

The third processing subunit is configured to scale the local video picture in equal proportion according to the display size of the M regions in the display container, so that the local video picture is completely displayed in the M regions ;

The fourth execution subunit is configured to determine the resolution of the captured video picture according to the display size of the N regions in the display container.

Optionally, the video synthesis apparatus further includes:

a first decoding subunit, configured to decode the pictures of the M regions and the pictures of the N regions, and respectively generate L pieces of texture information for the pictures of the M regions and the pictures of the N regions Perform decoding to generate L texture information respectively;

a fourth processing subunit, configured to calculate the texture coordinates and vertex coordinates of the L pieces of texture information according to the preset configuration information of the screen layout;

A fifth execution subunit is configured to generate the first play file according to the texture coordinates and the vertex coordinates.

Optionally, the video synthesis apparatus further includes:

a fourth obtaining subunit, configured to obtain a preset special effect picture for shooting video;

The fifth processing subunit is configured to load the special effect picture of the shot video into the N regions.

Optionally, the video synthesis apparatus further includes:

The sixth execution subunit is configured to superimpose and render the pictures of the M regions and the pictures of the N regions to obtain the rendered first play file.

According to a third aspect of the embodiments of the present application, there is provided a video synthesis device, comprising a processor and a memory for storing instructions executable by the processor, where the processor is configured to perform the steps of the above video synthesis method.

According to a fourth aspect of the embodiments of the present application, a non-transitory computer-readable storage medium is provided, when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can perform the steps of the above video synthesis method .

According to a fifth aspect of the embodiments of the present application, a computer program product is provided, including computer program code, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is caused to perform the steps of the above video synthesis method.

The technical solutions provided by the embodiments of the present application may include the following beneficial effects: by acquiring configuration information of a preset screen layout, wherein the configuration information includes the display size and display position of each area in the display container; The configuration information loads M preset local video images into M areas, and loads N shooting video images collected in real time into N areas, where L is a positive integer greater than 1, so The M and the N are both positive integers, and the sum of the M and the N is equal to the L; the first playback file is generated after rendering the pictures of the M regions and the pictures of the N regions . Render multiple real-time shooting videos and multiple local videos to generate video composite images. Video producers do not need professional skills to realize special effects clips for multi-screen display in the same display container. Display multiple images in the same display container, making the video Viewers can watch multiple videos at the same time, saving time in information ingestion.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

FIG. 1 is a flow chart of a video synthesis method according to an exemplary embodiment.

FIG. 2 is a flowchart illustrating a confirmation of a first layout format according to an exemplary embodiment.

FIG. 3 is a flowchart illustrating a third layout format confirmation according to an exemplary embodiment.

FIG. 4 is a schematic diagram of a third layout format according to an exemplary embodiment.

Fig. 5 is a flow chart of cropping a video according to an exemplary embodiment.

FIG. 6 is a flow chart of generating a first play file according to an exemplary embodiment.

FIG. 7 is a flow chart of loading a special effect screen of a shooting video according to an exemplary embodiment.

Fig. 8 is a block diagram of a video synthesis apparatus according to an exemplary embodiment.

Fig. 9 is a structural block diagram of a video synthesis device according to an exemplary embodiment.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as recited in the appended claims.

FIG. 1 is a flow chart of a video synthesis method according to an exemplary embodiment. As shown in FIG. 1 , the video synthesis method is used in a terminal and includes the following steps:

S1100. Acquire configuration information of a preset screen layout, wherein the configuration information includes the display size and display position of each area in the display container;

In the embodiment of the present application, the same display container is divided into two areas, and the display size and display position of the two areas in the display container are determined by the configuration information. Here, the configuration information determines the position and space occupied by each video image on the display screen when the two video data sources are output in the same image. For example: the display container is a display screen with a resolution of 1440*900. The obtained configuration information indicates that the first area is the left half of the screen which occupies 720*900, and the second area is the right half of the screen which occupies 720*900 part. It can be understood that there can be multiple data sources, including M local video images and N real-time shooting images. The corresponding display container is divided into L areas, where L is the sum of M and N. The configuration information determines that each area is displayed in the display. Display size and display position in the container.

S1200. Load M preset local video images into M areas according to the configuration information, and load N shooting video images collected in real time into N areas, where L is a positive value greater than 1 an integer, the M and the N are both positive integers, and the sum of the M and the N is equal to the L;

In step S1100, the size and position of each area on the display screen are determined through the configuration information. The specific loading process is that each area corresponds to a media buffer, and the original data of the cyclically read video is put into the corresponding media buffer for display in the next step.

S1300. Generate a first play file after rendering the pictures of the M regions and the pictures of the N regions.

Through step S1200, the media buffer of each area of the display container is loaded with local video data and real-time shooting data respectively. In this step, the first playback file is obtained by superimposing and rendering the L data. The playback file is used for real-time playback all the way, and the file is written all the way for saving.

Rendering is to draw images to the screen, which can realize the drawing of 3D objects on the screen. Rendering is done through renderers. The renderer reads the video data in the media buffer and completes the display of the video picture on the screen. In the embodiment of the present application, the renderer simultaneously reads the video data in the media buffer of each region, and superimposes the data of each channel to complete the drawing of the composite video picture on the same display container.

In this embodiment of the present application, the screen layout configuration information in the same display container is automatically adjusted according to the resolution of the display container. Please refer to FIG. 2 for the specific process. An implementation flow chart.

As shown in Figure 2, before step S1100, the following steps are also included:

S1101. Obtain the resolution of the display container;

The display container in the embodiment of the present application is the screen of the terminal or a part of the screen of the terminal. To read the resolution of the display container, you can obtain the resolution of the display container in the following ways: Obtain the model of the terminal device, search the database according to the model of the terminal device, and obtain the screen resolution of the terminal device. The resolution of the screen refers to the display on the screen. The number of pixels, the resolution is 1920X1080, which means that the number of pixels displayed in the horizontal direction is 1920px (pixel), and the number of pixels in the vertical direction is 1080px. Or, directly read the specification parameters of the display screen used by the terminal device to obtain the resolution of the terminal screen. In some embodiments, the display container may also be a preset partial area on the display screen.

S1102. Calculate the aspect ratio of the display container according to the resolution of the display container;

Get the width and height of the display container according to the resolution of the display container. Aspect ratio is the ratio of width to height, the width corresponds to the size in the horizontal direction, and the height corresponds to the size in the vertical direction. It is worth noting that the width and height of the mobile terminal are related to the placement state. When calculating the aspect ratio, it is necessary to first judge the placement state of the mobile terminal and determine the dimensions in the horizontal and vertical directions. The width and height of the mobile terminal will change according to the placement state. When calculating the aspect ratio, always consider the size in the horizontal direction as the width of the screen, and the size in the vertical direction as the height of the screen.

S1103. When the aspect ratio is greater than a preset first threshold, confirm that the preset first layout format is the configuration information of the screen layout, wherein the first layout format is to make the L areas in the The display containers are stacked in sequence in the width direction.

In the embodiment of the present application, the aspect ratio is different, and the configuration information corresponding to the automatic conversion of the screen layout is also different. When the aspect ratio is greater than the set threshold, for example, the set threshold is 1, the corresponding first layout format is left-right layout, that is, when the aspect ratio is greater than 1, each area is divided in a left-right manner. In the embodiment of the present application, an average division method is adopted. For example, if the resolution of the display container is 1440*960, the width in the horizontal direction is 1440, the height in the vertical direction is 960, the aspect ratio is 1440/960, the aspect ratio is greater than 1, and both M and N are 1, The display container is evenly divided on the left and right, that is, the local video image is displayed in the area of 720*960 on the left of the screen, and the real-time video image is displayed in the area of 720*960 on the right of the screen. It is easy to understand that left and right can be interchanged.

In the embodiment of the present application, if the aspect ratio is less than or equal to 1, the layout mode is different from that described above, and the second layout mode is adopted, and the second layout mode is the top-bottom layout. Following the previous example, if the mobile terminal is rotated, the resolution is still 1440*960, but the width in the horizontal direction is 960 and the height in the vertical direction is 1440. At this time, the aspect ratio is 960/1440, and the ratio is less than 1. The display container is divided according to the upper and lower way, that is, the real-time shooting video image is displayed in the area of 960*720 in the upper part of the screen, and the real-time shooting video image is displayed in the area of 960*720 in the lower part of the screen. It can be easily understood that up and down can be interchanged.

In some embodiments, not only the screen layout format is automatically selected according to the resolution of the display container, but also the screen layout format is determined according to the resolution of the local video picture and the captured video picture. Please refer to Figure 3 for details. 3 shows an implementation of determining the screen layout format according to the resolution of the local video and the captured video when M and N are both 1, the local video image is loaded in the first area, and the captured video image is loaded into the second area, according to an embodiment of the present application. flow chart of the way.

As shown in Figure 3, before step S1100, the following steps are also included:

S1111, acquiring the resolution of the local video picture and the captured video picture;

Video resolution refers to the pixel value on the width and height of the video screen. Common video resolutions include, 4:3, the resolution is 320*240 or 640*480, 16:9, the resolution is 640*360 , 720*480, 1024*600, 1280*720. The resolution of the video is usually written into the video file as an attribute parameter, and the resolution of the existing video is obtained by reading the resolution parameter in the video file. Since the other video is captured in real time by the video recording device, the resolution of the recorded video depends on the resolution setting on the video recording device, so the resolution of the real-time video is obtained by reading the recording resolution setting on the video recording device. to obtain.

S1112, calculating a width ratio or a height ratio of the local video frame and the captured video frame according to the resolution of the local video frame and the captured video frame;

In some scenarios, the resolution of the local video is much smaller than the resolution of the real-time shooting video. If the two occupy the same area of screen space when the same screen is displayed, it will cause some waste of display space, and the display effect will also be poor. not good. Therefore, by obtaining the resolution of the local video image and the captured video image, compare the width ratio or height ratio of the local video image and the captured video image, and set a different layout from the simple left-right layout or the top-bottom layout according to the ratio.

S1113. When the width ratio or the height ratio is smaller than a preset second threshold, determine that a preset third layout format is the configuration information of the screen layout, wherein the third layout format is to make the first layout format Regions are nested within a second region.

When the width of the local video frame is much smaller than the width of the captured video frame, or the height of the local video frame is much smaller than the height of the captured video frame, the third layout format is set as a frame layout method. The third layout format may be called a picture-in-picture layout format, as shown in Figure 4, that is, the real-time captured video occupies the entire space of the display container, and the local video is displayed in a corner of the display container. If the size of the local video image is too large, it will be displayed in the third layout format. Most of the real-time video image is covered by the local video image, and the display effect is not good. In order to achieve the best display effect, the width of the local video image and the captured video image is The ratio or height ratio is set to a threshold. When it is less than the threshold, the third layout format is adopted. In the embodiment of the present application, the threshold is set to 1/3, that is, when the width of the local video picture is less than 1/3 of the width of the captured video picture, the third layout format is adopted. The third layout format.

After the display container is divided, the resolution of the local video or the captured video is usually different from the resolution of the divided display area. If the size of the video image exceeds the size of the display area, the video image must be processed so that it can be Better display in the display area. Please refer to FIG. 5 for details. FIG. 5 is a flowchart of an implementation manner of processing a local video image and a video image captured in real time according to a display size according to an embodiment of the present application.

As shown in Figure 5, in step S1200, the following steps are also included:

S1201, obtaining the local video picture resolution;

As mentioned above, video resolution refers to the pixel value of the width and height of the video screen. Common video resolutions include: 4:3, with a resolution of 320*240 or 640*480; 16:9, resolution The rate is 640*360, 720*480, 1024*600, 1280*720. The resolution of the video is usually written into the video file as an attribute parameter, and the resolution of the existing video is obtained by reading the resolution parameter in the video file.

S1202, scaling the local video picture in equal proportions according to the display size of the M regions in the display container, so that the local video picture is completely displayed in the M regions;

After obtaining the resolution of the local video, match the display size, keep the original aspect ratio, and perform equal scaling. For example, if the resolution of the local video is 600*600 and the display size is 720*720, the width and height of the original video will be enlarged by 1.2 times.

S1203. Determine the resolution of the captured video image according to the display size of the N regions in the display container.

When the video image is captured in real time, the size of the captured image is limited during capturing, so that the captured video image can be well displayed on the display container when loaded into the display area.

As shown in Figure 6, in step S1300, the following steps are also included:

S1301. Decode the pictures of the M regions and the pictures of the N regions, respectively generate L pieces of texture information, decode the pictures of the M regions and the pictures of the N regions, and generate L pieces of texture information respectively texture information;

Decode the video data buffered in the media buffer corresponding to each region to generate texture information. Textures are the input information that the GPU uses to draw images. In computer graphics, a texture refers to a bitmap that represents details on the surface of an object. Because all textures in Direct3D are bitmaps, you can map any bitmap to the surface of a Direct3D primitive. For example, an application can create objects and make their surfaces appear wood-grained. Textures such as grass, dirt, and rocks can be affixed to the surface of the primitives that make up the mountain, so that you can get a real-looking hillside. Applications can also use textures to create other effects.

S1302, according to the configuration information of the preset screen layout, respectively calculate the texture coordinates and vertex coordinates of the L pieces of texture information;

When drawing an image, not only geometry coordinates but also texture coordinates must be defined for each vertex. Geometry coordinates determine where a vertex is drawn on the screen, while texture coordinates determine which texel in the texture image is assigned to that vertex. The texture coordinates and vertex coordinates of each texture are determined according to the configuration information of the screen layout obtained in the preceding steps.

S1303. Generate the first play file according to the texture coordinates and the vertex coordinates.

Get texture coordinates and vertex coordinates, draw the image through a vertex shader, and provide vertex coordinates and texture coordinates for each vertex. Vertex coordinates determine where on the screen a particular vertex should be rendered. Texture coordinates determine which texture unit in the texture image will be assigned to this vertex. The drawing results are displayed all the way to the user in real time, and all the way to the file for saving.

In some embodiments, some video special effects are added to the video shot in real time. For details, please refer to FIG. 7 , which is a flowchart of an implementation manner of adding video special effects according to an embodiment of the present application.

As shown in Figure 7, after step S1200, the following steps are also included:

S1211. Acquire a preset special effect picture for shooting a video;

Add video special effects to the video shot in real time. The video special effect files are saved locally. There are many kinds of video special effects, corresponding to multiple local files. Users can select the video effects they want to add before starting video recording.

S1212. Load the special effect picture of the shooting video into the N regions.

Load the selected video effect file into N areas, and when displayed, superimpose it on the captured video image captured in real time to achieve the preset effect.

Fig. 8 is a block diagram of a video synthesis apparatus according to an exemplary embodiment. Referring to FIG. 8 , the apparatus includes an acquisition unit 2100 , a processing unit 2200 , and an execution unit 2300 .

The obtaining unit 2100 is configured to obtain configuration information of a preset screen layout, wherein the configuration information includes the display size and display position of each area in the display container;

The processing unit 2200 is configured to load M preset local video images into M areas according to the configuration information, and load N shooting video images collected in real time into N areas, wherein the L is a positive integer greater than 1, the M and the N are both positive integers, and the sum of the M and the N is equal to the L;

The execution unit 2300 is configured to generate a first play file after rendering the pictures of the M regions and the pictures of the N regions.

In some embodiments, the video synthesis apparatus further includes: a first acquisition subunit, a first processing subunit, and a first execution subunit, wherein the first acquisition subunit is configured to acquire the resolution of the display container; a first processing subunit, configured to calculate the aspect ratio of the display container according to the resolution of the display container; a first execution subunit, configured to calculate the aspect ratio when the aspect ratio is greater than a preset first threshold , confirming that the preset first layout format is the configuration information of the screen layout, wherein the first layout format is that the L regions are sequentially stacked in the width direction of the display container.

In some embodiments, the video synthesis apparatus further includes: a second execution subunit, configured to confirm that the preset second layout format is the preset second layout format when the aspect ratio is less than or equal to the preset first threshold. The configuration information of the screen layout, wherein the second layout format is that the L regions are stacked in sequence in the height direction of the display container.

In some embodiments, the video synthesis apparatus further includes: a second acquisition subunit, a second processing subunit, and a third execution subunit, wherein the second acquisition subunit is configured to acquire the local video picture and the The resolution of the captured video picture; the second processing subunit is configured to calculate the width ratio or height ratio of the local video picture and the captured video picture according to the local video picture and the resolution of the captured video picture The third execution subunit is configured to, when the width ratio or the height ratio is smaller than the preset second threshold, determine that the preset third layout format is the configuration information of the screen layout, wherein the third The layout format is such that the first area is nested within the second area.

In some embodiments, the video synthesis apparatus further includes: a third acquisition subunit, a third processing subunit, and a fourth execution subunit, wherein the third acquisition subunit is configured to acquire the local video picture resolution; The third processing subunit is configured to scale the local video picture in equal proportion according to the display size of the M regions in the display container, so that the local video picture is completely displayed in the M regions ; a fourth execution subunit, configured to determine the resolution of the captured video picture according to the display size of the N regions in the display container.

In some embodiments, the video synthesis apparatus further includes: a first decoding subunit, a fourth processing subunit, and a fifth execution subunit, wherein the first decoding subunit is configured to compare the pictures of the M regions with The pictures of the N regions are decoded to generate L pieces of texture information respectively, and the pictures of the M regions and the pictures of the N regions are decoded to respectively generate L pieces of texture information; the fourth processing subunit is configured To calculate the texture coordinates and vertex coordinates of the L pieces of texture information respectively according to the configuration information of the preset screen layout; a fifth execution subunit is configured to generate the texture coordinates and the vertex coordinates according to the texture coordinates and the vertex coordinates First play file.

In some embodiments, the video synthesis apparatus further includes: a fourth acquisition subunit and a fifth processing subunit, wherein the fourth acquisition subunit is configured to acquire a preset special effect picture for shooting video; and a fifth processing subunit, is configured to load the shooting video special effect picture to the N regions.

In some embodiments, the video synthesizing apparatus further includes: a sixth execution subunit, configured to superimpose and render the pictures of the M regions and the pictures of the N regions, to obtain the rendered first playback file .

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 9 is a structural block diagram of a video synthesis device 900 according to an exemplary embodiment, where the video synthesis device 900 is a mobile terminal for video synthesis. For example, video synthesis device 900 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, and the like.

9, a video synthesis device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input/output (I/O) interface 912, a sensor component 914 , and communication component 916 .

The processing component 902 generally controls the overall operation of the video composition device 900, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

Memory 904 is configured to store various types of data to support operation at video composition apparatus 900 . Examples of such data include instructions for any application or method operating on the video composition device 900, contact data, phonebook data, messages, pictures, videos, and the like. Memory 904 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power supply component 906 provides power to the various components of video synthesis device 900 . Power supply components 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to video synthesis device 900 .

Multimedia component 908 includes a screen that provides an output interface between the video synthesis device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 908 includes a front-facing camera and/or a rear-facing camera. When the video synthesis apparatus 900 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 910 is configured to output and/or input audio signals. For example, audio component 910 includes a microphone (MIC) that is configured to receive external audio signals when video synthesis device 900 is in operating modes, such as call mode, recording mode, and speech recognition mode. The received audio signal may be further stored in memory 904 or transmitted via communication component 916 . In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

The I/O interface 912 provides an interface between the processing component 902 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 914 includes one or more sensors for providing status assessment of various aspects of video synthesis device 900 . For example, the sensor component 914 can detect the on/off state of the video composition device 900, the relative positioning of components, such as the display and keypad of the video composition device 900, the sensor component 914 can also detect the video composition device 900 or the video Changes in the position of a component of the composite device 900 , presence or absence of user contact with the video composite device 900 , video composite device 900 orientation or acceleration/deceleration and changes in the video composite device 900 temperature. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 916 is configured to facilitate wired or wireless communication between video composition device 900 and other devices. The video synthesis device 900 may access a wireless network based on a communication standard, such as WiFi, a carrier network (eg, 2G, 3G, 4G, or 5G), or a combination thereof. In one exemplary embodiment, the communication component 916 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, video synthesis device 900 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field Programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for carrying out the above method.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory 904 including instructions, executable by the processor 920 of the video synthesis apparatus 900 to accomplish the above method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

In some embodiments, a computer program product is provided, comprising computer program code, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the steps of the above-described video synthesis method.

Other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of this application that follow the general principles of this application and include common knowledge or conventional techniques in the technical field not disclosed in this application . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise structures described above and illustrated in the accompanying drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (16)

1. A video composition method for picture composition of L regions in the same display container, comprising:

acquiring configuration information of a preset picture layout, wherein the configuration information comprises a display size and a display position of each area in the display container, which are obtained by automatically adjusting according to the resolution of the display container;

loading M preset local video pictures into M areas according to the configuration information, and loading N shooting video pictures acquired in real time into N areas, wherein L is a positive integer larger than 1, M and N are both positive integers, and the sum of M and N is equal to L;

rendering the pictures of the M areas and the pictures of the N areas to generate a first playing file;

wherein, generating a first playing file after rendering the pictures of the M regions and the pictures of the N regions further includes:

decoding the pictures of the M areas and the pictures of the N areas to respectively generate L pieces of texture information;

respectively calculating texture coordinates and vertex coordinates of the L pieces of texture information according to the preset configuration information of the picture layout;

and generating the first playing file according to the texture coordinates and the vertex coordinates.

2. The video synthesis method according to claim 1, further comprising, before the obtaining configuration information of the preset screen layout:

acquiring the resolution of the display container;

calculating an aspect ratio of the display container according to a resolution of the display container;

and when the aspect ratio is larger than a preset first threshold value, confirming that a preset first layout format is configuration information of the picture layout, wherein the first layout format is that the L areas are sequentially stacked in the width direction of the display container.

3. The video synthesis method according to claim 2, further comprising, after the calculating the aspect ratio of the display container according to the resolution of the display container:

and when the aspect ratio is smaller than or equal to the preset first threshold value, confirming that a preset second layout format is configuration information of the picture layout, wherein the second layout format is that the L areas are sequentially stacked in the height direction of the display container.

4. The video synthesis method according to claim 1, wherein when M and N are both 1, the local video picture is loaded in a first area, the captured video picture is loaded in a second area, and before the obtaining of the configuration information of the preset picture layout, the method further comprises:

acquiring the resolution ratios of the local video picture and the shot video picture;

calculating the width ratio or the height ratio of the local video picture and the shot video picture according to the resolutions of the local video picture and the shot video picture;

and when the width ratio or the height ratio is smaller than a preset second threshold value, determining a preset third layout format as configuration information of the picture layout, wherein the third layout format is that the first area is nested in the second area.

5. The video synthesis method according to claim 1, wherein the loading M preset local video pictures into M regions and loading N captured video pictures collected in real time into N regions according to the configuration information further comprises:

acquiring the local video picture resolution;

scaling the local video picture according to the display sizes of the M areas in the display container to enable the local video picture to be completely displayed in the M areas;

and determining the resolution of the shot video picture according to the display sizes of the N areas in the display container.

6. The video synthesis method according to claim 1, wherein after the loading M preset local video pictures into M regions according to the configuration information and loading N captured video pictures collected in real time into N regions, the method further comprises:

acquiring a preset special effect picture of a shot video;

and loading the shot video special effect picture to the N areas.

7. The video synthesis method according to claim 1, wherein the rendering the M regions and the N regions generates a first playback file, and further comprising:

and superposing and rendering the pictures of the M areas and the pictures of the N areas to obtain the rendered first playing file.

8. A video compositing apparatus for picture compositing of L regions in the same display container, comprising:

an acquisition unit configured to acquire configuration information of a preset screen layout, wherein the configuration information includes a display size and a display position of each region in the display container, which are automatically adjusted according to a resolution of the display container;

the processing unit is configured to load M preset local video pictures into M areas according to the configuration information, and load N shooting video pictures collected in real time into N areas, wherein L is a positive integer greater than 1, M and N are both positive integers, and the sum of M and N is equal to L;

the execution unit is configured to render the pictures of the M areas and the pictures of the N areas and then generate a first playing file;

wherein, the device still includes:

a first decoding subunit configured to decode the pictures of the M regions and the pictures of the N regions, and generate L pieces of texture information, respectively;

a fourth processing subunit, configured to calculate texture coordinates and vertex coordinates of the L pieces of texture information, respectively, according to the configuration information of the preset screen layout;

a fifth execution subunit configured to generate the first playback file according to the texture coordinates and the vertex coordinates.

9. The video compositing device of claim 8, wherein said device further comprises:

a first acquisition subunit configured to acquire a resolution of the display container;

a first processing subunit configured to calculate an aspect ratio of the display container according to a resolution of the display container;

a first execution subunit configured to, when the aspect ratio is greater than a preset first threshold, confirm that a preset first layout format is configuration information of the screen layout, where the first layout format is such that the L regions are sequentially stacked in a width direction of the display container.

10. The video compositing device of claim 9, wherein said device further comprises:

a second execution subunit configured to, when the aspect ratio is less than or equal to the preset first threshold, confirm a preset second layout format as configuration information of the screen layout, wherein the second layout format is such that the L regions are sequentially stacked in a height direction of the display container.

11. The video compositing apparatus according to claim 8, wherein when both M and N are 1, the local video picture is loaded in a first area and the captured video picture is loaded in a second area, said apparatus further comprising:

a second acquisition subunit configured to acquire resolutions of the local video picture and the captured video picture;

a second processing subunit configured to calculate a width ratio or a height ratio of the local video picture to the captured video picture according to resolutions of the local video picture and the captured video picture;

a third execution subunit, configured to determine, when the width ratio or the height ratio is smaller than a preset second threshold, a preset third layout format as configuration information of the screen layout, where the third layout format is to make the first area be nested in the second area.

12. The video compositing device of claim 8, wherein said device further comprises:

a third obtaining subunit configured to obtain the local video picture resolution;

a third processing subunit, configured to perform equal scaling on the local video picture according to the display sizes of the M regions in the display container, so that the local video picture is completely displayed in the M regions;

a fourth execution subunit configured to determine a resolution of the captured video picture according to display sizes of the N regions in the display container.

13. The video compositing device of claim 8, wherein said device further comprises:

a fourth acquiring subunit configured to acquire a preset shooting video special effect picture;

a fifth processing subunit configured to load the captured video special effect picture into the N regions.

14. The video compositing device of claim 8, wherein said device further comprises:

and the sixth execution subunit is configured to perform superposition rendering on the pictures of the M regions and the pictures of the N regions to obtain the rendered first play file.

15. A video compositing device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the executable instructions, when executed by the processor, cause the processor to be configured as the steps of the video compositing method of any of claims 1-7.

16. A non-transitory computer readable storage medium having instructions which, when executed by a processor of a mobile terminal, enable the mobile terminal to perform a video compositing method, the method comprising the steps of the video compositing method according to any of claims 1 to 7.

Images