CN107622495A - Image processing method and device, electronic device, and computer-readable storage medium - Google Patents

Abstract

The invention discloses an image processing method for an electronic device. The image processing method comprises the following steps: acquiring a plurality of frames of three-dimensional scene images and depth images of a current user at a preset frequency; processing the multi-frame scene images and the multi-frame depth images to segment the character areas and the background areas except the character areas in each frame scene image to obtain multi-frame background area images, wherein the multi-frame background area images correspond to multi-frame preset three-dimensional images; and fusing each frame of preset three-dimensional image with the corresponding background area image to obtain a plurality of frames of combined images so as to output a video image. The invention also discloses an image processing device, an electronic device and a computer readable storage medium. According to the image processing method and device, the electronic device and the computer readable storage medium, the preset three-dimensional image replaces the character region in the three-dimensional scene image to obtain the multi-frame combined image, and the multi-frame combined image is formed into the video image to be output, so that the interestingness of image fusion is increased.

Description

Image processing method and device, electronic device, and computer-readable storage medium

technical field

The present invention relates to the technical field of image processing, in particular to an image processing method and device, an electronic device, and a computer-readable storage medium.

Background technique

The existing image fusion usually fuses the user's portrait with the background image, but this fusion method is less interesting.

Contents of the invention

Embodiments of the present invention provide an image processing method, an image processing device, an electronic device, and a computer-readable storage medium.

The image processing method in the embodiment of the present invention is used in an electronic device, and the image processing method includes:

Collect multi-frame 3D scene images and depth images of the current user at a preset frequency;

Processing multiple frames of the scene image and multiple frames of the depth image to segment the character area in each frame of the scene image and the background area except the character area to obtain multiple frames of background area images, and multiple frames of the background area The image corresponds to a plurality of frames of predetermined three-dimensional images; and

Merging each frame of the predetermined three-dimensional image with the corresponding background area image to obtain a multi-frame merged image to output a video image.

An image processing device according to an embodiment of the present invention is used for an electronic device, and the image processing device includes an imaging device and a processor. The imaging device is used to collect multiple frames of three-dimensional scene images and depth images of the current user at a preset frequency. The processor is configured to process multiple frames of the scene image and multiple frames of the depth image to segment the character area in each frame of the scene image and the background area except the character area to obtain multiple frames of background area images, The frame of the background area image corresponds to multiple frames of predetermined three-dimensional images, and each frame of the predetermined three-dimensional image is fused with the corresponding background area image to obtain multiple frames of merged images to output a video image.

An electronic device according to an embodiment of the present invention includes one or more processors, memory and one or more programs. Wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs include instructions for executing the above image processing method.

The computer-readable storage medium according to the embodiment of the present invention includes a computer program used in combination with an electronic device capable of taking pictures, and the computer program can be executed by a processor to implement the above-mentioned image processing method.

The image processing method, image processing device, electronic device, and computer-readable storage medium in the embodiments of the present invention, after acquiring the three-dimensional scene image and the depth image, use the depth information to segment the characters and the background of each frame of the scene image, so that the segmented The three-dimensional character area and the three-dimensional background area are more accurate, and the three-dimensional background area image obtained by each frame segmentation is fused with the corresponding predetermined three-dimensional image, that is, the predetermined three-dimensional image replaces the current user's character area in the scene image, and multiple The 3D merged image of the predetermined 3D image and the 3D background area image is fused, and the 3D merged image of multiple frames can also form a video image output. In this way, the fun of image fusion can be increased and the user experience can be improved.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 2 is a schematic structural diagram of an electronic device according to some embodiments of the present invention.

Fig. 3 is a schematic diagram of an image processing device according to some embodiments of the present invention.

Fig. 4 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 5 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 6(a) to Fig. 6(e) are schematic diagrams of scenes of structured light measurement according to an embodiment of the present invention.

Fig. 7(a) and Fig. 7(b) are schematic diagrams of scenes of structured light measurement according to an embodiment of the present invention.

Fig. 8 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 9 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 10 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 11 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 12 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 13 is a schematic diagram of an image processing device according to some embodiments of the present invention.

Figure 14 is a schematic diagram of an electronic device according to some embodiments of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

Please refer to FIGS. 1 to 2 together. The image processing method according to the embodiment of the present invention is used in an electronic device 1000 . Image processing methods include:

03: Collect multi-frame 3D scene images and depth images of the current user at a preset frequency;

05: Process multiple frames of scene images and multiple frames of depth images to segment the character area in each frame of the scene image and the background area except the character area to obtain multiple frames of background area images, and the multiple frames of background area images correspond to multiple frames of predetermined three-dimensional images; and

07: Merge each frame of the predetermined three-dimensional image with the corresponding background area image to obtain a multi-frame merged image to output a video image.

Referring to FIG. 3 , the image processing method of the embodiment of the present invention can be implemented by the image processing apparatus 100 of the embodiment of the present invention. The image processing device 100 according to the embodiment of the present invention is used in an electronic device 1000 . The image processing apparatus 100 includes an imaging device 10 and a processor 20 . Step 03 may be implemented by the imaging device 10 , and Step 05 and Step 07 may be implemented by the processor 20 .

That is to say, the imaging device 10 can be used to collect multiple frames of three-dimensional scene images and depth images of the current user at a preset frequency; the processor 20 can be used to process multiple frames of scene images and multiple frames of depth images to segment each frame of scene images The character area and the background area except the character area to obtain multiple frames of background area images, the multiple frames of background area images correspond to multiple frames of predetermined three-dimensional images, and each frame of predetermined three-dimensional images is fused with the corresponding background area image to obtain a multi-frame merged image to output video images.

Wherein, the preset frequency refers to the frame rate at which the imaging device 10 collects images per second, and the value of the frame rate may be 30 frames per second, 60 frames per second, 120 frames per second, etc. The higher the frame rate, the smoother the video image.

The background area image is obtained by dividing the three-dimensional scene image by the person area and the background area, therefore, the background area image is also a three-dimensional image.

In some embodiments, the predetermined three-dimensional image includes at least one of three-dimensional virtual characters, three-dimensional real characters, and three-dimensional animals and plants. The three-dimensional real person excludes the current user himself. Wherein, the three-dimensional virtual character can be a three-dimensional animation character, such as Mario, Conan, Big Head Son, Crayon Shin-chan, etc.; the three-dimensional real character can be a famous figure in three-dimensional image, such as Audrey Hepburn, Mr. Bean, Harry Potter, etc., the three-dimensional animals and plants can be three-dimensional animation animals or plants, such as Mickey Mouse, Donald Duck, pea shooter and so on.

The image processing device 100 according to the embodiment of the present invention can be applied to the electronic device 1000 according to the embodiment of the present invention. That is to say, the electronic device 1000 according to the embodiment of the present invention includes the image processing device 100 according to the embodiment of the present invention.

In some embodiments, the electronic device 1000 includes a mobile phone, a tablet computer, a laptop computer, a smart bracelet, a smart watch, a smart helmet, smart glasses, and the like.

After the image processing method, the image processing device 100 and the electronic device 1000 according to the embodiment of the present invention acquire the three-dimensional scene image and the depth image, the characters and the background of each frame of the scene image are segmented according to the depth information, so that the segmented three-dimensional character area And the three-dimensional background area is more accurate, and the three-dimensional background area image obtained by each frame segmentation is fused with the corresponding predetermined three-dimensional image, that is, the predetermined three-dimensional image replaces the character area of the current user in the scene image, and multiple frames of predetermined three-dimensional images are obtained The 3D merged image fused with the 3D background area image, and the multi-frame 3D merged image can also form a video image output, which can increase the fun of image fusion and improve user experience. In addition, since the merged image does not contain the actual portrait of the user, the privacy of the user can be protected to a certain extent.

Please refer to FIG. 4. In some embodiments, step 03 collects multiple frames of the current user's two-dimensional scene image and depth image at a preset frequency, including:

031: Shoot the current user at a preset frequency to obtain multiple frames of two-dimensional images;

032: Project structured light to the current user;

033: Take multiple frames of structured light images modulated by the current user at a preset frequency; and

034: Demodulate the phase information corresponding to each pixel of each frame of the structured light image to obtain a multi-frame depth image; and

035: Process multiple frames of two-dimensional images and multiple frames of depth images to obtain multiple frames of three-dimensional scene images.

Please refer to FIG. 3 again. In some embodiments, the image processing apparatus 100 includes an imaging device 10 . The imaging device 10 includes a visible light camera 11 and a depth image acquisition component 12 . The depth image acquisition component 12 includes a structured light projector 121 and a structured light camera 122 . Step 031 can be realized by the visible light camera 11 , step 032 can be realized by the structured light projector 121 , and step 033 , step 034 and step 035 can be realized by the structured light camera 122 .

That is to say, the visible light camera 11 can be used to shoot the current user at a preset frequency to obtain multiple frames of two-dimensional images; the structured light projector 121 can be used to project structured light to the current user; the structured light camera 122 can be used to shoot at a preset frequency Multiple frames of structured light images modulated by the current user, demodulate the phase information corresponding to each pixel of each frame of structured light images to obtain multiple frames of depth images, and process multiple frames of two-dimensional images and multiple frames of depth images to obtain multiple frames of three-dimensional images scene image.

Specifically, the visible light camera 11 captures a two-dimensional image of the current user, and the two-dimensional image is a grayscale image or a color image. After the structured light projector 121 projects a certain pattern of structured light onto the current user's face and body, a structured light image modulated by the current user will be formed on the surface of the current user's face and body. The structured light camera 122 captures multiple frames of modulated structured light images at a preset frame rate, and then demodulates each frame of structured light images to obtain a depth image corresponding to the frame of structured light images. After the light image is demodulated, a multi-frame depth image can be obtained. Among them, the pattern of structured light can be laser stripes, gray codes, sinusoidal stripes, non-uniform speckle, etc. The depth image representation contains the depth information of each person or object in the scene where the current user is. The scene range of the 2D image is the same as that of the depth image, and for each pixel in the 2D image, the depth information corresponding to the pixel can be found in the depth image. In this way, the processor 20 can perform three-dimensional modeling on the scene captured by the structured light camera 122 according to the depth information summarized in the depth image, and then combine the color information of the two-dimensional image to fill in the color of the three-dimensional modeled scene to obtain a three-dimensional Colored scene image.

It should be noted that, in a specific embodiment of the present invention, the visible light camera 11 and the depth image acquisition component 12 should use the same preset frequency to collect two-dimensional images and depth images respectively. In this way, three-dimensional multi-frame scene images and multi-frame The one-to-one correspondence between the frame depth images facilitates the fusion processing of the predetermined three-dimensional image and the background area image in step 07.

Please refer to FIG. 5 , in some embodiments, the step 034 of demodulating the phase information corresponding to each pixel of each frame of structured light image to obtain multiple frames of depth images includes:

0341: Demodulate the phase information corresponding to each pixel in each frame of structured light image;

0342: Convert phase information to depth information; and

0343: Generate a depth image from depth information.

Please refer to FIG. 2 again. In some implementation manners, Step 0341 , Step 0342 and Step 0343 can all be implemented by the structured light camera 122 .

That is to say, the structured light camera 122 can be further used to demodulate the phase information corresponding to each pixel in each frame of the structured light image, convert the phase information into depth information, and generate a depth image according to the depth information.

Specifically, compared with unmodulated structured light, the phase information of the modulated structured light has changed, and the structured light presented in the structured light image is the structured light after distortion, wherein the changed phase information It can represent the depth information of the object. Therefore, the structured light camera 122 first demodulates the phase information corresponding to each pixel in each frame of the structured light image, and then calculates the depth information according to the phase information, thereby obtaining the depth image corresponding to the frame of the structured light image.

In order to make those skilled in the art more clearly understand the process of collecting the depth image of the current user's face and body based on structured light, the following uses a widely used grating projection technology (stripe projection technology) as an example to illustrate its specific principles . Among them, grating projection technology belongs to surface structured light in a broad sense.

As shown in Figure 6(a), when using surface structured light projection, firstly generate sinusoidal fringes through computer programming, project the sinusoidal fringes to the measured object through the structured light projector 121, and then use the structured light camera 122 to capture the fringes The degree of curvature modulated by the object is then demodulated to obtain the phase of the curved fringe, and then the phase is converted into depth information to obtain a depth image. In order to avoid the problem of error or error coupling, it is necessary to calibrate the parameters of the depth image acquisition component 12 before using structured light for depth information acquisition. The calibration includes geometric parameters (for example, the relative relationship between the structured light camera 122 and the structured light projector 121). position parameters, etc.), the internal parameters of the structured light camera 122 and the internal parameters of the structured light projector 121, etc.

Specifically, in the first step, the computer is programmed to generate sinusoidal fringes. Since the distorted fringes need to be used to obtain the phase later, for example, the four-step phase shifting method is used to obtain the phase, so four fringes with a phase difference of π/2 are generated here, and then the structured light projector 121 time-sharing projects the four fringes to the measured object (the mask shown in FIG. 6( a )), the structured light camera 122 collects the picture on the left side of FIG. 6( b ), and at the same time reads the fringes of the reference surface shown on the right side of FIG. 6( b ).

The second step is to perform phase recovery. The structured light camera 122 calculates the modulated phase according to the collected four modulated fringe patterns (ie structured light images), and the obtained phase pattern at this time is a truncated phase pattern. Because the result obtained by the four-step phase-shift algorithm is calculated by the arctangent function, the modulated phase of the structured light is limited to [-π, π], that is, whenever the modulated phase exceeds [-π ,π], which will start all over again. The resulting main value of the phase is shown in Fig. 6(c).

Wherein, during the phase recovery process, transition elimination processing is required, that is, the truncated phase is restored to a continuous phase. As shown in Figure 6(d), the modulated continuous phase map is on the left, and the reference continuous phase map is on the right.

The third step is to subtract the modulated continuous phase from the reference continuous phase to obtain the phase difference (that is, phase information), which represents the depth information of the measured object relative to the reference plane, and then substitute the phase difference into the conversion of phase and depth formula (the parameters involved in the formula have been calibrated), the three-dimensional model of the object to be measured can be obtained as shown in Figure 6(e).

It should be understood that, in practical applications, according to different specific application scenarios, the structured light used in the embodiments of the present invention may be other arbitrary patterns besides the above-mentioned grating.

As a possible implementation manner, the present invention may also use speckle structured light to collect depth information of the current user.

Specifically, the method for obtaining depth information by speckle structured light is to use a substantially flat diffraction element, which has a relief diffraction structure with a specific phase distribution, and a stepped relief structure with two or more concavo-convex cross sections. The thickness of the substrate in the diffraction element is approximately 1 micron, and the height of each step is not uniform, and the height can range from 0.7 micron to 0.9 micron. The structure shown in Fig. 7(a) is the partial diffraction structure of the collimating beam splitting element of this embodiment. Fig. 7(b) is a cross-sectional side view along section A-A, and the units of the abscissa and ordinate are both micrometers. The speckle pattern generated by speckle structured light is highly random, and the pattern will change with the distance. Therefore, before using speckle structured light to obtain depth information, it is first necessary to calibrate the speckle pattern in space, for example, within the range of 0 to 4 meters from the structured light camera 122, take a reference plane every 1 cm, After the calibration is completed, 400 speckle images are saved. The smaller the calibration interval, the higher the accuracy of the acquired depth information. Subsequently, the structured light projector 121 projects the speckle structured light onto the object under test (that is, the current user), and the height difference of the surface of the object under test changes the speckle pattern of the speckle structured light projected on the object under test. After the structured light camera 122 shoots the speckle pattern projected on the object to be measured (that is, the structured light image), the cross-correlation operation is performed on the speckle pattern and the 400 speckle images saved after previous calibration, and then 400 correlation images are obtained. degree image. The position of the measured object in the space will show a peak on the correlation image, and the depth information of the measured object can be obtained by superimposing the above peaks and interpolating.

Since the ordinary diffraction element diffracts the light beam to obtain multiple beams of diffracted light, but the intensity of each beam of diffracted light varies greatly, and the risk of damage to human eyes is also large. Even if the diffracted light is diffracted twice, the uniformity of the obtained beam is low. Therefore, the projection effect of the light beam diffracted by the common diffraction element on the measured object is relatively poor. In this embodiment, a collimating beam-splitting element is used, which not only has the function of collimating the uncollimated beam, but also has the function of splitting light, that is, the uncollimated light reflected by the mirror passes through the collimating beam-splitting element and then Multiple collimated beams are emitted from different angles, and the cross-sectional areas of the emitted multiple collimated beams are approximately equal, and the energy flux is approximately equal, so that the projection effect of the scattered light after the diffraction of the beam is better. At the same time, the emitted laser light is dispersed to each beam, which further reduces the risk of damage to human eyes. Compared with other evenly arranged structured light, when the speckle structured light achieves the same collection effect, the speckle structured light consumes less lower power.

Please refer to FIG. 8 , in some implementations, step 05 processes the multi-frame scene image and the multi-frame depth image to segment the character area in each frame scene image and the background area except the character area to obtain the background area image including:

051: Identify the face area in each frame of the scene image;

052: Obtain the depth information corresponding to the face area from the scene image of the frame or the depth image corresponding to the scene image of the frame;

053: Determine the depth range of the character area according to the depth information of the face area;

054: Determine the character area connected to the face area and falling within the depth range according to the depth range of the character area;

055: Determine the background area image according to the character area and the frame scene image.

Please refer to FIG. 2 again, in some implementation manners, step 051 , step 052 , step 053 , step 054 and step 055 can all be implemented by the processor 20 .

That is to say, the processor 20 can be further used to identify the face area in each frame of the scene image, and obtain the depth information corresponding to the face area from the frame of the scene image or the depth image corresponding to the frame of the scene image, according to The depth information of the face area determines the depth range of the character area, determines the character area connected to the face area and falls within the depth range according to the depth range of the character area, and determines the background area image according to the character area and the frame scene image .

Specifically, firstly, the trained deep learning model can be used to identify the face area in each frame of scene image, and then, since each frame of three-dimensional scene image contains not only color information, but also depth information, it can be directly based on the The depth information of the human face area is obtained from the three-dimensional scene images of the frame, or the depth information of the human face area in each frame of the scene image may be determined according to the corresponding relationship between the depth image and the two-dimensional image. Since the face area includes features such as nose, eyes, ears, and lips, the depth data corresponding to each feature in the face area in the three-dimensional scene image or depth image is different. When using the image acquisition component 12, in the depth image captured by the depth image acquisition component 12, the depth data corresponding to the nose may be smaller, while the depth data corresponding to the ear may be larger. Therefore, the above-mentioned depth information of the face area may be a value or a range of values. Wherein, when the depth information of the face area is a value, the value may be obtained by taking an average value of the depth data of the face area; or, may be obtained by taking a median value of the depth data of the face area.

Since the character area includes the face area, that is to say, the character area and the face area are in a certain depth range, therefore, after the processor 20 determines the depth information of the face area, it can The depth range of the character area is set, and then the character area falling within the depth range and connected to the face area is extracted according to the depth range of the character area. After the character area is determined, the part of the scene image excluding the character area is the part of the background area, and the processor 20 extracts the part of the scene image excluding the character area to obtain the background area image.

In this way, the person area and the background area can be segmented from each frame of the scene image according to the depth information to obtain multiple frames of background area images. Since the acquisition of depth information is not affected by factors such as illumination and color temperature in the environment, the extracted image of the background area is more accurate.

Referring to Fig. 9, in some embodiments, the image processing method also includes the following steps:

061: Process each frame of the scene image to obtain a full-field edge image of each frame of the scene image; and

062: Correct the background area image corresponding to the full-field edge image of each frame according to the full-field edge image of the frame.

Please refer to FIG. 2 again, in some implementation manners, both step 061 and step 062 may be implemented by the processor 20 .

That is to say, the processor 20 is further configured to process each frame of the scene image to obtain a full-field edge image of each frame of scene image, and modify the background area image corresponding to the frame of the full-field edge image according to each frame of the full-field edge image.

The processor 20 first performs edge extraction on each frame of the scene image to obtain multiple frames of full-field edge images, wherein the edge lines in the full-field edge image include edge lines of the current user and background objects in the scene where the current user is located. Specifically, edge extraction can be performed on each frame of the scene image through the Canny operator. The core of the algorithm for edge extraction by Canny operator mainly includes the following steps: first, the scene image is convolved with a 2D Gaussian filter template to eliminate noise; then, the gradient value of the gray level of each pixel is obtained by using the differential operator, and Calculate the gradient direction of the gray level of each pixel according to the gradient value, through the gradient direction, you can find the adjacent pixels of the corresponding pixel along the gradient direction; then, traverse each pixel, if the gray value of a pixel is the same as the two before and after the gradient direction If the gray value of the adjacent pixel is not the largest, then this pixel is considered not to be an edge point. In this way, the pixel points at the edge position in the scene image can be determined, so as to obtain the edge image of the whole field after edge extraction.

Each frame of the scene image corresponds to a frame of the full-field edge image. Similarly, each frame of the scene image corresponds to a frame of the background area image. Therefore, the full-field edge image and the background area image are in one-to-one correspondence. After the processor 20 acquires the edge image of the entire field, it corrects the background region image corresponding to the edge image of the entire field according to the edge image of the entire field. The corrected person area determines the final background area image. It can be understood that the character area is obtained by merging all the pixels in the scene image that are connected to the face area and fall into the set depth range. In some scenes, there may be some pixels that are connected to the face area and fall into the depth range. objects within the range. Therefore, the full-field edge map can be used to correct the character area to obtain a more accurate character area, and then the background area can be determined based on the more accurate character area. In this way, the final image of the background region is also more accurate.

Further, the processor 20 may also perform a secondary correction on the corrected character area, for example, may perform dilation processing on the corrected character area, and expand the character area to preserve edge details of the character area. Subsequently, the processor 20 further determines the background area according to the more accurate person area, and the final image of the background area is more accurate.

Please refer to Fig. 10, in some embodiments, the image processing method of the embodiment of the present invention also includes:

063: Process each frame of the scene image and the depth image to extract the current user's motion information; and

064: Render the predetermined three-dimensional image according to the action information so that each frame of the predetermined three-dimensional image follows the current user's action;

Step 07. The step of merging the predetermined three-dimensional image of each frame with the corresponding background area image to obtain a combined image to output a video image includes:

071: Merge the rendered predetermined three-dimensional image of each frame with the corresponding background area image to obtain a merged image to output a video image.

Please refer to FIG. 2 again, in some implementation manners, step 063 , step 064 and step 071 may be implemented by the processor 20 . That is to say, the processor 20 can be used to process each frame of the scene image and the depth image to extract the action information of the current user, render the predetermined three-dimensional image according to the action information so that each frame of the predetermined three-dimensional image follows the current user's action, and convert each frame The rendered predetermined three-dimensional image is fused with the corresponding background area image to obtain a combined image to output a video image.

Wherein, the action information includes at least one of the current user's facial expressions and body movements. That is to say, the action information may be the current user's expression, or the current user's body movement, or the current user's expression and body movement.

Specifically, in step 05, the processor 20 has identified the face area of each frame of the scene image, and has segmented the character area and the background area, therefore, when executing step 063, the processor 20 processes each frame of the face area to identify the current user's expression, and process the character area in each frame of the scene image to obtain information about the current user's body movements. Wherein, the information of the current user's body movement can be obtained through template matching. Processor 20 matches the character region to a plurality of character templates. First match the head of the character area; after the head matching is completed, the next limb matching is performed on the remaining multiple character templates that match the head, that is, the matching of the upper body torso; after the upper body torso matching is completed, the The rest of the plurality of character templates that match the head and upper body torso are matched to the next body, that is, the matching of the upper body and the lower body, so as to determine the information of the current user's body movement according to the method of template matching. Subsequently, the processor 20 renders the predetermined three-dimensional image according to the recognized facial expressions and body movements of the current user, so that people, animals and plants in the predetermined three-dimensional image can follow and imitate the current user's facial expressions and body movements. Finally, the processor 20 fuses the rendered predetermined three-dimensional image with the background region image to obtain a merged image.

Since the person area where the current user is located in each frame of the scene image can be replaced by a three-dimensional person or animal or plant, and the person or animal or plant in each frame of the predetermined three-dimensional image can follow the scene image corresponding to the frame of the predetermined three-dimensional image Therefore, after multiple frames are merged to form a video image and played, the video image will display a three-dimensional character or animal or plant following the current user's action. In this way, the fun of image fusion is greatly improved, and a better visual experience is brought to users.

Please refer to FIG. 11. In some embodiments, step 07 fuses each frame of the predetermined three-dimensional image with the corresponding background area image to obtain a merged image to output a video image including:

072: Comparing the size of the predetermined three-dimensional image of each frame with the size of the character area in the corresponding scene image;

073: When the size of the frame of the predetermined three-dimensional image is larger than the size of the character area, reduce the frame of the predetermined three-dimensional image and fill it into the character area in the scene image for fusion to obtain a merged image; and

074: When the frame size of the predetermined 3D image is smaller than the size of the character area, enlarge the frame of the predetermined 3D image and fill it into the character area in the scene image to fuse to obtain a merged image, or fill the frame of the predetermined 3D image into the scene image character area, and use adjacent pixels of the character area to fill the gap between the frame predetermined three-dimensional image and the character area to obtain a merged image, and process multiple frames of the merged image to output a video image.

Please refer to FIG. 2 again, in some implementation manners, step 072 , step 073 , step 074 and step 075 can all be implemented by the processor 20 . That is to say, the processor 20 can also be used to compare the size of the character area in each frame of the predetermined three-dimensional image and the corresponding scene image, and when the size of the predetermined three-dimensional image of the frame is larger than the size of the character area, reduce the size of the frame predetermined three-dimensional image. The three-dimensional image is filled into the character area in the scene image to fuse to obtain a merged image. When the size of the predetermined three-dimensional image of the frame is smaller than the size of the character area, the predetermined three-dimensional image of the frame is enlarged and filled into the character area in the scene image to obtain a merged image. Merge images, or fill the frame of predetermined 3D images into the character area in the scene image, and use adjacent pixels of the character area to fill the gap between the frame of predetermined 3D image and the character area to obtain a merged image, and process multiple frames Merge images to output video image.

Specifically, since the collection distance between the current user and the visible light camera 11 is not fixed, the size of the character area in each frame of the scene image is also not fixed. In this way, before the background region image is fused with the predetermined 3D image, it is first necessary to compare the size of the character region corresponding to the predetermined 3D image and the fused background region image. Wherein, the size includes the character area and the height and width of the predetermined three-dimensional image. When the width and height of the predetermined three-dimensional image are larger than the size of the character area, an appropriate reduction value of the height and width can be determined according to the size of the character area, and the predetermined three-dimensional image is reduced according to the reduction value, so that the predetermined three-dimensional image fills the Into the part of the scene image where the character area is located. When the height and width of the predetermined three-dimensional image are smaller than the height and width of the character area, an appropriate height and width magnification value can be determined according to the size of the character area, and the predetermined three-dimensional image can be enlarged according to the magnification value to fill in the scene image The part where the character area is located; or, directly fill the predetermined 3D image into the part where the character area is located in the scene image with the original size, and then use the pixels around the character area to fill the gap between the predetermined 3D image and the character area. When the width of the predetermined 3D image is greater than the width of the character area, and the height of the predetermined 3D image is smaller than the height of the character area, the width of the predetermined 3D image can be appropriately reduced according to the height of the character area, and the predetermined 3D image can be appropriately enlarged according to the height of the character area height, and fill the part where the character area is located in the scene image with the predetermined three-dimensional image after resizing. When the height of the predetermined 3D image is greater than the height of the character area, and the width of the predetermined 3D image is smaller than the width of the character area, the height of the predetermined 3D image can be appropriately reduced according to the height of the character area, and the predetermined 3D image can be appropriately enlarged according to the width of the character area , and fill the part of the scene image where the character area is located with the predetermined three-dimensional image after resizing.

In some implementations, the predetermined three-dimensional image may be randomly selected by the processor 20, or selected by the current user.

After the processor 20 obtains the multi-frame merged image, the multi-frame merged image is sequentially arranged and stored, and the multi-frame merged image can be stored as a video format by the processor 20 to form a video image. When the video image is displayed on the display 50 ( As shown in Fig. 13 ), the user can watch a smooth video picture.

Please refer to Fig. 12, in some embodiments, the image processing method of the embodiment of the present invention also includes:

081: collect the voice information of the current user; and

082: Fuse video images with sound information to output video with sound.

Please refer to FIG. 3 again. In some implementations, the image processing device 100 further includes an acoustoelectric element 70 . Step 081 may be implemented by the acoustoelectric element 70 , and Step 082 may be implemented by the processor 20 . That is to say, the acoustoelectric element 70 can be used to collect the sound information of the current user, and the processor 20 can be used to fuse the video image and the sound information to output video with sound.

Specifically, when the imaging device is turned on to collect three-dimensional scene images and depth images, the acoustoelectric element 70 is also turned on simultaneously to collect voice information of the current user. In this way, the sound information collected by the acoustoelectric element 70 is synchronized with the video image formed by combining multiple frames of images. Subsequently, the processor 20 fuses the sound information and the video image to output a video with sound. When the video with audio is played on the display 50 (shown in FIG. 13 ) of the electronic device 1000 , the picture and sound in the video with audio can be played synchronously.

Please refer to FIG. 2 and FIG. 13 together. Embodiments of the present invention also provide an electronic device 1000 . The electronic device 1000 includes an image processing device 100 . The image processing device 100 can be realized by hardware and/or software. The image processing apparatus 100 includes an imaging device 10 and a processor 20 .

The imaging device 10 includes a visible light camera 11 and a depth image acquisition component 12 .

Specifically, the visible light camera 11 includes an image sensor 111 and a lens 112. The visible light camera 11 can be used to capture the color information of the current user to obtain multiple frames of two-dimensional images, wherein the image sensor 111 includes a color filter array (such as a Bayer filter array) , the number of lenses 112 may be one or more. When the visible light camera 11 acquires each frame of two-dimensional images, each imaging pixel in the image sensor 111 senses light intensity and wavelength information from the shooting scene to generate a set of original image data; the image sensor 111 takes the set of original image data After sending to the processor 20, the processor 20 performs operations such as denoising and interpolation on the original image data to obtain a color two-dimensional image. The processor 20 can process each image pixel in the raw image data one by one in a variety of formats, for example, each image pixel can have a bit depth of 8, 10, 12 or 14 bits, and the processor 20 can use the same or different Bit depth is processed for each image pixel.

The depth image acquisition component 12 includes a structured light projector 121 and a structured light camera 122 , and the depth image acquisition component 12 can be used to capture depth information of a current user to obtain a depth image. The structured light projector 121 is used to project the structured light to the current user, wherein the structured light pattern may be laser stripes, gray codes, sinusoidal stripes or randomly arranged speckle patterns and the like. The structured light camera 122 includes an image sensor 1221 and a lens 1222, and the number of the lens 1222 may be one or more. The image sensor 1221 is used to capture multiple frames of structured light images projected onto the current user by the structured light projector 121 . Each frame of structured light image can be sent by the depth acquisition component 12 to the processor 20 for processing such as demodulation, phase recovery, and phase information calculation to obtain the depth information of the current user.

In some embodiments, the functions of the visible light camera 11 and the structured light camera 122 can be realized by one camera, that is to say, the imaging device 10 only includes one camera and one structured light projector 121, and the above-mentioned camera can not only capture two-dimensional images , can also take structured light images.

In addition to using structured light to obtain depth images, depth images of the current user can also be obtained through binocular vision methods and depth image acquisition methods based on Time of Flight (TOF).

In addition, the image processing device 100 further includes a memory 30 . The memory 30 may be embedded in the electronic device 1000, or may be a memory independent of the electronic device 1000, and may include a direct memory access (Direct Memory Access, DMA) feature. The original image data collected by the visible light camera 11 or the data related to the structured light image collected by the depth image collection component 12 can be transmitted to the memory 30 for storage or buffering. The processor 20 can read the original image data from the memory 30 for processing to obtain a two-dimensional image, and can also read the data related to the structured light image from the memory 30 for processing to obtain a depth image, and can also read the original image data from the memory 30. The image data and the data related to the structured light image are processed to obtain a three-dimensional color scene image. In addition, the two-dimensional image, the scene image and the depth image can also be stored in the memory 30, so that the processor 20 can call and process at any time. For example, the processor 20 calls multiple frames of scene images and multiple frames of depth images to extract the background area, and The extracted multi-frame background region images are fused with corresponding predetermined three-dimensional images to obtain multi-frame merged images, and the multi-frame merged images are sequentially arranged or stored to form a video image. Wherein, the predetermined three-dimensional image, merged image, and video image may also be stored in the memory 30 .

The image processing device 100 may further include a display 50 . The display 50 can directly obtain video images from the processor 20 , and can also obtain video images from the memory 30 . The display 50 displays video images for users to watch, or is further processed by a graphics engine or a graphics processing unit (Graphics Processing Unit, GPU). The image processing device 100 also includes an encoder/decoder 60, which can encode and decode image data of two-dimensional images, scene images, depth images, merged images, video images, etc., and the encoded image data can be stored in memory 30 and may be decompressed by the decoder for display before the image is displayed on the display 50. The encoder/decoder 60 may be implemented by a central processing unit (Central Processing Unit, CPU), a GPU or a coprocessor. In other words, the encoder/decoder 60 may be any one or more of a central processing unit (Central Processing Unit, CPU), a GPU, and a coprocessor.

The image processing device 100 also includes a control logic 40 . When the imaging device 10 is imaging, the processor 20 analyzes the data acquired by the imaging device to determine image statistical information of one or more control parameters (eg, exposure time, etc.) of the imaging device 10 . The processor 20 sends the image statistical information to the control logic 40, and the control logic 40 controls the imaging device 10 to perform imaging with the determined control parameters. Control logic 40 may include a processor and/or a microcontroller executing one or more routines (eg, firmware). One or more routines may determine control parameters of imaging device 10 based on received image statistics.

The image processing device 100 also includes an acoustoelectric element 70 . The acoustic electric element 70 uses the principle of electromagnetic induction to convert sound into electric current output. When the current user makes a sound, the air inside the acoustic-electric element 70 will be driven to vibrate, causing a slight displacement between the coil and the magnetic core inside the acoustic-electric element 70 , thereby cutting the magnetic induction line to generate current. The acoustoelectric element 70 delivers electrical current to the processor 20, which processes the electrical current to generate audio information. The voice information can be sent to the memory 30 for storage. When the processor 20 fuses the video image and sound information to obtain a video with sound, the processor 20 can send the video with sound to the display 50 and the electroacoustic element (not shown), and the display 50 displays the video picture in the video with sound, and the electroacoustic The components play sound information synchronously.

Referring to FIG. 14 , an electronic device 1000 according to an embodiment of the present invention includes one or more processors 20 , a memory 30 and one or more programs 31 . One or more programs 31 are stored in memory 30 and configured to be executed by one or more processors 20 . The program 31 includes instructions for executing the image processing method of any one of the above-mentioned embodiments.

For example, the program 31 includes instructions for performing the image processing method described in the following steps:

03: Collect multi-frame 3D scene images and depth images of the current user at a preset frequency;

05: Process multiple frames of scene images and multiple frames of depth images to segment the character area in each frame of the scene image and the background area except the character area to obtain multiple frames of background area images, and the multiple frames of the background area images correspond to multiple frames of predetermined three-dimensional images; and

07: Merge each frame of the predetermined three-dimensional image with the corresponding background area image to obtain a multi-frame merged image to output a video image.

For another example, the program 31 also includes instructions for performing the image processing method described in the following steps:

051: Identify the face area in each frame of the scene image;

052: Obtain the depth information corresponding to the face area from the scene image of the frame or the depth image corresponding to the scene image of the frame;

053: Determine the depth range of the character area according to the depth information of the face area;

054: Determine the character area connected to the face area and falling within the depth range according to the depth range of the character area;

055: Determine the background area image according to the character area and the frame scene image.

A computer-readable storage medium according to an embodiment of the present invention includes a computer program used in conjunction with the electronic device 1000 capable of imaging. The computer program can be executed by the processor 20 to complete the image processing method in any one of the above-mentioned embodiments.

For example, the computer program can be executed by the processor 20 to complete the image processing method described in the following steps:

03: Collect multi-frame 3D scene images and depth images of the current user at a preset frequency;

05: Process multiple frames of scene images and multiple frames of depth images to segment the character area in each frame of the scene image and the background area except the character area to obtain multiple frames of background area images, and the multiple frames of the background area images correspond to multiple frames of predetermined three-dimensional images; and

07: Merge each frame of the predetermined three-dimensional image with the corresponding background area image to obtain a multi-frame merged image to output a video image.

For another example, the computer program can also be executed by the processor 20 to complete the image processing method described in the following steps:

051: Identify the face area in each frame of the scene image;

052: Obtain the depth information corresponding to the face area from the scene image of the frame or the depth image corresponding to the scene image of the frame;

053: Determine the depth range of the character area according to the depth information of the face area;

054: Determine the character area connected to the face area and falling within the depth range according to the depth range of the character area;

055: Determine the background area image according to the character area and the frame scene image.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present invention pertain.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment used. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. The program is processed electronically and stored in computer memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGAs), Field Programmable Gate Arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (18)

1. a kind of image processing method, for electronic installation, it is characterised in that described image processing method includes：

With the three-dimensional scene image and depth image of predeterminated frequency collection multiframe active user；

Depth image described in scene image described in multiframe and multiframe is handled to split people's object area in scene image described in every frame And except the background area beyond people's object area to obtain multiframe background area image, background area image pair described in multiframe Answer multiframe predetermined three-dimensional image；With

It will merge to obtain multiframe merging image to export with the corresponding background area image per predetermined three-dimensional image described in frame Video image.

2. image processing method according to claim 1, it is characterised in that described image processing method also includes：

Gather the acoustic information of the active user；With

The video image is merged to export sound video with the acoustic information.

3. image processing method according to claim 1, it is characterised in that described to be worked as with predeterminated frequency collection multiframe The step of three-dimensional scene image and depth image of preceding user, includes：

The active user is shot with the predeterminated frequency to obtain multiframe two dimensional image；

To active user's projective structure light；

The structure light image modulated with predeterminated frequency shooting multiframe through the active user；With

Phase information corresponding to each pixel of structure light image described in per frame is demodulated to obtain depth image described in multiframe；With

Depth image described in two dimensional image described in multiframe and multiframe is handled to obtain the three-dimensional scene image of multiframe.

4. image processing method according to claim 3, it is characterised in that demodulation structure light image described in per frame The step of phase information corresponding to each pixel is to obtain depth image described in multiframe includes：

Phase information corresponding to each pixel in demodulation structure light image described in per frame；

The phase information is converted into depth information；With

The depth image is generated according to the depth information.

5. image processing method according to claim 1, it is characterised in that the predetermined three-dimensional image includes three-dimensional void At least one of anthropomorphic thing, three-dimensional real person, three-dimensional animals and plants, the three-dimensional real person eliminate described work as Preceding user itself.

6. image processing method according to claim 1, it is characterised in that described image processing method also includes：

Scene image described in per frame and the depth image are handled to extract the action message of the active user；With

The predetermined three-dimensional image is rendered according to the action message so as to be followed per predetermined three-dimensional image described in frame described current The action of user；

It is described to merge to obtain merging image to export with the corresponding background area image per predetermined three-dimensional image described in frame The step of video image, includes：

Predetermined three-dimensional image described in every frame after rendering merges to obtain conjunction described in multiframe with the corresponding background area image And image is with output video image.

7. image processing method according to claim 6, it is characterised in that the action message includes the active user Expression and/or limb action.

8. image processing method according to claim 1, it is characterised in that predetermined three-dimensional image corresponds to multiframe described in multiframe The scene image, it is described to merge to obtain merging figure with the corresponding background area image per predetermined three-dimensional image described in frame As being included with the step of output video image：

Compare size of the predetermined three-dimensional image described in every frame with the size of people's object area in the corresponding scene image；

When the size of predetermined three-dimensional image described in the frame is more than the size of people's object area, predetermined three-dimensional described in the frame is reduced Image is simultaneously filled into people's object area in the scene image to merge to obtain merging image；

When the size of predetermined three-dimensional image described in the frame is less than the size of people's object area, amplify predetermined three-dimensional described in the frame Image is simultaneously filled into people's object area in the scene image to merge to obtain the merging image, or will be pre- described in the frame Determine people's object area that 3-D view is filled into the scene image, and filled out using the adjoining pixel of people's object area The space filled between predetermined three-dimensional image described in the frame and people's object area is to obtain described merging image；With

Merge image described in processing multiframe to export the video image.

9. a kind of image processing apparatus, for electronic installation, it is characterised in that described image processing unit includes：

Imaging device, the imaging device are used for the three-dimensional scene image and depth that multiframe active user is gathered with predeterminated frequency Image；

Processor, the processor are used to handle depth image described in scene image described in multiframe and multiframe to split described in every frame People's object area in scene image and except the background area beyond people's object area to obtain multiframe background area image, it is more Background area image described in frame corresponds to multiframe predetermined three-dimensional image；With

It will merge to obtain with the corresponding background area image per predetermined three-dimensional image described in frame and merge image to export video figure Picture.

10. image processing apparatus according to claim 9, it is characterised in that described image processing unit also includes acoustic-electric Element, the acoustoelectric element are used for the acoustic information for gathering the active user；

The processor is additionally operable to the acoustic information merge the video image to export sound video.

11. image processing apparatus according to claim 9, it is characterised in that the imaging device includes visible image capturing Head and depth image acquisition component, the depth image acquisition component includes structured light projector and structure light video camera head, described Visible image capturing head is used to shoot the active user with the predeterminated frequency to obtain multiframe two dimensional image；

The structured light projector is used for active user's projective structure light；

The structure light video camera head is used for：

The structure light image modulated with predeterminated frequency shooting multiframe through the active user；

Phase information corresponding to each pixel of structure light image described in per frame is demodulated to obtain depth image described in multiframe；With

Depth image described in two dimensional image described in multiframe and multiframe is handled to obtain the three-dimensional scene image of multiframe.

12. image processing apparatus according to claim 11, it is characterised in that the structure light video camera head is further used In：

Phase information corresponding to each pixel in demodulation structure light image described in per frame；

The phase information is converted into depth information；With

The depth image is generated according to the depth information.

13. image processing apparatus according to claim 9, it is characterised in that the predetermined three-dimensional image includes three-dimensional At least one of virtual portrait, three-dimensional real person, three-dimensional animals and plants, the three-dimensional real person eliminates described Active user itself.

14. image processing apparatus according to claim 9, it is characterised in that the processor is additionally operable to：

Scene image described in per frame and the depth image are handled to extract the action message of the active user；

The predetermined three-dimensional image is rendered according to the action message so as to be followed per predetermined three-dimensional image described in frame described current The action of user；

Predetermined three-dimensional image described in every frame after rendering merges to obtain conjunction described in multiframe with the corresponding background area image And image is with output video image.

15. image processing apparatus according to claim 14, it is characterised in that the action message includes the current use The expression and/or limb action at family.

16. image processing apparatus according to claim 9, it is characterised in that predetermined three-dimensional image described in multiframe is corresponding more Scene image described in frame, the processor are additionally operable to：

Compare size of the predetermined three-dimensional image described in every frame with the size of people's object area in the corresponding scene image；

When the size of predetermined three-dimensional image described in the frame is more than the size of people's object area, predetermined three-dimensional described in the frame is reduced Image is simultaneously filled into people's object area in the scene image to merge to obtain merging image；With

When the size of predetermined three-dimensional image described in the frame is less than the size of people's object area, amplify predetermined three-dimensional described in the frame Image is simultaneously filled into people's object area in the scene image to merge to obtain the merging image, or will be pre- described in the frame Determine people's object area that 3-D view is filled into the scene image, and filled out using the adjoining pixel of people's object area The space filled between predetermined three-dimensional image described in the frame and people's object area is to obtain described merging image；With

Merge image described in processing multiframe to export the video image.

17. a kind of electronic installation, it is characterised in that the electronic installation includes：

One or more processors；

Memory；With

One or more programs, wherein one or more of programs are stored in the memory, and be configured to by One or more of computing devices, described program include being used at the image that perform claim is required described in 1 to 8 any one The instruction of reason method.

A kind of 18. computer-readable recording medium, it is characterised in that the meter being used in combination including the electronic installation with that can image Calculation machine program, the computer program can be executed by processor to complete the image procossing described in claim 1 to 8 any one Method.