CN106358006A - Video correction method and video correction device - Google Patents

Abstract

The invention discloses a video correction method and device. Wherein, the method includes: collecting a first image and a second image of the user, wherein the first image is an image collected when the user watches the camera area, and the second image is an image collected when the user watches the screen area; extracting the second image The coordinate system in the first image and the coordinate system in the second image, and calculate the affine matrix from the coordinate system of the first image to the coordinate system of the second image; use the affine matrix to perform affine transformation processing on the collected initial video , to generate the corrected target video. The invention solves the technical problem that in the prior art, the video correction method for realizing eye contact between chatting users is complex and consumes a lot of computing resources.

Description

Video correction method and device

technical field

The present invention relates to the field of computers, in particular to a video correction method and device.

Background technique

With the development of Internet technology, people in multiple different regions can communicate face to face only through cameras and computers. However, when we use chat software (such as WeChat, Michat, MSN, etc.) There is a distance difference between the camera and the center of the screen, resulting in a lack of eye-to-eye communication between chat users. For example, the camera of a laptop computer is often set at the top of the screen. When user USER1 and user USER2 are chatting, user USER1 often looks at The center area of the screen, instead of the camera, then for user USER2, the display screen of user USER2 will show that the eyes of user USER1 are looking down, and eye contact is an indispensable part of human-to-human communication. It makes video chatting not as natural as usual face-to-face chatting.

In the prior art, the method of mirror reflection or virtual viewpoint drawing is often used to correct the video image, so as to realize the eye contact between chatting users, but the above method is complicated to realize, and the equipment is expensive, which is not suitable for people's daily video chat In many scenarios, it is difficult to popularize.

For the above-mentioned problems in the prior art, the video correction method for realizing eye contact between chatting users is complex and consumes a lot of resources, and no effective solution has been proposed so far.

Contents of the invention

Embodiments of the present invention provide a video correction method and device to at least solve the technical problems in the prior art that the video correction method for realizing eye contact between chatting users is complex and consumes a large amount of computing resources.

According to an aspect of an embodiment of the present invention, a video correction method is provided, the method includes: collecting a first image and a second image of the user, wherein the first image is an image collected when the user watches the area of the camera device, The second image is the image collected when the user watches the screen area; extract the coordinate system in the first image and the coordinate system in the second image, and calculate the affine matrix from the coordinate system of the first image to the coordinate system of the second image ; Use the affine matrix to perform affine transformation processing on the collected initial video to generate the corrected target video.

According to another aspect of the embodiments of the present invention, there is also provided a video correction device, including: a collection unit, configured to collect a first image and a second image of the user, wherein the first image is when the user watches the area of the camera equipment The collected image, the second image is the image collected when the user watches the screen area; the extraction unit is used to extract the coordinate system in the first image and the coordinate system in the second image, and calculate the coordinate system of the first image to An affine matrix of the coordinate system of the second image; a correction unit, configured to use the affine matrix to perform affine transformation processing on the collected initial video to generate a corrected target video.

In the embodiment of the present invention, the first image and the second image of the user are collected by using the method, wherein the first image is the image collected when the user watches the camera area, and the second image is the image collected when the user watches the screen area ; Extract the coordinate system in the first image and the coordinate system in the second image, and calculate the affine matrix from the coordinate system of the first image to the coordinate system of the second image; use the affine matrix to affine the collected initial video The corrected target video is generated by projective transformation processing, which solves the technical problem that the video correction method for realizing eye contact between chatting users in the prior art is complex and consumes a large amount of computing resources.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

Fig. 1 is the flow chart of the correction method of video according to the embodiment of the present invention; And

Fig. 2 is a schematic structural diagram of a video correction device according to an embodiment of the present invention.

detailed description

In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Embodiment one

According to an embodiment of the present invention, an embodiment of a video correction method is provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and, Although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

Fig. 1 is a flow chart of a video correction method according to an embodiment of the present invention. As shown in Fig. 1, the method includes the following steps:

Step S12, collecting a first image and a second image of the user, wherein the first image is an image collected when the user watches the area of the camera device, and the second image is an image collected when the user watches the area of the screen.

Specifically, in the above step S12, an imaging device may be used to collect two images of the user viewing different areas. The imaging device may be a camera arranged above the computer screen. In this embodiment, the user may be allowed to sit quietly in the In front of the computer screen, when the user looks down at the screen area (preferably the center of the screen), the first image of the user is collected by the camera; second image. It should be noted that, in the first image, the user's eyes focus on the center of the screen, and in the second image, the user's eyes focus on the camera above the screen.

Step S14, extracting the coordinate system in the first image and the coordinate system in the second image, and calculating an affine matrix from the coordinate system of the first image to the coordinate system of the second image.

Specifically, in the above step S14, a computer terminal can be used to obtain the above two images, and then extract the coordinate system in the above first image and the coordinate system in the second image. It should be noted that the above first image is in the form of The coordinate system with the camera as the origin, the above second image is the coordinate system with the screen as the center as the origin, and the computer terminal can calculate the coordinate system with the camera as the origin to the coordinate system with the screen as the origin according to the existing calculation algorithm Affine matrix between.

Step S16, using an affine matrix to perform affine transformation processing on the collected initial video to generate a corrected target video.

Specifically, in the above step S16, when the user is making a video call, this solution can obtain the user's initial video, and then extract the facial feature points and contours of each frame of the initial video, and then target the facial features of each frame of the initial video Affine transformation processing is performed on the points and contours to generate a corrected target video. Preferably, this solution can paste the corrected target video back to the face area of the original video. It should be noted that since this solution corrects the user's video, when multiple users are in video chat, the effect of eye contact can be achieved.

In this embodiment, the first image and the second image of the user are collected, wherein the first image is the image collected when the user watches the camera area, and the second image is the image collected when the user watches the screen area; the first image is extracted The coordinate system in the first image and the coordinate system in the second image, and calculate the affine matrix from the coordinate system of the first image to the coordinate system of the second image; use the affine matrix to perform affine transformation processing on the collected initial video, and generate The corrected target video solves the technical problems in the prior art that the video correction method for realizing eye contact between chatting users is complex and consumes a lot of resources.

Optionally, step S14, the step of extracting the coordinate system in the first image and the coordinate system in the second image may include:

Step S141, performing Hartley transform processing on the first image and the second image.

Step S142, performing stereo matching processing on the two images processed by the Hartley transform to obtain a disparity map between the first image and the second image.

What needs to be explained here is that the Hartley transform algorithm (also known as the Hartley transform algorithm) in this application is a completely symmetric real-number domain orthogonal triangular transform algorithm, which has better symmetry and computing power than the Fourier transform. efficiency.

Step S143, extracting the coordinate system in the first image and the coordinate system in the second image from the disparity map.

Specifically, in the above step S141 to step S143, the first image and the second image can be corrected by the Hartley correction method so that the epipolar lines of the two images are horizontal (or vertical), and then the disparity map can be obtained by using the stereo matching algorithm , smoothing and filtering the disparity map to obtain the smoothed disparity map. Finally, extract the positions of the eye feature points in the two images, and extract the coordinate system in the first image and the coordinate system in the second image from the disparity map .

It should be noted that this solution can also obtain the internal parameters of the camera through the camera calibration method (such as the triangulation calibration method). The internal parameter matrix contains the focal length and optical center information of the camera, and then converts the disparity map according to the focal length and optical center information of the camera. is the depth value, and then execute the solution of the above step S143.

Optionally, in step S142, before performing stereo matching processing on the two images processed by Hartley transform to obtain the disparity map between the first image and the second image, the method provided in this embodiment may further include:

Step S140, respectively rotating the first image and the second image clockwise by a preset angle.

Specifically, in this solution, since the stereo matching algorithm calculates the disparity in the horizontal direction, this solution can rotate the two images clockwise by a preset angle (for example, 90°), and then the left view (facing the center of the screen ) and the right view (facing the camera) using a stereo matching algorithm to obtain a vertical disparity map.

Optionally, after the corrected target video is generated in step S16, the method provided in this embodiment may further include:

Step S17, performing de-overlapping processing on the facial contour in the target video to generate a corrected target facial contour.

Specifically, in the above-mentioned step S17, this solution can optimize the facial contour in the target video, so as to eliminate the overlap and error of the facial contour in the target video.

Optionally, in step S17, the facial contour in the target video is de-overlapped, and the step of generating the corrected target facial contour may include:

Step S171, extracting multiple feature points from the facial contour in the target video.

Step S172, calculating the pixel density difference between each feature point and the first image.

In step S173, the feature point with the smallest pixel density difference is used as the feature point of the corrected target facial contour.

Specifically, in the above step S171 to step S173, 20 to 30 feature points can be extracted from the facial contour of each frame, and an N×N small block can be taken with each feature point as the center (N is an odd number, generally Odd numbers greater than 3 and less than 15, different values correspond to different calculation amounts and optimization effects, the larger the value, the greater the calculation amount, and the better the optimization effect; otherwise, the smaller the calculation amount, the worse the optimization effect, the selection of the number of feature points It can be selected according to the specific situation, usually 5 or 7), among the above N×N points, calculate the point with the smallest difference between the pixel density of the original image after correction, and then use this point as the point of the face contour after correction, so After M times of iterative calculation (M is a positive integer greater than 1, different values correspond to different calculation amounts and optimization effects, the larger the value, the greater the calculation amount, and the better the optimization effect; otherwise, the smaller the calculation amount, the better the optimization effect Poor, the number of M can be selected according to the specific situation, usually 3 or 4), and the optimized facial contour can be obtained.

To sum up, this solution proposes a technical solution to obtain the depth of the scene with a single ordinary webcam, and proposes to use image information to fill holes in the depth map. Finally, this solution can realize the situation without affecting the quality of the original video chat. Next, the technical effect of the chatter's eye-to-eye contact is realized.

Embodiment two

The present application also provides a video correction device, which can be used to implement the above video correction method, as shown in Figure 2, the device can include: a collection unit 20, used to collect the user's first image and images, wherein the first image is an image collected when the user watches the camera area, and the second image is an image collected when the user watches the screen area; the extraction unit 22 is used to extract the coordinate system in the first image and the second image coordinate system in the image, and calculate the affine matrix from the coordinate system of the first image to the coordinate system of the second image; the correction unit 24 is used to use the affine matrix to carry out affine transformation processing on the initial video collected, and generate correction after the target video.

In this embodiment, the first image and the second image of the user are collected, wherein the first image is the image collected when the user watches the camera area, and the second image is the image collected when the user watches the screen area; the first image is extracted The coordinate system in the first image and the coordinate system in the second image, and calculate the affine matrix from the coordinate system of the first image to the coordinate system of the second image; use the affine matrix to perform affine transformation processing on the collected initial video, and generate The corrected target video solves the technical problems in the prior art that the video correction method for realizing eye contact between chatting users is complex and consumes a lot of resources.

Optionally, the extraction unit may include: a first processing module, configured to perform Hartley transform processing on the first image and the second image; a second processing module, configured to perform stereo matching processing on the two images processed through the Hartley transform, A disparity map between the first image and the second image is obtained; a first extraction module is configured to extract a coordinate system in the first image and a coordinate system in the second image from the disparity map.

Optionally, the device provided in this embodiment further includes: a rotating unit, configured to respectively rotate the first image and the second image clockwise by a preset angle.

Optionally, the device provided in this embodiment further includes: a processing unit configured to perform de-overlapping processing on facial contours in the target video to generate a corrected target facial contour.

Optionally, the processing unit may include: a second extraction module, used to extract a plurality of feature points in the facial contour in the target video; a calculation module, used to calculate the pixel density difference between each feature point and the first image; The third processing module is used to determine the feature point with the smallest pixel density difference as the feature point of the corrected target facial contour.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be realized in other ways. Wherein, the device embodiments described above are only illustrative. For example, the division of the units may be a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or may be Integrate into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of units or modules may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media capable of storing program codes such as U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk.

The above is only a preferred embodiment of the present invention, it should be pointed out that, for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications can also be made. It should be regarded as the protection scope of the present invention.

Claims (10)

1. A correction method for video, characterized in that, comprising: Collecting a first image and a second image of the user, wherein the first image is an image collected when the user watches the camera area, and the second image is an image collected when the user watches the screen area; extracting a coordinate system in the first image and a coordinate system in the second image, and calculating an affine matrix from the coordinate system of the first image to the coordinate system of the second image; The affine transformation process is performed on the collected initial video by using the affine matrix to generate a corrected target video. 2. The method according to claim 1, wherein the step of extracting the coordinate system in the first image and the coordinate system in the second image comprises: Carrying out Hartley transform processing to the first image and the second image; performing stereo matching processing on the two images processed by the Hartley transform to obtain a disparity map between the first image and the second image; A coordinate system in the first image and a coordinate system in the second image are extracted in the disparity map. 3. The method according to claim 2, wherein the stereo matching process is carried out to the two images processed through the Hartley transform to obtain a disparity map between the first image and the second image The steps include: directly performing the stereo matching process on the two images processed by the Hartley transform to obtain a vertical disparity map between the first image and the second image; or After the two images processed by the Hartley transform are rotated clockwise by a preset angle, the stereo matching process is performed on the two images to obtain the horizontal level of the first image and the second image Disparity map. 4. The method according to claim 1, wherein, after generating the corrected target video, the method further comprises: De-overlapping processing is performed on the facial contour in the target video to generate a corrected target facial contour. 5. method according to claim 4, is characterized in that, de-overlapping is carried out to the facial contour in the target video, the step of generating the corrected target facial contour comprises: extracting a plurality of feature points in the facial contour in the target video; calculating the pixel density difference between each feature point and the first image; The feature point with the smallest pixel density difference is determined as the feature point of the corrected target facial contour. 6. A correction device for video, comprising: A collection unit, configured to collect a first image and a second image of the user, wherein the first image is an image collected when the user watches the area of the camera device, and the second image is an image collected when the user watches the area of the screen the captured image; an extraction unit, configured to extract a coordinate system in the first image and a coordinate system in the second image, and calculate an affine matrix from the coordinate system of the first image to the coordinate system of the second image; A correction unit, configured to use the affine matrix to perform affine transformation processing on the collected initial video to generate a corrected target video. 7. The device according to claim 6, wherein the extracting unit comprises: A first processing module, configured to perform Hartley transform processing on the first image and the second image; A second processing module, configured to perform stereo matching processing on the two images processed by the Hartley transform, to obtain a disparity map between the first image and the second image; A first extraction module, configured to extract the coordinate system in the first image and the coordinate system in the second image from the disparity map. 8. The device according to claim 7, further comprising: The rotating unit is used to rotate the first image and the second image clockwise by a preset angle respectively. 9. The device according to claim 6, further comprising: The processing unit is configured to perform de-overlapping processing on the facial contours in the target video to generate corrected target facial contours. 10. The device according to claim 9, wherein the processing unit comprises: The second extraction module is used to extract a plurality of feature points in the facial contour in the target video; A calculation module, configured to calculate the pixel density difference between each feature point and the first image; A third processing module, configured to determine the feature point with the smallest pixel density difference as the feature point of the corrected target facial contour.