WO2018176575A1 - Co-optical-center camera device, and seamless panoramic stitching assembly and method - Google Patents

Abstract

Provided are a co-optical-center camera device, and a seamless panoramic stitching assembly and method. The camera device comprises a camera, a first plane mirror, and a second plane mirror. Mirror surfaces of the first plane mirror and the second plane mirror are parallel to each other and perpendicular to a main optical axis of the camera. The first plane mirror is arranged in front of the camera at a preset distance, and the mirror surface thereof faces the camera. The second plane mirror is arranged at a location at which a lens of the camera is provided, and the mirror surface thereof faces the first plane mirror. A hole is arranged at the second plane mirror for the camera to capture an image. The camera device enables optical centers of a plurality of cameras to coincide, such that a video free of a parallax error can be produced, thereby performing seamless panoramic stitching.

Description

Camera device capable of common light center, panoramic seamless splicing component and method Technical field

The present invention relates to the field of virtual reality technology and computer vision, and in particular, to a camera device, a seamless seamless splicing component and a method.

Background technique

In recent years, the virtual reality technology boom has swept the world. It can be widely used in entertainment, education, communication, scientific research, military, medical and other industries. It not only brings a large-scale technological revolution, but also changes people's thinking habits. lifestyle. The world's major companies have entered the virtual reality field, and a large number of entrepreneurial teams have also flooded into the virtual reality industry, and a variety of virtual reality devices have emerged. However, high-quality virtual reality content is still very scarce, which makes virtual reality technology empty and has no use.

Panoramic video is one of the main contents of virtual reality. At present, most of the cameras used to capture panoramic video do not have a common center of light, so there is parallax between the various videos taken, resulting in the mosaic of panoramic video is not completely seamless. Small parallax can cause blurred images, and large parallax can cause problems with misalignment and ghosting. In particular, these problems become more serious when moving objects pass through the stitching seams of the panoramic video, giving the user a very bad experience.

The above disclosure of the background art is only for assisting in understanding the concepts and technical solutions of the present invention, and it does not necessarily belong to the prior art of the present patent application, and there is no clear evidence that the above content has been disclosed on the filing date of the present patent application. In this case, the above background art should not be used to evaluate the novelty and inventiveness of the present application.

Summary of the invention

In order to solve the above technical problem, the present invention provides a camera device with a common center of light, a seamless seamless splicing component and a method, which can overlap the optical centers of a plurality of cameras, thereby capturing a video without parallax, and then seamlessly Panorama stitching.

In order to achieve the above object, the present invention adopts the following technical solutions:

The invention discloses a camera device capable of common light center, comprising a camera, a first plane mirror and a second a plane mirror, the mirrors of the first plane mirror and the second plane mirror are parallel to each other, and are perpendicular to a main optical axis of the camera; wherein: the first plane mirror is located at a preset distance in front of the camera, and the mirror faces the a camera; the second plane mirror is located at a lens position of the camera, the mirror faces the first plane mirror, and the second plane mirror is provided with a hole for the camera to capture an image.

Preferably, the mirror size of the first plane mirror just completely covers the imaged field of view of the camera.

Preferably, the mirror size of the second plane mirror completely covers the field of view captured by the camera through the first plane mirror.

Preferably, the hole is disposed at a center of the second plane mirror, the size of the hole being greater than or equal to a size of a main lens of the camera.

Preferably, the back surface of the second plane mirror is in close contact with the lens position of the camera, and the size of the hole is just to block the shooting field of the camera.

The invention also discloses a panoramic seamless splicing assembly, comprising a plurality of the above-mentioned camera devices, wherein the plurality of camera devices are arranged opposite each other to form a circle, and the optical center of the camera in each of the camera devices is The distance of the center of the circle is equal to the sum of the distance from the optical center of the corresponding camera to the first plane mirror and the distance from the first plane mirror to the second plane mirror, respectively.

Preferably, 5 to 10 of said camera devices are included.

Preferably, the optical centers of the cameras in each of the camera devices are located on the same circle, and each of the cameras is equally spaced.

The invention additionally discloses a panoramic seamless splicing method, comprising the following steps:

S1: capturing a video by using the panoramic seamless splicing component, and performing feature extraction and feature matching on the first frame of the video captured by each of the camera devices in the panoramic seamless splicing component;

S2: performing homography alignment on the first frame of each video, and respectively calculating parameters of each of the camera devices;

S3: projecting, according to parameters of each of the camera devices, an image captured by each camera device to a spherical surface having a center of the circle as a center of the circle;

S4: merging the projected image to fuse the multiple images into a single panoramic image;

Steps S3 and S4 are repeated, and each frame image is sequentially projected and fused to obtain a seamless panoramic image frame, and a seamless panoramic video is generated.

Preferably, the step S1 specifically includes: extracting feature points of the first frame image of the video captured by each of the camera devices by using a SIFT, SURF or ORB feature extraction algorithm, and extracting feature points of the first frame image of each video Matching is performed; preferably, the multi-band fusion method is adopted by fusing the projected image in step S4.

Compared with the prior art, the beneficial effects of the present invention are: the camera device of the present invention discloses that the first plane mirror and the second plane mirror which are parallel to each other are disposed respectively at the front of the camera and the lens position, so that the camera is The optical center is moved to the outside of the physical size of the camera by two reflections, thereby making it possible to recombine the optical centers of the plurality of cameras, and thus the panoramic seamless splicing component based on the plurality of cameras of the present invention can be formed on the basis of the present invention. There is no parallax between the videos captured by each camera in the panoramic seamless splicing component, and the common overlapping area can be seamlessly spliced, and there is no blur or ghosting problem at the overlapping area after splicing, and splicing There is no problem with the seams, which greatly improves the stitching effect of the panoramic video; further realizes the shooting of the video without parallax, and then seamless panorama stitching. At the same time, there is no parallax between the multiple images captured by the panoramic seamless splicing component based on the camera device, and no need to perform seam searching and the like, the splicing speed can be improved, and the frame rate of the panoramic video can be improved, thereby improving the virtual reality. Experience.

DRAWINGS

1 is a schematic structural view of a camera device capable of a common light center according to a preferred embodiment of the present invention;

2 is a schematic diagram of an imaging optical path of a non-central region of an image sensor of the camera of FIG. 1;

3 is an equivalent diagram of an imaging optical path of a non-center region of the image sensor of the camera of FIG. 1;

4 is a schematic diagram of an imaging optical path of a central region of an image sensor of the camera of FIG. 1;

5 is a schematic structural view of a panoramic seamless splicing assembly according to a preferred embodiment of the present invention;

6 is a flow chart of a seamless seamless stitching method in accordance with a preferred embodiment of the present invention.

detailed description

The invention will now be further described with reference to the drawings in conjunction with the preferred embodiments.

As shown in FIG. 1 , it is a schematic structural diagram of a camera device capable of a common light center according to a preferred embodiment of the present invention. The camera device includes a camera A1, a first plane mirror A2, a second plane mirror A3, a first plane mirror A2 and a second plane mirror A3. Parallel to each other, both perpendicular to the main optical axis of camera A1, wherein: first plane mirror A2 is located at a preset distance in front of the camera A1, and its mirror surface faces the camera A1; the second plane mirror A3 is located at the lens position of the camera A1, and its mirror surface faces the first plane mirror A2, and a hole is formed in the second plane mirror A3 for Take a picture for camera A1.

Further, the mirror size of the first plane mirror A2 just completely covers the imaging field of view of the camera A1, that is, the light entering the inside of the camera A1 for imaging can only come from the light reflected by the first plane mirror A2. The mirror size of the second plane mirror A3 completely covers the field of view of the camera A1 reflected by the first plane mirror A2, that is, the light entering the inside of the camera A1 for imaging can only be reflected twice by the second plane mirror A3 and the first plane mirror A2. The light in the second plane mirror A3 is opened at the center thereof, and the size of the hole is smaller and smaller, so as not to obscure the shooting field of the camera A1, wherein the back surface of the second plane mirror A3 can be closely attached to the camera A1. At the lens position, the size of the hole may be equal to or slightly larger than the size of the main lens of the camera A1.

The optical center of the camera A1 is located at the center of the main lens, and the optical centers of the plurality of cameras cannot be overlapped without using a special device. Moreover, since the photosensitive element of the camera has a certain size, it can only be a certain field of view. The range is imaged. In the preferred embodiment of the present invention, the camera A1 in the common-light camera device is reflected by two plane mirrors, which is equivalent to moving to the rear for shooting, so that the optical center of the camera A1 is moved backward, so that multiple The camera can simultaneously capture multiple different directions at the same point in space, making seamless splicing of multiple channels possible.

As shown in FIG. 2, the light is subjected to two specular reflections to image on a non-center region of the image sensor A5 of the camera A1, wherein the light reflected by the edge of the first plane mirror A2 falls just at the edge of the image sensor A5, that is, The first plane mirror A2 just completely covers the field of view of the camera A1. The convex lens A4 represents the main lens of the camera A1, the optical center of the camera A1 is located at the center of the convex lens A4, and the extension line of all incident light rays of the second plane mirror A3 just intersects at one point, which is exactly where the optical center of the camera A1 moves to the rear. That is the virtual light heart.

As shown in FIG. 3, the first plane mirror A2 and the second plane mirror A3 are removed, and the convex lens A4 is moved to the position of the rear virtual optical center while maintaining the relative position of the image sensor A5 and the convex lens A4. As can be seen from the comparison with FIG. 2, the light received by the non-center region of the image sensor A5 is exactly the same as that of FIG. 2, and the formed image is completely identical; that is, the first plane mirror A2 and the second plane mirror A3 function positively. The camera A1 is moved to the position of the virtual optical center directly behind to shoot.

As shown in FIG. 4, the light received in the central area of the image sensor A5 passes through more than two mirrors. The surface reflection, the closer to the light of the center of the image sensor A5, the more the number of reflections, the Droost effect may occur; and since the central area is not the overlapping area of adjacent cameras, it does not affect the splicing.

1 to 4, the optical center of the camera A1 is moved to the virtual optical center directly behind by two specular reflections. From the imaging principle, the distance from the optical center of the camera A1 to the virtual optical center is equal to the light of the camera A1. The sum of the distance from the heart to the first plane mirror A2 and the distance from the first plane mirror A2 to the second plane mirror A3.

As shown in FIG. 5, the panoramic seamless splicing assembly of the present invention includes a plurality of the above-mentioned camera devices. The plurality of camera devices are arranged to face each other in a circle, and the virtual optical centers of the respective camera devices are located at the center of the circle, that is, The distance from the optical center of the camera in each camera device to the center of the circle is equal to the sum of the distance from the optical center of the corresponding camera to the first plane mirror and the distance from the first plane mirror to the second plane mirror, respectively, so that the respective camera devices are shared Heart set.

Further, the optical centers of the cameras in the respective camera devices are located on the same circle. By adjusting the distance between each set of plane mirrors, the distance of the optical center of each camera device is shifted by the radius of the circle, so that all the cameras can be made. The virtual optical center of the device is located at the center of the circle (i.e., at the center of the aforementioned circle). At this time, it is equivalent to the camera moving to the center of the circle to shoot a plurality of different directions, thereby obtaining a multi-channel non-parallax video. In some embodiments, the panoramic seamless splicing assembly can include from 5 to 10 camera devices, each camera device being equally spaced.

On the basis of the hardware device shown in FIG. 5, the preferred embodiment of the present invention discloses a seamless seamless splicing method. As shown in FIG. 6, the method includes the following steps:

S1: adopting the panoramic seamless splicing component shown in FIG. 5 to capture video, and performing feature extraction and feature matching on the first frame (first frame) of the video captured by each camera device in the panoramic seamless splicing component, specifically, Using SIFT (Size Invariant Feature Transform), SURF (Acceleration Robust Feature) or ORB feature extraction algorithm, feature points of the first frame image of the video captured by each of the camera devices are extracted, and the first frame image of each video is extracted. Feature points are matched;

S2: performing homography alignment on the first frame (first frame) of each video. Specifically, using a global homography transformation matrix, the feature points on the matching are aligned, thereby calculating internal parameters of each camera device. And external parameters;

S3: According to the internal parameters and external parameters of each camera device, respectively, each camera device is photographed The image is projected onto the spherical surface with the virtual optical center as the center of the sphere. Since each camera device has been totally optical, the images captured by each camera device will be projected onto the same spherical surface, and there is no parallax;

S4: merging the image after projection to merge multiple images into a single panoramic image. Specifically, a multi-band fusion method may be adopted. Since the images are non-parallax, the merged panorama is seamless. of;

Steps S3 and S4 are repeated, and each frame image is sequentially projected and fused to obtain a seamless panoramic image frame, thereby generating a seamless panoramic video.

The image image captured by the panoramic seamless splicing component shown in FIG. 5 is completely seamless after the above steps are synthesized, wherein since there is no parallax between the plurality of images, no seam searching or the like is required, thereby It can increase the stitching speed and thus increase the frame rate of panoramic video, which is crucial for the virtual reality experience.

The illuminable camera device of the present invention makes the optical center of the camera back to the physical size of the camera by adding a set of plane mirrors at the front of the camera and at the lens position, thereby making it possible to recombine the optical centers of the plurality of cameras. Furthermore, a panoramic seamless splicing component with a common center of light is designed. Since the video obtained by the panoramic seamless splicing component of the common optical center has the advantage of no parallax, the splicing effect of the panoramic video can be improved, the seamless panoramic video can be obtained, and the algorithm of the video splicing can be simplified, and the speed of the video splicing can be improved. The frame rate of the panoramic video.

The above is a further detailed description of the present invention in connection with the specific preferred embodiments, and the specific embodiments of the present invention are not limited to the description. It will be apparent to those skilled in the art that <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt;

Claims (10)

A camera device capable of common light center, comprising: a camera, a first plane mirror and a second plane mirror, wherein the mirror surfaces of the first plane mirror and the second plane mirror are parallel to each other, and are perpendicular to a main optical axis of the camera; among them: The first plane mirror is located at a preset distance in front of the camera, and the mirror faces the camera; The second plane mirror is located at a lens position of the camera, the mirror surface faces the first plane mirror, and the second plane mirror is provided with a hole for the camera to capture an image. A camera apparatus according to claim 1, wherein the mirror size of said first plane mirror just completely covers the imaged field of view of said camera. The camera apparatus according to claim 1, wherein a mirror size of said second plane mirror completely covers a field of view captured by said camera through said first plane mirror. The camera apparatus according to claim 1, wherein the hole is provided at a center of the second plane mirror, the size of the hole being greater than or equal to a size of a main lens of the camera. The camera device according to any one of claims 1 to 4, wherein a back surface of the second plane mirror is in close contact with a lens position of the camera, and the size of the hole is a film that does not block the camera. Vision. A panoramic seamless splicing assembly, comprising: a plurality of camera devices according to any one of claims 1 to 5, wherein a plurality of said camera devices are arranged facing away in a circle, each of said camera devices The distance from the optical center of the camera to the center of the circle is equal to the sum of the distance from the optical center of the corresponding camera to the first plane mirror and the distance from the first plane mirror to the second plane mirror, respectively. The panoramic seamless splicing assembly according to claim 6, comprising 5 to 10 of said camera devices. The panoramic seamless splicing assembly according to claim 6 or 7, wherein the optical centers of the cameras in each of the camera devices are located on the same circle, and each of the cameras is equally spaced. A panoramic seamless stitching method, comprising the steps of: S1: shooting a video using the panoramic seamless splicing component according to any one of claims 6 to 8, The first frame of the video captured by each of the camera devices in the panoramic seamless splicing component performs feature extraction and feature matching respectively; S2: performing homography alignment on the first frame of each video, and respectively calculating parameters of each of the camera devices; S3: projecting, according to parameters of each of the camera devices, an image captured by each camera device to a spherical surface having a center of the circle as a center of the circle; S4: merging the projected image to fuse the multiple images into a single panoramic image; Steps S3 and S4 are repeated, and each frame image is sequentially projected and fused to obtain a seamless panoramic image frame, and a seamless panoramic video is generated. The panoramic seamless splicing method according to claim 9, wherein the step S1 specifically comprises: extracting features of the first frame image of the video captured by each of the camera devices by using a SIFT, SURF or ORB feature extraction algorithm. And matching the feature points of the first frame image of each video; preferably, the multi-band fusion method is adopted for the fusion of the projected image in step S4.

Images