CN105323572A - Stereo image processing system, device and method - Google Patents

Abstract

A stereoscopic image processing system, device and method, wherein the proposed system includes multiple cameras set at different positions, which shoot a scene and generate continuous images after shooting objects in the scene. The image signal is transmitted to a stereoscopic image processing device, and the received image signal can be analyzed by the stereoscopic image forming unit therein, including first obtaining one or more object images in the scene, and obtaining the position and outline of each object at each camera shooting angle, as well as the distance between each object and each camera, thereby reconstructing the stereoscopic image of the object in the scene; then the type of the object can be identified by comparison, and even a moving object can be determined after a period of recording.

Description

Stereoscopic image processing system, device and method

technical field

The present invention relates to a stereoscopic image processing system, device and method, in particular to a stereoscopic image processing system, device and method constructed by a plurality of cameras arranged in a space.

Background technique

Stereoscopic image processing technology is becoming more and more mature with the development of hardware and technology. Stereoscopic images can not only provide a sense of space that cannot be replaced by flat images, but also provide better observation and monitoring in many fields. Features.

For example, in monitoring applications, stereoscopic images will provide more accurate monitoring images from more angles. One of the methods of producing stereoscopic images in the known technology includes the use of two cameras with different shooting angles, such as a multifunctional movable stereoscopic monitoring system disclosed in Taiwan Patent No. 364715 (announcement date: 1999/07/11). In addition to the mobile device discussed in this case, the method of generating stereoscopic images includes providing two sets of cameras, two sets of signal converters, and a main image display. Stereoscopic images on video displays. This type of stereoscopic image presentation is mainly a technique for producing stereoscopic visual effects by using the parallax of the left and right eyes of the human eye, and equipment such as polarized glasses, liquid crystal shutter glasses (liquid crystal shutter glasses) and the like must be used.

Another technology presents a stereoscopic image through image reconstruction. This type of stereoscopic image is mainly to shoot the same scene through different cameras. After image processing, the images shot by different cameras are reconstructed through splicing technology. The image and the depth of field of different angles of this kind of object can present a stereo picture, and related technologies such as the stereo vision panorama splicing device described in Taiwan Patent No. M373507 (announcement date 2010/02/01).

The stereo camera adopted in the above-mentioned known technology includes a left camera and a right camera, and a signal processor processes the signals from the left camera and the right camera. A plurality of stereoscopic images captured by the image are synthesized into a panoramic mosaic image. In this case, a mathematical model is used to give depth to the spliced images.

The previous technology of the monitoring system constructed by using three-dimensional images is as Taiwan Patent Publication No. 201136313 (publication date: 2011/10/16), in which a dual-lens camera is used to monitor the monitoring scene, and the left and right lenses of the dual-lens camera shoot the monitoring scene The left and right frames of the scene image can obtain the left and right frames of scene images containing visual depth information, and then use the spatial image processing method to filter out the same image information in the left and right frames of the scene images that contain exactly the same part or the content is close to the part, and analyze the difference The information of each part is stored according to the time, and the video compressed image at that time point is read when needed, and the three-dimensional digital image picture is reconstructed. The dual-lens camera used is a pupil-distance dual-lens camera, which is used to shoot images at the same scene angle of the surveillance scene through its left and right lenses, and is also a way to create a stereoscopic image by using the parallax caused by the distance between the left and right eyes of the human eye.

Contents of the invention

Different from the technology of producing stereoscopic images in the prior art, the stereoscopic image processing system, device and method proposed by the present invention especially propose multiple cameras shooting the same place and the same space, preferably more than two cameras, so that one place, Images from multiple angles in the space can be analyzed to obtain the size, depth of field, and spatial relationship of various objects in the space, so that the three-dimensional images in the space can be completely reconstructed.

According to an embodiment, the stereoscopic image processing system includes a plurality of cameras respectively arranged at multiple positions to photograph a scene for generating continuous images of the scene, and a stereoscopic image processing device. The stereoscopic image processing device is connected with multiple cameras to obtain the image signals shot by each camera, which further includes an input unit for receiving the image signals shot by each camera, and a stereoscopic image forming unit, the stereoscopic image forming unit is connected to the input unit, through the software The method analyzes the received image signal, obtains one or more object images in the scene, and obtains the position and outline of each object at the shooting angle of each camera, as well as the distance between each object and each camera, so as to reconstruct the stereoscopic image of the object in the scene .

Embodiments further use software methods to analyze image signals, thereby identifying one or more objects in the scene, constructing the outline of each object, determining the distance between each object and each camera, and combining images captured by each camera at the same time. It can even judge the moving objects in the scene by comparing the images before and after a time.

In the embodiment of the stereoscopic image processing method, firstly, the images of the same scene captured by multiple cameras at different positions are obtained first, and the position, outline, and each object in the scene captured by each camera at different positions are obtained through image processing The distance between the object and each camera, and the position, outline and distance information of at least one object in the scene can be spliced according to each camera to form a stereoscopic image with depth of field.

In one embodiment, in the stitched stereoscopic images, the characteristics of each object can be used to identify the type of the object, which further includes the step of judging the moving object in the scene by comparing the images before and after a time.

In another embodiment, multiple cameras can be installed on a vehicle to capture scenes in different directions, that is, to capture images of the scene outside the vehicle, obtain the distance of objects in the scene through image processing, and even use image processing to To obtain the depth of the object, images with depth of field in the scene outside the vehicle can also be obtained through splicing.

In order to further understand the technology, method and effect that the present invention adopts to achieve the intended purpose, please refer to the following detailed description and accompanying drawings of the present invention, and believe that the purpose, characteristics and characteristics of the present invention can be deepened and concretely However, the accompanying drawings and appendices are provided for reference and illustration only, and are not intended to limit the present invention.

Description of drawings

FIG. 1 shows a schematic diagram of an embodiment of a stereoscopic image processing system of the present invention;

Fig. 2 shows the implementation schematic diagram of each camera shooting scene of the stereoscopic image processing system of the present invention;

FIG. 3 shows a schematic diagram of an embodiment of a stereoscopic image processing device of the present invention;

FIG. 4 shows a functional block diagram of an embodiment of the stereoscopic image processing system of the present invention;

FIG. 5 shows a schematic diagram of an embodiment of applying the stereoscopic image processing device of the present invention to a vehicle;

The process shown in FIG. 6 describes one of the embodiment processes of the stereoscopic image processing method of the present invention;

The process shown in FIG. 7 describes the second process of the embodiment of the stereoscopic image processing method of the present invention.

[Description of Reference Signs]

Shooting scene 10 first camera 101

Second camera 102 Third camera 103

Fourth camera 104 objects 11, 12, 13

Scene 20 First Camera 201

Second camera 202 Third camera 203

Fourth camera 204 object 22

Stereo image processing device 25

Stereo image processing device 30 signal processing unit 301

Signal input unit 302 Memory unit 303

Communication unit 304 Cameras 311, 312, 313, 314

Handheld 38 Network 32

Cloud Server 34

Image capture device 41 input unit 42

Stereoscopic image forming unit 43 contour building module 431

Distance Judgment Module 432 Image Stitching Module 433

Object recognition unit 44 Feature extraction unit 45

Comparison unit 46 motion detection unit 47

Database 48 Output Unit 49

Vehicle 50 first camera 501

Second camera 502 Third camera 503

Fourth camera 504

Steps S601-S609 One of the embodiment procedures of the stereoscopic image processing method

Steps S701-S707 The second embodiment flow of the stereoscopic image processing method

detailed description

For the embodiment of the stereoscopic image processing system, please refer to the schematic diagram of the embodiment of the stereoscopic image processing system of the present invention shown in FIG. 1 .

This example shows a shooting scene 10, with multiple cameras on the periphery, especially an IP camera (IPCAM) that can transmit image signals through the network or other camera groups with communication capabilities, preferably in a space A plurality of high-resolution cameras, in which a first camera 101 , a second camera 102 , a third camera 103 and a fourth camera 104 are schematically shown, capture images of one or more objects 11 , 12 , 13 in the shooting scene 10 respectively.

According to the embodiment of the invention, the first camera 101, the second camera 102, the third camera 103 and the fourth camera 104 shoot continuous images at different angles respectively. When setting up these cameras, it can be considered to obtain all the objects in the shooting scene 10 completely. 11 , 12 , 13 are set at various angles, so the three-dimensional images of all objects 11 , 12 , 13 in the scene 10 can be completely obtained through analysis, identification and image splicing.

FIG. 2 is a schematic diagram showing the implementation of shooting scenes by each camera in the stereoscopic image processing system of the present invention.

The schematic diagram shows that there are a first camera 201, a second camera 202, a third camera 203 and a fourth camera 204 in the scene 20. There is an object 22 in the scene 20. The system includes a stereoscopic image processing device 25 implemented by a computer, which can Continuous images captured by the first camera 201 , the second camera 202 , the third camera 203 and the fourth camera 204 are obtained at any time, such as a digital video recorder (DVR).

Stereoscopic image processing device 25 is connected to multiple cameras 201, 202, 203, 204 to obtain image signals shot by each camera, and is provided with input terminals for connecting and receiving multiple cameras 201, 202, 203, 204, and is mainly provided with a stereoscopic image forming unit, including various software or firmware modules, It is used to analyze the received image signal, obtain images of one or more objects 22 in the scene 20, obtain the position and outline of each object 22 at each camera shooting angle, and the distance between each object and each camera, so as to reconstruct the scene 20 Stereoscopic images of one or more objects 22 .

When the stereoscopic image processing device 25 obtains the image signals shot by the connected cameras 201, 202, 203, and 204, it can be compressed and stored as data for subsequent interpretation, and it can also perform real-time image processing to reflect changes in the scene 20 in real time, so it can be used as a monitoring system. use. However, in particular, the stereoscopic image processing device 25 proposed in the embodiment of the present invention is a device capable of processing three-dimensional image signals. When images from different angles in the same scene 20 are obtained through the cameras 201, 202, 203, and 204, the image signals are analyzed by means of software or firmware. , to obtain the information of the object 22 in the scene 20 , including obtaining the position of the object 22 through image processing, the distance between the same object 22 and each camera 201 , 202 , 203 , 204 , and recognizing the image contour of the object 22 . Further, the stereoscopic image of each object can be calculated according to these information.

Another embodiment may not be limited to a specific indoor place, but is used for the implementation of a three-dimensional outdoor space. For example, a vehicle carrier is provided with multiple cameras shooting from different angles, and the above-mentioned stereoscopic image processing device 25 is loaded, which can Cooperate with the on-board computer, in-vehicle display system, etc., after shooting the surrounding scenery of the carrier, use image processing technology to obtain the outline, depth and relationship between the objects of each scene in a specific space range, and then construct a three-dimensional view of the surrounding scenery image. If it is used in vehicles, it can provide a reference for drivers to easily judge the surrounding environment; if it is used in monitoring, it can obtain more clear image analysis conditions.

FIG. 3 shows an embodiment diagram of the circuit unit of the stereoscopic image processing device of the present invention.

The stereoscopic image processing device 30 is a device implemented by a computer, and is provided with a signal processing unit 301 , a signal input unit 302 , a memory unit 303 and a communication unit 304 according to circuit functions. The signal processing unit 301 of the stereoscopic image processing device 30 serves as the core of computing data, through which the signal input unit 302 is connected and receives image signals generated by cameras 311, 312, 313, 314, and the memory unit 303 stores the received image signals.

The stereoscopic image processing device 30 is then connected to an external terminal device through the communication unit 304, such as connecting the handheld device 38 through the network 32, or connecting to the cloud server 34, so that the processed image signal, especially the processed stereoscopic image signal, can be processed in real time. The image, or the unprocessed compressed image or the file processed by other means is transmitted to the handheld device 38 and then displayed in real time, as the purpose of mobile monitoring. After the data is transmitted to the cloud server 34, it can be stored and backed up, and can be transmitted to other devices through the cloud server 34.

However, for terminal devices such as the handheld device 38, related monitoring or browsing applications, such as mobile applications (APP), can be installed in the handheld device 38. When receiving the stereoscopic image signal transmitted from the stereoscopic image processing device 30 After that, the stereoscopic image in the shooting scene can be displayed smoothly.

For the whole stereoscopic image processing system, wherein the processing technology is implemented by means of software or hardware, the functional block diagram of related embodiments can be referred to FIG. 4 .

One end of the system has an image capture device 41 , which is installed at different positions in a space or scene as mentioned above, and in principle is set at a position where it can be fully presented so that various objects placed in the scene can be three-dimensionally displayed.

The system is connected to the image capture device 41 (can be multiple) through the input unit 42, and the related image signal is processed by the stereoscopic image forming unit 43 in the system to form a reconstructed object stereoscopic image, such as outputting the signal through the output unit 49.

The stereoscopic image forming unit 43 mainly obtains the image signals taken by the image capture devices 41 , and obtains image parameters reflecting various objects in the scene from the signals. For example, the contour building module 431 implemented by software or firmware means constructs the contour of the object from the image signals of the same object obtained from different angles, and the presentation method can represent a contour with data parameters; the distance judgment module 432 is used to obtain each object and each image captured The distance parameter of the device 41, the distance parameter of the same object obtained from different image capture devices 41 can determine the depth of field (depth of field) of each part of each object.

The images captured by different adjacent image capture devices 41 in a scene will have overlapping parts, so image processing technology can be used to obtain the overlap of the frames captured by each image capture device 41, and then perform image collage Paste, such as the image stitching module 433 implemented by software or firmware in the system, to combine the images shot by different adjacent image capture devices 41 at the same time, and obtain a complete image of the shooting scene through image collage, and then The object of the invention to form a stereoscopic image is achieved by cooperating with the reconstructed stereoscopic image obtained from the image signals of each object obtained by different image capture devices 41 .

After the stereoscopic image is reconstructed, the system can also include an object recognition unit 44, which can recognize various objects in the scene, as well as the quantity and arrangement position of one or more objects; 45 After data analysis, the characteristics of the object are obtained to obtain what the object is, such as judging what kind of object the object is based on the appearance of the object.

The system can also use the comparison unit 46 to accurately obtain the detailed features of the objects according to the characteristic parameters of various objects recorded in the database 48. Obtain the information of relevant persons, which can be used for monitoring or access control.

It is worth mentioning that if it is used for monitoring or access control, the main purpose is identification of people entering and leaving, so the facial images of people entering and leaving can be recorded in this database 48; or it can be combined with some security databases as a judgment of illegal intrusion.

The system can use the motion detection unit 47 to determine the moving objects in the scene by comparing images before and after a time. In this part, the changed images can be obtained by comparing the frames of the continuous images before and after the time, and the moving objects and related moving tracks can be obtained from the changes of a period of time. Finally, it is output through the output unit 49, including outputting to the monitoring device of the terminal as mentioned above, or uploading to the cloud system for other uses.

To give an example here, the database 48 may be a database that is linked to a community network or a specific user database, thereby obtaining characteristics of the users therein. After the system obtains the image of the person in the shooting scene, it can identify the identity of the person through comparison with the database 48, thus achieving the functions of monitoring, security and deterrence. For example, the database 48 uses Google (Google ^TM ) and Facebook (Facebook ^TM ). When such a huge amount of data is introduced, the identity of the person can be identified through comparison of facial features.

5 shows a schematic diagram of an embodiment of applying the stereoscopic image processing device of the present invention to a vehicle, that is, the system formed by the above-mentioned multiple cameras can be applied to obtain images of the scene outside the vehicle, and obtain various objects in the scene (such as buildings, other vehicles, Pedestrian) image information, the overlapping part of different camera images can also successfully obtain the depth information of this part, and the distance between each object and the vehicle, the depth of the object and other information can be obtained by using image processing technology in the part where there is no overlapping image . In an embodiment, for example, the depth information of an object may be analyzed and obtained according to the light and shadow of the object in the scene, the positional relationship between the objects, and the like.

This example shows that a vehicle 50 is equipped with a plurality of image capturing devices, which respectively capture scenes from different directions, such as the first camera 501, the second camera 501, and the first camera 501, which respectively capture scenes in front of the vehicle, scenes on the left side of the vehicle, scenes behind the vehicle, and scenes on the right side of the vehicle. The second camera 502 , the third camera 503 and the fourth camera 504 .

Through the software and hardware devices described in Figure 4 above, image acquisition, processing, and splicing can be performed. For example, the distance of objects in the scene can be obtained through image processing, and the depth of objects can be obtained after image processing. Vehicles can also be obtained through splicing. Imagery with depth of field in an external scene. In the same way, one or more objects in the scene can be judged through image processing, and the position and outline of each object at the shooting angle of each image capture device, as well as the distance between each object and each image capture device can be obtained, thereby forming a three-dimensional image with depth of field. Images, these image information will provide the driver to monitor the surrounding environment, obtain images of one or more objects in the scene, and provide software to judge abnormal conditions.

The flow shown in FIG. 6 describes the flow of an embodiment of the stereoscopic image processing method of the present invention.

The system shown in Fig. 1 and Fig. 2 includes a first camera, a second camera, a third camera and a fourth camera in different positions, which can be respectively set in four corners of a certain scene, or in a vehicle or A specific scene is shot on a specific object, and the number of cameras can be changed according to the design of the actual scene or object. In principle, the design is based on the design that can smoothly obtain images from every corner of the scene.

The flow shows that in step S601, the system obtains images of the same scene shot by multiple cameras at different positions through the above-mentioned stereoscopic image processing device; From image signals shot at different angles, at least the distance between the outline of each object and each camera can be determined.

As another example in step S603, the stereoscopic image forming unit implemented by means of software or firmware in the stereoscopic image processing device can analyze the images obtained from the above-mentioned cameras (such as the first camera, the second camera, the third camera, and the fourth camera) through software methods. Image signal, identify one or more objects in the scene through the object recognition function (object recognition unit), construct the contour of each object (such as through the contour building module), and judge the distance between each object and each camera (such as through the distance judgment module ).

The embodiment shows that an object can be photographed by at least two cameras, so that different angles of an object can be obtained, and the depth (or width) of the object can be obtained according to the outlined outline and various distance data, and through Various depth data can be used to determine the positional relationship between objects, so the spatial relationship of various objects in the entire scene can be obtained to facilitate image stitching. For example, through the image stitching function (image stitching module) in the stereoscopic image processing device to combine the images captured by the above-mentioned at least two cameras at the same time, the method is mainly to first obtain an environmental image in a space to form a real-time stereoscopic image , the details include obtaining the edge, size, and distance of objects in the space, performing image stitching for each object within the effective irradiation range of each camera in multiple images, and constructing a three-dimensional image with depth of field according to the distance and size, and then according to the spatial relationship of each object. Such as step S605.

In addition to storing the stereoscopic image formed at each time point for future use, it can also cooperate with the image processing system to perform object recognition in real time or afterwards, such as step S607, such as first identifying the characteristics of the object and comparing it with the database to obtain object type. Finally, output the result, as in step S609.

In the process of forming a stereoscopic image, as shown in FIG. 7, at least the outline, distance, and size of each object in the scene are analyzed and judged through image processing, such as step S701; the embodiment can be added to splicing the images shot by each camera before proceeding. Analysis; then obtain the characteristics of each object through the feature analysis function, as in step S703; compare the database, as in step S705, and obtain and output the identification result, as in step S707. In addition, the information of moving objects can be obtained by comparing the images of before and after time.

The output result can be sent to a user device together with information such as the identified object type, size, position, movement, etc. For example, the user device is installed with an application program that can access the above-mentioned stereoscopic image processing device, and a real-time image signal is obtained by streaming technology .

To sum up, in order to obtain the spatial relationship of various objects in a space or scene, the stereoscopic image processing system proposed by the present invention includes a plurality of image capture devices installed in the space, and a device for processing image signals, especially The software module analyzes the image signal, identifies the object, and obtains information such as the position, outline, and distance of each object, and can combine the images shot by each camera at the same time through the image stitching technology. Thus, it is possible to continue to recognize object features and determine moving objects in the scene. In particular, in addition to constructing 3D images for indoor spaces, the invention does not exclude the possibility of constructing 3D images for outdoor scenes, including proposing the positions, sizes and spatial relationships of objects therein, and further obtaining the moving routes of objects therein.

The above description is only a preferred feasible embodiment of the present invention, and does not limit the scope of patent protection of the present invention. Therefore, all equivalent structural changes made by using the description and illustrations of the present invention are all included in the scope of the present invention. within range.

Claims (15)

1. A stereoscopic image processing system, characterized in that said system comprises: A plurality of image capture devices are respectively installed at multiple positions in different shooting directions to shoot a scene to generate continuous images for shooting the scene; and A stereoscopic image processing device, connected to the plurality of image capture devices, to obtain image signals captured by each image capture device, including: an input unit for receiving image signals captured by each image capture device; A stereoscopic image forming unit, connected to the input unit, analyzes the received image signal, obtains one or more object images in the scene, obtains the position and outline of each object at the shooting angle of each image capture device, and the relationship between each object and each object. The distance of the image capture device to reconstruct a stereoscopic image of the one or more objects in the scene. 2. The stereoscopic image processing system as claimed in claim 1, wherein the stereoscopic image forming unit analyzes the image signal by software, identifies the one or more objects in the scene by an object recognition unit, and identifies the one or more objects in the scene by an outline The building module constructs the outline of each object, judges the distance between each object and each image capture device through a distance judgment module, and combines the images shot by each image capture device at the same time through an image splicing module. 3. The stereoscopic image processing system according to claim 2, further comprising a feature acquisition unit for analyzing the features of each object, and comparing with a database to determine the type of each object. 4. The stereoscopic image processing system as claimed in claim 3, further comprising a motion detection unit, which compares images before and after a time to determine the moving objects in the scene. 5. The stereoscopic image processing system according to claim 1, wherein the stereoscopic image processing device further comprises a pair of communication units for outputting stereoscopic image signals. 6. The stereoscopic image processing system according to claim 1, wherein the system comprises a first camera, a second camera, a third camera, and a fourth camera arranged at different positions, and the stereoscopic image forms The unit analyzes the image signals obtained from the first camera, the second camera, the third camera, and the fourth camera images through a software method, and recognizes the one or more objects in the scene through an object recognition unit, and through an object recognition unit The contour building module constructs the contour of each object, judges the distance between each object and each camera through a distance judgment module, and combines the first camera, the second camera, the third camera and the fourth camera images through an image stitching module images taken at the same time. 7. A stereoscopic image processing device, characterized in that the device comprises: A stereoscopic image processing device, connected to multiple image capture devices to obtain image signals captured by each image capture device, including: an input unit for receiving image signals captured by each image capture device; A stereoscopic image forming unit, connected to the input unit, analyzes the received image signal, obtains one or more object images in the scene, obtains the position and outline of each object at the shooting angle of each image capture device, and the relationship between each object and each object. The distance of the image capture device to reconstruct a stereoscopic image of the one or more objects in the scene. 8. The stereoscopic image processing device as claimed in claim 7, wherein the stereoscopic image forming unit analyzes the image signal through a software method, identifies the one or more objects in the scene through an object recognition unit, and recognizes the one or more objects in the scene through an outline The building module constructs the outline of each object, judges the distance between each object and each image capture device through a distance judgment module, and combines the images shot by each image capture device at the same time through an image splicing module. 9 . The stereoscopic image processing device according to claim 8 , further comprising a feature acquisition unit for analyzing the features of each object and comparing with a database to determine the type of each object. 10 . The stereoscopic image processing device according to claim 9 , further comprising a motion detection unit, which compares images before and after a time to determine the moving object in the scene. 11 . 11. The stereoscopic image processing device according to claim 7, further comprising a pair of communication units for outputting stereoscopic image signals. 12. A stereoscopic image processing method, characterized in that said method comprises: Obtain images of the same scene captured by multiple image capture devices at different locations; Obtaining the position and outline of at least one object in the scene captured by each image capture device at different positions through image processing, and the distance between each object and each image capture device; and The position, outline and distance information of at least one object in the scene are spliced according to each image capture device to form a stereoscopic image with a depth of field. 13 . The stereoscopic image processing method according to claim 12 , further comprising a step of obtaining the feature of the at least one object, and a step of identifying the at least one object according to the feature of the at least one object by comparing with a database. 14 . The stereoscopic image processing method according to claim 13 , wherein after identifying the type of the at least one object, the result is output to a user device. 15. The stereoscopic image processing method according to claim 12, further comprising a step of judging the moving object in the scene by comparing the images before and after a time.