WO2012124852A1 - Stereo camera device capable of tracking path of object in monitored area, and monitoring system and method using same - Google Patents

Abstract

Disclosed are a stereo camera device capable of tracking the path of an object in a monitored area, and a method thereof. The stereo camera device of the present invention extracts a moving object to be monitored from a stereo image captured within a specific monitored area, and then generates and provides an alarm to a monitoring device when the extracted object enters a specifically monitored zone while tracking the movement of the corresponding object.

Description

Stereo camera device capable of tracking the path of an object on the surveillance area, surveillance system and method using the same

The present invention relates to a stereo camera apparatus capable of recognizing an object based on 3D depth map data obtained by using two cameras, and more particularly, to track a location of a recognized object. A stereo camera device, a surveillance system and method using the same.

In order to obtain a depth map on a three-dimensional space from an image, that is, a distance from a subject in a three-dimensional space, a method of using a stereo camera, a method of using a laser scan, or a method of using a time of flight (TOF) The back is known.

Among these, stereo matching using a stereo camera is a hardware implementation of a process of recognizing a stereoscopic object using two eyes, and a pair of images obtained by photographing the same subject with two cameras. It is a method of extracting information about depth (or distance) in space through the interpretation process of. To this end, binocular differences on the same Epipolar Line of images obtained from two cameras are calculated. The binocular difference includes distance information, and the geometrical characteristic calculated from the binocular difference becomes the depth. When the binocular difference value is calculated in real time from the input image, three-dimensional distance information of the observation space can be measured.

Known as a stereo matching algorithm, for example, "image matching method using a plurality of image lines" of the Republic of Korea Patent No. 0517876 or "binocular difference estimation method for three-dimensional object recognition" of the Republic of Korea Patent No. 0601958.

Applicant has already invented an image recognition apparatus capable of distinguishing and recognizing objects in a space, in particular, objects to be monitored by a manager using such a stereo matching algorithm, and Korean Patent Application Nos. 10-2010-0039302 and 10-. 2010-0039366 is pending.

Applicants have found that such camera apparatuses and image recognition apparatuses can generate information not previously experienced while performing various functions, and the applicant feels the need to attempt various applications of such information.

For example, in a conventional surveillance system installed in an office, a house, or a museum to monitor an intruder, movement information in the surveillance zone of the monitor may be divided into sensing information about points and lines acquired through a detector and sensing information about a space. Can be. Detection of points or lines is the detection of the opening of a door or window, or the detection of an object on an infrared line using an infrared sensor.

However, the detection of the space is to detect specific changes in the space by using a method of detecting temperature change through infrared rays, and so on, and to monitor the detailed movement in the space by simply detecting whether an object moves in the space. This is not possible.

SUMMARY OF THE INVENTION An object of the present invention is a stereo camera device for recognizing an object based on 3D depth map data acquired by using two cameras and simultaneously monitoring and tracking the position of the recognized object in space. To provide a monitoring system and method used.

The stereo camera device of the present invention for achieving the above object, a stereo camera having a first camera and a second camera for photographing the same surveillance zone to generate a pair of stereo digital image; And an image processor extracting an object moving in the surveillance zone while calculating distance information of each pixel through image processing on the stereo digital image output from the stereo camera.

Here, the image processor is configured to set at least one monitoring zone defined by a pixel area set to belong to the image and a predetermined distance range for each pixel of the pixel area, and based on the calculated distance information for each pixel. If the extracted object is determined to be located in the surveillance zone includes a surveillance zone-alarm unit for generating a surveillance zone entry alarm and output to the external monitoring device.

According to an embodiment, the surveillance zone entry alert may preferably include an identifier of the surveillance zone in which the extracted object is located.

The image processor may include a distance information calculator configured to calculate 3D depth map data using the stereo digital image; An object extractor configured to extract an area of a moving object by comparing one of the stereo digital images with a reference background image; And an object recognizing unit for calculating an area or a representative length of the extracted object and recognizing the object as a monitoring target when the calculated area or representative length of the object corresponds to a preset range.

Surveillance system according to another embodiment of the present invention, the stereo camera device and a monitoring device for receiving an alarm from the stereo camera device to display to the administrator in the form of a digital map.

According to another aspect of the present invention, there is provided a method for monitoring a stereo camera device, the method including: generating a pair of stereo digital images using two cameras photographing the same surveillance zone; Extracting an object moving in the surveillance zone while calculating distance information of each pixel through image processing on the stereo digital image output from the stereo camera; Setting at least one surveillance zone defined by a pixel area set to belong to the image and a preset distance range for each pixel of the pixel area; And generating a surveillance zone entry alarm when the extracted object is determined to be located in the surveillance zone based on the calculated pixel-by-pixel distance information, and outputting the generated surveillance zone alert to an external monitoring apparatus.

When the stereo camera apparatus of the present invention recognizes an object moving in a surveillance zone and recognizes a moving object, the stereo camera device may provide location information on the space of the object in addition to providing a simple alarm for recognition of the object.

Accordingly, an alarm may be generated by detecting an object moving on a specific surveillance zone, and the location and movement path of the corresponding surveillance zone may be tracked.

This feature enables a stereo camera to provide intelligent surveillance beyond the function of a surveillance sensor, and provide stereoscopic information to a manager beyond a simple alarm level, depending on the device coupled to the stereo camera device of the present invention.

In addition, since the movement information on the surveillance space generated by the present invention is provided in units of detailed surveillance zones, the position information can be displayed on the digital drawing even with a lighter system resource, such as a portable monitoring apparatus.

1 is a block diagram of a surveillance system including a stereo camera device according to an embodiment of the present invention;

2 is a flowchart provided to explain an alarm output method of the stereo camera device of the present invention;

3 is a view showing the monitoring zone (S) and the monitoring zone (L1, L2, L3) according to an embodiment of the present invention,

4 is an example of an image photographing the surveillance zone S of FIG. 3;

5 is a view provided to explain a method for calculating an area of an object; and

6 is a view provided to explain a method of extracting a central axis of an object.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

Referring to FIG. 1, the surveillance system 100 of the present invention includes a stereo camera device 130 and a monitoring device 150 connected through a predetermined network 110 to monitor a moving subject in a three-dimensional space, and to alert an alarm. Can be output and its location displayed on the digital drawing. This monitoring system 100 can be used not only for security purposes, but also for any application that tracks the location of a specific object entering a specific space and needs the location information.

The stereo camera device 130 is installed in a specific surveillance zone for crime prevention and other purposes, and the monitoring device 150 is preferably located in a manager area away from the surveillance zone.

The network 110 may be an internal dedicated communication network or a commercial public network such as the Internet, a mobile communication network, a public line network (PSTN), or a wired as well as a wireless network. Although not shown in FIG. 1, the stereo camera device 130 and the monitoring device 150 should include interface means for connecting to the network 110.

The stereo camera device 130 includes a stereo camera 131 and an image processor 140, and may recognize an object moving on the surveillance area to determine whether the object is a specific monitoring target. The stereo camera 131 includes a first camera 133, a second camera 135, and an image receiver 137.

The first camera 133 and the second camera 135 are a pair of cameras spaced apart from each other to photograph the same surveillance zone, and are called a stereo camera. The first camera 133 and the second camera 135 output an analog (or digital) stereo video signal of the surveillance zone to the image receiver 137.

The image receiver 137 converts a video signal (or image) of a continuous frame input from the first camera 133 and the second camera 135 into a digital image, and synchronizes the frame to the image processor 140 in synchronization with the frame. to provide.

The image processor 140 extracts an area of an object moving on a shooting area (monitoring area) from a pair of digital image frames output from the image receiver 137 to determine whether the object is of interest, and the stereo camera device 130. The above determination process may be performed in real time on all frames of the image (video) which are continuously input from the.

For the above process, the image processor 140 includes a distance information calculator 141, an object extractor 143, an object recognizer 145, an object tracker 147, and a monitoring zone-alarm 149. . Hereinafter, operations of the distance information calculator 141, the object extractor 143, the object recognizer 145, the object tracker 147, and the monitoring zone-alarm 149 will be described with reference to FIGS. 2 to 4. Explain.

The first camera 133 and the second camera 135 are arranged to photograph a specific surveillance zone. When the first camera 133 and the second camera 135 generate an analog image signal, the image receiver 137 converts the analog image signal into a digital image signal and then provides the image processor 140 in synchronization with a frame. (Step S201).

<Object extraction and recognition step: S203>

The distance information calculator 141 calculates 3D depth map data including distance information of each pixel from a pair of digital images received in real time from the image receiver 137.

The object extractor 143 and the object recognizer 145 extract a region of a moving object from at least one image of a pair of digital images input through the image receiver 137. As described above, the applicant has already issued the patent application Nos. 10-2010-0039302 and 10-2010-0039366 regarding object recognition using a stereo camera. According to this, the object extractor 143 first extracts the moving object using a conventionally known image processing technique. Extraction of a moving object is performed by obtaining a differential image obtained by subtracting a basic background image from a newly input image.

<Determine whether or not to monitor: S205>

The object extractor 143 and the object recognizer 145 determine whether the extracted object is an object of a type monitored by the manager. For example, it is determined whether the object is a person, a car, or an animal, or if it is a person, it is determined whether or not the person is over a certain height. If the monitoring target is a person, the object extraction unit 143 and the object recognition unit 145 extracts and recognizes an object that is specifically determined as a person.

As described above, according to the applicant's patent application Nos. 10-2010-0039302 and 10-2010-0039366 regarding object recognition using a stereo camera, the object extraction unit 143 detects the outline of the object from the difference image. The object recognition unit 145 calculates the area of the object or the representative length of the object by using the depth map data calculated by the object line extracting unit 143 and the outline of the object extracted by the distance information calculating unit 141.

The object recognition unit 145 may determine whether the extracted object is an object of interest by determining whether the calculated area or representative length of the object falls within a preset area or length range of the object of interest.

<Alarm occurrence and object tracking: S207, S209>

The object tracking unit 147 tracks the recognized movement of the object of interest and provides the location information to the monitoring zone-alarm unit 149. There are many different image processing techniques already known in the art regarding such motion tracking, and such a method can be used as appropriate.

In addition, even if the object tracking algorithm by the object tracking unit 147 in particular, the operation of the monitoring zone-alarm section 149 below is the distance information calculation unit 141, the object extraction unit 143 and the object recognition unit 145 It is also possible to repeat the operation described above with respect to the stereo image input in real time from the image receiving unit 137.

The monitoring zone-alarm unit 149 generates an alarm and outputs the alarm to the monitoring device 150 once the object of interest appears in the image. Here, the alarm may correspond to an alarm in a pure sense including information on the surveillance zone in which the stereo camera device 130 is installed and information indicating that the monitored object appears in the surveillance zone.

FIG. 3 shows a surveillance zone S in which the stereo camera device 130 of the present invention is installed, and a plurality of surveillance zones L1, L2, and L3 existing in the surveillance zone S. FIG. 3 is an example of an image P photographing the surveillance zone S of FIG. 3.

For example, referring to FIG. 3, assuming that the identified monitored object moves within the surveillance zone S and moves to m1, m2, m3, and m4, the surveillance zone-alarm unit 149 monitors the image of FIG. 4. Generate an alert when the target object appears in m1.

<Specific Surveillance Zone Boundary Modes: S211, S213>

If the monitoring zone-alarm 149 determines that the object being tracked enters the special monitoring zone based on the information provided by the object tracking unit 147, the monitoring zone-alarm unit 149 generates the monitoring zone entry alarm and together with the location information. Output to the monitoring device 150.

3 and 4, the surveillance zone S is determined according to the installation position of the stereo camera device 130 and the angle of view of the camera device 130, while the surveillance zones L1, L2, and L3 are surveillance zones. It is set by the administrator within (S).

The monitoring zones L1, L2, and L3 are specified by pixel ranges L1-1, L2-2, and L3-3 indicating a corresponding zone in the image, and a distance range from the stereo camera device 130 to the corresponding zone. do.

For example, in the case of the first monitoring zone L1, the pixel range L1-1 shown in FIG. 4 and the distance range d1 to d2 are specified. In the case of the second monitoring zone L2 and the third monitoring zone L3, the pixel ranges of the second monitoring zone L2-1 and L2-2 overlap somewhat, but the distance ranges are different, and thus may be distinguished from each other. The monitoring zone-alarm unit 149 may obtain the distance to the object by grasping the distance information for each pixel based on the depth map data provided from the distance information calculator 141.

Therefore, the monitoring zone-alarm unit 149 determines that the object is located at m1 when the object being tracked from the object tracking unit 147 is in the L1-1 pixel range and the distance to the object is in the range d1 to d2. The monitoring zone entry alarm is generated and output to the monitoring device 150.

Here, the watched zone entry alert may basically include information (eg, watched zone identifier) and detection time information about a watched zone to which an object enters from a plurality of watched zones. In addition, the surveillance zone entry information may include the image itself of at least one frame in which the corresponding sensing moment is captured.

Subsequently, even when the object moves to m2 → m3 → m4 and subsequently enters the second monitoring zone L2, the third monitoring zone L3, and the first monitoring zone L1, the monitoring zone-alarm unit 149 is also provided. The monitoring zone entry information is output to the monitoring device 150.

Object tracking and monitoring of the stereo camera device 130 of the present invention is performed by the above method.

The monitoring device 150 of the present invention may be connected to the stereo camera device 130 through the network 110, and may receive various alarms from the stereo camera device 130.

The monitoring device 150 may correspond to not only a general computer but also a mobile phone, a PDA, a smart phone, and other dedicated terminals carried by an individual.

The monitoring device 150 may include a display unit (b) to visually check whether the object enters a specific monitoring zone in the monitoring zone in the case of the monitoring zone entry warning.

In addition, the monitoring device 150 may visually display the movement of the object in the surveillance area to the manager using an alarm provided by the stereo camera device 130. For example, the monitoring device 150 uses an alarm provided by the monitoring zone-alarm unit 149 while preserving a digital map as shown in FIG. 3 in which the monitoring zone S and the monitoring zones L1, L2, and L3 are displayed. 3 may be displayed to the user.

Information of the degree of this drawing is useful because it can be a small enough capacity to be transmitted to the portable monitoring device 150 by wire or wireless, and to be processed and visually displayed by the portable monitoring device 150.

In addition, the monitoring apparatus 150 further includes a voice guidance processing unit (not shown), and when the object approaches a specific surveillance zone, a predetermined guidance message (eg, through a speaker (not shown) installed in the surveillance zone S) (eg, will be able to output "step back from ooo").

According to an embodiment, the monitoring device 150 may retain the function of the monitoring zone-alarm unit 149 of the stereo camera device 130. In this case, the monitoring apparatus 150 may further include a monitoring zone unit (not shown) that determines whether the object enters a specific monitoring zone by using information calculated and provided by the object tracking unit 147.

Hereinafter, the calculation of the area and the representative length of the object of step S205 will be briefly described based on the first patent application Nos. 10-2010-0039302 and 10-2010-0039366.

In calculating the area of an object, the actual area per pixel (hereinafter, referred to as a 'unit area' of a pixel) at a distance (do) at which the object is extracted in operation S203 is obtained, and then the pixel included in the outline of the object is calculated. This is done by multiplying numbers.

Referring to FIG. 5, the actual area M corresponding to the entire frame at the maximum depth D and the actual area m corresponding to the entire frame at the position do of the extracted object based on the existing background image ( do) is displayed. First, the actual area m (do) corresponding to the entire frame at a distance do where the object is located may be obtained as in Equation 1 below.

Equation 1

Here, M is an actual area corresponding to the entire frame (eg, 720 × 640 pixels) at the maximum distance do based on the existing background image.

Next, by dividing the actual area m (do) corresponding to the entire frame at the distance (do) where the object is located by the number of pixels (C, for example, 460,800 = 720 x 640) of the entire frame, The unit area m _{p (do)} of the pixel is obtained as in Equation 2 below.

Equation 2

Where Q is the total number of pixels. According to Equation 2, it can be seen that m _{p (do)} depends on the distance do to the corresponding object confirmed from the distance information of the 3D depth map data.

Finally, the area of the object can be obtained as shown in Equation 3 by multiplying the unit area m _{p (do)} of the pixel by the number qc of the pixels included in the outline.

Equation 3

Where qc is the number of pixels included in the object.

Hereinafter, the process of calculating the representative length of the object of step S205 will be briefly described.

<Extract central axis of moving object: step S209>

The object recognizing unit 145 extracts a media axis of an object having a width of 1 pixel by applying a skeletal or thinning algorithm to the object extracted by the object extracting unit 143. Various known techniques such as the Medial Axis Transform (MAT) algorithm or Zhang Suen algorithm can be applied.

For example, according to the central axis transformation, the central axis a of the object is a set of points having a plurality of boundary points among the respective points (or pixels) in the object R as shown in FIG. 6. Here, the boundary point refers to a point closest to the point in the object among the points on the outline B, and the points b1 and b2 on the outline become the boundary point of the point P1 in the object R. Therefore, the central axis algorithm is a process of extracting points having a plurality of boundary points and may be expressed as in Equation 4 below.

Equation 4

Here, P _ma is a central axis represented by a set of x, x is a point present in the object R, b _min (x) is the number of boundary points of the point x. Thus, the central axis is a set of points x whose number of boundary points is greater than one. Here, in order to calculate the boundary point, the structure of the skeleton may change somewhat according to a method of obtaining a distance from an internal point x to an arbitrary pixel on the outline (for example, 4-Distance, 8-Distance, Euclidean Distance, etc.). .

In addition, when the object is a relatively simple form, the center line may be extracted by extracting a peak value of the Gaussian value for the object.

Once the center line is extracted, the representative length of the object is obtained using the depth map data. The representative length of the object is a value calculated from an image as an actual length of an object set to represent the object, and may correspond to an actual length of a central axis, an actual width of an object, or an actual height of an object. However, the representative length of the object is affected by the position of the camera, the shooting angle, and the characteristics of the shooting area.

Further, the calculation of the actual length of an object is a method of obtaining the actual length per pixel (hereinafter referred to as the 'unit length' of a pixel) at a distance (do) where the object is located, and then multiplying the number of pixels representing the object. Is done. Here, the number of pixels representing the object may correspond to the number of pixels forming the central axis, the number of pixels to be the width or height of the object.

The width or height of the object, as the number of pixels representing the object, can be obtained through the range of the x-axis coordinate or the y-axis coordinate of the object area, and the length of the central axis is, for example, the number of pixels included in the central axis. It can be obtained by adding.

The unit length of a particular pixel varies from pixel to pixel (exactly depending on the depth of the pixel), and can be obtained as follows with reference to FIG. 5. Here, for convenience of explanation, it is assumed that the size of the image frame is 720x640 pixels.

5, the actual length Lmax corresponding to the vertical axis (or horizontal axis) of the entire frame at the maximum depth D based on the existing background image, and the vertical axis (or horizontal axis) of the entire frame at the position l of the extracted object. The corresponding actual length L (do) is indicated. First, the actual length L (do) corresponding to the vertical axis (or the horizontal axis) of the entire frame at the depth do where the object is located may be obtained as in Equation 5 below.

Equation 5

Here, L (do) is the actual length corresponding to the vertical axis (or horizontal axis) of the entire frame at the depth do, and Lmax is the vertical axis (or horizontal axis) of the entire frame at the maximum depth D based on the existing background image. The corresponding actual length.

Next, the actual length L (do) corresponding to the vertical axis (or horizontal axis) of the entire frame at the distance do is located, is the number of pixels Px, Py in the vertical axis (or horizontal axis) of the entire frame. = 720, Py = 640), the unit length L _p (do) of the pixels included in the object region can be obtained as in Equation 6 below.

Equation 6

Here, L _p (do) is the unit length of the pixel included in the object region located at the depth do, and Qy is the number of pixels along the vertical axis of the entire frame. According to Equation 5, it can be seen that L _p (do) depends on the depth to the corresponding object confirmed from the distance information of the 3D depth map data and the maximum depth on the map data.

When the unit length of the pixel is obtained, the object recognition unit 145 obtains the representative length of the object. The representative length of the object may be calculated by Equation 7 by multiplying the unit length L _p (do) of the pixel by the number qo of the pixels representing the object.

Equation 7

Here, qo is the number of pixels representing the object.

Although the above has been illustrated and described with respect to preferred embodiments of the present invention, the present invention is not limited to the above-described specific embodiments, it is usually in the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

Claims (5)

A stereo camera having a first camera and a second camera for photographing the same surveillance zone to generate a pair of stereo digital images; And And an image processing unit for extracting an object moving in the surveillance zone while calculating distance information of each pixel through image processing on the stereo digital image output from the stereo camera. The image processor, Setting at least one monitoring zone defined by a pixel area set to belong to the image and a preset distance range for each pixel of the pixel area, and the extracted object is monitored by the extracted object based on the calculated distance information for each pixel. And a monitoring zone-alarm unit which generates a monitoring zone entry alarm and outputs the monitoring zone entry alarm to the external monitoring device when it is determined to be located in the zone. The method of claim 1, And the surveillance zone entry alert includes an identifier of a surveillance zone in which the extracted object is located. The method of claim 1, The image processor, A distance information calculator configured to calculate 3D depth map data using the stereo digital image; An object extractor configured to extract an area of a moving object by comparing one of the stereo digital images with a reference background image; And And a object recognition unit calculating an area or a representative length of the extracted object and recognizing the object as a monitoring target when the calculated area or representative length of the object corresponds to a preset range. . A stereo camera having a first camera and a second camera for photographing the same surveillance zone, and generating a pair of stereo digital images, and the distance information of each pixel through image processing of the stereo digital images output from the stereo camera. A stereo camera device having an image processing unit for extracting an object moving in the surveillance zone while calculating; And Receiving an alarm from the stereo camera device includes a monitoring device for displaying to the administrator in the form of a digital map, The image processor, Setting at least one monitoring zone defined by a pixel area set to belong to the image and a preset distance range for each pixel of the pixel area, and the extracted object is monitored by the extracted object based on the calculated distance information for each pixel. And a monitoring zone-alarm unit for generating a monitoring zone entry alarm and outputting the monitoring zone entry alarm to the monitoring device when it is determined to be located in the zone. Generating a pair of stereo digital images using two cameras photographing the same surveillance zone; Extracting an object moving in the surveillance zone while calculating distance information of each pixel through image processing on the stereo digital image output from the stereo camera; Setting at least one surveillance zone defined by a pixel area set to belong to the image and a preset distance range for each pixel of the pixel area; And And generating a monitoring zone entry alarm and outputting the generated warning signal to an external monitoring apparatus when it is determined that the extracted object is located in the monitoring zone based on the calculated distance information for each pixel. How to monitor the device.

Images