WO2012133962A1 - Apparatus and method for recognizing 3d movement using stereo camera - Google Patents

Abstract

An apparatus and a method for recognizing 3D movement using a stereo camera are disclosed. The apparatus for recognizing 3D movement, according to the present invention, calculates information on the movement of a pointer by means of calculating the actual position, the direction of movement, the distance of movement, and the speed of the pointer, which corresponds to a specific area of the human body to be tracked, by using the stereo camera. The information which is extracted and calculated by the present invention enables the gesture of a person in a 3D space to be turned into a pattern and information.

Description

3D motion recognition device and recognition method using stereo camera

The present invention relates to a three-dimensional motion recognition apparatus and a recognition method that can recognize and inform a three-dimensional gesture of a user using a stereo camera.

The ability of computers, including computers, to recognize patterns of movement beyond simply recording human movements could be a new innovation in information technology. For example, it may be a fragmentary example that recent game devices detect a user's motion and proceed with the game in response to the motion.

As such, the technology for recognizing the user's three-dimensional gesture is difficult to guess the limit of the range of application, including various robots, as well as the user interface area for general information equipment.

Although there have been studies in this field in the past, most of the solutions are as far as the user uses a separate wearable interface, and the solution of directly interpreting the user's gesture using a camera is It has not been presented yet. However, it is clear that it is not too long to come true from science fiction films and the like, such as watching a three-dimensional stereoscopic image displayed in the form of a monitor or a hologram and virtually manipulating a device represented by the stereoscopic image.

An object of the present invention is to provide a three-dimensional gesture recognition apparatus and a recognition method that can recognize and inform the three-dimensional gesture of the user using a stereo camera.

A three-dimensional motion recognition method using a stereo camera to achieve the above object, comprising: generating a stereo image by photographing a moving object with a stereo camera; Calculating depth map data for each pixel in the stereo image; Extracting the moving object from the stereo image; Recognizing a pointer to be a motion recognition target from an object region of the image through image processing of the image; Tracking the change of the position of the pointer in the three-dimensional space by performing the calculating of the depth map data or recognizing a pointer for each image frame continuously generated in the generating step; And calculating and outputting information on the three-dimensional moving direction of the pointer by using the changed three-dimensional spatial position information of the tracked pointer.

Here, in the step of recognizing the pointer, it is preferable to extract the central axis of the object, and then recognize the end of the central axis as a pointer.

In the step of tracking the change of position, the position of the pointer in the three-dimensional space is preferably the coordinate of the pointer in each image frame and the depth in the coordinate extracted from the depth map data.

According to an embodiment of the present disclosure, the outputting may be performed by calculating a distance per unit pixel of the object region in the image and multiplying the number of pixels moved by the pointer to calculate the actual spatial movement distance of the pointer. The distance per unit pixel can be obtained by the following equation.

Where L _p (do) is the unit length of the pixels included in the object area located at the depth do, the Qxy is the number of pixels along the horizontal or vertical axis of the entire frame.

Motion recognition apparatus according to another embodiment of the present invention, the stereo camera unit for generating a stereo image by photographing a moving object with a stereo camera; A distance information calculator for calculating depth map data for each pixel in each frame of the stereo image which is continuously input from the stereo camera unit; An object extracting unit extracting the moving object from each of the image frames; A motion tracking unit for tracking a position change in the three-dimensional space of the pointer by performing a process for recognizing a pointer that is a motion recognition target among the object regions extracted by the object extraction unit for each of the image frames; And a motion information output unit for calculating and outputting information on the three-dimensional moving direction of the pointer by using the changed three-dimensional spatial position information of the tracked pointer.

The motion recognition apparatus of the present invention may recognize three-dimensional gestures of humans that may occur arbitrarily in a three-dimensional space, and generate information regarding a direction and a speed at which a specific human body part moves.

1 is a block diagram of a gesture recognition system according to an embodiment of the present invention;

2 is a flowchart provided for explaining a three-dimensional motion recognition method of the present invention;

3 is a view provided to extract the central axis of the object,

4 exemplarily shows an actual operating state of the recognition system of the present invention;

FIG. 5 is a diagram illustrating an image processing result for the images photographed in FIG. 4; and

6 is a view provided for the calculation of the area and representative length of an object.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

Referring to FIG. 1, the gesture recognition apparatus 100 may include a stereo camera unit 110 and an image processor 130 to recognize a user's three-dimensional motion in a three-dimensional space.

The motion recognition apparatus 100 of the present invention photographs the user using a stereo camera as shown in FIG. 4, and recognizes the user's three-dimensional gesture through image processing of the photographed image. Here, the recognition of the three-dimensional gesture means information about a moving direction, a moving distance, and a speed of a human body part (hereinafter, referred to as a 'pointer') of the user's body to be tracked (hereinafter, 'motion information' Means to create the In addition, the pointer may mean a part of the body such as a hand, an arm, a foot, a leg, a head, a finger, and may be a plurality of parts, such as two hands. The direction of movement includes rotation as well as any direction including up / down / left / right / front / rear in a three-dimensional virtual space with respect to the camera view.

In FIG. 4, the user moves his or her hand up and down. If the user's hand is set as a pointer in the gesture recognition apparatus 100, the gesture recognition apparatus 100 recognizes that the user's hand has moved at a certain speed by a certain distance in the Y direction and generates the information.

For this operation, the motion recognition apparatus 100 includes a stereo camera to generate a stereo image for the user and perform image processing.

Referring to FIG. 1, the gesture recognition apparatus 100 may include a stereo camera unit 110 and an image processor 130 to recognize a three-dimensional motion of a user's human pointer in a three-dimensional space.

The stereo camera unit 110 includes a first camera 111, a second camera 113, and an image receiver 115.

The first camera 111 and the second camera 113 are a pair of cameras spaced apart from each other to photograph the same area, and are called a stereo camera. The first camera 111 and the second camera 113 output an analog image signal photographing an area to the image receiver 115. The stereo camera may be used to extract the actual distance to the subject.

The image receiver 115 converts a video signal (or image) of a continuous frame input from the first camera 111 and the second camera 113 into a digital image and synchronizes the frame to the image processor 130 in synchronization with the frame. to provide.

According to an exemplary embodiment, the first camera 111 and the second camera 113 of the stereo camera unit 110 may be a camera that generates a digital video signal instead of an analog image. In this case, the image receiver 115 may be different. It provides an interface with the image processor 130 without conversion processing and serves to match frame synchronization of a pair of images.

The image processor 130 recognizes the user's pointer by using the digital stereo image continuously output from the stereo camera unit 110 and recognizes the user's three-dimensional motion by tracking the movement of the pointer.

For the above process, the image processor 130 includes a distance information calculator 131, an object extractor 133, an object recognizer 135, a motion tracker 137, and a motion information output unit 139. . Hereinafter, operations of the distance information calculator 131, the object extractor 133, the object recognizer 135, the motion tracker 137, and the motion information output unit 139 will be described with reference to FIG. 2.

First, the first camera 111 and the second camera 113 are arranged to photograph a specific space. When the first camera 111 and the second camera 113 generate an analog image signal, the image receiver 115 converts the analog image signal into a digital image signal and then provides the image processor 130 in synchronization with a frame. (S201).

<Create Depth Map Data: Step S203>

Motion recognition is performed in the order of (1) extracting moving objects, (2) determining whether an object is a human body, and (3) tracking a preset pointer part in the case of a human body, and (4) generating motion information for a pointer. Here, in the processes of (2) to (4), measured distance information from the first and second cameras 111 and 113 to the corresponding object is required.

To this end, the distance information calculator 131 obtains distance information of the capsular body captured by each pixel in units of pixels using a pair of digital images received in real time from the image receiver 115, and then uses a three-dimensional depth map. Compute (3D Depth Map) data. Therefore, the depth map data includes distance information for each pixel.

Here, the distance information of each pixel is binocular difference information obtained by a stereo matching method known in the art, and "image matching method using multiple image lines" of Korean Patent No. 0517876 or "3D object" of Korean Patent No. 0601958. The depth map data calculated by the distance information calculator 131 can be calculated using the graph cut algorithm presented in the binocular difference estimation method for recognition. Contains information about the actual distance.

<Extraction step of moving object: step S205>

The object extractor 133 extracts an area of a moving object by performing image processing on one image (or both images) of a pair of digital images input through the image receiver 115.

Applicant has already applied for patent applications 10-2010-0039302 and 10-2010-0039366 relating to a method for recognizing a moving object, especially a person, using a stereo camera.

According to this, the extraction of the moving object is performed by obtaining a differential image obtained by subtracting a background image from a newly input image. However, in the present invention, the background image may be a fixed value, but even if the image includes the region determined as the moving object, the background image may be the background image if the pointer is not recognized. However, the basic background image used to obtain the depth map data or the area or representative length of the object to be described below using the depth map data should be a preset image.

<Determine whether the extracted object is a human body: step S207>

When the object is extracted, the object extractor 133 and the object recognizer 135 first determine whether the extracted object is a human body. For example, it is determined whether the object is a human, an animal, or an object.

For object recognition, the object extractor 133 detects the outline of the object from the difference image, and the object recognizer 135 calculates the outline and distance information calculator 131 of the object extracted by the object extractor 133. The depth map data is used to determine the area of the object or the representative length of the object.

The object recognition unit 135 may determine whether the extracted object is a human body by determining whether the calculated area or representative length of the object falls within a preset area or length range of the human body.

Regarding the detection and calculation of the outline / area / representation of the object, the aforementioned patent applications Nos. 10-2010-0039302 and 10-2010-0039366 of the applicants mentioned above have outline detection, objects using stereo cameras, especially humans. It is presented on how to recognize the system, and the method will be described again below.

<Pointer Recognition Using Centerline of Object: Step S209>

The motion tracker 137 extracts a media axis of an object having a width of 1 pixel by applying a skeletonization or thinning algorithm to the object extracted by the object extractor 133. As the skeletalization algorithm, various known methods such as a Medial Axis Transform (MAT) algorithm using an outline or Zhang Suen algorithm can be applied.

For example, according to the central axis transformation, the central axis a of the object is a set of points having a plurality of boundary points among the respective points (or pixels) in the object R as shown in FIG. 3. Here, the boundary point refers to a point closest to the point in the object among the points on the outline B, and the points b1 and b2 on the outline become the boundary point of the point P1 in the object R. Therefore, the central axis algorithm is a process of extracting points having a plurality of boundary points and may be expressed as in Equation 1 below.

Equation 1

Where P _ma is a central axis represented by a set of x, x is a point present in the object R, and b _min (x) is the number of boundary points of the point x. Thus, the central axis is a set of points x whose number of boundary points is greater than one. Here, in order to calculate the boundary point, the structure of the skeleton may change somewhat according to a method of obtaining a distance from an internal point x to an arbitrary pixel on the outline (for example, 4-Distance, 8-Distance, Euclidean Distance, etc.). .

In addition, when the object is in a relatively simple form, the center line may be extracted by extracting a peak value of the Gaussian value for the object, and in this case, the edge detection of step S207 may be omitted.

When the central axis of the object is extracted, the motion tracking unit 137 recognizes the pointer using the central axis information. Since the pointer is preferably located at the end of the centerline of the human body, such as the head, hands, and feet, the pointer recognition of the motion tracker 137 may correspond to the end of the centerline of the extracted object.

<Tracking the movement of the pointer: steps S211 and S213>

Since motion recognition assumes movement, if the pointer recognized by the motion tracking unit 137 does not move, there is no information on the corresponding motion. Therefore, the motion tracking unit 137 repeatedly performs the image processing of steps S203 to S209 for all image frames generated in step S201 and continuously inputted until the tracking is finished until the pointer is recognized. It is determined whether the movement (S211).

As mentioned above, when the pointer is assigned to at least one of the head, the hand, and the foot, the pointer tracking of the motion tracking unit 137 is inputted in a chronological manner, as shown in FIG. Corresponds to tracking the movement of the end of the centerline in the image of.

FIG. 5 exemplarily illustrates an image processing result of images of the user photographed in FIG. 4. In FIG. 5, images M1, M2, and M3 obtained by sequentially processing image lines and extracting the center lines M1, M2, and M3 are illustrated, and two hands of the user are pointers m11, m12, m21, m22, m31, and m32. If set to). Accordingly, the motion tracker 137 tracks the movement of the two ends m11, m12, m21, m22, m31, and m32 of the center line in each of the images M1, M2, and M3.

In FIG. 5, since only the ends m11, m21, and m31 corresponding to the left hand are moving in each center line M1, M2, and M3, only motion information for the corresponding pointers m11, m21, and m31 will be generated.

The motion tracking unit 137 extracts the position information of each of the pointers m11, m12, m21, m22, m31, and m32 from the image of each frame that is continuously input and processed and provides the motion information output unit 139. . Here, the position information includes the coordinates of the pointer (or the pixel) in the image and the depth at the corresponding coordinate pixel. Here, the depth is extracted from the depth map data calculated by the distance information calculator 131 for the corresponding image frame. The pointer tracking of the motion tracking unit 137 continues while the pointers m11, m12, m21, m22, m31, and m32 are moved.

<Generate Motion Information of Pointer: Step S215>

The motion information output unit 139 is based on the location information of each of the pointers m11, m12, m21, m22, m31, and m32 in the image of each frame provided by the motion tracker 137, and the direction of movement of the pointer. The motion information of the pointer including the actual moving distance is calculated. In addition, the motion information output unit 139 calculates a moving speed based on the image frame period.

The direction of movement can naturally be extracted by drawing the movement vector of the pointer in the three-dimensional virtual space with the coordinates of the pointer and the depth information of the coordinates.

The actual moving distance of the pointer may use the horizontal distance, the vertical distance, and the depth map data per unit pixel of the pointer coordinate pixel. Here, the distance per unit pixel may use Equation 6 below. Equation 6 is the vertical length of each pixel, but the horizontal length may be obtained in the same manner.

As described above, since only the left hand pointers m11, m21 and m31 move, only motion information for the left hand pointers m11, m21 and m31 will be generated.

In the above method, a motion recognition method using a stereo camera of the 3D motion recognition apparatus 100 of the present invention is performed.

Hereinafter, the outline detection of the step S207, the calculation of the area and the representative length of the object will be briefly described based on the first patent application Nos. 10-2010-0039302 and 10-2010-0039366.

First, in order to detect the outline of the extracted object, the object extractor 133 detects the outline of the moving object by performing outline detection on the result of the subtraction operation of step S205. Edge detection is handled using different types of edges, depending on the borderline width and shape of the object.

The object extractor 133 may remove a noise by applying a morphology operation to a subtraction image and simplify an outline or a skeleton line to detect an outline. The morphology operation can basically use erosion operation to remove noise and dilation operation to fill small holes in an object.

In calculating the area of an object, the actual area per pixel (hereinafter, referred to as a 'unit area' of a pixel) at a distance (do) at which the object is extracted in operation S203 is obtained, and then the pixel included in the outline of the object is calculated. This is done by multiplying numbers.

Referring to FIG. 6, the actual area Nmax corresponding to the entire frame at the maximum depth D based on the basic background image, and the actual area N corresponding to the entire frame at the position of the extracted object do ( do) is displayed. First, the actual area N (do) corresponding to the entire frame at a distance do where the object is located may be obtained as in Equation 2 below.

Equation 2

Here, Nmax is an actual area corresponding to the entire frame (eg, 720 × 640 pixels) at the maximum distance do based on the existing background image.

Next, by dividing the actual area N (do) corresponding to the entire frame at the distance do by which the object is located by the number of pixels (eg, 460,800 = 720 × 640) of the entire frame, the object area is included in the object area. The unit area N _{p (do)} of the pixel is obtained as in Equation 3 below.

Equation 3

Where Q is the total number of pixels. According to Equation 3, it can be seen that N _{p (do)} depends on the distance do to the corresponding object confirmed from the distance information of the 3D depth map data.

Finally, the area of the object can be obtained as shown in Equation 4 by multiplying the unit area N _{p (do)} of the pixel by the number qc of the pixels included in the outline as described above.

Equation 4

Where qc is the number of pixels included in the object.

Hereinafter, the process of calculating the representative length of the object will be briefly described.

As described with reference to Equation 1, when the center line is extracted, the representative length of the object is obtained using the depth map data. The representative length of the object is a value calculated from an image as an actual length of an object set to represent the object, and may correspond to an actual length of a central axis, an actual width of an object, or an actual height of an object. However, the representative length of the object is affected by the position of the camera, the shooting angle, and the characteristics of the shooting area.

Further, the calculation of the actual length of an object is a method of obtaining the actual length per pixel (hereinafter referred to as the 'unit length' of a pixel) at a distance (do) where the object is located, and then multiplying the number of pixels representing the object. Is done. Here, the number of pixels representing the object may correspond to the number of pixels forming the central axis, the number of pixels to be the width or height of the object.

The width or height of the object, as the number of pixels representing the object, can be obtained through the range of the x-axis coordinate or the y-axis coordinate of the object area, and the length of the central axis is, for example, the number of pixels included in the central axis. It can be obtained by adding.

The unit length of a particular pixel varies from pixel to pixel (exactly depending on the depth of the pixel), and can be obtained as follows with reference to FIG. 6. Here, for convenience of explanation, it is assumed that the size of the image frame is 720x640 pixels.

In FIG. 6, the actual length Lmax corresponding to the vertical axis (or horizontal axis) of the entire frame at the maximum depth D based on the basic background image, and the vertical axis (or horizontal axis) of the entire frame at the position l of the extracted object. The corresponding actual length L (do) is indicated. First, the actual length L (do) corresponding to the vertical axis (or the horizontal axis) of the entire frame at the depth do where the object is located may be obtained as in Equation 5 below.

Equation 5

Here, L (do) is the actual length corresponding to the vertical axis (or horizontal axis) of the entire frame at the depth do, and Lmax is the vertical axis (or horizontal axis) of the entire frame at the maximum depth D based on the existing background image. The corresponding actual length.

Next, the actual length L (do) corresponding to the vertical axis (or the horizontal axis) of the entire frame at the distance (do) at which the object is located is determined by the number of pixels Qx, Qy in the vertical axis (or the horizontal axis) of the entire frame. = 720, Qy = 640), the unit length L _p (do) of the pixels included in the object region can be obtained as in Equation 6 below.

Equation 6

Here, L _p (do) is the unit length of the pixel included in the object region located at the depth do, and Qy is the number of pixels along the vertical axis of the entire frame. According to Equation 6, it can be seen that L _p (do) depends on the depth to the corresponding object confirmed from the distance information of the 3D depth map data and the maximum depth on the map data.

Previously, the vertical axis of the unit pixel used to calculate the movement distance of the pointer obtained in step S215 may be used as it is, and the horizontal axis length is obtained by inputting the horizontal axis Qx of the entire frame instead of Qy in Equation 6. It can be done.

When the unit length of the pixel is obtained, the object recognition unit 135 calculates the representative length of the object. The representative length of the object may be calculated by Equation 7 by multiplying the unit length L _p (do) of the pixel by the number qo of the pixels representing the object.

Equation 7

Here, qo is the number of pixels representing the object.

Although the above has been illustrated and described with respect to preferred embodiments of the present invention, the present invention is not limited to the above-described specific embodiments, it is usually in the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

Claims (5)

Generating a stereo image by photographing a moving object with a stereo camera; Calculating depth map data for each pixel in the stereo image; Extracting the moving object from the stereo image; Recognizing a pointer to be a motion recognition target from an object region of the image through image processing of the image; Tracking the change of the position of the pointer in the three-dimensional space by performing the calculating of the depth map data or recognizing a pointer for each image frame continuously generated in the generating step; And Computing and outputting information on the three-dimensional moving direction of the pointer using the changed three-dimensional spatial position information of the tracked pointer, 3D motion recognition method using a stereo camera. The method of claim 1, Recognizing the pointer, And extracting a central axis of the object and recognizing an end of the central axis as a pointer. The method of claim 1, In the step of tracking the position change, the position of the pointer in the three-dimensional space is a coordinate in the coordinates of the pointer in each image frame and the depth in the coordinates extracted from the depth map data using a stereo camera 3D motion recognition method. The method of claim 1, The outputting step may be performed by calculating a distance per unit pixel of the object area in the image and multiplying the number of pixels moved by the pointer, thereby calculating the actual spatial movement distance of the pointer. The distance per unit pixel is expressed by the following equation

To obtain, The L _p (do) is a unit length of a pixel included in the object area located at a depth do, the Qxy is a three-dimensional motion recognition method using a stereo camera, characterized in that the number of pixels in the horizontal or vertical axis of the entire frame. A stereo camera unit generating a stereo image by photographing a moving object with a stereo camera; A distance information calculator for calculating depth map data for each pixel in each frame of the stereo image which is continuously input from the stereo camera unit; An object extracting unit extracting the moving object from each of the image frames; A motion tracking unit for tracking a position change in the three-dimensional space of the pointer by performing a process for recognizing a pointer which is a motion recognition target among the object regions extracted by the object extraction unit for each of the image frames; And 3D motion recognition using a stereo camera, characterized in that it comprises a motion information output unit for calculating and outputting information about the three-dimensional movement direction of the pointer using the changed three-dimensional spatial position information of the tracked pointer. Device.

Images