CN106446837A - Hand waving detection method based on motion historical images - Google Patents

Abstract

The present invention proposes a hand-waving detection method based on motion history images. Firstly, the area of hand-waving detection is determined indirectly through pedestrian detection, then the main motion direction of the area is calculated through motion history images, and finally whether there is waving motion in sequence images is judged. . The method proposed by the invention has low complexity and high detection accuracy, and is suitable for the target confirmation stage before the drone target tracking.

Description

A Hand Waving Detection Method Based on Motion History Images

technical field

The invention relates to the technical fields of image processing and computer vision processing, in particular to a hand-waving detection method based on motion history images.

Background technique

With the rapid development of automation technology, the functions of drones are becoming more and more powerful, and automatic tracking and shooting of targets has become an important part of drone applications. The person or object to be tracked needs to be selected before the UAV is followed. In the traditional target selection, the method of manual frame selection is usually used to select the target. This method is simple in principle and relatively easy to implement, but it also has obvious defects: 1. The manual animation frame is inaccurate and depends on the operator's proficiency in the use of equipment and software; 2. The frame is often implemented on handheld devices such as mobile phones. It cannot be automatically recognized when it is separated from the mobile phone.

In order to realize the target selection of automatic tracking of pedestrians, a new idea is to let the tracked object perform certain behaviors such as waving, and then determine the tracking target through behavior (such as waving) detection.

Behavior detection is an important topic in computer applications, and has broad application prospects in human-computer interaction and virtual reality. In the early stage of its development, it mainly used hardware equipment as the front end to obtain initial motion information, such as data gloves used in hand motion detection systems. The hardware equipment required by this type of method is often relatively expensive, and needs to be worn so that the user's comfort is not high.

With the development of computer vision technology, software-based hand movement detection technology has also begun to develop. At present, the processing flow of the hand motion detection system is mainly divided into two parts. The first part is to extract the required information from the video sequence frame. The required information here mainly refers to the relative position of the moving object on the video frame. The current method mainly includes skin color division, Foreground division, geometry division, etc. The second part is to process the previously obtained image features through computational models, such as hidden Markav chain, wavelet transform, BP neural network and other methods.

At present, hand motion detection is mainly a combination of multiple algorithms, and generally requires the combination of hand feature extraction algorithms and motion detection algorithms. For applications with higher recognition requirements, more features and complex detection algorithms need to be extracted. However, the target waving detection algorithm for drone tracking needs to run on the mobile phone, which cannot provide higher computing power. Therefore, the existing methods are not suitable for target waving detection for drone tracking.

Contents of the invention

Aiming at the deficiencies of the existing hand-waving detection methods, the present invention proposes a hand-waving detection method based on motion history images.

The technical scheme adopted in the present invention is:

A hand-waving detection method based on motion history images, comprising the following steps:

S1 trains a pedestrian detector;

S2 When the UAV is tracking the target, detect pedestrians on the captured video images, sort the pedestrian candidate frames according to the score of the classifier, and select the pedestrian candidate frame with the highest score as the object of the initial waving detection;

S3 determines the waving detection area for the currently highest-scoring pedestrian candidate frame, and then performs waving detection in the waving detection area through motion history images;

S4 When the waving movement is detected, the overall process ends; if the waving behavior is not detected in the pedestrian candidate frame with the highest score, the pedestrian candidate frame with the second score in the ranking results of the pedestrian detection classifier in S2 is detected for waving, The waving detection method is the same as the waving detection method for the initial waving detection object; according to this rule, until the waving behavior is detected in a certain pedestrian candidate frame or all the pedestrian candidate frames have completed the waving detection but there is no waving behavior.

In S1 of the present invention, the training method of the pedestrian detector is: shoot different pedestrians through the camera on the drone, collect a total of no less than 500 pedestrian images as positive samples, and the number of pedestrians participating in the collection is no less than 100 . Then collect more than 200 images that do not include pedestrians from the Internet or various other databases as negative samples. Normalize the collected pedestrian images to an image with a size of 108*36, select the classic histogram of oriented gradient (HOG) feature extraction method to extract features from positive and negative samples, use svm for training, and obtain a pedestrian detector. (Note here: HOG+SVM was proposed by French researcher Dalal on the CVPR in 2005 for pedestrian detection, and the present invention uses this method completely.)

The method of S2 of the present invention is: when the unmanned aerial vehicle carries out target follow-up shooting, firstly carry out pedestrian detection to the photographed video image, the pedestrian detection process adopts the mode of sliding window (window size is 108*36) to carry out, extract the frame in the frame to be detected The HOG feature of the image is classified by the pedestrian detector to obtain a score of whether it is a pedestrian. If the score is greater than 0.7, the frame to be detected is used as the candidate frame. When there are multiple candidate frames, the pedestrian candidate frames are sorted according to the score of the classifier, and the pedestrian candidate frame with the highest score is selected as the object of the initial waving detection.

The method of S3 of the present invention is:

Firstly, a window for waving detection is determined according to the currently highest-scoring pedestrian candidate frame, that is, the window for pedestrian detection. The size of the window for waving detection is 36*36, and it is located at the upper left of the pedestrian detection window. The left vertices of the two windows, ie, the waving detection window and the pedestrian detection window, differ by 12 pixels on the x and y axes respectively.

Note that the video image taken by the drone is {P _n (x, y)|n=1, 2,...N}. The video images shot by the drone are arranged in time, and each image in the video image here represents the first frame, the second frame...the Nth frame image.

For each image in the video image, only the pixel values in the hand-waving detection window area are reserved, that is, all pixel values in the area outside the hand-waving detection window are set to 0.

Starting from the nth (n≥2) frame image, the motion detection of the nth (n≥2) frame image is performed using 3-frame difference, and the calculation formula is as follows:

D _n (x,y)=P _n-1 (x,y)-2P _n (x,y)+P _n+1 (x,y) (1)

Among them, P _n (x, y), P _n-1 (x, y), and P _n+1 (x, y) represent the nth frame image, the n-1th frame image, and the n+1th frame image, respectively.

Then D _n (x, y) is binarized using the classic otsu to obtain binary motion information A _n (x, y).

Next, waving detection is performed in the waving detection area through motion history images. The present invention proposes a calculation method of motion history images to better represent the direction information of motion, the method is as follows:

a. Define the motion history image H _n (x, y), the motion history image H _n (x, y) is a global description method about motion, where the value of each pixel is the motion time information of the pixel , the specific definition is as follows:

It can be seen from the above formula that the pixel gray level change information of the motion history image reflects the direction of the behavioral motion, the pixel brightness value of the recent motion is the largest, and the pixel of the long motion is cleared. Therefore, the motion history image well reflects the spatial characteristics and temporal information of the motion.

b. The wrong direction vector may be obtained on the boundary pixel. Therefore, before using formula 3 to calculate, first judge whether each pixel on the motion history image is a boundary pixel. The specific method is: on the motion history image, any The pixel point (x ₀ , y ₀ ) is the center, and there are 9 pixel points in the field with a size of 3×3. When all 9 pixel points are not zero, the pixel point (x ₀ , y ₀ ) is considered not Boundary pixel; otherwise, the pixel point (x ₀ , y ₀ ) is considered to be a boundary pixel;

When calculating the gradient direction, if the pixel is a boundary pixel, its gradient direction is directly set to 0; if the pixel is not a boundary pixel, the sobel operator is used to calculate its gradient direction, that is, the angle matrix θ _n (x,y ),as follows:

in is the convolution operation.

The above method can avoid the erroneous influence brought by the boundary pixels.

c. The range of the calculated angle matrix is 0 to 359. In order to simplify the calculation, a histogram with a size of 36 is used. Each pixel of the angle matrix θ _n (x, y) is traversed in turn, and the histogram is counted. Then the angle corresponding to the maximum value of the histogram is taken as the main motion direction ω _n of the image.

d. Starting from the second frame image in the image {P _n (x, y)|n=1,2,…N} transmitted from the video end, calculate the main direction of motion {ω _n of each image sequentially according to the above method |n=2,...,N}. N generally takes 30. Then calculate a direction index value:

Where the main direction of motion is between 46 degrees and 134 degrees, it is judged to be waving right; if the main direction of motion is between 226 degrees and 314 degrees, it is judged to be waving left.

When the number of non-zero values in the first 30 images reaches 15, the present invention determines that there is a hand-waving movement; otherwise, it is determined that there is no hand-waving movement.

The present invention proposes a hand-waving detection method based on motion history images. Firstly, the area of hand-waving detection is determined indirectly through pedestrian detection, then the main motion direction of the area is calculated through motion history images, and finally whether there is waving motion in sequence images is judged. . The method proposed by the invention has low complexity and high detection accuracy, and is suitable for the target confirmation stage before the drone target tracking.

Description of drawings

Fig. 1 is a schematic flow sheet of the present invention;

Fig. 2 is a schematic diagram of a hand-waving detection method based on motion history images;

Fig. 3 is a schematic diagram of determining the waving detection window according to the pedestrian detection window.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

The area captured by the drone is relatively large, and the waving area of the following object only takes up a small part of the entire picture. Therefore, in order to accurately determine whether there is a waving action, it is first necessary to determine an approximate area. Since the hand region is too small, direct hand detection is not feasible. Considering that the waving action generally occurs at the upper right of the target, the present invention proposes a hand waving detection method based on motion history images.

Firstly, train a pedestrian detector, and shoot different pedestrians through the camera on the drone. A total of no less than 500 pedestrian images are collected as positive samples, and the number of pedestrians participating in the collection is no less than 100. Then collect more than 200 images that do not include pedestrians from the Internet or various other databases as negative samples. Normalize the collected images of no less than 500 pedestrians to an image with a size of 108*36, select the classic histogram of oriented gradient (HOG) feature extraction method to extract the positive and negative samples, and use svm for training , to get a pedestrian detector.

When the UAV performs target tracking, firstly, pedestrian detection is performed on the captured video images. The pedestrian detection process is carried out by sliding window (window size is 108*36), and the HOG features of the image in the frame to be detected are extracted. The detector classifies and obtains a score of whether it is a pedestrian. If the score is greater than 0.7, the frame to be detected is used as a candidate frame. When there are multiple candidate boxes, the pedestrian candidate boxes are sorted according to the score of the classifier. First, select the pedestrian candidate frame with the highest score as the object of the initial waving detection, and perform waving detection on the waving area through the motion history image. If there is a waving behavior detected, the entire waving detection process ends and the follow-up tracking program is started; If the waving behavior is not detected in the pedestrian candidate frame, the waving detection method is performed on the pedestrian candidate frame with the second score. Waving behavior or all pedestrian candidate boxes have been detected for waving but there is no waving behavior.

Next, introduce the specific steps of the hand-waving detection proposed by the present invention:

First, the window for waving detection is determined according to the window for pedestrian detection. As shown in Figure 3, the size of the window for waving detection is 36*36, which is located at the upper left of the pedestrian detection window. The left vertices of the two windows, ie, the waving detection window and the pedestrian detection window, differ by 12 pixels on the x and y axes respectively.

Note that the video image taken by the drone is {P _n (x, y)|n=1, 2,...N}. The video images shot by the drone are arranged in time, and each image in the video image here represents the first frame, the second frame...the Nth frame image.

For each image in the video image, only the pixel values in the hand-waving detection window area are reserved, that is, all pixel values in the area outside the hand-waving detection window are set to 0.

Starting from the nth (n≥2) frame image, the motion detection is performed using 3-frame difference, and the calculation formula is as follows:

D _n (x,y)=P _n-1 (x,y)-2P _n (x,y)+P _n+1 (x,y) (1)

Then D _n (x, y) is binarized using the classic otsu to obtain binary motion information A _n (x, y).

Next, the present invention proposes a calculation method of motion history images to better represent motion direction information. The motion history image H _n (x, y) is a global description method about motion, in which the value of each pixel is the motion time information of the pixel, which is specifically defined as follows:

It can be seen from the above formula that the pixel gray level change information of the motion history image reflects the direction of the behavioral motion, the pixel brightness value of the recent motion is the largest, and the pixel of the long motion is cleared. Therefore, the motion history image well reflects the spatial characteristics and temporal information of the motion.

Use the sobel operator to calculate the gradient direction of each pixel of the motion history image, that is, the angle matrix θ _n (x, y):

in is the convolution operation.

The wrong direction vector may be obtained on the boundary pixel. Therefore, before using formula 3 to calculate, first judge whether each pixel on the motion history image is a boundary pixel. The specific method is: on the motion history image, take any pixel ( x ₀ , y ₀ ) as the center, there are 9 pixels in the field of size 3×3, only when the 9 pixels are not zero, the pixel (x ₀ , y ₀ ) is considered not to be a boundary pixel; otherwise, it is considered This point is the boundary pixel. When calculating the gradient direction, if the pixel is a boundary pixel, its gradient direction is directly set to 0 without calculation; if the pixel is not a boundary pixel, formula 3 is used for calculation. In this way, erroneous effects caused by boundary pixels can be avoided.

The calculated angle matrix ranges from 0 to 359. To simplify the calculation, a histogram with a size of 36 is used. Each pixel of the angle matrix θ _n (x, y) is traversed in turn, and the histogram is counted. Then the angle corresponding to the maximum value of the histogram is taken as the main motion direction ω _n of the image.

Starting from the second frame image in the image {P _n (x,y)|n=1,2,…N} transmitted from the video end, calculate the main direction of motion {ω _n |n of each image sequentially according to the above method =2,...,N}. N generally takes 30. Then calculate a direction index value:

Where the main direction of motion is between 46 degrees and 134 degrees, it is judged to be waving right; if the main direction of motion is between 226 degrees and 314 degrees, it is judged to be waving left.

When the number of non-zero values in the first 30 images reaches 15, the present invention determines that there is a hand-waving movement, and the overall process ends. If the current pedestrian is judged to have no waving movement, then the next pedestrian is waved according to the pedestrian detection score until the waving movement is detected or all pedestrians do not wave.

Claims (5)

1. a kind of detection method of waving based on motion history image is it is characterised in that comprise the following steps：

S1 trains a pedestrian detector；

S2, when unmanned plane carries out target with clapping, carries out pedestrian detection to the video image photographing, high according to grader score The low object as detection of waving for the first time for pedestrian candidate frame pedestrian candidate frame being ranked up, choosing wherein highest scoring；

S3 carries out detection zone determination of waving to present score highest pedestrian candidate frame, then passes through fortune in detection zone of waving Dynamic history image carries out waving to detect；

When waving motion is detected, overall flow terminates S4；If the pedestrian candidate frame of highest scoring is not detected by waving Behavior, then carry out to the pedestrian candidate frame of score second in the grader score height ranking results of pedestrian detection in S2 waving to examine Survey, detection method of waving is identical with the detection method of waving of detection object of waving for the first time；Rule according to this, until detect certain There is behavior of waving in pedestrian candidate frame or all of pedestrian candidate frame has all carried out waving to detect behavior of still all not waving Till.

2. the detection method of waving based on motion history image according to claim 1 is it is characterised in that in S1, pedestrian The training method of detector is：By the camera on unmanned plane, different pedestrians are shot, collection altogether is no less than 500 width Pedestrian image as positive model, the pedestrian's number participating in collection is no less than 100 people；

Then from network or other various databases collect the image of more than various 200 width not including pedestrian as negative sample This；

The pedestrian image collecting is normalized to the image that size is 108*36, choice direction histogram of gradients feature extraction side Method aligns negative sample and carries out feature extraction, is trained using svm, obtains a pedestrian detector.

3. the detection method of waving based on motion history image according to claim 1 and 2 is it is characterised in that in S2, When unmanned plane carries out target with clapping, first pedestrian detection is carried out to the video image shooting, pedestrian detection process adopts sliding window Mouthful mode carry out, window size is 108*36, extracts the HOG feature of image in frame to be checked, is carried out point by pedestrian detector Class, obtain one be whether pedestrian fraction, if this score is more than 0.7, with this frame to be detected as candidate frame；Multiple when existing During candidate frame, according to grader score height, pedestrian candidate frame is ranked up, chooses the pedestrian candidate frame of wherein highest scoring The object detecting as waving for the first time.

4. the detection method of waving based on motion history image according to claim 3 is it is characterised in that the method for S3 For：

First a detection window of waving is determined as the window of pedestrian detection according to present score highest pedestrian candidate frame, Detection window size of waving is 36*36, positioned at the upper left side of pedestrian detection window；Wave detection window and pedestrian detection window Left summit differs 12 pixels respectively in x, y-axis；

The video image that note unmanned plane shoots is { P_n(x, y) | n=1,2 ... N }, to each width image in video image, all only protect Stay the pixel value waved in detection window region, even all pixels value of detection window exterior domain of waving all is set to 0；

From the beginning of n-th (n >=2) two field picture, carry out the motion detection of n-th (n >=2) two field picture using 3 frame differences, computing formula is such as Under：

D_n(x, y)=P_n-1(x,y)-2P_n(x,y)+P_n+1(x,y) (1)

Wherein P_n(x,y)、P_n-1(x,y)、P_n+1(x, y) represents n-th frame image, the (n-1)th two field picture, the (n+1)th two field picture respectively；

Then using classical big law by D_n(x, y) binaryzation, obtains two-value movable information A_n(x,y)；

Next carry out waving to detect, method is as follows by motion history image in detection zone of waving：

A. define motion history image H_nThe value of (x, y), wherein each pixel is the move time information of this pixel, fixed Justice is as follows：

$H_n(x,y)=\left\{\begin{array}{cc}255,& \begin{array}{cc}if& A_n(x,y)=1\end{array}\\ max(0,H_{n-1}(x,y)-100),& else\end{array}\right.---\left(2\right)$

B. on motion history image, with any pixel point (x₀,y₀) centered on, in the field for 3 × 3 for the size, one has 9 pictures Vegetarian refreshments, when 9 pixels are all not zero, just thinks pixel (x₀,y₀) it is not boundary pixel；Otherwise it is assumed that this pixel (x₀,y₀) it is boundary pixel；

When calculating gradient direction, if pixel is boundary pixel, directly its gradient direction is set to 0；If pixel is not Boundary pixel, then calculate its gradient direction using sobel operator, namely angle matrix θ_n(x, y), as follows：

${\mathrm{\θ}}_n(x,y)=arctan\left(\frac{S_y(x,y)}{S_x(x,y)}\right)---\left(3\right)$

Wherein It is convolution fortune Calculate；

C. the scope of calculated angle matrix is 0 to 359, the histogram being 36 using size, successively to angle matrix θ_n (x, y) each pixel is traveled through, statistic histogram.Then using the angle corresponding to histogrammic maximum as this image Motion principal direction ω_n,

D. it is transmitted through image { the P coming from video end_n(x, y) | n=1,2 ... N in the second two field picture start, according to the method described above Calculate the motion principal direction { ω of each image successively_n| n=2 ..., N }, then calculate a direction index value：

$f\left({\mathrm{\&omega;}}_n\right)=\left\{\begin{array}{cc}1,& \begin{array}{cc}if& 46<{\mathrm{\&omega;}}_n<134\end{array}\\ -1,& \begin{array}{cc}if& 226<{\mathrm{\&omega;}}_n<314\end{array}\\ 0,& otherwise\end{array}\right.---\left(4\right)$

Wherein motion principal direction, between 46 degree and 134 degree, is judged as waving in the right side；Motion principal direction 226 degree and 314 degree it Between, it is judged as waving in a left side；

When non-zero value reaches N/2 in current N width image, then judge there is waving motion, otherwise, it is determined that being not wave to transport Dynamic.

5. the detection method of waving based on motion history image according to claim 4 is it is characterised in that in step S3, N Value is more than or equal to 30.