CN106228111A - A kind of method based on skeleton sequential extraction procedures key frame - Google Patents

Abstract

The invention relates to a method for extracting key frames based on a skeleton sequence, comprising: capturing human actions through a Kinect camera to obtain a three-dimensional skeleton sequence containing a plurality of skeleton nodes; subtracting the skeleton coordinates of adjacent frames to obtain the three-dimensional Skeleton motion vector: project the three-dimensional bone motion vectors of all bones on the three planes of the Cartesian orthogonal system, and perform probability statistics on the bone motion vectors according to the direction and amplitude on each projection plane to obtain a histogram ;According to the information entropy formula, calculate the information entropy value for the bone motion vector histogram of the adjacent frame, and define the frame with the local maximum value of the information entropy as the original frame; for each original frame of the whole three-dimensional skeleton sequence, calculate the interleaving coefficient , and weight the information entropy value of the original frame; get the key frame of the skeleton sequence human action. The invention can accurately, reliably and efficiently extract key frames of human actions.

Description

A Method of Extracting Key Frames Based on Skeleton Sequence

technical field

The invention belongs to the field of multimedia information processing and relates to a method for extracting key frames.

Background technique

With the advent of the Internet age and the rapid development of the computer industry, the market for computer intelligence is also booming. Machine learning, pattern recognition, data mining and other fields have broad development space in the current society. The human action detection and recognition of computers belonging to the field of pattern recognition has many applications in the world today, such as somatosensory games for human-computer interaction, intelligent monitoring, video retrieval, and so on. However, from the perspective of computer processing video, the amount of video information is often too large. In order to improve the speed of video processing and make video-based machine learning algorithms more applicable, it has become very popular in recent years to filter out key frames that contain more action information in videos for processing. The invention proposes a method for extracting action key frames in a video sequence based on a three-dimensional skeleton sequence for human actions.

In recent years, the camera industry has developed rapidly, and cameras capable of capturing depth information have become more and more widely used. After Microsoft released the Kinect camera in 2010, the depth camera has entered thousands of households, and a large number of scholars in the field of video and pictures in the world have gradually shifted their research direction to information processing based on RGB-D. With the continuous improvement of human skeleton tracking algorithms in depth video sequences, skeleton information, as a more abstract and high-level human body feature, has a wide range of applications because of its light insensitivity and more comprehensive three-dimensional characteristics. However, there is currently no keyframe extraction technique based on skeletal sequences.

Contents of the invention

In order to process video sequences more conveniently and allow computers to quickly and effectively recognize human actions, the present invention proposes a method for extracting key frames of human actions based on skeleton sequences. This method has more robust spatial characteristics than two-dimensional information. At the same time, thanks to the simplicity of bone information, it has high operating efficiency. The content of the invention is as follows:

A method for extracting key frames based on a skeleton sequence, comprising the following steps:

1) Capture human actions through the Kinect camera, perform skeleton tracking in the captured data stream, and obtain a 3D skeleton sequence containing multiple skeleton nodes;

2) For each skeletal node, subtract the skeletal coordinates of adjacent frames to obtain the skeletal motion vectors of each skeletal node between adjacent frames, and then calculate the three-dimensional skeletal motion vectors of all bones;

3) Project the three-dimensional bone motion vectors of all bones on the three planes of the Cartesian orthogonal system. On each projection plane, carry out probability statistics on the bone motion vectors according to the direction and amplitude, and obtain a histogram. It is defined as a histogram of bone motion vectors;

4) According to the information entropy formula, the information entropy value is calculated for the bone motion vector histogram of adjacent frames; in the entire video sequence, all the bone motion vector histogram information entropy values are arranged in sequence according to the video sequence, and drawn as Curve diagram, this curve is defined as an entropy curve, in the entropy curve, the local maximum value is obtained, and the frame with the information entropy having a local maximum value is defined as the original frame;

5) For each original frame i of the entire 3D skeleton sequence, according to its own information entropy value and the information entropy value of adjacent frames, the interleaving coefficient HI can be calculated by the interleaving method of the following formula:

$\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&PlusMinus;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&PlusMinus;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}$

Wherein, the information entropy value of the original frame is H(i), and let H(i±x) represent the information entropy value of x frames adjacent to the original frame i, the + sign represents the subsequent frame, and the - sign represents the previous frame; Let it be multiplied by the information entropy value of the original frame, so as to weight the information entropy value of the original frame;

6) The information entropy value H(i) of each original frame is multiplied by its interleaving coefficient HI for weighting, and the weighted original frame information entropy value is obtained;

7) According to the weighted original frame information entropy value, draw a new information entropy value curve, and find the frame corresponding to the local maximum value as the key frame of the skeleton sequence human action.

The invention can accurately, reliably and efficiently extract key frames of human actions.

Description of drawings

Figure 1 is the whole key frame extraction framework

Figure 2 is the key frame extracted by the present invention for the waving action on the MSRAction-3D data set, which is visualized by grayscale image

detailed description

1) The present invention adopts the 32-bit operating system of Windows8, the development IDE is VS2010, Kinect for Windows SDK v1.6 and OpenCV2.3.0 or higher versions are configured, and the NUI bone tracking method is used to carry out bone tracking in the data stream captured by Kinect , and output the sequence of human skeleton actions.

2) Each frame of the skeleton sequence contains the three-dimensional coordinates of 20 human skeleton nodes. For each skeletal node, the absolute difference between the three-dimensional coordinates of the bones in every two adjacent frames is the bone motion vector of the bone node between the two adjacent frames, and then the three-dimensional bone motion vectors of all 20 bone nodes can be obtained .

3) The bone motion vectors of all bone nodes between two frames are projected in three directions in the Cartesian orthogonal system, and the projections of the motion vectors of each bone node on a two-dimensional plane have different directions and vector sizes. On each two-dimensional plane, with the positive direction of the x-axis as the reference, every 45° rotation in the counterclockwise direction is defined as a direction, so far the plane can be divided into 8 directions. According to the experimental results, the present invention uses the maximum amplitude value of all bone motion vectors in each video sequence as a standard, and divides all bone motion vectors on two-dimensional planes into five size ranges. Thus, according to the size and direction of the skeletal motion vector, 40 categories are defined in turn (the order and number of the categories do not affect the result), and the skeletal motion vector of each two-dimensional projection surface can be classified into a category according to the direction and size .

On each projection surface, the number of bones contained in each category is counted, and a vector with a dimension of 40 (that is, a histogram) can be obtained. The vectors obtained by statistics on the three projection surfaces are connected to obtain a dimension of 120. Vector (i.e. histogram), which is defined as a histogram of bone motion vectors.

4) For each bone motion vector histogram, according to the information entropy formula: The entropy value corresponding to each bone motion vector can be calculated. Wherein H is the information entropy value, p _i is the proportion of the i-th category in the entire histogram in the 120-dimensional vector, n is the length of the histogram, and n=120 is taken in the present invention.

For the entire bone sequence, all information entropies are connected to obtain a curve composed of information entropy, which is defined as an entropy curve. Extract the local maximum value in the entropy curve, that is, the point in the entropy curve whose entropy value is greater than the entropy values of the left and right frames at the same time, and define the frame with the local maximum value in the bone sequence entropy curve as the original frame.

5) In the entropy curve, for each original frame, assuming that the frame is the i-th frame in the entire video, its entropy value is H(i), and H(i±x) represents the adjacent frame Entropy value for x frames (before or after). Calculate the interleaving coefficient HI of the frame according to the following interleaving formula. The interleaving coefficient reflects the difference between the skeletal motion of the original frame and the motion of its adjacent frames. The entropy value of the original frame is multiplied by the interleaving coefficient, so as to realize the weighting of the entropy value of the original frame. The formula for the interleaving coefficient is as follows:

$\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&PlusMinus;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&PlusMinus;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}$

6) The weighted original frame entropy values are sequentially connected according to the video sequence to obtain a new entropy curve. Take the frame corresponding to the local maximum value on the new entropy curve as the key frame of the human action sequence in the video.

The result that the present invention is tested on the MSRAction-3D data set is described below:

MSRAction-3D is an influential human action detection and recognition dataset, which contains 20 types of actions, providing depth information data and bone data. According to the key frame extraction method described above, the present invention extracts the key frames of the waving action. The hand waving action contains a total of 58 frames. Through the method of the present invention, a total of 8 key frames are extracted. Figure 2 is the result of visualizing the depth information corresponding to the key frames and then arranging them sequentially. From the results, we can see that the method of the present invention extracts a few frames that can represent the entire action from an action sequence that originally included multiple frames. Through the present invention, the processing of the entire video sequence can be converted into the processing of key frame sequences, thereby greatly reducing the redundancy of processing data, reducing the running time and space cost of algorithms, and improving the practicability of complex algorithms in video processing.

Claims (1)

1. A method for extracting key frames based on skeletal sequences, comprising the following steps: 1) Capture human actions through the Kinect camera, perform skeleton tracking in the captured data stream, and obtain a 3D skeleton sequence containing multiple skeleton nodes; 2) For each skeletal node, subtract the skeletal coordinates of adjacent frames to obtain the skeletal motion vectors of each skeletal node between adjacent frames, and then calculate the three-dimensional skeletal motion vectors of all bones; 3) Project the three-dimensional bone motion vectors of all bones on the three planes of the Cartesian orthogonal system. On each projection plane, carry out probability statistics on the bone motion vectors according to the direction and amplitude, and obtain a histogram. It is defined as a histogram of bone motion vectors; 4) According to the information entropy formula, the information entropy value is calculated for the bone motion vector histogram of adjacent frames; in the entire video sequence, all the bone motion vector histogram information entropy values are arranged in sequence according to the video sequence, and drawn as Curve diagram, this curve is defined as an entropy curve, in the entropy curve, the local maximum value is obtained, and the frame with the information entropy having a local maximum value is defined as the original frame; 5) For each original frame i of the entire 3D skeleton sequence, according to its own information entropy value and the information entropy value of adjacent frames, the interleaving coefficient HI can be calculated by the interleaving method of the following formula: $\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&PlusMinus;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&PlusMinus;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}$ Wherein, the information entropy value of the original frame is H(i), and let H(i±x) represent the information entropy value of x frames adjacent to the original frame i, the + sign represents the subsequent frame, and the - sign represents the previous frame; Let it be multiplied by the information entropy value of the original frame, so as to weight the information entropy value of the original frame; 6) The information entropy value H(i) of each original frame is multiplied by its interleaving coefficient HI for weighting, and the weighted original frame information entropy value is obtained; 7) According to the weighted original frame information entropy value, draw a new information entropy value curve, and find the frame corresponding to the local maximum value as the key frame of the skeleton sequence human action.