CN109977834A - The method and apparatus divided manpower from depth image and interact object - Google Patents

Abstract

The application proposes a kind of method and apparatus divided manpower from depth image and interact object, wherein method includes: to construct the manpower partitioned data set based on depth image using the dividing method based on color image；Using the manpower partitioned data set based on depth image, training obtains parted pattern, and parted pattern is made of encoder, attention TRANSFER MODEL and decoder；Depth image to be processed is split using parted pattern, obtains tag along sort figure corresponding with depth image to be processed, the value of each pixel is the types value of each pixel in tag along sort figure.The parted pattern that this method is obtained using the manpower partitioned data set based on depth image by training, depth image to be processed is split using parted pattern, realize the manpower and object segmentation of pixel scale, environmental robustness is improved, segmentation precision is higher, the manpower that is capable of handling under complex interaction situation and the case where object segmentation.

Description

Method and apparatus for segmenting human hands and interactive objects from depth images

technical field

The present application relates to the technical field of computer vision, and in particular, to a method and apparatus for segmenting human hands and interactive objects from depth images.

Background technique

Human hand segmentation is a fundamental problem in many research fields such as gesture recognition, hand tracking, and hand reconstruction. Compared with individual hand movements, the research on the interaction with the object is more important in the fields of human-computer interaction and virtual reality.

In recent years, general neural network-based semantic segmentation models have become more and more perfect, but the existing method models have low environmental robustness, poor segmentation accuracy, and cannot handle human hand segmentation in complex interactive situations.

SUMMARY OF THE INVENTION

This application proposes a method and device for segmenting a human hand and an interactive object from a depth image, which is used to solve the problem that the existing human hand segmentation model in the related art has low environmental robustness, poor segmentation accuracy, and cannot handle human hand segmentation in complex interactive situations. The problem.

An embodiment of the present application provides a method for segmenting a human hand and an interactive object from a depth image, including:

Using the segmentation method based on color image, construct a dataset of human hand segmentation based on depth image;

Using the depth image-based human hand segmentation data set, a segmentation model is obtained by training, and the segmentation model is composed of an encoder, an attention transfer model and a decoder;

Use the segmentation model to segment the depth image to be processed, and obtain a classification label map corresponding to the depth image to be processed, where the value of each pixel in the classification label map is the type value of each pixel , and the type value is used to represent the type to which the pixel belongs in the depth image to be processed.

In the method for segmenting a human hand and an interactive object from a depth image according to the embodiment of the present application, a segmentation method based on a color image is used to construct a human hand segmentation data set based on a depth image, and a segmentation model is trained by using the human hand segmentation data set based on the depth image, The segmentation model consists of an encoder, an attention transfer model and a decoder. The segmentation model is used to segment the depth image to be processed, and a classification label map corresponding to the depth image to be processed is obtained. The value of each pixel in the classification label map is The type value of each pixel point, according to the type value of each pixel point, the type to which each pixel point belongs can be determined. Therefore, the segmentation model obtained by training the human hand segmentation data set based on the depth image is used, and the segmentation model is used for processing. The depth image is segmented, which realizes the segmentation of human hands and objects at the pixel level, improves the environmental robustness, has high segmentation accuracy, and can handle the segmentation of human hands and objects in complex interactive situations.

Another embodiment of the present application provides an apparatus for segmenting a human hand and an interactive object from a depth image, including:

A building block for building a depth image-based hand segmentation dataset using a color image-based segmentation method;

a training module for using the depth image-based human hand segmentation data set to obtain a segmentation model through training, and the segmentation model is composed of an encoder, an attention transfer model and a decoder;

The identification module is used to segment the depth image to be processed by using the segmentation model, and obtain a classification label map corresponding to the depth image to be processed, and the value of each pixel in the classification label map is the value of each Type value of the pixel point, where the type value is used to represent the type of the pixel point in the depth image to be processed.

The device for segmenting a human hand and an interactive object from a depth image according to the embodiment of the present application constructs a depth image-based human hand segmentation dataset by using a color image-based segmentation method, and uses the depth image-based human hand segmentation dataset to train a segmentation model, The segmentation model consists of an encoder, an attention transfer model and a decoder. The segmentation model is used to segment the depth image to be processed, and a classification label map corresponding to the depth image to be processed is obtained. The value of each pixel in the classification label map is The type value of each pixel point, according to the type value of each pixel point, the type to which each pixel point belongs can be determined. Therefore, the segmentation model obtained by training the human hand segmentation data set based on the depth image is used, and the segmentation model is used for processing. The depth image is segmented, which realizes the segmentation of human hands and objects at the pixel level, improves the environmental robustness, has high segmentation accuracy, and can handle the segmentation of human hands and objects in complex interactive situations.

Additional aspects and advantages of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a method for segmenting a human hand and an interactive object from a depth image according to an embodiment of the present application;

2 is a schematic structural diagram of a segmentation model provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of an attention mechanism model provided by an embodiment of the present application;

4 is a schematic flowchart of another method for segmenting a human hand and an interactive object from a depth image according to an embodiment of the present application;

5 is a schematic diagram of a training process of a segmentation model provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of the effect of using contour error according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an apparatus for segmenting a human hand and an interactive object from a depth image according to an embodiment of the present application.

Detailed ways

The following describes in detail the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to be used to explain the present application, but should not be construed as a limitation to the present application.

The following describes the method and apparatus for segmenting a human hand and an interactive object from a depth image according to the embodiments of the present application with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of a method for segmenting a human hand and an interactive object from a depth image according to an embodiment of the present application.

As shown in Figure 1, the method for segmenting a human hand and an interactive object from a depth image includes:

Step 101 , using a color image-based segmentation method to construct a depth image-based human hand segmentation dataset.

Since the depth camera can collect color and depth images at the same time, the depth camera can be used to collect color images and depth images of the interaction between human hands and objects, so as to obtain multiple pairs of color images and depth images. Then, the depth image is processed based on the color image, and then the human hand segmentation dataset based on the depth image is obtained.

In order to improve the segmentation accuracy, in this embodiment, a fixed light source with the same brightness and color temperature may be used to collect objects with colors that are quite different from the skin of a human hand. For example, capture an image of a hand holding a blue pen with the same brightness and light source.

Step 102 , using the depth image-based human hand segmentation dataset to train to obtain a segmentation model.

After obtaining the human hand segmentation dataset based on depth images, the initial neural network model is trained with the dataset, and a segmentation model that meets the requirements is obtained.

Among them, in the training process, the loss function can be used to measure the prediction performance of the segmentation model.

In this embodiment, the segmentation model consists of an encoder, an attention transfer model and a decoder. Among them, the encoder uses a large convolutional network, and the decoder uses a deconvolution layer to restore high-level information to the image pixel scale.

FIG. 2 is a schematic structural diagram of a segmentation model provided by an embodiment of the present application. As shown in Figure 2, the segmentation model consists of an encoder, an attention transfer model, and a decoder. In this embodiment, an attention mechanism is added between the encoder and the decoder, and an attention feature map is constructed by fusing multi-scale image features, which is used to strengthen the same-layer connection between the encoder and decoder, which can improve the information transfer between the two. accuracy and validity.

FIG. 3 is a schematic structural diagram of an attention mechanism model provided by an embodiment of the present application. In Figure 3, the first layer, the second layer, ..., the i-1 layer feature maps are multiplied to obtain the bottom layer attention (FineAtt); the first layer, the second layer, ..., the i-1 layer each The layers include a scale scaling network (SqueezeNet, referred to as SN) and a bilinear down-sampling layer (Bilinear down-sampling, referred to as DS), where SN can normalize the dimension of the feature map. Multiply the feature maps of the i+1th layer, the i+2th layer, ..., the nth layer to obtain the high-level attention (CoarseAtt); among them, the i+1th layer, the i+2th layer, ..., the nth layer Each layer includes SN and an up-sampling layer (US for short). DS and US are used to downscale and upscale feature maps, respectively. The obtained FineAtt and CoarseAtt attention maps are concatenated with the feature map of the i-th layer and input to the decoder. The feature map scale of each layer from the 1st layer to the nth layer in Figure 3 is enhanced by this attention mechanism.

Step 103: Use the segmentation model to segment the depth image to be processed, and obtain a classification label map corresponding to the depth image to be processed.

In this embodiment, before the processed depth image is identified, the depth image to be processed may be acquired by a depth camera.

After the segmentation model is acquired, the depth image to be processed is input into the segmentation model obtained by training, and the segmentation model outputs a classification label map corresponding to the depth image to be processed. The size of the classification label map is the same as that of the depth image to be processed, and the value of each pixel in the classification label map is the type value of each pixel. The type value is used to characterize the type of the pixel in the depth image to be processed. In addition, the pixel coordinate value is implicit in the image pixel arrangement, and the value of each pixel in the input depth image is the depth value.

The type to which the pixel points belong in the depth image to be processed may include human hand, object, and background. During specific implementation, three types of human hand, object, and background can be represented by different type values. For example, use 0 for the background, 1 for the human hand, and 2 for the object.

In this embodiment, according to the type value of each pixel point and the type corresponding to the type value, the segmentation result of the human hand and the object in the depth image to be processed can be obtained, so as to realize the segmentation of the human hand and the interactive object.

As shown in Figure 2, the depth image to be processed is input into the deep network model, first through the encoder, then through the attention transfer model, and finally through the decoder to output the classification label map of the depth image to be processed. The type value of the pixel point, get the position of the human hand and the object, and realize the segmentation of the human hand and the object.

In the embodiment of the present application, according to the type value of each pixel in the depth image to be processed output by the segmentation model and the corresponding type of the type value, the pixels belonging to the human hand and the pixels belonging to the object can be determined, thereby realizing the The human hand and objects that interact in the processing image are separated, and the pixel-level human hand and object segmentation is realized. The segmentation accuracy is high, and it can segment the interacting human hands and objects in complex situations.

In one embodiment of the present application, a training dataset for human hand segmentation based on depth images can be constructed from color images. The following describes in detail with reference to FIG. 4 , which is a schematic flowchart of another method for segmenting a human hand and an interactive object from a depth image according to an embodiment of the present application.

As shown in Figure 4, the method for constructing a depth image-based hand segmentation dataset includes:

Step 301: Acquire multiple pairs of color images and depth images in a scenario of interaction between a human hand and an object.

In this embodiment, some objects that are quite different in color from the skin color of a human hand can be collected manually. Then, a depth camera is used to capture images of the human hand interacting with each object, thereby obtaining multiple pairs of color images and depth images. In addition, in order to increase the amount of data, for the same object, images of different interaction poses between the human hand and the object can be collected.

When capturing images with a depth camera, fix the lighting environment, such as using a fixed light source with the same brightness and color temperature, to ensure that the captured color images are clear and without shadows.

Step 302 , perform object segmentation based on the HSV color space on all color images, and obtain the type value of each pixel in each color image.

In this embodiment, the backgrounds in all color images and depth images can be removed first through a depth threshold, and images of human hands and objects are retained. Then, according to the existing RGB color space to HSV color space conversion formula, all acquired color images are converted to HSV color space. Among them, the parameters of the HSV color space are: hue (H), saturation (S), and lightness (V).

After that, segment the HSV color space corresponding to each color image to obtain the type value of each pixel in each color image. Specifically, the distribution of pixels of multiple pure-hand samples and interactive samples in HSV space is analyzed, and the overlapping area between samples is the area corresponding to the pixels of human hands, and multiple linear constraints are fitted. All color images were analyzed, and the pixels inside the constraints were marked as human hands, and the pixels outside the constraints were marked as objects.

Step 303: For each pair of color images and depth images, map each pixel in the color image to the corresponding pixel in the depth image to construct a depth image-based human hand segmentation training dataset.

For each pair of color image and depth image, pixel-align the color image and depth image, that is, estimate the camera internal and external parameters of the depth and color sensors respectively, and use an automatic labeling method to affine the depth point cloud to the color camera space. Generates a true classification label image based on a color image, which is also the true classification label map of the depth image corresponding to the color image. Among them, the type value of each pixel in the real classification label image can be 0 to represent the background, 1 to represent the hand, and 2 to represent the object.

In this embodiment, all depth images and their true classification label maps constitute a training dataset for human hand segmentation based on depth images.

Further, in order to improve the segmentation accuracy, in an embodiment of the present application, before performing the mapping, the depth image may be preprocessed, the morphological and contour filtering methods may be used for denoising, and the background in the depth image may be analyzed, Only the human hand and objects that interact with the human hand are kept.

After obtaining the data set for training the segmentation model, when training the model, the training data set for human hand segmentation based on depth images can be divided into training data set and test data set, wherein the number of depth images in the training data set is much larger than The number of depth images in the test dataset, the training dataset is used for training, and the test dataset is used to test the trained model.

Then, using the training data set, the initial segmentation model is trained and the first loss function is calculated. Among them, the first loss function adopts the softmax cross entropy loss function, as shown in the following formula (1):

Among them, _yi represents the real result, _xi represents the predicted value output by the segmentation model, the subscript i represents different types, and the subscript j also represents different types. For example, there are three types of pixel points, first calculate the loss of type value i=0, the loss is Compute the loss for type value i=1: Compute the loss for type value i=2: Then the loss of the model is

It should be noted that the first loss function may also be other loss functions capable of realizing the segmentation task.

Specifically, the depth images in the training dataset are input into the initial neural network model, and the network model outputs the predicted classification label map of the depth images. Then, according to the gap between the predicted classification label map and the real label map of the depth image, the gradient descent algorithm is used to feed back all the parameters in the network, and the network parameters are updated accordingly. The next time the depth image is input, the predicted class label map output by the network will be closer to the true class label map.

When the value of the first loss function no longer decreases after training, that is to say, when the performance of the model reaches the optimum by using the first loss function, the training is continued using the contour error as the loss function. Among them, the contour error is shown in the following formula (2):

Among them, B is the fuzzy operation, for example, Gaussian blur with σ=2.121 kernel of 5×5 can be used for Gaussian blurring; S is the contour extraction, for example, the Sober operator is used for contour extraction; M _labels is the real classification label map, and M _logits is The output of the network, specifically the predicted value of the pixel's type.

When the value of the contour error is stable and no longer decreases, the training can be stopped to obtain the segmentation model. Then, use the test set to test the segmentation model, specifically, the depth images in the test set can be input into the segmentation model for identification, and the Intersection-over-Union (IOU) scores of all the depth images in the test set are counted, Use the IOU score to judge whether the segmentation model meets the requirements.

Among them, IOU refers to the ratio of intersection and union. In this embodiment, it refers to the ratio of the intersection and union of the model prediction result and the real result, that is, the intersection of the model prediction result and the real result, and the model prediction result and The ratio of the union of the true results.

FIG. 5 is a schematic diagram of a training process of a segmentation model provided by an embodiment of the present application. The left side of Figure 5 is a schematic diagram of the data construction process, and the right side is a schematic diagram of the model training process. During data construction, the color images acquired with the depth camera are aligned with the depth images, and an automated labeling method is used to generate a map of true classification labels based on the color images, which are also the true classification labels of the aligned corresponding depth images. All the depth images and their ground-truth classified label images constitute the training dataset for human hand segmentation based on depth images.

During model training, the depth images in the dataset are input into the attention segmentation network, and the classification label map predicted by the network model is obtained, which is compared with the real classification label map and the loss is calculated, and the network parameters are iteratively updated step by step.

FIG. 6 is a schematic diagram of the effect of using a contour error according to an embodiment of the present application. In Figure 6, the objects and hands in the left column are real labels, the middle column is the network output without contour error, and the right column is the network output after contour error is used.

In the embodiment of the present application, when training the segmentation model, by first using a general loss function, when the value of the general loss function is stable, that is, the model is optimal under the loss function, the contour error is used as the loss function for training, and in The attention mechanism model is added to the segmentation model, thereby greatly improving the segmentation accuracy of the model.

Further, in order to enhance the generalization ability of the segmentation model, before using the training data set to train the segmentation model, a data augmentation operation may be performed on the training data set, and the depth images obtained by the data augmentation operation are added to the training data set.

The data augmentation operation includes at least one of freely rotating the depth image, adding random noise, and randomly flipping the depth image.

In order to realize the above embodiments, the embodiments of the present application further provide an apparatus for segmenting a human hand and an interactive object from a depth image. FIG. 7 is a schematic structural diagram of an apparatus for segmenting a human hand and an interactive object from a depth image according to an embodiment of the present application.

As shown in FIG. 7 , the apparatus for segmenting a human hand and an interactive object from a depth image includes: a building module 610 , a training module 620 , and a recognition module 630 .

A building module 610 is used to construct a depth image-based hand segmentation data set by using a color image-based segmentation method;

A training module 620, configured to use the depth image-based human hand segmentation data set to obtain a segmentation model by training, and the segmentation model is composed of an encoder, an attention transfer model and a decoder;

The identification module 630 is configured to use the segmentation model to segment the depth image to be processed, and obtain a classification label map corresponding to the depth image to be processed, where the value of each pixel in the classification label map is the value of each pixel. Type value of a pixel point, where the type value is used to characterize the type to which the pixel point belongs in the depth image to be processed.

In a possible implementation manner of the embodiment of the present application, the above-mentioned building module 610 is specifically used for:

Collect multiple pairs of color images and depth images in the interaction between human hands and objects;

Perform object segmentation based on HSV color space for all color images, and obtain the type value of each pixel in each color image;

For each pair of color image and depth image, map each pixel in the color image to the corresponding pixel in the depth image to construct a depth image-based human hand segmentation training dataset.

In a possible implementation manner of the embodiment of the present application, the depth image is preprocessed, including noise and background removal.

In a possible implementation of the embodiment of the present application, the depth image-based human hand segmentation dataset includes a training dataset and a test dataset, and the training module 620 is specifically used for:

Using the training data set, the initial neural network model is trained, and the first loss function is calculated, wherein the first loss function adopts the softmax cross entropy loss function;

When the value of the first loss function no longer decreases, continue training using the contour error as the loss function.

In a possible implementation manner of the embodiment of the present application, the device further includes:

The processing module is configured to perform a data augmentation operation on the training data set, wherein the data augmentation operation includes at least one of freely rotating the depth image, adding random noise, and randomly flipping the depth image.

It should be noted that the above description of the method embodiment for segmenting a human hand and an interactive object from a depth image is also applicable to the device for segmenting a human hand and an interactive object from a depth image in this embodiment, so it will not be repeated here.

The device for segmenting a human hand and an interactive object from a depth image according to the embodiment of the present application constructs a depth image-based human hand segmentation dataset by using a color image-based segmentation method, and uses the depth image-based human hand segmentation dataset to train a segmentation model, The segmentation model consists of an encoder, an attention transfer model and a decoder. The segmentation model is used to segment the depth image to be processed, and a classification label map corresponding to the depth image to be processed is obtained. The value of each pixel in the classification label map is The type value of each pixel point, according to the type value of each pixel point, the type to which each pixel point belongs can be determined. Therefore, the segmentation model obtained by training the human hand segmentation data set based on the depth image is used, and the segmentation model is used for processing. The depth image is segmented, which realizes the segmentation of human hands and objects at the pixel level, improves the environmental robustness, has high segmentation accuracy, and can handle the segmentation of human hands and objects in complex interactive situations.

Claims (10)

1. a method for segmenting human hands and interactive objects from a depth image, comprising: Using the segmentation method based on color image, construct a dataset of human hand segmentation based on depth image; Using the depth image-based human hand segmentation data set, a segmentation model is obtained by training, and the segmentation model is composed of an encoder, an attention transfer model and a decoder; Use the segmentation model to segment the depth image to be processed, and obtain a classification label map corresponding to the depth image to be processed, where the value of each pixel in the classification label map is the type value of each pixel , and the type value is used to represent the type to which the pixel belongs in the depth image to be processed. 2. method as claimed in claim 1 is characterized in that, described utilizing the segmentation method based on color image, constructs the hand segmentation data set based on depth image, comprises: Obtain multiple pairs of color images and depth images in the interaction scenario between human hands and objects; Perform object segmentation based on HSV color space for all color images, and obtain the type value of each pixel in each color image; For each pair of color images and depth images, each pixel in the color image is mapped to a corresponding pixel in the depth image, and a training dataset for human hand segmentation based on the depth image is constructed. 3. The method of claim 2, wherein after mapping each pixel in the color image to a corresponding pixel in the depth image, the method further comprises: The depth image is preprocessed, including noise and background removal. 4. The method of claim 2, wherein the depth image-based human hand segmentation dataset comprises a training dataset and a test dataset, and the depth image-based human hand segmentation dataset is used to train segmentation models, including: Using the training data set, the initial neural network model is trained, and the first loss function is calculated, wherein the first loss function adopts the softmax cross entropy loss function; When the value of the first loss function no longer decreases, continue training using the contour error as the loss function. 5. The method according to claim 4, wherein, before the training of the segmentation model using the training data set, the method further comprises: A data augmentation operation is performed on the training data set, and the data augmentation operation includes at least one of freely rotating the depth image, adding random noise, and randomly flipping the depth image. 6. A device for segmenting a human hand and an interactive object from a depth image, comprising: A building block for building a depth image-based hand segmentation dataset using a color image-based segmentation method; a training module for using the depth image-based human hand segmentation data set to obtain a segmentation model through training, and the segmentation model is composed of an encoder, an attention transfer model and a decoder; The identification module is used to segment the depth image to be processed by using the segmentation model, and obtain a classification label map corresponding to the depth image to be processed, and the value of each pixel in the classification label map is the value of each Type value of the pixel point, where the type value is used to represent the type of the pixel point in the depth image to be processed. 7. The apparatus of claim 6, wherein the building block is specifically used for: Obtain multiple pairs of color images and depth images in the interaction scenario between human hands and objects; Perform object segmentation based on HSV color space for all color images, and obtain the type value of each pixel in each color image; For each pair of color image and depth image, each pixel in the color image is mapped to the corresponding pixel in the depth image, and a training dataset for human hand segmentation based on the depth image is constructed. 8. The apparatus of claim 7, further comprising: A preprocessing module for preprocessing the depth image, including noise and background removal. 9. The apparatus of claim 7, wherein the depth image-based hand segmentation data set comprises a training data set and a test data set, and the training module is specifically used for: Using the training data set, the initial neural network model is trained, and the first loss function is calculated, wherein the first loss function adopts the softmax cross entropy loss function; When the value of the first loss function no longer decreases, continue training using the contour error as the loss function. 10. The apparatus of claim 9, further comprising: The processing module is configured to perform a data augmentation operation on the training data set, wherein the data augmentation operation includes at least one of freely rotating the depth image, adding random noise, and randomly flipping the depth image.