CN107300976A - A kind of gesture identification household audio and video system and its operation method - Google Patents

Abstract

The invention discloses a home theater system for gesture recognition, which includes a sound pickup module, a control module, audio-visual equipment, and a camera. The control module is respectively connected to the sound pickup module, audio-visual equipment, and camera; the camera is provided with a gesture recognition module, The gesture recognition module is connected with the control module. The invention also discloses an operation method of the gesture recognition home theater system. The gesture recognition home theater system is used to accurately control audio-visual equipment using a combination of voice and gestures.

Description

A gesture recognition home theater system and its operation method

technical field

The invention relates to a home theater system, in particular to a gesture recognition home theater system and an operating method thereof.

Background technique

In March 2017, Lenovo released the 65i3, which is a smart TV that can chat and realize complete voice control. The 65i3 integrates far-field voice and near-field remote controller voice, which can recognize, understand and respond . The 65i3 removes the extra equipment of the audio-visual equipment, such as the remote control, which enhances the comfort of the user experience and reduces the cost of the equipment. In addition, there is also a set of gesture operation instruction sets and recognition methods suitable for smart home environments. It is a gesture operation solution for all devices in the home, and it provides a set of gesture operations for smart home environments. Instruction set and its recognition method based on computer vision. The instruction set includes general operation method gestures and shortcut operation gestures. Under the general operation mode, a method of using binocular cameras to judge the device to be controlled based on the position of the human eyes and finger pointing is proposed. method.

my country's smart home started very late, the research is not deep enough, the technology is not mature enough, and there are still some obvious problems and defects in the development of smart home theater. Although the above-mentioned 65i3 is voice control, there are large differences in regional languages with different cultural backgrounds, so it cannot be promoted smoothly. The binocular camera used in the above-mentioned gesture operation instruction set and recognition method of the smart home environment needs to track and recognize human eyes, faces, fingers, and gestures, and the quality and complexity of its image processing cannot be guaranteed. Guarantee the validity of the order. In addition, there are many controlled furniture mentioned in the gesture operation instruction set and its recognition method of the smart home environment, but the types of human gestures are limited, and it is impossible to have detailed control instructions for all furniture equipment. The same gesture has different control commands for different kinds of furniture, which makes it inconvenient to use and operate two kinds of furniture at the same time, which will form "machine misunderstanding", which will make the user experience poor.

Contents of the invention

One of the objectives of the present invention is to provide a home theater system with gesture recognition, which is used to precisely control audio-visual equipment by combining voice and gesture.

To solve the above object, the present invention adopts the following technical solutions:

A gesture recognition home theater system, comprising a pickup module, a control module, audio-visual equipment and a video camera, the control module is respectively connected to the pickup module, audio-visual equipment and the video camera; After success, send the power-on signal to the control module; the control module is used to receive the power-on signal of the pickup module, and then sends a power-on command to the audio-visual equipment and the camera; the control module is used to receive the signal of the camera, and then sends an operation to the audio-visual equipment instruction; the camera is provided with a gesture recognition module, which is connected to the control module; the gesture recognition module is used to recognize the gesture image signal captured by the camera, and send the recognition result to the control module.

Preferably, the camera is a Kinect camera.

Preferably, the gesture recognition module includes an image preprocessing module, a gesture segmentation module, a gesture feature extraction module and a gesture matching module, and the image preprocessing module, gesture segmentation module, gesture feature extraction module and gesture matching module are connected in sequence.

Preferably, the gesture segmentation module includes a skin color segmentation module and a contour segmentation module, and the skin color segmentation module is connected to the contour segmentation module. The skin-color-based segmentation in the skin-color segmentation module and the hand-shape-based contour segmentation in the contour segmentation module can be interchanged smoothly, that is, the preprocessed image can first enter the skin-color segmentation module for skin-color-based segmentation, and then enter the contour segmentation module Carry out segmentation based on the contour of the hand shape, or first enter the contour segmentation module to perform segmentation based on the contour of the hand shape, and then enter the skin color segmentation module to perform segmentation based on skin color.

Preferably, a gesture library is provided in the gesture matching module, and the gesture library contains more than one gesture, and each gesture corresponds to a different operation instruction, and the operation instruction is used to send to the control module.

Preferably, the gesture library contains 14 different gestures, and the 14 different gestures correspond to the previous track or channel, the previous track or channel, the volume down by one level, and the volume up by one level , Decrease the volume by five levels, increase the volume by five levels, pop up the menu for system settings, confirm the selected option, select the previous option, select the next option, mute, release the mute, pause the video being played and shut down the operation instructions.

Another object of the present invention is to provide an operation method of a gesture recognition home theater system.

An operating method of a gesture recognition home theater system employs a convolutional neural network.

Preferably, the following steps are included:

1) The sound pick-up module receives and recognizes the voice signal for power-on, and sends the power-on signal to the control module after the recognition is successful, and the control module sends a power-on command to the audio-visual equipment and the camera;

2) The camera receives and recognizes the gesture image, and sends the recognition result to the control module after the recognition is successful, and the control module sends an operation command to the audio-visual equipment.

As preferably, in said step 2), after the camera receives the gesture image, the depth image in the gesture image is recognized, including the following steps:

a) performing a noise filtering operation on the depth image;

b) Segmenting the depth image after the noise filtering operation;

c) Extracting depth image features on the segmented depth image as samples;

d) Perform one-to-many comparison and matching between the extracted depth image feature samples and the data in the gesture library, and match the corresponding operation instructions.

Preferably, the depth image after the noise filtering operation in the step b) is segmented using two methods based on skin color and based on the shape of the hand.

The beneficial effects of the present invention are:

1. The gesture recognition home theater system of the present invention is equipped with a sound pickup module and a gesture recognition module, and combines two operation methods of voice and gesture. The user can adjust the state of the audio-visual equipment as long as the gesture is changed, such as increasing or decreasing the volume, switching channels , music playback pause, system setting, shutdown and a series of operations, the control is flexible and precise, and the user experience is good;

2. The operation method of the gesture recognition home theater system of the present invention adopts a convolutional neural network to train and recognize gestures, and the operation method has high precision and good stability for gesture recognition;

3. The gesture recognition home theater system of the present invention completely breaks away from the limitation of the remote control, saves the manufacturing cost of additional equipment, and is beneficial to market development.

Description of drawings

FIG. 1 is a structural block diagram of a gesture recognition home theater system provided by an embodiment of the present invention.

FIG. 2 is a structural block diagram of a gesture recognition module in a gesture recognition home theater system provided by an embodiment of the present invention.

FIG. 3 is a flowchart of the operation of the gesture recognition home theater system provided by the embodiment of the present invention.

Fig. 4 is a schematic diagram of each unit of the neural network in the embodiment of the present invention.

FIG. 5 is a schematic diagram of a neural network with one hidden layer in an embodiment of the present invention.

detailed description

The technical solutions provided by the present invention will be described in more detail below in conjunction with FIGS. 1-5 .

Embodiments of the present invention provide a home theater system for gesture recognition. As shown in FIG. Camera connection.

The sound pick-up module is used to recognize the start-up voice signal, and sends the start-up signal to the control module after the recognition is successful.

The control module is used for receiving the power-on signal of the pickup module, and then sends a power-on instruction to the audio-visual equipment and the Kinect camera; the control module is used for receiving the signal of the Kinect camera, and then sends an operation instruction to the audio-visual equipment.

By using the Kinect camera, the gesture recognition home theater system in the embodiment of the present invention can obtain the depth information of the gesture image, and can perform 3D processing on the image, and can judge the gesture action more accurately. It is smarter to make distinctions than to use ordinary cameras. As shown in Figure 1, the Kinect camera is provided with a gesture recognition module, which is connected to the control module; the gesture recognition module is used to recognize the gesture image signal captured by the Kinect camera, and sends the recognition result to the control module. As shown in Figure 2, the gesture recognition module includes an image preprocessing module, a gesture segmentation module, a gesture feature extraction module and a gesture matching module, and the image preprocessing module, gesture segmentation module, gesture feature extraction module and gesture matching module are connected in sequence. The image preprocessing module is used to preprocess the gesture images received by the Kinect camera. The preprocessing process is to enhance the image by removing noise. Because the sensor of the Kinect camera uses laser speckle technology, the acquired depth information often contains a lot of noise. , which will have a considerable impact on subsequent data processing and experiments, so it is necessary to perform noise filtering operations on the depth image in the preprocessing stage.

The gesture segmentation module includes a skin color segmentation module and a contour segmentation module, and the skin color segmentation module and the contour segmentation module are connected. Gesture segmentation has two methods based on skin color segmentation and hand shape contour segmentation. Both methods have great deficiencies when operated alone. Combining the two methods can make the segmentation of gesture images neither affected by the background color nor Accurate segmentation can be performed according to the shape and outline of the hand, that is, gesture segmentation combines two determination methods, avoiding the problem of low accuracy of machine recognition caused by the limitations of a single determination method. The image preprocessing module, skin color segmentation module, contour segmentation module and gesture feature extraction module are connected sequentially. The preprocessed image first enters the skin color segmentation module for skin color segmentation, then enters the contour segmentation module for hand shape contour segmentation, and finally Perform gesture feature extraction. The division based on skin color segmentation in gesture segmentation needs to pay attention to the distinction from the background color and the skin color distinction of different races. The gesture segmentation module can remove part of the hand redundant information and retain important information for feature extraction. In order to obtain the characteristics of the depth interval of the gesture image, the division of the interval must be considered. In the case of including the entire gesture image information, the length of each interval and the total number of intervals should be properly weighed to make the distinction as obvious as possible, so that it can be extracted Characterization of depth information for gesture images.

There is a gesture library in the gesture matching module. The gesture library contains 14 different gestures. Each gesture corresponds to a different operation instruction. The gesture library has many types of operation instructions to rotate. The gestures in the gesture library are designed according to people's living habits. The operation is simple and flexible, and it is easy to be accepted by people. 14 different gestures correspond to the previous track or channel, previous track or channel, volume down one level, volume up one level, volume down five levels, volume up five levels, pop-up menu Operation commands for system settings, confirm selected option, select previous option, select next option, mute, release mute, pause playing video and shut down. The operation instructions corresponding to gestures are as follows:

As shown in Figure 3, the workflow of the gesture recognition home theater system is as follows: the pickup module first receives the voice command for starting up, the Kinect camera is turned on to collect and receive gesture images, the image preprocessing module preprocesses the received gesture images, and gesture segmentation The module segments the preprocessed gesture image, and the gesture feature extraction module performs feature extraction on the segmented gesture image. The extracted features match the corresponding gesture features in the trained gesture library. After the matching is successful, the corresponding After the matching fails, the Kinect camera re-acquires and receives gesture images. After the matching is successful, the corresponding operating instructions are output to the control module, and the control module finally sends corresponding operating instructions to the audio-visual equipment.

An operating method of a gesture recognition home theater system employs a convolutional neural network. By collecting gestures as the training set, the convolutional neural network is trained. The difference between the convolutional neural network and the ordinary neural network is that the convolutional neural network includes a feature extractor composed of a convolutional layer and a sub-sampling layer. In a convolutional layer of CNN, it usually contains several feature planes, and each feature plane is composed of some rectangularly arranged neurons. The neurons of the same feature plane share weights, and the shared weights here are convolution kernels. The convolution kernel is generally initialized in the form of a random decimal matrix, and the convolution kernel will learn to obtain reasonable weights during the training process of the network. The immediate benefit of sharing weights is to reduce the connections between the layers of the network while reducing the risk of overfitting. Subsampling is also called pooling, usually in two forms: mean subsampling and maximum subsampling. Subsampling can be seen as a special kind of convolution process. Convolution and subsampling greatly simplify the model complexity and reduce the parameters of the model. The convolutional neural network consists of three parts, the first part is the input layer, the second part is composed of n convolutional layers and pooling layers, and the third part is composed of a fully connected multilayer perceptron classifier. Each unit in the neural network is shown in Figure 4, and the corresponding formula is as follows:

Wherein, this unit may also be called a Logistic regression model. When multiple units are combined and have a hierarchical structure, a neural network model is formed.

As shown in Figure 5, it shows a neural network with one hidden layer, which is a very simple three-layer neural network model, including an input layer, a hidden layer, and an output layer. Similarly, the hidden layer can be extended to three, four or more layers according to requirements. The above neural network model formula is as follows:

An operation method of a gesture recognition home theater system, comprising the following steps:

1) The sound pick-up module receives and recognizes the voice signal of starting up, and sends the starting up signal to the control module after the recognition is successful, and the control module sends a starting up command to the audio-visual equipment and the Kinect camera;

2) The Kinect camera receives and recognizes the gesture image, and sends the recognition result to the control module after the recognition is successful, and the control module sends an operation command to the audio-visual equipment.

Above-mentioned step 2) in, after Kinect camera receives gesture image, the depth image in gesture image is identified, comprises the following steps:

a) performing a noise filtering operation on the depth image;

b) Segmenting the depth image after the noise filtering operation;

c) Extracting depth image features on the segmented depth image as samples;

d) Perform one-to-many comparison and matching between the extracted depth image feature samples and the data in the gesture library, and match the corresponding operation instructions.

The depth image after the noise filtering operation in the above step b) is segmented using two methods based on skin color and based on the shape of the hand.

Claims (10)

1. A gesture recognition home theater system, characterized in that: It includes a sound pickup module, a control module, audio-visual equipment and a camera, and the control module is respectively connected with the sound pickup module, audio-visual equipment and the camera; The sound pick-up module is used to identify the start-up voice signal, and sends the start-up signal to the control module after the recognition is successful; The control module is used for receiving the power-on signal of the pickup module, and then sends a power-on command to the audio-visual equipment and the camera; the control module is used to receive the signal of the camera, and then sends an operation command to the audio-visual equipment; A gesture recognition module is provided in the camera, and the gesture recognition module is connected with the control module; The gesture recognition module is used to recognize the gesture image signal captured by the camera, and send the recognition result to the control module. 2. The gesture recognition home theater system according to claim 1, wherein the camera is a Kinect camera. 3. The gesture recognition home theater system according to claim 2, characterized in that: the gesture recognition module includes an image preprocessing module, a gesture segmentation module, a gesture feature extraction module and a gesture matching module, the image preprocessing module, The gesture segmentation module, the gesture feature extraction module and the gesture matching module are connected in sequence. 4. The gesture recognition home theater system according to claim 3, wherein the gesture segmentation module includes a skin color segmentation module and an outline segmentation module, and the skin color segmentation module is connected to the outline segmentation module. 5. The gesture recognition home theater system according to claim 4, wherein a gesture library is provided in the gesture matching module, and the gesture library contains more than one gesture, and each gesture corresponds to a different operation command, The operation command is used to send to the control module. 6. The gesture recognition home theater system according to claim 5, characterized in that: the gesture library contains 14 different gestures, and the 14 different gestures respectively correspond to the previous song or the previous channel and the previous song Track or previous channel, decrease volume by one level, increase volume by one level, decrease volume by five levels, increase volume by five levels, pop-up menu for system settings, confirm selected option, select previous option, select next option, mute , release mute, pause the video being played and shut down the operating instructions. 7. An operation method based on the gesture recognition home theater system according to claims 1-6, characterized in that a convolutional neural network is used in the operation method. 8. The operation method of the gesture recognition home theater system according to claim 7, comprising the following steps: 1) The sound pick-up module receives and recognizes the voice signal for power-on, and sends the power-on signal to the control module after the recognition is successful, and the control module sends a power-on command to the audio-visual equipment and the camera; 2) The camera receives and recognizes the gesture image, and sends the recognition result to the control module after the recognition is successful, and the control module sends an operation command to the audio-visual equipment. 9. The operation method of the gesture recognition home theater system according to claim 8, characterized in that: in the step 2), after the camera receives the gesture image, recognizing the depth image in the gesture image comprises the following steps: a) performing a noise filtering operation on the depth image; b) Segmenting the depth image after the noise filtering operation; c) Extracting depth image features on the segmented depth image as samples; d) Perform one-to-many comparison and matching between the extracted depth image feature samples and the data in the gesture library, and match the corresponding operation instructions. 10. The operation method of the gesture recognition home theater system according to claim 9, characterized in that: the depth image after the noise filtering operation in the step b) is aligned by using two methods based on skin color and based on the shape of the hand to split.