CN112287893B - A recognition method of sow lactation behavior based on audio and video information fusion - Google Patents

Abstract

The invention discloses a sow lactation behavior identification method based on audio and video information fusion, which comprises the following steps: acquiring overlooking video and audio data of lactating sows, extracting audio and video pictures from the original video and audio data, carrying out audio denoising and framing, acquiring an audio waveform chart sequence, and extracting a corresponding optical flow chart sequence according to the original video pictures; inputting the optical flow sequence and the video frame into a preset appearance-motion double-flow network to extract visual characteristics, inputting the audio waveform diagram sequence into a preset auditory characteristic extraction network, splicing and fusing the visual and auditory characteristics, inputting into a preset long-period and short-period memory network to extract time sequence visual and auditory characteristics, and finally, sending into a full-connection layer and a soft maximum to conduct behavior classification, and outputting a lactation behavior class. The invention utilizes the visual and auditory information in the lactation behavior to identify the sow lactation behavior in the pig farm environment, thereby acquiring the sow lactation information, being beneficial to timely finding out abnormal conditions of the lactation behavior and taking effective measures, and improving the economic benefit of the pig farm.

Description

A recognition method of sow lactation behavior based on audio and video information fusion

technical field

The invention relates to the technical field of intelligent animal husbandry, multi-modal information fusion and interactive behavior recognition, in particular to a sow-feeding behavior recognition method based on audio and video information fusion.

Background technique

The lactation behavior of sows is not only an important reflection of the health status of sows and maternal behavior, but also the main factor affecting the survival and growth of lactating piglets. Observing the status of sows' lactation behavior and timely detection of abnormal lactation behavior will help pig farm managers to make effective manual intervention decisions in a timely manner, improve the health status of sows and piglets, and thus improve the economic benefits of pig farms.

At present, the monitoring of sows' lactation behavior is mainly through manual on-site or video monitoring, and visual-based behavior monitoring methods. The manual observation method requires manual on-site or long-term observation of the behavior of sows through surveillance video, and records information such as the time, duration, and frequency of sows’ lactation behavior, and then judges whether the sows’ lactation behavior is abnormal through experience. This method is time-consuming and labor-intensive. , strong subjectivity and unable to adapt to the development of large-scale farming methods. Sow lactation behavior is an interactive behavior between sows and piglets. Currently, there are few reports on the recognition of sow lactation behavior based on vision. The method is mainly based on the visual information of sows during lactation, such as sow side Lying down, exposing the udder, piglets lie down on the sow's udder to suck milk, etc., automatically obtain the sow's lactation behavior information through algorithm analysis. However, due to changes in the ambient light of the pig house, crowding of piglets, and occlusion, some visual information cannot be obtained, which affects the judgment and analysis of breastfeeding behavior.

In summary, the existing monitoring methods for lactation behavior of sows are not suitable or difficult to identify the lactation behavior of sows. In view of this, there is an urgent need to provide an automatic identification method for the lactation behavior of sows, so as to accurately monitor the lactation behavior of sows. Since sows are accompanied by regular suckling sounds when breastfeeding, the present invention provides a method for recognizing sows’ lactating behaviors based on the fusion of audio and video information, through the cooperation and assistance of video and audio signals to achieve accurate breastfeeding behaviors of sows. monitor.

Contents of the invention

The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art that it is difficult to accurately identify the lactation behavior of sows, and proposes a method for recognizing the lactation behavior of sows based on the fusion of audio and video information, providing data support for making effective management decisions in a timely manner, and improving Pig health status and farm economics.

In order to achieve the above object, the technical solution provided by the present invention is: a method for recognizing a sow's lactation behavior based on audio and video information fusion, comprising the following steps:

1) Collect audio and video data of lactating sows;

2) Data preprocessing: first separate the audio and video data, then denoise the audio data, divide into frames and obtain the audio waveform sequence, and finally perform optical flow extraction on the video data to obtain the optical flow image sequence;

3) Input the video frame and optical flow image sequence into the preset appearance-motion dual-stream network for feature extraction to obtain visual features, and input the audio waveform sequence into the preset auditory feature extraction network to obtain auditory features;

4) Input the visual features and auditory features into the long-term short-term memory network for further feature fusion and extraction, and obtain temporal audio-visual features;

5) The time-series audiovisual features are sent to the fully connected layer and the soft maximum classifier for behavior classification, so as to realize automatic recognition of sow lactation behavior.

In step 1), a camera with recording function is installed directly above the pig house to collect video and audio data of the lactating sow looking down.

Described step 2) comprises the following steps:

2.1) Separate the audio and video data from the captured audio and video data;

2.2) processing the original audio signal with a bandpass filter to obtain a denoised audio signal corresponding to the original audio signal;

2.3) The denoised audio signal is divided into frames, the frame length is 30ms, and the frame overlaps by 10ms, and the audio signal is converted into an audio waveform sequence;

2.4) Using the optical flow method to obtain the optical flow image sequence of the lactating sow to be monitored according to the original image sequence of the lactating sow to be monitored.

Described step 3) comprises following two kinds of processing:

a. Input the video frame and optical flow image sequence into the preset appearance-motion dual-stream network, and extract the corresponding video frame and optical flow image sequence through the convolution layer, down-sampling layer and fully connected layer of the appearance-motion dual-stream network Appearance-movement features, and output a one-dimensional visual feature vector; wherein, before inputting the video frame and optical flow image sequence into the preset appearance-movement dual-stream network, the preset appearance-movement dual-stream network needs to be trained first, as follows:

Obtain the original video frame and optical flow image sequence marked with breast-feeding behavior; input the original video frame with breast-feeding behavior mark and the corresponding optical flow image sequence into the appearance-motion dual-stream network for training, and obtain the optimal appearance-motion dual-stream network Network parameters;

b. Input the audio waveform sequence into the preset auditory feature extraction network, and output the one-dimensional auditory feature vector through the convolutional layer, downsampling layer and fully connected layer of the auditory feature extraction network; wherein, after inputting the audio waveform sequence Before the preset auditory feature extraction network, it is necessary to train the preset auditory feature extraction network, as follows:

Obtain the original audio data marked with breastfeeding behavior; use a bandpass filter to denoise the original audio signal; divide the denoised audio signal into frames, the frame length is 30ms, and the frame overlap is 10ms; the framed The audio signal is converted into an audio waveform sequence to obtain the audio waveform sequence corresponding to the original audio signal; the audio waveform sequence with the breastfeeding behavior mark is input into the preset auditory feature extraction network for training, and the optimal auditory feature extraction network is obtained. Network parameters;

Described step 4) comprises the following steps:

4.1) Accumulate the one-dimensional visual feature vector and the one-dimensional auditory feature vector to obtain the visual and auditory features;

4.2) Send the audiovisual features into the preset long-term short-term memory network for feature extraction, and output the temporal audiovisual features;

Among them, before sending the visual and auditory features into the preset long-term short-term memory network, the preset network needs to be trained first, as follows:

Obtain the original video and audio sequence samples with behavioral markers, obtain the corresponding optical flow image sequence samples according to the original video sequence with behavioral markers, and obtain the denoised audio waveform diagram based on the original audio sequence samples with behavioral markers sequence samples;

Input the original video frame and optical flow sequence with behavioral markers into the preset appearance-motion dual-stream network to extract visual features, and input the audio waveform sequence samples into the preset auditory feature extraction network to extract auditory features;

The auditory features and visual features are piled up and spliced, and input into the preset long-term short-term memory network for training to obtain the optimal network parameters.

Described step 5) comprises the following steps:

5.1) Input the time-series audiovisual features into a fully connected layer for further feature extraction and integration, and obtain 2 feature values, corresponding to the feature values of breast-feeding behavior and non-breast-feeding behavior;

5.2) Input the feature values of breast-feeding behavior and non-breast-feeding behavior into the soft maximum classifier to calculate the probability values of the two features corresponding to breast-feeding behavior and non-breast-feeding behavior, and use the behavior category of the one with the highest probability as the behavior recognition result, so as to realize the Recognition of suckling behavior in pigs.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

1. The present invention provides a method for automatically identifying the lactation behavior of sows, thereby obtaining information on the lactation behavior of sows and further analyzing the ability of the lactation behavior.

2. The present invention integrates video and audio information, and the video and audio information cooperate and assist each other. Therefore, the extracted features are rich and accurate.

3. The present invention uses video frame sequence, optical flow sequence and audio sequence to analyze from the three aspects of spatial distribution, time series motion and sound respectively, so the recognition accuracy is high.

4. The present invention adopts the appearance-motion dual-stream network to fuse optical flow and video convolution features, which enhances the ability to express visual features.

5. The present invention adopts the long-short-term memory network to fuse visual and auditory features, which enhances the ability to express behavioral features in time series.

6. In the potential application method, the occurrence time, duration and frequency of sow's lactation behavior are counted within a fixed period of time to study the law of sow's lactation behavior. On the one hand, these lactation information can be used to predict the health and welfare of sows and piglets, on the other hand, it can provide data reference for judging the maternal ability of sows, and further provide decision support for breeding selection and feeding management in pig farms, so as to improve the economic benefits of pig farms .

7. For behaviors accompanied by specific sounds, the fusion of video and audio multi-modal data has strong applicability and can be used for research on other animal behaviors.

Description of drawings

Fig. 1 is the overall flow diagram of the method of the present invention.

Fig. 2 is a structure diagram of the method of the present invention.

Figure 3 is a schematic diagram of the appearance-motion dual-stream network.

Detailed ways

The present invention will be further described below in conjunction with specific examples.

As shown in Figures 1 and 2, the method for recognizing the lactation behavior of sows based on the fusion of audio and video information provided by the present invention realizes the recognition of the lactation behavior of commercial captive sows based on audio and video multimodal data, providing real-time monitoring and Analyze the lactation behavior and health status of sows to provide reference, which includes the following steps:

1) Collect audio and video data of lactating sows;

Specifically, a camera with a recording function is installed directly above the actual commercial pig house to continuously record video and audio of the daily behavior of lactating sows. In this embodiment, the Hikvision DS-2CD3T46FWD V2-I3 high-definition audio camera is used to obtain the video and audio data of the daily behavior including the lactation behavior of sows.

2) Data preprocessing: first separate the audio and video data, then denoise the audio data, divide into frames and obtain the audio waveform sequence, and finally perform optical flow extraction on the video data to obtain the optical flow image sequence; specifically include the following steps :

2.1) Use audio and video editing and conversion tools to separate audio and video from the acquired video and audio data, and extract audio data and video images;

2.2) Due to the noise of piglets, pigs in other stalls, and machinery and equipment in the actual pig house, in order to reduce the judgment of these noises on the sound of sows' lactation, the present invention first denoises the acquired original audio data. Before lactation, the sow will make a "humming and snorting" sound to attract the piglets to come to suck, and guide the piglets to switch between massage and sucking by changing the speed of grunting. Based on this nature, this embodiment uses The band-pass filter suitable for sows' lactation sounds filters the original audio to reduce the influence of other noises on the lactation sounds;

2.3) In order to be suitable for the two-dimensional image input of the convolutional network in this embodiment, this embodiment divides the time-series audio signal after denoising into frames, the frame length is 30ms, and the overlap between frames is 10ms, and the audio signal is converted into audio waveform sequence;

2.4) Using the optical flow method to obtain the optical flow image sequence of the sow to be monitored according to the original image sequence of the lactating sow to be monitored, each pixel in the optical flow image records the motion intensity and direction of the pixel.

3) Input the video frame and optical flow image sequence into the preset appearance-motion dual-stream network for feature extraction to obtain visual features, and input the audio waveform sequence into the preset auditory feature extraction network to obtain auditory features; the details are as follows:

a. Input the video frame and optical flow image sequence into the preset appearance-motion dual-stream network, and extract the video frame and optical flow image sequence through the convolutional layer, down-sampling layer and fully connected layer of the appearance-motion dual-stream network. Corresponding appearance-movement features, and output a one-dimensional visual feature vector;

In this embodiment, the preset appearance-motion dual-stream network structure is shown in Figure 3. The network structure is composed of appearance flow and motion flow, which take video frames and optical flow sequences as input respectively, and then go through 5 layers of convolutional layers and After the 5-layer downsampling layer, the feature map of the same dimension is output. The two-stream feature map is fused by splicing and sent to the 2-layer convolutional layer for further feature extraction and fusion. The fused feature map is sent to the continuous 2-layer full connection. layer, which outputs one-dimensional visual features representing appearance and motion;

The network needs to be trained before video frames and optical flow sequences are fed into the appearance-motion two-stream network structure. Since the network only implements feature extraction, in order to obtain the optimal network parameters of the network, before training the network, a full convolutional layer and a softmax (softmax) layer are added after the network for behavior classification. During the training process, the number of output behavior categories of the network is set to 2, representing breast-feeding behavior and non-breast-feeding behavior. Propagation and backward feedback training to obtain optimal network parameters for appearance-motion two-stream networks. Raw video frames and optical flow sequences are input into the trained appearance-motion two-stream network to output 1D visual features after convolutional, downsampling and fully connected layers.

b. Input the audio waveform sequence into the preset auditory feature extraction network, and output a one-dimensional auditory feature vector through the convolutional layer, downsampling layer and fully connected layer of the auditory feature extraction network;

In this embodiment, the preset auditory feature extraction structure consists of 5 convolutional layers, 5 downsampling layers and 2 fully connected layers;

The network needs to be trained before the audio waveform sequence is input into the preset auditory feature extraction network. Since the network only implements feature extraction, in order to obtain the optimal network parameters of the network, before training the network, a full convolutional layer and a softmax layer are added after the network for behavior classification, and the output behavior category is set The number is 2, which means breast-feeding behavior and non-breast-feeding behavior;

Obtain the original audio data marked with breastfeeding behavior, use a bandpass filter to denoise the original audio signal, divide the denoised audio signal into frames, the frame length is 30ms, and the frame overlap is 10ms, and the framed The audio signal is converted into an audio waveform sequence, and the audio waveform sequence with breastfeeding behavior markers is input into the preset auditory feature extraction network for forward propagation and backward feedback training to obtain the optimal network parameters of the auditory feature extraction network; audio The waveform image sequence is input to the convolutional layer, downsampling layer and fully connected layer of the trained auditory feature extraction network, and then the one-dimensional auditory feature is output.

4) Input the visual and auditory features into the long-term short-term memory network for further feature fusion and extraction, and obtain time-series audio-visual features, specifically including the following steps:

4.1) Accumulate and splice one-dimensional visual features and auditory features to obtain visual and auditory features;

4.2) Send audiovisual features into the preset long-term short-term memory network for feature extraction, and output one-dimensional time series features. The preset long-term short-term memory network is set and trained in advance; the long-term short-term memory network (LSTM ) is a temporal recurrent neural network suitable for processing and forecasting time series events. An LSTM consists of a cell and three controllers. The three controllers are the input gate, the forget gate and the output gate, which are used to protect and control the cell state. In this embodiment, the LSTM is designed for time series feature extraction, and the LSTM can be set according to actual needs, which is not specifically limited here.

When LSTM is trained, it can be combined with subsequent fully connected layers and soft maximum classifiers to form a classification network for network parameter training. The specific training steps of LSTM include:

Obtain the original video and audio sequence samples with behavioral markers, obtain optical flow image sequence samples with behavioral markers from the original video sequence with behavioral markers, and obtain behavioral markers from the original audio sequence samples with behavioral markers Denoised audio waveform sequence samples; input video and optical flow sequences with behavioral markers into the preset appearance-motion dual-stream network to extract visual features, and input audio waveform sequence samples into the preset auditory feature extraction network to extract auditory features ; The auditory features and visual features are piled up and spliced, and input into the preset long-term short-term memory network for forward propagation and backward feedback training to obtain optimal network parameters.

5) Send the time-series audiovisual features to the fully connected layer and the soft-max classifier for behavior classification to realize automatic recognition of sows’ lactation behavior, specifically including the following steps

5.1) Input the temporal audiovisual features into a fully connected layer for further feature extraction and integration, set the number of neurons in the fully connected layer to 2, and obtain 2 feature values, corresponding to the feature values of breastfeeding behavior and non-breastfeeding behavior;

5.2) Input the feature values of breast-feeding behavior and non-breast-feeding behavior into the soft maximum classifier to calculate the probability values of the two features corresponding to breast-feeding behavior and non-breast-feeding behavior, and use the behavior category of the one with the highest probability as the behavior recognition result, so as to realize the Recognition of suckling behavior in pigs.

In summary, the method for recognizing the lactation behavior of sows disclosed in the present invention includes collecting video and audio data from the top view of lactating sows, extracting audio and video images from the original video and audio data, audio denoising and framing, and obtaining audio waveforms Sequence, and extract the corresponding optical flow sequence based on the original video picture; input the optical flow sequence and video frame into the preset appearance-motion dual-stream network to extract visual features, input the audio waveform sequence into the preset auditory feature extraction network, and then splicing and fusion Visual and auditory features are input into the preset long-term and short-term memory network to extract time-series audiovisual features, and finally sent to the fully connected layer and soft maximum for behavior classification, and the output of the lactation behavior category is to realize the recognition of sow lactation behavior. This research is based on the fusion of video and audio multi-modal information to identify the lactation behavior of sows, provide reliable lactation information for pig farm managers, guide pig farm managers to make timely and effective management decisions, and improve the health and welfare of pigs At the same time, it also provides data support for the exploration of the law of sow's maternal sexual behavior, which is worthy of promotion.

The above-described embodiments are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, all changes made according to the shape and principles of the present invention should be covered within the protection scope of the present invention.

Claims (5)

1. A sow-feeding behavior recognition method based on audio-video information fusion, is characterized in that, comprises the following steps: 1) Collect audio and video data of lactating sows; 2) Data preprocessing: first separate the audio and video data, then denoise the audio data, divide into frames and obtain the audio waveform sequence, and finally perform optical flow extraction on the video data to obtain the optical flow image sequence; 3) Input the video frame and optical flow image sequence into the preset appearance-motion dual-stream network for feature extraction to obtain visual features, and input the audio waveform sequence into the preset auditory feature extraction network to obtain auditory features; wherein, the appearance- The motion dual-stream network consists of an appearance stream and a motion stream, which take video frames and optical flow sequences as input respectively, and then output feature maps of the same dimension after passing through 5 layers of convolutional layers and 5 layers of downsampling layers. The feature maps of the two streams are concatenated After fusion, it is sent to a 2-layer convolutional layer for feature extraction and fusion, and the fused feature map is sent to a continuous 2-layer fully connected layer to output a one-dimensional visual feature representing appearance and motion; 4) Input the visual features and auditory features into the long-term short-term memory network for further feature fusion and extraction, and obtain temporal audio-visual features; 5) The time-series audiovisual features are sent to the fully connected layer and the soft maximum classifier for behavior classification to realize automatic recognition of sows' lactation behavior. 2. A method for recognizing sows' lactation behavior based on audio and video information fusion according to claim 1, characterized in that: in step 1), a camera with a recording function is installed directly above the pigsty to collect data from lactating sows. Pigs look down on video and audio data. 3. A method for recognizing a sow's lactation behavior based on audio-video information fusion according to claim 1, wherein said step 2) comprises the following steps: 2.1) Separate the audio and video data from the captured audio and video data; 2.2) Process the original audio signal with a bandpass filter to obtain a denoised audio signal corresponding to the original audio signal; 2.3) Divide the denoised audio signal into frames with a frame length of 30ms and an overlap of 10ms between frames, and convert the audio signal into an audio waveform sequence; 2.4) Using the optical flow method to obtain the optical flow image sequence of the lactating sow to be monitored based on the original image sequence of the lactating sow to be monitored. 4. A method for recognizing a sow's lactation behavior based on fusion of audio and video information according to claim 1, wherein the step 3) includes the following two processes: a. Input the video frame and optical flow image sequence into the preset appearance-motion dual-stream network, and extract the corresponding video frame and optical flow image sequence through the convolution layer, down-sampling layer and fully connected layer of the appearance-motion dual-stream network Appearance-movement features, and output a one-dimensional visual feature vector; wherein, before inputting the video frame and optical flow image sequence into the preset appearance-movement dual-stream network, the preset appearance-movement dual-stream network needs to be trained first, as follows: Obtain the original video frame and optical flow image sequence marked with breast-feeding behavior; input the original video frame with breast-feeding behavior mark and the corresponding optical flow image sequence into the appearance-motion dual-stream network for training, and obtain the optimal appearance-motion dual-stream network Network parameters; b. Input the audio waveform sequence into the preset auditory feature extraction network, and output the one-dimensional auditory feature vector through the convolutional layer, downsampling layer and fully connected layer of the auditory feature extraction network; wherein, after inputting the audio waveform sequence Before the preset auditory feature extraction network, it is necessary to train the preset auditory feature extraction network, as follows: Obtain the original audio data marked with breastfeeding behavior; use a bandpass filter to denoise the original audio signal; divide the denoised audio signal into frames, the frame length is 30ms, and the frame overlap is 10ms; the framed The audio signal is converted into an audio waveform sequence to obtain the audio waveform sequence corresponding to the original audio signal; the audio waveform sequence with the breastfeeding behavior mark is input into the preset auditory feature extraction network for training, and the optimal auditory feature extraction network is obtained. Network parameters; Said step 4) includes the following steps: 4.1) Accumulate and stitch the one-dimensional visual feature vector and one-dimensional auditory feature vector to obtain visual and auditory features; 4.2) Send audiovisual features into the preset long-term short-term memory network for feature extraction, and output time-series audiovisual features; Among them, before sending the visual and auditory features into the preset long-term short-term memory network, the preset network needs to be trained first, as follows: Obtain the original video and audio sequence samples with behavioral markers, obtain the corresponding optical flow image sequence samples according to the original video sequence with behavioral markers, and obtain the denoised audio waveform diagram based on the original audio sequence samples with behavioral markers sequence samples; Input the original video frame and optical flow sequence with behavioral markers into the preset appearance-motion dual-stream network to extract visual features, and input the audio waveform sequence samples into the preset auditory feature extraction network to extract auditory features; The auditory features and visual features are piled up and spliced, and input into the preset long-term short-term memory network for training to obtain the optimal network parameters. 5. A method for recognizing a sow's lactation behavior based on audio-video information fusion according to claim 1, wherein said step 5) comprises the following steps: 5.1) Input the time-series audiovisual features into a fully connected layer for further feature extraction and integration, and obtain 2 feature values, corresponding to the feature values of breast-feeding behavior and non-breast-feeding behavior; 5.2) Input the feature values of breast-feeding behavior and non-breast-feeding behavior into the soft maximum classifier to calculate the probability values of the two features corresponding to breast-feeding behavior and non-breast-feeding behavior, and use the behavior category of the one with the highest probability as the behavior recognition result, so as to realize the Recognition of suckling behavior in pigs.