CN115620356A - A Framework and Method for Addressee Detection Based on Audio and Facial Input - Google Patents

Abstract

The invention belongs to the technical fields of audio-visual processing and machine learning, and discloses a framework and method for detecting a receiver based on audio and face input. The front end includes an audio stream encoder and a video stream encoder; the back end includes a cross-attention module; Linear fusion module and self-attention module; the framework of the present invention inputs variable-length audio and facial region information, and predicts the receiver in each frame by jointly analyzing audio and facial features. It uses datasets recorded in human-to-human and human-to-robot hybrid settings. Therefore, the framework can be applied and adapted to the robot to distinguish whether the robot is the addressee or not. It makes the robot have intelligent audio-visual perception ability, and improves the intelligence of the robot.

Description

A Framework and Method for Addressee Detection Based on Audio and Facial Input

technical field

The invention belongs to the technical fields of audio-visual processing and machine learning, and in particular relates to a frame and method for detecting a receiver based on audio and face input.

Background technique

A fundamental challenge for humanoid robots is to have an intelligent audio-visual perception capability system to assist natural interaction and cooperation with humans. One way to enrich this system is to let the robot recognize whether it is the recipient or not. It helps robots decide whether to respond to human utterances. Its main applications are guide robots, companion assistants, robot butlers, robot lifeguards, and mobile care robots. However, despite a small amount of prior work, this area has not been extensively explored with state-of-the-art methods for using effective communication cues in real-world settings. Although the research on AD at home and abroad has made important progress in recent years, there is still no research that combines audio and video (facial) features to explore AD. Previous work does not benefit much from existing audio and video information, long and short time clips. Most of these studies focus on segment-level (single image) information from 0.2s to 0.6s, and it is difficult to predict dialogue activities from a single image or a 0.2s video clip. However, in reality, people consider entire sentences spanning hundreds of video frames to judge whether a person is speaking to another person. For example, a 5-second video contains 15 words on average, and a short time segment of 0.2 seconds cannot even cover a complete word. Furthermore, existing frameworks use datasets recorded in conference rooms with fixed participants in human-to-human or human-to-robot settings, which are not suitable for human-robot interaction. Moreover, existing addressee detection works widely use statistical and rule-based methods, which are only applicable to specific tasks and cannot be applied to other situations, e.g., different actions and communication expressions and different numbers of participants.

Contents of the invention

The purpose of the present invention is to provide a framework and method for detecting a receiver based on audio and face input, so as to solve the above-mentioned technical problems.

In order to solve the above-mentioned technical problems, a kind of specific technical scheme of the receiver detection framework and method based on audio and face input of the present invention is as follows:

A framework for addressee detection based on audio and face input, which includes a dual-stream-based end-to-end framework ADNet, which is used to crop variable time lengths of face regions and corresponding audio clips as input, and predict human Whether speaking to the robot or to other people, ADNet includes a front end and a back end, the front end includes an audio stream encoder and a video stream encoder; the back end includes a cross-attention module; a bilinear fusion module and a self-attention module;

The video stream encoder is used to input N continuous face regions, and learns the long-term expression of facial region motion;

The audio stream encoder learns audio feature representations from temporal dynamics;

The cross-attention module is used to dynamically associate video and audio content;

The bilinear fusion module is used to fuse two modalities of video and audio;

The self-attention module is used to monitor listener activities from the background at the utterance level.

Further, the video stream includes two sub-modules: a visual front-end network module and a visual temporal convolution module, which are used to encode the video stream into a visual embedding E _v sequence with the same temporal resolution.

Further, including the fully connected layer, the fully connected layer projects the output of the self-attention network to the AD label sequence through the softmax operation.

Further, the visual front-end network module adopts 3D-ResNet, starts from spatio-temporal convolution, that is, 3D convolution layer, and then gradually reduces the spatial dimension through the 18-layer residual network ResNet18, learns the spatial information of each video frame, and The video frame stream is encoded as a frame-based embedding sequence; the visual temporal convolution module V-TCN is used to represent the temporal content in the long-term visual spatiotemporal stream, and the V-TCN includes five residual connection linear units ReLU, batch regression One BN and depth separable convolution layer DSConv1D, finally, add Conv1D layer to reduce the feature dimension to 128.

Further, the audio stream encoder adopts a ResNet-34 network comprising compression and excitation SE modules; the audio stream encoder uses Mel frequency cepstral coefficient MFCC, and each time step uses 13 Mel frequency bands, so Describe the ResNet-34 network input audio frame sequence to generate the audio embedding E _a sequence, the audio stream encoder feature dimension output is set to (1, 128), the design of ResNet34 uses hole convolution, so that the audio embedding E _a time resolution Matched with the visual embedding _Ev to facilitate the cross-attention module, MFCC features were extracted using a 25ms analysis window with a stride of 10ms and 100 audio frames per second.

Further, the core part of the cross-attention network is the attention layer, and the input is the query (Q _a , Q _v ), key (K _a , K _v ) and value (V _a , V _v ) vector, the output is an audio attention feature: audio cross attention ACA, and a visual attention feature: visual cross attention VCA;

where d represents the dimensions of Q, A, and V. ACA learns by using the target sequence in the video stream to generate queries, and the source sequence in the audio stream to generate keys and values, thereby generating new interactive audio features, and vice versa. Then, new visual features are generated in a similar way; finally, feed-forward layers, residual connections and layer normalization are added before the two cross-attention modules are fed into the fusion layer to generate the final cross-attention network.

Further, the 128-D audio and 128-D visual attention-per-frame features generated by video and audio stream cross-attention are concatenated with bilinear fusion, BLF = f ^blp (ACA _ij , VCA _ij ), and then summed over the positions to Join features along time:

捕获了相应空间位置的相乘交互作用，在视 听交叉注意力层融合音频和视觉注意力特征生成融合特征E_av后，增加 BLF。The resulting features

Capturing the multiplicative interactions of the corresponding spatial locations, the BLF is augmented after the audio-visual cross-attention layer fuses audio and visual attention features to generate the fused feature E _av .

Further, the self-attention module is used to input the fusion feature E _av of BLF to model the time information at the level of audio-visual discourse.

Further, considering AD as a frame-level classification task, the predicted label sequence is compared with the ground-truth label sequence by cross-entropy loss. The loss function is as follows, where P _i and y _i are j ^th video frame j ∈ [a,N ] prediction and ground truth AD labels, N is the number of video frames;

The present invention also discloses a method for deep learning based on the audio and facial input receiver detection framework, comprising the following steps:

Step 1: Build a dataset recorded in a human-to-human and human-to-robot hybrid setting, extend the existing MuMMER dataset by adding spatiotemporal annotations of dialogue activities, or use a custom A dataset recorded in scenes where conversations between robots take place. The dataset is annotated by human annotators generating addressee labels. Each speaking face in the frame is annotated as a bounding box area (x, y), where x represents width and Y represents height; the active time axis describes Audio waveforms to mark the start and end timestamps of the speech, with manually selected start and end timestamps ([(t _s0 , t _e0 ), (t _s1 , t _e1 ), ..., (t _sn , t _en )] ), and mark the activity of each face according to the activity in the boundary of the selected voice segment, where n is the last voice segment in the video, in the data set, speaking to the robot is marked as 0, and the subject talks to the robot, The robot is the receiver; speaking to the subject is marked as 1, and the subject talks to another subject; the data set is randomly divided, 60% is used for training, 20% is used for verification, and 20% is used for testing;

Step 2: Construct a dual-stream-based deep learning framework ADNet, which uses long-term and short-term audio-visual features to predict the addressee, this framework includes audio and video stream encoders for extracting and embedding audio and video information , an audiovisual cross-attention method for cross-modal interaction, a bilinear fusion for combining two modalities, and a self-attention method for capturing long-term speech activity;

Step 3: Train an end-to-end model using the specified E-MuMMER dataset or a custom dataset.

An audio and facial input-based receiver detection framework and method of the present invention have the following advantages: the inventive framework inputs variable-length audio and facial region information, and predicts the region in each frame by jointly analyzing audio and facial features Receiver. It uses datasets recorded in human-to-human and human-to-robot hybrid settings. Therefore, the framework can be applied and adapted to robots to distinguish whether a robot is a callee or not. It makes the robot have intelligent audio-visual perception ability, and improves the intelligence of the robot.

Description of drawings

Fig. 1 is a schematic diagram of the receiver detection framework based on audio and facial input of the present invention;

Fig. 2 is a schematic diagram of the video stream encoder framework of the present invention;

FIG. 3 is a schematic diagram of an audio stream encoder framework of the present invention;

Figure 4 (a) is a schematic diagram of the cross-attention module framework of the present invention;

Figure 4 (b) is a schematic diagram of the framework of the self-attention module of the present invention;

Fig. 5 is a schematic diagram of the annotation interface of the E-MuMMER of the present invention.

detailed description

In order to better understand the purpose, structure and function of the present invention, below in conjunction with accompanying drawing, a kind of receiver detection frame and method based on audio frequency and face input of the present invention are described in further detail.

An audio and facial input-based addressee detection framework that takes variable-length audio and facial region information as input and predicts the addressee in each frame by jointly analyzing audio and facial features. It uses datasets recorded in human-to-human and human-to-robot hybrid settings. Thus, the framework is applicable and applicable to bots to distinguish whether a bot is a callee or not. The implementation steps of the framework are as follows.

Step 1: Construct a dataset recorded in a human-to-human and human-to-robot hybrid setting, extending the existing MuMMER dataset by adding spatiotemporal annotations of dialogue activities. Custom datasets can also be used, but should be recorded in scenarios where human-to-human and human-to-robot conversations occur. The dataset is annotated by a human annotator using the interface shown in Figure 5 to generate the addressee labels. Each speaking face in the frame should be annotated as a bounding box region (x, y). x means width and y means height. The active timeline below the window depicts the audio waveform, marking the start and end timestamps of the speech. Manually select speech segments with start and end timestamps ([(t _s0 , t _e0 ), (t _s1 , t _e1 ), ..., (t _sn , t _en )]), and according to the selected speech segment boundaries The activity of each face marks the activity of each face, where n is the last speech segment in the video. There are two speech activity types in E-MuMMER: "talk to/talk to robot" and "talk to/talk to subject". In the data set, speaking to the robot is marked as 0, the subject is talking to the robot, and the robot is the receiver; speaking to the subject is marked as 1, and the subject is talking to another subject. The dataset is randomly split with 60% for training, 20% for validation and 20% for testing. When there is no speech (no highlighted waveform, see Figure 5) and the robot is talking, no annotations will be displayed.

Step 2: Construct a dual-stream based deep learning framework (called ADNet) that uses long- and short-term audio-visual features to predict recipients. This framework includes audio and video stream encoders for extracting and embedding audio and video information, audiovisual cross-attention methods for cross-modal interaction, bilinear fusion for combining two modalities, and A self-attention approach to temporal phonological activity.

Step 3: Train an end-to-end model using the specified E-MuMMER dataset or a custom dataset.

As shown in Figure 1, the front-end network consists of a video stream encoder (VSE) and an audio stream encoder (ASE). The backend network consists of a cross-attention (CA) module, a bilinear fusion (BLF) module, a self-attention (SA) module and an addressee detector (AD) module on each stream. ADNet is a two-stream based end-to-end framework that takes as input variable temporal lengths (segments) of cropped face regions and corresponding audio segments, and predicts whether the human is speaking to the robot or to someone else. ADNet consists of an audio stream encoder (ASE) and a video stream encoder (VSE) at the front end. These dual streams take frame-based audio and video signals and encode them into audio and video embeddings that represent temporal context. The back-end network consists of three modules: 1) cross-attention module (dynamically associates video and audio content), 2) bilinear fusion module (fusing two modalities), and 3) self-attention module (at the utterance level Monitor callee activity from the background).

As shown in Figure 2, the video stream encoder inputs a series of N consecutive 112×112 grayscale cropped face regions. This encoder is designed to learn long-term representations of motion in facial regions. The video stream includes two sub-modules: a visual front end and a visual temporal network. The aim is to encode a video stream into a sequence of visual embeddings _Ev with the same temporal resolution. For the visual front-end network, we adopted 3D-ResNet. It starts with spatio-temporal convolution (i.e. 3D convolutional layer (3DConv)), and then gradually reduces the spatial dimension through an 18-layer residual network (ResNet18). The goal is to learn the spatial information of each video frame and encode a stream of video frames into a sequence of frame-based embeddings. The Visual Temporal Convolutional Module (V-TCN) represents temporal content in long-term visual spatiotemporal streams. The V-TCN network consists of five residual connected linear units ReLU, batch normalization (BN) and depthwise separable convolutional layers (DSConv1D). Finally, a Conv1D layer is added to reduce the feature dimension to 128.

Audio stream encoders learn audio feature representations from temporal dynamics. As shown in Figure 3, the audio stream encoder employs a ResNet-34 network containing compression and excitation (SE) modules. The audio stream encoder uses Mel-frequency cepstral coefficients (MFCCs), using 13 mel-frequency bands per time step. The ResNet-34 network inputs a sequence of audio frames to generate a sequence of audio embeddings E _a . The feature dimension output of the audio stream encoder is set to (1, 128). ResNet34 is designed with dilated convolutions to match the temporal resolution of the audio embedding E _a with the visual embedding E _v to facilitate the underlying attention module. MFCC features are extracted using a 25ms analysis window with a stride of 10ms and 100 audio frames per second.

As shown in Fig. 4(a), audio-visual cross-attention is added after audio-visual feature embedding to handle dynamic audio-visual interactions in the temporal dimension. The embedded features E _v and E _a aim to distinguish events suitable for audio and visuo-speech activities. The core part of the cross-attention network is the attention layer. The input is the query (Q _a , Q _v ), key (K _a , K _v ) and value (V _a , V _v ) vectors of the audio and visual embeddings respectively projected by the linear layer. As shown in Equation 1 and Equation 2, the output is an audio attention feature (Audio Cross Attention (ACA)) and a visual attention feature (Visual Cross Attention (VCA)).

where d represents the dimensions of Q, A and V. As shown in Equation 1 and Equation 2, ACA learns to generate new interactive audio features by employing target sequences in video streams to generate queries and source sequences in audio streams to generate keys and values, and vice versa. New visual features are generated in a similar manner. Finally, feed-forward layers, residual connections and layer normalization are added before the two cross-attention modules are fed into the fusion layer to generate the final cross-attention network.

The 128-D audio and 128-D visual attention-per-frame features generated by cross-attention from video and audio streams are concatenated with bilinear fusion. Here the bilinear fusion BLF=f ^blp (ACA _ij ,VCA _ij ) aims to compute the external matrix product of two cross-modal attention features at each pixel location, and then sum over the locations to connect along the time direction feature.

捕获了相应空间位置的相乘交互作用。在视 听交叉注意力层融合音频和视觉注意力特征生成融合特征(E_av)后，增加 BLF。The resulting features

The multiplicative interaction of the corresponding spatial locations is captured. BLF is added after the audio-visual cross-attention layer fuses audio and visual attention features to generate fused features (E _av ).

Referring to Fig. 4(b), we adopt the self-attention module to input the fused features (E _av ) of BLF to model the audiovisual discourse-level temporal information. The self-attention architecture is the same as the cross-attention network, except for queries (Q _av ), keys (K _av ) and values (V _av ) from joint audiovisual features, as shown in Fig. 4(b). This module aims to distinguish between frames of conversations with the robot and conversations with other agents.

Finally, a fully connected layer is added, and then the output of the self-attention network is projected to the AD label sequence through a softmax operation. We regard AD as a frame-level classification task. The predicted label sequence is compared to the ground truth label sequence via a cross-entropy loss. The loss function is shown in Equation 5, where P _i and y _i are the predicted and ground-truth AD labels for j ^th video frame j ∈ [a,N]. N is the number of video frames.

It can be understood that the present invention is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, the features and embodiments may be modified to adapt a particular situation and material to the teachings of the invention without departing from the spirit and scope of the invention. Therefore, the present invention is not limited by the specific embodiments disclosed here, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims (10)

1. A receiver detection framework based on audio and face input, said framework comprising an end-to-end framework ADNet based on dual streams, ADNet is used to clip variable time lengths of face regions and corresponding audio clips as input, and Predict whether humans are speaking to robots or to other people, characterized in that ADNet includes a front end and a back end, the front end includes an audio stream encoder and a video stream encoder; the back end includes a cross-attention module; bilinear A fusion module and a self-attention module; the video stream encoder is used to input N continuous face regions, and learn the long-term representation of facial region motion; The audio stream encoder learns audio feature representations from temporal dynamics; The cross-attention module is used to dynamically associate video and audio content; The bilinear fusion module is used to fuse two modalities of video and audio; The self-attention module is used to monitor listener activities from the background at the utterance level. 2. The receiver detection framework based on audio and face input according to claim 1, wherein the video stream includes two sub-modules: a visual front-end network module and a visual time convolution module, which are used to convert the video stream Encoded as a sequence of visual embeddings E _v with the same temporal resolution. 3. The receiver detection framework based on audio and face input according to claim 1, characterized in that it includes a fully connected layer, and the fully connected layer projects the output of the self-attention network to the AD label sequence through a softmax operation. 4. The receiver detection framework based on audio and face input according to claim 2, characterized in that, the visual front-end network module adopts 3D-ResNet, starting from spatio-temporal convolution, i.e. 3D convolutional layer, and then passing The 18-layer residual network ResNet18 gradually reduces the spatial dimension, learns the spatial information of each video frame, and encodes the video frame stream into a frame-based embedding sequence; the visual temporal convolution module V-TCN is used to represent long-term visual Temporal content in the spatio-temporal stream, V-TCN includes five residual connected linear units ReLU, batch normalization BN and depth separable convolutional layer DSConv1D, and finally, adding the Conv1D layer to reduce the feature dimension to 128. 5. The receiver detection framework based on audio and face input according to claim 1, wherein the audio stream encoder adopts the ResNet-34 network comprising compression and excitation SE modules; the audio stream encoder Using the Mel frequency cepstral coefficient MFCC, each time step uses 13 Mel frequency bands, the ResNet-34 network inputs the audio frame sequence to generate the audio embedding E _a sequence, and the audio stream encoder feature dimension output is set to (1, 128), ResNet34 is designed with atrous convolution to match the temporal resolution of the audio embedding E _a with the visual embedding E _v to facilitate the cross-attention module, using a 25ms analysis window to extract MFCC features with a stride of 10ms, producing 100 audio frames per second. 6. The receiver detection framework based on audio and face input according to claim 1, wherein the core part of the cross-attention network is the attention layer, and the input is the audio and visual embeddings respectively projected by the linear layer Query (Q _a , Q _v ), key (K _a , K _v ) and value (V _a , V _v ) vectors, output as audio attention features: audio cross attention ACA, and visual attention features: visual cross attention Attention VCA;

where d represents the dimensions of Q, A, and V. ACA learns by using the target sequence in the video stream to generate queries, and the source sequence in the audio stream to generate keys and values, thereby generating new interactive audio features, and vice versa. Then, new visual features are generated in a similar way; finally, feed-forward layers, residual connections and layer normalization are added before the two cross-attention modules are fed into the fusion layer to generate the final cross-attention network. 7. The receiver detection framework based on audio and face input according to claim 6, characterized in that, the 128-dimensional audio and 128-dimensional visual attention generated by video and audio stream cross-attention per frame feature and bilinear Fusion connection, BLF=f ^blp (ACA _ij , VCA _ij ), then sum over locations to concatenate features along time direction:

The resulting features

Capturing the multiplicative interactions of the corresponding spatial locations, the BLF is augmented after the audio-visual cross-attention layer fuses audio and visual attention features to generate the fused feature E _av . 8. The receiver detection framework based on audio and face input according to claim 7, characterized in that, the fusion feature E _av of the self-attention module input into the BLF is used to model the time information of the audio-visual discourse level. 9. The addressee detection framework based on audio and face input according to claim 7, characterized in that AD is considered as a frame-level classification task, and the predicted label sequence is compared with the ground truth label sequence by cross-entropy loss , the loss function is as follows, where P _i and y _i are the predicted and ground truth AD labels of j ^th video frame j ∈ [a,N], N is the number of video frames;

. 10. A method utilizing the receiver detection framework based on audio and face input as described in any one of claims 1-9 to carry out deep learning, characterized in that it comprises the steps of: Step 1: Build a dataset recorded in a human-to-human and human-to-robot hybrid setting, extend the existing MuMMER dataset by adding spatiotemporal annotations of dialogue activities, or use a custom The data set is recorded in the scene where the dialogue between robots occurs, and the human annotator generates the label of the receiver to annotate the data set. Each speaking face in the picture is annotated as a bounding box area (x, y), x Indicates the width, Y indicates the height; the active time axis describes the audio waveform to mark the start and end timestamps of speech, manually select the start and end timestamps ([(t _s0 , t _e0 ), (t _s1 , t _e1 ), ..., (t _sn , t _en )]) and label the activity of each face according to the activity in the boundary of the selected speech segment, where n is the last speech segment in the video, and in the dataset, Speaking to the robot is marked as 0, the subject talks to the robot, and the robot is the receiver; speaking to the subject is marked as 1, the subject talks to another subject; the data set is randomly divided, 60% of which are used for training and 20% for verification , 20% for testing; Step 2: Construct a dual-stream-based deep learning framework ADNet, which uses long-term and short-term audio-visual features to predict the addressee, this framework includes audio and video stream encoders for extracting and embedding audio and video information , an audiovisual cross-attention method for cross-modal interaction, a bilinear fusion for combining two modalities, and a self-attention method for capturing long-term speech activity; Step 3: Train an end-to-end model using the specified E-MuMMER dataset or a custom dataset.

Images