JP7688715B2 - Detecting interlocutors in multi-human computer interaction scenes - Google Patents

Description

The present invention belongs to the field of computer technology, and in particular to the detection of interlocutors in multi-human computer interaction scenes.

In the process of linguistic interaction, there must be both a speaker and an interlocutor from whom the speaker expects a response. In particular, in the process of human-computer interaction, the robot responds after receiving voice information.

For example, when one person interacts with a robot, the robot is inevitably the interlocutor when the human speaks. Therefore, the robot can directly process the received voice information and respond. Such functionality is already being used in some smart devices and has been highly effective.

However, the interaction between a group of humans and a robot is more complicated than the interaction between a single person and a robot. Because human-to-human and human-to-robot interactions exist simultaneously, the robot cannot judge whether the person speaking is speaking to it or not, and as a result, it can only mechanically respond to all the speech it receives, which seriously affects the interaction and experience between users. In such a case, the human has no choice but to repeatedly use the wake word to have multiple interactions with the robot, which reduces the efficiency of the interaction.

To solve the above technical problems, an embodiment of the present invention provides an apparatus and method for interlocutor detection in a multi-human computer interaction scene.

According to one embodiment of the present invention, there is provided an apparatus for interlocutor detection in a multi-human computer interaction scene, the multi-human computer interaction relating to a human group including multiple humans and at least one robot, wherein the apparatus comprises an audio-video collection module for collecting time-stamped video frame data and time-stamped audio frame data in real time, where a plurality of video frames included in the video frame data and a plurality of audio frames included in the audio frame data are synchronized according to timestamps, a text generation module for generating time-stamped text information based on the audio frame data, a face processing module for detecting faces in each video frame included in the video frame data by a machine vision method and tracking the same person in the plurality of video frames to obtain face sequence data, a text feature extraction module for extracting text semantic features from the time-stamped text information by a machine learning or deep learning method, and a text feature extraction module for extracting text semantic features from the time-stamped text information by a machine learning or deep learning method. The learning method includes an audio feature extraction module for extracting voice audio features from the audio frame data, a face feature extraction module for extracting person facial features including time series features and spatial features of the person's face from the face sequence data, a speaker detection module for recognizing a speaker at a current time in the human group based on the person's facial features and the voice audio features in the face sequence data, and obtaining information of the speaker at the current time, and a conversation partner recognition module for recognizing a conversation partner of the speaker at the current time in the human group based on scene features, the text semantic features, the voice audio features, and the person's facial features in the face sequence data, and detecting whether the conversation partner of the speaker at the current time is a robot, in which the scene features include speaker information and conversation partner information at a previous time. The scene features may also be stored in a scene database to be called by the conversation partner recognition module.

Further, the audio/video collection module includes a video collection module for collecting time-stamped video frame data in real time using a camera, and an audio collection module for collecting time-stamped audio frame data using a micro. Optionally, the video frame data is stored in a video frame database in chronological order, and the audio frame data is stored in an audio frame database in chronological order.

The face processing module further includes a face detection module for detecting faces in video frames included in the video frame data using a deep learning method and assigning a unique fixed identifier to a same face detected from two or more video frames to represent the person, and a face tracking module for tracking the same person in multiple video frames based on the detection result output by the face detection module and obtaining time-stamped face sequence data. By assigning a unique fixed identifier to the same face, the person can be represented by its original id even if the person disappears and then reappears in the scene field of view. Optionally, the time-stamped face sequence data is stored in a face database.

Further, the speaker detection module includes a first multimodal fusion module for fusing the facial features and the voice audio features of the person according to a timestamp based on the face sequence data into a first multimodal feature, and a speaking state detection module for inputting the first multimodal feature into a deep learning network and determining a speaker and corresponding speaker information at the current time by predicting the speaking state of each person in the human group one by one at the current time. Optionally, the speaker information at the current time is stored in a speaker database. For example, the speaker database may store the speaker information according to a timestamp.

Furthermore, the interlocutor recognition module includes a second multimodal fusion module for fusing the face features of the person, the voice audio features, the text semantic features, and the scene features into a second multimodal feature according to a timestamp based on the face sequence data, and an interlocutor detection module for inputting the second multimodal feature into a deep learning network, predicting one by one whether each person in the human group and each of the robots is an interlocutor of the speaker at the current time, and determining the interlocutor information at the current time accordingly. Optionally, the interlocutor information at the current time is stored in a interlocutor database for being called by other modules or output as a result. For example, the interlocutor database may store the interlocutor information according to a timestamp.

Further, the text generation module includes a speech recognition module for generating time-stamped text information corresponding to a plurality of hierarchies based on the audio frame data, where the plurality of hierarchies include a word level, a sentence level, a dialogue topic level, etc. Optionally, the text information is stored hierarchically in chronological order using a text database.

According to another embodiment of the present invention, there is provided a method for detecting interlocutors in a multi-human computer interaction scene, the multi-human computer interaction relating to a human group including multiple humans and at least one robot. Here, the method includes a step S1 of collecting time-stamped video frame data in real time by an audio/video collection module, for example using a camera, collecting time-stamped audio frame data, for example using a microcomputer, and synchronizing a plurality of video frames included in the video frame data and a plurality of audio frames included in the audio frame data according to time stamps; a step S2 of generating time-stamped text information at different hierarchical levels, such as a word level, a sentence level, and a dialogue topic level, by performing speech recognition on the audio frame data in real time by a text generation module, and extracting text semantic features from the time-stamped text information by a text feature extraction module; and a step S3 of detecting the appearance of each video frame included in the video frame data by a machine vision method by a face processing module. The method includes a step S3 of detecting a face in the human group, tracking the same person in multiple video frames to obtain face sequence data, extracting facial features of the person from the face sequence data by a facial feature extraction module, and extracting voice audio features from the audio frame data by an audio feature extraction module; a step S4 of recognizing a speaker at a current time in the human group based on the facial features and the voice audio features of the person by a machine learning or deep learning method by a speaker detection module, and obtaining information of the speaker at the current time; and a step S5 of recognizing a conversation partner of the speaker at the current time in the human group by a machine learning or deep learning method by a conversation partner recognition module, based on scene features, the text semantic features, the voice audio features, and the facial features of the person, and detecting whether the conversation partner of the speaker at the current time is a robot. Here, the scene features include information of the speaker and conversation partner at a previous time.

Furthermore, in step S1, the video frame data may be published in the form of a Robot Operating System (ROS) topic, and the video frame data may be obtained in real time by subscribing to an image topic, and the audio frame data may be published in the form of a ROS topic, and the audio frame data may be obtained in real time by subscribing to an audio topic. In step S2, face detection may be performed using YOLO (You Only Look Once), and multiple targets are tracked using a Deep Simple Online Realtime Tracking (Deep SORT) model, and an ID is assigned to each person as a result of tracking, and the ID of each person is unique and fixed throughout the entire process.

Furthermore, step S4 may specifically include a step of fusion coding the facial features and the voice audio features of the person according to a timestamp based on the face sequence data to obtain a first multimodal feature, and a step of predicting a speaker at a current time in the human group based on the first multimodal feature using a deep learning method.

Furthermore, the step S5 may specifically include a step of fusion-coding the scene features, the text semantic features, the voice audio features and the facial features of the person according to the timestamp based on the face sequence data, i.e., performing multimodal feature fusion to obtain second multimodal features, and a step of predicting, one by one, the probability that each person in the human group is a conversation partner of the speaker at the current time based on the second multimodal features using a deep learning method. Optionally, the coding and decoding are performed using a Transformer method.

According to the device and method for detecting interlocutors in a multi-human computer interaction scene of the embodiment of the present invention, it is possible to predict interlocutors in a multi-human computer interaction scene where the number of people changes from moment to moment. Specifically, by associating feature information of different dimensions using a multimodal fusion module, it is possible to extract information that is useful for determining interlocutors. In addition, by making predictions using a deep learning method without requiring complex artificial feature extraction processing, it is possible to effectively improve prediction efficiency in the usage process.

FIG. 2 is a schematic diagram of a multi-human and robot interaction scene according to an embodiment of the present invention. 1 is a module schematic diagram of an apparatus for detecting interlocutors in a multi-human computer interaction scene according to an embodiment of the present invention; 1 is a flowchart of a method for detecting interlocutors in a multi-human computer interaction scene according to an embodiment of the present invention; FIG. 2 is a schematic diagram of a selectable model architecture of a partner recognition module according to an embodiment of the present invention.

In order to better understand the purpose, structure and function of the present invention, the following provides a more detailed description of the apparatus and method for interlocutor detection in a multi-human computer interaction scene according to an embodiment of the present invention with reference to the drawings.

Figure 1 shows a schematic diagram of an example of a multi-human-robot interaction scene. In Figure 1, squares represent objects in the scene, isosceles triangles represent people in the scene, the vertex angles may be used to recognize the orientation of people, and circles with R represent robots. As shown in Figure 1, the human-computer interaction in the scene involves four people and one robot. As can be understood by those skilled in the art, Figure 1 is merely an example of a multi-human-computer interaction scene, and the number of people and robots actually participating in the human-computer interaction should not be limited thereto and may change from time to time.

2 shows a functional module diagram of an apparatus for detecting interlocutors in a multi-human computer interaction scene according to an embodiment of the present invention. As shown in FIG. 2, the apparatus includes an audio-video collection module 110, a text generation module 120, a face processing module 130, a text feature extraction module 140, an audio feature extraction module 150, a face feature extraction module 160, a speaker detection module 170 and an interlocutor recognition module 180.

Here, the audio/video collection module 110 can, for example, use a camera to collect time-stamped video frame data in real time (wherein the video frame data includes, for example, video frames of color images), and can use a micro to collect time-stamped audio frame data. In some embodiments, as shown in FIG. 2, the video frame data and the audio frame data may be stored in chronological order in the video frame database 101 or the audio frame database 102, respectively. Also, the multiple video frames included in the video frame data and the multiple audio frames included in the audio frame data are synchronized according to the timestamps. In other words, video and audio collected at the same time should be synchronized according to the timestamps.

The text generation module 120 can generate time-stamped text information corresponding to different hierarchies such as word level, sentence level, dialogue topic level, etc. based on the audio frame data, for example by speech recognition. In some embodiments, the text information may be stored in a text database 104, as shown in FIG. 2.

The face processing module 130 can use machine vision methods to detect faces in video frames, e.g., color images, and track the same person in multiple video frames to obtain face sequence data. In some embodiments, the face sequence data can be stored in a face database 103, as shown in FIG. 2. Here, the multiple video frames can be consecutive video frames, e.g., multiple video frames captured by a camera within a certain time period. However, the multiple video frames can also be non-consecutive video frames, which can effectively realize person tracking even when the person leaves the scene and returns.

The text feature extraction module 140 can input the time-stamped text information corresponding to different hierarchies into a natural language deep learning network to extract time-stamped text semantic features. In some implementations, after obtaining the text information, the text may be regarded as a word sequence and coded, for example, using a GloVe word encoder, to obtain a text semantic feature vector of a certain length (e.g., 128 dimensions).

The audio feature extraction module 150 may input the time-stamped audio frame data into a deep learning network to extract time-stamped voice audio features. For example, the audio frame data may be first divided into overlapping audio segments, and feature extraction may be performed on the audio segments to obtain Mel-Frequency Cepstral Coefficients (MFCCs) as input for further audio feature extraction. For example, the MFCCs may be input into a deep learning network to generate a voice audio feature vector of a particular length (e.g., 128 dimensions) based on the input MFCCs.

The facial feature extraction module 160 can extract time-stamped facial features of a person by inputting the facial sequence data into a deep learning network. Here, the facial features of a person may include time series and spatial features of the person's face. For example, the facial sequence data of each person can be regarded as one image block sequence, and the image block sequence can be converted into a visual feature code by a deep learning network, and then the visual feature code and the position code can be added to obtain the corresponding facial feature of the person. In addition, the facial feature of a person can be represented as a feature vector of a certain length (e.g., 128 dimensions).

The speaker detection module 170 may use a machine learning or deep learning method to recognize a speaker in a human group at a current time based on the facial features and the voice audio features of the person in the face sequence data, and obtain information about the speaker at the current time. In some embodiments, the information about the speaker at the current time may be stored in a speaker database 105, as shown in FIG. 2. For example, the speaker database 105 may store the speaker information according to a timestamp.

The interlocutor recognition module 180 can recognize an interlocutor of the speaker in the human group at the current time based on scene features, the text semantic features, the voice audio features, and the facial features of the person in the face sequence data through machine learning or deep learning methods, and detect whether the interlocutor of the speaker at the current time is a robot. In some embodiments, the interlocutor information may be stored in an interlocutor database 106, as shown in FIG. 2.

Specifically, as shown in FIG. 2, the audio/video collection module 110 may include a video collection module 111 and an audio collection module 112. Here, the video collection module 111 may collect video frames, for example color images, with time stamps in real time using, for example, a camera. The audio collection module 112 may collect audio frame data with time stamps using, for example, a micro. In addition, the video frame database 101 may be used to store the video frame data with time stamps in a time series in order to be called by other modules, for example, the face processing module 130, and the audio frame database 102 may be used to store the audio frame data with time stamps in a time series in order to be called by other modules, for example, the text generation module 120, the audio feature extraction module 150, etc.

Specifically, as shown in FIG. 2, the face processing module 130 may include a face detection module 131 and a face tracking module 132. Here, the face detection module 131 may detect faces in video frames included in the video frame data using a deep learning method, and assign a unique fixed identifier to the same face detected from two or more video frames to represent the person, and the face tracking module 132 may track the same person in multiple video frames based on the detection result output by the face detection module 131, and obtain time-stamped face sequence data. By assigning a unique fixed identifier to the same face, even if the person disappears in the scene field and then reappears, the person can be represented by the original id. In some embodiments, the face database 103 may be used to store the time-stamped face sequence data, for example to be called by other modules of the face feature extraction module 160, as shown in FIG. 2.

Specifically, as shown in FIG. 2, the speaker detection module 170 may include a first multimodal fusion module 171 and a speaking state detection module 172. The first multimodal fusion module 171 may fuse the face features and voice audio features of the person according to a time stamp based on the face sequence data into a first multimodal feature, and the speaking state detection module 172 may input the first multimodal feature into a deep learning network and predict the speaking state of each person in a human group at a current time one by one to determine the speaker and corresponding speaker information at the current time. In some embodiments, the speaker database 105 may be used to store the speaker information at the current time, for example, to be called by other modules of the speaking partner recognition module 180, as shown in FIG. 2.

In some embodiments, the combination method may also incorporate a person's facial features and voice audio features into the first multimodal feature. For example, if the person's facial features and voice audio features are both 128-dimensional vectors, the first multimodal feature obtained by feature combination is a 256-dimensional vector.

Specifically, as shown in FIG. 2, the interlocutor recognition module 180 may include a second multimodal fusion module 181 and an interlocutor detection module 182. The second multimodal fusion module 181 may fuse the face features of the person, the voice audio features, the text semantic features, and the scene features from the scene database 107 into a second multimodal feature according to a time stamp based on the face sequence data, and the interlocutor detection module 182 may input the second multimodal features into a deep learning network, predict one by one whether each person in the human group and each robot is an interlocutor of the speaker at the current time, and determine the interlocutor information at the current time accordingly. In some embodiments, as shown in FIG. 2, the interlocutor database 106 may be used to store the interlocutor information at the current time, for example, to be called by other modules of the scene database 107. Alternatively, the interlocutor information at the current time may be directly output as a result.

Also, as shown in FIG. 2, the scene database 107 may store speaker and interlocutor information from previous times for use by the interlocutor recognition module 180.

Specifically, as shown in FIG. 2, the text generation module 120 may include a speech recognition module 121. The speech recognition module 121 may perform speech recognition based on the audio frame data and generate time-stamped text information corresponding to different hierarchies such as a word level, a sentence level, and a dialogue topic level. In some embodiments, as shown in FIG. 2, the text database 104 may be used to hierarchically store the time-stamped text information in a chronological order, for example, to be called by other modules of the text feature extraction module 140.

Figure 3 shows a flowchart of a method for interlocutor detection in a multi-human computer interaction scene according to an embodiment of the present invention. As shown in Figure 3, the method may include steps S1 to S5.

In step S1, the audio/video collection module 110 collects time-stamped video frame data in real time using, for example, a camera, and collects time-stamped audio frame data using a microcomputer. Here, a plurality of video frames included in the video frame data and a plurality of audio frames included in the audio frame data may be stored in chronological order in a video frame database or an audio frame database. In this way, video and audio collected at the same time can be synchronized according to the timestamps.

Specifically, the video frame at the current time may be a color image acquired in real time during actual operation. For example, in a robot system using a Robot Operating System (ROS), color images collected by a monocular camera may be published in the form of a ROS topic, so that color images can be acquired in real time by subscribing to the image topic. Audio information collected by the array micro may also be published in the form of a ROS topic, so that audio information can be acquired in real time by subscribing to the audio topic.

In step S2, the text generation module 120 performs real-time speech recognition on the audio frame data to generate time-stamped text information at different levels, such as the word level, sentence level, and dialogue topic level, and the text feature extraction module 140 extracts text semantic features from the time-stamped text information. In some embodiments, the text information may be stored in the text database 104.

In step S3, the face processing module 130 detects a face in the video frame data using a machine vision method, tracks the same person in multiple video frames to obtain face sequence data, the face feature extraction module 160 extracts the person's face features from the face sequence data, and the audio feature extraction module 150 extracts voice audio features from the audio frame data.

In one exemplary embodiment, YOLO may be used for face detection, and Deep SORT may be used for multi-target tracking. As a result of tracking, an ID is assigned to each person, and each person's ID is unique and fixed throughout the entire process.

In step S4, the speaker detection module 170 uses a machine learning or deep learning method to recognize the speaker in the human group at the current time based on the person's facial features and the voice audio features, and obtains information about the speaker at the current time.

Specifically, step S4 may further include a step of fusion-coding the facial features and the voice audio features of the person according to the time stamp based on the face sequence data, i.e., performing multimodal feature fusion to obtain a first multimodal feature, and a step of predicting a speaker at a current time in a human group based on the first multimodal feature using a deep learning method.

In step S5, the conversation partner recognition module 180 uses a machine learning or deep learning method to recognize the conversation partner of the speaker in the human group at the current time based on the scene features, the text semantic features, the voice audio features, and the facial features of the person, and detects whether the conversation partner of the speaker at the current time is a robot.

Specifically, step S5 may further include a step of fusion-coding the scene features, the text semantic features, the voice audio features and the facial features of the person according to the timestamp based on the face sequence data, i.e., performing multimodal feature fusion to obtain second multimodal features, and a step of predicting, one by one, the probability that each person in the human group is a conversation partner of the speaker at the current time based on the second multimodal features using a deep learning method.

In one exemplary embodiment, a deep learning method for predicting based on the first/second multimodal features may be implemented using a Transformer model, which is well known to those skilled in the art. In general, a Transformer model includes an input, an encoder, a decoder, and an output.

Here, the input of the Transformer model is a coded sequence. For example, for video frame data, it is common to block frame images into an image sequence, with the collection time of each frame image being an element of the image sequence. For text information, characters are first coded into a word sequence, and then each word in the word sequence is word coded to generate a text code sequence. Audio frame data also needs to be coded into an audio sequence before it can be used as input for the Transformer model.

The encoder in the Transformer model is mainly composed of six-layer code modules. Each code module mainly includes one multi-head self-attention mechanism layer and one fully connected feed-forward layer, and residual connection and normalization are added to both. Here, the multi-head self-attention mechanism layer takes the sequence code of the previous layer as input, and generates q, k, and v values in the search key value triad (query, key, value) through the fully connected layer. The q, k, and v values may all be feature vectors with a length of 64. Between sequences, attention is calculated for each k using each q, and the calculation formula is:
where d _k represents the length of the feature vector and is equal to 64.

Similarly, the decoder in the Transformer model is mainly composed of six-layered decoding modules. Each decoding module includes two multi-head self-attention mechanism layers and one fully-connected forward propagation layer. The input of the decoder includes the output of the encoder and the previous output of the decoder. In particular, the output of the decoder is the output of the Transformer model.

Below, we briefly explain the application of the Transformer model in an embodiment of the present invention, taking as an example predicting interlocutor based on a second multimodal feature.

As shown in FIG. 4, in order to effectively recognize the speaker's interlocutors, the input data includes the speaker's face image sequence, the other person's face image sequence, the audio frame data of the corresponding time period, and the text information of the corresponding time period. First, feature extraction is performed on the image information, audio information, and text information, respectively, to obtain the corresponding person's face feature vector, voice audio feature vector, and text semantic feature vector; then, in the multimodal fusion module, all feature vectors are combined to realize multimodal fusion, thereby obtaining second multimodal features corresponding to the speaker and each other person; then, the second multimodal features obtained by fusion are coded by the Transformer encoder to obtain second multimodal code feature vectors of the speaker and each other person; finally, the second multimodal code feature vector is transmitted into the Transformer decoder to predict the probability that each other person is the speaker's interlocutor. Here, the prediction by the Transformer decoder may be an order prediction. For example, the probability that the robot is a conversation partner may be predicted first, and then the probability that each other person is a conversation partner may be predicted. In some embodiments, as shown in FIG. 4, the result of the conversation partner prediction for the previous person may be re-input to the Transformer decoder, and used as the input when the Transformer decoder performs conversation partner prediction for the next person. In other words, when performing conversation partner recognition, the Transformer decoder predicts each person in the human group other than the speaker one by one. The first output result of the Transformer decoder is the probability that the robot is a conversation partner, and the subsequent output results are the probabilities of each other person in turn. If the probability indicated by the output result of the Transformer decoder is greater than a preset threshold, the corresponding robot or person is considered to be a conversation partner. For example, if the first output result is greater than the preset threshold, it indicates that the robot is a conversation partner of the speaker at the current time.

As will be understood, the present invention has been described through several embodiments, and those skilled in the art will recognize that various modifications or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to suit particular situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present invention are within the scope of protection of the present invention.

Claims (9)

1. An apparatus for interlocutor detection in a multi-human computer interaction scene involving a human group including multiple humans and at least one robot, comprising:
an audio/video collection module (110) for collecting time-stamped video frame data and time-stamped audio frame data in real time, wherein a plurality of video frames included in the video frame data and a plurality of audio frames included in the audio frame data are synchronized according to the timestamps;
a text generation module (120) for generating time-stamped text information based on the audio frame data;
a face processing module (130) for detecting a face in each video frame included in the video frame data using a machine vision method and tracking the same person in multiple video frames to obtain face sequence data;
a text feature extraction module (140) for extracting text semantic features from the time-stamped text information by machine learning or deep learning methods;
an audio feature extraction module (150) for extracting voice audio features from the audio frame data using machine learning or deep learning methods;
a facial feature extraction module (160) for extracting, by machine learning or deep learning methods, facial features of a person from the face sequence data, including temporal and spatial features of the person's face;
a speaker detection module (170) for recognizing a speaker at a current time in the human group based on the facial features and the voice audio features of the person in the face sequence data by a machine learning or deep learning method, and obtaining information of the speaker at the current time;
a partner recognition module (180) for recognizing a partner of a speaker in the human group at the current time based on scene features, the text semantic features, the voice audio features and the facial features of the person in the face sequence data by a machine learning or deep learning method, and detecting whether the partner of the speaker at the current time is a robot or not, wherein the scene features include speaker information and partner information of a previous time ;
The interlocutor recognition module (180)
a second multimodal fusion module (181) for fusing the person's facial features, the voice audio features, the text semantic features, and the scene features into second multimodal features according to time stamps based on the face sequence data;
and a partner detection module (182) for inputting the second multimodal features into a deep learning network, predicting one by one whether each person in the human group and each robot is a partner of a speaker at the current time, and determining partner information at the current time accordingly . The audio-video acquisition module (110)
a video acquisition module (111) for acquiring said time-stamped video frame data in real time using a camera;
an audio capture module (112) for capturing said time-stamped audio frame data using a microphone;
and/or a video frame database (101) for storing said video frame data in chronological order;
2. The apparatus of claim 1, further comprising: an audio frame database (102) for storing said audio frame data in chronological order. The face processing module (130)
a face detection module (131) for detecting faces in video frames included in the video frame data using a deep learning method and assigning a unique fixed identifier to a face detected in two or more video frames to represent the person;
a face tracking module (132) for tracking the same person in the plurality of video frames based on the detection result output by the face detection module (131) to obtain time-stamped face sequence data;
2. The apparatus of claim 1, further comprising a face database (103) for storing said time-stamped face sequence data. The speaker detection module (170)
a first multimodal fusion module (171) for fusing the facial features of the person and the voice audio features into a first multimodal feature according to a time stamp based on the face sequence data;
a speaking state detection module (172) for inputting the first multi-modal features into a deep learning network and predicting the speaking state of each person in the human group at a current time, one by one, to determine a speaker and corresponding speaker information at the current time;
2. The apparatus of claim 1, further comprising a speaker database (105) for storing information of said speakers according to time stamps. a conversation partner database (106) for storing information of said conversation partners according to time stamps;
The apparatus of claim 1 further comprising: a scene database (107) for storing said scene features. The text generation module (120) includes a speech recognition module (121) for generating text information with time stamps corresponding to a plurality of hierarchical levels based on the audio frame data, the plurality of hierarchical levels including a word level, a sentence level, and a dialogue topic level;
2. The apparatus of claim 1, further comprising a text database (104) for storing said text information in a hierarchical time sequence. A method for interlocutor detection in a multi-human computer interaction scene involving a human group including multiple humans and at least one robot, comprising:
A step S1 of collecting time-stamped video frame data and time-stamped audio frame data in real time by an audio/video collection module (110), wherein a plurality of video frames included in the video frame data and a plurality of audio frames included in the audio frame data are synchronized according to the timestamps;
A step S2 of generating time-stamped text information based on the audio frame data in real time by a text generation module (120) and extracting text semantic features from the time-stamped text information by a text feature extraction module (140);
Step S3: detecting a face in each video frame included in the video frame data by a face processing module (130) using a machine vision method, tracking the same person in multiple video frames to obtain face sequence data, extracting face features of the person from the face sequence data by a face feature extraction module (160), and extracting voice audio features from the audio frame data by an audio feature extraction module (150);
Step S4: Recognizing a speaker in the human group at a current time based on the person's facial features and the voice audio features by a speaker detection module (170) using a machine learning or deep learning method to obtain information of the speaker at the current time;
Step S5 of recognizing a speaker's conversation partner in the human group at the current time based on scene features, the text semantic features, the voice audio features and the person's facial features by a conversation partner recognition module (180) using a machine learning or deep learning method, and detecting whether the speaker's conversation partner at the current time is a robot or not, wherein the scene features include speaker information and conversation partner information at a previous time ;
The step S5 is
co-coding the scene features, the text semantic features, the voice audio features and the person's facial features according to a time stamp based on the face sequence data to obtain a second multi-modal feature;
and predicting, one by one, the probability that each person in the human group is a conversation partner of the speaker at the current time based on the second multimodal feature using a deep learning method . In step S1,
The video frame data is published in the form of a ROS topic, and the video frame data is obtained in real time by subscribing to an image topic;
The audio frame data is published in the form of a ROS topic, and the audio frame data is obtained in real time by subscribing to the audio topic;
8. The method according to claim 7, wherein in step S2, face detection is performed using YOLO, and multiple target tracking is performed using a Deep SORT model, and an ID is assigned to each person as a result of the tracking, and the ID of each person is unique and fixed throughout the entire process. The step S4 is
co-coding the facial features and the voice audio features of the person according to a time stamp based on the face sequence data to obtain a first multi-modal feature;
and predicting, with a deep learning method, a speaker at a current time in the human group based on the first multi - modal features.