JP2023117068A - Speech recognition device, speech recognition method, speech recognition program, speech recognition system - Google Patents

Abstract

The present invention provides a speech recognition device, a speech recognition method, a speech recognition program, and a speech recognition system that display the utterances of a speaker who is a sound source identified from an image included in video data. In a voice recognition system in which an information processing device, an imaging device, and a display device are connected via a network or the like, a voice recognition processing unit 230 of an information processing terminal 200A, which is the information processing device, A notable speaker determination unit 247 that determines a notable speaker based on a face image of a person detected from an image indicated by image data included in the data; and audio data of a notable speaker from among audio data included in the video data. The utterance content recognition result output unit 233 displays the text data converted from the utterance content recognition result output unit 233 on a display device. [Selection diagram] Figure 5

Description

The present invention relates to a speech recognition device, a speech recognition method, a speech recognition program, and a speech recognition system.

In recent years, a technique is known in which a speaker who is a sound source is specified from an image, the voice uttered by the specified speaker is converted into a character image, and the character image is displayed on a display unit. Specifically, for example, when a person specified in an image is moving his mouth, a system is known that converts the voice of the person into text and displays the text.

With the above-described conventional technology, when there are multiple persons moving their mouths in the image, it is not possible to select a speaker of interest. For this reason, when focusing on a specific person, there is a possibility that the utterance content of the focused person may not be displayed appropriately in the conventional technology.

The technology disclosed has been made in view of the above circumstances, and aims to display the utterance content of a specific speaker.

The disclosed technology includes a target speaker determination unit that determines a target speaker based on a face image of a person detected from an image represented by image data included in video data, and voice data included in the video data, and a speech recognition result output unit for displaying text data converted from the speech data of the speaker of interest on a display device.

It is possible to display the utterance contents of a specific speaker.

1 is a diagram showing an example of a speech recognition system according to a first embodiment; FIG. FIG. 2 is a first diagram illustrating an overview of speech recognition according to the first embodiment; FIG. FIG. 10 is a diagram illustrating a case where the speech recognition system is applied to smart glasses; FIG. 4 is a second diagram for explaining an overview of speech recognition according to the first embodiment; It is a figure explaining the function of smart glasses. 4 is a first flow chart explaining the operation of the smart glasses of the first embodiment; 4 is a second flowchart for explaining the operation of the smart glasses of the first embodiment; FIG. 4 is a diagram explaining determination of a speaker of interest in the first embodiment; 9 is a third flowchart for explaining the operation of the smart glasses of the first embodiment; FIG. FIG. 4 is a first diagram for explaining an example of operation of smart glasses; FIG. 11 is a second diagram for explaining an example of operation of smart glasses; FIG. 11 is a third diagram illustrating an example of operation of smart glasses; FIG. 14 is a fourth diagram illustrating an example of operation of smart glasses; FIG. 11 is a fifth diagram for explaining an example of operation of smart glasses; FIG. 11 is a sixth diagram explaining an example of operation of smart glasses; FIG. 11B is a seventh diagram illustrating an example of operation of smart glasses. It is a figure explaining recognition of the speech content of 1st embodiment. It is a figure which shows an example of the translation system of 2nd embodiment. It is a figure which shows an example of the utterance content recording system of 3rd embodiment.

(First embodiment)
A first embodiment will be described below with reference to the drawings. FIG. 1 is a diagram showing an example of the speech recognition system of the first embodiment.

The speech recognition system 100 of this embodiment includes an information processing device 200, an imaging device 300, and a display device 400. The information processing device 200, the imaging device 300, and the display device 400 are connected via a network or the like. ing.

In the speech recognition system 100 of this embodiment, the information processing device 200 has a speech recognition processing section 230 .

The imaging device 300 of this embodiment acquires video data and transmits it to the information processing device 200 . Video data includes image data (moving image data) and audio data.

In this embodiment, the video data may be captured by the user of the speech recognition system 100 using the imaging device 300 and transmitted to the information processing device 200 . Therefore, the image data included in the video data includes the speaker that the user pays attention to.

The information processing apparatus 200 of the present embodiment uses the speech recognition processing unit 230 to identify a speaker that the user of the speech recognition system 100 pays attention to based on the image data included in the video data acquired from the imaging device 300 . Then, the information processing device 200 converts only the voice data of the utterer of interest to the user into text data by the voice recognition processing unit 230, and causes the display device 400 to display the text data. Note that the display device 400 may be, for example, a display included in the information processing device 200 .

As described above, in the present embodiment, the user identifies the speaker of interest from the image data, and converts only the voice data of the identified speaker into text data and outputs the text data. Therefore, according to the present embodiment, it is possible to display the utterance content of a specific utterer focused on by the user.

An outline of speech recognition by the information processing apparatus 200 of the present embodiment will be described below with reference to FIG. FIG. 2 is a first diagram explaining an outline of speech recognition according to the first embodiment.

An image 21 shown in FIG. 2 is an example of an image represented by image data acquired by the imaging device 300 .

The image 21 includes an image of person A and an image of person B. FIG. The information processing apparatus 200 of the present embodiment identifies an image of a person whose face image is positioned close to the center of the image 21 among the images of the person included in the image 21 as a person of interest.

In the example of FIG. 2, the facial image of person A is closer to the center of image 21 than the facial image of person B is. Therefore, in FIG. 2, person A is identified as the speaker of interest.

Note that the center of the image may be the position where the diagonal lines of the rectangle indicated by the image intersect. Also, in the following description, a speaker to be noted may be referred to as a speaker of interest. A speaker of interest is, in other words, a specific speaker that the user of the speech recognition system 100 is paying attention to.

In the present embodiment, when person A is identified as the speaker of interest, the person A, who is the speaker of interest, is identified using a moving image showing the movement of the lips of person A and audio data acquired by the imaging device 300 . is converted into text data 23 and displayed. Therefore, according to this embodiment, it is possible to convert the utterance content of the speaker of interest into text data with high accuracy.

At this time, in the present embodiment, the image 21 and the text data 23 may be superimposed and displayed.

Next, a case where the speech recognition system 100 of the present embodiment is applied to smart glasses will be described with reference to FIG.

FIG. 3 is a diagram illustrating a case where the speech recognition system is applied to smart glasses. In FIG. 3, the speech recognition system 100 is described as smart glasses 100A.

4 is a diagram illustrating an example of a hardware configuration of the smart glasses of FIG. 3; FIG. The smart glasses 100</b>A of this embodiment are glasses-type wearable terminals including an glasses-type display device 300</b>A, an information processing terminal 200</b>A, and a cable 150 . In the example of FIG. 3, the glasses-type display device 300A and the information processing terminal 200A are connected by the cable 150, but the present invention is not limited to this. The glasses-type display device 300A and the information processing terminal 200A may perform wireless communication.

The glasses-type display device 300</b>A includes a camera (imaging device) 110 , a microphone (sound collecting device) 120 , a display (display device) 130 and an operation member 140 . That is, the glasses-type display device 300A includes an imaging device and a display device.

The camera 110 acquires image data of the line-of-sight direction of the wearer wearing the smart glasses 100A. The microphone 120 acquires audio data around the smart glasses 100A. Display 130 displays text data output from information processing terminal 200A. The display 130 of this embodiment may be an optical see-through display. The operation member 140 may be a physical button or the like, and various operations are performed on the glasses-type display device 300A.

Also, in the present embodiment, the camera 110 and the microphone 120 are provided separately, but the present invention is not limited to this. The microphone 120 of this embodiment may be built into the camera 110 . In this case, the camera 110 acquires video data including image data and audio data.

Cable 150 transmits image data acquired by camera 110 and audio data acquired by microphone 120 to information processing terminal 200A. Also, the cable 150 transmits various types of information from the information processing terminal 200A to the glasses-type display device 300A.

Information processing terminal 200A includes information input/output interface (I/F) 201 , memory 202 , operation device 203 , storage 204 , power supply 205 , CPU (Central Processing Unit) 206 and network interface (I/F) 207 .

An information input/output interface (I/F) 201 is an interface for transmitting and receiving various data between the information processing terminal 200A and the glasses-type display device 300A. A memory 202 stores temporary information such as audio data and image data (moving image data). The operation device 203 is used by the wearer of the smart glasses 100A to perform various operations such as execution of applications and power on/off. The operation device 203 may be realized by, for example, a touch panel or the like.

The storage 204 stores various models and the like, which will be described later. The power supply 205 supplies power to each device of the smart glasses 100A. The CPU 206 executes various processes and controls the operation of the smart glasses 100A as a whole.

200 A of information processing terminals implement|achieve the function of the speech-recognition process part 230 because CPU206 reads and runs the program stored in the storage 204 grade|etc.,.

A network interface 207 is an interface for accessing a communication network.

Note that the smart glasses 100A shown in FIG. 3 include the glasses-type display device 300A and the information processing terminal 200A, but are not limited to this. In the smart glasses 100A, the glasses-type display device 300A may have all the configurations of the information processing terminal 200A.

Next, an outline of speech recognition by the smart glasses 100A will be described with reference to FIG. FIG. 4 is a second diagram illustrating an overview of speech recognition according to the first embodiment.

In the smart glasses 100A of this embodiment, the information processing terminal 200A identifies the speaker of interest based on the image data acquired from the camera 110 . Then, the information processing terminal 200A causes the display 130 to display the text data converted from the speech data of the speaker of interest acquired from the microphone 120 .

In the example of FIG. 4, the wearer P of the smart glasses 100A is looking at the person A. In the example of FIG. In addition, in the example of FIG. 4, the person A and the person B are present in the line-of-sight direction of the wearer P. As shown in FIG. In this case, the image captured by the camera 110 of the smart glasses 100A is an image in which the image of the person A that the wearer P pays attention to is positioned at the center.

Therefore, in the smart glasses 100A, the person A is identified as the speaker of interest, only the voice data of the speaker of interest is converted into text data, and only the text data is displayed on the display 130 .

In the present embodiment, by applying the voice recognition system 100 to the smart glasses 100A in this way, the wearer P of the smart glasses 100A simply faces the person of interest, and the person of interest of the wearer P becomes the person of interest. identified by the speaker.

Further, in this embodiment, the display 130 of the smart glasses 100A is of an optical see-through type. Therefore, in the present embodiment, the text data 23 can be visually recognized by the wearer P without obstructing the wearer's P view.

Note that the display 130 may not be of the optical see-through type, and may display an image indicated by the image data acquired by the camera 110 and the text data 23 in a superimposed manner.

Also, the smart glasses 100A may be, for example, a retinal scanning spectacles-type projection device. In this case, the display 130 is unnecessary, and the text data 23 may be directly projected onto the retina of the wearer P by the optical system.

Next, with reference to FIG. 5, functions of the smart glasses 100A of the present embodiment will be described. FIG. 5 is a diagram illustrating functions of smart glasses. Specifically, FIG. 5 shows functions of an information processing terminal 200A included in the smart glasses 100A.

200 A of information processing terminals of this embodiment have the speech-recognition process part 230. FIG. The speech recognition processing unit 230 includes a video input unit 231, a speech input unit 232, a speaker-of-interest identification unit 240, a lip feature amount acquisition unit 250, an acoustic feature amount acquisition unit 260, a person identification unit 270, and a multimodal recognition unit 280 (first first speech recognition unit), speech recognition unit 290 (second speech recognition unit), and speech content recognition result output unit 233 .

The video input unit 231 acquires image data (moving image data) captured by the camera 110 . The voice input unit 232 acquires voice data collected by the microphone 120 . At this time, the audio input unit 232 may acquire the audio data as monaural uncompressed data sampled under a predetermined condition.

The utterance content recognition result output unit 233 causes the display 130 to display the text data that is the recognition result of the utterance content by the multimodal recognition unit 280 and the text data that is the recognition result of the utterance content by the speech recognition unit 290 .

The attention speaker identification unit 240 identifies the attention speaker from the video data acquired by the video input unit 231 . The attention speaker identification unit 240 includes an image conversion unit 241, a face area recognition unit 242, a face area detection model 243, a face position determination unit 244, a lip area extraction unit 245, a facial feature point estimation model 246, and an attention speaker determination unit 247. have

The image conversion unit 241 converts moving image data into time-series frame images. Note that the image conversion unit 241 may convert RGB image data into grayscale image data or may convert the number of pixels in order to speed up processing.

The face region recognition unit 242 uses the face region detection model 243 to recognize regions (face regions) containing face images in the acquired time-series frame images. The face area detection model 243 is a model for detecting a face image from an image, and is a model in which a neural network is trained in advance using a large amount of data. The face image detected here is the face image of a person who is a candidate for the speaker of interest.

The face position determination unit 244 determines the position of the face area in the image indicated by the image data acquired by the camera 110, and acquires information indicating the position of the face area.

A lip region extraction unit 245 detects a lip region including an image of the lip portion in the face image using the facial feature point estimation model 246, and extracts an image within the lip region from the face image within the face region. .

The face feature point estimation model 246 is a model that acquires the coordinates of the contours of the eyes, nose, and lips from the face image and detects the coordinates around the lips.

In this embodiment, the lip region extraction unit 245 extracts the image of the lip region, but is not limited to this. For example, in the face image, if the image of the lip portion is hidden by a person's hand or the like, the lip region extraction unit 245 extracts the image of the region corresponding to the facial parts such as the eyes and nose. good too. Areas corresponding to facial parts such as eyes and nose may be detected by facial feature point estimation model 246 .

The attention speaker determination unit 247 determines the attention speaker based on the position of the person's face area in the image and the movement of the mouth indicated by the image (moving image) in the lip area. The details of the processing of the attention speaker identification unit 240 will be described later.

The lip feature quantity acquisition unit 250 of the present embodiment acquires a lip feature quantity indicating the movement of the mouth from the image within the lip region in the image acquired by the camera 110 .

The lip feature quantity acquisition unit 250 has a lip pixel number conversion unit 251 , a lip feature quantity calculation unit 252 , and a lip feature quantity calculation model 253 .

A lip pixel number conversion unit 251 converts the image in the extracted lip region into an image of a predetermined size. In other words, the lip pixel number conversion unit 251 enlarges or reduces the image in the lip region, which varies in size depending on the distance between the camera 110 and the person being photographed, so that the image has a uniform size. .

The lip feature amount calculation unit 252 uses the lip feature amount calculation model 253 to calculate the lip feature amount. Specifically, the lip feature amount calculation unit 252 inputs the moving image data showing the time-series images in the lip area whose size has been changed to the lip feature amount calculation model 253, and recognizes the utterance content. To calculate the effective lip feature amount. A lip feature amount is a multidimensional vector that is output from the lip feature amount calculation model 253 by inputting moving image data in the lip region to the lip feature amount calculation model 253 .

The acoustic feature amount acquisition unit 260 detects an utterance period, which is a period in which a person speaks, from the audio data acquired by the audio input unit 232, and acquires an acoustic feature amount of the audio data in the utterance period.

The acoustic feature quantity acquisition unit 260 has a voice utterance segment detection unit 261 , a voice utterance segment detection model 262 and an acoustic feature quantity calculation unit 263 .

The voice utterance period detection unit 261 uses the voice utterance period detection model 262 to detect the utterance period from the input voice data.

The acoustic feature amount calculation unit 263 calculates an acoustic feature amount from the speech waveform of the section detected as the utterance section. The acoustic features may be, for example, Mel frequency cepstrum coefficients (MFCC), logarithmic Mel filter bank features (FBANK), logarithmic Mel filters, and the like.

The person identification unit 270 acquires information for identifying the person who has spoken from the acoustic feature amount. The person identification unit 270 has a speaker embedded information calculation unit 271 , a speaker embedded information calculation model 272 and an off-screen speaker estimation unit 273 .

The speaker-embedded information calculation unit 271 uses the speaker-embedded information calculation model 272 to calculate speaker-embedded information (embedding) representing the voice quality of the speaker. The speaker embedding information is information for specifying the speaker, and may be, for example, a feature quantity with a certain number of dimensions extracted by a method such as i-vector, d-vector, or x-vector.

The out-of-screen speaker estimation unit 273 uses the speaker embedded information when the orientation of the face of the wearer P of the smart glasses 100A changes and the image of the speaker of interest is no longer included in the image acquired by the camera 110. to estimate the speaker. The details of the processing of the off-screen speaker estimation unit 273 will be described later.

The multimodal recognition unit 280 recognizes the utterance content of the target speaker using the lip feature amount and the acoustic feature amount. The multimodal recognition unit 280 has a feature amount integration unit 281 , a multimodal speech content recognition unit 282 , and a multimodal speech content recognition model 283 .

The feature amount integration unit 281 integrates the acoustic feature amount acquired by the acoustic feature amount acquisition unit 260 and the lip feature amount acquired by the lip feature amount acquisition unit 250 to obtain a multimodal feature amount. A multimodal feature amount is a feature amount including a plurality of types of feature amounts. More specifically, the multimodal features include acoustic features and lip features.

A multimodal speech content recognition unit 282 recognizes speech content using a multimodal speech content recognition model 283 . More specifically, the multimodal utterance content recognition unit 282 of the present embodiment recognizes utterance content using acoustic features extracted from voice data and lip features extracted from video data.

The speech recognition unit 290 recognizes the speech content from the speech data acquired by the speech input unit 232 based on the acoustic feature quantity acquired by the acoustic feature quantity acquisition unit 260 . The voice recognition unit 290 has a voice utterance content recognition unit 291 and a voice utterance content recognition model 292 .

The voice utterance content recognition unit 291 recognizes the utterance content using the acoustic feature amount when the lip feature amount of the person set as the speaker of interest is not acquired. Specifically, the voice utterance content recognition unit 291 uses the voice utterance content recognition model 292 to recognize the utterance content based on the voice data, and passes the recognition result to the utterance content recognition result output unit 233 .

In this embodiment, if the image extracted by the lip area extraction unit 245 is an area corresponding to an image of a facial part other than the mouth, the lip feature amount may not be calculated.

In the present embodiment, the face region detection model 243, face feature point estimation model 246, voice utterance section detection model 262, and speaker embedded information calculation model 272 may be models using known techniques.

Next, operation of the smart glasses 100A of the present embodiment will be described with reference to FIG. FIG. 6 is a first flow chart explaining the operation of the smart glasses of the first embodiment.

The process of FIG. 6 is executed, for example, when the wearer P of the smart glasses 100A performs an operation of instructing the start of the process of recognizing the utterance content of the speaker of interest.

In the smart glasses 100A of the present embodiment, the information processing terminal 200A acquires image data (moving image data) and audio data using the video input unit 231 and the audio input unit 232 (step S601).

Subsequently, the information processing terminal 200A performs processing for detecting an utterance period using the voice utterance period detection unit 261 (step S602).

In step S602, when the speech period is not detected, the information processing terminal 200A returns to step S601.

When the speech period is detected in step S602, the information processing terminal 200A repeats the processes from step S605 to step S607 for the number of persons whose face images are detected (step S604).

200 A of information processing terminals detect the face area containing a face image in the image which the image data which the video input part 231 acquired by the face area recognition part 242 (step S605).

Subsequently, the information processing terminal 200A detects the lip area from the face area using the lip area extraction unit 245 (step S606). If the lip region is not detected in the face region, the lip region extraction unit 245 may detect regions corresponding to images of facial parts (eyes, nose, etc.) other than the mouth. In other words, the lip region extraction unit 245 should detect a region corresponding to a partial image of the face from the face region.

Further, in the present embodiment, detecting the lip region may be synonymous with extracting the lip image within the lip region from the face image within the face region.

Subsequently, the information processing terminal 200A selects the attention speaker by the attention speaker determination unit 247 (step S607). Details of the processing in step S607 will be described later.

200 A of information processing terminals repeat the process from step S605 to step S607 for the number of people (step S608). In this embodiment, the speaker of interest is determined by repeating this process.

Next, the information processing terminal 200A uses the acoustic feature acquisition unit 260 to calculate acoustic features from the voice data of the person identified as the speaker of interest (step S609).

Subsequently, the information processing terminal 200A uses the speaker-embedded information calculation unit 271 to calculate the speaker-embedded information of the speaker of interest (step S610). Note that the speaker-embedded information calculation unit 271 may retain the speaker-embedded information of the speaker of interest after the speaker of interest is determined. Further, the speaker-embedded-information calculation unit 271 may delete the held speaker-embedded information when the speaker of interest is no longer the speaker of interest.

Subsequently, the information processing terminal 200A determines whether or not the lip region is detected by the lip feature amount acquisition unit 250 (step S611). In other words, information processing terminal 200A determines whether the image extracted by lip area extraction section 245 is an image within the lip area.

In step S611, if the lip region is not detected, the information processing terminal 200A uses the speech recognition unit 290 to recognize the utterance content based on the speech data (step S612), and proceeds to step S615, which will be described later.

When the lip area is detected in step S611, the information processing terminal 200A uses the lip feature amount acquisition unit 250 to calculate the lip feature amount from the image in the lip area (step S613).

Subsequently, the information processing terminal 200A uses the multimodal recognition unit 280 to recognize the speech content using the acoustic feature amount calculated in step S609 and the lip feature amount calculated in step S613 (step S614).

Subsequently, the information processing terminal 200A outputs the text data of the recognition result by the speech content recognition result output unit 233 (step S615), and ends the process. In other words, the utterance content recognition result output unit 233 causes the display 130 to display the text data of the recognition result, and ends the process.

As described above, in the present embodiment, only the speech data of the person who is the target speaker is used as the speech data for recognizing the utterance content.

Next, with reference to FIG. 7, processing of the attention speaker determination unit 247 of this embodiment will be described. FIG. 7 is a second flowchart for explaining the operation of the smart glasses of the first embodiment. FIG. 7 shows details of the processing in step S607 of FIG.

In the information processing terminal 200A of the present embodiment, the attention speaker determination unit 247 determines whether or not a plurality of face areas are detected in step S605 (step S701).

If a plurality of face areas are not detected in step S701, that is, if only one face area is detected, the speaker-of-interest determination unit 247 proceeds to step S704, which will be described later.

In step S701, when a plurality of regions are detected, the speaker-of-interest determination unit 247 determines the x-coordinate of the center of the lip region in one face image and the center point of the image indicated by the image data acquired by the image input unit 231. is calculated (step S702).

If the lip region extraction unit 245 extracts a region corresponding to a partial image of the face instead of the lip region, the x coordinate of the center point of this region is replaced by the x coordinate of the center of the lip region. should be used instead of

Subsequently, the attention speaker determining unit 247 determines whether or not the calculated distance is the smallest among the distances calculated for the plurality of face regions (step S703). In other words, the attention speaker determination unit 247 determines whether or not the calculated distance is smaller than the previously calculated distance. That is, here, the person closest to the center of the image captured by camera 110 is detected.

If the distance is not the minimum in step S703, the attention speaker determination unit 247 determines that the person corresponding to this face area does not correspond to the attention speaker (step S705), and terminates the process.

If the distance is the minimum in step S703, the speaker-of-interest determination unit 247 determines whether or not the lips are moving from the image in the region extracted by the lip region extraction unit 245 (step S704). That is, here, the attention speaker determining unit 247 determines whether or not the person corresponding to the face area is speaking.

In step S704, if the lips are not moving, the attention speaker determination unit 247 proceeds to step S705. A case where the lips are not moving indicates that the person is not speaking.

In step S704, if the lips are moving, the attention speaker determination unit 247 selects this face area as the attention speaker's face area (step S706), and ends the process.

Determination of the attention speaker by the attention speaker determination unit 247 will be further described below with reference to FIG. FIG. 8 is a diagram explaining determination of a speaker of interest in the first embodiment.

An image 81 shown in FIG. 8 is an image represented by image data acquired by the video input unit 231 . A point o in the image 81 is the center point of the image 81, and the coordinates of the center point are (x1, y1). Note that the coordinates of the center point o in the present embodiment may be coordinates when the upper left vertex of the image 81 is set as the origin, for example.

FIG. 8 shows a case where the face area of person A and the face area of person B are detected in step S605 of FIG. In this case, the information processing terminal 200A detects the lip area from each face area in step S606 of FIG. In the example of FIG. 8, a lip area Ra is extracted from the face area of person A, and a lip area Rb is extracted from the face area of person B. FIG.

Here, for example, the attention speaker determination unit 247 first selects the face area of the person B, and calculates the distance Lb between the x-coordinate of the center point of the lip area Rb and the x-coordinate of the center point o. At this time, since the distance Lb is the smallest, when the lips of the person B are moving, the person B is selected as the speaker of interest.

Next, the attention speaker determination unit 247 selects the face area of the person A, and calculates the distance La between the x-coordinate of the center point of the lip area Ra and the x-coordinate of the center point o. At this time, the distance La is smaller than the distance Lb. Therefore, the attention speaker determination unit 247 excludes person B from the attention speaker, and determines person A as the attention speaker when the lips of person A are moving.

As described above, in this embodiment, a person whose face image is detected at a position closest to the center of the image captured by the camera 110 is specified as the speaker of interest. The center of the image captured by the camera 110 is, in other words, the line-of-sight direction of the wearer of the smart glasses 100A. That is, in the present embodiment, the person closest to the line-of-sight direction of the wearer of the smart glasses 100A is determined as the speaker of interest. Then, in this embodiment, only the utterances by the speaker of interest are converted into text data.

Therefore, according to the present embodiment, even if the image captured by the camera 110 includes a plurality of people, the user of the smart glasses 100A identifies the person who is paying attention, and the identified person It is possible to display only the utterance content of the person on the display 130 . In other words, it is possible to appropriately display the utterance content of a specific speaker that the user of the speech recognition system 100 pays attention to without obstructing the user's field of vision.

Further, in this embodiment, only the utterance content of the speaker of interest is displayed on the display 130, so that the amount of information displayed on the display 130 can be suppressed from becoming excessive.

In addition, in this embodiment, since the utterance content is recognized using both the lip feature amount and the acoustic feature amount of the speaker of interest, accuracy in recognizing the utterance content can be improved.

Furthermore, in the present embodiment, when the lip region is not extracted, a region corresponding to a partial image of the face is used as a substitute. can do.

Next, the operation of the smart glasses 100A after the target speaker is determined will be described with reference to FIG. FIG. 9 is a third flow chart explaining the operation of the smart glasses of the first embodiment. The process shown in FIG. 9 is a process that is periodically executed after the attention speaker is determined by the process of FIG.

In the smart glasses 100A of the present embodiment, the information processing terminal 200A performs a process of detecting an utterance period using the voice utterance period detection unit 261 (step S901). Subsequently, the information processing terminal 200A uses the acoustic feature quantity calculation unit 263 to calculate the acoustic feature quantity from the voice data acquired in the speech period (step S902).

Next, the information processing terminal 200A uses the speaker-embedded information calculation unit 271 to calculate the speaker-embedded information of the person who spoke in the utterance period (step S903).

Next, the information processing terminal 200A uses the off-screen speaker estimation unit 273 to determine whether or not the current image of the speaker of interest is included in the image indicated by the image data acquired by the video input unit 231 ( step S904). That is, here, it is determined whether or not the speaker of interest stays in the line-of-sight direction of the wearer.

At this time, the off-screen speaker estimation unit 273, for example, performs face recognition processing on the image indicated by the image data acquired by the video input unit 231, and determines whether or not the face image of the speaker of interest is included. may be determined.

If the image of the speaker of interest is not included in step S904, the information processing terminal 200A proceeds to step S909, which will be described later.

The case where the image of the speaker of interest is not included means that the speaker of interest disappears from the field of view of the wearer of the smart glasses 100A due to the speaker of interest moving or the wearer changing the direction of the head, or indicates to move to the corner of

If the image of the speaker of interest is included in step S904, the information processing terminal 200A determines whether or not the lip area of the speaker of interest has been detected by the lip region extraction unit 245 (step S905). In step S905, if the lip region is not detected, the information processing terminal 200A proceeds to step S911, which will be described later.

In step S905, when the lip area is detected, the information processing terminal 200A uses the lip feature amount acquisition unit 250 to calculate the lip feature amount from the image extracted from the lip area (step S906), and proceeds to step S907. .

The processing in steps S907 and S908 in FIG. 9 is the same as the processing in steps S614 and S615 in FIG. 6, so description thereof will be omitted.

In step S904, if the target speaker is not included in the image, the information processing terminal 200A causes the off-screen speaker estimation unit 273 of the person identification unit 270 to detect that the target speaker is no longer included in the image for less than 10 seconds. (step S909). Note that 10 seconds is an example of a preset time, and is not limited to this.

If it is less than 10 seconds in step S909, the off-screen speaker estimation unit 273 determines whether or not the speaker-embedded information calculated in step S903 matches the speaker-embedded information of the speaker of interest (step S910). The speaker-embedded information of the speaker of interest is the speaker-embedded information calculated in step S610 of FIG.

Here, the off-screen speaker estimation unit 273 may, for example, calculate the cosine similarity of two speaker embedding information, etc., and determine that the two match when the calculated value is equal to or greater than a predetermined threshold. .

If both match in step S910, the information processing terminal 200A causes the speech recognition unit 290 to perform speech utterance recognition using the acoustic feature amount calculated in step S902 (step S911), and proceeds to step S908. .

If the two do not match in step S910, the information processing terminal 200A proceeds to step S912, which will be described later.

In step S909, if it is not less than 10 seconds, that is, if 10 seconds or more have passed since the speaker of interest left the line-of-sight direction of the wearer of the smart glasses 100A, the information processing terminal 200A cancels the determination of the speaker of interest. (step S912), and the process ends.

In other words, in this embodiment, when the face image of the target speaker is not detected from the image represented by the image data acquired by the video input unit 231 for a set time or longer, the determination of the target speaker is canceled. .

Canceling the determination of the speaker of interest means, in other words, returning from the state in which the speaker of interest has been determined to the initial state in which the speaker of interest has not been selected.

In this embodiment, in this way, even if the speaker of interest temporarily deviates from the line-of-sight direction of the wearer of the smart glasses 100A, it is determined from the voice whether or not it is the utterance of the speaker of interest, The recognition result of the utterance content can be displayed on the display 130 .

Examples of operations of the smart glasses 100A will be described below with reference to FIGS. 10 to 17 .

FIG. 10 is a first diagram illustrating an example of operation of smart glasses. FIG. 10 shows a state in which person A is positioned in the line-of-sight direction of person P wearing smart glasses 100A, and only the face area of person A is detected in the image captured by camera 110 .

In this case, the smart glasses 100A detect one face region from the image captured by the camera 110, and identify person A corresponding to this face region as the speaker of interest. The smart glasses 100</b>A detect the lip region 22 , perform multimodal speech recognition processing using the acoustic feature amount and the lip feature amount, and display text data 23 of the recognition result on the display 130 .

FIG. 11 is a second diagram illustrating an example of operation of smart glasses. FIG. 11 shows a state in which a predetermined set time (for example, 10 seconds) has passed since person A, who is the speaker of interest, has left the line-of-sight direction of person P wearing smart glasses 100A.

In this case, the smart glasses 100A determine the person A to be the target speaker based on only the voice data of the person A, perform voice recognition processing using the acoustic feature amount calculated from the voice data, and convert the text data 23a of the recognition result into Display on the display 130 .

FIG. 12 is a third diagram illustrating an example of operation of smart glasses. FIG. 12 shows a state in which a predetermined set time or more has elapsed after person A, who is the speaker of interest, has left the line-of-sight direction of person P wearing smart glasses 100A.

In this case, the smart glasses 100A cancel the determination of the target speaker for person A, and return to the initial state in which the target speaker is not determined. Therefore, nothing is displayed on the display 130 .

FIG. 13 is a fourth diagram illustrating an example of operation of smart glasses. In FIG. 13, person A is positioned in the line-of-sight direction of person P wearing smart glasses 100A, and in the image captured by camera 110, only the face area of person A is detected, and the lip area of person A is not detected. Indicates status.

In this case, the smart glasses 100A detect one face region from the image captured by the camera 110, and identify person A corresponding to this face region as the speaker of interest. In addition, since the lip region of person A is not detected, the smart glasses 100A perform speech recognition processing using the acoustic feature amount calculated from the speech data, and display the text data 23a of the recognition result on the display 130.

FIG. 14 is a fifth diagram explaining an example of operation of smart glasses. FIG. 14 shows a state in which face areas of person A and person B are detected in an image captured by camera 110 .

In this case, the smart glasses 100A measure the distance between the x-coordinate of the center point of the lip region of person A and the x-coordinate of the center point of the image captured by camera 110, and the x-coordinate of the center point of the lip region of person B. , and the distance from the x-coordinate of the center point of the image captured by the camera 110 .

Next, the smart glasses 100A determine the person with the shortest distance as the speaker of interest. In FIG. 14, person A is determined as the target speaker. Then, the smart glasses 100A detect the lip region 22 of the person A, perform multimodal speech recognition processing using the acoustic feature amount and the lip feature amount, and display the text data 23 of the recognition result on the display 130.

FIG. 15 is a sixth diagram illustrating an example of operation of smart glasses. In FIG. 15, in the image captured by the camera 110, between the person A and the person B, the person closer to the center point of the image captured by the camera 110 changes from the person A to the person B due to the movement of the wearer P's head. indicates the case.

In this case, the smart glasses 100A detect the lip region 22B of the person B, perform multimodal speech recognition processing using the acoustic feature amount and the lip feature amount, and display text data 23B of the recognition result on the display 130.

FIG. 16 is a seventh diagram illustrating an example of operation of smart glasses. FIG. 16 shows a state in which the mouth of person A, who is closer to the center point of the image captured by camera 110 than person A and person B, is hidden in the image captured by camera 110 .

In this case, the smart glasses 100A obtain the x-coordinate of the central point of the partial area of the facial image of the person A instead of the x-coordinate of the central point of the lip area of the person A, and the x-coordinate and the camera 110 A distance from the x-coordinate of the center point of the captured image is calculated. Next, the smart glasses 100A determine Person A as the speaker of interest based on this distance.

Then, the smart glasses 100</b>A perform speech recognition processing using the acoustic feature amount calculated from the speech data of the person A, and display the text data 23 a of the recognition result on the display 130 .

As described above, in the present embodiment, even when a plurality of people are present in the line-of-sight direction of the wearer of the smart glasses 100A, or when the mouth of the person whom the wearer is paying attention to is hidden, It is possible to cause the display 130 to display text data indicating the contents of the speech of the speaker.

Next, with reference to FIG. 17, recognition of utterance content of the smart glasses 100A of the present embodiment will be described. 17A and 17B are diagrams for explaining recognition of utterance content according to the first embodiment. FIG.

The voice recognition processing unit 230 of the smart glasses 100A of the present embodiment inputs the video extracted from the lip region of the person identified as the speaker of interest to the lip feature amount calculation unit 252, and acquires the lip feature amount 171. . In addition, in this embodiment, the speech waveform of the person identified as the speaker of interest is input to the acoustic feature amount calculator 263 to acquire the acoustic feature amount 172 .

Then, the speech recognition processing unit 230 combines the lip feature quantity 171 and the acoustic feature quantity 172 in the feature quantity integration unit 281 to obtain a multimodal feature quantity 173 .

Next, the speech recognition processing unit 230 inputs the multimodal feature quantity 173 to the multimodal speech content recognition unit 282, generates text data indicating the speech content using the multimodal speech content recognition model 283, and generates text data. is output to the speech content recognition result output unit 233 .

In addition, in the present embodiment, when the lips are hidden or the image captured by the camera 110 does not include the image of the speaker of interest and the image within the lip region cannot be used, the acoustic features extracted by the acoustic feature amount calculation unit 263 Quantity 172 is input to speech utterance content recognition unit 291 . The voice utterance content recognition unit 291 uses the voice utterance content recognition model 292 to generate text data representing the content of the utterance, and outputs the text data to the utterance content recognition result output unit 233 .

As described above, in this embodiment, the accuracy of speech recognition can be improved because the speech content recognition processing method is switched depending on whether or not the lip area of the speaker of interest can be detected.

Further, in this embodiment, the lip feature amount calculation model 253, the multimodal utterance content recognition model 283, and the voice utterance content recognition model 292 are composed of moving image data of the lip region, voice data, and correct text data. is a trained model in which a neural network is trained using , as learning data.

Further, in the present embodiment, voice data is acquired for each utterance period and the content of utterance is recognized, but the present invention is not limited to this. In this embodiment, for example, when the voice data of a plurality of persons are acquired at the same time, only the voice data of the target speaker may be selected based on the face image of the person detected from the image data. .

(Second embodiment)
A second embodiment will be described below with reference to the drawings. The second embodiment is a translation system to which the smart glasses 100A of the first embodiment are applied.

FIG. 18 is a diagram showing an example of the system configuration of the translation system of the second embodiment. A translation system 500 of this embodiment includes smart glasses 100A and an automatic translation device 700 . The smart glasses 100A and the automatic translation device 700 are connected via a network or the like, for example.

When the automatic translation device 700 of the present embodiment receives the text data of the first language and the selection of the language, it translates the text data of the first language into the selected language (second language), Output text data in the second language.

In the translation system 500 shown in FIG. 18, in the smart glasses 100A, based on the image data and the voice data, the text data of the result of recognizing the speech content of the speaker of interest is used as the text data of the first language. 700. At this time, the smart glasses 100A may accept the selection of the second language in advance. In that case, the smart glasses 100A transmit information indicating the second language to the automatic translation device 700 together with the text data of the first language.

The automatic translation device 700 receives text data in the first language and information indicating the second language, converts the text data in the first language into text data in the second language, and transmits the data to the smart glasses 100A. do.

Smart glasses 100A causes display 130 to display the text data in the second language received from automatic translation device 700 .

In the present embodiment, by linking the smart glasses 100A and the automatic translation device 700 in this way, the wearer of the smart glasses 100A can use a second language different from the first language used by the speaker of interest. , it is possible to display the utterance contents of the speaker of interest.

(Third embodiment)
A third embodiment will be described below with reference to the drawings. The third embodiment is a minutes creation system to which the smart glasses 100A of the first embodiment are applied.

FIG. 19 is a diagram showing an example of the system configuration of the utterance content recording system of the third embodiment. A speech content recording system 600 of the present embodiment includes smart glasses 100A and a speech content recording device 700A. The smart glasses 100A and the utterance content recording device 700A are connected via a network or the like, for example.

In the present embodiment, the smart glasses 100A are used, for example, to hold the contents of speeches made by teachers as text data in lectures at educational institutions. In this case, the wearer of the smart glasses 100A can store the content of the teacher's utterance as text data in the storage unit of the utterance content recording device 700A simply by directing the line of sight toward the teacher T who is giving the lecture. can.

In addition, in this embodiment, for example, when a plurality of people are present on a stage set up in a space such as an auditorium, the utterance contents of a specific person can also be used for saving text data. can be done.

In this embodiment, by linking the smart glasses 100A with the utterance content recording device 700A, for example, even in a scene where a plurality of people speak in random order, the utterance content of the speaker of interest can be recorded. can be saved as text data.

Note that the smart glasses 100A can also be applied to embodiments other than those described above. For example, the smart glasses 100A are useful when the wearer P has a hearing impairment.

Each function of the embodiments described above may be implemented by one or more processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays) and conventional circuit modules.

Moreover, the devices described in the embodiments are only representative of one of several computing environments for implementing the embodiments disclosed herein.

In one embodiment, the information processing apparatus 200 (information processing terminal 200A) includes multiple computing devices such as a server cluster. Multiple computing devices are configured to communicate with each other over any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein. Similarly, information processing apparatus 200 may include multiple computing devices configured to communicate with each other.

Further, the information processing apparatus 200 can be configured to share the disclosed processing steps in various combinations. For example, a process performed by information processing apparatus 200 may be performed by another information processing apparatus. Similarly, the functions of information processing device 200 may be performed by other information processing devices. Also, each element of the information processing device and other information processing devices may be integrated into one information processing device, or may be divided into a plurality of devices.

Although the present invention has been described above based on each embodiment, the present invention is not limited to the requirements shown in the above embodiments. These points can be changed without impairing the gist of the present invention, and can be determined appropriately according to the application form.

100 voice recognition system 100A smart glasses 110 camera 120 microphone 130 display 200 information processing device 200A information processing terminal 230 voice recognition processing unit 231 video input unit 232 voice input unit 233 utterance content recognition result output unit 240 attention speaker identification unit 250 lip features quantity acquisition unit 260 acoustic feature quantity acquisition unit 270 person identification unit 280 multimodal recognition unit 290 speech recognition unit

Japanese Patent Application Laid-Open No. 2021-026050

Claims (14)

a speaker-of-interest determination unit that determines a speaker of interest based on a facial image of a person detected from an image represented by image data included in video data;
a speech recognition result output unit for displaying text data converted from the speech data of the speaker of interest among the speech data contained in the video data on a display device. a lip region extraction unit for detecting a lip region including an image of the lip portion of the person from the face image of the person identified as the speaker of interest;
2. The apparatus according to claim 1, further comprising a first speech recognition unit that converts the speech data of the speaker of interest into text data by using the image data representing the image within the lip region and the speech data of the speaker of interest. voice recognition device. When the lip region is not detected from the face image,
3. The speech recognition apparatus according to claim 2, further comprising a second speech recognition section for converting the speech data of the speaker of interest into text data by using the speech data of the speaker of interest. The display device is a spectacles-type display device, and the spectacles-type display device is provided with an imaging device that captures an image in a line-of-sight direction of the wearer wearing the spectacles-type display device,
4. The speech recognition device according to any one of claims 1 to 3, wherein said video data is video data acquired by said imaging device. The attention speaker determination unit
When a plurality of face images are detected from an image indicated by image data included in the video data, a center point of a partial region of the face image and a center of the image indicated by the image data included in the video data 5. The speech recognition apparatus according to any one of claims 1 to 4, wherein the face image having the smallest distance from the point is set as the face image of the speaker of interest. a speaker-embedded information calculation unit for calculating speaker-embedded information for identifying a person who uttered the voice data;
estimating the speaker of the audio data based on the embedded speaker information calculated from the audio data when the face image of the speaker of interest is not detected from the image indicated by the image data included in the video data; 6. The speech recognition device according to any one of claims 1 to 5, further comprising a speaker estimation unit. The off-screen speaker estimation unit
A period during which the facial image of the speaker of interest is not detected from the image indicated by the image data included in the video data is less than a predetermined set time, and the facial image of the speaker of interest is detected. When the speaker-embedded information calculated from the speech data acquired in a state where the speech data is not synchronized with the speaker-embedded information of the target speaker, the speech data is determined to be the speech data of the target speaker. 7. A speech recognition apparatus according to claim 6. The off-screen speaker estimation unit
8. The determination of the speaker of interest is canceled when the face image of the speaker of interest is not detected from the image represented by the image data included in the video data for the predetermined set time or longer. A speech recognition device as described. A speech recognition system comprising an information processing device, an imaging device capable of communicating with the information processing device, and a display device capable of communicating with the information processing device,
The information processing device is
a speaker-of-interest determination unit that determines a speaker of interest based on a face image of a person detected from an image represented by image data included in video data acquired by the imaging device;
an utterance content recognition result output unit that causes the display device to display text data converted from the speech data of the speaker of interest among the speech data included in the video data. A speech recognition method by a computer, the computer comprising:
determining a speaker of interest based on a facial image of a person detected from an image represented by image data included in video data;
A speech recognition method, wherein text data converted from the speech data of the speaker of interest among the speech data included in the video data is displayed on a display device. determining a speaker of interest based on a facial image of a person detected from an image represented by image data included in video data;
A speech recognition program for causing a computer to execute a process of displaying text data converted from the speech data of the speaker of interest among the speech data contained in the video data on a display device. Smart glasses having an information processing terminal, an imaging device connected to the information processing terminal, and a display device connected to the information processing terminal,
The information processing terminal
a speaker-of-interest determination unit that determines a speaker of interest based on a face image of a person detected from an image represented by image data included in video data acquired by the imaging device;
and an utterance content recognition result output unit that causes the display device to display text data converted from the speech data of the speaker of interest among the speech data included in the video data. Translation including smart glasses having an information processing terminal, an imaging device connected to the information processing terminal, and a display device connected to the information processing terminal, and a translation device capable of communicating with the smart glasses A system, wherein the information processing terminal included in the smart glasses includes:
a speaker-of-interest determination unit that determines a speaker of interest based on a face image of a person detected from an image represented by image data included in video data acquired by the imaging device;
an utterance content recognition result output unit that outputs text data in a first language converted from the speech data of the speaker of interest among the speech data included in the video data to the translation device;
A translation system in which text data in a second language translated from text data in the first language is displayed on the display device in the translation device. Smart glasses having an information processing terminal, an imaging device connected to the information processing terminal, and a display device connected to the information processing terminal, and an utterance content recording device capable of communicating with the smart glasses. an utterance content recording system comprising: the information processing terminal included in the smart glasses,
a speaker-of-interest determination unit that determines a speaker of interest based on a face image of a person detected from an image represented by image data included in video data acquired by the imaging device;
An utterance content recognition result of causing the display device to display text data converted from the speech data of the speaker of interest among the voice data included in the video data, and outputting the text data to the utterance content recording device. an output unit;
The utterance content recording device is
An utterance content recording system having a storage unit for storing the text data output from the information processing terminal.

Images