RU218617U1 - Intelligent high-performance speech recognition device for the deaf and deaf-blind, containing microphones - Google Patents

Abstract

The utility model relates to assistive devices intended for use by people with hearing, hearing and vision disabilities, namely, to an array of microphones used independently or in the designs of these devices. The essence of the utility model is that an array of microphones for a speech recognition device for the deaf and deaf-blind, made in a constructive unity and including a block for autorun processes, a command telemetry module, a speech recognition module, a microphone control module, a module for outputting text information to a Braille display and a block control, which checks the performance of these modules and the sound capture unit and the device control unit, additionally contains four microphones for sound capture and four microphones for noise reduction, and the built-in computer is equipped with a tuning unit that is connected to the device control unit, as well as blocks for autorun processes and checks performance. The utility model achieves the technical result of improving speech recognition.

Description

The utility model relates to assistive devices intended for use by people with hearing, hearing and vision disabilities, namely, to an array of microphones used independently or in the designs of these devices.

Microphone array - a device consisting of a group of microphones, each of which is directed in a certain direction, while they work as a single unit and capture the sound signal from different points in the room. Microphone arrays are widely used in various areas where it is necessary to organize remote recording and high-quality sound transmission.

Microphone arrays are widely used in assistive devices for the hearing-impaired, deaf-blind and deaf-blind.

Thus, the use of a microphone array in a device for audiovisual navigation of deaf-blind people is disclosed in RU 192148 (G10L 21/10, publication date 09/05/2019). According to the utility model, the array consists of at least 3 microphones, the computing module is configured to receive data from the array of microphones in the form of sound frames, read in such a way as to ensure hardware synchronization of sound reading from all channels. However, the design features of the microphone array are not fully described.

A hearing aid is known in the art that includes a microphone array of five directional microphones, whereby a hearing impaired person can understand someone speaking directly in front of him. Background noise coming from other directions is suppressed by the array (see EP 1025744 A1, 08/09/2000, G02C 11/06; H04R 25/00). The electrical output signals of the microphones are applied to at least one transmission channel belonging to the ear. Means are provided for obtaining two array outputs from the outputs of the microphones, the array having two main sensing directions at an angle to each other, each of which is associated with the output of the array. Each output signal of the array is fed into its own transmission path belonging to one ear of the hearing impaired person.

The disadvantage of the known solution is that the device does not use beamforming technology to improve the signal-to-noise ratio.

It is known that noise reduction in relation to hearing aids means the attenuation of unwanted signals and the strengthening of useful signals. The desired signals are usually speech that the hearing aid user is trying to understand. Unwanted signals can be any ambient sounds that interfere with the main speaker. These unwanted sounds can be other speakers, restaurant noise, music, traffic noise, etc.

Three basic principles of noise reduction apply to hearing aids: directional beamforming, spectral subtraction, and pitch-based speech enhancement. The purpose of beamforming in a hearing aid is to create the illusion of "tunnel hearing" in which the listener hears what they are looking at but does not hear sounds coming from other directions. If he looks in the direction of the desired sound—for example, at someone he is talking to—then other distracting sounds, such as other speakers, will be attenuated. The beamformer then separates the desired "online" (line of sight) target signal from the unwanted "offline" interferer signals so that the target can be amplified while the jammer is attenuated.

In addition, this device does not have a tuning function for detecting speech and sound signals using a built-in computer, as well as the ability to process data and transfer to a smartphone or tablet for further output in text or tactile form using a built-in computer.

The microphone array is an embedded device and cannot be used stand-alone.

The proposed solution is an improved refinement of the applicant's microphone array used in a portable speech and sound recognition device (see RU 198673, 07/21/2020, G10L 21/00), made in a constructive unity and including a housing, a microphone array with a noise filtering module, a block memory of recognized speech, a battery with a charger, a block for displaying information on the display and a block for speech conversion. The device contains a board of LEDs, made with the possibility of their visibility through the slots in the housing cover, covered with glass; a single-board computer, on the platform of which an autorun process block is installed, which is interconnected with a command-telemetry module, a sound capture module, a microphone control module, a module for outputting text information to a Braille display; a block for monitoring the operability of these modules and a block for launching, as well as a block for recognizing sounds and a block for controlling a device, connected through modules with a block for autorun processes. This device differs in that the housing includes a cover and a bottom, at the bottom of which boards of a microphone array, a single-board computer, a battery, a board of buttons and a charger-discharge device are mounted, fastened together. And also by the fact that the electrical connection of the boards and the battery is made by wires, contains boards for measuring the battery voltage and indicating the current state of the device, contains an HDMI connector for connecting an external display, a USB connector for connecting a Braille display in a wired way, a LAN connector for a wired connection to the Internet and a connector for charging the battery, a mobile phone display or monitor screen is used to transmit recognized text or control the device, and that a Braille display is used to transmit recognized text or control the device.

Thanks to this design, the portable device recognizes not only speech, but also other sounds. Information about the recognized sounds comes in a converted form - in the form of printed text or braille dotted font on the display. In addition to the printed notification, the device informs the user about the source of the sound by light direction from the LEDs.

However, this solution does not use beamforming technology to improve the signal-to-noise ratio, and also lacks a trim function to determine speech and audio signals, which degrades speech recognition. This decision was taken as the closest analogue.

The task to be solved by the present utility model is to create an intelligent high-performance microphone array for speech and audio recognition, which improves the quality in terms of recognizing the interlocutor's speech by using beamforming technology to improve the signal-to-noise ratio and introducing an adjustment function to determine speech and sound signals.

The solution of the tasks set provided the achievement of the technical result, which consists in increasing speech recognition in comparison with the closest analogue.

The achievement of the technical result became possible due to the fact that in an intelligent array of microphones it was proposed to apply beamforming technology to increase the signal-to-noise ratio and highlight the direction to the speaker using four microphones in the array for sound capture and four microphones for noise reduction. This utility model is also distinguished by the presence of a built-in computer with a tuning unit for determining speech and sound signals, processing data and transferring it to a smartphone or tablet, which is connected to the device control unit, as well as to autorun processes and performance check units.

At the same time, common features with the closest analogue are the implementation of the device in a constructive unity and the presence of the following blocks and modules: an autorun process module, a command-telemetry module, a speech recognition module, a microphone control module, a module for outputting text information to a Braille display and a control unit that checks the performance the specified modules and the audio capture unit and the device control unit.

An intelligent high performance microphone array for a speech recognition device for the deaf and deafblind is disclosed in the following illustrations, where:

Fig. 1 is a general view of an intelligent high-performance array of microphones;

Fig. 2 - front view of the array;

Fig. 3 - side view of the array;

Fig. 4 - view of the PMMiK_MR board from above and below:

Fig. 5 shows how the beamforming algorithm works to improve the signal-to-noise ratio;

Fig. 6 - top and bottom view of the PSSK_MR board;

Fig. 7 is a block diagram of the array of microphones;

Fig. 8 is a diagram of the interaction between the boards and the software of the array of microphones.

In FIG. 1 shows the claimed intelligent high-performance array of microphones, made in the form of a complete technological assembly of boards without the use of wires and cables. The device includes: a microphone array and recognition module controller board (PMMIC_MR) 1 with an interface for connecting an indication 4; board for interfacing with the computer of the recognition module (PSSK_MR) 2; recognition module connector board (PR_MR) 3 with interfaces for connecting external peripheral devices via interfaces: HDMI 8 for connecting an external display, USB connector for connecting a Braille display in a wired way 7, LAN connector for wired connection to the Internet 6, connector for charging the battery 10 , as well as a connector for connecting an external microphone 9 and a connector for connecting a remote control 11.

In FIG. 2 and 3 show the features of placement and interconnection of the microphone array board and the recognition module controller (PMMiK_MR) 1 and the recognition module computer interface board (PSSK_MR) 2 using interboard connectors 17 and 21; PC interface board of the recognition module (PSSK_MR) 2 and connector board of the recognition module (PR_MP) 3 using interboard connectors 22 and 23.

In FIG. Figure 4 shows the design of the microphone array board and the controller of the recognition module (PMMiK_MR) 1 with an interface for connecting an indication 4 and placed on it a sound capture chip (MSZ) 15, a memory chip 16, a command-telemetry controller 12, an interboard connector 17, a ring of LEDs 13 and buttons 14.

New is the use of beamforming technology to improve the signal-to-noise ratio and highlight the direction to the speaker uses four microphones for sound pickup 19 and four microphones for noise reduction 18. Beamforming technology's signal processing algorithm from the microphone array can attenuate unwanted signals and enhance useful signals, what improves the quality of recognition of speech and other sounds consists in highlighting the direction to the speaker with a subsequent increase in the signal-to-noise ratio by bringing the noise and useful signals from each microphone of the array to a single phase and adding the amplitudes of the useful signals with the subtraction of the noise components. The essence of the beamforming technology shown in Fig. 5 consists in recording sound on a spatially spaced array of microphones, due to which the signal from the signal source arrives at each microphone with its own time shift t _i . Since the mutual arrangement of the microphones in the array is known, it is possible to identify the matching configurations of the useful signal on each microphone and the phase shift t _i between them, due to which, knowing the speed of sound propagation in the air, it is possible to determine the direction to the speaker and compensate for the phase difference due to the signal shift by known values of t _i and sum the signals. In this case, the signal coming from an external signal/interference source from another direction (noise) will have a significantly lower gain, inversely proportional to the number of microphones in the array due to different signal phases from the other direction.

In FIG. 6 shows the composition of the interface board with the computer of the recognition module (PSSK_MR) 2 with an embedded computer (VC) 5, on which the functions of the tuning unit 29 are implemented for determining speech and sound signals, processing data and transferring it to a smartphone or tablet for further output in text or tactile form, transmission of recognized text via the Internet. The use of tuner 29 helps to improve the accuracy of the speech capture system and saves system resources and transmitted traffic through the use of voice detection technology based on the use of a neural network trained to detect voice and distinguish it from extraneous noise (if only extraneous sounds are heard instead of speech ). Unlike voice detector solutions that use frequency filtering, this method does not reject sounds outside the specified frequency range, but searches for pre-trained speech patterns in the frequency spectrum captured by the microphone.

Also, as part of the service software, embedded PC, a process autorun block is installed, which is interconnected with a command-telemetry module, a sound capture module, a microphone control module, a module for outputting text information to a Braille display; a unit for monitoring the operability of said modules and a launch unit, as well as a sound recognition unit and a device control unit associated with the process autorun unit. In addition, the board contains a four-port USB hub 23, a PCM 20 sound card for capturing sound from an external microphone. In addition, the board has HDMI 8 connectors for connecting an external display, board-to-board connectors 21 and 22.

In the electrical block diagram of Fig. 7 shows the diagram of electrical connections between the boards PMMiK_MR 1, PSSK_MR 2 with built-in computer (VC) 5, PR_MR 3 made using interboard connectors 17, 21, 22 and 23. The diagram also displays the types of interfaces, the number of lines used, supply voltages, service voltage converters.

From the point of view of the joint operation of all boards as a single whole, the logic is as follows: after switching on using buttons 14 and controller 12, power is supplied to the built-in computer 5. After loading the functional and service software, sound is continuously captured from microphones 18 and 19, which enters sound capture chip 15 with the settings stored in the memory chip 16, from where the processed sound goes to a single-board PC, where the recognition procedure, adjustments to determine speech and sound signals, data processing and transmission to a smartphone or tablet for further output in text or tactile form, or transmission of recognized text via the Internet.

To control the settings and functions of the device, commands from the buttons 14 are used, which are processed by the controller 12 and send commands for execution to the built-in computer 5. In addition, to display information about the state of the device, the controller 12 controls the indication through the indication connection interface 4. To ensure the operability of the connected external peripherals connected via interfaces 6, 7, 8, 9, 10 additionally involve microcircuits 20, 23, data from which is sent to the built-in computer 5. All communications and signals are provided through board-to-board connectors without the use of cables, which achieves high technological performance models.

The device can be powered either from a battery from the portable device or from a standard power supply with an output voltage of 5V connected to a household power supply network with a voltage of 220 V and a frequency of 50 Hz, and if the battery is connected, charging will occur through an array of microphones.

In the array of microphones, in addition to the regular software and hardware for a portable device (transformation unit, memory unit, text information output unit, keyboard, display, etc.), which implement standard functions for transmitting information and processing it to the platform of the embedded computer automatic start when the computer is booted command and telemetry module 24, speech recognition module 25, a specialized kernel module 26 for working with a sound capture chip 15, a module for displaying text information on a Braille display 27, a block for monitoring the performance of all main processes 28, as well as a tuning block 29, a client module for working as a server 30, a module for interacting with applications for a smartphone or tablet 31.

Autostart of all processes is carried out using block 32.

The functionality of the intelligent array of microphones is provided by the sound capture units 33 and the device control unit 34.

The sound capture unit 33 implements sound capture from the microphone array and controller board (PMMiK) 1 using the core module 26, detection of the beginning and end of voice activity, sending sound to the speech recognition module 25 using the tuning unit 29, outputting the result of speech recognition to an external display and Braille display using module 27, send to external devices using control unit 34, to smartphones and tablets using module 31 or to a server for communication via the Internet via module 30, and enter the text of the user's response using Braille display or keyboard or from the server side via the Internet.

The implementation of the voice activity detector is known from the prior art and is as follows:

- conversion of an audio signal into the frequency domain;

- suppression of high-frequency noise in the signal;

- suppression of noise in the captured signal in the low frequency region;

- analysis of the purified audio signal for the presence of speech.

Speech recognition can be implemented using any cloud-based speech recognition platform.

Any HDMI-connected screen and/or Braille display can be used to display text information.

The device control unit 34 implements the ability to control the device via Bluetooth Low Energy according to the GATT profile, allows you to connect the device to a Wi-Fi network, start playing a search mode signal on it, and publishes a distribution via Bluetooth Low Energy with speech recognition results. Thus block 34 configures and controls modules 24, 25, 26, 27, 30 and 31, as well as block 29.

The device interface can display a list of all recognized phrases, battery charge level, microphone sensitivity level, the status of the presence or absence of an active sound source at the moment, the status of active connections.

In FIG. 8 schematically shows the functional diagram of the software of the proposed portable device, which includes 6 modules 24, 25, 26, 27, 30 and 31 and five blocks 28, 29, 32, 33 and 34. The autorun process block 32 installed on the platform of the embedded computer 5 initializes the command and telemetry module 24, the speech recognition module 25, the core module 26 and the module for outputting text information to the Braille display 27 and the tuning blocks 29 and device control 34. The tuning block 29 initializes, if necessary, the sound capture block 33 and adjusts it. Block 28 checks the operability of the indicated modules 24, 25, 26, 27, 30 and 31 and tuning 29 and device control 34 and, if necessary, gives a command to restart through the start block 32. The tuning block 29 and the device control block 34, which provide the functionality of the device, are directly connected , as well as through modules 24, 25, 30, 31 with blocks of autorun processes 32 and health checks 28.

The device is used as follows.

Through an array of microphones, a sound capture unit, a speech recognition module, a single-board computer, a memory unit and a module for outputting text information to a Braille display, the results of speech recognition and audio signals are instantly transmitted to a display connected via an HDMI input, including the display of mobile phones and /or a Braille display, in the form of text, which makes the user aware of everything that is happening around him. This text can be scrolled through, or you can return to the right moments. It is also possible to enter the text of the user's response using a Braille display or a keyboard connected via USB. Moreover, the LED backlight always indicates the direction of the side from which the sound source is located, which allows the user to respond to it in a timely manner. Additionally, the device is equipped with LEDs for faster and more convenient understanding of the location of the sound source - they highlight this direction. For ease of use, the device includes four control buttons that are responsible for turning on, changing the font size, scrolling text on the screen and resetting the mobile phone connection.

The utility model can be implemented using the specified device by people who are deaf or hard of hearing and sight.

In the first case, the user turns on the device, connects a display, such as a mobile phone, wirelessly via Bluetooth or a monitor screen via HDMI output. Then, on the displayed screen interface, selects the sensitivity settings of the microphone array that are relevant to him. During operation, the user will receive real-time information about what is happening around - the device will automatically convert speech and audio signals into text on the display. The LEDs will visually indicate the direction of the sound source to respond quickly. And additionally, the direction to the sound source can be indicated verbally on the display or screen. It is also possible to type a response by a user with speech defects using a keyboard connected to the USB port of the device.

In the second case, dedicated to people with both hearing and visual impairments, the user connects a braille display to the device wirelessly via Bluetooth or wired via a USB input. All recognized audio signals and speech, as well as the direction of the sound source, will be instantly processed into text and transferred to the Braille display, thereby allowing the user to freely navigate and perceive information from surrounding people and objects. It is also possible to type a response by a user with speech impediments using a Braille display.

As shown by the experiments, the proposed device recognizes speech approximately 2 times better than the closest analogue.

The beamforming system was tested on test specimens of the closest analogue speech recognition device. As a result of testing, the level of noise suppression out of the radiation pattern was -12 dB. A tuning block was also tested, which gave an increase in recognition accuracy by 16%, system speed by 5% and reduced the amount of useless (non-speech) signal for recognition in typical use by 30%.

Claims (1)

An intelligent high-performance speech recognition device for the deaf and deaf-blind, containing microphones, made in a constructive unity and including an autorun process unit, a command and telemetry module, a speech recognition module, a microphone control module, a module for outputting text information to a Braille display and a control unit that checks the performance of these modules and a sound capture unit and a device control unit, characterized in that it contains four microphones for sound capture and four microphones for noise reduction, and the built-in computer is equipped with a tuning unit that is connected to the device control unit, as well as blocks for autorun processes and health checks.

Images