JP2019008274A - Voice information processing system, control method of voice information processing system, program of voice information processing system and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To present a technical idea for more accurately grasping a situation in which a conversation is taking place while optimizing by analyzing, analyzing, recognizing, evaluating, and correcting voice contents with high accuracy and at low cost. thing. SOLUTION: An input means for inputting voice information related to voice, a pre-stage processing means for performing pre-stage processing on the input voice information so as to facilitate identification processing, and voice information processed by the pre-stage processing means. Optimized by performing predetermined processing, performing task processing based on the first information, correcting the first information when the evaluation of the task processing is not sufficient, and repeating a series of processing until the evaluation is sufficient. It is provided with an optimization means to be used. [Selection diagram] Fig. 5

Description

  The present invention relates to a voice information processing system, a method for controlling the voice information processing system, a program for the voice information processing system, and a recording medium according to a voice information processing system in conversation.

  In recent years, the development of speech information processing technology has been remarkable. For example, a voice dialogue system (for example, refer to Patent Document 1) that allows a user to easily grasp the state of the system and realizes a smooth dialogue between the user and the system, accepts complaints by voice, A dialog recording system (see, for example, Patent Document 2) that can be transmitted to a processor in a form that can be used for the above processing, and an electronic device (for example, Patent Document 3) that can smoothly interact with a user. Reference).

  The invention described in Patent Document 1 includes a microphone, voice input means, voice analysis means, voice recognition means, syntax analysis means, intention extraction means, dialogue management means, problem solving means, response sentence generation means, voice synthesis means, voice output In the voice dialogue system comprising means, a speaker, and a plurality of halfway response processing means, the plurality of halfway response processing means are voice input means, voice analysis means, voice recognition means, syntax analysis means, and intention extraction means as input system means. The processing result of any one or a plurality of means is input, and the processing result is output to one or a plurality of means among a speech output means, a speech synthesizing means, and a response sentence generating means, which are output means. It is.

  The invention described in Patent Document 2 is a recording device that records voice data of a dialog, and information that performs processing for generating an identifier for identifying a specific portion of the recorded voice data and causing the recording device to record the identifier. A processing device, and the information processing device receives an identifier generation request for the audio data recorded in the recording device, generates an identifier, and associates the identifier with the audio data to be recorded in the recording device. In the recording apparatus, voice data and identifier data are recorded, and text data obtained by voice recognition of the voice data by the voice recognition unit is recorded.

  In the invention described in Patent Document 3, the refrigerator includes a microphone and a speaker, has an interactive function of acquiring voice and speaking according to the acquired voice, and the refrigerator is within a predetermined range near the refrigerator. A position specifying unit for specifying the position of the user, a microphone control unit for adjusting the sensitivity of the microphone so as to have a value corresponding to the position of the user specified by the position specifying unit, and a speaker control unit for adjusting the volume of the speaker Is provided.

Japanese Patent No. 345497 JP 2000-067064 A JP 2017-069835 A

  However, in the invention described in Patent Document 1, the user can recognize that his / her utterance is input as a voice by a parrot return response or a mutual response, and can make the next utterance with peace of mind. Therefore, no evaluation is made for irregular utterances with mixed noises and echoes, and fixed phrases are not modified.

  In addition, the invention described in Patent Document 2 determines whether it is equal to or greater than a predetermined reference value. When the reference value is less than the reference value, it is determined that the speech is interrupted, and then when the reference value is exceeded, Although it is determined that the utterance has begun and a cueing signal is output, there is no processing for grasping the emotion of the other party in the dialogue or improving the mistake in hearing.

  Furthermore, although the invention described in Patent Document 3 makes a passive response to speak in response to the acquired voice, it does not talk to the other party when active.

  This application is intended to solve such problems, and is optimized by analyzing, analyzing, recognizing, evaluating, and correcting speech content with high accuracy and at a low cost. The aim is to present a technical idea that more accurately grasps the situation.

  In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that input means for inputting sound information relating to sound and pre-stage processing for performing pre-stage processing for facilitating identification processing on the input sound information. A predetermined processing is performed on the voice information processed by the processing means and the preceding processing means, the task processing is performed based on the first information, and the first information is corrected when the evaluation of the task processing is not sufficient. And an optimization means for optimizing by repeating a series of processes until the evaluation becomes sufficient.

  Here, the sound is a sound wave including a physical sound (for example, a sound of hitting a desk or a door), a human voice and noise (for example, a siren, a cry of an animal, a crushing sound, etc.).

  Further, the first information is a scenario design related to application software (hereinafter referred to as “application”), which means is selected from various means, and in what order is executed, evaluated, and evaluated. This is information including a flow that repeats when this is insufficient.

  According to a second aspect of the present invention, in addition to the configuration of the first aspect, the optimization means includes a first evaluation means for evaluating a result of the task processing, and the first evaluation means when the evaluation is not sufficient. And a repeating unit that repeats a series of processing from the preceding processing unit to the correcting unit.

  The invention according to claim 3 is the case where the processing capability of the voice input / output device on the client side can cope with the response to the analysis of the contents of the voice in the room in addition to the configuration according to claim 1 The voice input / output device performs information processing, and when the processing capability of the voice input / output device is not compatible, the cloud side includes a determination unit that performs information processing.

  The invention described in claim 4 is characterized in that, in addition to the configuration described in claim 1, an external system for managing the setting of the indoor environment, intentional interpretation, and dialogue is provided.

  Here, intentional interpretation refers to interpretation that estimates the speaker's intention and reflects the estimation result.

  According to a fifth aspect of the present invention, in addition to the configuration according to the first aspect, the external system includes an imaging unit that images the outside of the voice input / output device casing, a vibrating unit that vibrates the casing, and the casing. And at least one projection means for projecting an image onto a wall outside the housing.

  The invention described in claim 6 is characterized in that, in addition to the configuration described in claim 1, external content can be used for intentional interpretation and management of the dialog.

  The invention according to claim 7 performs the information processing process including the analysis process, the analysis process, and the recognition process of the content of the voice in addition to the configuration of the claim 1, and includes the age and gender of the speaker. Inference means for inferring attributes, collection means for collecting logs used when designing the second information, analysis means for analyzing the logs, second means for evaluating the response and the second information And an evaluation means, and performing optimization based on continuous evaluation based on the evaluation.

  Here, the second information refers to information about a flow that is used among various means, is processed in which order, is evaluated, and is repeated when it is not sufficient.

  The recognition process consists of recognition of laughter, applause, call, etc. in addition to the speaker from the collected voice information, and analysis, analysis, recognition, etc. of environmental sound, and speaker identification, gender estimation, age In addition to performing estimation and the like, various information regarding the birthplace and the like is provided from intonation determination.

  According to an eighth aspect of the present invention, in addition to the configuration according to the seventh aspect, the inference means includes an interpretation means for intentionally interpreting the dialogue with the speaker, and a management for managing the dialogue with the speaker. Means.

  Here, the management of dialogue with the speaker is to record what kind of emotion the speaker has about what kind of utterance in order to improve customer satisfaction, and the voice input / output device on the client side is placed in the call center. This includes alerting the operator and reporting to the administrator when it is being used. Also, when using the client's voice input / output device for a meeting, if the attendees become emotional, take a break to calm down, or speak a voice to the effect of calmness Including that.

  According to a ninth aspect of the present invention, in addition to the configuration according to the fourth aspect, the environment determining unit includes a size determining unit that determines the size of the room, and a noise level recognizing unit that recognizes the noise level in the room. Reverberation level recognition means for recognizing the reverberation level in the room.

  According to a tenth aspect of the present invention, in addition to the configuration according to the fifth aspect, an image display means provided on the housing for displaying an image is provided.

  The invention described in claim 11 is characterized in that, in addition to the configuration described in claim 5, a fingerprint authentication unit provided in the housing for recognizing a user is provided.

  The invention according to claim 12 has the processing capability of the voice input / output device on the client side in addition to the configuration according to claim 1, such as the processing speed of the processor, memory size, sensor type, microphone array, number of speakers, LED And the number of built-in cameras, the number of application software, and front-end signal processing compatible with devices of other companies.

  According to a thirteenth aspect of the present invention, in addition to the configuration of the first aspect, characteristic extraction means for extracting a speech method as a feature from the speaker's voice, and storing the feature in association with the speaker information. And speaker identification means for identifying the speaker by comparing the characteristics of the newly input voice with the speaker information stored in the storage means.

  The invention described in claim 14 is characterized in that, in addition to the configuration described in claim 7, an emotion identifying means for identifying the speaker's emotion is provided.

  The invention according to claim 15 is provided on the outer periphery of the casing in addition to the structure according to claim 14, and is lit around so that a part of the emission color on the track is different from the emission color of the remaining part. And a light emitting means for emitting light so that the part of the emission color stops in the direction of the speaker when the speaker is detected.

  According to the sixteenth aspect of the present invention, voice information related to voice is input, pre-processing is performed on the input voice information to facilitate identification processing, and predetermined processing is performed on the pre-processed voice information. To perform task processing based on the first information, modify the first information when the evaluation of the task processing is not sufficient, and optimize by repeating a series of processing until the evaluation is sufficient It is characterized by doing.

  The invention according to claim 17 is a computer-readable program, wherein the computer has input means for inputting voice information related to voice, and the preceding stage that facilitates identification processing for the input voice information. Pre-processing means for performing processing, performing predetermined processing on the audio information processed by the pre-processing means, performing task processing based on the first information, and when the evaluation of the task processing is not sufficient, It is a program of a voice information processing system for functioning as an optimization unit that corrects information and optimizes by repeating a series of processes until the evaluation becomes sufficient.

  The invention according to claim 18 is a recording medium on which the program according to claim 17 is recorded.

  According to the present invention, it is possible to optimize by analyzing, analyzing, recognizing, and evaluating speech content with high accuracy and at low cost, and more accurately grasping a situation where conversation is being performed.

It is an example of the block diagram of the whole audio | voice information processing system which concerns on one Embodiment of this invention. It is an example of the hardware block diagram of the cloud side server used for the audio | voice information processing system shown in FIG. FIG. 2 is an example of a hardware block diagram of the voice information processing system shown in FIG. 1. It is an example of the software block diagram of the audio | voice information processing system shown in FIG. FIG. 3 is an example of a software stack diagram showing processing contents of the voice information processing system shown in FIG. 2. It is an example of the external view of the audio | voice information processing system shown in FIG. It is an example of the flowchart which shows the whole operation | movement in the audio | voice information processing system shown in FIG. It is another example of the flowchart which shows the whole operation | movement in the audio | voice information processing system shown in FIG. It is another example of the flowchart which shows the whole operation | movement in the audio | voice information processing system shown in FIG.

  Embodiments of the present invention will be described with reference to the drawings.

<Configuration>
<Entire system>
FIG. 1 is an example of a configuration diagram of an entire audio information processing system according to an embodiment of the present invention. The voice information processing system according to an embodiment of the present invention is configured by connecting a cloud-side server 20 and a client-side voice input / output device 100 via a network 10.

  A voice information processing system may be configured by linking the voice input / output device 100 and a smartphone, or may be connected to the Internet using a Wi-Fi router. In addition, as a communication means between the voice input / output device 100 and the smartphone, any method such as wireless, infrared, or wired may be used. Further, when there are a plurality of audio information processing systems to which the present application is applied, for example, there are as many audio input / output devices 100 as there are.

<Hardware configuration of cloud server and voice input / output device>
Next, details of the cloud-side server 20 will be described with reference to FIG. As shown in FIG. 2, the cloud server 20 includes a database (hereinafter referred to as “DB”) 21, a processor 22, an output device 23, an input device 24, an interface 26, and the like. The processor (also referred to as “computer”) 22 performs processing of data related to management of voice information, and the DB 21 stores data such as information related to management of voice information, a control program, and the like. The output device 23 includes a display, a printer, and the like, and outputs various types of information as necessary. The input device 24 includes a keyboard, a barcode reader, a scanner, and the like. The input device 24 inputs information as necessary, but includes all devices that can input information. Note that the cloud-side server 20 may be composed of a single system or a plurality of systems as long as it can finally perform business as a voice information processing system.

<Hardware configuration of voice input / output device>
Next, details of the voice input / output device 100 will be described with reference to FIG. As shown in the figure, the voice input / output device 100 mainly includes an expansion unit 201, a storage unit 202, a microphone unit 203, a microphone control unit 204, a signal processing unit 205, a communication unit 206, a voice generation unit 207, and an inaudible sound generation. A unit 208 and a display unit 209 are provided. The display unit 209 may include an LED (Light Emitting Diode) 210 and an LCD (Liquid Crystal Display) 211. The LED 210 may be ring-shaped.

  The voice input / output device 100 may further include an imaging unit 212, a personal authentication unit 213, an IR (Infrared) unit 214, a projection unit 215, a vibration unit 216, and a rotation unit 217 indicated by broken lines.

  The expansion unit 201 is a member for connecting a USB (Universal Serial Bus) memory or a USB device to the voice input / output device 100.

  The storage unit 202 is a member that stores data such as a control program of the voice input / output device 100, voice data, personal data, and image data. For example, a ROM (Read Only Memory) and a RAM (Random Access Memory): Examples thereof include a rewritable memory (HDD), a hard disk drive (HDD), and a solid state drive (SSD).

  The microphone unit 203 includes at least one microphone, and the microphone control unit 204 performs control such as AGC (Automatic Gain Control) and forming.

  The signal processing unit 205 performs processing such as ambient noise removal on the audio signal from the microphone, and after performing accurate recognition processing, stores the processed information in the storage unit 202 and generates audio from the audio generation unit 207. Or performing speaker identification processing or emotion identification processing of the audio signal from the microphone, or processing the processed voice information together with the arrival direction information, speaker identification information, and emotion identification information in the storage unit 202, and at the same time the display unit 209 Display conforming to the. In addition, the information can be transmitted from the communication unit 206 or the expansion unit 201 to the outside, and detailed information analysis can be performed by cloud processing or the like. With these processes, it is possible to block out audio information from a noise source existing in a specific direction, or to record only information from a specific direction on the contrary.

  In addition, the storage unit 202 has a multi-layer configuration, and it is possible to organize related information such as the arrival direction of voice information to be recorded, speaker identification, and emotion identification.

  The signal processing unit 205 includes a communication unit 206 for wireless communication with an external device using Wi-Fi, Bluetooth (registered trademark), and the like, and an expansion unit 201 connected to the external device with a hard wire. It is possible to collect noise and input the received noise from the expansion port to reduce the influence of ambient noise, or to communicate with an external device via the USB port.

  Ultrasonic waves are generated from the non-audible sound generation unit 208, and the distance to the speaker or the wall due to the reflection can be measured.

  The LED 210 of the display unit 209 may be a ring-shaped LED that is lit around, blinks, or changes the emission interval or emission color. The LCD 211 may be provided on a top plate or a side surface of the housing of the voice input / output device 100, and may be color or monochrome.

  The imaging unit 212 is a member that captures the situation around the audio input / output device 100, and includes, for example, a CCD (Charge Coupled Device) camera. The image captured by the imaging unit 212 may be a moving image or a still image.

  The personal authentication unit 213 is a member that identifies a user's fingerprint or voiceprint, and even if it is a fingerprint identification device provided on the top plate of the voice input / output device 100, a voiceprint identification device that identifies a voiceprint from the voice of the speaker (Or software).

  The IR unit 214 is an infrared sensor, and can be used as a human sensor for monitoring intrusion of people and detecting visitors.

  The projection unit 215 is a projector that is provided in the housing of the voice input / output device 100 and projects a map and an agenda on an indoor whiteboard, a wall, and a screen, for example, for a meeting and travel explanation.

  The vibration unit 216 alerts the user by vibrating the casing of the client-side voice input / output device 100. Examples of the vibration unit 216 include a piezoelectric element and a motor having an eccentric cam on the output shaft.

  The rotating unit 217 includes a base provided on the bottom surface of the voice input / output device 100, a rotating shaft provided on the base, and a motor provided on the rotating shaft for rotating the casing. The rotation unit 217 can change the orientation of the projection unit 215 and the LCD 211.

<Software configuration of cloud server and voice input / output device>
The software configuration of the cloud server and the voice input / output device will be described with reference to FIG.

<Cloud server>
The cloud server 20 includes an input unit 41, an output unit 42, a storage unit 43, a determination unit 44, an optimization unit 45, a translation unit 49, a first control unit 50, an inference unit 51, and a communication unit 52. ing.

  The optimization unit 45 includes an evaluation unit 46, a correction unit 47, and a repetition unit 48. Inference means 51 includes emotion identification means 51a, orientation detection means 51b, speaker identification means 51c, collection means 51d, interpretation means 51e, management means 51f, size determination means 51g, noise level recognition means 51h, and reverberation level recognition means 51i. Is provided.

  The input means 41 inputs information as necessary, but includes all devices that enable input of information, and is realized by the input device 24 shown in FIG.

  The output means 42 is a means for outputting various information as necessary, and is realized by the output device 23 shown in FIG.

  The storage means 43 is means for storing data such as information related to the control program of the cloud-side server and management of voice information, and is realized by the database 21 shown in FIG. The control program is set so as to say a greeting first when it is activated, that is, for example, when a person is detected.

  The determination means 44 performs information processing in the voice input / output device 100 on the client side when the processing capability of the voice input / output device 100 on the client side is compatible when analyzing and responding to the voice content in the room, This is means for determining that the cloud-side server 20 performs information processing when the processing capability of the client-side voice input / output device 100 is not compatible, and is realized by the processor 22 shown in FIG.

  Here, the processing capability of the voice input / output device 100 on the client side is the processing speed of the processor, memory size, sensor type, microphone array, number of speakers, number of LEDs, built-in camera, number of application software, device of other companies Includes front-end signal processing that can handle

  When the optimization unit 45 performs predetermined processing on the voice information processed by the pre-processing unit so that the identification processing becomes easy, performs task processing based on the first information, and the task processing is not sufficiently evaluated The first information is corrected and optimized by repeating a series of processes any number of times until the evaluation becomes sufficient, and is realized by the database 21 and the processor 22 shown in FIG.

  The first information is the scenario design related to the application, which one of the various methods is selected, how and how it is executed, evaluated, and the flow that is repeated as many times as the results are insufficient. It is information to include.

  The evaluation means 46 of the optimization means 45 is a means for evaluating the result of task processing. The correcting means 47 of the optimizing means 45 is means for correcting the first information when the result is not sufficient. The repetition means of the optimization means 45 is means for repeating a series of processes from the evaluation means 46 to the correction means 47 any number of times.

  The translation means 49 is means for automatically identifying a language, for example, translating from Japanese to multilingual other than Japanese, and translating multilingual other than Japanese into Japanese. The database 21 shown in FIG. And the processor 22. The translation means 49 converts Japanese speech into multilingual speech other than Japanese, or converts multilingual speech other than Japanese into Japanese speech, so to speak, an interpretation function (or simultaneous interpretation). Function). In this case, the speech is synthesized by speech synthesis means. However, it is also possible to determine the gender and age of the speaker and pronounce the speech corresponding to the young and old. The translation means 49 can also record the translation contents in a text format when translating. The text data is preferably associated with the speaker. The text data may be color-coded according to the speaker.

  The first control unit 50 is a unit that performs overall control of each unit of the cloud-side server 20, and is realized by the processor 22 illustrated in FIG.

  The inference means 51 is a means for performing information processing including speech content analysis processing, analysis processing, and recognition processing, and inferring attributes including the age and gender of the speaker. The database 21 shown in FIG. And the processor 22.

  Emotion identification means 51a is means for identifying the speaker's emotion, and is realized by the database 21 and the processor 22 shown in FIG. It is preferable to record the emotion of the speaker identified by the emotion identifying means 51a in association with the speech of the speaker.

  The direction detection means 51b is a means for detecting the direction of the speaker as viewed from the voice input / output device 100, and is realized by the input device 24 and the processor 22 shown in FIG.

  The speaker identifying unit 51c stores the feature obtained by the feature extracting unit that extracts the difference from the average acoustic model of the speech as a feature from the speaker's voice in association with the speaker information, and newly inputs the feature. The voice characteristics are compared with the speaker information stored in the storage means 43 to identify the speaker, and is realized by the database 21 and the processor 22 shown in FIG. Speaker identification may be realized by face image recognition processing or fingerprint authentication processing by providing an imaging unit 73 and fingerprint authentication unit 72 for imaging in the voice input / output device 100 described later.

  The collecting unit 51d is a unit that collects a log used when designing the second information, and is realized by the database 21 and the processor 22 illustrated in FIG. As described above, the second information refers to information about a flow to be repeated among the various means, which means are used, in which order they are processed, evaluated, and the result is not sufficient.

  The interpretation means 51e is a means for intentionally interpreting the dialogue with the speaker, and is realized by the database 21 and the processor 22 shown in FIG.

  The management means 51f is a means for managing dialogues with speakers, and is realized by the database 21 and the processor 22 shown in FIG.

  The size determination means 51g is a means for determining the indoor size, and is realized by the interface 26 and the processor 22 shown in FIG. The size determination means 51g intermittently generates non-audible sounds in the client-side voice input / output device 100 via the interface 26, collects the reflected sound from the surroundings with a microphone, and the client-side voice input / output device 100 You may make it grasp | ascertain an environment (two-dimensional azimuth | direction and distance).

  The noise level recognition means 51h is a means for recognizing the indoor noise level, and is realized by the interface 26 and the processor 22 shown in FIG. The noise level recognizing means 51h can obtain information on the noise level before the noise removal processing from the room voice obtained by the microphone of the voice input / output device 100 on the client side via the interface 26. . Based on the noise level in the room, it can be determined whether the environment of the voice input / output device 100 on the client side is, for example, reception, conference room, call center room, or other.

  The reverberation level recognition means 51i is a means for recognizing the reverberation level in the room, and is realized by the interface 26 and the processor 22 shown in FIG. The reverberation level recognizing means 51i can obtain the information of the reverberation level from the room sound obtained by the microphone of the client-side voice input / output device 100 via the interface 26. The environment of the voice input / output device 100 on the client side can be determined by the reverberation level in the room.

  The communication means 52 is means for exchanging information between the cloud-side server 20 and the client-side voice input / output device 100 via the network 10, and can be realized by the interface 26 shown in FIG.

<Voice input / output device>
The voice input / output device 100 on the client side mainly includes input means 61, output means 62, pre-stage processing means 63, light emitting means 64, communication means 65, second control means 66, storage means 67, input / output means 68, and detection means. 69 is provided. The voice input / output device 100 may further include an image display means 71, a fingerprint authentication means 72, an imaging means 73, an external information input means 74, a vibration means 75, and a rotation means 76.

  The input means 61 is means for inputting voice information related to voice, and is realized by the microphone unit 203 and the microphone control unit 204 shown in FIG. The input means 61 performs beam forming processing, blind sound source separation processing, reverberation suppression processing, noise suppression processing, echo cancellation (barge-in) processing, and voice section detection (Voice Activity Detection: VAD) processing.

  The output unit 62 is a unit that generates a non-audible sound (a sound of 20 kHz to 40 kHz, preferably 30 kHz), and is realized by the non-audible sound generation unit 208 shown in FIG. Examples of the non-audible sound generating unit 208 include an ultrasonic speaker using a piezoelectric element in addition to an ultrasonic speaker using a moving coil.

  The pre-processing unit 63 is a unit that removes noise from the sound from the microphone, cancels echo, performs beam forming processing, blind sound source separation processing, and reverberation suppression processing. The microphone control unit 204 shown in FIG. It is realized by.

  The light emitting means 64 is provided on the outer periphery of the housing, and turns on so that a part of the light emission color on the track is different from the light emission color of the remaining part, or a part of the light emission color when the speaker is detected. It is means for emitting light so as to stop in the direction of the speaker, and is realized by the LED 210 shown in FIG.

  The communication means 65 is means for exchanging information between the client-side voice input / output device 100 and the cloud-side server 20 via the network 10, and is realized by the communication unit 206 shown in FIG.

  The second control means 66 is a means for comprehensively controlling the voice input / output device 100, and is realized by the signal processing unit 205 shown in FIG. An example of the signal processing unit 205 is a processor.

  The storage means 67 is a means for storing a program for overall control of the voice input / output device 100, and is realized by the storage unit 202 shown in FIG. The storage unit 67 includes, for example, ROM, RAM, HDD, and SSD, and may be configured to store voice information, personal information, image information, and fingerprint information.

  The input / output means 68 is means for connecting a USB flash memory or a USB device, and is realized by the expansion unit 201 shown in FIG.

  The detection means 69 is a means for detecting a person who approaches or passes the voice input / output device 100, and includes the IR unit 214 shown in FIG. An example of the detection unit 69 is a human sensor.

  The image display means 71 is a means for displaying an image such as a still image or a moving image including character information, and is realized by the LCD 211 shown in FIG.

  The fingerprint authentication means 72 is a means for recognizing a user, and is realized by the personal authentication unit 213 shown in FIG. An example of the fingerprint authentication means 72 is a fingerprint sensor.

  The imaging means 73 is a digital camera and is realized by the imaging unit 212 shown in FIG.

  The external information input unit 74 is a unit for inputting content from the outside, and is realized by the extension unit 201 shown in FIG.

  The vibration means 75 is a means for vibrating the casing of the voice input / output device 100, and is realized by the vibration unit 216 shown in FIG.

  The rotating means 76 is means for rotating (turning) the casing of the voice input / output device 100 around a vertical central axis, and is realized by the rotating unit 217 shown in FIG.

  The projection means 77 is a means for projecting an image onto an indoor screen, whiteboard, wall surface, etc., and is realized by the projection unit 215 shown in FIG.

<Software stack>
The configuration of the analysis processing, analysis processing, recognition processing, and the like described above will be described according to the processing contents of the software stack diagram of FIG. The processing contents include a usage log collection / analysis unit 503, an intention interpretation / dialog management technology unit 504, a speech recognition unit 505, a speaker identification unit 506, an environmental sound recognition unit 507, an emotion analysis unit 508, and a front-end signal processing technology unit 509. , A microphone array processing technology unit 510, a multi-microphone array processing technology unit 510, a multi-microphone hardware unit 512, a sensor 515, and the like. The intentional interpretation / dialog management unit 504 may be connected to an external system 513 or an external content 514. The scenario design is evaluated according to the dialog applications 501-1 to 501-n. At that time, use log collection and analysis to improve continuously. In the figure, although three arrows are described for continuous improvement, the intentional interpretation / dialog management technology unit 504, the speech recognition unit 505, the speaker identification unit 506, the environmental sound recognition unit 507, the emotion analysis unit 508, The front-end signal processing technology unit 509, the microphone array processing technology unit 510, and the multi-microphone hardware unit 512 are also described with arrows because the order is changed or a part thereof is omitted and continuous improvement is performed. May be.

  The multi-microphone hardware 512 and the front-end signal processing technology unit 509 can flexibly support products of other companies.

  The microphone array processing technology unit 510 includes beam forming processing, blind sound source separation processing, reverberation suppression processing, and the like, and the front-end signal processing technology unit 509 includes noise suppression processing, echo cancellation (barge-in) processing, voice section detection (“VAD”). The speech recognition unit 505 that includes processing and the like and converts speech into characters and the like includes multilingual processing, automatic language identification processing, multilingual mixed processing, and the like, and the speaker identification unit 506 performs pre-learning. Processing, clustering processing, speaker verification (authentication) processing, etc., emotion analysis unit 508 is composed of emotion multi-class classification processing, emotion mapping processing, intonation recognition processing, etc., and environmental sound recognition unit 507 is Laughter recognition processing, overlap detection processing, scene estimation processing, and abnormal sound inspection processing are acoustic sensory inspection processing, normal / abnormal sound recognition processing It consists of processing, and the like.

  Although the configuration of the voice processing has been described above, the voice processing performs processing such as recognition of characteristic sounds such as laughter, applause, and bells, speaker identification, gender estimation, intonation determination, and the like. However, in order to enable each of the above processing, as sound preprocessing, processing such as voice extraction, noise suppression, reverberation suppression, sound source localization, that is, beam forming for extracting a sound at a specified angle is performed. Further, the recognition result information obtained by each process is analyzed and managed by the usage log collection / analysis technology unit 503 and the intention interpretation / dialog management technology unit 504 in cooperation with the related system, for example.

  In addition to collecting a person's speech, for example, using a microphone, the speaker produces, for example, a super audible sound or a non-audible sound that cannot be heard by a person's hearing, for example, and a reflection of the super audible sound or non-audible sound. The sound is collected by, for example, a microphone, and this information is analyzed and recognized by voice processing, and the material, distance, and the like of the object to be reflected are grasped regarding the surrounding situation of the voice input / output device 100. In order to enable these, for example, a plurality of microphones are mounted, and for example, a microphone mounting portion (not shown) in which a microphone is mounted in a circular shape in the horizontal direction is mounted in the vertical direction, for example, in two layers, so that the vertical direction Can be detected. In addition, a super audible sound or a non-audible sound to be generated is formed into, for example, a pulse shape, and a plurality of information sounds peculiar to a certain voice input / output device 100 are added to an interval between a certain pulse sound and the next pulse sound. The voice input / output device 100 can be identified.

  Here, according to the present embodiment, which processing is used, such as voice processing such as speaker identification and emotion identification, usage log collection, analysis, and intention interpretation, in which order, and reverberation after removing noise. The user can freely select whether the noise is removed after performing the processing or the reverberation processing.

<Voice information processing system>
The system illustrated in FIG. 6 is a system including the voice input / output device 100 and the cloud-side server 20.

  The voice input / output device 100 includes an LED (Light Emitting Diode) ring 102 disposed on the outer periphery of the top plate of the housing 101 in which a large number of through holes are formed, and a peripheral surface on the same plane of the housing 101. A plurality (for example, 16 but not limited) of PDM microphones 103-1 to 103-16 arranged in a circle and a speaker group (a squawker 104 </ b> S and a tweeter 104 </ b> S) arranged in the casing 101 toward the lower end. ) And a conical reflector 105 convex upward on the bottom surface of the housing 101. Various circuit boards are provided in the housing 101. Reference numeral 106 denotes an LED as a power lamp. Reference numeral 107 denotes a power cord, which can be mounted on a battery.

  The casing 101 is cylindrical in the figure, but is not limited, and may be a prismatic shape, a cone shape, a truncated pyramid shape, or a truncated cone shape.

  The LED ring 102 is a large number of three-color LED multi-color light emitting devices, and is formed in an annular shape. The LED ring 102 emits light in a different emission color from some of the adjacently arranged LEDs and the other remaining LEDs, or rotates or stops along a circular path, and is lit around It is possible to do. For example, it is possible to turn on the speaker in white with several LEDs and turn on the remaining LEDs in blue. However, the present invention is not limited to this. The illuminance may be changed according to the intensity of the person's voice.

  Since 16 PDM microphones 103-1 to 103-16 are arranged at equal intervals on the same plane of the casing 101, the sound source can be identified in the left-right direction within 20 degrees from the reflected sound. It is possible to collect human voice as a sound source in the vicinity of the voice input / output device 100.

The squawker 104S is a speaker that generates normal sound, and the tweeter 104T is a speaker that generates inaudible sound (ultrasonic waves). The reflection plate 105 is a member that radially reflects the sound and non-audible sound from the squawker 104S and the tweeter 104S to the outside of the housing 101. When the non-audible sound from the tweeter 104 is reflected by the reflecting plate 105 and is emitted to the outside of the housing 101 and reflected by the speaker, the sound is directed to the housing 101 and collected by the PDM microphones 103-1 to 103-16. It can function like a radar (or sonar). This non-audible sound may be pulse modulated.
The cloud system 300 includes a server group 301 having a plurality of servers, and may perform software processing such as deep learning processing of the voice input / output device 100.

<Operation 1>
The subject of the operation of the flowchart shown in FIG.

  In this system, a user can set a scenario as first information in advance. The scenario is, for example, a story indicating how to respond to a dialogue between the speaker and the voice input / output device 100.

  First, the user creates a plurality of information (for example, first information, second information, and third information) in the cloud-side server 20 (step S10).

  When the power switch of the voice input / output device 100 is turned on (step S11), the processor 22 determines whether there is a mode setting signal from the outside of the voice input / output device 100, for example, with a smartphone (not shown) (step S12).

  When the processor 22 determines that there is a mode setting signal from the outside (step S12 / YES), one of the call center mode, the reception mode, the conference mode,... Is set (step S13), and the mode from the outside is set. When it is determined that there is no setting signal (step S12 / NO), the process proceeds to step S14.

  In step S14, the processor 22 determines whether or not the processing capability of the voice input / output device 100 on the client side is sufficient, and determines that the processing capability of the voice input / output device 10 is sufficient (step S14 / YES). When processing is performed by the voice input / output device 100 on the client side (step S15) and it is determined that the processing capability of the voice input / output device 100 on the client side is not sufficient (step S14 / NO), processing is performed on the cloud side (step S16). The process proceeds to step S17.

  The processor 22 determines whether or not it has been optimized (step S17). If it has not been optimized (step S17 / NO), the processor 22 selects the first information (step S18), and makes the identification process easy. Execute the process. For example, at least one of beam forming, blind source separation, and reverberation suppression is selected as the first information, and the order is determined and executed. It is assumed that the order is determined and executed by appropriately selecting from noise suppression, echo cancellation, and speech section detection (step S19).

  Next, the task is evaluated (step S20), it is determined whether or not the evaluation is sufficient (step S21), and if it is determined that the evaluation is not sufficient (step S21 / NO), the process returns to step S10 and the cloud side 1 information is corrected, and steps S10 to S23 are repeated any number of times.

  When the processor 22 determines that the evaluation of the task is sufficient in step S21 (step S21 / YES), the processor 22 selects the second information (step S22) and executes the identification process. The identification process is executed by selecting at least one of the second information, for example, voice recognition, speaker identification, emotion analysis, and environmental sound recognition, determining the order, and executing (step S23). ).

  Next, the task is evaluated (step S24), it is determined whether the evaluation is sufficient (step S25), and if it is determined that the evaluation is not sufficient (step S25 / NO), the process returns to step S10, and the cloud side 2 is corrected, and Steps S10 to S24 are repeated any number of times.

  When the processor 22 determines that the evaluation of the task is sufficient in step S24 (step S24 / YES), the processor 22 selects the third information (step S26) and executes the interactive application. As for the dialogue application, at least one of usage log collection, analysis, intentional interpretation, and dialogue management is selected as the third information, and the order is determined and executed (step S27).

  Next, the task is evaluated (step S28), it is determined whether the evaluation is sufficient (step S29), and if it is determined that the evaluation is not sufficient (step S29 / NO), the process returns to step S10, and the cloud side The third information is corrected, and steps S10 to S28 are repeated any number of times.

  When the optimization is completed (step S30), the processor 22 executes the application (step S31), and if not finished (step S32 / NO), returns to step S12, and if finished (step S32 / YES), ends. To do. In this case, the power switch may be automatically turned off.

  Here, the flowchart shown in FIG. 7 is only an example and is not limited. For example, if the indoor environment is the same and there are changes in the number of speakers, even if there is a change in the indoor environment, there is no change in the number of speakers or people, the mode used is changed, the microphone used Even if there is a change in the number or grade, continuous improvement is possible by replacing Step S18 to Step S21, Step S22 to Step S25, and Step S26 to Step S29 as appropriate, or omitting some of them. Can be applied flexibly.

  For example, when processing is performed based on the flowchart shown in FIG. 7, beam forming and blind sound source separation are not performed when there is one microphone of the microphone unit, and noise suppression is performed when the performance of the microphone is not high performance. Is performed only once. When there are a large number of microphones in the microphone unit, for example, 16 or more, beam forming and blind sound source processing are performed, and processing such as noise suppression and echo cancellation is performed.

<Operation 2>
8 is the processor 22 of the cloud-side server 20. The difference between the flowchart shown in FIG. 8 and the flowchart shown in FIG. 7 is that steps S18 to S21 and steps S26 to S29 are interchanged. This is an example of processing that is performed when the result is insufficient even if processing is performed in the flowchart shown in FIG. Even if such a process is performed, continuous improvement is made and a flexible response is possible.

<Operation 3>
The flowchart shown in FIG. 9 is placed on the reception desk, conference room, or call center operator desk with the power switch turned off after the use of the voice input / output device 100 is finished, and again on the next business day. It is assumed that the power switch is turned on.

  When the power switch is turned on (step S91), the processor 22 determines whether there is no room environment, speaker change, number of people, and mode change (step S92). If there is no change (step S92 / YES) ) The application is executed (step S93).

  The processor 22 determines whether or not to end after executing the application (step S94). If it is ended, the processor 22 ends (step S94 / YES). If not, the processor 22 returns to step S92 (step S94 / NO).

  When there is an indoor environment change, speaker change, number of persons, and mode change (step S92 / NO), the processor 22 determines whether or not the processing capability of the voice input / output device 100 on the client side is sufficient (step S95). ).

  When the processor 22 determines that the processing capability of the voice input / output device 100 on the client side is sufficient (step S95 / YES), the processor 22 performs processing (step S96), and the voice input / output device on the client side When it is determined that the processing capacity of 100 is not sufficient (step S96 / NO), the processing is performed on the cloud side (step S97), and the process proceeds to step S98.

  The processor 22 determines whether or not there is a mode setting signal (step S98), and if it is determined that there is a mode setting signal from the outside (step S98 / YES), the call center mode, the reception mode, the conference mode,. If any mode is set (step S99) and it is determined that there is no mode setting signal from the outside (step S98 / NO), the process proceeds to step S14 (see FIG. 7).

  By the processing in step S92, useless processing is omitted and efficiency is improved.

  The voice input / output device 100 according to the present invention described above is realized by a control program that causes a computer to execute processing. As an example, the case where the function of the present invention is realized by a program will be described below.

A computer-readable program,
Computer
Input means for inputting voice information related to the voice;
Pre-stage processing means for performing pre-stage processing that facilitates identification processing on the input voice information;
Predetermined processing is performed on the audio information processed by the preceding processing means, task processing is performed based on the first information, and if the evaluation of the task processing is not sufficient, the first information is corrected and the evaluation becomes sufficient Optimization means to optimize by repeating a series of processing until
A program of a voice information processing system for functioning as

  Thus, the voice input / output device 100 according to the present invention can be realized anywhere as long as there is a computer environment capable of executing the program.

  Such a program may be stored in a computer-readable recording medium.

<Recording medium>
Here, examples of the recording medium include computer-readable recording media such as CD-ROM, flexible disk (FD), and CD-R, semiconductor memories such as flash memory, RAM, ROM, and FeRAM, and HDD.

  CD-ROM is an abbreviation for Compact Disc Read Only Memory. The flexible disk means a flexible disk. CD-R is an abbreviation for CD Recordable. FeRAM is an abbreviation for Ferroelectric RAM and means a ferroelectric memory. HDD is an abbreviation for Hard Disc Drive.

  The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there.

  The present invention is not only capable of knowing the situation of the object in detail based mainly on information by voice, but also making it possible to recognize matters that are expected to occur in the future. It is applicable to the scene that needs to grasp the situation mainly by voice.

10 ... Network 20 ... Cloud side server 21 ... Database (DB)
22 ... Processor 23 ... Output device 24 ... Input device 26 ... Interface 100 ... Voice input / output device 100
DESCRIPTION OF SYMBOLS 101 ... Housing 102 ... LED ring 103-1 to 103-16 ... PDM microphone 104 ... Speaker group 104S ... Squawker 104T ... Tweeter 105 ... Reflector plate 106 ... Power lamp 107 ・ ・ ・ Power cord 201 ・ ・ ・ Expansion unit 202 ・ ・ ・ Storage unit 203 ・ ・ ・ Microphone unit 204 ・ ・ ・ Mic control unit 205 ・ ・ ・ Signal processing unit 206 ・ ・ ・ Communication unit 207 ・ ・ ・Sound generation unit 208 ・ ・ ・ Non-audible sound generation unit 209 ・ ・ ・ Display unit

Claims (18)

An input means for inputting voice information about the voice;
Pre-stage processing means for performing pre-stage processing for facilitating identification processing on the input voice information;
Predetermined processing is performed on the audio information processed by the pre-processing unit, task processing is performed based on the first information, and the first information is corrected when the evaluation of the task processing is not sufficient, and the evaluation An audio information processing system comprising: optimization means for optimizing by repeating a series of processes until the signal becomes sufficient. The optimization means includes
First evaluation means for evaluating a result of the task processing;
Correction means for correcting the first information when the evaluation is not sufficient;
The speech information processing system according to claim 1, further comprising: a repeating unit that repeats a series of processing from the preceding processing unit to the correcting unit.   When analyzing and responding to the contents of the voice in the room, if the processing capability of the voice input / output device on the client side is compatible, the voice input / output device performs information processing, and the processing capability of the voice input / output device The voice information processing system according to claim 1, further comprising a determination unit that performs information processing on the cloud side when the information cannot be handled.   The audio information processing system according to claim 1, further comprising an external system that manages indoor environment settings, intentional interpretation, and dialogue.   The external system includes an imaging unit that images the outside of the voice input / output device, a vibration unit that vibrates the casing, a rotating unit that rotates the casing, and a projection unit that projects an image on a wall outside the casing. The speech information processing system according to claim 1, comprising at least one of the following.   The audio information processing system according to claim 1, wherein external contents can be used for intentional interpretation and management of the dialogue. An inference means for performing an information processing process including an analysis process, an analysis process, and a recognition process of the content of the voice, and inferring an attribute including a speaker's age and sex;
A collecting means for collecting logs used when designing the second information;
Analyzing means for analyzing the log;
Second evaluation means for evaluating the response and the second information, and
The voice information processing system according to claim 1, wherein the voice information processing system is optimized through continuous improvement based on the evaluation. The inference means is
Interpretation means for intentionally interpreting the conversation with the speaker;
The voice information processing system according to claim 7, further comprising management means for managing a dialogue with the speaker. The environment judgment means includes
Size determining means for determining the size of the room;
Noise level recognition means for recognizing the noise level in the room;
The sound information processing system according to claim 4, further comprising reverberation level recognition means for recognizing the reverberation level in the room.   The audio information processing system according to claim 5, further comprising an image display unit provided in the housing for displaying an image.   6. The voice information processing system according to claim 5, further comprising fingerprint authentication means provided in the housing for recognizing a user.   The processing capability of the voice input / output device on the client side is the processing speed of the processor, memory size, sensor type, microphone array, number of speakers, number of LEDs, built-in camera, number of application software, front that can support other companies' devices The voice information processing system according to claim 1, further comprising end signal processing. Feature extraction means for extracting speech from the voice of the speaker as a feature;
Speaker identification means for identifying the speaker by storing the characteristic in association with the speaker information, and comparing the newly input voice characteristic with the speaker information stored in the storage means; The voice information processing system according to claim 1, wherein:   8. The voice information processing system according to claim 7, further comprising emotion identifying means for identifying the speaker's emotion.   It is provided on the outer periphery of the housing, turns around so that a part of the emission color on the orbit is different from the emission color of the remaining part, and when the speaker is detected, the part of the emission color 15. The voice information processing system according to claim 14, further comprising light emitting means for emitting light so as to stop in a direction. Enter audio information about the audio,
For the input voice information, perform pre-processing that facilitates identification processing,
Apply predetermined processing to the audio information processed in the previous stage, perform task processing based on the first information, modify the first information when the evaluation of the task processing is not sufficient, and the evaluation is sufficiently performed A control method for a speech information processing system, characterized by optimizing by repeating a series of processes until A computer-readable program,
Computer
Input means for inputting voice information related to the voice;
Pre-stage processing means for performing pre-stage processing for facilitating identification processing on the input voice information,
Predetermined processing is performed on the audio information processed by the pre-processing unit, task processing is performed based on the first information, and the first information is corrected when the evaluation of the task processing is not sufficient, and the evaluation Optimization means to optimize by repeating a series of processes until the
Program for voice information processing system to function as   A recording medium on which the program according to claim 17 is recorded.

Images