JP2018180424A - Speech recognition apparatus and speech recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a "voice recognition device and a voice recognition method" capable of targeting only the spoken voice of a specific occupant for triggerless voice recognition without performing complicated registration work of a voice profile. SOLUTION: In a voice recognition device 100 capable of performing voice recognition of voice input from a microphone 200 installed in a vehicle, a face photograph of a user to be voice-recognized and a target to be voice-recognized. When at least one of the seats is registered as a recognition target, the image taken by the camera installed in the car is recognized, and it is determined that the mouth of the occupant corresponding to the registered recognition target is moving. By recognizing the voice input from the microphone 200 installed in the vehicle, it is possible to target only the spoken voice of a specific occupant to the target of triggerless voice recognition without performing complicated registration work of the voice profile. I made it possible. [Selection diagram] Fig. 1

Description

  The present invention relates to a voice recognition device and a voice recognition method, and is particularly suitable for use in a voice recognition device that does not require a trigger such as an operation of a speech button or a specific operation.

  The vehicle is equipped with various electronic devices such as an audio device, an air conditioner, and a navigation device. In addition, there is also provided a system in which the operation of the electronic device can be performed by voice recognition in order to avoid one-hand operation or the like when operating these electronic devices. Using this voice recognition technology, the driver can operate various electronic devices without releasing his / her hand from the steering wheel (without manually operating the operation unit such as the remote controller or the operation panel).

  The speech recognition device usually recognizes a specific word or phrase uttered by the user, a simple command sentence, etc. (hereinafter, these are collectively referred to as "word") as a speech command. The electronic device performs control in accordance with the word (utterance command) recognized by the speech recognition device. The speech recognition apparatus compares the speech pattern of each recognition target word registered in the speech recognition dictionary with the speech of the user to search for a speech pattern having the highest similarity to the speech, and the speech pattern The word associated with is recognized as the word of the utterance voice.

  In the conventional voice recognition apparatus, when the user presses an equipped speech button, the speech recognition mode is set, and the user's speech voice input from the microphone is recognized to execute a command. It is also known that the voice recognition mode is set to be triggered by a specific operation such as hitting a hand instead of the operation of the speech button. Recently, voice recognition (hereinafter referred to as triggerless voice recognition) has been provided which eliminates the need for a trigger such as an operation of a speech button or a specific operation at the time of voice recognition. In triggerless speech recognition, the microphone is always on, the input speech is identified, and it is determined whether the word corresponds to a speech command.

  In addition, when there are a plurality of occupants in the car, speech voices of the plurality of occupants may be simultaneously input to the microphone. In such a case, in consideration of the possibility of causing a malfunction, a plurality of microphones and an imaging device for determining the presence or absence of an occupant of the in-vehicle seat from the in-vehicle image are provided, and voice input and imaging from the plurality of microphones A voice recognition system has been devised in which the seat position of a person who has input a voice is specified from the seating information from the device (see, for example, Patent Document 1).

  Further, when there are a plurality of occupants in the car, the microphone is always on in triggerless speech recognition, so conversational voices between the occupants are also input from the microphone in addition to the speech for speech recognition. On the other hand, when the driver talks with a passenger, the size and change of the driver's voice, the height of the voice, the difference in the size of the voice from the noise section, the speech start part or the speech end part The driver's profile data is created using voice characteristics, vocalization speed, etc., and after detecting the operation of the talk switch by the driver, the driver's vocalization is made by comparing the driver's condition with the profile data. There is known a speech recognition apparatus that determines whether the speech is directed to a passenger or a speech input directed to an in-vehicle system, and selectively recognizes speech for the speech input directed to the in-vehicle system (see See, for example, Patent Document 2).

  In addition, first to third occupant intention recognizing means for recognizing the intention of the occupant based on the sound emitted from the occupant, the movement of the lip of the occupant, and the gesture of the occupant are provided, and the first to third occupants are provided. There is also known a control device for a vehicle that controls the operation of a vehicle-mounted device in accordance with the recognition result of the intention recognition means (see, for example, Patent Document 3).

JP 2004-354930 A JP 2008-250236 A JP, 2011-232637, A

  When there are a plurality of occupants in the vehicle interior, the microphone is always on in triggerless voice recognition, and therefore the voices uttered by any occupant are also input from the microphone and are considered as voice recognition targets. Therefore, there is a problem that it is not possible to set only the voice of a specific occupant as the target of triggerless speech recognition.

  Note that voiceprints and characteristics of speaking style related to a specific passenger are registered as a voice profile, and only the voice of a specific passenger is targeted for voice recognition by comparing the voice input from the microphone with the voice profile. Is considered. However, in this case, it is necessary to create and register a voice profile for each word to be recognized as a speech command, and the registration work becomes extremely complicated.

  The present invention has been made to solve such a problem, and it is possible to make only a specific occupant's speech voice a target of triggerless speech recognition without performing complicated registration work of the voice profile. The purpose is to

  In order to solve the above-described problems, in the present invention, at least one of a face photograph of a user who is a target of voice recognition and a seat which is a target of the voice recognition is registered as a recognition target. Recognizes the image taken by the installed camera, and when it is determined that the passenger's mouth corresponding to the registered recognition object is moving, to recognize the voice input from the microphone installed in the car I have to.

  According to the present invention configured as described above, if a face picture or a seat is registered as an object to be subjected to voice recognition, a specific occupant to which a face picture is registered among voices constantly input from the microphone, or Only voice input from the microphone when it is determined that the mouth of a specific occupant sitting in the registered seat is moving is subjected to voice recognition processing. As a result, it is possible to make only a specific occupant's uttered voice a target of triggerless voice recognition without performing complicated registration work of the voice profile.

It is a block diagram showing an example of functional composition of a speech recognition device by this embodiment. It is a figure which shows the structural example of the vehicle-mounted system which applies the speech recognition apparatus of this embodiment. It is a figure which shows the face photography selection screen displayed by the recognition object registration part of this embodiment. It is a figure which shows the seat selection screen displayed by the recognition object registration part of this embodiment. It is a flowchart which shows the operation example of the speech recognition apparatus by this embodiment. It is a flowchart which shows the operation example of the image recognition part by this embodiment. It is a flowchart which shows the operation example of the speech recognition control part by this embodiment.

  Hereinafter, an embodiment of the present invention will be described based on the drawings. FIG. 1 is a block diagram showing an example of a functional configuration of the speech recognition apparatus 100 according to the present embodiment. FIG. 2 is a view showing an example of the configuration of an in-vehicle system to which the speech recognition apparatus 100 of the present embodiment is applied. The in-vehicle system is a system mounted on a vehicle, and the voice recognition device 100 of this embodiment, the microphone 200, the front seat camera 301 (corresponding to the camera in claims), the rear seat camera 302 (claims) And the touch panel 400 are configured.

  As shown in FIG. 2, the speech recognition apparatus 100 of the present embodiment is installed on a center console or a dashboard of a vehicle. The microphone 200 is installed near the front seat of the vehicle. The microphone 200 can pick up voices of both the occupant seated in the front seat and the occupant seated in the rear seat. The front seat camera 301 is provided at a position where it can capture the face of an occupant seated in the front seat, such as a ceiling near the front seat. The rear seat camera 302 is provided at a position where it can capture the face of an occupant seated in the rear seat, such as a ceiling near the rear seat. The touch panel 400 is provided at a position at which at least an occupant of the front seat, such as a dashboard of a vehicle, can perform touch operation.

  The speech recognition apparatus 100 according to the present embodiment recognizes the speech of an occupant (a word such as a specific word, a phrase or a simple command sentence) input from the microphone 200 as a speech command, and is mounted on a vehicle based on the recognition result. The electronic device 500 is controlled. The electronic device 500 to be controlled by the voice recognition device 100 is, for example, a navigation device, an audio device, an air conditioner, or the like.

  As shown in FIG. 1, the voice recognition apparatus 100 of this embodiment has a list registration unit 11 (corresponding to an occupant list registration unit in the claims), a recognition target registration unit 12, and an image recognition unit 13 as its functional configuration. The voice processing unit 14, the voice recognition unit 15, the electronic device control unit 16, and the voice recognition control unit 17 are provided. The voice recognition device 100 further includes an occupant list storage unit 21, an in-vehicle configuration list storage unit 22, and a recognition target storage unit 23 as storage media.

  Each of the functional blocks 11 to 17 can be configured by any of hardware, a DSP (Digital Signal Processor), and software. For example, when configured by software, each of the functional blocks 11 to 17 actually comprises a CPU, a RAM, a ROM and the like of a computer, and a program stored in a storage medium such as a RAM, a ROM, a hard disk or a semiconductor memory Is realized by operating.

  The list registration unit 11 registers face photographs of a plurality of users as the occupant list 21a. The occupant list 21a is a list having information (hereinafter referred to as "user identification information") for identifying a plurality of users in association with a user's face picture. The face photograph of one user means image data generated by photographing the face of the one user. As described later, since a face picture is used for user's face recognition, it is desirable that the face picture is image data generated by photographing the user's face from the front.

  The occupant list storage unit 21 stores an occupant list 21a. In the present embodiment, “registering the user's face photograph as the occupant list 21a” means registering the combination of the user identification information and the face photograph in the occupant list 21a stored by the occupant list storage unit 21. Do.

  In the present embodiment, the list registration unit 11 performs the following processing to register a face picture. That is, a face picture (image data) is stored in advance in a predetermined folder of the speech recognition apparatus 100. Then, the list registration unit 11 receives a user identification information in response to a user's instruction on the touch panel 400, and a user interface capable of designating one of the face photographs stored in the predetermined folder. Is displayed on the touch panel 400. Hereinafter, the user who makes an input to the user interface is referred to as an “input person”. The input person inputs the user identification information of the user to be registered using the user interface, and specifies the facial photograph of the user to be registered among the facial photographs stored in the predetermined folder. The designation of the face photograph is performed, for example, by designation of the file name of the face photograph. The list registration unit 11 acquires the input user identification information and acquires the designated face photograph. Next, the list registration unit 11 registers the user identification information and the face picture in the occupant list 21 a stored in the occupant list storage unit 21 in association with each other. For example, the owner of the vehicle is a user who is likely to get on the vehicle and registers in the occupant list 21a in advance for a user who is permitted to be a target of voice recognition.

  The process of the list registration unit 11 described above is an example, and another process may be performed to register a face picture. For example, when acquiring a face picture, the list registration unit 11 may execute the following process. That is, for a user whose face picture has not been registered, a face is photographed by the front seat camera 301 or the rear seat camera 302 to generate image data. Then, the list registration unit 11 may be configured to acquire image data generated based on the photographing result of the front seat camera 301 or the rear seat camera 302 as a face picture. In this case, it is not necessary for the user to store the face photograph in the predetermined folder described above in advance, and the fact that the front seat camera 301 and the rear seat camera 302 are mounted on the vehicle is used. A face photograph of the passenger who is on board can be registered as appropriate. Also, for example, the list registration unit 11 may communicate with a communicable external device, acquire information necessary for registering a facial photograph from the external device, and register the facial photograph based on the acquired information.

  In addition, the list registration unit 11 registers a plurality of seats existing in the vehicle as the in-vehicle configuration list 22a. The in-vehicle configuration list 22a is a list having information (hereinafter referred to as "seat position information") indicating the position of a seat in the vehicle for each of a plurality of seats present in the vehicle.

  As shown in FIG. 1, the in-vehicle configuration list storage unit 22 stores an in-vehicle configuration list 22a. In the present embodiment, "registering the seat as the in-vehicle configuration list 22a" registers seat position information of each of a plurality of seats existing in the vehicle in the in-vehicle configuration list 22a stored in the in-vehicle configuration list storage unit 22. It means that. As an example, the list registration unit 11 displays a user interface for inputting seat position information on the touch panel 400, and based on the input to the user interface, the seat position information of each of a plurality of seats present in the vehicle The in-vehicle configuration list 22a stored by the list storage unit 22 is registered.

  The position of the seat in the vehicle is basically fixed depending on the type of the vehicle. Based on this, an in-vehicle configuration list 22a having contents corresponding to the vehicle type of the vehicle may be registered in advance by a company providing the speech recognition apparatus 100, a company providing the vehicle, or the like. In addition, the list configuration registration unit 11 may perform the following processing regarding the registration of the in-vehicle configuration list 22b. That is, the list registration unit 11 displays a user interface for inputting a vehicle type of a vehicle on the touch panel 400, and allows the user to input the vehicle type of the vehicle. The list registration unit 11 acquires the vehicle type input to the user interface, and acquires the in-vehicle configuration list 22a corresponding to the acquired vehicle type. For example, a database having an in-vehicle configuration list 22a having contents corresponding to a vehicle type is stored in the speech recognition apparatus 100 for each vehicle type, and the list registration unit 11 selects an in-vehicle configuration list 22a corresponding to the vehicle type input by the user from the database. To get Also, for example, the list registration unit 11 communicates with an external device (for example, a server on the Internet) having the in-vehicle configuration list 22a for each vehicle type, and acquires the in-vehicle configuration list 22a from the external device. The list registration unit 11 stores the acquired in-vehicle configuration list 22a in the in-vehicle configuration list storage unit 22.

  The recognition target registration unit 12 performs any of the following processing according to the operation modes M1 to M3. That is, in the case of the operation mode M1, the recognition target registration unit 12 registers, as a recognition target, a face picture of a user who is a target of voice recognition. Further, in the case of the operation mode M2, the recognition target registration unit 12 registers, as a recognition target, a seat to which voice recognition is to be performed. Further, in the case of the operation mode M3, the recognition target registration unit 12 registers, as a recognition target, a seat to which voice recognition is to be performed, in addition to a face photograph of the user to which voice recognition is to be performed. As described later, in the present embodiment, the user corresponding to the information registered as the recognition target by the recognition target registration unit 12 is the target of the speech recognition, and the speech recognition by the speech recognition unit 15 is performed while the user speaks. Is done.

  The processing of the recognition target registration unit 12 will be described below for each of the operation modes M1 to M3. The user can select an operation mode by a predetermined means. For example, the user can select an operation mode by performing a touch operation on the touch panel 400.

(Processing of recognition target registration unit 12 in operation mode M1)
In the operation mode M1, based on the occupant list 21a stored in the occupant list storage unit 21, the recognition target registration unit 12 registers, as a recognition target, a face photograph of a user who is to be subjected to voice recognition.

  More specifically, when registering recognition targets, the recognition target registration unit 12 acquires the occupant list 21 a stored by the occupant list storage unit 21. Next, the recognition target registration unit 12 displays the face photo selection screen G1 on the touch panel 400 based on the acquired occupant list 21a.

  FIG. 3 is a view showing an example of the face picture selection screen G1. As shown in FIG. 3, the face picture selection screen G1 has a block BL1 for each user registered in the occupant list 21a. The user identification information N1 is displayed on the left side of each block BL1, and the face photo image F1 is displayed on the right side. The facial photograph image F1 is a thumbnail image of a facial photograph (image data). In each block, a check box CK1 is displayed. The user can check the check box CK1 or can uncheck the check box by touch-operating the check box CK1. In addition, the face picture selection screen G1 has a determination button BK1 for determining an input on the screen. Hereinafter, a person who operates the face picture selection screen G1 is referred to as an “operator”. A person who operates a seat selection screen G2 described later is similarly expressed as "operator".

  The operator checks the check box CK1 corresponding to each user for one or a plurality of users who are to be subjected to voice recognition, and then performs the touch operation on the confirmation button BK1. When it is detected that the confirm button BK1 has been touched, the recognition target registration unit 12 acquires a face picture corresponding to the checked check box CK1. Next, the recognition target registration unit 12 registers the acquired face picture (a face picture of the user who is the target of voice recognition) in the recognition target storage unit 23. As a result, the recognition target storage unit 23 stores the face photographs of the user who is the target of the speech recognition.

  As described above, in the operation mode M1, the recognition target registration unit 12 registers, as a recognition target, a face photograph of the user selected from the occupant list 21a by the user operation. The operator can use a simple operation of touching the face picture selection screen G1 to select the user, so that a specific occupant among the occupants who get on the vehicle can be subjected to the voice recognition.

(Processing of recognition target registration unit 12 in operation mode M2)
In the operation mode M2, based on the in-vehicle configuration list 22a stored in the in-vehicle configuration list storage unit 22, the recognition target registration unit 12 registers a seat to be subjected to voice recognition as a recognition target.

  More specifically, when registering the recognition target, the recognition target registration unit 12 acquires the in-vehicle configuration list 22a stored in the in-vehicle configuration list storage unit 22. Next, the recognition target registration unit 12 causes the touch panel 400 to display the seat selection screen G2 based on the acquired in-vehicle configuration list 22a.

  FIG. 4 is a diagram showing an example of the seat selection screen G2. As shown in FIG. 4, the seat selection screen G2 has an image in which the inside of the vehicle is schematically represented in a state in which a seat on which a passenger can sit is clearly indicated. A check box CK2 is displayed on the seat selection screen G2 in association with each seat. Further, the seat selection screen G2 has a determination button BK2 for determining an input to the screen.

  The operator checks the check box CK2 corresponding to each seat for one or more seats for which voice recognition is to be performed, and then performs a touch operation on the confirmation button BK2. The phrase "target seat for speech recognition" means that the user seated on the seat is targeted for speech recognition. When it is detected that the confirm button BK2 has been touched, the recognition target registration unit 12 acquires seat position information of a seat corresponding to the checked check box CK2. Next, the recognition target registration unit 12 registers the acquired seat position information (information indicating the position of the seat to which voice recognition is performed) in the recognition target storage unit 23. The process of registering the seat position information in the recognition target storage unit 23 corresponds to the process of registering a seat which is a target of voice recognition as a recognition target. As a result, in the recognition target storage unit 23, the seat position information of each seat to be subjected to the voice recognition is stored.

  As described above, in the operation mode M2, the recognition target registration unit 12 registers the seat selected by the user operation from the in-vehicle configuration list 22a as the recognition target. The operator can use the seat selection screen G2 as a touch operation to designate a seat, thereby making it possible to make an occupant, who is seated in a specific seat among the seats present in the vehicle, an object of voice recognition.

(Processing of recognition target registration unit 12 in operation mode M3)
In the operation mode M3, the recognition target registration unit 12 uses the occupant list 21a stored by the occupant list storage unit 21 and the in-vehicle configuration list 22a stored in the in-vehicle configuration list storage unit 22 to perform voice recognition for the user. In addition to the face picture, a seat to be subjected to voice recognition is registered as a recognition target.

  That is, in the operation mode M3, the recognition target registration unit 12 causes the touch panel 400 to display the face picture selection screen G1 and sets the face picture of the user who is the target of voice recognition as the recognition target based on the user operation on the screen. It is registered in the recognition target storage unit 23. Next, the recognition target registration unit 12 causes the touch panel 400 to display the seat selection screen G2, and based on a user operation on the screen, the recognition target storage unit sets a seat (seat position information) to be a target for voice recognition. Register at 23 As a result, the recognition target storage unit 23 stores a face picture of each user who is the target of voice recognition, and stores seat position information of each of the seats that are the target of voice recognition. Become.

  As described above, in the operation mode M3, the recognition target registration unit 12 registers the seat selected by the user operation as the recognition target in addition to the face photograph selected by the user operation. As described later, when the user corresponding to the face photograph selected by the user operation sits on the seat selected by the user operation, the user is a target for performing voice recognition and the user utters In the meantime, processing related to speech recognition is performed. The operator recognizes a specific occupant seated in a specific seat as a voice recognition target by a simple operation of touching the facial photograph selection screen G1 and the seat selection screen G2 to select a facial photograph and a seat. Can.

  The process of the recognition target registration unit 12 has been described above for each of the operation modes M1 to M3. In the above description, the face picture or the seat is selected by the touch operation of the screen by the operator. In this case, the user's touch operation on the screen corresponds to "user operation". In this regard, for example, the selection may be performed by gesture input. As an example, an infrared sensor used for recognizing a gesture is provided in the voice recognition apparatus 100, and the recognition target registration unit 12 appropriately recognizes a gesture performed by the user based on a detection value of the infrared sensor. The recognition of the gesture may be performed based on the imaging results of the front seat camera 301, the rear seat camera 302, and the like. At the time of face photograph registration, the recognition target registration unit 12 displays on the touch panel 400 a user interface capable of making a gesture to select a user who registers the face picture. For example, the recognition target registration unit 12 displays the face picture selection screen G1 shown in FIG. 3 on the touch panel 400. Then, the recognition target registration unit 12 detects a gesture by the operator, and acquires a face photograph of the user selected by the operator based on the detected gesture. For example, in the case of displaying the face picture selection screen G1 as a user interface, a gesture for selecting a user to be a target for checking or unchecking the check box CK1 is checked as a gesture. Alternatively, a gesture for removing the check from the check box CK1 and a gesture for operating the confirmation button BK1 are set in advance. Then, the recognition target registration unit 12 recognizes the user selected by the operator based on the recognized gesture, and registers the face photograph of the selected user. Also at the time of seat registration, the recognition target registration unit 12 executes the same process. In this case, the gesture performed by the user corresponds to "user operation".

  The image recognition unit 13 recognizes the images taken by the front seat camera 301 and the rear seat camera 302 installed in the car, and the mouth of the occupant corresponding to the recognition target registered by the recognition target registration unit 12 moves. Determine if there is. At the time of the determination, the image recognition unit 13 executes different processing according to the operation mode. Hereinafter, the processing of the image recognition unit 13 will be described in detail for each of the operation modes M1 to M3.

(Process of image recognition unit 13 in operation mode M1)
In the operation mode M1, the image recognition unit 13 recognizes the face image of the occupant taken by the front seat camera 301 and the rear seat camera 302, and the same occupant's mouth as the user whose face picture is registered as the recognition object moves. It is determined whether the

  In the present embodiment, in the operation mode M1, the image recognition unit 13 executes the following processing. That is, the front seat camera 301 shoots an area corresponding to the front seat of the vehicle at a predetermined cycle, and outputs image data to the image recognition unit 13 based on the shooting result. Similarly, the rear seat camera 302 shoots an area corresponding to the rear seat of the vehicle at a predetermined cycle, and outputs image data to the image recognition unit 13 based on the shooting result. Hereinafter, the front seat camera 301 and the rear seat camera 302 may be collectively referred to simply as “camera”. Further, the image data that the front seat camera 301 and the rear seat camera 302 output based on the shooting result is expressed as “shot image data”.

  Then, the image recognition unit 13 analyzes the photographed image data according to the input of the photographed image data from the camera, and recognizes the face image included in the data. The face image recognized here is an image of the face of the vehicle occupant. Face image recognition is suitably performed using existing face recognition techniques. Next, the image recognition unit 13 acquires the face photograph stored in the recognition target storage unit 23 and, among the recognized face images, the face photograph registered in the recognition target storage unit 23 (a user who is a target for performing voice recognition) It is determined whether a face image having the same identity as that of The determination is appropriately performed using existing face recognition technology such as face recognition technology that applies pattern matching using a feature point of an image of a face part in a face photograph as a template. If the recognized face image has the same identity as the face picture registered in the recognition target storage unit 23, the occupant corresponding to the face image is the same as the user corresponding to the face picture registered in the recognition target storage unit 23. It is a person, and it is a person who is the target of speech recognition. Hereinafter, among the occupants who get on the vehicle, the occupants who are the target of voice recognition are referred to as “recognition target occupants”.

  When the captured image data includes the face image of the recognition target occupant, the image recognition unit 13 tracks and analyzes the face image of the recognition target occupant included in the captured image data input from the camera in a predetermined cycle. It is determined whether the target occupant's mouth is moving.

  In the present embodiment, the image recognition unit 13 performs the following processing to determine whether the mouth of the recognition target occupant is moving. That is, the image recognition unit 13 identifies a mouth area (hereinafter referred to as "mouth area") in the face image of the recognition target occupant included in the photographed image data, and tracks and analyzes the mouth area. Based on the result of analysis of the mouth area, the image recognition unit 13 determines that the mouth is moving when the mouth is moving continuously for a predetermined time or more, and the mouth continues for a predetermined time or more. If it is not moving (the mouth is closed or is open), it is determined that "the mouth is not moving". Here, human beings are characterized by moving their mouth continuously when speaking. Therefore, when the recognition target occupant's mouth is moving continuously, there is a high possibility that the recognition target occupant is speaking. Based on this, the image recognition unit 13 determines whether or not the recognition target occupant is speaking by determining whether or not the mouth of the recognition target occupant is moving. In addition, since it is a process performed in order to determine whether it is the state which is uttering whether the determination of whether the mouth is moving is performed, for example, a predetermined process is performed and the mouth at the time of eating The movement of the mouth or the movement of the mouth which is different from the movement of the mouth such as the movement of the mouth while smiling may be determined as "the mouth is not moving", and the accuracy of the determination may be raised Of course.

  If the image recognition unit 13 determines that the mouth of the recognition target occupant is moving, the image recognition unit 13 notifies the voice recognition control unit 17 that the mouth of the recognition target occupant is moving. In the present embodiment, at the start of a series of successive mouth movements, the image recognition unit 13 notifies that the mouth movements have been started (hereinafter referred to as “operation start notice”), and at the end, the mouth Notification that the movement of the movement has been stopped (hereinafter referred to as “operation end notification”). In addition, since the image recognition unit 13 determines that the mouth is moving when the mouth moves continuously for a predetermined time (for example, 0.5 seconds) or more, the timing when the mouth actually starts to move is determined. There is a time lag between when the operation start notification is made. As described later, the voice recognition control unit 17 executes a process corresponding to the time lag.

(Process of image recognition unit 13 in operation mode M2)
In the operation mode M2, the image recognition unit 13 recognizes the face image of the occupant photographed by the front seat camera 301 and the rear seat camera 302, and the occupant's mouth in the seat registered as the recognition target is moving or not. Determine if

  In the present embodiment, in the operation mode M2, the image recognition unit 13 executes the following processing. That is, the image recognition unit 13 acquires seat position information stored in the recognition target storage unit 23. Next, the image recognition unit 13 executes the following processing based on the acquired seat position information and photographed image data input from the camera. That is, the image recognition unit 13 recognizes the face image included in the photographed image data, and specifies the face image of the occupant seated in the seat at the position indicated by the seat position information among the face images. Here, the occupant corresponding to the identified face image is a recognition target occupant. Next, the image recognition unit 13 tracks and analyzes the face image of the recognition target occupant in the photographed image data, and determines whether or not the mouth of the recognition target occupant is moving. The image recognition unit 13 appropriately sends the above-described operation start notification and operation end notification to the voice recognition control unit 17 according to the determination as to whether or not the mouth of the recognition target occupant is moving.

(Process of image recognition unit 13 in operation mode M3)
In the operation mode M3, the image recognition unit 13 recognizes the face image of the occupant photographed by the front seat camera 301 and the rear seat camera 302, and the occupant in the seat registered as the recognition target is a face photograph as the recognition target Is a registered user, and it is determined whether or not the passenger's mouth is moving.

  In the present embodiment, in the operation mode M3, the image recognition unit 13 executes the following processing. That is, the image recognition unit 13 acquires a face picture and seat position information stored in the recognition target storage unit 23. Next, the image recognition unit 13 executes the following processing based on the acquired face photograph and seat position information, and the photographed image data input from the camera. That is, the image recognition unit 13 recognizes the face image included in the photographed image data, and specifies the face image of the occupant seated in the seat at the position indicated by the seat position information among the face images. Next, the image recognition unit 13 determines whether or not the specified face image (a face image of an occupant sitting on a seat at a position indicated by the seat position information) has the same identity as the acquired face picture. The occupant corresponding to the face image determined to have the same identity as the acquired face photograph is a user who is seated on the seat registered as the recognition target and corresponds to the face photograph registered as the recognition target, and the voice It is a person to be recognized. That is, the said passenger | crew is a "recognition object passenger | crew."

  Next, the image recognition unit 13 tracks and analyzes the face image of the recognition target occupant in the photographed image data, and determines whether or not the mouth of the recognition target occupant is moving. The image recognition unit 13 appropriately sends the above-described operation start notification and operation end notification to the voice recognition control unit 17 according to the determination as to whether or not the mouth of the recognition target occupant is moving.

  The audio processing unit 14 performs necessary audio processing such as A / D conversion on the audio collected by the microphone 200 to generate an audio signal, and performs buffering in the buffer 31. The buffer 31 is a storage area formed in volatile memory such as RAM. As a result of the processing by the audio processing unit 14, audio signals based on the audio collected by the microphone 200 during a predetermined period retroactively from the current time are accumulated in the buffer 31. Hereinafter, a set of audio signals stored in the buffer 31 is referred to as "recorded audio data".

  The voice recognition unit 15 recognizes the voice input from the microphone 200 installed in the car. In the present embodiment, the voice recognition unit 15 sequentially reads the recorded voice data stored in the buffer 31, and whether or not the voice recorded in the recorded voice data includes a word (utterance command) registered in advance. Determine In the present embodiment, the word is a character string or the like that causes the electronic device 500 to execute a specific process corresponding to the word. As a word, for example, when the electronic device 500 is a navigation device, a word for executing processing of displaying the current position of a vehicle (for example, “display current position”) or a word for starting guidance of a route to a destination ("Navi start" as an example) is registered in advance. The above-described determination by the voice recognition unit 15 is appropriately performed using existing voice recognition technology, such as comparison between a voice pattern of a word registered in advance and a voice pattern of voice recorded in recorded voice data. If the speech recognition unit 15 determines that a word is included, the speech recognition unit 15 notifies the electronic device control unit 16 of the word.

  When the word is notified from the speech recognition unit 15, the electronic device control unit 16 outputs, to the electronic device 500, a control signal that causes the electronic device 500 to execute the process corresponding to the notified word.

  The voice recognition control unit 17 controls the voice recognition unit 15 to perform processing when the image recognition unit 13 determines that the occupant's mouth corresponding to the recognition target is moving.

  In the present embodiment, the voice recognition control unit 17 controls the voice recognition unit 15 based on the operation start notification and the operation end notification by the image recognition unit 13. More specifically, when notified of the start of operation, the voice recognition control unit 17 causes the voice recognition unit 15 to determine a predetermined time from the timing when the operation start notification is received for the voice recording voice data stored in the buffer 31 (for example, The reading is performed from the position before 0.5 seconds) to start speech recognition (determination as to whether or not the speech contains a word). Furthermore, when the voice recognition control unit 17 receives the operation end notification, the voice recognition unit 15 reads the voice recording data and reads the voice based on the read voice data to a position corresponding to the timing when the operation termination notification is received. After recognition is performed, speech recognition is stopped. As a result, the voice recognition unit 15 recognizes the voice collected by the microphone 200 in a period from the timing when the operation start notification is given to the timing when the operation end notification is given from the timing before the predetermined timing. To be done. As described above, the period from the operation start notification to the operation end notification is a period determined by the image recognition unit 13 that the mouth of the recognition target occupant is moving. Based on this, in the present embodiment, the voice recognition unit 15 is used for the voice collected by the microphone 200 during a period in which the mouth of the recognition target occupant is moving (a period in which the probability of the recognition target occupant speaking is high). Speech recognition is performed.

  The reason why the voice recognition control unit 17 causes the voice recognition unit 15 to read the recorded voice data stored in the buffer 31 from a position ahead of the current time by a predetermined time is as follows. That is, as described above, a time lag occurs between the timing when the recognition target occupant's mouth actually starts moving and the timing when the image recognition unit 13 performs the operation start notification. Then, by causing the voice recognition unit 15 to read the recorded voice data from a position preceding the current time (at the time when the operation start notification is made) by a predetermined time, the recognition target occupant is caused by the time lag. Can be prevented from leaking out from the target of speech recognition by the speech recognition unit 15.

  In addition, the voice recognition control unit 17 controls the voice recognition unit 15 during a period until the next operation start notification is received after the image recognition unit 13 notifies the operation completion, and the voice recognition unit 15 stops voice recognition. Do. The period from the operation end notification to the next operation start notification is a period determined by the image recognition unit 13 that the mouth of the recognition target occupant has not moved.

  As described above, in the present embodiment, while the voice collected by the microphone 200 during the period when the recognition target occupant's mouth is moving is the target of voice recognition by the voice recognition unit 15, the period other than that period The voice collected by the microphone 200 is not considered as a target of voice recognition by the voice recognition unit 15. This produces the following effects.

  That is, as described above, when the recognition target occupant's mouth is moving, the probability that the recognition target occupant is speaking is high. Then, according to the present embodiment, it is possible to set only the speech of the occupant who is instructed to be the target of voice recognition, which is the recognition target occupant, as the target of the voice recognition. Conventionally, in order to target only speech voices of specific occupants as the target of speech recognition, it was necessary to create and register speech profiles for each word for each of the occupants who are the target of speech recognition, According to the present embodiment, it is possible to set only the speech of a specific occupant as the target of speech recognition (triggerless speech recognition) without performing such registration. For this reason, there is no need for the complicated work of creating and registering the voice profile, and the convenience of the user is high.

  FIG. 5 is a flowchart showing an operation example of the speech recognition apparatus 100 according to the present embodiment. The flowchart shown in FIG. 5 shows an example of the overall operation of the speech recognition apparatus 100 when setting regarding speech recognition is performed after the speech recognition apparatus 100 is powered on.

  As shown in FIG. 5, after the power of the speech recognition apparatus 100 is turned on, the recognition target registration unit 12 performs registration regarding the user who is the target of speech recognition according to the user's instruction (step S1). As described above, in the case of the operation mode M1, the recognition target registration unit 12 registers the face picture of the user who is the target of the speech recognition in the recognition target storage unit 23, and in the operation mode M2, the target of the speech recognition The seat to be registered is registered in the recognition target storage unit 23. In the case of the operation mode M3, the face photograph of the user who is the target of voice recognition and the seat that is the target of the voice recognition are registered in the recognition target storage unit 23. .

  After registration by the recognition target registration unit 12, the image recognition unit 13 starts recognition of the face image included in the photographed image data and determination as to whether the mouth of the recognition target occupant is moving based on the recognition (step S2). Further, the audio processing unit 14 starts buffering of the audio signal based on the audio collected by the microphone 200 into the buffer 31 (step S3). Further, the voice recognition control unit 17 starts control of the voice recognition unit 15 according to the determination result of the image recognition unit 13 (step S4). By the above, the process (hereinafter, referred to as “specific voice recognition process”) relating to the voice recognition in which the speech voice of the recognition target occupant is the target of recognition is started.

  FIG. 6 is a flowchart showing an operation example of the image recognition unit 13 at the time of execution of the specific voice recognition process.

  As shown in FIG. 6, the image recognition unit 13 acquires the information registered in the recognition target storage unit 23 (step S11). The information registered in the recognition target storage unit 23 is a face photograph in the case of the operation mode M1, is seat position information in the case of the operation mode M2, and is a face photograph and a seat position in the case of the operation mode M3. It is information. Hereinafter, regardless of the operation mode, the information registered in the recognition target storage unit 23 is generically referred to as “registration information”.

  Next, the image recognition unit 13 performs the above-described processing based on the registration information acquired in step S11 and the photographed image data input from the camera, and the registration among the face images included in the photographed image data is performed. A face image corresponding to the recognition target occupant specified by the information is specified (step S12). In step S12, a plurality of face images may be registered. In this case, the image recognition unit 13 executes the processes of steps S13 to S16 described below in parallel for each of the plurality of face images.

  Next, the image recognition unit 13 tracks and analyzes the face image identified in step S22 for the photographed image data intermittently input from the camera, and whether or not the continuous movement of the recognition target occupant's mouth is started Is monitored (step S13). The process of step S13 is continuously performed until the start of continuous movement of the recognition target occupant's mouth is detected. When it is detected that the continuous movement of the recognition target occupant's mouth has been started (step S13: YES), the image recognition unit 13 performs the above-described operation start notification (step S14).

  Next, the image recognition unit 13 monitors whether or not the continuous movement of the recognition target occupant's mouth has stopped (step S15). The process of step S15 is continuously performed until the stop of continuous movement of the recognition target occupant's mouth is detected. When it is detected that the continuous movement of the recognition target occupant's mouth is stopped (step S15: YES), the image recognition unit 13 performs the above-described operation end notification (step S16). After the process of step S16, the process procedure returns to step S13. As a result of the above processing being performed, the image recognition unit 13 executes the operation start notification and the operation end notification at an appropriate timing according to the movement of the recognition target occupant's mouth.

  FIG. 7 is a flowchart showing an operation example of the speech recognition control unit 17 at the time of execution of the specific speech recognition process.

  As shown in FIG. 7, the voice recognition control unit 17 monitors whether or not there is an operation start notification from the image recognition unit 13 (step S21). The process of step S21 is continuously performed until the operation start notification is received. If there is an operation start notification (step S21: YES), the speech recognition control unit 17 causes the speech recognition unit 15 to start executing speech recognition (step S22). As described above, at the start of voice recognition, the voice recognition control unit 17 causes the voice recognition unit 15 to read the recorded voice data from a position preceding the timing when the operation start notification is given by a predetermined time.

  Next, the voice recognition control unit 17 monitors whether the operation recognition has been notified by the image recognition unit 13 (step S23). The process of step S23 is continuously performed until the operation end notification is received. If the operation end notification has been received (step S23: YES), the voice recognition control unit 17 causes the voice recognition unit 15 to stop the execution of voice recognition (step S24). As described above, when the voice recognition is stopped, the voice recognition control unit 17 causes the voice recognition unit 15 to read the recorded voice data up to the position corresponding to the timing at which the voice end notification is made and the voice based on the read voice data Once recognition is performed, execution of speech recognition is stopped. After the processing of step S24, the processing procedure returns to step S21. As a result of the above processing being performed, the voice recognition unit 15 performs voice recognition of the voice collected by the microphone 200 only while the mouth of the recognition target occupant is moving.

  As described above in detail, in the present embodiment, at least one of the face photograph of the user who is the target of voice recognition and the seat which is the target of the voice recognition is registered as a recognition target and installed in the car Recognize an image captured by the selected camera, and recognize voice input from the microphone 200 installed in the vehicle when it is determined that the passenger's mouth corresponding to the registered recognition object is moving I have to.

  As a result, if a face picture or a seat is registered as an object to be subjected to voice recognition, among voices constantly input from the microphone 200, the user can sit in a specific occupant or a registered seat in which the face picture is registered. Only voice input from the microphone 200 when it is determined that the specific occupant's mouth is moving is subjected to voice recognition processing. As a result, it is possible to set only the speech of a specific occupant as the target of speech recognition (triggerless speech recognition) without performing complicated registration work of the speech profile.

  In the embodiment described above, the screen used when the user selects a face picture has been described using the example of FIG. 3, but the contents of the screen are limited to the contents of the screen illustrated in FIG. It is not a thing. The same applies to the screen used when the user selects a seat.

  In the embodiment described above, in the operation mode M3, when the occupant in the seat registered as the recognition target is the user whose face photograph has been registered as the recognition target, the image recognition unit 13 selects the occupant, It was judged as a recognition target occupant. In this regard, the image recognition unit 13 may execute the following process. That is, the image recognition unit 13 recognizes the face image of the occupant photographed by the camera, and determines whether or not the mouth of the same occupant as the user whose face picture is registered as the recognition target is moving, and as the recognition target It is determined whether the passenger's mouth in the registered seat is moving. In this case, the occupant sitting in the seat registered as the recognition target and the occupant corresponding to the user whose face photograph is registered as the recognition target are each determined as the recognition target occupant, and the mouth of the recognition target occupant moves. Under the control of the speech recognition control unit 17, speech recognition is performed by the speech recognition unit 15 for a corresponding period.

  Further, in the above-described embodiment, one microphone 200 is provided in the vehicle, and one microphone 200 is configured to pick up an utterance voice of an occupant getting in the vehicle. In this regard, the following configuration may be employed. That is, the microphones 200 are provided in the vicinity of each of the seats of the vehicle. A microphone 200 provided in the vicinity of one seat is a microphone 200 for picking up the speech of the occupant seated in the one seat. The microphone 200 can be switched on / off by the image recognition unit 13.

  Then, the image recognition unit 13 performs the above-described processing to determine whether the recognition target occupant is moving the mouth, and further determines that the movement of the recognition target occupant's mouth is started, the recognition target The microphone 200 provided in the vicinity of the seat where the occupant sits is turned on, and the voice recognition control unit 17 is notified of the operation start. In the embodiment described above, the image recognition unit 13 determines that the mouth is moving when the mouth moves continuously for a predetermined time or more, but in the present configuration, the image recognition unit 13 recognizes the occupant When transitioning to the moving state from the state in which the mouth of the mouth is not moving, it is determined that the continuous movement of the mouth has been started, and the microphone 200 is immediately turned on to issue an operation start notification. If the image recognition unit 13 determines that the continuous movement of the mouth has ended after the start of the movement of the mouth, the image recognition unit 13 turns off the microphone 200 and sends an operation end notification to the voice recognition control unit 17. The voice recognition control unit 17 reads the recorded voice data from the position corresponding to the timing at which the operation start notification has been made to the position corresponding to the timing at which the operation end notification is received, based on the notification from the image recognition unit 13. And, voice recognition based on the read out recorded voice data is performed.

  As a result, while the mouth of the recognition target occupant is moving, the microphone 200 located in the vicinity of the recognition target occupant performs sound collection, and the voice recognition unit 15 collects the collected sound (voice of the recognition target passenger). Speech recognition is performed. As a result, since the speech of the recognition target occupant can be collected by the microphone 200 located in the vicinity of the recognition target occupant, the possibility that noise is contained in the microphone 200 can be reduced, and voice recognition with high accuracy can be realized. Moreover, it can suppress that the microphone 200 is turned on unnecessarily, and can suppress consumption of a resource effectively.

  In addition, any of the above-described embodiments is merely an example of embodying the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the scope or main features of the present invention.

11 List registration unit (Occupant list registration unit)
12 recognition target registration unit 13 image recognition unit 15 voice recognition unit 17 voice recognition control unit 22 in-vehicle configuration list storage unit 100 voice recognition device

Claims (8)

A recognition target registration unit that registers, as a recognition target, a face photograph of a user who is a target of speech recognition;
A voice recognition unit that recognizes voice input from a microphone installed in the car;
An image recognition unit that recognizes an image captured by a camera installed in the vehicle, and determines whether a passenger's mouth corresponding to the recognition target registered by the recognition target registration unit is moving;
And a voice recognition control unit configured to control processing of the voice recognition unit when it is determined by the image recognition unit that the mouth of the occupant corresponding to the recognition target is moving. Voice recognition device.   The image recognition unit recognizes a face image of the occupant photographed by the camera, and determines whether or not the mouth of the same occupant as the user whose face picture is registered as the recognition target is moving. The speech recognition apparatus according to claim 1. The occupant list registration unit for registering face photographs of a plurality of users as an occupant list is further provided.
The voice recognition apparatus according to claim 1 or 2, wherein the recognition target registration unit registers a face picture of a user selected from the occupant list by a user operation as the voice recognition. The recognition target registration unit registers, as the recognition target, a seat on which the voice recognition is to be performed, instead of the user's face picture.
The image recognition unit recognizes a face image of the occupant photographed by the camera, and determines whether or not the mouth of the occupant in the seat registered as the recognition target is moving. The speech recognition device according to claim 1. The in-vehicle configuration list storage unit further stores a plurality of seats as the in-vehicle configuration list,
The voice recognition device according to claim 4, wherein the recognition target registration unit registers a seat selected by a user operation from the in-vehicle configuration list as the recognition target. The recognition target registration unit registers, as the recognition target, a seat to which the voice recognition is to be performed, in addition to the face picture of the user.
The image recognition unit recognizes the face image of the occupant photographed by the camera, and the occupant in the seat registered as the recognition target is a user whose face photograph has been registered as the recognition target, and the occupant The speech recognition apparatus according to claim 1, wherein it is determined whether or not the mouth of the mouse is moving. The recognition target registration unit registers, as the recognition target, a seat to which the voice recognition is to be performed, in addition to the face picture of the user.
The image recognition unit recognizes the face image of the occupant photographed by the camera, determines whether or not the mouth of the same occupant as the user whose face picture is registered as the recognition target is moving, and the recognition target The speech recognition apparatus according to claim 1, wherein it is determined whether or not the mouth of the occupant in the seat registered as is moving. A first step of registering, as a recognition target, at least one of a face picture of a user who is a target of speech recognition and a seat which is a target of the speech recognition;
The image recognition unit of the voice recognition device recognizes an image captured by a camera installed in the car, and whether or not the mouth of the occupant corresponding to the recognition target registered by the recognition target registration unit is moving A second step of determining
The voice recognition control unit of the voice recognition device recognizes the voice input from the microphone installed in the vehicle when the image recognition unit determines that the occupant's mouth corresponding to the recognition object is moving. And a third step of performing control so as to perform processing of the speech recognition unit.