JP7586367B1 - Audio processing device, audio processing method, and program - Google Patents

Abstract

A speech processing device, a speech processing method, and a program are provided that are capable of easily correcting the results of simultaneous interpretation in multiple languages.
[Solution] A voice processing device comprising: a voice conversion unit that converts voice data indicating the content of a user's speech in a first language into text data indicating the translation results into a plurality of second languages different from the first language through simultaneous interpretation; a monitoring processing unit that accepts corrections to the text data for one specified second language among the plurality of second languages; and a translation unit that translates the corrected text data into the second language that was not specified at the time of correction, and obtains text data indicating the translation results in which the corrections are reflected for each of the plurality of second languages.
[Selected figure] Figure 2

Description

The present invention relates to an audio processing device, an audio processing method, and a program.

In recent years, advances in AI (Artificial Intelligence) have improved the accuracy of AI-based speech recognition and machine translation, but the recognition and translation results may still contain misrecognitions or mistranslations.

In this regard, the following Patent Document 1 discloses a technique for correcting the results of speech recognition. This technique is used, for example, when correcting subtitles generated based on the results of speech recognition in video data that includes audio data.

JP 2019-148681 A

By the way, simultaneous interpretation is possible by combining speech recognition and machine translation. However, even in simultaneous interpretation, the recognition or translation results may contain misrecognition or mistranslation. For this reason, in simultaneous interpretation of multiple languages, if the simultaneous interpretation results contain at least one misrecognition or mistranslation, the checker must correct the simultaneous interpretation results for each language, which is not easy to do.

In view of the above problems, the object of the present invention is to provide a speech processing device, a speech processing method, and a program that can easily correct the results of simultaneous interpretation in multiple languages.

In order to solve the above-mentioned problems, a voice processing device according to one aspect of the present invention is a voice processing device that includes a voice conversion unit that converts voice data indicating the contents of a user's utterance in a first language into text data indicating the translation results into a plurality of second languages different from the first language by simultaneous interpretation, a monitoring processing unit that accepts corrections to the text data for one specified second language among the plurality of second languages, and a translation unit that translates the corrected text data into the second language that was not specified at the time of correction and obtains text data indicating the translation results in which the corrections are reflected for each of the plurality of second languages.

A speech processing method according to one aspect of the present invention is a speech processing method executed by a computer, including a speech conversion process for converting speech data indicating the contents of a user's speech in a first language into text data indicating the translation results into a plurality of second languages different from the first language by simultaneous interpretation, a monitoring process for accepting corrections to the text data for one specified second language among the plurality of second languages, and a translation process for translating the corrected text data into the second language that was not specified at the time of correction, and acquiring text data indicating the translation results in which the corrections are reflected for each of the plurality of second languages.

A program according to one aspect of the present invention is a program for causing a computer to function as: a speech conversion means for converting speech data indicating the contents of a user's speech in a first language into text data indicating the translation results into a plurality of second languages different from the first language by simultaneous interpretation; a monitoring processing means for accepting corrections to the text data for a specified one of the plurality of second languages; and a translation means for translating the corrected text data into the second language that was not specified at the time of correction, and acquiring text data indicating the translation results in which the corrections are reflected for each of the plurality of second languages.

The present invention makes it possible to easily correct the results of simultaneous interpretation in multiple languages.

FIG. 1 is a diagram illustrating an example of a configuration of a subtitle display system according to a first embodiment. 1 is a block diagram showing an example of a functional configuration of a sound processing device according to a first embodiment. FIG. 4 is a diagram showing an example of a monitoring screen according to the first embodiment; FIG. 11 is a diagram illustrating an example of a correction operation procedure according to the first embodiment. 4 is a sequence diagram showing an example of a processing flow in the subtitle display system according to the first embodiment. FIG. FIG. 11 is a diagram illustrating an example of a configuration of a subtitle display system according to a second embodiment. FIG. 11 is a block diagram showing an example of a functional configuration of a sound processing device according to a second embodiment. FIG. 11 is a diagram showing an example of a monitoring screen according to the second embodiment. FIG. 11 is a diagram illustrating an example of a correction operation procedure according to the second embodiment. FIG. 11 is a diagram illustrating an example of change of conversion priority according to the second embodiment. FIG. 11 is a sequence diagram showing an example of a processing flow in a subtitle display system according to a second embodiment. FIG. 11 is a sequence diagram showing an example of the flow of a display preparation process according to the second embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings.

<<1. First embodiment>>
A first embodiment will be described with reference to Figures 1 to 5. A subtitle display system will be described below. The subtitle display system is a system for translating a user's speech into multiple languages different from the language used by the user (multilingual translation) and displaying subtitles in each language (multilingual subtitles).
In the following, a first embodiment will be described by taking as an example a case where subtitles obtained by translating the speech content (lecture content) of a user (lecture speaker) giving a lecture at a lecture into multiple languages are distributed to users (listeners) listening to the lecture. In the first embodiment, the language spoken by the lecturer (first language) is any one language (e.g., any language other than Japanese), and the language of the subtitles translated into multiple languages (second language) is any multiple languages (e.g., multiple languages including Japanese).

<1-1. Configuration of the subtitle display system>
The configuration of a subtitle display system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a subtitle display system according to the first embodiment.

As shown in FIG. 1, the subtitle display system 1 includes a sound collection device 10, a sound processing device 20, a simultaneous interpretation engine 21, a machine translation engine 22, a monitoring terminal 30, and a display device 40.
Each device and terminal are connected to each other via a wired connection, a wireless connection, or a network connection so as to be able to transmit and receive various information. The network may be, for example, a local area network (LAN), a wide area network (WAN), a telephone network (such as a mobile phone network or a landline telephone network), a regional Internet Protocol (IP) network, the Internet, or the like.

(1) Sound collection device 10
The sound collection device 10 is a device that collects the sound generated when a speaker speaks, and is, for example, a microphone.
The sound collection device 10 is connected to the sound processing device 20 via a wired connection or a wireless connection so as to be able to communicate with the sound processing device 20. When the sound collection device 10 collects the voice of the lecturer, it converts the collected voice into data and transmits the converted voice data to the sound processing device 20.

(2) Audio Processing Device 20
The voice processing device 20 is a device that performs processing for displaying the results of simultaneous interpretation as subtitles. The voice processing device 20 is realized by, for example, one or more servers (e.g., cloud servers), PCs (Personal Computers), etc. In the device, various processes are executed by a program for functioning as the voice processing device 20.
The voice processing device 20 executes various processes based on the voice data received from the sound collection device 10. The voice processing device 20 executes, for example, a voice conversion process, a monitoring process, a machine translation process, a subtitle display process, and the like.

The voice conversion process is a process in which voice data is converted into text data. In the voice conversion process, the voice processing device 20 performs voice recognition and machine translation of the voice data received from the sound collection device 10 using the functions of the simultaneous interpretation engine 21 described below, and obtains text data indicating the translation result. In the voice conversion process, one piece of text data expressed in a first language is machine translated into multiple pieces of text data expressed in multiple second languages different from the first language.

The monitoring process is a process executed by the checker to monitor the results of the voice conversion process on the voice data (voice conversion result). In the monitoring process, the voice processing device 20 displays a monitoring screen on the monitoring terminal 30. The monitoring screen is a screen that can accept a selection operation for whether or not to display the text data converted from the voice data, and an operation for correcting the text data.

The checker can monitor the voice conversion result by checking the voice conversion result displayed on the monitoring screen of the monitoring terminal 30. During monitoring, the checker determines whether the text data converted from the voice data in the voice conversion process can be displayed as subtitles (displayability). The criteria for determining whether displayability is possible include, for example, whether there is a misrecognition or misconversion in the voice conversion result, or whether the meaning of the text data is understandable. If there is a misrecognition or misconversion in the voice conversion result, or if the meaning of the text data is incomprehensible, the checker determines that the text data cannot be displayed as subtitles. On the other hand, if there is no misrecognition or misconversion in the voice conversion result and the meaning of the text data is understandable, the checker determines that the text data can be displayed as subtitles.

Depending on the result of the judgment, the checker performs an operation to select whether or not the text data can be displayed on the monitoring screen, and performs a correction operation as necessary. If it is judged that the text data can be displayed, the checker performs an operation to select "displayable" on the monitoring screen. On the other hand, if it is judged that the text data cannot be displayed, the checker performs an operation to select "displayable" on the monitoring screen, and corrects the text data so that it can be displayed. Once the correction operation by the checker is completed, the displayability of the corrected text data is automatically changed to "displayable" by the audio processing device 20.

The monitoring screen may be provided with an automatic selection function for whether or not to display. The automatic selection function is a function that automatically selects whether to display or not display after a predetermined time has elapsed since text data was displayed on the monitoring screen. The predetermined time can be set by any user (e.g., a checker or an administrator) on the monitoring screen to any time (e.g., 3 seconds). Also, whether to automatically select whether to display or not display can be set by any user on the monitoring screen.

The voice conversion result displayed on the monitoring screen is text data in only one language. The checker only needs to monitor, on the monitoring screen, the text data in one second language among the text data in multiple second languages converted by the voice conversion process. Note that the text data in the second language to be displayed on the monitoring screen among the text data in multiple second languages can be appropriately selected according to, for example, the language that the checker can handle.

Machine translation processing is processing in which text data is translated. In machine translation processing, the voice processing device 20 machine translates the text data that has been determined to be undisplayable in the monitoring processing and has been corrected, using the functions of the machine translation engine 22 described below, to obtain text data indicating the translation result. In machine translation processing, one piece of text data expressed in a first language is machine translated into multiple pieces of text data expressed in multiple second languages different from the first language. Therefore, the voice processing device 20 can obtain text data in which the corrections are reflected in multiple other languages from one piece of text data corrected in one language, using machine translation processing.

The subtitle display process is a process in which subtitles are displayed. In the subtitle display process, the voice processing device 20 causes the display device 40 to display text data indicating the translation result in the machine translation process as subtitles.
Note that the audio processing device 20 may cause the display device 40 to display only subtitles in one specified second language, or may cause the display device 40 to display subtitles in a plurality of second languages.

(3) Simultaneous Interpretation Engine 21
The simultaneous interpretation engine 21 is an engine (program) that simultaneously translates a first language into a second language. The simultaneous interpretation engine 21 converts the voice data of the first language input from the voice processing device 20 into text data of the first language by voice recognition, and machine translates (converts) the text data of the first language into text data of the second language. The simultaneous interpretation engine 21 machine translates one first language into multiple different second languages. In other words, the simultaneous interpretation engine 21 generates multiple text data of the second language from text data of one first language.
The function of the simultaneous interpretation engine 21 may be provided by a device or terminal different from the voice processing device 20, or may be provided by the voice processing device 20.

(4) Machine translation engine 22
The machine translation engine 22 is an engine (program) that performs machine translation from a first language to a second language. The machine translation engine 22 performs machine translation (conversion) of text data in the first language input from the speech processing device 20 into text data in the second language. The machine translation engine 22 performs machine translation from one first language to multiple different second languages. In other words, the machine translation engine 22 generates text data in multiple second languages from text data in one first language.
The function of the machine translation engine 22 may be provided by a device or terminal different from the speech processing device 20, or may be provided by the speech processing device 20.

(5) Monitoring terminal 30
The monitoring terminal 30 is a terminal used by a checker for monitoring. The monitoring terminal 30 is, for example, a PC, a smartphone, a tablet terminal, etc. In the monitoring terminal 30, various processes are executed by a program for functioning as the monitoring terminal 30.
The monitoring terminal 30 displays a monitoring screen based on the screen information received from the sound processing device 20, and accepts various operations related to monitoring by the checker.

For example, a monitoring screen is displayed on the monitoring terminal 30 by an application for utilizing a monitoring function (hereinafter, also referred to as a "monitoring app") The checker can monitor the results of the simultaneous interpretation by operating the monitoring screen displayed on the monitoring terminal 30 by the monitoring app.
The functions of the monitoring app may be provided by installing the monitoring app on each terminal (i.e., a native app) or may be provided by a web system (i.e., a web app). In the case of a web app, the monitoring app is managed by a server, and its functions are provided via a web browser.

(6) Display device 40
The display device 40 is a device that displays subtitles. The display device 40 may be, for example, a display device such as a screen 41, or may be a device having a display such as a smartphone 42. The display device 40 is communicably connected to the audio processing device 20, and displays subtitles based on screen information received from the audio processing device 20.

For example, a subtitle display screen is displayed on the display device 40 by an application for utilizing a subtitle display function (hereinafter, also referred to as a "subtitle display application") The audience can check the subtitles showing the results of the simultaneous interpretation by operating the subtitle display screen displayed on the display device 40 by the subtitle display application.
The function of the subtitle display application may be provided by installing the subtitle display application in each device (i.e., a native application), or may be provided by a Web system (i.e., a Web application). In the case of a Web application, the subtitle display application is managed by a server, and the function is provided via a Web browser.

<1-2. Functional configuration of the voice processing device>
The configuration of the subtitle display system 1 according to the first embodiment has been described above. Next, the functional configuration of the audio processing device 20 according to the first embodiment will be described with reference to Fig. 2 to Fig. 4. Fig. 2 is a block diagram showing an example of the functional configuration of the audio processing device 20 according to the first embodiment.
As shown in FIG. 2 , the voice processing device 20 includes a communication unit 210 , a storage unit 220 , a first control unit 230 , and a second control unit 240 .

(1) Communication unit 210
The communication unit 210 has a function of transmitting and receiving various information. The communication unit 210 is communicably connected to the sound collection device 10, the simultaneous interpretation engine 21, the machine translation engine 22, the monitoring terminal 30, and the display device 40, and transmits and receives various information.

(2) Storage unit 220
The storage unit 220 has a function of storing various information. The storage unit 220 is configured by a storage medium provided as hardware in the audio processing device 20, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read/write memory (RAM), a read only memory (ROM), or any combination of these storage media.

The storage unit 220 stores, for example, conversion candidate information. The conversion candidate information is information that indicates conversion candidates for text data. The conversion candidate information accumulates text that is a conversion candidate based on the correction history by the checker.
The conversion candidates are stored, for example, on a lecture-by-lecture basis. The conversion candidates may be deleted when the lecture ends, or may remain stored. The conversion candidates that remain stored may be used for another lecture.

(3) First control unit 230
The first control unit 230 has a function of controlling processes related to simultaneous interpretation. The first control unit 230 is realized, for example, by causing a central processing unit (CPU) or a graphics processing unit (GPU) provided as hardware in the speech processing device 20 to execute a program.
As shown in FIG. 2, the first control unit 230 includes a voice data acquisition unit 231 and a voice conversion unit 232 .

(3-1) Voice data acquisition unit 231
The voice data acquisition unit 231 has a function of acquiring voice data. The voice data acquisition unit 231 acquires the voice data received by the communication unit 210 from the sound collection device 10, and inputs the voice data to the voice conversion unit 232.

(3-2) Voice conversion unit 232
The voice conversion unit 232 has a function of executing a voice conversion process. The voice conversion unit 232 executes the voice conversion process using the simultaneous interpretation engine 21. In the voice conversion process, the voice conversion unit 232 inputs the voice data indicating the contents of the speaker's speech in the first language acquired by the voice data acquisition unit 231 to the simultaneous interpretation engine 21, and acquires text data output from the simultaneous interpretation engine 21 as a translation result. In this way, the voice conversion unit 232 can convert the voice data indicating the contents of the speaker's speech in the first language into text data indicating the translation results into a plurality of second languages different from the first language by simultaneous interpretation.
The voice conversion unit 232 inputs the text data in the second language obtained by the voice conversion process to the monitoring processing unit 241.

(4) Second control unit 240
The second control unit 240 has a function of controlling processes related to monitoring and subtitle display. The second control unit 240 is realized, for example, by causing a CPU or a GPU included in the audio processing device 20 as hardware to execute a program.
As shown in FIG. 2, the second control unit 240 includes a monitoring processing unit 241 , a machine translation unit 242 , and a subtitle processing unit 243 .

(4-1) Monitoring Processing Unit 241
The monitoring processing unit 241 has a function of executing a monitoring process. In the monitoring process, the monitoring processing unit 241 transmits screen information from the communication unit 210 to the monitoring terminal 30 and displays a monitoring screen on the monitoring terminal 30. The monitoring processing unit 241 accepts, via the monitoring screen displayed on the monitoring terminal 30, a selection operation of whether or not to display text data from the checker and an operation to correct the text data.

The monitoring processing unit 241 displays on the monitoring screen the text data of one language that has been designated in advance as a target for monitoring, among the text data of the plurality of second languages input from the voice conversion unit 232. For this reason, the monitoring processing unit 241 accepts corrections to the text data for the one second language designated among the plurality of second languages.
Note that one language designated in advance as a target for monitoring is, for example, a language desired by the checker. In the first embodiment, as an example, it is assumed that the checker is Japanese and the language desired by the checker is Japanese.

The monitoring processing unit 241 displays, on the monitoring screen, a UI (User Interface) for receiving a selection operation as to whether or not the text data displayed on the monitoring screen should be displayed on the display device 40. The UI is, for example, a button, but may also be a check box, a pull-down, or the like.
When the monitoring processing unit 241 receives a selection operation for display or non-display, it controls the display of subtitles of the text data that is the target of the operation. By the control of the monitoring processing unit 241, subtitles of text data for which "not possible" has been selected as the display option on the monitoring screen are not displayed on the display device 40, and subtitles of text data for which "possible" has been selected as the display option on the monitoring screen are displayed on the display device 40.

The monitoring processing unit 241 displays a UI for displaying text data on the monitoring screen. The UI is, for example, a text field. The monitoring processing unit 241 may display the text data in the text field on the monitoring screen, and accept a selection of whether or not to display the text field through an operation on the text field. For example, the monitoring processing unit 241 switches whether or not to display when the inside of the text field is selected, and switches whether or not to display when the outside of the text field is selected after the inside of the text field is selected.

The monitoring processing unit 241 also accepts correction operations for the text data by operations on the text field in which the text data is displayed. For example, when the inside of a text field is selected by the checker, the monitoring processing unit 241 accepts corrections to the text data displayed in the selected text field. The checker can correct the text data, for example, by manual input.

When a portion of text data to be corrected is selected on the monitoring screen, the monitoring processing unit 241 may display conversion candidates near the portion to be corrected. The monitoring processing unit 241 inserts text selected by the checker from the conversion candidates into the portion to be corrected. In this way, the checker can correct text data not only by manual input, but also by selecting conversion candidates.

The monitoring processing unit 241 adds, to the conversion candidates, any text that is not included in the conversion candidates and is detected as a difference when comparing the text data before and after correction. In this case, the monitoring processing unit 241 adds the new conversion candidate to the conversion candidate information stored in the storage unit 220. As a result, text that is a conversion candidate is accumulated in the conversion candidate information based on the correction history by the checker.

If the text data displayed in the text field does not need to be corrected and "Yes" is selected for "Displayable", the monitoring processing unit 241 inputs the text data in the second language obtained by the voice conversion process by the voice conversion unit 232 to the subtitle processing unit 243. On the other hand, if the text data displayed in the text field needs to be corrected and "No" is selected for "Displayable", the monitoring processing unit 241 inputs the corrected text data to the machine translation unit 242.

When "no" is selected as the display option for text data for which subtitles are already displayed on the display device 40, the monitoring processing unit 241 hides the subtitles for the text data for which "no" is selected, and makes it possible to accept correction operations.

Here, the monitoring screen according to the first embodiment will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the monitoring screen according to the first embodiment.

The buttons B1, B2, and pull-down PD on the monitoring screen G1 shown in Fig. 3 are UIs for making settings related to automatic selection of display/non-display. The button B1 is a button for setting automatic selection of possible (o) as display/non-display. The button B2 is a button for setting automatic selection of not possible (x) as display/non-display. The pull-down PD is a pull-down for setting a predetermined time from when text data is displayed on the monitoring screen G1 until when display/non-display is automatically selected.
3, as an example, button B1 is on, button B2 is off, and the predetermined time is set to three seconds. In this case, "Yes" is automatically selected as the display possibility three seconds after the text data is displayed on the monitoring screen. On the other hand, if button B1 is off and button B2 is on, "No" is automatically selected as the display possibility three seconds after the text data is displayed on the monitoring screen.

3 are UIs for the checker to correct the displayed text data. The monitoring processing unit 241 displays, in the text field, text data in one language that is designated in advance as a target for monitoring, among the text data in the second language input from the speech conversion unit 232.
In the example shown in FIG. 3, four pieces of text data converted from four pieces of voice data are displayed in chronological order in text fields F1 to F4, respectively.

Buttons B3 and B4 on the monitoring screen G1 in FIG. 3 are UIs that allow the checker to select whether or not to display the text data. Button B3 is a button for selecting "not possible" (x) as whether or not to display. Button B4 is a button for selecting "possible" (o) as whether or not to display. Buttons B3 and B4 are displayed for each text field F. Note that the selection of buttons B3 and B4 may not only be made by the checker, but may also be switched under the control of the monitoring processing unit 241 after a predetermined time has elapsed, when editing the text data starts, or after editing the text data.

In the example shown in FIG. 3, as an example, buttons B3-1 to B3-4 and buttons B4-1 to B4-4 are displayed in each of the text fields F1 to F4.
In the example of buttons B3-1 and B4-1 in text field F1, the checker has determined that the text data displayed in text field F1 is correct and can be displayed, and button B4-1 has been selected.
In the example of buttons B3-2 and B4-2 in text field F2, the checker determines that the meaning of the text data displayed in text field F2 is incomprehensible, and button B3-2 is selected.
In the example of buttons B3-3 and B4-3 in text field F3, the checker has determined that the text data displayed in text field F3 is correct and can be displayed, and button B4-3 has been selected.
In the example of buttons B3-4 and B4-4 in text field F4, the checker determined that the text data displayed in text field F4 needed to be corrected and could not be displayed, and button F3-4 was selected, but after the text data was corrected, button B4-4 was selected under the control of the monitoring processing unit 241.

Now, referring to FIG. 4, the correction operation procedure according to the first embodiment will be described. FIG. 4 is a diagram showing an example of the correction operation procedure according to the first embodiment. FIG. 4 shows an example in which a checker corrects text data displayed in text field F5. Note that, between buttons B3-5 and B4-5, it is assumed that button B4-5 is selected as the initial selection.

As shown in FIG. 4, first, the checker checks the text data displayed in the text field F5, and since correction is required, selects (touches) any position in the text field F5 (step S1). In the example shown in FIG. 4, it is assumed that the correction location (position P) is selected.

After the checker selects the text field F5, the monitoring processor 241 selects the text field F5 (e.g., displays a thick frame), switches the display/non-display selection from button B4-5 to button B3-5, displays a cursor K inside the text field F5, and displays a window W1 showing conversion candidates near the correction location (step S2). If the position of the cursor K is not aligned with the correction location, the checker moves the position of the cursor K to the correction location.
In addition, if the subtitles of the text data displayed in text field F5 are already being displayed on display device 40, the subtitles of the text data displayed in text field F5 are made invisible (deleted) at the time when the part to be corrected is selected by the checker.

The checker deletes the text to be corrected (step S3). In the example shown in FIG. 4, the checker deletes the text indicating "concatenation."

The checker selects the correct text from among the conversion candidates shown in window W1 (step S4). In the example shown in FIG. 4, the checker selects the text indicating "linkage."

After the checker selects the correct text, the monitoring processing unit 241 inserts the selected text into the correction location inside the text field F5 (step S5). If the correct text is not among the conversion candidates displayed in the window W1, the checker can manually input the correct text into the correction location.

As the correction is complete, the checker selects the outside of text field F5 (step S6). As a result, the monitoring processing unit 241 returns text field F5 to an unselected state (cancels the bold frame display) and switches the display/non-display selection from button B3-5 to button B4-5. Furthermore, the monitoring processing unit 241 inputs the corrected text data to the machine translation unit 242.

(4-2) Machine translation unit 242
The machine translation unit 242 has a function of executing a machine translation process. The machine translation unit 242 executes the machine translation process using the machine translation engine 22. In the machine translation process, the machine translation unit 242 inputs the text data corrected by the monitoring processing unit 241 to the machine translation engine 22, and acquires the text data output from the machine translation engine 22 as a translation result. In this way, the machine translation unit 242 can translate the corrected text data into a second language that was not specified at the time of correction, and acquire text data indicating the translation result in which the correction is reflected for each of the multiple second languages.
The machine translation unit 242 inputs the corrected text data obtained by the monitoring process and the multiple pieces of text data in a second language obtained by the machine translation process to the subtitle processing unit 243.

(4-3) Subtitle processing unit 243
The subtitle processing unit 243 has a function of executing a subtitle display process. In the subtitle display process, the subtitle processing unit 243 displays subtitles on the display device 40 using the text data in the second language converted by the speech conversion unit 232 and input from the monitoring processing unit 241, or the text data in the second language input from the machine translation unit 242.
When using text data in a plurality of second languages input from the monitoring processing unit 241, the subtitle processing unit 243 can display text data indicating a translation result for which correction is determined not to be required in the monitoring process as subtitles on the display device 40. On the other hand, when using text data in a plurality of second languages input from the machine translation unit 242, the subtitle processing unit 243 can display text data indicating a translation result for which correction is determined to be required in the monitoring process and in which the correction is reflected as subtitles on the display device 40.

<1-3. Processing flow>
The functional configuration of the audio processing device 20 according to the first embodiment has been described above. Next, a process flow in the subtitle display system 1 according to the first embodiment will be described with reference to Fig. 5. Fig. 5 is a sequence diagram showing an example of a process flow in the subtitle display system 1 according to the first embodiment.

As shown in FIG. 5, first, the sound collection device 10 transmits audio data of the collected audio to the audio processing device 20 (step S101). The audio data acquisition unit 231 of the audio processing device 20 acquires the audio data that the communication unit 210 receives from the sound collection device 10.

Next, the voice conversion unit 232 of the voice processing device 20 requests the simultaneous interpretation engine 21 to perform simultaneous interpretation of the voice data acquired by the voice data acquisition unit 231 (step S102). The voice conversion unit 232 transmits the voice data to the simultaneous interpretation engine 21 via the communication unit 210.

Next, the simultaneous interpretation engine 21 simultaneously interprets (speech recognition and machine translation) the voice data received from the voice processing device 20 and transmits the result of the simultaneous interpretation to the voice processing device 20 (step S103). The result of the simultaneous interpretation is text data in multiple second languages.

Next, the monitoring processing unit 241 of the voice processing device 20 performs a process of displaying the monitoring screen (step S104). The monitoring processing unit 241 transmits screen information to the monitoring terminal 30 via the communication unit 210, and displays the monitoring screen.

The monitoring terminal 30 displays a monitoring screen based on the screen information received from the sound processing device 20 (step S105).
After displaying the monitoring screen, the monitoring terminal 30 accepts corrections to the text data made by the checker on the monitoring screen, and transmits correction information indicating the contents of the corrections to the voice processing device 20 (step S106).

The monitoring processing unit 241 determines whether or not the text data has been corrected, depending on whether or not the communication unit 210 receives correction information from the monitoring terminal 30 (step S107). If there has been a correction (step S107/YES), the process proceeds to step S108. On the other hand, if there has been no correction (step S107/NO), the process proceeds to step S110.

When the process proceeds to step S108, the machine translation unit 242 of the speech processing device 20 requests the machine translation engine 22 to perform machine translation of the text data corrected by the checker (step S108). The machine translation unit 242 transmits the text data to the machine translation engine 22 via the communication unit 210.

Next, the machine translation engine 22 machine translates the text data received from the speech processing device 20 and transmits the results of the machine translation to the speech processing device 20 (step S109). The results of the machine translation are text data in multiple second languages. After transmission, the process proceeds to step S111.

When the process proceeds to step S110, the monitoring processing unit 241 determines whether the text data can be displayed (step S110). If the display is possible (step S110/YES), the process proceeds to step S111. On the other hand, if the display is not possible (step S110/NO), the process ends.

When the process proceeds to step S111, the subtitle processor 243 of the audio processing device 20 executes a subtitle display process (step S111). The subtitle processor 243 transmits text data in a plurality of second languages to the display device 40 via the communication unit 210.
The display device 40 displays the text data in the second language received from the audio processing device 20 as subtitles (step S112).

The process flow according to the first embodiment has been described above.
As described above, the voice processing device 20 according to the first embodiment includes a voice conversion unit 232 that converts voice data indicating the content of a user's speech in a first language into text data indicating the translation results into a plurality of second languages different from the first language by simultaneous interpretation, a monitoring processing unit 241 that accepts corrections to the text data for one specified second language among the plurality of second languages, and a machine translation unit 242 that translates the corrected text data into a second language that was not specified at the time of correction and obtains text data indicating the translation results in which the corrections are reflected for each of the plurality of second languages.

With this configuration, in the case of simultaneous interpretation in multiple languages, if the simultaneous interpretation result contains a misrecognition or mistranslation, the checker can correct the simultaneous interpretation result in only one of the multiple languages of the translated word, and can also correct the simultaneous interpretation results in the other languages.
Therefore, the speech processing device 20 according to the first embodiment makes it possible to easily correct the results of simultaneous interpretation in multiple languages.

Furthermore, if the simultaneous interpretation of the speaker's speech contains a misrecognition or mistranslation, the checker can correct the misrecognition or mistranslation before the simultaneous interpretation result (subtitles) is presented to the audience. This ensures that the audience will not be shown the simultaneous interpretation result that contains the misrecognition or mistranslation, but only the corrected simultaneous interpretation result that does not contain the misrecognition or mistranslation.
Therefore, the speech processing device 20 according to the first embodiment enables the user to correctly understand the content of the simultaneous translation.

<<2. Second embodiment>>
The first embodiment has been described above. Next, the second embodiment will be described with reference to Figs. 6 to 12. In the second embodiment, a method of correcting text data on a monitoring screen that is different from that in the first embodiment will be described. In the following, descriptions that overlap with those in the first embodiment will be omitted as appropriate.
In the second embodiment, the language spoken by the speaker (first language) is any one language (e.g., Japanese), and the language of the multilingual translated subtitles (second language) is any multiple languages (e.g., languages other than Japanese).

<2-1. Configuration of subtitle display system>
A configuration of a subtitle display system 1a according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of a subtitle display system 1a according to the second embodiment.
As shown in FIG. 6, the subtitle display system 1a includes a sound collection device 10, a sound processing device 20a, a machine translation engine 22, a voice recognition engine 23, a conversion candidate API (Application Programming Interface) 24, a monitoring terminal 30, and a display device 40.

(1) Sound collection device 10
The sound collector 10 according to the second embodiment is similar to the sound collector 10 according to the first embodiment, and therefore a description thereof will be omitted.

(2) Audio processing device 20a
The audio processing device 20a according to the second embodiment differs from the audio processing device 20 according to the first embodiment in the process it executes, due to a different correction method on the monitoring screen. Based on the audio data received from the sound collection device 10, the audio processing device 20a performs not only audio conversion processing, monitoring processing, machine translation processing, and subtitle display processing, but also morphological analysis processing, conversion priority processing, and an automatic conversion processing unit.

(3) Machine translation engine 22
The machine translation engine 22 according to the second embodiment is similar to the machine translation engine 22 according to the first embodiment, and therefore a description thereof will be omitted.

(4) Voice recognition engine 23
The voice recognition engine 23 is an engine (program) that performs voice recognition on voice data. The voice recognition engine 23 converts the voice data in the first language input from the voice processing device 20a into text data in the first language by voice recognition.
The function of the voice recognition engine 23 may be provided by a device or terminal different from the voice processing device 20a, or may be provided by the voice processing device 20a.

(5) Conversion candidate API 24
The conversion candidate API 24 is an API that acquires conversion candidates for the reading of text data. The conversion candidate API 24 acquires conversion candidates corresponding to the reading of the text data of the first language based on information indicating the reading of the text data of the first language input from the voice processing device 20a.
The function of the conversion candidate API 24 may be provided by a device or terminal different from the voice processing device 20a, or may be provided by the voice processing device 20a.

(6) Monitoring terminal 30
The monitoring terminal 30 according to the second embodiment is similar to the monitoring terminal 30 according to the first embodiment, and therefore a description thereof will be omitted.

(7) Display device 40
The display device 40 according to the second embodiment is similar to the display device 40 according to the first embodiment, and therefore a description thereof will be omitted.

<2-2. Functional configuration of the voice processing device>
The configuration of the subtitle display system 1a according to the second embodiment has been described above. Next, the functional configuration of the audio processing device 20a according to the second embodiment will be described with reference to Fig. 7 to Fig. 10. Fig. 7 is a block diagram showing an example of the functional configuration of the audio processing device 20a according to the second embodiment.
As shown in FIG. 7, the voice processing device 20a includes a communication unit 210a, a storage unit 220a, a first control unit 230a, and a second control unit 240a.

(1) Communication unit 210a
The communication unit 210a is communicatively connected to the sound collection device 10, the machine translation engine 22, the voice recognition engine 23, the conversion candidate API 24, the monitoring terminal 30, and the display device 40, and transmits and receives various information.

(2) Storage unit 220a
The storage unit 220a differs from the storage unit 220 according to the first embodiment in that it also stores priority information. The priority information is information indicating the priority of automatic conversion for each morpheme into which text data is divided by morphological analysis. The priority is calculated based on the past correction history during monitoring by the checker.

(3) First control unit 230a
As shown in FIG. 7, the first control unit 230a includes a voice data acquisition unit 231a and a voice conversion unit 232a.

(3-1) Voice data acquisition unit 231a
The voice data acquisition unit 231a according to the second embodiment is similar to the voice data acquisition unit 231 according to the first embodiment, and therefore a description thereof will be omitted.

(3-2) Voice conversion unit 232a
The speech conversion unit 232a differs from the speech conversion unit 232 according to the first embodiment in that it does not perform simultaneous interpretation but only performs speech recognition.
The voice conversion unit 232a executes a voice recognition process as a voice conversion process using the voice recognition engine 23. In the voice recognition process, the voice conversion unit 232a inputs voice data indicating the contents of the speaker's speech in the first language acquired by the voice data acquisition unit 231a to the voice recognition engine 23, and acquires text data output from the voice recognition engine 23 as a voice recognition result. In this way, the voice conversion unit 232a can convert the voice data indicating the contents of the user's speech in the first language into text data indicating the contents of the user's speech in the first language by voice recognition.
The voice conversion unit 232a inputs the text data in the first language obtained by the voice recognition process to the monitoring processing unit 241a.

(4) Second control unit 240a
As shown in FIG. 7, the second control unit 240 a includes a monitoring processing unit 241 a , a machine translation unit 242 a , a subtitle processing unit 243 a , a morphological analysis unit 244 , a conversion priority processing unit 245 , and an automatic conversion processing unit 246 .

(4-1) Monitoring processing unit 241a
During the monitoring process, the monitoring processor 241a accepts a correction operation for the text data from the checker via the monitoring screen displayed on the monitoring terminal 30, but does not accept a selection operation for whether or not to display the text data. Also, the monitoring processor 241a divides the text data into a plurality of texts and displays them on the monitoring screen, and accepts corrections to the text data in units of the divided texts. For this reason, the method of correcting the text data on the monitoring screen differs between the first embodiment and the second embodiment.

The monitoring processing unit 241a displays a UI for displaying text data on the monitoring screen. The UI is, for example, a text field. The monitoring processing unit 241a displays a plurality of text fields for one piece of text data on the monitoring screen. The monitoring processing unit 241a divides the text data into texts in morpheme units based on the result of morpheme analysis by the morpheme analysis unit 244 described later. The morpheme units are, for example, parts of speech units. In this case, the monitoring processing unit 241a divides the text data into parts of speech units based on information indicating the result of the morpheme analysis unit 244 dividing the text data into parts of speech units (hereinafter, also referred to as "part of speech information"). The part of speech information includes, for example, the text after division and information indicating the reading of each text.
After the division, the monitoring processor 241a displays the divided texts in their respective text fields. The monitoring processor 241a accepts correction operations for each of the text fields for one piece of text data. This allows the checker to correct the text data on a morpheme-by-morpheme basis.

When text to be corrected in the text data is selected on the monitoring screen, the monitoring processing unit 241a displays conversion candidates near the selected text. The monitoring processing unit 241a acquires conversion candidates using the conversion candidate API 24. The monitoring processing unit 241a inputs part-of-speech information acquired by the morpheme analysis unit 244 to the conversion candidate API 24, and acquires information indicating the conversion candidates (hereinafter also referred to as "conversion candidate information") output from the conversion candidate API 24. The conversion candidate API 24 refers to the reading of the text included in the part-of-speech information, and acquires homonyms as conversion candidates. This allows the monitoring processing unit 241a to display conversion candidates on the monitoring screen based on the acquired conversion candidate information.
When a conversion candidate displayed on the monitoring screen is selected, the monitoring processor 241a replaces the text to be corrected with the text selected from the conversion candidates. In this way, the checker can correct the text data on a morpheme-by-morpheme basis by selecting a conversion candidate.

The monitoring processing unit 241a also displays a text input field along with displaying the conversion candidates, and accepts corrections by inputting text into the input field. This allows the checker to appropriately correct the text data by inputting appropriate text into the input field, for example, when there is no appropriate conversion candidate among the conversion candidates.

When displaying text data on the monitoring screen, the monitoring processing unit 241a may convert the text data based on past correction records before displaying the text data. For example, it is assumed that the text data to be displayed contains text corresponding to text previously corrected in other text data. The corresponding text is, for example, a homonym. In this case, the monitoring processing unit 241a converts the corresponding text in the text data to be displayed into text previously corrected before displaying the text data. That is, when displaying second text data obtained by speech recognition performed after the first text data is corrected, if the second text data contains second text corresponding to the first text corrected in the first text data, the monitoring processing unit 241a converts the second text into the corrected first text before displaying the second text data.
As a result, text that has been corrected in the past is automatically converted by the monitoring processing unit 241a, which reduces the load of monitoring by the checker.

The monitoring processing unit 241a may display not only the text data converted from the voice data, but also the text data obtained by translating the text data into a different language on the monitoring screen. When the text data converted from the voice data is corrected on the monitoring screen, the monitoring processing unit 241a obtains text data indicating the translation result reflecting the correction using the machine translation engine 22, and updates the display of the translation with the obtained text data.
The translation that the monitoring processor 241a displays on the monitoring screen may be a translation of one of the multiple second languages, or may be a translation of multiple languages.

Here, the monitoring screen according to the second embodiment will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of the monitoring screen according to the second embodiment.

The monitoring screen G2 shown in FIG. 8 displays, as an example, text data obtained by executing a voice recognition process for two pieces of voice data, and the translation results obtained by machine translation of each piece of text data.

The first voice data is, for example, a voice indicating the spoken content "The company provides a conductor." The voice data is converted into text data "The company provides a conductor" by the voice recognition process of the voice conversion unit 232a. The monitoring processing unit 241a divides the text data into parts of speech based on the result of morphological analysis and then displays it in the text field F11. In the example shown in FIG. 8, the text data is divided into seven parts of speech (words) of "company,""at,""is,""conductor,""for,""attach," and "masu" and displayed in the text field F11.
Furthermore, the monitoring processing unit 241a displays text data indicating the result of the text data being translated by the machine translation unit 242a in the text field F12. In the example shown in Fig. 8, the text data is translated into "There is a conductor in the company." and displayed in the text field F12.
The text data displayed in the text field F11 is text data obtained by performing voice recognition on speech data indicating the contents of a speech by a lecturer between certain arbitrary times T1 and T2, and is the first text data. Time T2 is assumed to be a time later than time T1.
The text data displayed in the text field F12 is text data showing the translation result into a preset arbitrary language among a plurality of languages (a plurality of second languages) that are the subject of multilingual translation. In the second embodiment, as an example, it is assumed that the translation result from Japanese to English is set to be displayed.

The second voice data is, for example, a voice indicating the spoken content of "The company emblem is mainly attached with an important formula." The voice data is converted into text data of "The company emblem is mainly attached with an important formula" by the voice recognition process of the voice conversion unit 232a. The monitoring processing unit 241a divides the text data into parts of speech based on the result of morphological analysis and then displays it in the text field F13. In the example shown in FIG. 8, the text data is divided into nine parts of speech (words) of "company emblem,""is,""mainly,""important,""na,""form,""de,""attach," and "masu," and is displayed in the text field F13.
Furthermore, the monitoring processing unit 241a displays text data indicating the result of the translation of the text data by the machine translation unit 242a in the text field F14. In the example shown in Fig. 8, the text data is translated into "We put the company emblem mainly at important ceremonies." and displayed in the text field F14.
The text data displayed in the text field F13 is text data obtained by performing voice recognition on the speech data indicating the contents of the speech of the speaker between any given times T3 and T4, and is the second text data. Time T3 is assumed to be later than time T2, and time T4 is assumed to be later than time T3.

Now, referring to FIG. 9, a correction operation procedure according to the second embodiment will be described. FIG. 9 is a diagram showing an example of a correction operation procedure according to the second embodiment. FIG. 9 shows an example of correcting the text data in the text field F11 in FIG. 8.

As shown in FIG. 9, first, the checker checks the text data displayed in the text field F11 and selects (touches) the text that needs to be corrected (step S11). In the example shown in FIG. 9, it is assumed that "conductor" is selected as the text to be corrected.

After the checker selects the text, the monitoring processor 241a displays a conversion candidate window W11, a text field F15, an add button B5, and a delete button B6 near the selected text (step S12). The checker selects appropriate text for correction from among the conversion candidates displayed in the conversion candidate window W11. If there is no appropriate text among the conversion candidates, the checker inputs appropriate text into the text field F15 (input field) and presses the add button B5. If the checker wants to delete the text selected in the text field F11, the checker presses the delete button B6.
If the text selected by the checker from the text field F11 is automatically converted text, the text before the automatic conversion is displayed in the text field F15. In this case, the checker can return the text selected in the text field F11 to the text before the automatic conversion by pressing the Add button B5.

After the checker makes the correction, the monitoring processor 241 displays the text selected in the text field F11 as the corrected text (step S13). Furthermore, the monitoring processor 241a displays, in the text field F12, text data obtained by translating the corrected text data shown in the text field F11.
In the example shown in Fig. 9, "Conductor" is corrected to "Company Emblem." As a result, in the example shown in Fig. 8, the text recognized as "Conductor" in the voice recognition of the second voice data is automatically converted to "Company Emblem" and then displayed in the text field F13.

(4-2) Machine translation unit 242a
The machine translation unit 242a translates the text data converted from the voice data of the first language by the voice recognition process by the voice conversion unit 232a into a second language different from the first language, and obtains text data indicating the translation result. In this way, the machine translation unit 242a can obtain text data indicating the translation result to be displayed on the monitoring screen together with the text data converted from the voice data.
Furthermore, when text data displayed on the monitoring screen is corrected, the machine translation unit 242a translates the corrected text data and obtains text data indicating the translation result reflecting the corrections. This allows the checker to check the corrected translation on the monitoring screen, and allows the machine translation unit 242a to obtain text data reflecting the corrections that can be displayed as is on the display device 40 as subtitles.

(4-3) Subtitle processing unit 243a
The subtitle processing unit 243a according to the second embodiment is similar to the subtitle processing unit 243 according to the first embodiment, and therefore a description thereof will be omitted.

(4-4) Morphological analysis unit 244
The morpheme analysis unit 244 has a function of performing morpheme analysis. The morpheme analysis unit 244 performs morpheme analysis on the text data converted from the voice data. The morpheme analysis unit 244 converts the text data into, for example, Divide text into multiple parts of speech.

(4-5) Conversion Priority Processing Unit 245
The conversion priority processing unit 245 has a function of performing processing related to priority information. Priority information is information indicating the priority of text used for conversion in automatic text conversion. The priority is determined based on the history of past corrections. When text data displayed on the monitoring screen is corrected, the conversion priority processing unit 245 changes the priority (weight) according to the content of the correction.

Here, referring to FIG. 10, a change in conversion priority according to the second embodiment will be described. FIG. 10 is a diagram showing an example of a change in conversion priority according to the second embodiment. The table on the left side of FIG. 10 shows an example of a priority before the change, and the table on the right side shows an example of a priority after the change.

10, pre-conversion text, post-conversion text, and priority (weight) are shown as priority information. The priority information indicates that when pre-conversion text is included in speech-recognized text data, the pre-conversion text is converted to post-conversion text with a higher priority.
The table on the left shows that the priority of converting "park" to "lecture" is "1.05", the priority of converting "park" to "performance" is "1.00", the priority of converting "park" to "oral performance" is "0.95", and the priority of converting "lecture" to "performance" is "1.00". Multiple pieces of priority information are shown for "park". If "park" is included in the text data that has been speech-recognized, it is automatically converted to "lecture", which has the highest priority.
When the voice-recognized text data is corrected by the checker, the conversion priority processing unit 245 changes the priority information. For example, assume that the voice-recognized text data includes "lecture" and the checker corrects "lecture" to "performance." In this case, the conversion priority processing unit 245 changes the table shown on the left side of Fig. 10 to the table shown on the right side. Because "lecture" has been corrected to "performance," the conversion priority processing unit 245 increases the priority of conversion to "performance" and decreases the priority of conversion to anything other than "performance."

(4-6) Automatic conversion processing unit 246
The automatic conversion processing unit 246 has a function of automatically converting text. Before the text data divided into a plurality of texts is displayed, the automatic conversion processing unit 246 refers to priority information. When the plurality of texts includes a text to be automatically converted, the automatic conversion processing unit 246 selects a text according to priority from among the texts used for past corrections, and converts the target text with the selected text. The text used for past corrections is the corrected text when the checker corrected the text of the text data in the past, and is the converted text in the priority information shown in FIG. 10.
The text according to the priority selected by the automatic conversion processing unit 246 is, for example, the text with the highest priority. When only one piece of priority information exists for one text, the automatic conversion processing unit 246 selects the text indicated by the priority information and performs automatic conversion. When multiple pieces of priority information exist for one text, the automatic conversion processing unit 246 selects the text indicated by the priority information with the highest priority and performs automatic conversion.

<2-3. Processing flow>
The functional configuration of the voice processing device 20a according to the second embodiment has been described above. Next, the flow of processing according to the second embodiment will be described with reference to Figs.

(1) Processing flow in the subtitle display system 1a The processing flow in the subtitle display system 1a according to the second embodiment will be described with reference to Fig. 11. Fig. 11 is a sequence diagram showing an example of the processing flow in the subtitle display system 1a according to the second embodiment.

As shown in FIG. 11, first, the sound collection device 10 transmits audio data of the collected audio to the audio processing device 20a (step S201). The audio data acquisition unit 231a of the audio processing device 20a acquires the audio data received by the communication unit 210a from the sound collection device 10.

Next, the voice conversion unit 232a of the voice processing device 20a performs a voice conversion process (step S202). The voice conversion unit 232a transmits the voice data acquired by the voice data acquisition unit 231a to the voice recognition engine 23 via the communication unit 210a and requests voice recognition. The voice recognition engine 23 performs voice recognition on the voice data received from the voice processing device 20a and transmits the result of the voice recognition to the voice processing device 20a.

Next, the machine translation unit 242a of the voice processing device 20a performs a machine translation process (step S203). The machine translation unit 242a transmits the text data acquired by the voice conversion unit 232a to the machine translation engine 22 via the communication unit 210a and requests machine translation. The machine translation engine 22 performs machine translation of the text data received from the voice processing device 20a and transmits the result of the machine translation to the voice processing device 20a.

Next, the audio processing device 20a performs a display preparation process (step S204). The display preparation process is a preparation process for displaying the monitoring screen. The details of the display preparation process will be described later.

Next, the monitoring processing unit 241a of the audio processing device 20a performs a process of displaying the monitoring screen (step S205). The monitoring processing unit 241a transmits screen information to the monitoring terminal 30 via the communication unit 210a, and displays the monitoring screen.

The monitoring terminal 30 displays a monitoring screen based on the screen information received from the sound processing device 20a (step S206).
After displaying the monitoring screen, the monitoring terminal 30 accepts corrections to the text data made by the checker on the monitoring screen, and transmits correction information indicating the contents of the corrections to the voice processing device 20a (step S207).

The monitoring processing unit 241a determines whether or not the text data has been corrected, depending on whether or not the communication unit 210a receives correction information from the monitoring terminal 30 (step S208). If there has been a correction (step S208/YES), the process proceeds to step S209. On the other hand, if there has been no correction (step S208/NO), the process proceeds to step S212.

When the process proceeds to step S209, the machine translation unit 242a of the voice processing device 20a performs a machine translation process (step S209). The machine translation unit 242a transmits the text data corrected by the checker to the machine translation engine 22 via the communication unit 210a and requests machine translation. The machine translation engine 22 performs machine translation of the text data received from the voice processing device 20a and transmits the result of the machine translation to the voice processing device 20a.

Next, the monitoring processing unit 241a performs an update process for the monitoring screen (step S210). The monitoring processing unit 241a updates the translation display on the monitoring screen based on the results of the machine translation obtained by the machine translation unit 242a.

Next, the conversion priority processing unit 245 of the audio processing device 20a updates the priority (step S211). Based on the correction information that the communication unit 210a receives from the monitoring terminal 30, the conversion priority processing unit 245 updates the priority of the priority information corresponding to the correction information.

When the process proceeds to step S212, the subtitle processor 243a of the audio processing device 20a executes a subtitle display process (step S212). The subtitle processor 243a transmits text data in a plurality of second languages to the display device 40 via the communication unit 210a.
The display device 40 displays the text data in the second language received from the audio processing device 20a as subtitles (step S213).

(2) Flow of Display Preparation Processing A flow of the display preparation processing according to the second embodiment will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of the flow of the display preparation processing according to the second embodiment.

As shown in FIG. 12, first, the morphological analysis unit 244 of the speech processing device 20a performs morphological analysis on the text data acquired by the speech conversion unit 232a (step S301).

Next, the automatic conversion processing unit 246 of the audio processing device 20a acquires the priority information stored in the memory unit 220a (step S302).

The automatic conversion processing unit 246 refers to the priority information and checks whether or not there is a part of speech (text) with a high priority for automatic conversion in the morphologically analyzed text data (step S303). If there is a part of speech with a high priority for automatic conversion (step S303/YES), the process proceeds to step S304. On the other hand, if there is no part of speech with a high priority for automatic conversion (step S303/NO), the process proceeds to step S305.

If the process proceeds to step S304, the automatic conversion processing unit 246 performs automatic conversion (step S304). After the automatic conversion, the process proceeds to step S305.

When the process proceeds to step S305, the monitoring processing unit 241a of the speech processing device 20a transmits the part-of-speech information acquired by the morphological analysis unit 244 to the conversion candidate API 24 via the communication unit 210a (step S305).

The conversion candidate API 24 obtains conversion candidate information based on the part of speech information received from the speech processing device 20a and transmits it to the speech processing device 20a (step S306).

The processes from step S302 to step S306 are performed on a part-of-speech basis (text basis) based on the results of the morphological analysis in step S301. Therefore, the processes from step S302 to step S306 are repeated until conversion candidate information has been obtained for all parts of speech. When conversion candidate information has been obtained for all parts of speech, the display preparation process ends.

The process flow according to the second embodiment has been described above.
As described above, the voice processing device 20a according to the second embodiment includes a voice conversion unit 232a that converts voice data indicating the content of a user's speech into text data by voice recognition, and a monitoring processing unit 241a that divides the text data into multiple texts, displays the text data, and accepts corrections to the text data in divided text units. When displaying second text data obtained by voice recognition performed after corrections are made to the first text data, if the second text data contains second text corresponding to the first text corrected in the first text data, the monitoring processing unit 241a converts the second text into the corrected first text and displays it.

With this configuration, when the checker corrects a recognition error that occurred in speech recognition of certain speech data, even if the same recognition error occurs in speech recognition of other speech data after the correction, it is automatically converted into correct text. This eliminates the need for the checker to make the same correction every time the same recognition error occurs.
Therefore, the voice processing device 20a according to the second embodiment makes it possible to reduce the burden of correcting erroneous recognition in voice recognition.

Furthermore, if the simultaneous interpretation of the speaker's speech contains a misrecognition or mistranslation, the checker can correct the misrecognition or mistranslation before the simultaneous interpretation result (subtitles) is presented to the audience. This ensures that the audience will not be shown the simultaneous interpretation result that contains the misrecognition or mistranslation, but only the corrected simultaneous interpretation result that does not contain the misrecognition or mistranslation.
Therefore, the speech processing device 20a according to the second embodiment enables the user to correctly understand the content of the simultaneous translation.

<<3. Modified Examples>>
The embodiment has been described above. Next, modified examples of the above-mentioned embodiment will be described. The modified examples described below may be applied to the embodiment alone or in combination with each other. The modified examples may be applied in place of the configuration described in the embodiment, or may be applied in addition to the configuration described in the embodiment.

The first and second embodiments described above may be implemented in combination. For example, the monitoring screen of the first embodiment may display a translation in the same manner as the monitoring screen of the second embodiment.
Furthermore, the monitoring screen of the second embodiment may display a UI capable of receiving a selection operation from the checker as to whether or not to display text data, similar to the monitoring screen of the first embodiment.

In the above-described first embodiment, an example is described in which the first language is a language other than Japanese and the second language is a plurality of languages including Japanese, but the present invention is not limited to such an example. For example, the first language may be Japanese and the second language may be a language other than Japanese.
In the above-described second embodiment, an example is described in which the first language is Japanese and the second language is a plurality of languages other than Japanese, but the present invention is not limited to such an example. For example, the first language may be a language other than Japanese and the second language may be a plurality of languages including Japanese.
In addition, in each of the above-described embodiments, the display device 40 may be capable of displaying subtitles in a first language.

The above describes the modified examples of the embodiment.
It should be noted that some or all of the functions of the subtitle display system 1, 1a and the audio processing device 20, 20a in the above-described embodiments may be realized by a computer. In this case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.
Additionally, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, etc., and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases.
Furthermore, the above program may be for realizing a part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in a computer system, or may be realized by using a programmable logic device such as an FPGA (Field Programmable Gate Array).

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the present invention.

1, 1a...subtitle display system, 10...sound collection device, 20, 20a...sound processing device, 21...simultaneous interpretation engine, 22...machine translation engine, 23...speech recognition engine, 24...conversion candidate API, 30...monitoring terminal, 40...display device, 41...screen, 42...smartphone, 210, 210a...communication unit, 220, 220a...storage unit, 230, 230a...first control unit, 231, 231a...speech data acquisition unit, 232, 232a...speech conversion unit, 240, 240a...second control unit, 241, 241a...monitoring processing unit, 242, 242a...machine translation unit, 243, 243a...subtitle processing unit, 244...morpheme analysis unit, 245...conversion priority processing unit, 246...automatic conversion processing unit

Claims (9)

a speech conversion unit that converts speech data indicating the contents of a user's speech in a first language into text data indicating a translation result into a plurality of second languages different from the first language by simultaneous interpretation;
a monitoring processing unit that accepts corrections to the text data for one designated second language among the plurality of second languages;
a translation unit that translates the corrected text data into the second language that was not specified at the time of correction, and obtains text data indicating a translation result in which the correction is reflected for each of the second languages;
An audio processing device comprising: a subtitle processing unit that displays the text data indicating the translation result reflecting the correction as a subtitle on a display device;
The audio processing device of claim 1 further comprising: the monitoring processing unit displays, on the monitoring terminal, a monitoring screen capable of accepting a correction operation for the text data converted from the voice data and a selection operation for whether or not to display the text data.
The audio processing device according to claim 1 . Subtitles of the text data for which "cannot be displayed" is selected on the monitoring screen are not displayed on a display device, and subtitles of the text data for which "can be displayed" is selected on the monitoring screen are displayed on the display device.
The audio processing device according to claim 3 . the monitoring processing unit displays the text data in a text field on the monitoring screen, switches the display enable/disable state to disabled when an inside of the text field is selected, and switches the display enable/disable state to enabled when an outside of the text field is selected after the inside of the text field is selected.
The audio processing device according to claim 3 . When a portion of the text data to be corrected is selected on the monitoring screen, the monitoring processing unit displays conversion candidates near the portion of the text data to be corrected, and inserts text selected from the conversion candidates into the portion of the text data to be corrected.
The audio processing device according to claim 3 . the monitoring processing unit adds, to the conversion candidates, text that is not included in the conversion candidates and is detected as a difference by comparing the text data before correction with the text data after correction.
The audio processing device according to claim 6. a speech conversion process for converting speech data representing the contents of a user's speech in a first language into text data representing the results of translation into a plurality of second languages different from the first language by simultaneous interpretation;
a monitoring process for receiving corrections to the text data for a specified one of the second languages;
a translation process of translating the corrected text data into the second language that was not specified at the time of correction, and acquiring text data indicating a translation result in which the correction is reflected for each of the second languages;
16. A computer-implemented method for audio processing comprising: Computer,
a speech conversion means for converting speech data representing the contents of a user's speech in a first language into text data representing the results of translation into a plurality of second languages different from the first language by simultaneous interpretation;
a monitoring processing means for receiving corrections to the text data for one designated second language among the plurality of second languages;
a translation means for translating the corrected text data into the second language that was not specified at the time of correction, and obtaining text data indicating a translation result in which the correction is reflected for each of the second languages;
A program to function as a

Images