JP2015018079A - Subtitle voice generation apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a subtitle voice generation apparatus which can generate subtitle voice alleviating discomfort to a user.SOLUTION: A subtitle voice generation apparatus includes a voice analysis part which analyzes a condition of a manner of speaking of a person on the basis of voice data to be input, and a synthetic voice generation part which generates subtitle voice being synthetic voice on the basis of subtitle data corresponding to the voice data and the analysis result by the voice analysis part.

Description

  The present invention relates to an apparatus for generating caption audio.

  2. Description of the Related Art Conventionally, there is known a caption audio generation device that generates a caption audio by synthesizing a subtitle character string included in a broadcast signal or the like.

  For example, in Patent Document 1, a subtitle portion is extracted from a video signal, a character string included in the extracted subtitle portion is character-recognized, a character string of the character-recognized subtitle is voice-synthesized, and subtitle audio is transmitted from a speaker. A caption extraction device for output is disclosed.

  In this caption extraction device, the characteristics of the sound quality are analyzed from the input sound signal, and the sound quality closest to the sound quality is selected from the sound database and synthesized. For example, when the voice output when a Japanese subtitle is displayed in a foreign movie is an actress's voice, voice synthesis is performed with the female voice quality based on the voice quality characteristics of the voice.

  As a result, the synthesized speech that tends to be inorganic can be given some individuality.

JP 2003-333445 A (page 5)

  However, in a movie or the like, it is normal for a person to speak while changing the speaking speed, voice strength, etc. However, in the above-mentioned Patent Document 1, this point is not taken into account when synthesizing speech, and a viewer who listens to subtitle speech. A sense of incongruity.

  In view of the above problems, an object of the present invention is to provide a caption audio generation device that can generate caption audio that reduces a sense of discomfort given to a user.

  In order to achieve the above object, a subtitle audio generating apparatus according to the present invention includes an audio analysis unit that analyzes a state of speech of a person based on input audio data, subtitle data corresponding to the audio data, and the audio analysis unit. And a synthesized speech generation unit that generates subtitle speech that is synthesized speech based on the analysis result of the above.

  According to such a configuration, the state of the person's speaking can be reflected in the subtitle sound, so that it is possible to generate subtitle sound that reduces a sense of discomfort given to the user.

  In the above-described configuration, the person's speaking state may be voice speed and / or voice strength.

  According to such a configuration, the speed of the voice and / or the strength of the voice can be reflected in the subtitle sound. In particular, when the speed of the voice is reflected in the caption audio, it is possible to suppress the difference between the movement of the mouth of the person and the caption audio in the video, and it is possible to suppress a sense of discomfort for the user watching the video.

  In any one of the above-described configurations, the speech analysis unit detects a plurality of persons based on the speech data, and the synthesized speech generation unit stores a plurality of synthesized speech patterns that are stored and prepared in advance in a storage unit. The subtitle sound may be generated by assigning the synthesized sound pattern to each of the detected plurality of persons.

  According to such a configuration, when a plurality of persons appear, it is possible to generate subtitle sound as if the plurality of persons are speaking, and it is possible to further suppress a sense of discomfort for the user. In addition, since a plurality of synthesized speech patterns prepared in advance are assigned, subtitle speech can be generated quickly.

  In the above configuration, the synthesized speech generation unit may prepare the plurality of synthesized speech patterns in advance on the storage unit based on program information.

  According to such a configuration, a plurality of synthesized voice patterns can be prepared as necessary according to the program, and preparation more than necessary can be suppressed.

  Further, the plurality of synthesized speech patterns may be synthesized speech patterns having different voice qualities prepared for each gender.

  According to such a configuration, when a plurality of persons with different genders appear, subtitle sound can be generated as if a plurality of persons with different genders are talking.

  In any one of the configurations described above, when the synthesized sound generation unit detects that there is no sound or detects that a person is not shown in the video, the synthesized sound generating unit outputs the monotonous subtitle sound based on the subtitle data. It may be generated.

  According to such a configuration, it is possible to generate caption audio such as narration when there is no audio or when a person is not shown in the video.

  In any of the above-described configurations, sound based on the sound data may be generated from a built-in speaker, and sound based on the subtitle sound may be output from an external output terminal.

  According to such a configuration, for example, a user who wants to hear a sound in a foreign language can hear the sound from the built-in speaker, and a user who wants to hear the sound in his / her own language can hear the subtitle sound output from the external output terminal.

  In any of the above configurations, sound based on the sound data may be generated from a built-in speaker, and the subtitle sound may be transmitted to an external mobile device using a radio signal.

  According to such a configuration, for example, a user who wants to hear a sound in a foreign language can hear the sound from the built-in speaker, and a user who wants to hear the sound in his / her own language can hear the subtitle sound output from the mobile device at hand.

  According to the present invention, it is possible to generate subtitle sound that reduces a sense of discomfort given to a user.

It is a block diagram which shows schematic structure of the television apparatus which concerns on 1st Embodiment of this invention. It is a flowchart regarding the audio | voice analysis process which concerns on embodiment of this invention. It is a figure which shows an example of the some synthetic speech pattern which concerns on embodiment of this invention. It is a block diagram which shows schematic structure of the television apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram which shows schematic structure of the television apparatus which concerns on 3rd Embodiment of this invention. It is a block diagram which shows schematic structure of the television apparatus which concerns on 4th Embodiment of this invention.

<First Embodiment>
An embodiment of the present invention will be described below with reference to the drawings. Hereinafter, a television device will be described as an example of the caption audio generation device. FIG. 1 is a block diagram showing a schematic configuration of the television apparatus according to the first embodiment of the present invention. 1 includes a tuner 11, a demodulator 12, a separator 13, a video decoder 14, a data decoder 15, an audio decoder 16, a video output unit 17, a display unit 18, and an OSD. An (on-screen display) unit 19, a voice analysis unit 20, a synthesized voice generation unit 21, a voice output unit 22, and a speaker 23 are provided. The antenna 11 is connected to the tuner 11.

  The tuner 11 corresponds to, for example, at least one of terrestrial digital broadcast, BS digital broadcast, and CS digital broadcast, and selects a broadcast signal of a desired channel from a high-frequency broadcast signal input from the antenna 2.

  The demodulator 12 performs processing such as digital demodulation and error correction on the broadcast signal of the channel selected by the tuner 11, generates a transport stream, and outputs the transport stream to the separator 13.

  The separation unit (demultiplexer) 13 separates the transport stream input from the demodulation unit 12 into a video stream, an audio stream, caption data, and the like.

  The video decoder 14 decodes the video stream input from the separation unit 13 and outputs the generated video data to the video output unit 17.

  The data decoder 15 decodes the caption data input from the separation unit 13 and outputs the generated caption text data to the OSD unit 19.

  The OSD unit 19 generates display data for on-screen display such as menu display and outputs the generated display data to the video output unit 17. The OSD unit 19 can also generate caption display data based on the caption text data input from the data decoder 15, and outputs the caption display data to the video output unit 17.

  The video output unit 17 superimposes the display data input from the OSD unit 19 on the video data input from the video decoder 14, converts the superimposed video data into a video signal suitable for the display unit 18, and displays the display unit 18. Output to. In some cases, only video data input from the video decoder 14 without being superimposed or display data input from the OSD unit 19 is converted into a video signal and output to the display unit 18.

  The display unit 18 is, for example, a liquid crystal display, and displays a video based on the video signal input from the video output unit 17. As a result, video of a broadcast program including subtitles and various videos such as a menu screen are displayed on the display unit 18.

  The audio decoder 16 decodes the audio stream input from the separation unit 13 and outputs the generated audio data to the audio output unit 22. The audio output unit 22 converts the audio data input from the audio decoder 16 into an audio signal suitable for the speaker 23 and outputs the audio signal to the speaker 23. The speaker 23 generates sound based on the audio signal input from the audio output unit 22. Thereby, the sound of the broadcast program is generated from the speaker 23.

  The audio decoder 16 can also output the generated audio data to the audio analysis unit 20. The voice analysis unit 20 analyzes the voice data input from the voice decoder 16 and notifies the voice synthesis unit 21 of the analysis result.

  The synthesized voice generation unit 21 generates subtitle voice data that is a synthesized voice based on the caption text data input from the data decoder 15 and the analysis result of the voice notified from the voice analysis unit 20, and outputs it to the voice output unit 22. . In this case, the audio output unit 22 converts the subtitle audio data input from the synthesized audio generation unit 21 into an audio signal suitable for the speaker 23 and outputs the audio signal to the speaker 23. Thereby, subtitle sound is generated from the speaker 23.

  Next, the caption audio output operation in the television apparatus 1 according to the present embodiment will be described in more detail.

  For example, it is assumed that the sound of a certain broadcast program is only in English, and the subtitles are in Japanese and English. In this case, as the normal mode, the video data output from the video decoder 14 is output from the OSD unit 19 based on the Japanese caption text data output from the data decoder 15 (the language of the caption text data is selectable). The subtitle display data is superimposed on the video output unit 17, and a broadcast video including Japanese subtitles is displayed on the display unit 18 (selection not to display subtitles is also possible). In the normal mode, audio data in English is output from the audio decoder 16 to the audio output unit 22, and broadcast audio in English is generated from the speaker 23.

  If you want to watch the broadcast in English, you can watch it in the normal mode, but if you want to watch in Japanese, you can move to the subtitle audio output mode described below. The transition to the caption audio output mode is performed by, for example, a control unit (not shown) of the television device 1 according to an operation by a remote control device (not shown). When the subtitle audio output mode is entered, the audio decoder 16 does not output audio data to the audio output unit 22 but outputs only to the audio analysis unit 20. Thereby, the sound by the sound data of the broadcast signal (the sound in English in the above case) is not output from the speaker 23. In the caption audio output mode, the video display is the same as in the normal mode.

  Here, analysis processing by the voice analysis unit 20 will be described with reference to a flowchart shown in FIG. The voice analysis unit 20 includes a buffer (not shown), and the voice analysis unit 20 analyzes the voice data input from the voice decoder 16 and stored in the buffer. Further, the process of FIG. 2 is repeatedly performed.

  When the flowchart shown in FIG. 2 is started, first, in step S1, the speech analysis unit 20 acquires speech features such as frequency features of speech from the speech data to be analyzed.

  Next, in step S2, the voice analysis unit 20 determines whether the voice feature acquired in step S1 matches the voice feature acquired in the past.

  If the voice features do not match in step S2 (N in step S2), the process proceeds to step S3, and the voice analysis unit 20 determines whether the voice is a male voice or a female voice based on the voice data to be analyzed. I do.

  After step S3, in step S4, the voice analysis unit 20 detects the speed of the voice and the strength of the voice based on the voice data to be analyzed.

  Then, in step S5, the voice analysis unit 20 registers the newly detected person of male or female in association with the voice feature acquired in step S1. Here, registration is performed according to the sex result determined in step S3. For example, when it is determined that the voice is male, males A, B, C,... Are registered in the order of registration, and when it is determined that the voice is female, females A, B, C,. . Note that the voice feature acquired in the past for determination in step S2 is a voice feature registered here.

  In step S6, the voice analysis unit 20 notifies the synthesized voice generation unit 21 of the person newly registered in step S5 and the speed and strength of the voice detected in step S4.

  If the voice features match in step S2 (Y in step S2), the process proceeds to step S7, where the voice analysis unit 20 detects the speed of the voice and the strength of the voice based on the voice data to be analyzed. Then, in step S8, the voice analysis unit 20 determines the person corresponding to the voice feature that matches the voice feature acquired in step S1 (that is, the person detected in the past), and the speed of the voice detected in step S7. The synthesized voice generation unit 21 is notified of the strength.

  After step S6 or step S8, the process is completed (end).

  Then, the synthesized speech generation unit 21 generates caption speech data that is synthesized speech based on the analysis result notified from the speech analysis unit 20 in step S6 or step S8 and the caption text data input from the data decoder 15. .

  More specifically, as shown in FIG. 3, for example, the synthesized speech generation unit 21 stores and prepares a plurality of synthesized speech patterns having different voice qualities for each gender in advance in a storage unit (not shown). A synthesized speech pattern corresponding to the person notified from the analysis unit 20 is selected and used to generate caption audio data. For example, if the person notified from the voice analysis unit 20 is “male C”, the synthesized voice pattern “male voice C” in FIG. 3 is selected.

  Then, the audio output unit 22 converts the subtitle audio data input from the synthesized audio generation unit 21 into an audio signal suitable for the speaker 23, thereby generating subtitle audio from the speaker 23.

  As described above, in the present embodiment, the television apparatus 1 (caption sound generation apparatus) analyzes the voice speed and strength (state of person's speaking) based on the sound data input from the sound decoder 16. And a synthesized speech generation unit 21 that generates captioned speech that is synthesized speech based on the caption text data corresponding to the speech data and the analysis result by the speech analysis unit 20.

  Thereby, since the speed and strength of the voice (state of how the person speaks) can be reflected in the subtitle sound, it is possible to generate subtitle sound that reduces a sense of discomfort given to the user.

  In particular, when the speed of the voice is reflected in the caption audio, it is possible to suppress the difference between the movement of the person's mouth in the image and the caption audio, and to suppress the uncomfortable feeling given to the user watching the image.

  Also, the voice analysis unit 20 detects a plurality of persons based on the voice data (the process of FIG. 2), and the synthesized voice generation unit 21 stores a plurality of synthesized voice patterns (stored and prepared in advance in the storage unit). For example, from FIG. 3), a synthesized voice pattern is assigned to each of the detected plurality of persons to generate subtitle sound.

  Thereby, when a plurality of persons appear, subtitle sound as if the plurality of persons are speaking can be generated, and the uncomfortable feeling given to the user can be further suppressed. In addition, since a plurality of synthesized speech patterns prepared in advance are assigned, subtitle speech can be generated quickly.

  For example, as shown in FIG. 3, the plurality of synthesized speech patterns may be synthesized speech patterns having different voice qualities prepared for each gender.

  Thereby, when a plurality of persons with different genders appear, subtitle sound as if a plurality of persons with different genders are speaking can be generated.

Second Embodiment
Next, a second embodiment of the present invention will be described. FIG. 4 is a block diagram showing a schematic configuration of the television apparatus according to the second embodiment of the present invention. Hereinafter, differences from the first embodiment (FIG. 1) of the television apparatus 1 ′ shown in FIG. 4 will be mainly described.

  The television apparatus 1 ′ illustrated in FIG. 4 includes a network interface 24. The network interface 24 can be connected to the Internet 3 and communicates with the server device 4 via the Internet 3.

  The data decoder 15 'outputs the caption text data to the synthesized speech generation unit 21' as in the first embodiment. At the same time, the data decoder 15 ′ decodes SI (Service Information) separated from the transport stream by the separation unit 13, and generates EPG (Electronic Program Guide) information included in the decoded SI from the synthesized speech generation unit. To 21 '.

  The synthesized voice generation unit 21 ′ transmits, for example, detailed program information included in the input EPG information to the server device 4 via the Internet 3 using the network interface 24.

  Here, the server device 4 has a database in which person names and genders are associated. And the server apparatus 4 searches the program performer contained in the received program detailed information using the said database, and detects the number of the performers for every sex (for example, 10 men, 8 women, etc.). Then, the server device 4 sends the detection result to the synthesized speech generation unit 21 ′ via the Internet 3 and the network interface 24.

  The synthesized speech generation unit 21 ′ prepares the synthesized speech pattern having the number of patterns for each gender in advance in a storage unit (not shown) according to the received detection result (for example, the detection result is 10 males). If there are eight people and eight women, the synthetic voice pattern shown in FIG. 3 is prepared for 10 patterns for men and 8 patterns for women).

  Then, the synthesized speech generation unit 21 ′ selects the prepared synthesized speech pattern according to the person notified from the speech analysis unit 20, and uses it for generating caption audio data.

  As described above, according to the present embodiment, the synthesized speech generation unit 21 ′ prepares a plurality of synthesized speech patterns in advance based on EPG information (program information). As a result, a plurality of synthesized voice patterns can be prepared as necessary according to the program, and preparation more than necessary can be suppressed.

<Third Embodiment>
Next, a third embodiment of the present invention will be described. FIG. 5 is a block diagram showing a schematic configuration of the television apparatus according to the third embodiment of the present invention.

  The difference between the television apparatus 1 ′ ′ shown in FIG. 5 and the first embodiment (FIG. 1) is the sound output form.

  The television apparatus 1 ′ illustrated in FIG. 5 includes an audio output unit 25, an external output terminal 26, an audio output unit 27, and a speaker 28. A terminal of the headphone 5 can be attached to and detached from the external output terminal 26.

  In the first embodiment (FIG. 1), when the subtitle audio output mode is entered, the audio decoder 16 does not output audio data to the audio output unit 22, and only the subtitle audio is output from the speaker 23. In the subtitle audio output mode, the audio decoder 16 outputs audio data to the audio output unit 27 in addition to the audio analysis unit 20.

  As a result, the synthesized speech generation unit 21 generates subtitle speech that is synthesized speech and outputs it to the audio output unit 25. Then, the audio output unit 25 outputs a subtitle audio signal to the headphones 5 connected to the external output terminal 26 via the external output terminal 26. Accordingly, subtitle sound is generated from the headphones 5.

  At the same time, the audio output unit 27 converts the audio data input from the audio decoder 16 into an audio signal suitable for the speaker 28 and outputs the audio signal to the speaker 28. Thereby, the sound of the broadcast program is generated from the speaker 28.

  According to the present embodiment as described above, sound of a broadcast program (for example, sound in English) is generated from the speaker 28 of the television apparatus 1 ′ ′, and subtitle sound (for example, sound in Japanese) is generated from the headphones 5. Therefore, a user who wants to listen to the broadcast program sound, for example, wants to learn a foreign language, can hear the sound from the speaker 28, and a user who wants to hear the subtitle sound can hear the sound generated from the headphones 5.

  If the headphone 5 listens to the caption sound, the user who uses the headphones suppresses the sound from the speaker 28 from being heard, and the user who listens to the sound from the speaker 28 suppresses the sound of the caption sound from being heard. it can.

  Further, even a blind user who cannot understand the language of the audio of the broadcast program can appreciate the broadcast by listening to the subtitle audio generated from the headphones 5.

  In the present embodiment, it is also possible to apply a configuration (FIG. 4) capable of connecting to the Internet as in the second embodiment.

<Fourth embodiment>
Next, a fourth embodiment of the present invention will be described. FIG. 6 is a block diagram showing a schematic configuration of a television apparatus 1 ′ ″ ′ according to the fourth embodiment of the present invention.

  A difference from the third embodiment (FIG. 5) of the television apparatus 1 ′ ′ illustrated in FIG. 6 is that a wireless communication unit 30 is provided.

  In the present embodiment, in the subtitle audio output mode, the subtitle audio data is output from the synthesis generation unit 21 to the audio output unit 29 and the audio signal is output from the audio output unit 27 to the speaker 28 as in the third embodiment. Is output. The audio output unit 29 outputs the input subtitle audio data to the wireless communication unit 30.

  The wireless communication unit 30 performs wireless communication with the mobile device 6 in accordance with standards such as Bluetooth and Wi-Fi. The mobile device 6 is, for example, a smartphone or a mobile phone.

  The wireless communication unit 30 transmits the audio data input from the audio output unit 29 to the mobile device 6 using a radio signal conforming to the corresponding standard. Thereby, subtitle sound is generated from the built-in speaker of the mobile device 6.

  According to the present embodiment as described above, sound of a broadcast program (for example, sound in English) is generated from the speaker 28 of the television apparatus 1 ′ ″, and subtitle sound (for example, sound in Japanese) is generated from the mobile device 6. To do. Accordingly, a user who wants to listen to the broadcast program sound, for example, wants to learn a foreign language, can hear the sound from the speaker 28, and a user who wants to hear the subtitle sound can hear the sound generated from the built-in speaker of the mobile device 6 at hand. . In particular, subtitle sound can be heard from the mobile device 6 in a room different from the room where the television apparatus 1 ′ ′ is installed.

  Moreover, even a blind user who cannot understand the language of the audio of the broadcast program can appreciate the broadcast by listening to the subtitle audio generated from the mobile device 6 at hand. In addition, if the user listens to the sound through the headphones connected to the mobile device 6, the user who uses the headphones is prevented from hearing the sound from the speaker 28, and the user who is listening to the sound from the speaker 28. The subtitle sound can be suppressed from being heard.

  Note that the mobile device 6 may be a speaker device compatible with Bluetooth, for example.

  Further, in the present embodiment, it is also possible to apply a configuration (FIG. 4) capable of connecting to the Internet as in the second embodiment.

<Fifth Embodiment>
In the first to fourth embodiments, the synthesized speech generation unit 21 (or 21 ′) may further have the following configuration.

  When the synthesized voice generating unit 21 detects that there is no voice based on the voice data output from the voice decoder 16, the synthesized voice generating unit 21 determines that the voice is narrated, and generates synthesized voice from the subtitle text data with a constant voice speed and strength. . At this time, a predetermined pattern of male or female is used as the synthesized voice pattern.

  Alternatively, when the synthesized audio generation unit 21 detects that no person is shown in the video based on the video data output from the video decoder 14, it may determine that the voice is narrated as described above.

  According to the present embodiment, when it is determined that the narration is determined, monotonous subtitle sound is generated, and the user can listen to the subtitle sound that makes the user think that it is narration.

  As mentioned above, although embodiment of this invention was described, if it is in the range of the meaning of this invention, embodiment can be variously changed.

  For example, the present invention is not limited to a television device as long as it can receive a broadcast signal, and may be applied to, for example, a hard disk recorder, an optical disk recorder, a set top box, and the like. Also, the video and audio and the subtitle are not limited to the broadcast signal, but may be based on the reproduction signal.

DESCRIPTION OF SYMBOLS 1 Television apparatus 2 Antenna 3 Internet 4 Server apparatus 5 Headphone 6 Mobile device 11 Tuner 12 Demodulation part 13 Separation part 14 Video decoder 15 Data decoder 16 Audio decoder 17 Video output part 18 Display part 19 OSD part 20 Audio analysis part 21 Synthetic voice production | generation Unit 22 audio output unit 23 speaker 24 network interface 25 audio output unit 26 external output terminal 27 audio output unit 28 speaker 29 audio output unit 30 wireless communication unit

Claims (8)

A voice analysis unit that analyzes the state of the person's speech based on the input voice data;
A synthesized voice generation unit that generates subtitle voice that is a synthesized voice based on the caption data corresponding to the voice data and the analysis result by the voice analysis unit;
A subtitle sound generating apparatus comprising:   The subtitle sound generating apparatus according to claim 1, wherein the person's way of speaking is voice speed and / or voice strength. The voice analysis unit detects a plurality of persons based on the voice data,
The synthesized speech generation unit generates the subtitle speech by assigning the synthesized speech pattern to each of the plurality of detected persons from a plurality of synthesized speech patterns stored and prepared in advance in a storage unit. The caption audio generation device according to claim 1 or 2, characterized in that   4. The caption audio generation device according to claim 3, wherein the synthesized audio generation unit prepares the plurality of synthesized audio patterns in advance on the storage unit based on program information.   5. The caption audio generation device according to claim 3, wherein the plurality of synthesized speech patterns are synthesized speech patterns of different voice qualities prepared for each gender.   The synthetic voice generation unit generates the monotonic subtitle audio based on the subtitle data when detecting that there is no audio or detecting that no person is shown in the video. The caption audio generation device according to any one of claims 1 to 5.   The subtitle sound generation according to any one of claims 1 to 6, wherein sound based on the sound data is generated from a built-in speaker and sound based on the subtitle sound is output from an external output terminal. apparatus.   The sound based on the sound data is generated from a built-in speaker, and the subtitle sound is transmitted to an external mobile device using a radio signal. Subtitle audio generator.

Images