JP2001175280A - Caption display device and storage medium for caption control - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To display subtitles as regularly as possible while correcting a time lag that occurs when subtitle creation processing is performed in parallel. SOLUTION: A plurality of voice writing means 4-1 to 4-N
When the subtitles are processed in parallel to process the audio signals divided in time order to obtain subtitles, the order in which the subtitles are completed may not coincide with the time order. Therefore, by assigning consecutive numbers to the divided audio signals (audio sequence number generation device 3) and providing a buffer storage device 7 for temporarily storing the newly written subtitles, the subtitles are displayed in a correct time sequence and periodically. I made it possible.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a caption display device and a storage medium for caption control.

More specifically, the present invention relates to a subtitle display device and a subtitle control storage medium for displaying subtitles corresponding to audio on a television screen in real time.

[0003]

2. Description of the Related Art Hitherto, it has been desired to add subtitles to television news almost in real time. In order to realize such a service, it is essential to write down the contents spoken by the announcer at high speed.

On the other hand, a method of sequentially writing down the contents of an announcer's utterance using a voice recognition device has been considered. In addition, a technique of rewriting at high speed using human power, that is, using a word processor has been considered. Then, by sequentially inputting the written sentences obtained by the voice rewriting device such as the voice recognition device or the word processor to the subtitle superimposing device, it becomes possible to add subtitles to the television screen in real time.

[0005]

However, when the utterance content is written using a single voice writing means, there is a problem that the subtitle display is delayed with respect to the voice because the writing speed is not sufficient. I do. In particular, when manual correction is performed after output from the voice recognition device as a voice writing means, the problem of time delay becomes serious.

In order to cope with such a time delay problem, for example, a set of a speech recognition device and a correction device is arranged in parallel, an input speech signal is divided, and each divided portion is divided in parallel. By recognizing the voice and correcting the voice recognition result, it is possible to create subtitles at high speed.
In addition, such a parallel voice writing method can be effectively applied even when a word processor is used.

However, although it is effective to operate the voice writing means in parallel as described above, in this case, the processing time of each parallel part is not constant. Therefore, if these processing results are displayed as captions as they are,
A problem arises in that the subtitle order is out of order. Furthermore, if the subtitles obtained in parallel are superimposed on the television screen as they are, there is a problem that the subtitles cannot be displayed at regular intervals.

Accordingly, an object of the present invention, in view of the above points, is to make it possible to display subtitles as regularly as possible while correcting a time lag that occurs when subtitle creation processing is performed in parallel. An object of the present invention is to provide a caption display device and a storage medium for caption control.

[0009]

To achieve the above object, the present invention according to claim 1 is a subtitle display device for displaying subtitles corresponding to audio on a television screen in real time, A subtitle display device comprising: a plurality of subtitle writing units for writing divided audio sections in parallel; and an alignment unit for displaying the subtitle fragments respectively written by the plurality of subtitle writing units in a correct order.

[0010] According to a second aspect of the present invention, in the subtitle display device according to the first aspect, the alignment means includes a buffer means for temporarily storing subtitles written in accordance with the divided audio signals.

According to a third aspect of the present invention, in the caption display device according to the first aspect, when a significant output delay occurs in a part of the plurality of subtitle writing means, the display interval is reduced. There is provided an omitting means for omitting the display in order to keep the intervals at equal intervals.

According to a fourth aspect of the present invention, in the subtitle display device according to the third aspect, when the subtitle display is omitted by the omitting means, a special symbol for calling a viewer's attention is displayed on the screen. I do.

According to a fifth aspect of the present invention, there is provided the subtitle display device according to the first aspect, further comprising an unfinished subtitle due to a delay in output from a part of the plurality of subtitle writing units. If only the subtitles have been created, the display device is provided with display control means for maintaining the display intervals at equal intervals by displaying the unfinished subtitles as they are.

According to a sixth aspect of the present invention, in the subtitle display device according to the fifth aspect, when the unfinished subtitle is displayed, a special symbol for calling a viewer's attention is displayed on the screen. .

According to a seventh aspect of the present invention, in the caption display device according to the first aspect, in addition to the above, a line end symbol corresponding to the number of characters of one line on the screen display is inserted in the middle of the character string of the subtitle fragment. An insertion means is provided so that subtitles are not unnaturally divided on the screen.

The present invention according to claim 8 is to provide a plurality of subtitle writing steps for writing in parallel time-division audio sections when displaying subtitles corresponding to audio in real time on a television screen; And a sorting step of displaying the subtitle fragments respectively written in the subtitle writing step in the correct order, in the form of a readable program.

According to a ninth aspect of the present invention, in the subtitle control storage medium according to the eighth aspect, the arranging step includes a step of temporarily storing subtitles written in accordance with the divided audio signals.

According to a tenth aspect of the present invention, in the subtitle control storage medium according to the eighth aspect, when a significant output delay occurs in a part of the plurality of subtitle writing means, a display is performed. There is an omitting step of omitting the display in order to keep the intervals at equal intervals.

[0019] According to an eleventh aspect of the present invention, in the subtitle control storage medium according to the tenth aspect, when the subtitle display is omitted in the omission step, a special symbol for calling a viewer's attention is displayed on the screen. To be displayed.

According to a twelfth aspect of the present invention, there is provided the storage medium for subtitle control according to the eighth aspect, wherein the output is delayed due to a delay in output from some of the plurality of subtitle writing means. If only subtitles have been created, a display control step of maintaining the display intervals at equal intervals by displaying the unfinished subtitles as they are is provided.

According to a thirteenth aspect of the present invention, in the subtitle control storage medium according to the twelfth aspect, when the unfinished subtitle is displayed, a special symbol for calling a viewer's attention is displayed on the screen. indicate.

According to a fourteenth aspect of the present invention, in the subtitle control storage medium according to the eighth aspect, in addition to the above, a line end symbol corresponding to the number of characters of one line on the screen display is provided in the middle of the character string of the subtitle fragment. An insertion step for inserting is provided so that subtitles are not unnaturally divided on the screen.

[0023]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is an overall configuration diagram of a caption display system to which the present invention is applied. In FIG. 1, reference numeral 1 denotes a television program audio output device, which supplies television audio, that is, an input to be processed by the present system.

Reference numeral 2 denotes a voice dividing device, which time-divides a voice signal for each pause portion of the voice. Reference numeral 3 denotes an audio sequence number generation device, and reference numerals 4-1 to 4-N denote subtitle fragment output devices.

The audio signal divided by the audio dividing device 2 is
The audio sequence number generation device 3 and the N subtitle fragment creation devices 4-1 to 4-N are sequentially input. Here, N is the number of caption fragment output devices to be operated in parallel, and is an arbitrary number of 2 or more.

The audio sequence number generator 3 generates an integer value starting from 1 corresponding to the time-division audio signal. That is, to each of the caption fragment creation devices 4-1 to 4-N, two data of an audio signal and an integer number indicating the order of the audio signal are input.

FIG. 2 shows a subtitle fragment output device 4 (4-1 to 4-4).
-N) is a block diagram showing a detailed configuration. The N subtitle fragment output devices 4 (4-1 to 4-N) all have the same configuration. That is, each subtitle fragment output device 4
(4-1 to 4-N) includes a character string creation device 40 and a line end symbol insertion device 41.

The character string creation device 40 is a device that outputs a character string corresponding to the input audio signal one character at a time. In addition, a binary symbol called a character string creation state is output. That is, S (Start) is output when the output of a character string is started with a voice input, and E (End) is output when the output is completed.

Each subtitle fragment output device 4 (4-1 to 4-
N) outputs the input speech sequence number as it is.

The line end symbol insertion device 41 is a device for inserting a line end symbol according to the number of characters when a character string is displayed on a television screen or the like. Here, one line 1 on the TV screen
It is assumed to be displayed in six characters, and an end-of-line symbol is inserted every 16 characters.

In order to concretely realize the subtitle fragment output device 4 (4-1 to 4-N), for example, in addition to using a speech recognition system as shown in FIG. 3, a word processor as shown in FIG. It is preferred to use. 3 and 4, the speech recognition device 140 and the word processor 2
Reference numeral 40 corresponds to the character string creation device 40 of FIG. In the case of the speech recognition device 140, the writing of the character string is automatically executed. In addition, immediately after the speech recognition device, the above-described correction device can be added. Further, when the word processing device 240 is used, the operator writes down the voice.

In each case, for a voice input and a voice sequence number input, a newly written character string grouped as one line for every 16 characters, a binary symbol indicating the state of creation of the character string, and a voice sequence number Is output.

Reference numeral 7 denotes a buffer storage device for temporarily storing outputs from the subtitle fragment output devices 4 (4-1 to 4-N). The character string, character string creation state, and voice sequence number, which are input data to the buffer storage device 7, are managed as a set of data. That is, if you specify a voice sequence number,
You can check the corresponding character string and the character string creation status at that moment. When a character string is specified, its voice sequence number and character string creation state can be confirmed.

Reference numeral 8 denotes a subtitle display control device. The subtitle display control device 8 executes the following processes 1 to 3 on the subtitle character strings stored in the buffer storage device 7. Display subtitles correctly in chronological order. Display subtitles at regular intervals as much as possible. To change the display according to the degree of completion of subtitle data.

More specifically, by performing processing described in detail later,
The subtitle data is output to the subtitle superimposing device 9.

The caption superimposing device 9 is a device for superimposing a caption signal on a television screen or the like. The caption superimposing device 9 has only a function of superimposing an input character string on a television screen while performing a line feed with a predetermined number of characters. The character string data to be actually superimposed and the display interval thereof are displayed in a caption display. Determined by the control device 8. The line feed in the caption superimposing device 9 is performed every 16 characters.

Reference numeral 10 denotes a storage medium (a ROM, a hard disk device, or the like) that stores a control procedure of the entire system in the form of a program. By reading the program, the execution timing and execution contents in each device are determined. .

Next, the detailed operation of the caption display control device 8 will be described with reference to the flowchart shown in FIG. The subtitle display control device 8 has a function of displaying subtitles composed of 16 characters per line on a television screen in units of two lines while executing the above-described processes (1) to (4).

The processing contents of each step shown in FIG. 5 are as follows.

Step S801 (initial processing):
The voice sequence number counter count is set to 0.

Step S802 (confirmation of output candidate): Here, it is confirmed that subtitle data is present in the buffer. Subtitle fragments that do not always follow the time sequence are input to the buffer. To output subtitles in chronological order, it is preferable to output them so that the audio sequence numbers are continuous. That is, the character string having the smallest voice sequence number in the buffer is an output candidate. However, as described in step S807, those for which the display processing has been completed are not candidates.

Step S803 (waiting for data): If there is no data in the buffer, the process waits until a preset number of data, ie, B data are accumulated. B can be any number from 0. The larger the number is, the later the start of displaying subtitles is. However, since the number of data is large, there is a high possibility that data can be displayed at regular intervals. It should be noted that the number of data is a unit managed by a voice sequence number, and is not a line specified by a line feed symbol.

Step S804 (time continuity check): It is determined whether or not the current output candidates are temporally continuous. Specifically, assuming that the current voice sequence number is n,

[0045]

[Equation 1] It is confirmed whether or not n = count + 1 (1) holds.

Step S805 (time wait): If the current data is not continuous, wait for C seconds. Then, the continuity is checked again in step S806. The value of C is manually specified in advance.

Step S806 (time continuity check): Same as step S804.

Step S807 (data loss processing): In step S806, continuity is checked again. If continuity is not maintained, data continuity is abandoned, and the process proceeds to a process of outputting the current candidate. Therefore, a special symbol XXX is used to indicate that the data was discontinuous.
X is output to the caption superimposing device 9 for D / 2 seconds. D is subtitle 2
The number of seconds to display the line, given in advance. Special symbols indicate half the time. Also, discontinued co
The data of unt + 1 is determined to have been processed by displaying XXXX, and is not displayed even if it is input to the buffer with a delay.

Step S808 (two-line display output): If the current output candidate data is being input to the buffer, the input is stopped. The character string creation state at this time is stored. It is assumed that the output candidate is m-line data with 16 characters as a line unit. These are arranged in chronological order. Also, 2
In principle, display subtitles for each line. The display time of two lines is D seconds. The displayed line is deleted from the buffer. The following processing (1) to processing (3) are performed on the m rows until they become empty.

Process (1): One line at the beginning is taken out,
This is A. If A ends at a period, it outputs A to the caption superimposing device 9 for F seconds. F is a shorter number of seconds than D,
For example, it is set to 2/3 seconds. The output data is deleted from the buffer and the process proceeds to processing (3). If A does not end at a period, the process proceeds to processing (2).

Process (2): One more row is taken out and set to B. A and B are output to the caption superimposing device 9 for D seconds. If B is empty, only A is output for F seconds. The output data is deleted from the buffer and the process proceeds to processing (3).

Process (3): The contents of A and B are deleted and the process returns to process (1).

Step S809 (Confirmation of Completeness of Output Candidate Data): It is confirmed whether or not the caption data for which the output processing in step S808 has been completed is complete. For this, the character string creation state symbol stored in step S808 is confirmed. If E, the character string is completed. If it is S instead of E, that is, if a part of the character string is not yet completed, a special symbol. . . Is output for D / 2 seconds. This symbol indicates that the character string was interrupted.

Step S810 (post-display processing): The counter is incremented by one. That is,

[0055]

## EQU2 ## It is assumed that count ← count + 1 (2).

Next, the above operation will be described with reference to FIG.

FIG. 6 shows a state in which data is stored in the buffer storage device 7 at a certain moment, and it is assumed that the lowest data is the latest data. In the figure, three alphabetical characters are used as one line of data, which is interpreted as 16 characters. It is also assumed that the processing up to voice sequence number 2 has already been completed.

Since the data of the voice sequence number 3 is 3 lines and does not meet the special condition, AAA and BBB are displayed for D seconds and CCC is displayed for F seconds.

Next, the display shifts to the display of the data of the voice number 4.
First, since DDD ends at a period, this is displayed for F seconds. Next, the remaining EEE is displayed for F seconds.

Here, the counter becomes 5, but the voice sequence number of the next data is 6, which is discontinuous. Therefore, data is waited for C seconds in the process of step S805. Here, if the data of the voice sequence number 5 does not come, the special symbol XXXX is output. Then, the data of the voice sequence number 6 is displayed. (If the data of the voice sequence number 5 arrives, the data of the voice sequence number 6 is displayed after the execution of the display processing.) The data of the voice sequence number 6 is two lines, and the FFF and GGG are displayed for D seconds. I do.

Next, the process proceeds to the display of the data of the voice sequence number 7. This data is currently being written to the buffer. Therefore, first, writing to the buffer is prohibited. Since this data consists of two lines, HHH and III are displayed for D seconds. Next, by confirming the data integrity in step S809, it is found that the character string creation state is S, and the special symbol. . . Is output for D / 2 seconds.

Effects of Embodiment In the above-described embodiment, when the subtitle character string fragments are input irregularly in time from the parallel audio writing means, that is, the subtitle fragment creation device, The following effects (1) to (5) are achieved.

(1) Captions that correctly reflect the order of audio can be created. Since the subtitles are displayed in this order using the audio sequence number, the subtitles can be displayed in the correct order. However,
If the creation of a subtitle fragment is significantly delayed, the display of that part is omitted and the special symbol XXXX is displayed. This caption is not displayed even if it is subsequently input, so that no time reversal occurs.

(2) The regularity of the display interval can be ensured. Basically, two lines of subtitles are displayed at intervals of D seconds. However, when a line ends at a period and when there is only one line of data, only that line is displayed for F seconds shorter than D. The special treatment of punctuation is due to readability considerations. There are two cases where the display interval deviates from this principle.

First, there is a case where data is lost in the buffer. At this time, it waits until the predetermined B data is accumulated. The display starts as soon as this number is reduced to zero, but the data in the buffer may soon be exhausted. When the number is increased, the punctuality of the display is improved, but it takes time until data is accumulated in the empty buffer, and subtitles cannot be obtained during this time.

The other is a case where time continuity of output candidates is not maintained. In this case, it waits for an extra C seconds.

It should be noted that in each case, the processing takes extra time. At least D seconds or more can be displayed for two lines with the specified display time, and there is no problem that the display is too fast to read.

(3) Regularity can be ensured by omitting subtitles. When trying to display the caption data of a certain number and it is not in the buffer, it waits for C seconds. If this value is set to about D seconds of display instead of a large value, there is no uncomfortable feeling about the extension of the display interval due to the waiting time. If subtitle data does not come, the subtitle is not displayed. This can prevent a significant delay in subtitle display. In this case, by displaying the special symbol XXXX, the viewer can know that the subtitle is partially missing.

(4) Unfinished subtitles can be displayed. When a subtitle is displayed, if the input is an unfinished subtitle or a partial input, it is displayed. Thereby, the punctuality of the subtitle display is secured. When such an unfinished subtitle is displayed, a special symbol is displayed. . . Can be notified to the viewer that the caption is incomplete.

(5) The effect of inserting the end-of-line symbol can be obtained. In the description so far, it has been described that the subtitle screen display is 16 characters per line, and the subtitle fragment output device 4 inserts a line end symbol into the character string every 16 characters in accordance with this. Suppose that a line end symbol is inserted every 13 characters, for example, regardless of the screen display. In this case, when the subtitle data of two lines is sent to the display device, the subtitle data is displayed as a character string of two lines composed of one line composed of 16 characters and 10 characters. The next display starts from the 11th character on the second line after D seconds. That is, an unnatural character string display in which the display is restarted from the middle of the caption is obtained. By supplying a character string from the buffer to the subtitle display device with the number of characters corresponding to one line of the subtitle superimposition device 9,
Such a problem can be prevented from occurring.

[0071]

As described above, according to the present invention, it is possible to correct a time lag that occurs when subtitle creation processing is performed in parallel and to display subtitles as regularly as possible. A display device and a storage medium for caption control can be realized.

[Brief description of the drawings]

FIG. 1 is an overall configuration diagram of a caption display system to which the present invention is applied.

FIG. 2 is a detailed configuration diagram of a subtitle fragment output device.

FIG. 3 is a diagram showing a caption fragment output device using a speech recognition device.

FIG. 4 is a diagram showing a caption fragment output device using a word processor.

FIG. 5 is a flowchart showing a control procedure of the caption display control device.

FIG. 6 is an operation explanatory diagram of the present embodiment.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 TV program audio output apparatus 2 Audio division apparatus 3 Audio sequence number generation apparatus 4-1 ..., 4-N Subtitle fragment output apparatus 7 Buffer storage apparatus 8 Subtitle display control apparatus 9 Subtitle superposition apparatus 10 Control program storage medium 40 Character string creation device 41 Line end symbol insertion device 140 Speech recognition device 142 Information source (acoustic model, language model, kanji pronunciation dictionary) 240 Word processor 242 Information source (dictionary)

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Jun Goto 1-10-11 Kinuta, Setagaya-ku, Tokyo Inside the Japan Broadcasting Corporation Broadcasting Research Institute (72) Inventor Atsushi Imai 1-10-11 Kinuta, Setagaya-ku, Tokyo No. Japan Broadcasting Corporation Broadcasting Research Institute (72) Inventor Toru Toki 1-10-11 Kinuta, Setagaya-ku, Tokyo Japan Broadcasting Corporation Broadcasting Research Institute (72) Inventor Akio Ando 1-kinuta, Setagaya-ku, Tokyo No. 10-11 Japan Broadcasting Corporation F-Term in the Broadcasting Research Institute (reference) 5C023 AA18 BA01 BA16 CA04 CA05 DA03 5D015 AA01 AA02 9A001 BB06 HH15 HH17

Claims (14)

[Claims] 1. A subtitle display device for displaying subtitles corresponding to audio on a television screen in real time, a plurality of subtitle writing units for writing time-divided audio sections in parallel, and the plurality of subtitle writing units And a sorting means for displaying the subtitle fragments newly written by the means in a correct order. 2. The subtitle display device according to claim 1, wherein the arrangement unit has a buffer unit for temporarily storing subtitles written in accordance with the divided audio signals. . 3. The subtitle display device according to claim 1, further comprising: when a significant output delay occurs in a part of the plurality of subtitle writing units, to maintain display intervals at equal intervals. A caption display device comprising an omitting means for omitting the display. 4. The subtitle display device according to claim 3, wherein when the subtitle display is omitted by the omitting means, a special symbol for calling attention to a viewer is displayed on the screen. Subtitle display device. 5. The subtitle display device according to claim 1, further comprising a case where only incomplete subtitles are created due to a delay in output from a part of the plurality of subtitle writing units. A display control means for maintaining the display intervals at equal intervals by displaying the unfinished subtitles as they are. 6. The subtitle display device according to claim 5, wherein, when the unfinished subtitle is displayed, a special symbol for calling a viewer's attention is displayed on a screen. Display device. 7. The caption display device according to claim 1, further comprising: an insertion unit that inserts an end-of-line symbol corresponding to the number of characters in one line of the screen display, in the middle of the character string of the subtitle fragment. A subtitle display device, wherein the subtitle is not unnaturally divided above. 8. When displaying subtitles corresponding to audio on a television screen in real time, a plurality of subtitle writing steps for writing time-divided audio sections in parallel and a plurality of subtitle writing steps are respectively written. And storing the subtitle fragments in the correct order in the form of a readable program. 9. The subtitle control storage medium according to claim 8, wherein the arranging step includes a step of temporarily storing a subtitle written down corresponding to the divided audio signal. Storage media. 10. The subtitle control storage medium according to claim 8, further comprising: maintaining a display interval at an equal interval when a significant output delay occurs in a part of the plurality of subtitle writing units. A subtitle control storage medium provided with an omission step of omitting the display for the purpose. 11. The subtitle control storage medium according to claim 10, wherein when the subtitle display is omitted in the omitting step,
A storage medium for caption control, wherein a special symbol for calling a viewer's attention is displayed on a screen. 12. The subtitle control storage medium according to claim 8, wherein only unfinished subtitles are created due to a delay in output from a part of the plurality of subtitle writing units. If not, a subtitle control storage medium including a display control step of displaying the unfinished subtitle as it is to keep display intervals at equal intervals. 13. The subtitle control storage medium according to claim 12, wherein when the unfinished subtitle is displayed, a special symbol for calling a viewer's attention is displayed on a screen. Subtitle control storage medium. 14. The subtitle control storage medium according to claim 8, further comprising an insertion step of inserting an end-of-line symbol corresponding to the number of characters of one line on a screen display, in the middle of the character string of the subtitle fragment. ,
A subtitle control storage medium wherein subtitles are not unnaturally divided on a screen.