JP2000324395A - How to add timing information to subtitles - Google Patents

Abstract

(57) [Summary] [Problem] For each of presentation unit subtitles obtained by dividing a subtitle sentence text as a base of a subtitle at an appropriate position according to a predetermined presentation format, high-precision timing information corresponding to the division position is provided. It is an object to provide a method of automatically adding timing information to captions. SOLUTION: By providing reference timing information to each part at least to a subtitle sentence text before division as a base of a subtitle and dividing the subtitle sentence text at an appropriate place according to a predetermined presentation format. Perform presentation subtitles,
At least one of the start point / end point of each presentation unit subtitle is based on reference timing information and character information including at least one of a character type and the number of characters presented by the presentation unit subtitle, or a phonetic symbol string. Timing information calculated by analogy.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a timing for subtitles applied to a subtitle program production system for producing a subtitle program assuming a form in which an almost common digitized original is used for both announcement and subtitle. Regarding the method of giving information,
The subtitle sentence text, which is the basis of the subtitles to which the timing information regarding the presentation of subtitles has been added to each part such as the beginning of the sentence, is provided for each of the presentation unit subtitles after division at an appropriate position according to a predetermined presentation format, and the division position corresponding to the division position The present invention relates to a method for providing timing information to subtitles, which can automatically provide high-precision timing information.

[0002]

2. Description of the Related Art Today, it is generally referred to as an advanced information society, but hearing impaired persons are in a situation where it is difficult to obtain information as compared with healthy persons.

That is, for example, a TV broadcast program which is widely used as an information medium is exemplified, and when the ratio of a subtitle program to all TV broadcast programs in Japan is referred to, it reaches 33 to 70% in Europe and the United States. Only 10
At present, it is extremely low at about%.

[0004]

Now, all T in Japan
The reason why the ratio of subtitle programs to V broadcast programs is lower than in Europe and the United States is mainly due to the lack of subtitle program production technology. Specifically, there is also a problem unique to Japanese, and most of the subtitle program production processes are manually performed, requiring a great deal of labor, time, and cost.

[0005] Therefore, the present inventors conducted a substantive investigation on the current production of subtitled programs in an attempt to investigate the cause of hindrance to the development of subtitled program production technology.

[0008] The left side of FIG. 8 shows a subtitle program production flow that is currently generally performed. In step S101, the subtitle program creator receives three subtitle manuscript creation materials, that is, program data in which the time code is superimposed on video, a program tape in which the time code is recorded on an audio channel, and a program script. Note that in the drawings, “time code” may be abbreviated as “TC”.

In step S103, the expert, such as a person who has broadcast experience, based on the subtitle manuscript preparation material received in step S101, (1) transcribes the summary of the program announcement, and (2) prepares the separately specified subtitle presentation. Caption presentation image according to the standard manuscript preparation procedure, (3) Start
The work of inputting the end time code is sequentially performed to create a subtitle manuscript.

In step S105, the input operator creates digitized subtitles based on the subtitle manuscript created in step S103.

In step S107, step S1
The digitized subtitles created in step 05 are previewed and corrected in the presence of the caption production manager in charge, the manuscript creator, and the input operator to make the completed subtitles.

[0010] Recently, by training a human resource called a caption operator who has been involved in both the summary transcription of program announcements and the digitization of subtitles, the improved current subtitle production flow shown on the right side of FIG. It has been implemented.

That is, in step S111, the subtitle program maker receives two subtitle manuscript creation materials, a program tape in which a time code is recorded on an audio channel, and a program script.

In step S113, the caption operator reproduces the program tape having the time code recorded on the audio channel, and clicks the mouse button at the start point of the dialog to extract the start time code from the audio channel at that point. Record. Furthermore, while listening to the dialogue and inputting it as summary electronic data, clicking the mouse button again at the dialogue point corresponding to the break point based on the subtitle manuscript creation procedure, extracting the end point time code from the audio channel at that point Record. These operations are repeated until the end of the program, and the subtitles of the entire program are digitized.

In step S117, step S1
The digitized subtitles created in step 05 are previewed and modified under the attendance of the caption production manager in charge and the caption operator to obtain completed subtitles.

In the latter improved current subtitle production flow, the caption operator uses only a program tape in which a time code is recorded on an audio channel to summarize and convert the dialogue into words and to divide it into presentation units. By clicking the mouse button at the timing of the dialog corresponding to the start point / end point of the subtitle, each time code of the audio channel is taken out and recorded, which can be said to be an effective subtitle production flow with considerably reduced labor.

The reason why a particularly large number of man-hours are required in a series of processing flows in the above-described current subtitle production flow is that steps S103 to S105 or step S11.
3. (1) Transcript of summary of program announcement, (2)
These are the work processes of subtitle presentation imaging and (3) entry of the start / end time code, and these work processes largely depend on the knowledge and experience of the skilled person.

However, among the subtitle programs currently being broadcast,
There are some programs in which an announcement manuscript is created in advance, and the manuscript is assumed to be actual broadcast subtitles with little modification. For example, when actually examining an information program with subtitles called "The Life of the Earth", it is assumed that the announcement sound and subtitle content are almost the same, and that a common manuscript is used for both the announcement and subtitles I can do it.

As described above, the announcement sound and the subtitle contents are very similar, and a substantially common manuscript is used for both the announcement and the subtitle, and when a program in which the manuscript is digitized is assumed, (1) ) Will hardly need to be transcribed. In this case, the remaining operations are (2) subtitle presentation imaging and (3) start / end time code entry. The inventors of the present invention have made intensive studies to simplify each of these work processes, and as a result, a new technology that can automate the process of (3) starting and ending time code entry without human intervention. It came to come to mind.

The present invention has been made in view of the above-described circumstances, and is intended to convert a subtitle sentence text serving as a base of a subtitle in which timing information relating to the presentation of a subtitle is added to each part such as a head of a sentence in an appropriate manner according to a predetermined presentation format. It is an object of the present invention to provide a method of providing timing information to subtitles, which can automatically provide high-precision timing information corresponding to the divided portion to each of the presentation unit subtitles divided at the portion.

[0019]

Means for Solving the Problems In order to solve the above-mentioned problems, the invention of claim 1 relates to a method for producing a subtitle program.
At least the subtitle sentence text that is the basis of the subtitles, at each of the presentation unit subtitles divided at an appropriate location according to a predetermined presentation format, timing information to the subtitles used when adding timing information corresponding to the division location A method of providing, wherein reference timing information is provided to each part of the subtitle sentence text before division at an appropriate part according to the predetermined presentation format, and the subtitle sentence text is divided at the appropriate part. By performing the presentation unit subtitles, based on the timing information serving as the reference and the character information including the character type and the number of characters or the pronunciation symbol string presented by each presentation unit subtitle, each presentation unit after division at the appropriate location The timing information to be assigned to at least one of the start point and the end point of the caption is calculated by analogy, and each of the caption texts is presented after being divided at the appropriate place. Position for each of the subtitle, and summarized in that the automatically given timing information described above analogy operation.

According to the first aspect of the present invention, reference timing information is added to each portion of the subtitle text before division at an appropriate portion according to a predetermined presentation format, and the subtitle text is added at the appropriate portion. Performing the presentation unit subtitles by dividing, based on the timing information serving as the reference and the character information including the character type and the number of characters or the pronunciation symbol string presented by each presentation unit subtitle, after division at the appropriate place The timing information to be applied to at least one of the start point and the end point of each presentation unit subtitle is calculated by analogy, and the timing calculated by analogy is calculated for each of the presentation unit subtitles after the subtitle sentence text is divided at the appropriate position. Since the information is automatically added, the subtitle sentence text is divided into appropriate sub-subtitles according to the predetermined presentation format. Timing information attaching method of the response was highly accurate timing information of the automatically given subtitles can be obtained.

According to a second aspect of the present invention, there is provided the method for adding timing information to subtitles according to the first aspect, wherein at least one of a start point and an end point of each presentation unit subtitle after division at the appropriate position. Based on the timing information serving as the reference and the character information including the character type and the number of characters presented by each of the presentation unit subtitles, in calculating the timing information to be given to the other characters, other characters including kanji, Arabic numerals, and English characters By converting the reading time of characters into a predetermined magnification obtained from a statistical survey with respect to the reading time of characters including hiragana or katakana, the start point / end point of each presentation unit subtitle after division at the appropriate part The gist of the present invention is to calculate the timing information to be given to at least one of them by analogy.

According to the second aspect of the present invention, when the timing information to be assigned to at least one of the start point and the end point of each presentation unit subtitle after division at an appropriate location is calculated by analogy, Based on the character information including the character type and the number of characters presented by each of the presentation unit subtitles, the reading time of other characters including kanji, Arabic numerals and English characters is statistically calculated with respect to the reading time of characters including hiragana or katakana. By converting the time to a predetermined magnification obtained from a typical survey, the timing information to be assigned to at least one of the start point and the end point of each of the divided presentation unit subtitles at the appropriate part is calculated by analogy.
As a result that it is not necessary to apply the synchronization detection technique that requires a complicated and constant processing time for all the subtitle characters, it is possible to expect to maintain good immediacy regarding the presentation of subtitles.

According to a third aspect of the present invention, there is provided the method for adding timing information to subtitles according to the second aspect, wherein the predetermined magnification obtained from the statistical investigation is about 1.86. Make a summary.

According to the invention of claim 3, the predetermined magnification obtained from the statistical investigation can be set to, for example, about 1.86.

According to a fourth aspect of the present invention, there is provided the method of adding timing information to a subtitle according to the first aspect, wherein at least one of a start point and an end point of each presentation unit subtitle after division at the appropriate position. In calculating the timing information to be applied to the analogy, the reading time corresponding to the phoneme of each phonetic symbol is calculated based on the reference timing information and the character information including the phonetic symbol string presented by each presentation unit subtitle. By referring to the phoneme time table tabulated using a statistical method, by accumulating each phoneme time of the phonetic symbol string included in each presentation unit subtitle, the presentation unit subtitle of each presentation unit after division at the appropriate place The gist of the present invention is to calculate the timing information to be applied to at least one of the start point and the end point by analogy.

According to the fourth aspect of the present invention, when the timing information to be added to at least one of the starting point and the ending point of each of the divided presentation unit subtitles at an appropriate location is calculated by analogy, , Character information including a phonetic symbol string presented by each presentation unit subtitle,
By referring to the phoneme time table in which the reading times corresponding to the phonemes of each phonetic symbol are tabulated using a statistical method, by accumulating the phoneme times of the phonetic symbol strings included in each presentation unit subtitle, The timing information to be assigned to at least one of the start point and the end point of each presentation unit subtitle after division at the appropriate part is calculated by analogy. Therefore, similar to the invention of claim 2, all subtitle characters are targeted. As a result of not requiring the application of the synchronization detection technique that requires a complicated and constant processing time, it is possible to expect to maintain good immediacy regarding the presentation of subtitles.

According to a fifth aspect of the present invention, there is provided the method for adding timing information to subtitles according to any one of the second to fourth aspects, wherein the timing information is inferred by using a time ratio technique. The gist is to be calculated.

According to the fifth aspect of the present invention, the timing information is calculated by analogy using a time ratio method.
Therefore, it is possible to implement a relatively high-precision analogy calculation of timing information by a simple method.

[0029]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a method for providing timing information to subtitles according to the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional block diagram of an automatic subtitle program production system which embodies the method of adding timing information to subtitles according to the present invention. FIG. 2 shows the average number of readings for an actual TV news sentence. FIG. 3 is a diagram showing a result of a survey, FIG. 3 is a diagram showing a trial calculation result of a time error in a timing information adding method focusing on a character type, FIG. 4 is a diagram showing a divided subtitle sentence for explanation of the present invention, and FIG. FIG. 6 and FIG. 7 are diagrams illustrating an example of a phoneme time table used in the timing information adding method focusing on a symbol string, and FIG. 6 and FIG.

As a predetermined presentation format adopted in the embodiment of the present invention, a presentation format in which the limited number of characters N per line is 15 characters and a presentation unit subtitle consisting of two lines is totally and totally replaced is exemplified. The following description will proceed.

As described above, among subtitle programs currently being broadcast, an announcement manuscript is created in advance, and there are a number of programs that are presumed to have actual subtitles without any substantial modification of the manuscript. There is. For example, when actually examining an information program with subtitles called "Travel of the Earth", the announcement sound and subtitle content are almost the same, and it is estimated that almost the same manuscript is used for both the announcement and subtitle I can do it.

Therefore, the present inventors use the same manuscript for both the announcement and subtitles, because the announcement sound and the contents of the subtitles are very similar, and the manuscript is digitized. Assuming a program, at least the timing information on the presentation of subtitles at the beginning of the sentence is the subtitle sentence text that is the basis of the subtitles given to various places, for each of the presentation unit subtitles after division at an appropriate place according to the predetermined presentation format Thus, a method of adding timing information to subtitles, which can automatically add high-precision timing information corresponding to the division, has been conceived.

Here, the background that led to the present invention will be described. When considering the problem of subtitle sentence text division from the viewpoint of subtitles that are easier to read and understand, it is obvious that subtitles are easier to read and understand. The question is what kind of thing is. We have not yet found a quantitatively clear answer to this question, but through valuable experience in producing experimental subtitle programs and subtitle evaluation experiments, qualitative but important factors are becoming clearer.

From the viewpoint of readability and comprehension of subtitles, generally, a certain number of characters or more is presented simultaneously.
It is said that it is good that the presentation lasts for the required time, but the number of characters and the presentation duration greatly affect how the presented subtitles are read.

For example, assuming that a hearing-impaired person watches a TV program with subtitles, the video information and the audio information are alternately viewed through the visual sense, so that the subtitles can be viewed only intermittently. Absent. For this reason, it is desirable to present the audio information as subtitles that are easier to read and understand, so that the ratio of viewing subtitles is reduced as much as possible, so that more video can be viewed accordingly.

The way of viewing subtitles in this case also depends on the presentation format of the subtitles. For example, when a presentation format in which two-line presentation unit subtitles are exchanged at a time is illustrated, and an attempt is made to capture all the subtitles presented, In general, pre-reading, post-reading, or both are performed centering on a reference subtitle character (for example, a character corresponding to the progress of a voice announcement).

The reason for performing pre-reading, post-reading, or both is that there is an overlooking operation time in which the user is looking away from the subtitles including watching the video or blinking or looking aside. The length of the missed operation time is empirically considered to be about 0.5 to 2 seconds.

Here, assuming that the subtitle presentation speed is 200 characters / minute, the maximum time of 2 seconds is equivalent to about 7 characters. It can be seen that there is a risk of overlooking.

From the above, it is necessary that 14 characters continuous from the reference subtitle character be the minimum presentation unit, and the gazing point returns to the subtitle again, and the subtitle is read and recognized. Assuming 7 characters, 24
It can be seen that it is desirable to simultaneously present subtitles of about 29 characters on the screen. By the way, in the current subtitle broadcasting, one line is 15
Many characters are presented in two lines, and up to about 30 characters are presented.

Further, assuming that it takes at least 2 seconds from the presentation of the subtitles to the actual reading according to the above analysis results, if the subtitles of 7 characters or less are presented only for the time corresponding to the number of characters, May not be able to read this subtitle at all. For example, due to the characteristics of Japanese, negative words are placed at the end of sentences in negative sentences, so divisions in which this negative word part corresponds to the above state may have a very bad effect, and such divisions are possible. It must be avoided as much as possible.

As a countermeasure, it is desirable to apply a technique of not dividing the number of characters into a small number or increasing the presentation time with a small number of characters.

The next problem is, for example, the handling of a silent section between sentences, that is, a pause. If there is a long pause in the caption text, it is highly likely that before and after the pause are caption texts related to mutually different contents, so it is not preferable to present a caption spanning the pose. Conversely, when there is a very short pause, it is more preferable to treat it as a continuous subtitle sentence before and after this pause, since there is a high possibility that the sentence is a subtitle sentence related to mutually common contents. For this reason, it is desirable to apply a caption sentence division method that takes into account the length of the pause time.

Further, it is desirable that a group of characters be presented on the same line without being divided as much as possible. Examples of this include not only ordinary words, but also continuous kanji, katakana, Arabic numerals, and alphabetic characters, such as (xxx) or "xxx"
Ruby, abbreviations for formal names, annotations, etc. are also included in this category.

As described above, when the caption text is divided for the purpose of obtaining a caption that is easier to read and understand, it is necessary to sufficiently consider the above-described elements. However, with the division of the caption sentence text, there arises a new problem that it is necessary to add timing information corresponding to the divided portion to each of the divided presentation unit subtitles at an appropriate portion.

Therefore, the present invention employs a synchronous detection technology for announcement sound and subtitle sentence text proposed in the present invention, and a statistical feature analysis method for reading and pronunciation of Japanese, and the like, to provide a predetermined presentation format. , Automatic presentation of high-precision timing information corresponding to the divided portion is realized for each of the presentation unit subtitles divided at an appropriate portion according to the above.

Before the description of this embodiment, terms used in the following description are defined. In the description of this embodiment, the entire set of subtitle sentences to be presented is referred to as “subtitle text”. In the caption text, a subset of the caption text separated by appropriate punctuation is referred to as “unit subtitle text”, and the subtitle that is the presentation unit on the display screen of the display is referred to as “presentation subtitle”. say,
When expressing individual subtitles of each line included in the presentation unit subtitles, this is called "presentation unit subtitle line", and when expressing any character in the presentation unit subtitle line, this is called "subtitle character" I will. When presenting a single-line presentation unit subtitle on the display screen, the “presentation unit subtitle line” and “presentation unit subtitle line” are synonymous. In this case, the expression “presentation unit subtitle line” is used I will not do it.

First, an automatic subtitle program production system 11 that embodies the method for adding timing information to subtitles according to the present invention.
Will be described with reference to FIG.

As shown in the figure, the automatic subtitle program production system 11 includes an electronic document recording medium 13, a synchronization detection device 15, an integration device 17, a morphological analysis unit 19, a division rule storage unit 21, , A program material VTR, for example, a digital video tape recorder (hereinafter, referred to as “D-VTR”) 23.

The digitized original recording medium 13 is composed of, for example, a hard disk storage device, a floppy disk device, or the like, and stores subtitle sentence text representing the entire set of subtitles to be presented. In this embodiment, since it is assumed that a substantially common digitized manuscript is used for both the announcement and the subtitle, the contents of the subtitle sentence text stored in the digitized manuscript recording medium 13 are presented. It is assumed that not only does it match the target subtitle, but also matches the announcement sound recorded on the material VTR.

The synchronization detecting device 15 has a function of detecting the time synchronization between the caption sentence with the synchronization detection point and the announcement sound read out. More specifically, when the synchronization detection device 15 receives the subtitle sentence with the synchronization detection point provided by the integration device 17, the synchronization detection device 15 relates to the announcement sound corresponding to the subtitle sentence fetched from the program material VTR. And its timecode,
It has a function of detecting timing information of a designated synchronization detection point, that is, a time code, detecting a pause point included in the announcement sound, and transmitting the detected time code and pause point to the integrating device 17. ing.

The synchronous detection of the time code as the above-mentioned timing information is performed by applying the synchronous detection technology between the announcement sound and the caption text including the speech recognition processing for the announcement sound which has been researched and developed by the present inventors. By doing so, it can be realized with high accuracy.

That is, as shown in FIG. 6, the flow of the detection of the subtitle transmission timing is as follows. First, the subtitle sentence text described in the kana-kanji mixed sentence is pronounced using a reading technique used in speech synthesis or the like. Convert to a symbol string. The "Japanese reading system" is used for this conversion. next,
Refer to the acoustic model (HMM: Hidden Markov Model) that has been learned in advance,
To convert these phonetic symbol strings into a speech model (HMM) called a word string pair model. Then, by performing the comparison and collation through the announcement sound to the word string pair model using the “maximum likelihood collation system”, the synchronization detection of the caption transmission timing is performed.

The algorithm (word string pair model) used for detecting the caption sending timing employs a keyword spotting technique. As a keyword spotting technique, a method has been proposed in which a posterior probability of a word is obtained by a forward / backward algorithm, and a local peak of the word likelihood is detected. As shown in FIG. 7, the word string pair model is applied to the point where it is desired to synchronize subtitles and audio, that is, word string 1 (Keywords1) and word string 2 (Keywords2) are connected before and after the synchronization point. The model is designed to observe the likelihood at the middle point (B) of the word string, detect its local peak, and obtain the utterance start time of the word string 2 with high accuracy.
The word sequence is formed by connecting phoneme HMMs, and the garbage (Garbage) portion is formed as parallel branches of all phoneme HMMs. When the announcer reads the manuscript, a pause is inserted between the word strings 1 and 2 because the position of the breath is arbitrarily determined so that the contents can be easily understood. The detection of the pause time can be easily achieved by a well-known technique from the start and end time codes in which a sound and its time code are supplied from the material VTR and the sound level is continuous below a specified level.

The integrating device 17 stores the digitized original recording medium 1
3. A unit subtitle sentence extraction function for sequentially extracting unit subtitle sentences based on the required number of characters starting from the beginning of the subtitle sentence text read from unit 3, and a unit subtitle extracted by using the unit subtitle sentence extraction function A presentation unit subtitle conversion function of converting a sentence into a presentation unit subtitle according to a desired presentation format, and a time transmitted from the synchronization detection device 15 for the presentation unit subtitle converted by exhibiting the presentation unit subtitle function. A timing information adding function of adding timing information using a code and a pause point.

The morphological analysis unit 19 divides the unit caption sentence described in the kanji kana mixed sentence into morphemes, and expresses each of the morphemes divided by performing the splitting function. It has an additional information addition function to add additional information such as shape, part of speech, reading, and standard expression, and an information element sequence acquisition function to group each morpheme into clauses and clauses and obtain some information element strings. ing. As a result, the unit caption sentence is composed of a surface sequence, a symbol sequence (part of speech), a standard sequence,
And an information element sequence.

The division rule storage unit 21 has a function of storing a division rule that is referred to when optimizing a line feed / page break position for a unit subtitle sentence.

The D-VTR 23 has a function of reproducing and outputting video, audio, and their time codes from a program material VTR tape on which the program material is recorded.

Next, the internal configuration of the integrating device 17 which plays a major role in the automatic subtitle program production system 11 will be described.

The integrating device 17 includes a unit subtitle sentence extracting unit 33
, Presentation unit subtitle conversion unit 35, timing information addition unit 3
7 are included.

The unit subtitle sentence extraction unit 33 reads, for example, 70 to 90 units of subtitle sentence text read from the digitized original recording medium 13 and arranged in the order of presentation time.
It has a function of sequentially extracting text sentences to be processed in units by using, for example, information on a delimitable portion, using the caption character as a guide. Note that, as the delimitable portion information, morphological analysis data with phrase data obtained by the morphological analysis unit 19 and division rules (line feed / page break data) stored in the division rule storage unit 21 can also be used. . Here, regarding the above-described division rule (line break / page break data), the recommended line break / page break point defined by the division rule (line break / page break data) is first after a punctuation mark and secondly by a reading point. After the third, between clauses, and fourth, between morpheme parts of speech. When applying the division rule (line feed / page break data), priority is given from the top of the above-described description order. It is preferably applied.

The presentation unit captioning unit 35 generates a unit caption based on the unit caption sentence extracted by the unit caption sentence extraction unit 33, the delimitable location information added to the unit caption sentence, information from the synchronization detection device 15, and the like. It has a presentation unit subtitle conversion function of converting the unit subtitle sentences extracted by the sentence extraction unit 33 into at least one or more presentation unit subtitles according to a desired presentation format.

The timing information adding unit 37 uses the time code and the pause point transmitted from the synchronization detection device 15 for the presentation unit subtitle converted by the presentation unit subtitle conversion unit 35, and uses a timing interpolation method described later. And has a timing information adding function of adding timing information by using.

Next, a method of adding timing information to subtitles according to the present invention will be described with reference to FIGS.

As described above, the synchronous detection of the timing information related to the caption corresponding to the announcement voice is performed between the announcement voice including the voice recognition processing for the announcement voice researched and developed by the present inventors and the caption text. Although it can be realized with high accuracy by applying the synchronization detection technology, this synchronization detection process is quite complicated and requires a certain processing time. Therefore, it is necessary to target all start / end time codes of each presentation unit subtitle. In applying the synchronization detection technology as
There is a possibility that the number of synchronization detection points becomes excessive and the immediacy regarding the presentation of subtitles is impaired.

Here, a method of adding timing information to subtitles is analyzed by paying attention to the timing of adding timing information to subtitles.
It can be broadly divided into a timing information adding mode based on subtitles before division.

In the timing information adding mode based on the divided subtitles, since the subtitles to be added are determined, synchronization detection is performed by comparing the announcement sound and the subtitle sentence text at the start point / end point, and the start point is determined. / It is only necessary to add timing information for each end point.

In this mode, timing information is directly assigned to subtitles, which is the most reliable and has high synchronization accuracy. On the other hand, when producing subtitles in accordance with various presentation formats based on the same subtitle sentence text, However, there is a problem that the immediacy relating to the presentation of subtitles may be impaired as a result of the need to perform synchronous detection requiring a complicated and constant processing time for each presentation format.

On the other hand, the timing information adding mode based on the subtitles before division is suitable for producing subtitles according to various presentation formats from the same subtitle text. In this case, first, for the subtitle text before splitting,
For example, at appropriate intervals at the beginning of a sentence or the like, reference timing information is added by applying a synchronization detection technique, and then the subtitle sentence text is divided into appropriate portions according to a predetermined presentation format. By performing the presentation unit subtitles, based on the timing information as a reference, character information including the character type and the number of characters presented by the presentation unit subtitles, or phonetic symbol string, by applying the interpolation method described later The timing information calculated by analogy is used as the starting point of each presentation unit subtitle /
Since the procedure of assigning to at least one of the end points is performed, it is not necessary to apply a synchronous detection technique that requires a complicated and constant processing time for all the start points / end points of each presentation unit subtitle. It can be expected that the promptness of the presentation of the information is well maintained.

Here, the timing information interpolation providing method based on the subtitles before division can be further roughly classified into a timing information providing method focusing on a character type and a timing information providing method focusing on a phonetic symbol string. it can. In the following description, a timing information providing method focusing on a character type is referred to as a first timing information providing method, while a timing information providing method focusing on a phonetic symbol sequence is sometimes referred to as a second timing information providing method. I will add that.

In the first timing information adding method, the character type and the number of characters are used as the character information presented by the presentation unit subtitle, and the timing information is calculated by analogy when reading other characters including kanji, Arabic numerals, and alphabetic characters. The time is calculated based on the reading time of characters including hiragana or katakana, as shown in FIG. 2, for example, as shown in FIG. 2, obtained from a statistical survey on the number of pronunciations of these character types included in actual TV news sentences. Time conversion to a predetermined magnification such as .86 times, and is added to subtitles based on the integrated value of the reading time represented by Hiragana or Katakana, the converted reading time represented by other characters, and reference timing information. The timing information is calculated by analogy. Then, the result of the analogy calculation is added as timing information to the presentation unit subtitle after the division.

Regarding the first timing information adding method,
4 will be described in more detail by exemplifying the news sentence shown in FIG. 4. Reference timing information is added in advance to the beginning “t” of the divided subtitle sentence 1 and the end of the sentence “a” of the divided subtitle sentence 2, Assuming that the respective pieces of timing information are TB and TE, the timing information TM at the beginning of the sentence “i” of the divided subtitle sentence 2 is estimated by the following procedure.

First, the number of characters of hiragana or katakana included in the divided subtitle sentence 1 is 12, the number of characters of other kanji etc. is 7, the number of characters of hiragana or katakana included in the divided subtitle sentence 2 is 11, and other characters are included. Number of characters such as kanji is 3
It is. Assuming that the reading time of Hiragana or Katakana is "1", the total reading time TR1, T of the divided subtitle sentences 1, 2
R2 is obtained by the following equations (1) and (2).

TR1 = (12 * 1) + (7 * 1.86) = 25 (Expression 1) TR2 = (11 * 1) + (3 * 1.86) = 16.6 (Expression 2) The total reading time T of the divided subtitle sentences 1 and 2 as the calculation result
Utilizing R1 and TR2, the timing information TM of the head “i” of the divided subtitle sentence 2 which is a division point between the divided subtitle sentences 1 and 2
Is calculated by the following equation 3 using the time ratio method.

TM = TB + (TE−TB) * TR1 / (TR1 + TR2) = TB + (TE−TB) * 0.6 (Equation 3) In this way, the timing information of the head “i” of the divided subtitle sentence 2 The TM can be calculated by analogy, and the analogy calculation result TM is added to the beginning “i” of the divided subtitle sentence 2 as timing information. Note that the timing information TM at the beginning of the subtitle sentence 2 of the divided subtitle sentence 2 can be handled as the timing information at the end of the sentence “e” of the divided subtitle sentence 1.

Here, in the first timing information adding method using the average number of readings of a predetermined character type obtained by the statistical method, the subtitle sentence is determined regardless of the number of other characters including Chinese characters, Arabic numerals, and English characters. For example, as a result of applying the same magnification such as about 1.86 times, a time error may inevitably occur. Therefore, the effect of the time error in the first timing information providing method will be considered.

First, it is premised that the subtitle sentence text that is the basis of the subtitles is provided with accurate timing information detected by using the synchronization detection technique at intervals of 30 characters. Line presentation unit subtitles, accurate timing information shall be given to each of the starting point of the second line of the previous page and the ending point of the first line of the current page of interest. Assume the value is 1.862. Then, on the premise described above, a trial calculation was made of the time error given by the timing information corresponding to the start point on the first line of the current page of interest when the average number of readings is different from the standard value. FIG. 3 shows the calculation result of the time error. In FIG. 3, the second line of the previous page is all kanji, the first line of the current page is all hiragana, and the case where the subtitle speed is 5, 6, 7 characters / second is shown.

As shown in the figure, the maximum time error in this trial calculation is 0.162 seconds (delay), but the present inventors are studying separately and evaluating the allowable range of the time error in the subtitle presentation timing. From the experimental results, a time error of about ± 1.0 second can be considered to be within an allowable range. Therefore, the first timing information addition using the average number of readings of the character type obtained by the above-described statistical method is performed. The method can be said to be a simple but fairly practical approach.

Next, in the second timing information adding method focusing on the phonetic symbol sequence, the phoneme of each phonetic symbol is used as the character information presented by the presentation unit subtitle using the phonetic symbol sequence included in each presentation unit subtitle. The timing information to be added to the subtitles is calculated by analogy with reference to a phoneme time table as shown in FIG. 5, for example, in which the reading time corresponding to each is tabulated using a statistical method, and the analogy calculation result is used as timing information. Is added to the presentation unit subtitle after division.

The second timing information adding method is as follows.
4 will be described in more detail by exemplifying the news sentence shown in FIG. 4. Reference timing information is added in advance to the beginning “t” of the divided subtitle sentence 1 and the end of the sentence “a” of the divided subtitle sentence 2, Assuming that the respective pieces of timing information are TB and TE, the timing information TM at the beginning of the sentence “i” of the divided subtitle sentence 2 is estimated by the following procedure. In addition, the Japanese reading result in FIG. 4 is a phonetic symbol string delimited by “,”, and “t”, “a”, “i”... .
For this phoneme, the phoneme time table shown in FIG. 5 obtained from the analysis of the speech database is prepared in advance, so that the phoneme sequence as the Japanese reading result and the phoneme time as the reading time corresponding to the phoneme are prepared. And based on the subtitle sentence 2
The timing information TM at the beginning of the sentence "i" can be calculated by analogy using the interpolation method described below.

That is, the total reading time T obtained from the phoneme strings of readings 1 and 2 corresponding to the divided subtitle sentences 1 and 2, respectively.
R3 and TR4 are obtained by the following equations (4) and (5).

TR3 = Tt + Ta + Ti +... + Ti + Tt + Te (Equation 4) TR4 = Ti + Tk + Te +. Time 5.62731
6, and “Ta” is a reading time 7.130941 corresponding to the phoneme “a” in the phoneme time table;
Similarly, the reading time corresponding to each phoneme can be extracted from the phoneme time table.

By utilizing the total reading time TR3, TR4 obtained from the phoneme sequence of reading 1, 2 corresponding to each of the divided subtitle sentences 1, 2 as the integration result, the divided subtitle sentences 1, 2 are used. The timing information TM of the beginning of the subtitle sentence 2 of the subtitle sentence 2 which is the division point is calculated by analogy with the following equation 6 using the method of the time ratio.

TM = TB + (TE−TB) * TR1 / (TR1 + TR2) (Equation 6) In this manner, the timing information TM of the beginning “i” of the divided subtitle sentence 2 can be calculated by analogy. The calculation result TM is interpolated to the head “i” of the divided subtitle sentence 2 as timing information. Note that the timing information TM at the beginning of the subtitle sentence 2 of the divided subtitle sentence 2 can be handled as the timing information at the end of the sentence “e” of the divided subtitle sentence 1.

Here, when a time error of the timing information provided by the second timing information providing method was calculated by a simple experiment, it was confirmed that the time error converged to about 0.4 seconds. The information provision method is
In view of the result of the evaluation experiment that the time error of about ± 1.0 second is within an allowable range, this is a fairly practical and effective method, similar to the above-described first timing information adding method focusing on the character type. Can be said.

As described above, according to the method of adding timing information to subtitles according to the present invention, the synchronous detection technology of announcement sound and subtitle sentence text proposed in the present invention, and statistical features related to reading and pronunciation of Japanese. By applying an analysis method or the like, it is possible to automatically give high-precision timing information corresponding to the divided portion to each of the divided presentation unit subtitles at an appropriate portion according to a predetermined presentation format.

It is needless to say that the present invention is not limited to the above-described embodiment, but can be embodied in other forms by appropriately changing it within the scope of the claims.

[0088]

As described above in detail, according to the first aspect of the present invention, each of the presentation unit subtitles obtained by dividing the subtitle sentence text at an appropriate location according to a predetermined presentation format corresponds to the division location. It is possible to obtain a method of adding timing information to subtitles, which can automatically add the highly accurate timing information described above.

According to the second aspect of the present invention, it is not necessary to apply a synchronous detection technique which requires a complicated and constant processing time for all subtitle characters, so that good immediacy regarding presentation of subtitles is maintained. Can be expected.

On the other hand, according to the invention of claim 4, according to claim 2,
As in the invention of the first aspect, as a result of not requiring the application of the synchronization detection technique which requires a complicated and constant processing time for all subtitle characters, it is possible to expect to maintain good immediacy regarding the presentation of subtitles.

According to the fifth aspect of the invention, there is an extremely excellent effect that a relatively high-precision analogy calculation of timing information can be realized by a simple method.

[Brief description of the drawings]

FIG. 1 is a functional block configuration diagram of an automatic subtitle program production system that embodies a method of adding timing information to subtitles according to the present invention.

FIG. 2 is a diagram showing a survey result of an average number of readings for an actual TV news sentence.

FIG. 3 is a diagram illustrating a trial calculation result of a time error in a timing information adding method focusing on a character type;

FIG. 4 is a diagram showing a divided subtitle sentence for explanation of the present invention.

FIG. 5 is a diagram illustrating an example of a phoneme time table used in a timing information adding method focusing on a phonetic symbol string.

FIG. 6 is a diagram provided for describing a technique for detecting a synchronization of subtitle transmission timing for an announcement sound;

FIG. 7 is a diagram provided for describing a technique for detecting a synchronization of subtitle transmission timing with respect to an announcement sound;

FIG. 8 is an explanatory diagram relating to a current subtitle production flow and an improved current subtitle production flow.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 11 Automatic closed-caption program production system 13 Electronic original recording medium 15 Synchronization detection device 17 Integrated device 19 Morphological analysis unit 21 Division rule storage unit 23 Digital video tape recorder (D-V
TR) 33 unit subtitle sentence extraction unit 35 presentation unit subtitle conversion unit 37 timing information addition unit

 ──────────────────────────────────────────────────続 き Continued on the front page (71) Applicant 000006013 Mitsubishi Electric Corporation 2-3-2 Marunouchi, Chiyoda-ku, Tokyo (71) Applicant 000004352 Japan Broadcasting Corporation 2-2-1 Jinnan, Shibuya-ku, Tokyo (72 Inventor Eiji Sawamura 2-31-19 Shiba, Minato-ku, Tokyo Communication and Broadcasting Corporation (72) Inventor Takao Kadoma 2-31-19 Shiba in Minato-ku Tokyo, Japan Communication and Broadcasting Corporation (72) Inventor Takahiro Fukushima Tokyo 2-31-19 Shiba, Minato-ku, Tokyo Communication and Broadcasting Organization (72) Inventor Ichiro Maruyama 2-31-19 Shiba, Minato-ku, Tokyo Inside (72) Inventor Terumasa Ehara 2 Shiba, Minato-ku, Tokyo −31−19 Communications and Broadcasting Corporation (72) Inventor Katsuhiko Shirai 2-31-19, Shiba, Minato-ku, Tokyo Communications and Broadcasting Corporation F-term (reference) 5C023 AA18 AA38 BA11 BA16 CA01 CA05 5C025 CA09 CB10 DA10

Claims (5)

[Claims] When producing a subtitle program, at least a subtitle sentence text as a basis of the subtitle is divided into an appropriate portion according to a predetermined presentation format, and each presentation unit subtitle is divided into timing information corresponding to the division portion. Is a method of providing timing information to subtitles used when adding, for each part of the subtitle sentence text before division at an appropriate place according to the predetermined presentation format, the reference timing information is added, Performing the presentation unit subtitle by dividing the subtitle sentence text at the appropriate part, based on the timing information serving as the reference, and character information including the character type and the number of characters or the pronunciation symbol string presented by each presentation unit subtitle, Calculating analogous timing information to be given to at least one of the start point and the end point of each of the divided presentation subtitles at the appropriate part; Makubun for each of the presentation units subtitle after division the text in the right location, timing information attaching method of the subtitle, characterized in that the automatically given timing information described above analogy operation. 2. The method according to claim 1, wherein timing information to be added to at least one of a starting point and an ending point of each of the divided presentation unit subtitles at the appropriate position is analogized. In calculating, based on the timing information serving as the reference and the character information including the character type and the number of characters presented by each presentation unit subtitle, the reading time of other characters including Chinese characters, Arabic numerals, and English characters, By converting the reading time of characters including katakana into a predetermined magnification obtained from a statistical survey, at least one of the starting point and the ending point of each presentation unit subtitle after division at the appropriate location is provided. A method for adding timing information to subtitles, comprising performing an analogy calculation of timing information. 3. The method according to claim 2, wherein the predetermined magnification obtained from the statistical investigation is about 1.86.
A method for adding timing information to subtitles, characterized in that the timing information is doubled. 4. The method of adding timing information to subtitles according to claim 1, wherein the timing information to be assigned to at least one of a start point and an end point of each of the divided presentation unit subtitles at the appropriate place is analogized. In calculating, based on the timing information serving as the reference and character information including a phonetic symbol string presented by each presentation unit subtitle, a reading time corresponding to each phoneme of each phonetic symbol is calculated using a statistical method. By summing the phoneme times of the phonetic symbol strings included in each presentation unit subtitle while referring to the converted phoneme time table, at least one of the start point / end point of each presentation unit subtitle after division at the appropriate location A method for adding timing information to subtitles, wherein a timing information to be added to one of the subtitles is calculated by analogy. 5. The method according to claim 2, wherein the timing information is calculated by analogy using a time ratio method. How to add timing information to subtitles.