JPH04199262A - Picture displaying method for multimedia document processing system - Google Patents

Abstract

PURPOSE:To enable a user to easily understand the content of a multimedia document by automatically displaying the picture information related to the text information of the multimedia document on a display when the text information is read out in a synthesized voice. CONSTITUTION:The process regarding the link and display of text information and a related medium in a multimedia document is performed by means of a document edition processing section 103, displaying information analyzing section 104, displaying information selecting section 109, and link information table 102. Then, when the text information of the multimedia document is read out in a synthesized voice, figures, tables, and pictures are successively displayed on a screen in accordance with the content of the document. Therefore, a user can listen to the voice reading out the text information while the user watches the figures, tables, and pictures automatically displayed on the screen and can easily understand the content of the multimedia document.

Description

[Detailed Description of the Invention] [Field of Industrial Application] The present invention relates to a multimedia document processing system that handles multimedia documents, and in particular, when text information of a multimedia document is synthesized and read out, the text The present invention relates to an image display method for a multimedia document processing system that automatically displays image information related to images on a display.

[Conventional technology]

Examples of text-to-speech synthesis systems include MITalk developed at MIT in the United States and the Japanese text-to-speech synthesis system developed by NTT Basic Research Laboratories.
The basic technology is introduced in ``Speech Information Engineering'' (co-authored by Kochi, Kei, et al.) published by TT Technology Transfer Co., Ltd.

Furthermore, as a commercially available system, MICRON
There is an electronic voice output device for the PC-9800, ``Onjibu'' by IC3, which has a function to read out text information created with a Japanese word processor using text-to-speech synthesis processing.

Further, hypertext technology is known as a method for linking various multimedia data to text information. For example, famous products include M, P
There is rGuideJ by Technology Inc., which allows you to link a word in a text with other sentences, pictures, videos, etc., and when you click on the part where this word is displayed with the mouse, you can see the linked document. It has a function that allows the displayed content to be replaced or to display figures, videos, etc. at the same time (references include Nikkei Baito April 1990 issue, P. 220~
P, 230).

[Problem to be solved by the invention]

When a document created using a word processor is a multimedia document that includes multimedia data such as figures, tables, and images, the content of the text information is generally described in relation to the figures, tables, and images in the document. . Regarding such documents, it is difficult to understand the content of the document based only on the content of text information. When text information in a multimedia document is read out by a text-to-speech synthesis system, the user needs to refer to related diagrams and image information as appropriate depending on the content being read out while listening.

Hypertext is a system that links information related to a certain part of a document and allows users to refer to related information as appropriate. However, in conventional hypertext, when a user wants to refer to related information while following a document with his or her eyes, the user clicks on a relevant location in the text with a mouse to display the related information. On the other hand, the process of reading aloud involves the passage of time, and it is necessary for the user to read information related to the content that is being read aloud at the moment.

An object of the present invention is to provide a method for reading out text information in a multimedia document by a text-to-speech synthesis system.
It is an object of the present invention to provide an image display method for automatically displaying related information such as images in FIG. 2 and Table 2 that the user should refer to on a display according to the content being read out.

[Means to solve the problem]

In order to achieve the above object, the present invention relates to a range of textual information in a multimedia document.
Tables or image information (hereinafter referred to as "related media") are linked, and the text information is stored as link information to display related media before and after the corresponding range, and the text information is synthesized into speech. This makes it possible to display related media when link information of related media is detected in the process of processing from the beginning.

[Effect]

A diagram associated with a range of textual information.

Table 2 When image information is linked, link information is embedded before and after the specified range of text information. This link information is detected in the process in which the text information is sequentially processed by the speech synthesis processing system from the beginning, and is sent to the image information display control section in Figure 9. Figure 2 Table 9 The system determines whether the text information in the range linked to the image information is currently being processed and read aloud by the speech synthesis processing system, and while that range is being processed and read out by the text-to-speech synthesis processing system. , linked Figure 2 Table.

Image information will be displayed on the display. In this way, the timing for displaying the related image information in Figure 9, Table 2, according to the content being read out can be determined, and the user can adjust the currently read out audio to the displayed image in Figure 2, Table 2. You can listen while listening.

〔Example〕

Hereinafter, one embodiment of the present invention will be described in detail using the drawings.

FIG. 1 shows an overall configuration diagram of a system having the functions of multimedia document editing processing, linking and display processing of text contents and related media, and text-to-speech synthesis processing.

The multimedia document editing process is carried out by the document editing processing section 10.
It is done in 3. The user uses the mouse 110. keyboard 1
11 and the display 112 to input editing commands to the document editing processing section 103 to perform creation and editing of the multimedia document. create.

The edited multimedia document is stored in the multimedia document database 101. Edit again.

When processing, use the multimedia document database 10
1, a document to be edited is extracted and processed by the document editing processing unit 103.

Processing related to linking and displaying text content and related media is performed by the document editing processing unit 1039 and the display information analysis unit 104.
9 display information control unit 109 and link information table 10
2. The user specifies text information and related media and issues a link instruction to the document processing editing unit 103. The document processing editing department 103 that received the link instruction,
Control information for displaying related media during speech synthesis processing is recorded in the text information of the document currently being edited by the document editing processing unit 103. Further, link information indicating which related media has been linked is recorded in the link information table 102. The display information analysis section 104 sends text information to the text analysis section 105 while checking the control information recorded in the text by the processing of the document processing and editing section 103 during speech synthesis processing. When display information is detected,
Sending the text information to the text analysis section 105 is stopped, the speech synthesis process is temporarily interrupted, and control information is sent to the display information control section 109.

The display information control unit 109 refers to the link information table 102 based on the received control information, determines the related media linked to the corresponding part of the text, and selects the corresponding related media from the document in the document editing processing unit 103. is extracted and displayed on the display 112.

Although detailed link information is stored in the link information table here, information for determining the media may also be stored in the text of the sentence.

Further, the link information table may also include address information indicating the range of text, and the address of the text being outputted as audio may be checked and compared with the address information in the table.

Text-to-speech synthesis processing for converting text information into audio output is performed by the text analysis unit 105. Voice control information generation unit 10
6. The following operations are performed by the speech synthesis control parameter file 1.07 and the speech synthesis section 108. The text analysis unit 105 performs syntactic and semantic analysis of text information, and analyzes readings and word/clause boundaries.

Generates information such as grammar and basic accents. The speech control information generation section 106 receives the information generated by the text analysis section 105, generates speech synthesis control parameters regarding phoneme information and @rhythm information, and sends them to the speech synthesis section 108. The speech synthesis unit 108 converts the received speech synthesis control parameters into an audio signal and outputs it from the speaker 113. Parameter information that has to wait for processing by the speech synthesis unit 108 due to speaking speed is once stored in the speech synthesis control parameter file 107, and the speech synthesis unit 108 is then able to process the next parameter information. After that, it is sent to the speech synthesis section 108. Furthermore, the status of accumulation of parameters waiting to be processed in the speech synthesis control parameter file 107 can be monitored by the display information analysis unit 104 in order to perform speech synthesis accompanied by display processing of related media.

Next, regarding the processing related to linking and displaying text content and related media, we will focus on the operations performed in the document editing processing unit 103, display information analysis unit 1042, display information control lit 109, and link information table 102. The mechanism of automatically displaying related media in accordance with link and text-to-speech synthesis processing will be explained in detail.

FIG. 2 is an example of a screen displaying a multimedia document on the display 112. Reference numeral 201 indicates a command menu, and a multimedia document consisting of data such as Jl 203°, text 2041, and image 205 is displayed in a window 202. Each area has an attribute regarding the area number and the media type of figure/table/text/image. FIG. 4 shows an example of attribute information held by the area in FIG. 203.

At this time, an example of the user's operation when linking a certain range of the text 204 and the diagram 203 is as follows.

First, the user specifies in the text 204 the range related to FIG. 203.

In FIG. 2, the shaded area 206 is the designated range.

Next, the command menu: 1. When selecting "1 display attribute" in 201, a message indicating that the related media to be linked is selected is displayed, and the user follows this message and clicks with the mouse in Figure 203'l. Then, a display target tropism setting window 30 as shown in FIG.
1 is displayed. Since the figure 203 is specified as the display target, the field 303 for specifying the display target type contains "1 figure".
is automatically selected. .

The user sets a name in the column 302 as shown in FIG. 20:3. 1@:302. After completing the 30:3 setting,
Select “Complete” from the command menu 201. Above is the diagonal line fat! of figure 203 and text 204! 206 linking is completed.

The operation of the system during such a link setting operation is as follows.

The window 202 displaying the document in FIG.
The document editing processing section 103 shown in FIG. 1 is in control. When a target range is specified in the text 204 and "display attribute" is selected from the command menu 201, the document editing processing unit 103 displays on the screen a message indicating that the link target has been specified. Furthermore, the user
When you start and specify, the document editing processing unit 103 executes the following:
(A window 301 as shown in the figure is displayed on the display. At this time, the media type of the area is determined by referring to the attribute information of the startled area as shown in Figure 4.)
1, and 1-] before and after specifying the range of text information.
Control information 501 as shown in the figure. 505 and link information 502 and 506 are generated. Figure 5 shows the sequence of character information constituting the range specified text I = - information, and the character 503 "Figure" is the first character of the specified text [-information range, and is the same <504 ".

” is the last character in the specified range of text information. 1 document editing processing unit 103 is displayed in hexadecimal, 1'FFF
The character code IJ is recognized as link information and does not indicate the beginning of the specified range, and rFFF2J is recognized as link information indicating the end of the specified range. The link information 502 and 506 of the link information 502 and 506 are respectively recognized as a code that uniquely represents the link relationship with the related meso and ia.This code is uniquely attached by the system when the link information is generated. When displaying a document on the display 112, the document editing processing unit 103 does not display these control information and link information, and does not edit them as one of the characters.On the other hand, based on the referenced area attribute, the media type 402 When the document editing processing unit 103 generates the display target attribute setting window 3, the determined result is displayed in the paulownia 3O3 indicating the display target type.The user sets the display target name in the display target name 302, and issues the command When “Complete” is specified in the menu 201, document ms processing 81
! 103 leaves the display target attribute setting window 301;
Generated link information, specified area number,.

Link information table 1 for media type and display target name
Store in 02. An example of this storage is shown in FIG.

Next, a mechanism for displaying related media in text-to-speech synthesis processing will be explained along the processing flow shown in FIG. 7.

First, the display information analysis section 04 performs document editing processing &[;1
From 03 onwards, it is suggested to make the text into voice.
For the 7 reports, the presence of punctuation marks is detected in order from the first character information. When a punctuation mark is found or the last character specified to be vocalized is detected, the character information up to that point is grouped together and recognized as one phrase ('701). Next, it is checked whether there is any character information representing the display information in this clause, that is, (71) rFFFIJ in the example of FIG. 5 (702). If the display information is not in the relevant clause,
The data of this clause is sent to the text/analysis unit 105 to proceed with audio output processing (711). If the display information is in the relevant clause, the display information analysis unit] 04 waits to directly send this text information to the TEGIST analysis 81S105,
Perform screen display processing for related media first. When the display information analysis unit 104 detects display information in a clause, it waits until all parameters of the speech synthesis control parameter file 107 are sent to the speech synthesis unit 108c. After monitoring that all the parameters have been sent to the speech synthesis section 108, the speech information analysis section . Display information fee Jl
fli ], 09-Q Li.

It is determined whether the received display information is the information on the beginning of the display, that is, in the example of FIG. Regarding the display information code of the received display information, that is, in the example of FIG. 5, it is rooo.
1", the equivalent data is stored in link information table 1.
02 from the display information code column 601 and access the data having this display information code (705). Then, an area having an area attribute matching the area number 603 searched here is accessed from among the documents in the document processing editing unit 103 (706). Display information control unit 109
also generates a window for displaying related media,
Information about the area in the document accessed here is displayed in this window (707). The display information is deleted from the character information of the relevant phrase (710), and the text analysis section 1.05
and the speech synthesis process proceeds (711).

On the other hand, if the display information received by the display information control unit 109 is information about the display path, that is, in the example of FIG.
If it is F F F 2J (708), the related media display process is finished and the display window is closed (70
9). Then, the speech synthesis process proceeds in the same way as when the display information at the beginning of the display is received (71
0. 711). This process continues until the end of the range specified for reading text information color (
712).

According to this embodiment, since text information and related media can be linked in the same environment as document editing, a user-friendly environment for link setting is provided. Furthermore, since the text-to-speech synthesis process is performed for each clause, when the speech synthesis process is temporarily interrupted for related media display processing, the content to be read out will have some good separation. As a result, stopping the reading does not give the user an unnatural feeling.

[Effects of the Invention] According to the present invention, when the text information of a multimedia document is read aloud by speech synthesis processing, the images in FIG. 7 and Table 7 are displayed one after another on the screen according to the content, so the user can automatically The content of the multimedia document can be easily understood because the user can listen to the voice reading while looking at the images in FIG. 1 and Table 7 associated with the displayed text. Furthermore, since the user can freely set the text information reading range for displaying related media, the related media can be displayed in accordance with the reading speed.

[Brief explanation of the drawing]

FIG. 1 is an overall configuration diagram of a multimedia document processing system according to an embodiment of the present invention, FIG. 2 is a diagram showing an example of a screen display of a multimedia document, FIG. 3 is a diagram showing a display target attribute setting window, and FIG. Figure 4 shows the attribute information of each area in the document, Figure 5 shows text information embedded with display information indicating the start and end of display of related media, and Figure 6.
FIG. 7 is a configuration diagram of a link information table, and FIG. 7 is a flowchart showing a process for displaying related media during text-to-speech synthesis processing. 101... Multimedia document database, 102
...Link information table, 103...Document editing processing section, 1.04...Display information analysis section, 105...Text analysis section, 106...Audio control information generation section, 108...Audio Synthesis department.

Claims (1)

[Claims] 1. An image of a multimedia document processing system that stores multimedia documents including text information and image information, and has an editing function, a display function, and an audio output function of the text information. A display method that associates a certain range of text information with related image information via link information, and displays image information linked to the text based on the link information when outputting audio of the certain range of text. An image display method for a multimedia document processing system, characterized in that: 2. The image display method for a multimedia document processing system according to claim 1, wherein the link information is recorded before and after a certain range of the text.