JP2001160064A - Document processing device, document processing method, recording medium - Google Patents

Abstract

(57) [Summary] [PROBLEMS] Realization of document output according to video data output. SOLUTION: The content of electronic document data corresponding to the video data is presented and output in conjunction with the output of the video data. The electronic document data is presented in a form such as display of a text or a summary, display of a telop, and reading aloud. The presentation of the electronic document is switched according to the time code of the video data.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to electronic document processing, and more particularly, to a document processing apparatus for interlocking and outputting video data and related electronic document data, a document processing method thereof, and an operation program of the document processing method. It relates to a recording medium that has been used.

[0002]

2. Description of the Related Art Conventionally, WWW (World Wide Web) has been provided as an application service for providing hypertext information in a window format on the Internet.

[0003] The WWW is a system for executing document processing for creating, publishing, or sharing a document, and showing the way of a new style document. However, from the viewpoint of practical use of documents, advanced document processing beyond WWW, such as classification and summarization of documents based on the contents of the documents, is required. For such advanced document processing, mechanical processing of the contents of the document is indispensable.

However, mechanical processing of the contents of a document is still difficult for the following reasons. First
HTML, a language that describes hypertext
(Hyper Text Markup Language) specifies the expression of a document, but hardly specifies the content of the document. Second, a network of hypertexts formed between documents is not always easy for a reader of the document to understand the contents of the document. Third
In general, the author of a document writes without considering the convenience of the reader, but the convenience of the reader of the document is not adjusted to the convenience of the author.

[0005] As described above, the WWW is a system showing the way of a new document. However, since the document is not mechanically processed, advanced document processing cannot be performed.
In other words, in order to perform advanced document processing, it is necessary to process the document mechanically.

Therefore, a system for supporting mechanical processing of documents has been developed based on the results of natural language research, with the goal of mechanical processing of documents. As a document processing based on natural language research, mechanical document processing using tags attached to a document has been proposed on the assumption that attribute information about the internal structure of the document by the author of the document, so-called tags are added. I have.

[0007]

By the way, with the recent spread of computers and the progress of networking, text processing, creation of text documents, labeling, change, and the like have been performed by using an index depending on the contents of the documents. There is a demand for enhanced document processing to be performed. For example, it is desired to summarize a document or classify a document according to a user's request. Also,
Processing that is more suitable for users, such as display styles suitable for the form of the document viewer or display unit that displays documents and their summaries, read-aloud functions that accompany display, and interlocking of display of documents and images with read-aloud is required. ing. Furthermore, if it is possible to output video data such as moving images and still images in association with document data, it is expected that users will be able to provide more diverse, sophisticated, and more accurate information. Therefore, it is desired to cooperate with video data in addition to various types of document processing.

[0008]

SUMMARY OF THE INVENTION The present invention has been proposed in view of such circumstances, and has as its object to output document data in an appropriate mode together with the output of video data. And

For this reason, the document processing apparatus of the present invention comprises a determining means for determining the correspondence between electronic document data and video data, a video output control means for controlling output of video data, and a control of the video output control means. Document output control means for controlling the presentation and output of the contents of the electronic document data corresponding to the video data determined by the determination means in conjunction with the output of the video data based on the video data. The determining means determines the correspondence between the electronic document data and the video data based on the video data designation information added to the electronic document data or the time code added to the video data. Further, the document output control means performs control such that display of a text or a summary of electronic document data is executed. Alternatively, control is performed so that the display of a telop for the main text or the summary text of the electronic document data is executed. Alternatively, control is performed so that the text-to-speech output of the text or the summary sentence of the electronic document data is executed. The document output control means monitors the time code of the video data output under the control of the video output control means, and switches the presentation output portion of the electronic document data.

According to the document processing method of the present invention, a discrimination procedure for discriminating a correspondence between electronic document data and video data, a video output control procedure for controlling output of video data,
In conjunction with the output of the video data based on the control in the video output control procedure, the document output control procedure for performing control to present and output the contents of the electronic document data corresponding to the video data determined by the determination procedure, Is done.

It is also assumed that the recording medium of the present invention has recorded thereon an operation control program having the above-described procedures. Further, together with the operation control program, electronic document data or video data is recorded, and these can be provided to the document processing apparatus.

[0012]

Embodiments of the present invention will be described below in the following order. 1. 1. Configuration of document processing device 2. Document data structure 3. Manual classification process for document data 3-1 Processing procedure 3-2 Index creation 3-3 Document browsing / classification creation / classification operation 3-4 Classification model creation / registration 4. Automatic classification processing for document data 4-1 Processing procedure 4-2 Automatic classification Speech processing 6. 6. Summary creation / display processing 7. Output processing of video file 7-1 Video file and tag 7-2 Video output processing 8. Functional block configuration of document processing device 9. Video and linked document display output processing 10. Video and linked document reading output processing Modified example

1. 1. Configuration of Document Processing Apparatus As shown in FIG. 1, a document processing apparatus 1 according to an embodiment of the present invention includes a main body 10 including a control unit 11 and an interface 12, and an input unit that receives an input from a user and sends the input to the main body 10. 20 and a communication unit 21 for transmitting and receiving signals to and from the outside
A display unit 30 for displaying an output from the main body 10; and a recording / reproducing unit 3 for recording / reproducing information on / from a recording medium 32.
1, an audio output unit 33, and a hard disk drive (HDD).

The main body 10 has a control unit 11 and an interface 12, and constitutes a main part of the document processing apparatus 1. The control unit 11 has a CPU 13 that executes processing in the document processing apparatus 1, a RAM 14 that is a volatile memory, and a ROM 15 that is a non-volatile memory. The CPU 13 temporarily stores data, if necessary, in the RAM 14 according to a procedure recorded in the ROM 15, for example, and performs control for executing the program. The operation of the control unit 11 is as follows.
As will be described later in detail, there are a classification process for the supplied document data, a summary creation process, a process of generating a speech reading file for a reading operation, and a document analysis required for these processes. Further, output control processing of video data related to the document data is also performed. The programs and application software necessary for these operations are stored in the ROM 15, the HDD 34, or the recording medium 32. The document processing program used by the control unit 11 is stored in a ROM
15 or a recording medium 32 or HDD
34, for example, the communication unit 2
It is also conceivable to download a document processing program provided by an external server or the like from a network such as the Internet via the Internet.

The interface 12 includes a control unit 11, an input unit 20, a communication unit 21, a display unit 30, and a recording / reproducing unit 3.
1. Connected to the audio output unit 33 and HDD. Under the control of the control unit 11, the interface 12 inputs data from the input unit 20, inputs and outputs data to and from the communication unit 21, outputs data to the display unit 30, and controls the recording / reproducing unit 31. Each operation of data input / output, data output to the audio output unit 33, and data input / output to the HDD 34 is performed. More specifically, it adjusts the timing of data input / output between the control unit 11 and each of the above units, converts the format of data, and the like.

The input section 20 is a section for receiving an input from the user to the document processing apparatus 1. This input unit 20
Is composed of, for example, a keyboard and a mouse. The user uses the input unit 20 to input characters such as keywords using a keyboard, or to display the display unit 30 using a mouse.
For example, the user can select an element of the electronic document that is displayed in. Hereinafter, an electronic document handled by the document processing apparatus 1 is referred to as “document data” or simply “document”. The “element” is an element constituting a document, and includes, for example, a document, a sentence, and a word.

The communication unit 21 is a unit that receives a signal transmitted from the outside to the document processing apparatus 1 via, for example, a communication line, and transmits a signal to the communication line. The communication unit 21 receives one or a plurality of data such as documents transmitted from the outside, and sends the received data to the main body 10. Of course, data is transmitted to an external device via a communication line. Also, one or a plurality of video data (video files) accompanying or independently of the document data
May be captured by the communication unit 21.

The display section 30 is a section for displaying characters and image information as an output of the document processing apparatus 1. The display unit 30 is, for example, a cathode ray tube (CR).
T), a liquid crystal display (LCD), or the like, for example, displays one or more windows, and displays characters, figures, and the like on these windows.

The recording / reproducing section 31 records / reproduces data on / from a recording medium 32 such as a so-called floppy disk. Here, the recording medium 32
An example of a floppy disk (magnetic disk) is given as an example of an optical disk, a magneto-optical disk,
Recording medium 3 if it is a portable medium such as a memory card.
2 can be applied as an example. Then, the recording / reproducing unit 31
Any recording / reproducing device (disk drive, card drive, etc.) according to the media may be used.

When the recording medium 32 stores a document processing program for processing a document,
The recording / reproducing unit 31 can read out the document processing program from the recording medium 32 and supply it to the control unit 11. If document data or video data is recorded on the recording medium 32, the recording / reproduction unit 31 can read out the data and supply the read data to the control unit 11. That is, the document processing apparatus 1 is an input mode of the document data and the video data which is different from the reception of the document data and the video data by the communication unit 21. Further, the control unit 11 can cause the recording / reproducing unit 31 to record the document data processed by the document processing apparatus 1 on the recording medium 32.

The voice output unit 33 is a part for outputting a document as an output of the document processing apparatus 1 as a reading voice. That is, when a voice signal generated by a voice synthesis process based on document information (a reading file to be described later) is supplied by the control unit 11, the voice output unit 33 performs output processing of the voice signal to perform display. Together with the unit 30, it functions as an output unit of the document processing apparatus 1.

The HDD 34 provides a large-capacity recording area in the document processing apparatus 1. The HDD 34 includes the control unit 11
The recording / reproduction of information is performed based on the control of. This HDD
Reference numeral 34 denotes an application program for various processes executed by the control unit 11, for example, a program for voice synthesis, or the like, or for example, document data or video data loaded into the document processing apparatus 1. Etc. can be used as a part for storing.

2. Document Data Structure Next, the structure of document data in this example will be described. In this example, document processing is performed with reference to a tag that is attribute information assigned to the document. The tags used in this example include syntactic
c) There are tags and semantic / pragmatic tags that enable mechanical understanding of the content of documents between multiple languages.

Some syntactic tags describe the internal structure of a document. As shown in FIG. 2, the internal structure by tagging is configured such that each element such as a document, a sentence, and a vocabulary element is linked to each other by a normal link and a reference / referenced link. In the figure, a white circle “○” indicates an element, and the lowest white circle is a vocabulary element corresponding to the minimum level word in the document. The solid line is a normal link indicating a connection between elements such as a document, a sentence, and a vocabulary element, and the dashed line is a reference link indicating a dependency relationship by reference / reference. The internal structure of a document is document (documemt), subdivision (subdivision), paragraph (paragraph), sentence (sentenc
e), subsentential segment (subsentential s
egment), ..., vocabulary elements. Of these, subdivisions and paragraphs are optional.

On the other hand, as the semantic / pragmatic tagging, there is a method of describing information such as a meaning like a meaning of a polysemy. Tagging in this example is performed by HTML (Hyper Te
XML (Extensible Mar) similar to xt Markup Language
kup Language).

An example of tagging is shown below, but tagging a document is not limited to this method. In the following, examples of English and Japanese documents are shown, but the description of the internal structure by tagging can be similarly applied to other languages.

For example, “Time flies like an arro
The sentence "w." can be tagged as follows. <> is a tag added to the document.

<Sentence><noun phrase word meaning = “time0”> time </ noun phrase><verbphrase><verb word meaning = “fly1”> flies <// verb><adverbphrase><adverb word meaning = like0> like < / Adverb><
Noun phrase> an <noun meaning = “arrow0”> arrow </ noun></ noun phrase></ adverb phrase></ verb phrase>. </ Sentence>

Here, <sentence>, <noun>, <noun phrase>, <
Verbs, <verb phrases>, <adverbs>, <adverb phrases> are sentences, nouns, noun phrases, verbs, verb phrases, adjectives / adverbs (including prepositional or postpositional phrases), adjective phrases / adverbial phrases, respectively. Is shown. In other words, the syntactic structur of a sentence
e).

These tags are arranged immediately before the front end and immediately after the end of the element. The tag placed immediately after the end of the element indicates that it is the end of the element by the symbol "/". Elements are syntactic constructs: phrases, clauses, and sentences. Note that the meaning (word sense) = “time0” corresponds to the word “ti”.
It refers to the plural meanings of “me”, that is, the 0th meaning of the plural meanings.
Has at least the meanings of a noun, adjective, and verb. Here, it indicates that the word "time" is a noun (= the 0th meaning). Similarly, the word "orange" has at least the meaning of plant name, color, and fruit, but these can also be distinguished by their meaning.

In this example, the syntactic structure of the document data can be displayed in the window 101 on the display unit 30, as shown in FIG. In this window 101, vocabulary elements are displayed on the right half 103, and the internal structure of the sentence is displayed on the left half 102.

For example, as shown in FIG.
1 has a sentence “A” in which the internal structure is described by tagging.
In C City, where his B meeting was over, some popular and general newspapers have stated on paper that they will voluntarily regulate their photographic coverage. Is displayed. An example of tagging this document is as follows:

<Document><sentence><adverb phrase relation = “place”
><Nounphrase><adverb phrase phrase = “C city”><adverb phrase relation = “subject”><noun phrase identifier = “B
Kai ”><adverb phrase relation =“ affiliation ”><person name identifier =
"/ Mr. A"> Mr. A // person name </ adverb phrase><organization name identifier = “B society”> B society </ organization name> // noun phrase>
/ Adverb phrase> finished </ adverb phrase><place name identifier = "C city"> C city </ place name></ noun phrase>, </ adverb phrase><adverb phrase relation = "subject"><noun phrase Identifier = "press" Syntax = "parallel"><nounphrase><adverbphrase> Some </ adverb phrase> popular paper </ noun phrase> and <noun> general paper </ noun><
/ Noun phrase> is </ Adverb phrase><Adverbphrase> Relation = "Object"><Adverb phrase Relationship = "Content" Subject = "press"><Adverb phrase Relation = "Object"><Nounphrase><AdverbPhrase><
Noun co-reference = "B-kai"> so </ noun></ adverb phrase>
Photo reporting </ noun phrase></ adverb phrase> Self-regulating </ adverb phrase> Policy </ adverb phrase><adverb phrase relation = "position"></ adverb phrase> on paper. </ Text></text>

By being tagged in this way, the structure of the document is expressed by each pair of tags <> to </>. For example, <document> to </ document> indicates the range of one document, and similarly, <sentence> to </ sentence> indicates the range of one sentence. Also, for example, <Noun phrase identifier = “B meeting”>
According to </ noun phrase>, the part "Mr. A's B meeting"
It is expressed as a noun phrase with "kai" as the identifier. That is, the internal structure of the sentence shown on the left half surface 102 in FIG. 3 is expressed by the tagging.

Further, in this document, "part of popular paper and general paper" is expressed in parallel by a tag of syntactic = "parallel". The definition of parallel is to share a dependency relationship. If nothing is specified, for example, <noun phrase relation = x><noun> A </ noun><noun> B <// noun></ noun phrase> indicates that A has a dependency on B Represent. Relation = x represents a relation attribute.

The relation attribute describes the interaction between syntactic, meaning, and rhetorical. Grammar functions such as subjects, objects, and indirect objects, subject roles such as an actor, a subject, a beneficiary, etc., and rhetorical relations such as a reason and a result are described by the relation attributes. In this example, the subject, object,
Describe relative attributes for relatively easy grammar functions such as indirect objects.

In this document, "Mr. A"
For proper nouns such as "B Association" and "C City", place names,
Attributes are described by tags such as person names and organization names. By adding tags such as place name, person name, organization name, etc.,
It is expressed that the word is a proper noun.

3. Manual Classification Processing on Document Data 3-1 Processing Procedure In the document processing apparatus 1 of this example, when document data is taken in from the outside by, for example, the communication unit 21 (or the recording / reproducing unit 31), the document data is converted according to the content. Perform classification processing. In the following description, it is described that document data from the outside is taken in via the communication unit 21.
The description is the same when the document data is supplied from the outside in the form of a portable medium such as a floppy disk and the document data is taken in from the recording / reproducing unit 31.

As the classification process, there are a manual classification process in which the user manually classifies according to the contents of the document data, and an automatic classification process in which the document processing apparatus 1 automatically classifies. These classification processes are performed based on a classification model described later. However, in the document processing apparatus 1, no classification model exists in an initial state. Therefore, at the time of the initial state, a classification process including creation of a classification model is required as a manual classification process. After the classification model is generated, the input document data can be automatically classified. First, a manual classification process that needs to be performed first will be described. That is, the manual classification process means that when the document processing apparatus 1 in the initial state receives document data sent from the outside, the control unit 11 creates a classification model and classifies the document data based on a user operation. Is performed.

FIG. 4 shows the overall processing procedure as the manual classification processing. The detailed processing of each processing step will be described later.

Step F11 in FIG.
2 shows a document receiving process by the receiving unit 21 of FIG. In step F11, the receiving unit 21 receives one or a plurality of documents transmitted via, for example, a communication line. The receiving unit 21 sends the received document to the main body 10 of the document processing device. The control unit 11 stores the supplied document data or document data in the RAM 14 or the HDD 34.

When a document is received, a video file containing video data (moving image or still image data) related to the document may be received (of course, only video data may be received). When a video file is received, the control unit 11 takes in the video file and stores it in the RAM 14 or the HDD 34.
Processing relating to the video file will be described later in detail.

In step F12, the control section 11 of the document processing apparatus 1 extracts the features of the plurality of documents sent from the receiving section 21 and creates feature information, that is, an index of each document. The control unit 11 stores the created index in, for example, the RAM 14 or the HDD 34.
As will be described later, the index includes proper nouns, meanings other than proper nouns, and the like, which are characteristic of the document, and can be used for document classification and search.

The document browsing in step F13 is a process executed as required by the user. That is, it is performed according to a user operation. Step F13 and the next step F14 are processes based on a user operation.
The user can view the content of the input document data on the screen of the display unit 30 by performing a required operation. While browsing the document, the user can instruct various processes such as summarization, which will be described later, by operating the icons and the like on the screen. The process will proceed to the classification operation. In step F14, in response to the user performing an operation of setting a classification item (the classification item is also referred to as a category in this specification), the control unit 11
Generates / displays classification items. The user also performs an operation of sorting the document data into the set classification items, and the control unit 11 sorts and displays the document data accordingly.

In step F15, the control unit 11 creates a classification model in accordance with the classification item creation and classification operation performed by the user in step F14. The classification model includes a plurality of classification items (categories) for classifying documents, and associates each document index with each category index (index of each document created in step F12) to change the classification state. This is the specified data.
After generating such a classification model, step F16
Then, the classification model is registered. That is, the control unit 11 performs the registration by storing the classification model in, for example, the RAM 14. By the processing of FIG. 4 described above, manual classification and creation of a classification model have been performed for one or a plurality of pieces of document data input when the document processing state 1 is in the initial state. The processing of step F12 and subsequent steps in FIG. 4 will be described in detail.

3-2. Index Creation In step F14, the control unit 11 creates an index for the input document data. First, a specific example of an index created for certain document data will be described.

<Index date = "AAAA / BB / CC"
Time = “DD: EE: FF” Document address = “1234”><User operation history Maximum summary size = “100”><Number of selected elements = “10”> Picturetel </
Selection> ... </ User operation history><Summary> Tax reduction scale, not touching-Prime Minister X's interview </ Summary><Word Meaning = "0003" Central activity value = "140.6"> No touch </ word><Word meaning = “0105” identifier = “X” central activity value =
“67.2”> Prime Minister </ word><person name identifier = “X” word meaning = “6103” central activity value = “150.2”> X Prime Minister </ word / person name><word meaning = “5301” central activity value = “ 120.6 ”> sought </ word><word meaning =“ 2350 ”identifier =“ X ”central activity value =
“31.4”> Prime Minister </ word><word meaning = “9582” central activity value = “182.3”> emphasized </ word><word meaning = “2595” central activity value = “93.6”> touch </ word><Word meaning = “9472” central activity value = “12.0”> forecasted </ word><word meaning = “4934” central activity value = “46.7”> not touched </ word><word meaning = “0178” Central activity value = "175.7"> Explained </ word><word Meaning = "7248" Identifier = "X" Central activity value =
“130.6”> I </ word><word Meaning = “3684” Identifier = “X” Central activity value =
“121.9”> Prime Minister </ word><word meaning = “1824” Central activity value = “144.4.”> Appealed </ word><word meaning = “7289” Central activity value = “176.8”> showed Word></index>

In this index, <index> and </ index> indicate the start and end of the index, <date> and <time> indicate the date and time when this index was created, <summary> and </ Summary> indicates the start and end of the summary of the contents of this index, respectively. Also, <word> and <
/ Word> indicates the beginning and end of the word. Further, for example, the meaning = “0003” indicates that it is the third meaning. The same applies to other cases. As mentioned above,
Since the same word may have a plurality of meanings, a number is predetermined for each meaning in order to distinguish them, and the corresponding meaning is represented by a number.

<User operation history> and </ user operation history> indicate the start and end of the user's operation history, and <select> and </ select> indicate the start and end of the selected element. , Respectively. The maximum summary size = “100” indicates that the maximum size of the summary is 100 characters, and the number of elements = “10” indicates that the number of selected elements is 10.

As in this example, the index includes proper nouns, meanings other than proper nouns, and the like, which are characteristic of the document. For example, the process of step F12 for creating such an index will be described with reference to FIGS. FIG. 5 shows an index creation process for one document data. Therefore, when a process is performed for a plurality of document data, the process of FIG. 5 is performed for each document data. Further, FIG. 7 shows a detailed process of step F31 in FIG. 5, and FIG. 8 shows a detailed process of step F43 in FIG.

As the index creation processing in step F12 in FIG. 4 described above, first, active diffusion in step F31 in FIG. 5 is performed. This active diffusion is a process in which the central activity value of an element is spread based on the internal structure of a document with respect to document data, so that an element related to an element having a high central activity value also has a high central activity value. It is. That is, after giving a central activity value as an initial value to each element constituting the document, the central activity value is diffused based on the internal structure of the document, specifically, the link structure. Since this central activity value is determined according to the internal structure by tagging, it is used for extracting features of a document. The control unit 11 determines in step F3
As 1, the active diffusion is performed, and the central active value of each element obtained as a result of the active diffusion is stored in, for example, the RAM 14.
Will be stored.

FIG. 6 shows the active diffusion in step F31.
8 will be described in detail. First, an example of a link structure between elements is shown in FIG. In FIG.
The periphery of elements E1 and E2 is shown as part of the structure of the elements and links that make up the document. E1
E8 is an example of an element, in which element E
1 and E2 will be described.

It is assumed that the central activity value of element E1 is e1, and the central activity value of element E2 is e2. The elements E1 and E2 are connected by a link L12 (the normal link or the reference link described above). The end point of the link L12 connected to the element E1 is T12, and the end point of the link L12 connected to the element E2 is T21. Element E1 further includes elements E3, E4,
E5 is connected to links L13, L14, and L15, respectively. The end points of the links L13, L14, and L15 on the element E1 side are denoted by T13, T14, and T, respectively.
It is assumed to be 15. Element E2 is composed of elements E6 and E6.
Links L26, L27, L2 for E7 and E8 respectively
8 are connected. Each link L26, L27, L28
At the element E2 side are denoted by T26 and T26, respectively.
27 and T28. The active diffusion processing of FIGS. 7 and 8 will be described using an example of such a link structure.

In step F41 of FIG. 7, when starting the active diffusion for the document data to be indexed, the control unit 11 first initializes the central active value for all elements of the document data. As the initial value of the central activity value, for example, a high value is given to a proper noun, an element selected (clicked) by a user, or the like. Further, the control unit 11 sets the end point activation value of the end point T (xx) of the link connecting the elements to 0 for the reference link and the normal link. The control unit 11 stores, for example, the RAM 14 with the initial value of the endpoint activation value thus assigned.

In step F42, the controller 11
Initializes a counter for counting the elements Ei constituting the document. That is, the count value i of the counter for counting elements is set to one. If i = 1, this counter will refer to the first element (eg, element E1 in FIG. 6).

In step F43, the control unit 11
Executes a central activation value update process for calculating a new central activation value for the element referenced by the counter. This center activation value update process will be described in detail with reference to FIG. 8 by taking the process for the element E1 as an example. The center activation value updating process is a process of updating the end point activation value of the element, and calculating a new center activation value using the updated end point activation value and the current center activation value.

In step F51 of FIG.
Is an element Ei (for example, in this case E
In 1), a counter for counting the number of links having one end connected thereto is initialized. That is, the count value j of the counter for counting the links is set to one. When j = 1, this counter refers to the first link L (yy) connected to the element Ei. In the example of FIG.
For example, the link L12 is referred to as the first link for the element E1.

In step F52, the control section 11 determines whether or not the link being referred to, that is, the link L12 connecting the elements E1 and E2, is a normal link by referring to the tag of the related attribute. If the link L12 is a normal link, the control unit 11 proceeds to step F53, and if the link L12 is a reference link, proceeds to step F54.

If the link L12 is determined to be a normal link and the process proceeds to step F53, the control section 11 performs a process of calculating a new endpoint activation value of the endpoint T12 connected to the normal link L12 of the element E1. Endpoint T12
Are the endpoint activation values of all the endpoints connected to the link other than the link L12 among the endpoint activation values of the linked element E2 (in this case, T26 and T2).
7, each end point activation value t26, t27, t28 of T28)
And the central activation value e2 of the element E2, and the value obtained by this addition is divided by the total number of elements included in the document. The control unit 11 performs such an operation using each endpoint activity value and each center activity value read from the RAM 14, thereby calculating a new endpoint activity value for the endpoint connected to the normal link. The calculated end point activation value is stored in the RAM 14. That is, the end point T1
The end point activation value t12 of No. 2 is updated.

On the other hand, if it is determined in step F52 that the link L12 is a reference link and the process proceeds to step F54, the control unit 11 similarly sets a new endpoint activation of the endpoint T12 of the element E1 connected to the normal link L12. The processing for calculating the value is performed. The calculation for calculating the end point activity value is as follows. That is, in this case, the endpoint activation values t12 of the endpoint T12 are the endpoint activation values of all the endpoints connected to the link other than the link L12 among the endpoint activation values of the linked element E2 (in this case, T2
6, each end point activation value t26, t27, t of T27, T28
28) and the central activation value e2 of the element E2. (That is, the point where there is no division is different from the case of the above-mentioned normal link.) The control unit 11 performs such an operation using each of the end point activation values and each of the center activation values read from the RAM 14, A new endpoint activity value is calculated for the endpoint connected to the reference link, and the calculated endpoint activity value is stored in the RAM.
14 is stored. That is, the end point activation value t1 of the end point T12
Update 2.

After performing the processing in step F53 or F54, the control unit 11 performs the processing in step F5 through the determination processing in step F55 (if the determination result is NO).
Proceeding to 7, the count value j is incremented, and the process returns to step F52. That is, subsequently, by setting the count value j = 2, the second link (for example, link L13) for the element E1 is referred to.
In the same manner as described above, the end point activation value t13 of the end point T13 connected to the link L13 is calculated / updated in the processing after step F52.

In step F55, the control unit 11 determines whether or not new end point activation values for all links have been calculated for the element Ei (E1) currently being referred to by the count value i, and branches the processing. Because
The process of updating the endpoint activation values is performed until all endpoint activation values of the element Ei being referred to are updated. That is, by repeating the process while incrementing the count value j in step F57, for the element E1, for example, the endpoint activation values t12, t13, t14, and t15 are updated for the endpoints T12, T13, T14, and T15, respectively. When all of them have been updated, the process proceeds from step F55 to F56.

In step F56, a new central activity value ei of the element Ei is calculated by using the updated endpoint activity values in response to all the endpoint activity values for the element Ei being obtained. The new central activation value ei of the element Ei is the current central activation value e of the element Ei.
It is determined by the sum of new endpoint activation values of all endpoints of i and element Ei. For example, in the case of the element E1 in FIG. 6, the new central activation value e1 (new) is e1 (new) = e1 + t12 + t13 + t14 + t15.

The control unit 11 calculates the central activation value ei of the element Ei being referred to by the current count value i in this way. Then, the control unit 11 causes the RAM 14 to store the calculated new central activation value ei. That is, the central activation value ei of the element Ei is updated. (However, at this point, the old center activation value is also held because it is used in the process of step F45 described later.)

When the processing shown in FIG. 8 is performed as the central activation value updating processing in step F43 in FIG. 7, the processing of the control unit 11 proceeds to step F44 in FIG.
The control unit 11 determines whether or not the central activation value update processing has been completed for all elements in the document. Specifically, control unit 11 determines whether or not count value i has reached the total number of elements included in the document. Control unit 1
When the central activation value update processing has not been completed for all the elements, the process proceeds to step F47, where the count value i is incremented, and
Return to For example, after the processing for the element E1 is completed as described above, the count value i is set to 2 and the element E2 is referred to this time. Then, for the element E2, the central activation value update processing of step F43 (that is, the processing of FIG. 8) is performed in the same manner as described above. Although the details will not be described because they are redundant description, in the case of the link example of FIG. 6, in the case of the element E2, the end point activation values t21, t26, t27 of the end points T21, T26, T27, T28 in the processing of FIG. After t28 is updated,
A new central activity value e2 (new) is calculated as e2 (new) = e2 + t21 + t26 + t27 + t28, and is updated.

In the processing shown in FIG. 7, the count value i is incremented in step F47 and the central activation value update processing in step F43 is repeated while the reference element is changed. Will be updated.

When the update of the central activation value has been completed for all the elements in the document, the processing is terminated at step F44.
To F45. In step F45, the control unit 11 calculates the average value of the change in the central activity values of all the elements included in the document, that is, the change in the newly calculated central activity value from the original central activity value. For example, the control unit 11 reads the old central activation value stored in the RAM 14 and the updated new central activation value for all elements included in the document. Then, for each element, the difference between the new central activity value and the old central activity value is obtained, and the sum of the differences is divided by the total number of elements to calculate the average value of the change in the central activity value of all elements. The control unit 11 causes the RAM 14 to store, for example, the average value of the change in the central activity value of all elements calculated in this way.

Subsequently, at step F46, the control unit 11
Determines whether the average calculated in step F45 is within a preset threshold. When the average value is within the threshold value, the control unit 11 ends a series of steps as the active diffusion process. When the average value is not within the threshold value, the control unit 11 returns to step F42 and performs the above-described series of processes. Perform the process again.

This series of active diffusion processing can be said to be processing for raising the central activity value of an element related (linked) to an element having a high central activity value. However, by performing this activation diffusion only once, when considering the purpose of the index creation processing, among the elements that should be able to raise the central activation value,
In some cases, the central activity value may not be sufficiently raised. For example, in one activation diffusion, for an element directly linked to an element for which the initial value of the central activation value is set high, the central activation value is raised to some extent, but for an element that is not directly linked, it is Even if the element is important as an index, the central activity value may not be sufficiently raised. Therefore, by performing the active diffusion process a plurality of times as necessary based on the determination in step F46, the central activity value can be converged as a whole, and important elements that cannot be increased in the central activity value As much as possible. The reason why the central activity value is converged as a whole in the active diffusion process a plurality of times is that the central active value of each element updated in the active diffusion process is based on the central activity value of each element, and further the active activity diffusion process is performed for each element. This is because the central activity value is updated. However, if such activation diffusion processing is performed too many times, the central activation values of all the elements converge and become almost the same, which is inappropriate.
For this reason, in the processing of steps F45 and F46, the end timing of the active diffusion process is determined based on the change so as to obtain the average value of the change of the central active value, so that the active diffusion suitable for index creation is performed. Will be realized.

When the active diffusion processing as shown in FIGS. 7 and 8 (ie, step F31 in FIG. 5) is completed, the controller 1
The process 1 proceeds to step F32 in FIG. In step F32, the control unit 11 determines in step F3
Based on the central activity value of each element obtained in step 1, an element whose central activity value exceeds a preset threshold is extracted. The control unit 11 causes the RAM 14 to store the extracted elements.

Subsequently, in step F33, the control section 11 reads out the elements extracted in step F32, for example, from the RAM 14. Then, the control unit 11 extracts all proper nouns from the extracted elements and adds them to the index. Proper nouns have special properties such as having no meaning and not appearing in dictionaries, and are therefore treated separately from words other than proper nouns. As described above, the word meaning corresponds to each of a plurality of meanings of the word. Whether or not each element is a proper noun can be determined based on a tag attached to the document. For example, in the internal structure by tagging shown in FIG. 3, "A", "B" and "C"
“City” is a proper noun since the related attributes of the tags are “person name”, “organization name”, and “place name”, and the control unit 11 adds the extracted proper noun to the index, and The result is stored in the RAM 14.

In the next step F34, the control unit 1
1 extracts the meanings other than proper nouns from the elements extracted in step F32, adds them to the index, and stores the result in the RAM 14.

By the above processing, for example, an index as in the above specific example is generated. That is, the index finds the features of the tagged document and arranges the features, and the features of the document are determined based on the central activity value diffused according to the internal structure of the document. It will be. Since such an index includes meanings and proper nouns representing features representative of a document, it can be used when referring to a desired document. The index includes, in addition to the meanings and proper nouns representing the characteristics of the document, the document in the RAM 14 (or H
The document address indicating the position stored in DD34) is included.

3-3 Document Browsing / Category Creation / Classification Operation The index creation processing described above with reference to FIGS. 5 to 8 is performed in step F12 in FIG. Therefore, FIG.
As for the manual classification processing of step F13 and step F1
The process proceeds to the process of No. 4, that is, the process of browsing and manual classification by the user as described above.

As described above, in step F13 of FIG. 4, the user can browse the document displayed on the display unit 30. In step F14, the user can perform an operation of setting a classification item and an operation of sorting document data into the set classification item. The operations performed in steps F13 and F14,
The corresponding process of the control unit 11 and a display example of the display unit 30 are as follows.

FIGS. 9 and 10 show specific examples of the display on the display unit 30. FIG. First, FIG. 9 is a display example of a document classification window 201 corresponding to a classification model described later in detail. That is, a graphic user interface (graphic user interface) used for displaying a document classification.
GUI). This document classification window 201
The operation button display 202 includes a position reset button 202a for returning a screen window state to an initial position, a browser button 202b for calling a browser (browser) for reading the contents of a document, An exit button 202c is displayed.

This document classification window 201
Document classification areas 203, 204, 205,... Are formed as small windows corresponding to the classification items corresponding to the classification model. The document classification area 203 is an area for displaying “other topics”. The “other topics” document classification area 203 is an area where documents that have not been classified yet are presented. For example, step F in FIG.
Each document received at 11 (that is, a document to be classified) is presented in the "other topics" document classification area 203. The document classification area 204 is an area where documents classified as, for example, “business news” are presented. The document classification area 205 is an area in which documents classified as, for example, “political news” are presented. In addition to these, document classification areas without reference numerals in the figure are areas where documents corresponding to specific classification items are presented.

Each of these document classification areas 203 and 204
.., The documents classified into the classification items (categories) set in the respective document classification areas are presented by the icon of the document and the title of the document. If there is no title, a summary of one sentence is displayed. Also, the size of each of the document classification areas 203, 204,... Is not fixed,
When the user moves the delimiting frames 211, 212, 213,... Separating the document classification areas by a drag operation or the like, the document classification areas 203, 204,.
Can be arbitrarily changed. The user can arbitrarily increase or decrease the number of document classification areas.

Each of the document classification areas 203, 204,.
The title (for example, "political news") can be arbitrarily set and changed by the user. Note that the number of document classification areas and each title correspond to the classification items of the classification model described later. In other words, the user sets or deletes a document classification area or performs a title setting in the classification window 201 by operating the input unit 20 using a mouse, a keyboard, or the like.
The number and title of the classification items of the classification model are set.

FIG. 10 shows an example of a browsing window 301 for a user to browse the contents of document data. For example, the user clicks on a certain document in the classification window 201 of FIG. 9 to select the document, and then clicks the browser button 202b, so that the control unit 11
Opens a browsing window 301 for displaying the selected document as shown in FIG.

In the browsing window 301, a file name display unit 3 for displaying the file name of the document data file
02, a document display unit 303 for displaying the document data of the file name, a summary display unit 304 for displaying a summary of the document displayed on the document display unit 303, and a keyword display unit 305 for inputting / displaying a keyword. . As the operation button display 306, a summary creation button 306a for instructing creation of a summary sentence, an undo button 306b for performing an undo operation (operation cancellation), a reading button 306c for executing a reading operation, and video data A video button 306d or the like for instructing the display of is displayed.

In such a browsing window 301,
The user can browse the document displayed on the document display unit 303. If the entire document cannot be displayed, a part of the document is displayed. Of course, you can browse the full text by scrolling. In addition, by clicking the summary creation button 306a, the user can cause a summary sentence for the document displayed on the document display unit 303 to be created and displayed on the summary display unit 304. Further, by clicking the read-aloud button 306c, the user can cause the text or the summary of the document displayed on the document display section 303 to be read aloud. Further, by clicking the video button 306d, the user can cause the display of the text of the document displayed on the document display unit 303 or the display of the video data related to the summary displayed on the summary display unit 304. it can. The processing of the control unit 11 for creating the summary sentence, the processing for the reading operation, and the processing for the video output operation will be described later.

The classification window 201 and the browsing window 301 as described above are displayed on the display unit 20 at any time according to the user's operation, not limited to the manual classification processing in FIG. As for the manual classification process,
The user can confirm the type and content of the received document in the classification window 201 and the browsing window 301. Specifically, one or a plurality of documents received in step F11 in FIG. 4 are subjected to the index creation processing in step F12, and then the classification window 2 shown in FIG.
01 "other topics" document classification area 20
3 is displayed. In this classification window 201,
The user manually classifies each of the documents displayed in the document classification area 203. If, for example, the content is not known only by the title of the document, the user checks the content of the document by using the browsing window 301 in FIG. . The browsing performed as needed by the user in this manner is the process of step F13 in FIG.

In step F14, the user can arbitrarily add, update, or delete a classification item on the classification window 201, and in accordance with the operation, the control unit 11 causes the displayed document classification area 203, 204 ...
・ Change the display mode (number, area, title, etc.). The setting / change of the classification item (title of the document classification area) by the user is reflected in the classification model described later.

After setting the classification items as necessary, the user sorts each document displayed in the document classification area 203 into each document classification area. That is, the documents are classified manually by the user. Specifically, the icon of the document displayed in the document classification area 203 of “other topics” is dragged to the document classification area corresponding to a desired classification item (category) using, for example, the mouse of the input unit 20. Do it by doing. For example, the user sets a document classification area titled “Sports”, and changes the icon of a sports-related document displayed in the document classification area 203 of “Other topics” to the document classification area of “Sports”. Perform an operation like dragging. The icons and titles of the documents that have been manually classified in this manner are thereafter displayed in the dragged destination document classification area.

3-4 Creation / Registration of Classification Model As described above, when the user performs the manual classification operation, the control unit 11 determines in step F15 of FIG. 4 that the classification model is composed of a plurality of categories based on the user's classification operation. Create That is, the control unit 11 collects indices of the plurality of documents classified into each category and generates a classification model. Then, the plurality of documents are classified into each category of the classification model.

The classification model is composed of a plurality of classification items (categories) for classifying documents. Then, for each category, a data format is provided in which the classified documents are shown.
An index is formed for each document in step F12 and the like, and the classification model is, for example, as shown in FIG.
As shown in (1), the data structure is such that the indexes of the documents classified for each category are associated with each other.
In FIG. 11A, the category is “sports”.
“Company”, “Computer”, etc. are set, but these are the classification items set by the user in the classification window 201 as described above. Of course, even if the user does not set, it is set in advance (that is, displayed as a document classification area in the classification window).
There may be categories. Each of the classification items is associated with an index IDX1, IDX2,.
That is, the index of the document classified by the user as described above is associated with each classification item.

The index associated with each classification item is the index of the document displayed in the document classification area of the classification item in the classification window 201. For example, the index IDX1 is associated with the category "sports" because the user creates a document classification area with the title "sports" in the classification window 201, and furthermore, sets the icon of the document of the index IDX1 to "sports". Is manually dragged to a document classification area having a title of "".

As described above, the index of each document includes proper nouns, meanings other than proper nouns, document addresses, and the like. Then, for example, as shown in FIG.
One or a plurality of indices are associated with one classification item. Since the indexes include proper nouns, meanings, document addresses, and the like, the classification model can be represented as shown in FIG. 11B.

That is, as shown in FIG. 11B, the classification model has a structure having columns of proper nouns, meanings other than proper nouns, and document addresses as category indexes corresponding to each category. Then, in the classification model, the proper noun “Mr. A” for each category “sports” “society” “computer” “plant” “art” “event”
... "," Mr. B, ... "," Company C, Company G, ...
"," D type, ... "," Mr. E, ... "and" F
A proper noun such as "Mr." is also assigned.
546), ground (2343), ... "," labor (3112), unique (9821), ... "," mobile (2102), ... "," cherry blossom 1 (11111), orange 1 ( 9911) "," Sakura 2 (11112), Orange 2 (9912) ", and" Sakura 3 (11113) "are also assigned to each category. Further, the document addresses" SP1, SP2, SP3,... " “S01,
S02, S03,... "," CO1, CO2, CO
3, ... "," PL1, PL2, PL3, ... ",
"AR1, AR2, AR3, ..." and "EV1,
EV2, EV3,... "Are also assigned to each category.

Note that "Sakura 1", "Sakura 2" and "Sakura 3"
The first meaning (11111) and the second meaning (1111)
112) and the third meaning (11113). “Orange 1” and “Orange 2” indicate the first meaning (9911) and the second meaning (9912) of “orange”. For example, "orange 1" represents a plant orange, and "orange 2" represents an orange color. The reason that the meaning is used instead of the word itself in cases other than proper nouns is that the same word may have a plurality of meanings.

In step F15 in FIG. 4, for example, such a classification model is generated according to a manual classification operation by the user. Then, the classification model is registered as Step F16,
That is, it is recorded in the RAM 15 (or the HDD 34). The generation / registration of the classification model in this way means that the classification of the document has been performed.

Note that, as described above, step F in FIG.
After the creation / registration of the classification model is performed as F15 and F16, the classification model is sequentially updated in accordance with automatic classification processing described later, editing of a classification item by a user, or manual classification operation. Become. When the classification model is updated, the update date and time are recorded in the classification model. FIG. 11 shows that the update date and time “19:05 on December 10, 1998
6 minutes and 10 seconds "are recorded.

4. 4. Automatic Classification Processing for Document Data 4-1 Processing Procedure In the document processing apparatus 1 of the present example, after the classification model is once created as described above, for example, the document data taken in from the outside by the communication unit 21 is automatically converted. The automatic classification process of classifying into is possible. That is, the automatic classification process described below is a process of, when the document processing apparatus 1 receives document data sent from the outside, classifying the document data with respect to a classification model. In this example, the automatic classification process described below may be performed each time one document is received, or may be performed each time a plurality of predetermined documents are received. The automatic classification process may be performed on all the documents received so far when the operation of opening is performed.

FIG. 12 shows the entire processing procedure as the automatic classification processing. Step F21 in FIG. 12 illustrates a document receiving process by the receiving unit 21 of the document processing apparatus 1. In this step F21, the receiving section 21 receives one or a plurality of documents transmitted via a communication line, for example. The receiving unit 21 sends the received document to the main body 10 of the document processing device. The control unit 11 stores the supplied document data or document data in the RAM 14 or the HDD 34. When one or a plurality of video files are received together with the document data (or independently of the document data), the control unit 11 fetches the video file and stores the video file in the RAM 14 or the HDD 3.
4 will be stored.

Subsequently, the process proceeds to step F22, where the control unit 11
Creates an index for the document taken in step F21.

In step F23, the control section 11 converts each indexed document based on the classification model into
Automatic classification into one of the categories of the classification model. Then, the control unit 11 causes the RAM 14 to store the result of the classification, for example. Details of the automatic classification will be described later.

In step F24, the control unit 11 updates the classification model based on the result of the automatic classification of a new document in step F23. Then, in step F25, the control unit 11 registers the classification model updated in step F24. For example, the classification model is stored in the RAM 14.

By the above-described processing in FIG. 12, the automatic classification processing is performed so that the document data input to the document processing state 1 is classified on the classification model. That is, in this automatic classification process, an index is created for the received document, and after the automatic classification is performed, proper nouns, meanings,
The document address and the like are associated with a certain category on the classification model as shown in FIG. 11 (the classification model is updated).

The processes in steps F21 and F22 are the same as steps F11 and F12 in the above-described manual classification process. That is, as the index creation processing in step F22, the processing described with reference to FIGS. 5 to 8 is performed, and a repeated description thereof will be omitted. The update of the classification model in step F24 depends on the classification result of the automatic classification in step F23. Hereinafter, automatic classification in step F23 will be described in detail as processing different from the above-described manual classification processing.

4-2 Automatic Classification FIG. 13 shows the detailed processing of the automatic classification in step F23 in FIG. In step F61 of FIG.
Defines the number of intersections of the set of proper nouns included in the category Ci of the classification model and the set of proper nouns among the words extracted from the document received in step F21 and put into the index by P ( Ci). Then, the control unit 11 calculates the number P (Ci) thus calculated.
Is stored in the RAM 14.

In step F62, the control unit 11
Refers to the degree of association between all meanings included in the index of the document and all the meanings included in each category Ci by referring to the table of association between meanings shown in FIG. The total sum R (Ci) of the degrees is calculated. That is, the control unit 11 calculates the total sum R (Ci) of the degrees of association between all the meanings of words other than the proper noun in the classification model. The control unit 11 then calculates the total sum R (C
i) is stored in the RAM 14.

Here, the degree of association between meanings will be described. The degree of association between meanings may be calculated in advance for the meanings included in the electronic dictionary included in the document processing apparatus 1 by the processing in FIG. 14 and the result may be stored as shown in FIG. That is, if the control unit 11 executes the process of FIG. 14 once in advance, it can be used in the automatic classification process of FIG.

The processing of FIG. 15 executed by the control unit 11 in advance is as follows. First, in step F71, the control unit 11 uses the description of the meaning of a word in the electronic dictionary and creates a meaning network using this dictionary. That is, a meaning network is created from the reference relation between the explanation of each meaning in the dictionary and the meaning appearing in the explanation.
The internal structure of the network is described by tagging as described above. For example, the control unit 11 of the document processing apparatus sequentially reads meanings and descriptions of the electronic dictionary stored in the RAM 14 and creates a network. The control unit 14 causes the RAM 14 to store the meaning network created in this manner.

The network is not only created by the control unit 11 of the document processing apparatus using a dictionary, but also by the receiving unit 21.
The recording / reproducing section 31 reproduces the data received from the outside from the recording medium 32. Further, the electronic dictionary can be obtained by receiving from the outside by the receiving unit 21 or by reproducing from the recording medium 32 by the recording / reproducing unit 31.

In Step F72, Step F7
The central activity value corresponding to each semantic element is spread on the semantic network created in step 1. By this activity diffusion, the central activity value corresponding to each meaning is given according to the internal structure by tagging given by the dictionary. The diffusion processing of the central activity value is the processing described with reference to FIG.

In Step F73, Step F7
One semantic Si constituting the semantic network created in 1 is selected, and in the subsequent step F74,
The initial value of the central activity value ei of the vocabulary element Ei corresponding to this meaning Si is changed, and the difference △ ei of the central activity value at this time is calculated.

Further, in step F75, a difference Δej of the central activation value ej of the element Ej corresponding to another meaning Sj corresponding to the difference Δei of the central activation value ei of the element Ei in step F74 is determined. In step F76, the difference Δe obtained in step F75
quotient ｊej obtained by dividing j by △ ei obtained in step F74
/ △ ei is the degree of association between meanings of the meaning Si with respect to the meaning sj.

In step F77, one meaning Si
It is determined whether or not the calculation of the degree of association between meanings has been completed for all pairs of the word and another meaning Sj. When the calculation of the relevance between meanings is not completed for all pairs of meanings, the process returns to step F73, and the calculation of the relevance between meanings is continued for the pairs for which the calculation of the relevance is not completed. In the loop from step F73 to step F77, the control unit 11 sets a necessary value to, for example, RA
M14 is sequentially read out, and the degree of association between meanings is calculated as described above. The control unit 11 causes the RAM 14 to sequentially store the calculated meaning-to-sense associations, for example. Then, when the calculation of the degree of inter-word meaning is completed for all the word-sense pairs, this series of processing is ended from step F77.

The calculation of the degree of association between meanings is performed by a certain one.
It can be said that, when the central activity value of one meaning is changed, the meaning in which the central activity value changes along with the meaning has a high degree of relevance. That is, if the central activity value of a certain meaning is changed in step F74, the central activity value of the related (linked) meaning changes accordingly, and if the degree of the change is examined, a certain meaning is obtained. The degree of relevance of each other meaning with respect to is understood. (The central activation value of an element Ei reflects the central activation value and the endpoint activation value of the linked element as described in the above description of the activity diffusion, and the element Ei endpoint activation value is updated. In addition, since the value is obtained from the sum of the end point activity value of the element Ei and the current center activity value, the larger the degree of association with the link destination is, the larger the amount of change of the center activity value is.) By performing the processing for all other meanings, it is possible to calculate the degree of association for all pairs (combinations) of meanings.

The calculated degree of association between meanings is defined between each meaning as shown in FIG. In the table of FIG. 15, the degree of association between meanings is normalized to take a value from 0 to 1. In this table, as an example, the degree of inter-semantic association between “computer”, “television”, and “VTR” is shown. The relevance between meanings of "computer" and "television" is 0.55, the relevance between meanings of "computer" and "VTR" is 0.25, and the relevance between meanings of "television" and "VTR" is 0.60. It is.

When the processing of step F62 in FIG. 13 is performed using the degree of association between the meanings calculated in advance as described above, the control unit 11 then proceeds to step F63 where the inter-document classification of the document with respect to the category Ci is performed. Relevance Rel (C
i) is calculated as Rel (Ci) = m1P (Ci) + n1R (Ci). Here, the coefficients m1 and n1 are constants and represent the degree of contribution of the respective values to the inter-document-class relevance. The control unit 11 performs the calculation of the above expression using the number P (Ci) of common sets calculated in step F61 and the sum R (Ci) of the degree of association between meanings calculated in step F62, and calculates the degree of association between document classifications. Calculate Rel (Ci). The control unit 11 causes the RAM 14 to store the thus calculated inter-document-class relevance Rel (Ci).

The values of these coefficients m1 and n1 can be, for example, m1 = 10 and n1 = 1. The values of the coefficients m1 and n1 can also be estimated using a statistical method. That is, the control unit 11 determines the degree of association Rel between document classifications for a plurality of pairs of coefficients m and n.
Given (Ci), the coefficient can be obtained by optimization.

In Step F64, the control unit 11
Is the document class relevance Rel (C
When i) is the largest and the value of the degree of association Rel (Ci) between the document classifications exceeds a certain threshold, the documents are classified into the category Ci. That is, the control unit 11 creates the inter-document-class relevance for each of the plurality of categories, and when the maximum inter-document-class relevance exceeds the cabinet value, the control unit 11 assigns the document with the maximum inter-document-class relevance. Classify into category Ci. As a result, the document is automatically classified into a required category. If the maximum degree of association between document classifications does not exceed the threshold value, no document classification is performed.

The processing shown in FIG.
If the automatic classification in step F23 is performed,
The series of automatic classification is completed by updating and registering the classification model accordingly at 24 and F25. That is, the document data received by the document processing apparatus 1 is automatically classified, and the user can confirm the received document data in a required document classification area, for example, in the classification window 201 of FIG. Become.

5. Read-aloud process Next, a process of reading aloud document data will be described. As described above, the user can browse the text of the document by selecting the document and opening the browse window 301 as shown in FIG. For example, the viewing window 301 can be opened from the classification window 201 described with reference to FIG.

For example, by clicking a browser button 202b while a certain document is selected in the classification window 201, the document display unit 3 is displayed as shown in FIG.
A browsing window 301 displaying the text of the document selected in 03 is opened. When the entire text of the document cannot be displayed on the document display unit 303, a part of the document is displayed. At the point in time when a summary sentence has not been created, the summary display section 304 is blank as shown in FIG. The creation and display operation of the summary sentence will be described later.

As described above, when the document processing apparatus 1 receives the tagged document, the document processing apparatus 1 can display the text and a summary sentence described later in detail, and present the received document to the user by voice. You can also. In other words, the CPU 13 activates a voice reading program among the electronic document processing programs recorded in the ROM 15 or the HDD 34, thereby reading the document through a series of steps as shown in FIG. . First, here, a simplified description of each step is given, and thereafter, each step is explained in detail using a specific document example.

As a process of the control unit 11, step F101 in FIG. 16 is a document receiving / storing process similar to step F11 in FIG. 4 (or step F21 in FIG. 12). As described above, the received document (tagged document) is subjected to manual or automatic classification processing. Similarly, in step S101, FIG. It is written. Here, the order of the classification process and the reading process and the like are not particularly defined as the processing procedure.

It is necessary that a document (reception document) to be subjected to the text-to-speech processing is provided with a tag necessary for performing speech synthesis, as described later. However, after receiving the tagged text, the document processing apparatus 1 can also create a document by newly adding a tag necessary for performing speech synthesis to the document. Hereinafter, the tagged document prepared by receiving or creating in this manner is referred to as a tag file.

In the document reading process of the document processing apparatus 1, subsequently, in step F102, a reading file is generated based on the tag file under the control of the CPU 13. As will be described later, the reading file is generated by deriving attribute information for reading from a tag in the tag file and embedding the attribute information.

Subsequently, in step F103, under the control of the CPU 13, the document processing apparatus 1 performs processing suitable for the speech synthesis engine using the reading file.
This speech synthesis engine may be configured by hardware or may be realized by software. When the speech synthesis engine is realized by software, the application program is stored in the ROM 15 or the HDD 34 in advance. In the document processing apparatus 1 of this example, a functional block configuration formed to execute a reading process, a summary creation / display process, a video output process, and the like, which will be described later, will be described later with reference to FIG. The speech synthesis engine corresponds to the speech synthesis engine 601 in FIG.

Subsequently, the document processing apparatus 1 proceeds to step F104.
In, processing is performed according to an operation performed by a user using a user interface described later. Document processing device 1
Can read out a given document by performing such processing. Each of these steps will be described in detail below.

First, reception or creation of a tagged document in step F101 will be described. The document processing apparatus 1 receives, for example, a document (a document to which a tag required for performing speech synthesis has already been added) from the communication unit 21. Alternatively, the document processing apparatus 1 receives a tagged document, and newly adds a tag necessary for performing speech synthesis to the document to create the document.

As an illustrative example, it is assumed that the following tag file in which Japanese and English documents are tagged is received or created.

First, a Japanese document serving as a source of a tag file is as follows. "[Nice aging] / 8 cancer metastases can be suppressed! Cancer has been the leading cause of death in Japan for more than a decade.
The mortality rate is increasing with age. When thinking about the health of the elderly, you cannot avoid cancer.
Characterizing cancer is cell proliferation and metastasis. There are two types of human cells: "cancer genes", which act as accelerators in automobiles and rapidly grow cancer, and "cancer suppressing genes" that act as brakes. There is no problem if both are balanced. Loss of normal regulatory function and nonbreaking mutations in the cell initiate cancer growth. In the elderly, the mutation accumulates over the years, increasing the proportion of cells with cancerous conditions, leading to more cancers. By the way, without the other characteristic, metastasis, cancer doesn't have to be so afraid. This is because a complete cure is possible just by resection. Here is the importance of suppressing metastasis. This metastasis does not occur simply by increasing the number of cancer cells. Cancer cells dissolve proteins (proteins) between cells, create their own way, and enter blood vessels and lymph vessels. In recent years, it has been elucidated that complicated movements such as searching for a new “dwelling house” while circulating are performed. "

An example of an English document serving as a source of a tag file is as follows: “During its centennial year, The ABC Journal will
report events of the past century that stand as m
ilestones of American business history.THREE COMP
UTERS THAT CHANGED the face of personal computing
were Iaunchedin 1977. That year the PC A II, PC B
and PC C came to market.The computers were crude
by today's standerds.PC A ll owners, for example,
had touse their television sets as screens and sto
red data on audiocassettes.

When the document processing apparatus 1 receives a document to which such a Japanese or English document is tagged,
It is possible to perform the classification process, display the text as shown in FIGS. 26 and 27 described later, or create and display a summary sentence.

Here, the above-mentioned Japanese or English documents are each constituted as a tag file as shown in FIG. 18 or FIG. As a tag file of a Japanese document, FIG. 18 (a) shows an excerpt of the heading “[nicely aging] / 8 cancer metastasis, suppressed !?”, and FIG. 18 (b) It shows an excerpt from the last paragraph in the document, "This metastasis, cancer cells are being elucidated in recent years." The remaining paragraphs have been omitted. It should be noted that the actual tag file is composed of one file from the heading to the last paragraph.

In the heading portion shown in FIG.
The tag <heading> indicates that this part is a heading. The tag files shown in FIGS. 18A and 18B are basically attached with tags in the same manner as the tag file example used when the document data structure was described with reference to FIG. Although a detailed description of each of the above-described tags is omitted, tags required for performing speech synthesis are added to required locations.

For example, first, as shown in the portion of “Example 1” in the figure,
Some are given when information indicating a reading pseudonym is given to the original document, such as “protein”.
In other words, in this case, the pronunciation = “nul
Reading attribute information “l” is described, and a tag for prohibiting partial reading of “(tanpaku)” is added. Also, the tags required to perform speech synthesis include:
As shown in the "Example 2" and "Example 3" in the figure, technical terms such as "lymphatic vessels" and difficult-to-read parts such as "dwelling house" that may be read aloud incorrectly There is. In other words, in this case, "Rinpakuda" or "Sumie"
In order to prevent the pronunciation from being read out, pronunciation attribute information indicating the pronunciation pseudonym of pronunciation = “Rinkan” and pronunciation = “Sumika” is described.

On the other hand, as a tag necessary for performing speech synthesis in the tag file shown in FIG. 19, as shown in a portion shown as “Example 4” in the figure, a pronunciation = “ Two ”reading attribute information is described. This is described in order to prevent "II" from being read out as "second" when it is desired to be read out as "two".

For example, if a document includes a quote, such a tag file includes, although not shown,
A tag indicating that the sentence is a quote is attached. Further, when there is a question sentence in the document, for example, a tag indicating that the sentence is a question sentence is added to the tag file, though not shown.

In step F101 shown in FIG. 16, the document processing apparatus 1 receives or creates a document to which a tag necessary for performing speech synthesis has been added as in the above example. .

Next, the generation of a reading file in step F102 of FIG. 16 will be described. The document processing device 1 derives attribute information for reading out from a tag in the tag file, and generates a reading-out file by embedding the attribute information. Specifically, the document processing apparatus 1 finds tags indicating the beginning of a paragraph, a sentence, or a phrase of a document, and embeds attribute information for reading aloud in correspondence with these tags. When the document processing apparatus prepares the summary of the document, the head of the part included in the summary is found from the document, and the attribute information for increasing the volume at the time of reading out is embedded. Can be emphasized.

The document processing device 1 generates a reading file as shown in FIG. 20 or 21 from the tag file shown in FIG. 18 or FIG. FIG. 20 (a)
(B) corresponds to the portion shown in FIGS. It goes without saying that the actual reading file is composed of one file from the above-mentioned heading portion to the last paragraph.

In the reading file shown in FIG. 20, attribute information of Com = Lang *** is embedded corresponding to the head of the document. This attribute information indicates the language describing the document. Here, the attribute information is Com = Lang = JPN, indicating that the language describing the document is Japanese. In the document processing apparatus, by referring to the attribute information, it is possible to select an appropriate speech synthesis engine corresponding to the language for each document.

The read-out file has embedded therein attribute information such as Com = begin_p, Com = begin_s, and Com = begin_ph. These pieces of attribute information indicate the beginnings of paragraphs, sentences, and phrases of the document, respectively. The document processing device 1 identifies the beginning of these paragraphs, sentences, and phrases based on the tags in the tag file described above. In the reading file, for example, portions where tags representing syntactic structures of the same level appear continuously, such as <adjective verb phrase><nounphrase> in the above-described tag file, correspond to each. Without the number Com = begin_ph being embedded,
It is put together and one Com = begin_ph is embedded.

Further, the read-out file includes Com = be
Attribute information of Pau = 500, Pau = 100, and Pau = 50 is embedded corresponding to gin_p, Com = begin_s, and Com = begin_ph, respectively. These pieces of attribute information indicate that pause periods of 500 ms, 100 ms, and 50 ms are provided for reading out. That is, the document processing apparatus 1 provides pauses of 500 ms, 100 ms, and 50 ms at the beginning of a paragraph, a sentence, and a phrase of a sentence, respectively, so that the document is read out by the speech synthesis engine. Information. Note that these pieces of attribute information are embedded corresponding to Com = begin_p, Com = begin_s, and Com = begin_ph. Therefore, for example, a portion where tags representing the same level of syntactic structure appear continuously, such as <adverb phrase><nounphrase> in the tag file, is regarded as one phrase, and the corresponding number of Pau = One Pau = 50 is embedded together without being embedded. Also,
For example, Pau = *** is embedded in portions where tags representing different levels of syntactic structure appear continuously, such as <paragraph><sentence><nounphrase> in the tag file. Therefore, when reading out such a portion, the document processing apparatus 1 provides a pause period of 650 milliseconds obtained by adding the pause periods of the paragraphs, sentences, and phrases of the document, for example. in this way,
By providing a pause period corresponding to a paragraph, a sentence, and a phrase, the document processing apparatus 1 can perform reading without discomfort in consideration of a break between the paragraph, the sentence, and the phrase. The pause period need not be 600 ms, 100 ms, and 50 ms at the beginning of the paragraph, sentence, and phrase of the document, respectively, and can be changed as appropriate.

Further, in the reading-out file, “(protein)” is removed and the pronunciation = “rin” corresponding to the pronunciation attribute information of pronunciation = “null” described in the tag file. According to the reading attribute information of "kan" and pronunciation = "Sumika", "lymphatic vessels"
"Dwellers" are "Rinkan" and "Sumika", respectively.
Has been replaced by By embedding such reading attribute information, the document processing device 1 prevents reading errors due to inadequate dictionaries referenced by the speech synthesis engine.

Further, in the reading-out file, attribute information for designating only this quotation to use another speech synthesis engine based on a tag indicating that the quotation is included in the document is included. It may be embedded. Further, attribute information for raising the intonation of the ending of the sentence may be embedded in the reading-out file based on a tag indicating that the sentence is a question sentence. Furthermore, in the reading-out file, if necessary, attribute information for converting a so-called "Dama-tona" style to a "Dama-tona" style can be embedded. In this case, instead of embedding such attribute information in the reading-out file, the document processing apparatus 1 converts the style of "Dana-tona" into the style of "Dan-Ma-tona" and converts the speech reading-out file. You may make it generate | occur | produce.

On the other hand, in the read-out file shown in FIG. 21, attribute information Com = Lang = ENG is embedded at the beginning of the document, indicating that the language describing the document is English. ing. The attribute information of Com = Vol = *** is embedded in the file for reading out.
This attribute information indicates the volume at the time of reading. For example,
Com = Vol = 0 indicates that the text is read out at the default volume of the document processing apparatus. Com = Vol = 80 indicates that the default volume is read out at a volume increased by 80%. Any Com = Vol = *** is valid until the next Com = Vol = ***. Further, in the reading file, "II" is replaced by "two" in accordance with the reading attribute information of pronunciation = "two" described in the tag file.

The document processing apparatus 1 generates such a reading file through a series of steps shown in FIG. First, the document processing apparatus 1 executes step F201.
The CPU 13 analyzes the received or created tag file. Here, the document processing apparatus 1 determines the language in which the document is described, and searches for the head of the paragraph, sentence, and phrase of the document, and the reading attribute information based on the tag. Subsequently, in step F202, the document processing apparatus 1 embeds attribute information Com = Lang = *** at the head of the document by the CPU 13 in accordance with the language in which the document is described.

Next, the document processing apparatus 1 executes step F203.
, The CPU 13 replaces the head of the paragraph, sentence and phrase of the document with the attribute information in the reading file. That is, the document processing apparatus 1 replaces <paragraph>, <sentence>, and <*** phrase> in the tag file with Com = begi
Replace with n_p, Com = begin_s and Com = begin_ph.

Further, the document processing apparatus 1 executes step F20.
In 4, the same level of syntactic structure appears by the CPU 13 and the same Com = begin _ ***
Combine into one Com = begin _ ***. Subsequently, in step F205, the document processing apparatus 1 embeds Pau = *** in correspondence with Com = begin _ *** by the CPU 13. That is, the document processing apparatus 1 embeds Pau = 500 before Com = begin_p, embeds Pau = 100 before Com = begin_s,
Embed Pau = 50 before gin_ph. And the document processing device 1
Is determined by the CPU 13 in step F206.
Replace with correct reading based on reading attribute information. That is, the document processing apparatus 1 removes “(protein)” based on the pronunciation attribute information “pronunciation =“ null ”, and based on the pronunciation attribute information“ pronunciation = “Rinkan” and pronunciation = “Sumika”. "Lymphatic vessels", "houses"
Substitute “Rinkan” and “Sumika”, respectively.

In step F102 shown in FIG. 16, the document processing apparatus 1 automatically generates a reading file by performing the processing shown in FIG. The document processing device 1 causes the RAM 14 to store the generated reading file.

Next, the processing using the reading file in step F103 of FIG. 16 will be described.
The document processing device 1 uses the read-out file to
Under the control of the CPU 13, processing suitable for the speech synthesis engine stored in advance in the M15, the HDD 34, or the like is performed.
Specifically, the document processing apparatus 1 selects a speech synthesis engine to be used based on the attribute information of Com = Lang = *** embedded in the reading file. The speech synthesis engine is provided with an identifier according to the type of language or male / female voice, and the information is recorded in the HDD 34 as, for example, an initialization file. The document processing device 1 refers to the initialization file and selects a speech synthesis engine having an identifier corresponding to the language. .

The document processing apparatus 1 converts Com = begin _ *** embedded in the file for reading out into a format suitable for the speech synthesis engine. For example, the document processing device 1
Mark Com = begin_ph with a number in the 10000s, such as Mark = 10000. Also mark Com = begin_s with a number in the 1000s, such as Mark = 1000, and Com = begin_p
Is marked with a number in the 100s, such as Mark = 100. This means that the heads of <phrase>, <sentence>, and <paragraph> are indicated by numbers in the 10000s, 1000s, and 100s, respectively.
The head of <sentence> and <paragraph> can be identified. Furthermore, in the reading file, the volume attribute information is
Since the volume is represented by the default volume percentage such as Voll = ***, the document processing apparatus 1 converts the percentage information into absolute value information based on the attribute information and obtains the absolute value information.

In step F103 shown in FIG. 16, the document processing apparatus 1 performs processing using such a file for reading out, so that the speech synthesis engine can read out the file for reading out. It will be converted to a format.

Next, the operation using the user interface in step F104 of FIG. 16 will be described. The document processing apparatus 1 activates the speech synthesis engine when the user operates, for example, a mouse or the like of the input unit 20 and clicks on the read-aloud button 306c shown in FIG. Then, the document processing apparatus 1 displays a reading window 401 for a user interface on the display unit 30 as shown in FIG.

As shown in the figure, the read-out window 401 has a play button 42 for reading out a document.
0 and a stop button 421 for stopping reading out
And pause button 4 to pause reading aloud
22. In addition, the reading window 401
Are a cue button 411, a fast rewind button 412, and a fast forward button 413 for cueing, fast rewinding, and fast forwarding in sentence units, a cueing button 414 for cueing, fast rewinding, and fast forwarding in paragraph units, and a fast rewinding button. A button 415 and a fast forward button 416, and a cue button 417 and a fast reverse button 4 for cueing, fast rewinding, and fast forwarding in phrase units
18 and a fast forward button 419.

Further, a reading window 401 is provided with a selection switch 4 for selecting whether to read a whole sentence or a summary sentence created as described above.
23,424. Further, an image display area 403 for displaying an image at the time of reading is set, and an image of a person who is reading is displayed, and a telop display area 402 is displayed.
Is provided so that characters can be displayed in a telop corresponding to the reading voice.

Although not shown here, for example, it has buttons for increasing / decreasing voice, buttons for increasing / decreasing reading speed, buttons for changing voices such as male / female voices, and the like. Is also good.

The document processing apparatus 1 performs a reading operation by the speech synthesis engine in response to the user clicking / selecting these various buttons / switches by operating the mouse or the like of the input unit 20, for example. For example, the document processing device 1
Starts reading the document by clicking the play button 420 by the user. Specifically, the control unit 1
1 supplies the audio signal generated by the audio synthesis processing to the audio output unit 33 and outputs it as audio. In addition, the document processing apparatus 1 stops or pauses the reading process when the stop button 421 and the pause button 422 are clicked.

When the user presses the cue button 411 during the reading, the user jumps to the head of the currently read sentence and reads the sentence again. Cue button 4
In the same manner, for 14 and 417, the program jumps to the beginning of the currently read paragraph or phrase, and is read again. The control unit 11 controls the cue buttons 411, 414,
Regarding the operation 417, the jump destination is recognized by the above-mentioned marking. In other words, when the cue button 411 relating to a sentence is operated, the control unit 11 traces the sentence back from the middle position of the sentence currently being read out.
Search for 000's mark. When a mark in the 1000's range is detected, reading out is resumed from there. In the case of paragraphs and phrases,
Reading is resumed by looking for a mark in the 10000s range. By such processing, for example, it is possible to respond to a request that a user wants to repeatedly reproduce a desired portion in a document.

The document processing apparatus 1 executes the processing in step F1 in FIG.
At 04, in response to the user performing an operation using the user interface in the reading window 401, the text-to-speech engine reads the document. In this manner, the document processing apparatus 1 can read out a desired document by the speech synthesis engine without any discomfort.

By the way, the document to be read may be the body of the document data, or may be a summary sentence generated as described later. The text or the summary text is selected by clicking the selection switches 423 and 424. In any case, the processing in steps F102 and F103 in FIG. The text-to-speech by the speech synthesis engine becomes possible.

In this example, the reading file is generated from the received or created tag file.
Instead of generating such a reading file, reading may be directly performed based on the tag file.
In this case, after receiving or creating the tag file, the document processing device 1 uses a speech synthesis engine and, based on tags indicating paragraphs, sentences, and phrases attached to the tag file,
A certain pause is provided at the beginning of a paragraph, sentence or phrase to read out. By doing so, the document processing device 1
Can be read directly based on the tag file without generating a voice reading file.

6. Summary Creation / Display Process Next, a process of creating a summary sentence for document data will be described. As described above, the user can browse the text of the document by selecting the document and opening the browse window 301 as shown in FIG. In the browsing window 301, the text and the summary of the selected document are displayed. However, at the point in time when a summary sentence has not been created, the summary display unit 304 is blank as shown in FIG.

When the user clicks on the summary creation button 306a in the browsing window 301, the document display unit 30
A summary sentence for the document displayed in 3 is created,
This is displayed on the summary display unit 304 as shown in FIG. In other words, the control unit 11 activates a digest creation engine (described later with reference to FIG. 37) composed of software or hardware, and in response to a digest creation operation of the user,
A summary sentence creation process as described below is performed, and after creation, control for displaying the summary sentence is performed. The process of creating an abstract from a document is performed based on the internal structure of the document by tagging.

The summary sentence is usually displayed in the summary display section 304.
Is generated according to the size of And a text display unit 303
And the area of the summary display section 304 is determined by the user
Can be changed by moving. That is, the summary sentence is displayed in the summary display unit 30 at the time when the summary creation is instructed.
4 is created with a size (document length) corresponding to the size of the document No. 4. For example, in FIG. 26, the summary display unit 304 is very narrow, but when the user moves the partition frame 312 to widen the summary display unit 304 and instructs the creation of a summary, for example, as shown in FIG. A summary sentence having a document length corresponding to the area size is created and displayed on the summary display unit 304 expanded as described above.

However, for example, as shown in FIG.
If the summary sentence is created according to the size when 04 is very narrow, the summary sentence often has insufficient content. In other words, since an abstract cannot be created with a sufficient document length, the sentence does not substantially function as an abstract. Therefore, in this example, the summary display unit 30
If the window size (the document length that can be displayed in the window) as 4 is equal to or less than a predetermined value, for example, if the number of lines on the screen as shown in FIG. A summary sentence having a document length exceeding the size of 304 is created, and a telop creation engine (to be described later with reference to FIG. 37) constituted by software or hardware converts the summary into a telop and displays it.

Hereinafter, the operation of creating and displaying a summary sentence of this example including such a telop display will be described.

[0164] When summary creation button 306a is clicked on browsing window 301, controller 11 (summary creation engine) starts the summary creation process of FIG. When the summary creation processing of FIG. 23 ends, the summary sentence display processing of FIG. 24 is subsequently executed, and in some cases, the telop display processing of FIG. 25 is performed during the summary sentence display processing.

In the digest creation processing of FIG. 23, first, in step F81, the control unit 11 performs active diffusion. In this example, the document is summarized by employing the central activity value obtained by the activity diffusion as the importance. That is, in a document given an internal structure by tagging, by performing active diffusion, a central activation value corresponding to the internal structure by tagging can be given to each element. The active diffusion process performed in step F81 is the same as the process described with reference to FIGS.
As described above, the active diffusion is a process of giving a high central activity value to an element related to an element having a high central activity value. That is, the active diffusion is an operation on the central activity value such that the central activity value is equal between the anaphor (co-reference) expression and its antecedent, and the central activity value attenuates otherwise. Since the central activity value is determined according to the internal structure by tagging, it can be used for analyzing a document in consideration of the internal structure by tagging.

Next, in step F82, the size wsy of the summary display unit 304 at that time is checked, and it is checked whether the size wsy of the summary display unit is equal to or larger than a predetermined size wsth set in advance. The predetermined size wsth is
Normally, the summary is set as a window size that can display a sufficient document length (number of characters). An appropriate value as the predetermined size wsth is determined appropriately in terms of system design.

First, a case where the summary display portion size wsy is equal to or larger than a predetermined size wsth will be described. This is a process when a summary sentence according to the summary display unit size wsy is created. In this case, the process of the control unit 11 proceeds to step F83,
Sets the size wsy (specifically, the maximum number of characters that can be displayed on the summary display unit 304) of the summary display unit 304 of the browsing window 301 currently displayed on the display unit 30 as a variable (maximum number of characters) ws. I do. The control unit 11
Is the summary string (an internal register that holds the summary sentence) s
Is initialized and the initial value s (0) is set to "". The control unit 11 records the maximum number of characters ws and the initial value s (0) of the character string s thus set in the RAM 14.

Next, the process proceeds to step F85, where the control unit 11
Sets the count value i of the counter that counts the sentence skeleton extraction process to “1”. Then, in step F86, the control unit 11 extracts the skeleton of the sentence having the i-th highest average central activity value from the sentence based on the count value i of the counter. The average central activity value is the average of the central activity values of the elements constituting one sentence. Control unit 11
Is, for example, a character string s (i-1) recorded in the RAM 14.
Is read out, and the character string of the skeleton of the extracted sentence is added to this character string s (i-1) to obtain S (i). Then, the control unit 11 converts the character string s (i) thus obtained into R
Record in AM14. In the first time, since the character string s (i-1) has the initial value s (0), the skeleton of the sentence extracted this time is stored in the RAM 14 as the character string S (i).
When the process of step F84 is performed thereafter, the skeleton of the extracted sentence is changed to the character string S (i), and the character string S (i) up to that point (that is, the character string S (i-
1)). At the same time, the control unit 11 determines in this step F84 a list L (i) of the elements not included in the skeleton of the sentence in the order of the central activation value.
And the list L (i) is recorded in the RAM 14.

That is, in this step F86, the summarization algorithm uses the result of activity diffusion to
The sentences are selected in descending order of the average central activity value, and the skeleton of the selected sentence is extracted. The skeleton of a sentence is composed of essential elements extracted from the sentence. Required elements can be the head of the element, the subject, the object (obj
ect), indirect object, owner (poss
An element having a relationship attribute of essor, cause, condition, or comparison, and an element directly included when the coordination structure is an essential element, constitute an essential element. Then, by connecting the essential elements of the sentence, a skeleton of the sentence is generated and added to the summary.

At step F87, the control section 11 determines whether or not the length of the character string s (i) is larger than the maximum number of characters ws of the summary display section 104 of the browsing window 301.
This step F87 is a determination process for creating a summary sentence according to the size of the summary display unit 304.

If the length of the character string s (i) has not reached the maximum number of characters ws, the control unit 11 proceeds to step F
Proceed to 88. In step F88, the control unit 11 determines in the document the central activity value of the element of the sentence having the (i + 1) -th highest average central activity value and the highest central activity value of the list L (i) created in step F86. Compare the central activity values of the elements. That is, the sentence having the next highest average central activity value after the sentence adopted as the summary in step F86 (that is, the sentence which is a candidate to be added to the next summary sentence) and the sentence adopted as the summary in step F86 The central activity values of elements excluded from the summary as not scaffolds are compared.

In the process of step F88, the part to be added next to the character string as the summary sentence is set in the immediately preceding step F88.
This is a process of determining whether to select from sentences not adopted as a skeleton in the sentence adopted in 86 or to select from other sentences.

In the sentence having the (i + 1) -th highest average central activity value, the list L is higher than the central activity value of the element.
If the highest central activity value in (i) is the higher central activity value, the portion to be added next to the character string as the summary sentence is used as the skeleton in the sentence immediately preceding step F86. Try to choose from those that have not been adopted. Therefore, the process of the control unit 11 proceeds to step F90, in which the element having the highest central activity value in the list L (i) is stored in the character string S (i) stored at that time
And a character string SS (i). At this time, the element added to the character string SS (i) is deleted from the list L (i). Then, in step F91, the character string S
It is determined whether S (i) is greater than the maximum number of characters ws. If not, the process returns to step F88.

In step F88, if the (i + 1) -th element having the highest central activity value is higher than the highest central activity value in the list L (i), the summary sentence In step F89, the part to be added next to the character string is selected from a sentence different from the sentence adopted in step F86 immediately before.
, The count value i is incremented, and the process returns to step F86. That is, in step F88, (i + 1)
For the sentence having the second highest average central activity value, the skeleton is extracted in step F86 and added to the character string S (i).

As described above, in step F87 or step F9, while adding the character string as a skeleton element or other element of the sentence in step F86 or step F90 on the basis of the element having the higher central activation value.
1, the character string S (i) or SS (i) is converted to the maximum number of characters ws
By comparing with, a character string close to the maximum number of characters ws but not exceeding the maximum number of characters ws is created.

For example, if the character string S (i) exceeds the maximum number of characters ws in step F87, the process of the control section 11 proceeds to step F92, and the character string S (i- 1) is a summary sentence. In other words, this means that the maximum number of characters ws has been exceeded by adding the skeleton of the sentence in step F86.
The character string S (i-1) before adding the skeleton is determined to be a character string that is close to the maximum number of characters ws but does not exceed the maximum number of characters ws, and is used as a summary sentence.

If the character string S (i) exceeds the maximum number of characters ws in step F87 when the character string S (i) is generated for the first time (i = 1) in step F86, Since the character string S (i-1) becomes the character string S (0) as the initial value set in step F83, a summary sentence could not be created substantially. This is because the size of the summary display unit 304 is too small. Therefore, the user enlarges the area of the summary display unit 304 on the screen, and then clicks the summary creation button 306a again to return to FIG. What is necessary is just to make a process start. However, in the case of this example, in step F82, the summary display unit 3
04 is determined to be too small. In such a case, as will be described later, the process is to generate a summary sentence having a document length longer than the document length that can be displayed in the size of the summary display unit 304. It rarely happens that a summary sentence cannot be created.

If the character string S (i) does not exceed the maximum number of characters ws in step F87, as described above, the control unit 1
The process of step 1 proceeds to step F88, in which the next portion to be added to the character string is determined. When the process proceeds to step F91 as described above, it is determined whether or not the character string SS (i) has exceeded the maximum number of characters ws. Where the string SS
If (i) exceeds the maximum number of characters ws, the process of the control unit 11 proceeds to step F93, and the character string S (i) before adding a certain element in the immediately preceding step F90 is used as a summary sentence. Become. In other words, this means that the maximum number of characters ws has been exceeded by adding the element in step F90. Therefore, the character string S (i) before the addition of the element is close to the maximum number of characters ws but is not the maximum number of characters ws. Is determined to be a character string that does not exceed, and is used as a summary sentence.

The above processing is described in steps F82 → F8.
The description has been made assuming that the process proceeds from 3 to F85. Therefore, FIG.
At the point of time when the process of step 3 is completed, a summary sentence having a document length suitable for the size of the summary display unit 304 has been created. The contents of the summary sentence include the skeleton of one or more sentences having a high average central activity value and elements having a high central activity value other than the skeleton. The summary sentence created in this way is stored in the RAM 14, and the entire summary sentence is fixedly displayed on the summary display unit 304 as shown in FIG.

That is, when the digest creation processing of FIG. 23 is completed, the process of the control unit 11 proceeds to the digest sentence display processing of FIG. In the summary sentence display processing, first, step F94
Then, it is confirmed whether or not the size wsy of the summary display unit 304 matches the maximum number of characters ws at that time. That is, the maximum number of characters ws used in the immediately preceding summary creation process
Is the size wsy of the summary display unit 304 in step F83.
It is determined whether or not is substituted. wsy = w
The case of s is a case where a summary sentence suitable for the size of the summary display unit 304 at that time is created by the above processing. Therefore, in this case, the process proceeds to step F96, and the generated summary text is displayed on the summary display unit 304. As a result, the entire summary sentence is fixedly displayed as shown in FIG. That is, when the user sets the size of the summary display unit 304 in advance and instructs the creation of a summary, the summary of the document length according to the size of the summary display unit 304 is fixed (that is, switched as telop or switched). (No movement).

When the user looks at the displayed summary and wants to see a more detailed summary, or wants to see a shorter summary, the browsing window 30 is displayed.
After increasing or decreasing the size (area) of the first summary display unit 304, the user may click the summary creation button 306a again. Then, as long as the size wsy of the summary display unit 304 is equal to or larger than the predetermined size wsth, a summary having a document length corresponding to the size of the summary display unit 304 at that time is created and displayed by the above-described processing of FIG. Will be done.

By the way, for example, when the user clicks on the summary creation button 306a in a state where the summary display section 304 has a very small size as shown in FIG. 26 and the summary creation processing of FIG. Step F82
In the case where it is determined that the size wsy of the summary display unit 304 is smaller than the predetermined size wsth, the following processing is performed.

In this case, the processing of the control unit 11 is performed in step F
Proceeding to 84, a preset fixed value wsfix is substituted as the maximum number of characters ws. The fixed value wsfix is a value corresponding to the number of characters set as a minimum required as a summary sentence, and an appropriate value is set in terms of system design. For example, wsfix = wsth may be set. Further, a fixed value wsfix = (wsfix−i) ×
V, the fixed value initial value wsfix-i is held, and the user can arbitrarily input the value of the coefficient V, so that the user can arbitrarily set the fixed value wsfix.

The control unit 11 initializes a summary character string (an internal register for holding a summary sentence) s, and
(0) = “” is set. The control unit 11 sets the maximum number of characters ws and the initial value s of the character string s set as described above.
(0) is recorded in the RAM 14.

Subsequently, the control unit 11 sets the variable i to 1 in step F85, and then proceeds to the processing after step F86.
Since this is as described above, repeated description will be avoided. That is, also in this case, a summary sentence having the number of characters closest to the maximum number of characters ws is created without exceeding the maximum number of characters ws. However, in this case, the maximum number of characters ws = w
sfix, the summary display unit 30 at that time is displayed.
Therefore, a summary sentence having a number of characters (document length) exceeding the number of characters that can be displayed on the summary display unit 304 is not generated.

When a summary sentence is created by the processing of steps F86 to F93, the control unit 11 executes a summary sentence display process of FIG. In this case, since the maximum number of characters ws is not the summary display portion size wsy, the process proceeds from step F94 to F95, and a process of displaying the generated summary text as a telop using the telop creation engine is performed.

FIG. 25 shows an example of the telop display processing in step F95 in detail. First, the control unit 11 (telop creation engine) sets a variable n = 1 in step F501. In step F502, the n-th sentence is extracted from the generated summary sentence. Therefore, first, the first sentence constituting the summary sentence is extracted.

Next, in step F503, it is determined whether or not the sentence length Ls (the number of words) of the extracted sentence exceeds the summary display portion size wsy. For example, assuming that the first sentence of one or a plurality of sentences forming the summary sentence is equal to or less than the number of words that can be displayed in the current summary display unit size wsy, the process of the control unit 11 proceeds to step F504. move on. Then, the telop display time T1 is calculated by multiplying the sentence length Ls by a predetermined coefficient k.

Subsequently, in step F505, the control section 11 starts counting of the internal timer, and displays the first sentence on the summary display section 304 in step F506. At this time, the sentence length Ls is the summary display section size wsy
Because of the following, the entire first sentence is displayed on the summary display unit 304 as a telop. During this display operation, the control unit 11 monitors the time count value in step F507, and proceeds to step F515 when T1 is counted by the timer. Then, Step F5
In step F516, it is determined whether the display of the last sentence of the summary sentence has been completed.
, And the process returns to step F502.
Therefore, the second sentence of the sentences constituting the summary sentence is extracted next.

Subsequently, the control unit 11 makes a determination in step F503 for the second sentence. Here, if the sentence length of the second sentence exceeds the summary display unit size wsy (that is, the entire sentence at a time) Is not possible), the process of the control unit 11 proceeds to step F508, and multiplies the sentence length Lwsy, which can be displayed on the summary display unit, by a predetermined coefficient k to calculate the telop display time T1. Subsequently, in step F509, the control unit 11 starts counting of the internal timer, and in step F510, causes the summary display unit 304 to display the second sentence as a telop. However, at this time, since the sentence length Ls exceeds the summary display portion size wsy, only the displayable head portion is displayed. That is, a portion corresponding to the sentence length Lwsy from the beginning of the second sentence is displayed. During this display operation, the control unit 11 monitors the time count value in step F511, and when the time T1 is counted by the timer, the control unit 11 proceeds to step F512.
Proceed to.

In step F512, the displayed sentence is scrolled by one character. And step F31
At 3, wait for T2 time. Then, it is determined whether or not up to the last character of the second minute has been displayed in step F514,
If the last character has not been reached yet, step F51
Return to 2 and scroll one character. The T2 time as the standby time is a time that defines the scroll timing, which is also a value set in the system design. For example, if the time is about 0.5 to 1 second, the sentence is scrolled at an appropriate speed. Will go.

As a result of the processing in steps F508 to F514, for a sentence that cannot be displayed on the summary display unit 304, the display is scrolled one character at a time every T2 after the top is first displayed for T1. It will be displayed until the last character of the sentence.

When the last character of the sentence is displayed by scrolling, the process of the control unit 11 proceeds from step F514 to F515, and determines whether or not the display of the last sentence of the summary sentence is completed. If not, step F51
In step F50, after incrementing the variable n in 6,
Return to 2. Therefore, the third sentence among the sentences constituting the summary sentence is extracted next. Then, for the third sentence, the telop display is executed by the processing of steps F504 to F507 or steps F508 to F514.

If it is determined in step F515 that the display of all the sentences constituting the summary sentence has been completed, the process proceeds to step F517, where the head of the summary sentence is displayed.
That is, a state is displayed in which the displayable top portion is displayed in the summary display portion size wsy. In step F518, a scroll bar 307 described later is displayed, and the telop display process ends.

The display operation realized by the above telop display processing is as follows. For example, it is assumed that the digest creation button 306a is clicked in the browsing window 301 in which the digest display unit 304 is very narrow as shown in FIG. 26, and a digest is created. At this time, a summary sentence longer than the document length that can be displayed on the summary display unit 304 is created by the processing of FIG. 23 described above.

For example, it is assumed that the following summary sentence is generated for the text displayed on the text display unit 303 in FIG. "[Nice aging] / 8 cancer metastasis,
Can be suppressed! ? Characterizing cancer is cell proliferation and metastasis. Human cells have "oncogenes" and "cancer suppressor genes." In the elderly, normal regulation is lost over the years and non-braking mutations accumulate in cells, increasing the proportion of cells and leading to more cancers. Metastasis does not occur simply by increasing the number of cancer cells. Cancer cells dissolve proteins and so on, making a path and entering blood vessels and lymph vessels. From the aspect of cancer diagnosis and treatment (nm2
The expectations of 3) are gathering. "

Such a summary sentence is composed of the following seven sentences as follows. In the above-described telop display processing of the present example, a telop is formed for each sentence. [Nice aging] / 8 cancer metastasis can be suppressed! ? Characterizing cancer is cell proliferation and metastasis. Human cells have "oncogenes" and "cancer suppressor genes." In the elderly, normal regulation is lost over the years and non-braking mutations accumulate in cells, increasing the proportion of cells and leading to more cancers. Metastasis does not occur simply by increasing the number of cancer cells. Cancer cells dissolve proteins and so on, making a path and entering blood vessels and lymph vessels. Expectations of (nm23) have been gathered from the aspect of cancer diagnosis and treatment.

By the processing in FIG. 25, first, the first sentence is extracted and displayed on the summary display unit 304 as shown in FIG. The telop display states of the summary display unit 304 after the state of FIG. 25 will be described with reference to FIGS.

The sentence of FIG. 28 satisfies the sentence length Ls ≦ wsy (the entire sentence can be displayed).
In the processing of 504 to F507, the display state of FIG. 29A is continued for T1 time calculated according to the sentence length. Subsequently, since the sentence length also becomes the sentence length Ls ≦ wsy, the display of FIGS. 29B and 29C is executed by the processing of steps F504 to F507 for each T1 time calculated according to the sentence length.

Since the next sentence has a sentence length Ls> wsy (the entire sentence cannot be displayed), step F50 in FIG.
8 to F514 are performed. That is, first, FIG.
After the head of the sentence is displayed for the calculated T1 time as shown in FIG. 29, every T2 time, as shown in FIG. 29 (e) → (f),
The display scrolls one character at a time. Thereafter, the scrolling proceeds in the same manner, and the display of the sentence is completed when the state of FIG. 29G is reached.

Next, the sentences are sequentially extracted, and since they satisfy the sentence length Ls ≦ wsy, in the processing of steps F504 to F507, respectively, the time T1 calculated according to the sentence length is shown in FIG. ) (I) and (j) are displayed.

As described above, each sentence is sequentially displayed as a telop. If the sentence is too long to be displayed, the portion that could not be displayed is displayed by scrolling. Then, when the telop display of all the sentences constituting the summary sentence is completed, the head of the summary sentence is displayed as shown in FIG. 30 and the scroll bar 307 is displayed as steps F517 and F518 in FIG. That is,
After the entire summary is presented to the user by the above telop display, a scroll bar 307 is provided so that the user can scroll the characters on the summary display unit 304 arbitrarily and check the content of the summary. It is to be.

The creation and display of the summary sentence of the present embodiment are executed as described above. This makes it possible to present a summary suitable for the user. First, the user displays the summary display 3
By setting the size of the summary 04 arbitrarily, the document length of the generated summary sentence can be set. In other words, when the user wants to view a detailed summary, he or she clicks the summary creation button 306a after widening the summary display section 304. On the other hand, when he wants to see a summary of simple content, he narrows the summary display section 304 and then clicks the summary creation button 306a. Just click. Thereby,
A summary sentence of the size desired by the user is generated, and is displayed instead of a telop.

If the user has made the summary display section 304 too narrow, a summary text of the minimum required document length is created regardless of the window size, and is displayed in a telop as described above. Will be done. Therefore, for example, even when the user has to make the summary display section 304 considerably narrower in order to make the text display section 303 wider, the user can view the summary text with appropriate contents. Thereby, the contents can be accurately recognized.

In the telop display processing of this example, the summary sentence is converted into a telop in units of sentences. However, since the display time (T1) of one sentence is calculated according to the sentence length, each sentence ( Telop) is an appropriate display time for the user to read. Furthermore, when the sentence is long, it is preferable that the user recognizes one sentence in order to execute the scroll display (one sentence follows because the subsequent part is displayed in a moving manner instead of a switching display). Is easy for users to recognize).

FIG. 2 shows an example of the telop display process.
Not only the example of 5, but also various processing examples are conceivable. For example, a telop may be formed not in units of sentences but in units of phrases, phrases, or the like, or display divisions may be changed according to the length of sentences or phrases. It is also conceivable to scroll the entire summary from the beginning to the end.
In any case, the entire summary may be presented to the user by switching display or moving display, or a combination thereof.

The setting of the display period, the switching timing, and the scroll timing of each telop can be variously considered. As described above, in this example, the text and the summary text can be read aloud, but the timing of the telop display performed by the telop creation engine can be set in accordance with the output timing of the read voice. For example, in the reading process as described above, a reading file is generated from a tag file, and the reading file is converted into a form suitable for a speech synthesis engine. Speaking is performed according to the speed at which the user feels, and paragraphs, sentences, and phrases are separated. Therefore, by setting such a paragraph, sentence, and phrase division, a telop division similar to the reading speed, and a display period, a telop display that is easy for the user to read can be realized. As a specific example of the method, as described above, the text-to-speech processing is performed by the speech synthesis engine (however, the output of the text-to-speech voice is not performed), and the CPU 13 refers to the text-to-speech timing to determine the telop display division and switching timing. Just set it. For example, during the period when a certain phrase is being read out by the speech synthesis engine (actually no voice is output), a character string as the phrase is displayed as a telop, and when the read-out voice advances to the next phrase, Is switched to the telop display of the next phrase. A configuration for realizing such interlocking between the speech synthesis engine and the telop creation engine will be described later with reference to FIG.

In the above example, the process of displaying a telop in the summary display in some cases has been described in connection with the summary creation process and its display process. It is also possible to execute without association. That is, if the processing described with reference to FIG. 25 is performed on the text or the summary sentence of certain document data, the telop display can be performed, and the telop display operation is performed independently according to the operation of the user. May be. Therefore, the telop display operation of the present example is not necessarily performed in the manner shown in FIGS.
The process is not only executed according to the size of the summary display unit 304 described in 4 but can be arbitrarily executed by the user regardless of, for example, the size of the summary display unit 304 or the summary creation processing. . Of course, the telop display can be performed together with the reading operation.

[0209] 7. Output Processing of Video File 7-1 Video File and Tag The above description is about the processing mainly for only the document data in the document processing apparatus 1 of the present embodiment, but in the document processing apparatus 1 of the present embodiment, Video data related to document data can be output in cooperation with or independently of such various processes, for example, display output of a text, display output of a summary sentence, output of a read-out voice, and the like. Hereinafter, various processes regarding video data will be described.

First, a data file as video data (hereinafter, referred to as a video file) and a tag structure added to the document data to associate the video file with the document data will be described. As mentioned above,
For example, when document data is taken into the document processing apparatus 1 from the communication unit 21 or the recording medium 32, a video file may be taken at the same time. A video file may be input separately from the document data. 4 and 12
As described above, the document data and the video file taken into the document processing apparatus 1 are stored in the RAM 14 or the HDD 3.
4 or the like. Then, the control unit 11 can display the text of the document, create and display the summary, display the text, etc. as described above by extracting the stored document data as necessary. In addition, a video file corresponding to the document data to be processed can be extracted and displayed and output.

FIG. 31 shows an image of document data and a video file stored in the document processing apparatus 1.
As document data, that is, tag files, various document data such as tag files TG1, TG2, TG3,... On the other hand, as video files, for example, video files V1, V2
··············· Various files are stored. The contents of the video file are various and are arbitrarily created by the server that supplies the document data and the video data. For example, the video file V1 includes moving image data VM1 to VM4 as a plurality of scenes. The video file V3 has five moving image data VM5
VM9 is the video file V5 is one moving image data VM
12 are included. The video file V2 includes a plurality of still image data VS1 to VS7. In the video file V4, a plurality of still image data VS8, VS9
And a plurality of moving image data VM10 and VM11. Of course, there are various other video files. The video data is, for example, MPEG data.

[0212] The moving image data is assumed to include audio data. However, for still image data,
Basically, audio data is not included, but audio data related to a still image may be added. The audio data referred to here is audio data recorded in correspondence with an image as a moving image or a still image, and is an audio data in which the contents of a text or a summary of related document data are read out. (This is not a read-out voice signal generated by the voice synthesis process as described above, and is not a voice signal having the same content as the read-out voice). In other words, audio data refers to audio data accompanying video data, and does not directly match document data.
For example, it refers to audio data included in normal video data, such as ambient sound, sound effect, and sound of announcement / description at the time of recording the recorded video. (However, depending on the information provider's intention, audio data having a content as if the document data was read out as it is could be added to video data corresponding to certain document data.)

Each video file is, for example, a time /
A time code TC as minutes / seconds / frames is attached for each frame constituting video data. For example, in the video file V1, as shown in the figure, “00:00:00:
00 ”to“ 00: 03: 39: 05 ”time code T
C is attached. The time code TC functions as an address of each part in the video file. For example, by specifying a time code, an arbitrary portion (arbitrary section) can be reproduced. For example, when reproducing the moving image data VM2 in the video file V1, the time code of the start frame and the end frame of the section as the moving image data VM2 may be specified.

As described above, each video file has 1
Alternatively, the document data (tag file) is composed of a plurality of moving image and still image data, and a video file name, each moving image data,
A tag for specifying still image data is attached. That is, as the tag file, related video data can be specified by tags in arbitrary units such as the entire document, paragraphs, sentences, phrases, and the like. The video data related to the document data can be determined and output based on the designated tag (referred to as a video tag).

In addition, the fact that video data related to document data can be determined by the video tag means that the relevant document data can be determined from the video data at that time. Further, since the time code is included in the video tag attached to the document data, a related document portion can be specified for each scene defined by the time code.

FIGS. 32 and 33 show examples of tag files with video tags. FIG. 32 shows an example in which a video tag is added to a tag file having the same contents as in FIG. 18 described above. In the case of FIG. 18, <document> to </ document> are added as tags defining the document range. ">", Indicating that video data related to the entire document data is included in the video file V3 shown in FIG.

Also, as shown in the portion as (Example 10), <heading><scene = “VM5” in = “00: 0”
0: 00: 00: 00 "out =" 00: 00: 16: 1
0 ″>... </ Scene></heading>, and the video data related to the heading of the document data is included in the video file V3 shown in FIG. It is indicated that the data is VM5, and that in = “00: 00: 00: 00” and o
By ut = “00: 00: 03: 12”, the start time (start frame) and end time (end frame) of the scene as the moving image data VM5 in the video file V3, that is, the time code is specified.

Further, as shown in the portion as (Example 11), <paragraph><scene = “VM9” in = “00: 0”
5:29:01 ”out =“ 00: 06: 41: 0 ”
0 ″>... </ Scene></paragraph>, and the tags shown in FIG.
It is shown that the video data related to the paragraph shown in (b) is the moving image data VM9 included in the video file V3 shown in FIG. Also, in = “0”
0: 05: 29: 01 "out =“ 00: 06: 4
1:00 "specifies the start time (start frame) and end time (end frame) of the scene as the moving image data VM9 in the video file V3, that is, the time code.

With such a tag structure, the control unit 11
Can determine video data related to the tag file, and can reproduce and output the data as necessary.

FIG. 33 shows an example in which a video tag is added to the tag file having the same contents as in FIG. 19 described above. In this example, one corresponding still image data is specified for each sentence in the document data.
First, <document video file = “V2”>... <
The tag “/ document>” indicates that video data related to this document is included in the video file V2. And each sentence is a tag <sentence>-</ sentence>,
Although the range is shown, as shown in (Example 12) to (Example 16), the tag for each sentence is <Sentence Scene = "VS1"> ... </ Sentence><Sentence Scene = "" VS2 "> ... </ sentence><sentence scene =" VS3 "> ... </ sentence><sentence scene =" VS4 "> ... </ sentence><sentence scene =" VS5 " ... </ Sentence>, that is, it indicates that the still image data VS1, VS2, VS3, VS4, and VS5 correspond to each sentence.

In the video file V2, each of the still image data VS1 to VS7 has a data name "VS1".
..Assuming that a management structure that can be designated as "VS7" is adopted, a video tag that designates such a scene only by a data name may be used. Similarly, it may be specified by a time code in a video tag. Conversely, when the playback section of the moving image data can be specified only by the data name, it is not always necessary to specify the time code as described above.

With the tag structure shown in FIGS. 32 and 33, the control section 11 can determine video data related to each sentence of the tag file, and can reproduce and output the video data as necessary.

The example of FIG. 32 is an example in which one piece of moving image data is associated with each of the units such as a headline and a paragraph of a document. FIG. In this example, association of video data with a tag file is possible in various variations. That is, if a video tag is added to a tag such as <** phrase> to </ ** phrase>, it is possible to associate certain video data in an arbitrary unit such as a phrase unit. If a video tag is added to tags <document> to </ document> indicating the range of the entire document, one video data can be associated with the entire document. In addition, it is naturally possible to associate a plurality of pieces of moving image data or combine still image data and moving image data in one tag file.

7-2 Video Output Processing According to the tag file as in the above example, the document processing apparatus 1 can output video data in association with the tag file. Hereinafter, output processing for video data will be described. At the time of outputting the video data, the display of the text or the summary sentence, the display of the telop, the reading out of the text data and the like can be simultaneously performed with respect to the document data. These operations will be described later in detail. Here, an operation when the document processing apparatus 1 outputs only video data will be described.

FIG. 35 shows the video output processing by the control unit 11. This process is started when the video button 306d is clicked while the viewing window 301 is displayed as shown in FIG. 26 described above, for example.
Needless to say, the video button 306d may be clicked while the summary is created and displayed as shown in FIG.

In response to clicking on video button 306d, control unit 11 firstly proceeds to step F6 in FIG.
01, the tag file of the document data displayed on the browsing window 301 is confirmed, and the tag file includes a video tag as shown in FIGS. 32 and 33, or as shown in FIGS. Check if the video tag does not exist.

If the browsing window 301 is open for a tag file having no video tag, it is determined that there is no video data related to the displayed document data. Finish the output process. That is, in this case, clicking the video button 306d is an invalid operation.

However, if the video data is included in the tag file of the document data displayed in the browsing window 301, the video data related to the document data is specified. Advances the process to step F602, and generates a video output file. The video output file is, for example, as shown in FIG.
(A) As shown in (b), a file is generated by extracting a video tag from a tag file. FIG. 34A shows a video output file generated from the tag file shown in FIG. 32. In this case, the above-described video tag is extracted,
As shown in the figure, the file content specifies the moving image data VM5 to VM9 in the video file V3. In addition,
FIGS. 32 (a) and 32 (b) show only portions where the moving image data VM5 and VM9 are specified in the tag file. However, the example of FIG. 34 (a) shows the respective document portions shown in FIGS. 32 (a) and 32 (b). This is an example in which three paragraphs exist between the moving image data and the moving image data VM6, VM7, and VM8 are specified for each paragraph.

FIG. 34B shows a video output file generated from the tag file shown in FIG. 33. In this case,
As shown in the figure, the file content specifies the still image data VS1 to VS5 in the video file V2.

[0230] Such a video output file is a file for executing a video playback operation by the video engine in the document processing apparatus 1. The video engine may be configured by hardware or realized by software. If the video engine is implemented by software, the application program
It is stored in the ROM 15 or the HDD 34 in advance. The functional configuration including the video engine will be described later with reference to FIG.

FIG. 34 shows an example of the structure of the video output file, and the video output file is not actually limited to such a structure. In the processing of FIG. 35 and the video output processing in the case where the document data is output together with the output of the document data, which will be described later, the video data is output based on the video output file. Reproduction of video data may be performed based on a file.

Subsequently, the control unit 11 determines in step F603 that
Start the video engine. In step F604, the control unit 11 opens a video window 501 on the display unit 30, for example, as shown in FIG. This video window 5
01, a file name display section 502 for displaying a file name of the document data file, and a video display section 50 for displaying video data related to the document data having the file name.
3. A video bar 50 indicating the position of the image displayed on the video display unit 503 (the position where the reproduction of the moving image data proceeds).
4. a start position display section 521 for displaying a time code as a start position of a displayed image; a current position display section 522 for displaying a time code as a current reproduction position (current position) of the displayed image; An end position display section 523 that displays a time code as an end position of the displayed video is displayed. Further, in order for the user to instruct an image reproducing operation, a reproducing button 50 is provided.
5, a pause button 506, a stop button 507, a fast-reverse button 508, a fast-forward button 509, a cue button 510 in a return direction, a cue button 511 in a feed direction, and the like are displayed.

When such a video window 501 is opened, the control section 11 performs a process corresponding to the operation of the user interface as step F605.
For example, according to the video output file as shown in FIG. 34A, the reproduction of the moving image data VM5 of the video file V3 shown in FIG. 31 is designated, but the reproduction button 505 is clicked in the video window 501. In this case, the control unit 11 sends the moving image data V
The reproduction of M5 is started. Thereby, the video display unit 50
3 displays an image of the moving image data VM5. At this time, the audio data included in the moving image data VM5 is output as audio from the audio output unit 33. FIG.
4 (a), when the reproduction of the moving image data VM5 is completed, the moving image data VM6
Is output. Hereinafter, the moving image data VM7, VM
8. The reproduction of the VM 9 is performed.

If the pause button 506 is clicked during the reproduction of the moving image data VM5, for example,
Suspends reproduction of the moving image data VM5 by the video engine. When the stop button 507 is clicked, the control unit 11 stops the reproduction of the moving image data VM5 by the video engine. When the fast reverse button 508 or the fast forward button 509 is clicked, the control unit 11 causes the video engine to perform fast reverse or fast forward of the reproduction operation of the moving image data VM5. When the cue button 510 in the return direction or the cue button 511 in the feed direction is clicked,
The control unit 11 determines, for example, the beginning of the preceding or succeeding scene (or a specific point to be a point) or another moving image data (VM6)
Etc.) to the beginning (or forward), and playback from that location is executed. In addition, the user can change the video bar 504
When the display portion of the current position is dragged, the video engine VM5 is moved in the backward or forward direction of the playback position of the moving image data VM5 accordingly.

When the video data to be displayed is still image data, for example, based on a video output file as shown in FIG. Since the operation is substantially meaningless, it may not be displayed. In that case, the display of the still image data is started by clicking the play button 505, and when a plurality of still image data is specified by the tag file,
Still image data may be switched in response to clicking on the cue button 510 in the return direction or the cue button 511 in the feed direction.

By the processing according to the user interface operation, the user can view the video data in an arbitrary state. Then, by displaying the video data related to the document data displayed in the browsing window 301 in the video window 501 in this manner, the user can see the video data related to the document content, and the document content (information Content) can be known more accurately and in detail.

[0237] 8. Functional Block Configuration of Document Processing Device Various operations realized in the document processing device 1 have been described above. To be more specific, they are creation / display of an abstract for document data, display of a telop for a text or an abstract, reading out of a text or an abstract, and output of video data. In order to enable these operations and an interlocked output operation of video data and document data, which will be described later, a function formed as a software configuration (or a hardware configuration) and a file group configuration in the control unit 11, for example. The block is shown in FIG. FIG. 3
Reference numeral 7 denotes only a portion related to an output operation from the display unit 30 or the audio output unit 33, and functional blocks for implementing, for example, a document reception process and a classification process are omitted.

In this case, the above-described speech synthesis engine 601, telop creation engine 602, summary creation engine 603, and video engine 604 are provided as shown. In addition, a video-linked document creation engine 620 is provided to perform a linked output operation of video data and document data described later. An audio control unit 605 is provided as an audio signal output processing unit for the audio output unit 33, and a display control unit 606 is provided as an image signal output processing unit for the display unit 30. Further, a part that performs processing for a user operation (an operation by clicking a mouse or the like as the input unit 20) on a button on various displayed windows is provided as the user interface 607.
Then, a controller 600 for controlling these functional blocks is formed.

The files include the reading file 608, the body tag file 609, the summary tag file 610, the video file 611, and the video output file 612 described above. As described above, the body tag file 609 and the video file 611 are taken from the communication unit 21 or the recording / reproducing unit 31 (recording medium 32). In addition, the summary sentence tag file 61
0 is generated from the body tag file 609 by the summary creation engine 603. Speech file 608
Is generated by converting the body text tag file 609 or the summary sentence tag file 610. The video output file 612 is generated by converting (specifically, extracting a video tag) from the text tag file 609 or the summary sentence tag file 610.

Further, for a linked output operation of video data and document data, which will be described later, a video linked output document file 621 is generated by the video linked document creation engine 620.
Is created. This video linked output document file 62
Reference numeral 1 denotes a file in which the text tag file 609 or the summary tag file 610 is converted and used as a fixed display or a telop display of a document, or as a file for reading out and outputting.

The following describes each operation described above and the processing of the functional blocks in FIG. 37 for realizing the linked output operation of video data and document data, which will be described later.

Text-to-Speech Processing or Text-to-Summarized Sentence In the text-to-speech processing described with reference to FIGS.
10, a reading-out file 608 is generated. Then, the speech synthesis engine 601 refers to the reading file 608 based on the instruction of the controller 600 and performs a speech synthesis process based on the reading file 608. The output level of the generated synthesized voice signal (speech voice signal) Yout is adjusted by the voice control unit 605, and the like.
It is supplied to the audio output unit 33 and output. Further, the controller sends a reading window 401 from the display control unit 606.
Is output and displayed on the display unit 30. Further, information on the user operation on the reading window 401 is taken in by the user interface 607 and transmitted to the controller 600, and is transmitted to the controller 600.
0 controls the operation of the speech synthesis engine 601 according to a user operation.

Summary Creation Processing In the summary creation processing described with reference to FIG. 23, the controller 600 instructs the summary creation engine 603 to execute summary creation for the body tag file 609. As a result, a summary tag file is formed. By transmitting the size information of the summary display unit 304 to the summary creation engine 603, the controller 600 performs the summary generation process according to the size of the summary display unit 304 as described above.

Summarized Fixed Display Processing As described with reference to FIGS. 23, 24 and 27, when the size of the summarized display unit 304 is equal to or larger than the predetermined value wsth,
Summary sentence tag file 610 of the document length according to the size
Is generated and it is fixedly displayed. In this case, the generated summary sentence tag file 610 is processed as a display document output Sout by the summary creation engine 603 and supplied to the display control unit 606. The display control unit 606 combines the image with the image of the browsing window 300 and displays the image on the display unit 30 as shown in FIG.

23. Telop Display Processing According to Summary Display Size As described with reference to FIGS. 23 to 30, when the size of the summary display unit 304 is smaller than the predetermined value wsth, a summary sentence having a document length exceeding that size is used. A tag file 610 is generated, and it is displayed as a telop. in this case,
The generated summary sentence tag file 610 is subjected to telop processing by the telop creation engine 602. Then, it is sequentially output as a telop display document output Tout. The display control unit 606 combines the telop display document output Tout with the image of the browsing window 300, and causes the display unit 30 to perform the display as shown in FIGS.

Summary Text / Text Subtitle Display Processing As described above, regardless of the size of the summary display unit 304, telop display of a text or a summary text can also be performed. In that case, the body tag file 609
Alternatively, the telop creation engine 602 performs telop conversion processing on the summary sentence tag file 610. Then, it is sequentially output as a telop display document output Tout. The display control unit 606 combines the telop display document output Tout with the image of the browsing window 300 and other required windows, and causes the display unit 30 to execute the telop display.

-Telop Display Processing at Reading Timing As a modification of the telop display, it has been described that the telop progress timing can be adjusted to the reading voice timing. In this case, the reading-out file 608 converted from the summary sentence tag file 610 is used as a shared file of the speech synthesis engine 601 and the telop creation engine 602. (A signal system indicated by a broken line in the figure is formed.) Under the control of the controller 600, the speech synthesis engine 601 performs speech synthesis / speech output using the speech file 608. However, in this case, the voice control unit 60
Numeral 5 sets the output level of the generated synthesized voice signal Yout to zero so that no read-out voice is output to the user. Telop creation engine 60
2 uses the reading file 608 to create a telop. At this time, the telop creation engine 602 and the speech synthesis engine 601 mutually exchange timing signals. In other words, the telop creation engine 602 executes telop creation / telop display document output Tout based on the timing signal from the reading-out file 608. As a result, the telop display based on the reading speed is executed. Note that the reading speed (that is, the telop speed in this case) is set in the user interface 607.
When the controller 600 changes the settings in the speech synthesis engine 601 based on the information from, the change can be made.

Video Output Processing The video output processing described with reference to FIGS. 35 and 36 is performed by the processing of the video engine 604 in accordance with an instruction from the computer 600 based on information from the user interface 607. The video engine 604 refers to the video output file 612 generated from the text tag file 609 or the summary sentence tag file 610, determines video data to be reproduced, and reads the video file 611. The read video data is transmitted to the video engine 6
The video signal Vout is processed as an output video signal Vout and supplied to the display control unit 606. Then, the display control unit 606 combines the image with the image of the video window 501,
It is displayed on the display unit 30 as shown in FIG. The audio data included in the video data is also processed as an output audio signal Aout by the video engine 604 and supplied to the audio control unit 605 to perform processing such as level adjustment. Then, the sound is output from the sound output unit 33 as reproduced sound. During video output, information on the user operation on the video window 501 is captured by the user interface 607 and transmitted to the controller 600, and the controller 600 controls the operation of the video engine 601 according to the user operation.

Fixed display of document data linked to video data As described later with reference to FIG. 40, in this example, a text or a summary as document data can be fixedly displayed and output in synchronization with the output of video data. This is a process for fixedly displaying a corresponding portion in the document data for each scene output as video data. For this purpose, the document portion must be specified corresponding to the position of the video data, that is, the section defined by the time code. For this reason, the video-linked document creation engine 620
Video-linked output document file 621 in which a document portion corresponding to the time code of the video data to be output is specified
Generate This can be generated based on the tag file to which the video tag is attached as described above. In the case of this linked output, the video engine 604 refers to the video output file 612 generated from the body tag file 609 or the summary sentence tag file 610, determines video data to be reproduced, and determines the video file 611.
Is read. The read video data is processed by the video engine 604 as an output video signal Vout, and supplied to the display control unit 606. Then, the image is combined with the image of the video window 501 by the display control unit 606, and is displayed on the display unit 30 as shown in FIG. The audio data included in the video data is also processed as an output audio signal Aout by the video engine 604 and supplied to the audio control unit 605 to perform processing such as level adjustment. Then, the sound is output from the sound output unit 33 as reproduced sound. During video output, information on the user operation on the video window 501 is captured by the user interface 607 and transmitted to the controller 600.
The operation of the video engine 601 is controlled according to a user operation. When such video data is output, the video engine 604 sends the video linked document creation engine 620
Is supplied with a time code TC. The video-linked document creation engine 620 checks the output timing (time code) specified in the video-linked output document file 621 while monitoring the time code TC, and displays a document portion corresponding to the time code TC on the display document output Sout.
Is supplied to the display control unit 606. The display document output Sout is sent to the display control unit 606 in the video window 5.
01 and are displayed on the display unit 30 as shown in FIG. Note that, in response to a user operation on the video window 501, video data is temporarily stopped,
Although fast rewind and cueing are performed, the controller 600 also controls the video-linked document creation engine 620 based on the operation information, so that the video-linked document creation engine 620 pauses video data, In accordance with a situation such as fast forward, fast rewind, or cueing, a corresponding part of the display document output Sout is performed.

-Document data telop display linked with video data This will also be described later with reference to FIG. 40. In this example, a text or summary sentence as document data can be displayed in a telop display in synchronization with the output of video data. . This is a process of displaying a corresponding portion in the document data by telop for each scene output as video data. Also in this case, since the document portion must be specified corresponding to the section defined by the time code of the video data, the video-linked document creation engine 620 outputs the document corresponding to the time code of the output video data. A video-linked output document file 621 whose part is specified is generated. Although a duplicate description is avoided, in the case of this linked output, the video engine 604 refers to the video output file 612 to perform the output processing of the video data similarly to the case of the document data fixed display linked with the video data. Do. At the time of outputting such video data, the time code TC is supplied from the video engine 604 to the telop creation engine 602. The telop creation engine 602 checks the output timing (time code) specified in the video-linked output document file 621 while monitoring the time code TC, and displays a document portion corresponding to the time code TC as a telop display document output Tout. It is supplied to the control unit 606. The display control unit 606 outputs the telop display document output Tout to the video window 5.
01, and is displayed on the display unit 30. Further, when video data is temporarily stopped, fast-forwarded, fast-returned, or cueed in response to a user operation on the video window 501, the controller 600 also controls the telop creation engine 602 based on the operation information. By doing so, the telop creation engine 602 outputs the telop display document output T of the corresponding part according to the situation such as pause, fast forward, fast reverse, or cueing of the video data.
Perform out.

Reading of Document Data Linked to Video Data This will be described with reference to FIG. 44. In this example, a text or a summary as document data can be read and output in synchronization with the output of video data. This is a process of reading out a corresponding portion in the document data for each scene output as video data. Again,
Since the document part must be specified corresponding to the section defined by the time code of the video data, the video-linked document creation engine 620 specifies the document part corresponding to the time code of the output video data. Then, a document file 621 for video linked output is generated. Although a duplicate description is avoided, in the case of this linked output, the video engine 604 also performs the video output file 612 similarly to the case of the document data fixed display linked with the video data.
To output video data. When outputting such video data, the video engine 60
4 to the speech synthesis engine 601 with the time code T
C is supplied. The voice synthesis engine 601 checks the output timing (time code) specified in the video-linked output document file 621 while monitoring the time code TC, and outputs a synthesized voice signal (speech voice) for the document portion corresponding to the time code TC. Signal) Yout is supplied to the voice control unit 605. In this case, the audio control unit 605 mutes the output of the audio data accompanying the video data. Alternatively, the volume level is set to be very low as compared with the reading voice. Further, when video data is temporarily stopped, fast-forwarded, fast-returned, or cueed in response to a user operation on the video window 501, the controller 600 also controls the speech synthesis engine 601 based on the operation information. By doing so, the speech synthesis engine 601 outputs the read-out speech signal Yout of the corresponding part according to the situation such as pause, fast forward, fast reverse, or cueing of the video data.

As described above, the summary creation / display of the document data, the telop display of the text or the summary, the reading of the text or the summary, the output of the video data, and the output of the document linked with the video data (Fixed display, telop display, reading out).

The functional blocks shown in FIG. 37 are merely examples, and the configuration and operation of the functional blocks for realizing each of the above operations are not necessarily limited to such examples.

[0254] 9. Video and Linked Document Display Output Processing An example of a case in which document data is fixedly displayed or displayed in telop in conjunction with video data will be described below.

Now, an example will be given in which a certain tag file TG1 as document data has a configuration shown in FIG. In this case, it is assumed that the tag file TG1 is formed by, for example, five paragraphs and each of the document portions TX1 to TX5. And for each document part,
As described with reference to FIG. 32, a video tag is attached, and as shown in FIG. 38A, the moving image data VM
It is assumed that 5 to VM9 (see FIG. 31) are associated with each other. In the case of such a tag file structure, each paragraph TX1
TX5 is a document portion of the interlocking unit for the video data. (Hereinafter, the linked unit document TX1, the linked unit document T
Note that the linked unit document is not necessarily a paragraph unit of the document data, but is defined by the addition status of the video tag in the tag file. May be a linked unit document.

On the other hand, the video file V3 specified by the video tag of the tag file TG1 includes the moving image data VM5
To VM9 are configured as shown in FIG. For example, as the time code TC in the video file V3, “00: 00: 00: 00” to “00: 0”
The range of “0:16:10” is the moving image data VM5 forming one scene, and “00: 00: 16: 11”
The range from “00: 02: 20: 01” is the moving image data VM6 forming one scene.

In the case where video data is output based on such a tag file TG1 and document data is output in conjunction therewith, the output proceeds as shown in FIG. 39 in terms of time. In the case of this interlocking operation, the output of video data is mainly performed, and therefore, as shown in FIG.
5, the reproduction of VM6, VM7, VM8, and VM9 proceeds sequentially. As can be seen from the figure, during the reproduction of the moving image data VM5, that is, as the time code TC, “00:
During the period from “00:00:00” to “00: 00: 16: 10”, the portion of the linked document unit TX1 is output in fixed display or telop display. Then, from the timing when the time code TC becomes “00: 00: 16: 11”, that is, from the time when the output of the moving image data VM6 starts, “0”
0: 02: 20: 01 ”, the linked document unit T
X2 is displayed and output. Similarly, the linked document units TX3, TX4, and TX5 are displayed and output in the playback periods of the moving image data VM7, VM8, and VM9, respectively. In the case of the reading output described later, the linked document units TX1 to TX5 are read and output in the same period.

In FIG. 39, moving image data VM5 to VM
9 shows a case where the playback of the normal playback progresses normally. However, if the user performs an operation such as fast forward or cueing in the middle,
As the video data advances / retreats accordingly,
The output linked document unit is also switched accordingly. For example, if the cueing operation of the moving image data VM6 is performed during the output of the moving image data VM8 and the linked document unit TX4, the video data output is switched to the output of the moving image data VM6, and the document data output is changed to the linked document unit TX2. Output.

FIG. 40 shows a processing example of the control unit 11 when the document data is fixedly displayed or displayed as a telop in association with the video data.

For example, when certain document data is displayed in the browsing window 301 as shown in FIG. 26, when the user clicks the video button 306d, the output of the video data and the output of the document data linked therewith are performed. Shall be started. Note that, in the case of outputting only the video data described with reference to FIG. 35, the operation is started by such an operation. However, actually, the video button 306d for starting the output of only the video data in the browsing window 301 is displayed. In addition, a video button for starting the linked output described below may be displayed so that the operation can be distinguished. Alternatively, when the video button 306d is shared, whether to output video data alone or to perform linked output may be defined by mode setting or the like. There are various user operation modes (operation modes set in the document processing apparatus 1) for video data independent output, video data and linked output of document data display, and further, linked output of reading of video data and document data described later. Thought,
What is necessary is just to determine in consideration of the function which can be realized, the convenience on design, the operability, etc.

For example, when linked output of video data / document data is started when certain document data is designated as the document data displayed in the browsing window 301, the control unit 11 first sets the state shown in FIG. In step F701, the presence / absence of a video tag for the designated document data is confirmed. If the video tag does not exist, the video file related to the document data does not exist, and the process ends.

When a video tag exists and, for example, the output of a video file related to the document data displayed in the browsing window 301 and the linked output of the document are executed,
The process of the control unit 11 proceeds to step F702, and generates a video output file 612 as a file for executing the video playback operation by the video engine 604.
This is similar to step F602 in FIG.
As shown in FIGS. 4 (a) and 4 (b), a process for extracting a video tag from a tag file (text tag file 609) as document data and generating a video output file is performed.

Subsequently, the control unit 11 determines in step F703 that
The video linked document creation engine 620 is started. Then, in step F704, the video-linked document creation engine 620
Causes the video-linked output document file 621 to be generated. For example, FIG. 41 shows an example of a document file 621 for video-linked output generated from a tag file to which a video tag is attached as shown in FIG. 32 (text tag file 609). That is, the document file 621 for video-linked output as in this example.
Has a file structure in which a time code specified by a video tag and a corresponding document (linked unit documents TX1 to TX5 in FIG. 38) are paired from a body tag file. For example, in the portion shown in FIG. 41A, “0” specified by tags TCin and TCout specifying a time code is used.
0: 00: 00: 00 "to" 00: 00: 16: 10 "
, That is, the moving image data VM5 and the linked document unit TX1 of “[nicely aging] / 8 cancer transfer, suppressed !?” FIG. 41
In the part shown in (b), the time code “00: 05: 2”
9:01 "to" 00: 06: 41: 00 ", that is, the moving image data VM9 and" this metastasis, cancer cells ...
・ It is being elucidated in recent years. Linked document unit TX5
Are shown to correspond. The structure of the video-linked output document file 621 is not limited to such an example, and may be any content that indicates the relationship between the time code and the linked document unit.

Subsequently, in step F705, the control unit 11 branches the process depending on whether the display output of the document data is the fixed display output or the telop display output. Whether to use the fixed display or the telop display may be automatically set by the control unit 11, or may be selectable by the user. In the case of automatic setting, for example, the setting may be made based on the relationship between the size of the document display area 530 for displaying a document in the video window 501 in FIG. 42 and the document length of the displayed document (interlocking unit document).
Alternatively, the fixed display and the telop display may be determined by the user selecting in advance. Further, an operation button for linking video data and fixed display in the browsing window 301 and the like and an operation button for linking video data and telop display are separately provided so that the user can determine which button is clicked. Good. To execute ticker display,
The control unit 11 activates the telop creation engine 602 in step F706.

The control unit 11 determines in step F707 that
Activate the video engine 604, and then proceed to step F70
In step 8, a video window 501 as shown in FIG. Video window 5 in this case
01 has a document display unit 530 added to the various displays shown in the video window 501 of FIG.

Then, the control unit 11 controls the video window 5
After opening the file 01, in step F709, the video output by the video engine 604 is started. As a result, a reproduced image is displayed on the video display unit 503 as shown in FIG. When linking the fixed display, the control unit 11 executes a time code check by the video-linked document creation engine 620 in step F710, and executes a document output process according to the time code of the video data in steps F711 and F712. Let the linked document creation engine 620 execute it. As a result, as shown in FIG. 42, a document in a linked document unit corresponding to the scene currently displayed is displayed on the document display unit 530. Then, as the processing of steps F710, F711, and F712 is performed successively, as described with reference to FIG. 39, the document portion to be displayed is displayed at the required time code timing in accordance with the progress of the video data. , For each linked document unit.

When executing the telop display, the control section 11 causes the telop creation engine 602 to execute the processing of steps F710, F711 and F712. Then, a telop display for each linked document unit is executed in the document display area 530. In this telop display,
Document display area 5 capable of displaying a plurality of lines as shown in FIG.
Alternatively, for example, as shown in FIG. 43, the document display area 530 may be set to a state in which one line of document can be displayed, and the telop display may be executed there.

If the output of the video data based on the video file 611 has been completed, or if the reproduction of the video data has been completed in response to the operation of the user, the control section 11 proceeds from step F713 to F714 and fixes it. The linked document display by display or telop display is also terminated, and a series of processing is completed. Through such output processing, the user can view video data related to certain document data, and also view the display of a corresponding portion (in linked document units) in the document data in accordance with the scene of the video data. Can be.

Although not shown in detail in the flowchart of FIG. 40, during the processing of steps F710 to F712, operations such as fast-forwarding and cueing of video data are performed, and the reproduction position of the video data changes. When the corresponding linked document unit changes, the displayed linked document unit is switched as described above.

10. Next, an example of a case where the document data is read aloud in synchronization with the video data will be described. In this case, as described with reference to FIG. 39, the output of the video data is mainly performed, and the video data VM
5, the corresponding linked document units TX1, TX2, TX3, TX4, TX at timings corresponding to the sequential progress of the reproduction of VM6, VM7, VM8, and VM9.
5 is read aloud.

FIG. 44 shows a processing example of the control section 11 when reading out document data in conjunction with video data. For example, when certain document data is designated as the document data displayed in the browsing window 301,
When the linked output of the video data / document data is started, the control unit 11 first determines in step F801 of FIG.
Check whether there is a video tag for the specified document data. If the video tag does not exist, the video file related to the document data does not exist, and the process ends.

When a video tag exists and, for example, the output of a video file related to the document data displayed in the browsing window 301 and the linked output of the document are executed,
The process of the control unit 11 proceeds to step F802, and generates a video output file 612 as a file for performing the video playback operation by the video engine 604.
This is similar to step F602 in FIG.
As shown in FIGS. 4 (a) and 4 (b), a process for extracting a video tag from a tag file (text tag file 609) as document data and generating a video output file is performed.

Subsequently, the control unit 11 determines in step F803 that
The video linked document creation engine 620 is started. Then, in step F804, the video-linked document creation engine 620
Causes the video-linked output document file 621 to be generated. In this case, the video-linked output document file 621
Is a file in which the relationship with the time code is specified in the reading file 608 described above. For example, FIG.
FIG. 45 shows an example of a video-linked output document file 621 generated from a tag file (text tag file 609) to which a video tag is attached as in FIG. That is, as in this example, the document file 621 for video-linked output includes tags TCin, TCout specifying the time code as in FIG.
38, the time code specified by the video tag and the corresponding document (interlocking unit documents TX1 to TX in FIG. 38)
5) is paired. Then, a file structure is added with information for reading as described with reference to FIG. Of course, also in the case of reading out, the structure of the video linked output document file 621 is not limited to such an example, but shows the relationship between the time code and the linked document unit, and further adds information for reading out. Content.

Subsequently, in step F805, the control section 11 activates the speech synthesis engine 601. Then, the control unit 11 activates the video engine 604 in step F806, and subsequently opens the video window 501 on the display unit 30 in step F807. In this case, the video window 501 does not need the document display area 530 as shown in FIG. 42, and may be, for example, the video window 501 as shown in FIG.

Then, the control section 11 controls the video window 5
After opening the file 01, the video output by the video engine 604 is started in step F808. As a result, a reproduced image is displayed on the video display unit 503 as shown in FIG. In step F809, the control unit 11 causes the voice synthesis engine 601 to execute a time code check, and in steps F810 and F811, causes the voice synthesis engine 601 to execute a process of outputting a read-out voice Yout according to the time code of the video data. As a result, a document in a linked document unit corresponding to the scene currently displayed is read out. Then, as the processing of steps F809, F810, and F811 is performed successively, as described with reference to FIG. 39, the reading of each linked document unit is performed at the required time code timing in accordance with the progress of the video data. Is being executed.

If the output of the video data based on the video file 611 has been completed or if the reproduction of the video data has been completed in response to the operation of the user, the control section 11 proceeds from step F812 to F813 and reads out the speech. Output of the linked document is terminated, and a series of processing is completed. Through such output processing, the user can view video data related to certain document data, and read out a corresponding portion (linked document unit) in the document data according to the scene of the video data. You can hear

Although not shown in detail in the flowchart of FIG. 44, during the processing of steps F809 to F811, operations such as fast-forwarding and cueing of video data are performed, and the playback position of the video data changes. When the corresponding linked document unit changes, the linked document unit to be read out is switched as described above.

[0278] 11. Modification Example The document processing apparatus 1 of the present embodiment
In the above, the linked output operation of the video data and the document data executed in the above has been described. However, various modifications of the processing operation for the linked output can be considered.

First, as described above, as the document data output to be linked with the video data, the fixed display, the telop display, and the reading aloud have been described. For example, the user may be allowed to make a selection while the video window 501 is open. For example, as the video window 501, as shown in FIG.
5 is displayed, and when outputting video data, it is possible to select whether to use only video data, to link document display (fixed or telop), or to read out a document.
Then, the user selects a check box 52 of a desired option.
In response to clicking on 6, it is set which process is to be executed. For example, in the state of FIG.
Since the option for linking the document display with the video data output is selected, if the play button 505 is clicked in this state, the above-described processing in FIG. 40 is performed. When the play button 505 is clicked in a state where only the video data is selected, the processing of FIG. 35 is performed. Further, an option for linking the reading of the document with the output of the video data is selected, and the play button 505 is pressed. When clicked, the processing in FIG. 44 is performed.

Further, output relating to document data linked to video data may be executed in a complex manner. For example, when playing back video data, fixed display and reading out are simultaneously performed for each linked document unit corresponding to the video data,
It is conceivable that the telop display and the reading out are performed simultaneously.

Also, as shown in FIG. 39, each linked document unit which is document data output according to the video data playback section is extracted from the document data body in accordance with the specified time code section. However, for example, a summary sentence may be generated for each linked document, and the summary sentence may be displayed or read out.

When document data (linked document unit) is output together with video data, the output timing and display period of the document data in linked document unit are as follows.
Various considerations are possible depending on the time length of the corresponding portion of the video data and whether the image is a moving image or a still image. In the case of temporal progress such as telop display and reading, for example, the time length until the completion of the telop display or reading in the linked document unit and the time length until the completion of the corresponding video data part always match. Not something. For this reason, the telop display speed and the reading speed can be variably set in accordance with the progress status of the video data, or a part of the telop display or reading document can be omitted according to the switching of the video data content (scene). Conversely, it is also possible to temporarily stop the progress of the telop display and the reading out.

When reading out the document data together with the video data, the reading-out file 60 is used as the video-linked output document file 621 as described above.
A file corresponding to the file No. 8 is created, and the creation timing may be variously considered depending on the processing capability of the control unit 11 and the balance with other processes. That is, as in the example shown in FIG. 44, it may be created before the start of video data output, but may be created for each linked document unit at a required timing after the start of video data output according to the progress of the time code. Good. Of course, if possible, real-time processing may be performed according to the video data.

The hardware or software configuration of the document processing apparatus 1 according to the embodiment described above and processing examples can be variously considered. For example, there are various specific device examples that constitute the document processing apparatus 1. For example, if the input unit 20 in the document processing apparatus 1 is taken as an example,
Not only a keyboard and a mouse but also other devices such as a tablet, a light pen, a wireless commander device using infrared rays, and the like are conceivable.

[0285] In the embodiment, the description has been made assuming that a document or a video file is transmitted from the outside to the communication unit 22 via a telephone line or the like. However, the present invention is not limited to this. For example, it can be applied to a case where a document or a video file is transmitted via a satellite or the like.
The data may be read from the recording medium 32 in the recording / reproducing unit 31 or a document or a video file may be written in the ROM 15 in advance.

Further, in the embodiment, an example of a method of tagging a document has been described, but it is needless to say that the present invention is not limited to this tagging method. Furthermore, in the above-described embodiment, Japanese and English sentences have been exemplified, but it goes without saying that the present invention is not limited to these languages. As described above, it goes without saying that the present invention can be appropriately changed without departing from the spirit of the present invention.

Further, in the present invention, a disk-shaped recording medium or a tape-shaped recording medium in which an operation control program for executing the above-mentioned linked output processing of video data / document data is provided as the recording medium 32. Things. Of course, the recording medium 32 may be an optical disk, a magneto-optical disk, a magnetic tape, a memory card such as a flash memory, a memory chip, or the like, in addition to the floppy disk. Also, the HDD 34 shown in FIG.
Similarly, the recording medium of the present invention can be obtained.
Further, the operation control program can be provided also via a network communication such as the Internet, and therefore, the present invention can be applied to a program server side or a recording medium in a communication process.

According to such a recording medium 32 or the like, by providing the above-described operation control program to the document processing apparatus 1, each of the user-side document processing apparatuses 1 can realize the above-described linked output operation. Become. Further, by making the operation control program including the above-described classification processing and various window display operations, the document processing apparatus 1 that realizes the above-described document processing method can be easily realized using, for example, a general-purpose personal computer. it can.

As the recording medium 32 and the like, those which record document data and video data to be processed by the program together with the operation control program for realizing the above-mentioned processing are also useful. That is, when the electronic document data is provided from the recording medium 32 to the document processing apparatus 1, the operation control program and the video file are also provided at the same time. Corresponding document output processing and video output processing can be executed.

[0290]

As will be understood from the above description, the present invention has the following effects. That is, according to the document processing apparatus and the document processing method of the present invention, the contents of the electronic document data corresponding to the video data are presented and output in conjunction with the output of the video data. The present invention has the effect that a document can be presented together with the video by the user, and various and advanced information contents can be provided accurately. In particular, with regard to the electronic document data, a suitable operation for the user is realized by being presented in a form such as display of a text or a summary, display of a telop, and reading out. Also, the correspondence between the electronic document data and the video data is determined by the video data designation information added to the electronic document data or the time code added to the video data, so that the electronic document data and the video data can be reliably linked. Output processing based on the above. Furthermore, by monitoring the time code of the video data being output and switching the presentation output portion of the electronic document data, more appropriate and advanced linked output such as document presentation corresponding to each scene as video content can be achieved. It becomes possible.

Further, according to the recording medium of the present invention, a program for realizing the document processing method of the present invention can be provided, and the document processing apparatus of the present invention can be easily realized using, for example, a general-purpose personal computer. Become like Thus, the general user can easily enjoy the above effect.

[Brief description of the drawings]

FIG. 1 is a block diagram of a document processing apparatus according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a document structure used in the embodiment.

FIG. 3 is an explanatory diagram of a window for displaying a sentence structure according to the embodiment;

FIG. 4 is a flowchart of a manual classification process according to the embodiment.

FIG. 5 is a flowchart of an index creation process according to the embodiment.

FIG. 6 is an explanatory diagram of an activation value of an element according to the embodiment.

FIG. 7 is a flowchart of an active diffusion process according to the embodiment.

FIG. 8 is a flowchart of a central activation value update process according to the embodiment.

FIG. 9 is an explanatory diagram of a classification window according to the embodiment.

FIG. 10 is an explanatory diagram of a browsing window according to the embodiment.

FIG. 11 is an explanatory diagram of a classification model according to the embodiment.

FIG. 12 is a flowchart of an automatic classification process according to the embodiment.

FIG. 13 is a flowchart of automatic classification according to the embodiment.

FIG. 14 is a flowchart of a word meaning association degree calculation process according to the embodiment;

FIG. 15 is an explanatory diagram of a degree of association between meanings according to the embodiment.

FIG. 16 is a flowchart of a document reading process according to the embodiment;

FIG. 17 is a flowchart of a reading file generation process according to the embodiment;

FIG. 18 is an explanatory diagram of an example of a tag file according to the embodiment;

FIG. 19 is an explanatory diagram of an example of a tag file according to the embodiment;

FIG. 20 is an explanatory diagram of an example of a reading file according to the embodiment;

FIG. 21 is an explanatory diagram of an example of a reading file according to the embodiment;

FIG. 22 is an explanatory diagram of a reading window according to the embodiment.

FIG. 23 is a flowchart of a summary creation process according to the embodiment.

FIG. 24 is a flowchart of a summary sentence display process according to the embodiment.

FIG. 25 is a flowchart of a telop display process of the embodiment.

FIG. 26 is an explanatory diagram of a display example of a browsing window according to the embodiment.

FIG. 27 is an explanatory diagram of a display example including a summary of a browsing window according to the embodiment.

FIG. 28 is an explanatory diagram of a telop display example in a browsing window according to the embodiment;

FIG. 29 is an explanatory diagram of a telop display operation of the embodiment.

FIG. 30 is an explanatory diagram of a display example after displaying a telop in a browsing window according to the embodiment.

FIG. 31 is an explanatory diagram of a tag file and a video file according to the embodiment.

FIG. 32 is an explanatory diagram of a tag file with a video tag according to the embodiment.

FIG. 33 is an explanatory diagram of a tag file with a video tag according to the embodiment.

FIG. 34 is an explanatory diagram of a video output file according to the embodiment.

FIG. 35 is a flowchart of a video output process according to the embodiment.

FIG. 36 is an explanatory diagram of a video window by the video output processing according to the embodiment.

FIG. 37 is an explanatory diagram of functional blocks of the document processing apparatus according to the embodiment;

FIG. 38 is an explanatory diagram of a relationship between a tag file and a video file according to the embodiment.

FIG. 39 is an explanatory diagram of document output linked to video output according to the embodiment.

FIG. 40 is a flowchart of a video and linked document display output process of the embodiment.

FIG. 41 is an explanatory diagram of a video-linked output document file according to the embodiment;

FIG. 42 is an explanatory diagram of a video and linked document display state according to the embodiment.

FIG. 43 is an explanatory diagram of a video and linked document telop display state according to the embodiment;

FIG. 44 is a flowchart of video and linked document read-aloud output processing of the embodiment.

FIG. 45 is an explanatory diagram of a video-linked output document file according to the embodiment;

FIG. 46 is an explanatory diagram of a modified example of the video window of the embodiment.

[Description of Signs] 1 Document processing apparatus, 10 main body, 11 control unit, 12
Interface, 13 CPU, 14 RAM, 15
ROM, 20 input unit, 21 communication unit, 22 portable terminal reader, 30 display unit, 31 recording / reproducing unit, 32 recording medium, 33 audio output unit, 34 HDD, 101 window, 201 classification window, 301 browsing window, 303 document display Section, 304 summary display section, 401
Reading window, 501 Video window, 6
00 controller, 601 speech synthesis engine, 602
Telop creation engine, 603 summary creation engine,
604 video engine, 605 audio control unit, 606
Display control unit, 607 user interface, 6
20 video linked document creation engine, 621 video linked document file for output

Continued on the front page F term (reference)

Claims (19)

[Claims] A determination unit configured to determine a correspondence between electronic document data and video data; a video output control unit configured to control output of the video data; and an output of the video data based on the control of the video output control unit. And a document output control means for performing control for presenting and outputting the contents of the electronic document data corresponding to the video data determined by the determination means. 2. The method according to claim 1, wherein the determining unit determines a correspondence between the electronic document data and the video data based on video data designation information added to the electronic document data or a time code added to the video data. The document processing device according to claim 1. 3. The document processing apparatus according to claim 1, wherein the document output control means performs control so as to display a text or a summary of the electronic document data. 4. The document processing apparatus according to claim 1, wherein the document output control means performs control so as to display a telop for a text or an abstract of electronic document data. 5. The document processing apparatus according to claim 1, wherein the document output control means controls so that a text-to-speech output of a text or a summary sentence of the electronic document data is executed. 6. The document output control means monitors the time code of the video data output under the control of the video output control means, and switches the presentation output portion of the electronic document data. The document processing device according to claim 1. 7. A determination procedure for determining a correspondence between electronic document data and video data, a video output control procedure for controlling output of video data, and an output of video data based on the control in the video output control procedure. A document output control procedure for performing a control for presenting and outputting the content of the electronic document data corresponding to the video data determined in the determination procedure in conjunction with the document processing method. 8. In the determining step, the correspondence between the electronic document data and the video data is determined based on the video data designation information added to the electronic document data or the time code added to the video data. Claim 7
Document processing method described in. 9. The document processing method according to claim 7, wherein in the document output control procedure, control is performed such that display of a text or a summary of electronic document data is executed. 10. The document processing method according to claim 7, wherein in the document output control procedure, control is performed such that a telop is displayed for a text or an abstract of electronic document data. 11. The document processing method according to claim 7, wherein in the document output control procedure, control is performed such that a text-to-speech output of a text or a summary of electronic document data is executed. 12. The document output control step is characterized in that a time code of video data output is monitored under the control of the video output control means, and a presentation output portion of electronic document data is switched. The document processing method according to claim 7. 13. A procedure for determining the correspondence between electronic document data and video data, a video output control procedure for controlling output of video data, and an output of video data based on control in the video output control procedure. And a document output control procedure for controlling to present and output the contents of the electronic document data corresponding to the video data determined by the determination procedure in conjunction with the operation control program. Recording medium characterized by the above-mentioned. 14. The control method according to claim 1, wherein the determining step determines the correspondence between the electronic document data and the video data based on video data designation information added to the electronic document data or a time code added to the video data. 14. The recording medium according to claim 13, wherein the program is recorded. 15. The operation control program according to claim 13, wherein in the document output control procedure, an operation control program for controlling display of a text or an abstract of electronic document data is recorded. A recording medium according to claim 1. 16. The document output control procedure is characterized in that an operation control program for controlling a display of a telop for a text or an abstract of electronic document data is recorded. The recording medium according to claim 13. 17. The method according to claim 17, wherein the document output control procedure records an operation control program for performing control so that a text-to-speech output of a text or a summary of electronic document data is executed. Item 14. The recording medium according to Item 13. 18. In the document output control procedure, a time code of video data output under the control of the video output control means is monitored to switch a presentation output portion of electronic document data. 14. The recording medium according to claim 13, wherein the program is recorded. 19. The recording medium according to claim 13, wherein video data or electronic document data is recorded together with the operation control program.