JP2010061587A - Similar document determination device, similarity determination method and program therefor - Google Patents

Abstract

Similarity determination of document data such as PowerPoint is easily performed.
A similarity determination apparatus pays attention to a document title, a page title, and a page text of document data, and performs similarity determination between document data. At this time, the similarity determination device determines whether or not the document data have the same document title, the ratio of the number of pages of the same page title, and the ratio of the same page text. It is determined whether the document data is similar. This determination result is recorded in similar document information and used for search processing of similar document data.
[Selection] Figure 1

Description

  The present invention relates to a technique for retrieving similar document data.

Conventionally, as a method for retrieving document data that is the basis of document data, there are the following methods. (1) A method of retrieving document data that is the source of the document data using CSV (Concurrent Versions System) or the like (see Non-Patent Document 1). (2) A method of searching similar document data of the document data using tags included in a structured document such as SGML (Standard Generalized Mark-up Language) or XML (Extensible Markup Language) (see Non-Patent Document 2) . As a similar document data search technique, there is a method of using the appearance frequency of words in document data (see Non-Patent Document 3).
CSV (Concurrent Versions System), [online], [July 30, 2008 search], Internet, <URL: http://www.linkclub.or.jp/~tumibito/soft-an/cvs/cvs- man / cvs-ja_1.html # SEC1> Tomita et al., Full-text search system that can rank structured documents, IEICE Technical Report 2000-DBS-122, P361-368, IEICE, July 2000 Hashiashi et al., Retrieval Expression Expansion Method Using Word Contribution in Similarity Between Documents, IPSJ Journal vol.40 No.SIG8 (TOD4), P63-73, November 1999

  Here, for document data created with presentation software such as PowerPoint (registered trademark) and having a structure such as a document title, page title, and page text, the document data that is the basis of the document data (derived document) There is a need to search (data). However, in order to retrieve document data such as PowerPoint (registered trademark) by CSV described in Non-Patent Document 1, it is premised that document data is created in advance by CSV. Further, when using the technique described in Non-Patent Document 2, document data such as PowerPoint (registered trademark) must be once converted into a structured document such as XML, and a common tag used for each document data is used. It is necessary to make it. Furthermore, the technique described in Non-Patent Document 3 is unsuitable for searching document data such as PowerPoint (registered trademark) that does not necessarily have many words. Therefore, the present invention solves the above-described problem, makes it easy to determine similarity between document data of document data such as PowerPoint (registered trademark), and facilitates retrieval of document data when the document data is the basis. The purpose is to do.

  In order to solve the above-described problem, when the similar document determination device receives input of one or more document data via the input unit, the document analysis unit extracts the document title, the page title, and the page text from the document data, Document analysis information of the document data is created. When receiving input of new document data (document data to be compared), the document analysis unit extracts the document title, page title, and page text from the document data, and the determination unit displays the document title. Similarity determination is performed focusing on the page title and page text. That is, the determination unit determines whether the document data indicated by the document analysis information is (1) the same document title as the document title of the new document data, or (2) the same page title as the page title of the new document data. (3) Based on a similar pattern composed of any one or a combination of the same page text ratio as the page text of the new document data, whether or not the similar document data applies to any similar pattern Determine. When the determination unit determines that the document data is similar document data corresponding to any one of the similar patterns, the determination unit outputs a determination result including the similar pattern. As described above, the similar document determination device performs similarity determination focusing on the document title, the page title, and the page text, so that the document data is similar to each other without converting the document data into a structured document such as XML. Can be determined.

  The document analysis unit of the similar document determination apparatus extracts the first page title of the input document data as the document title of the document data. Therefore, even when there is no content having the attribute of document title in the document data, the document title can be extracted and document analysis information can be created.

  The document analysis unit of the similar document determination device extracts the page text of the first page of the document data as the document title of the document data when there is no page title of the first page of the input document data. Document analysis information can be created. Therefore, since there is a high possibility that the document analysis information includes the document title, the determination unit can easily determine the similarity between the document data using the document title included in the document analysis information.

  In addition, when there is a page without a page title in the document data, the document analysis unit of the similar document determination apparatus extracts the page text of this page as a page title and creates document analysis information. Therefore, since there is a high possibility that the document analysis information includes a page title, the determination unit can easily determine similarity between document data using the page title included in the document analysis information.

  Further, the similar document information creation unit of the similar document determination device, based on the determination result by the determination unit, for each piece of identification information of the input document data, the identification information of the similar pattern and the similar document data corresponding to the similar pattern Similar document information associated with identification information is created and stored in the storage unit. In this way, when the similar document search unit receives an input of a search request indicating a search condition including at least one of the identification information of the similar pattern and the identification information of the document data, the search condition is satisfied. A set of document data and similar document data can be retrieved from similar document information. Then, since the search result is displayed by the display processing unit, the user of the similar document determination device can select a document data (similar document data) group corresponding to a predetermined similar pattern, or document data corresponding to a similar pattern with respect to predetermined document data. A group of (similar document data) can be confirmed.

  Further, the rearrangement processing unit of the similar document determination device refers to the last edit date and time of the document data indicated in the document analysis information, and sets the document data retrieved from the similar document information and the set of similar document data in the set. When similar document data is rearranged in order of oldest or newest date and time, the retrieved document data and similar document data, and there are multiple pairs of different pairs of document data for the same similar document data The groups are rearranged in order of oldest or newest date and time of the last editing of document data in the group. For example, the display processing unit sorts the similar document data in ascending order of the last edit date and sorts the document data paired with the similar document data in the order of the last edit date. The document data group derived from the document data can be displayed in a state close to the order derived from the derivation source document data. That is, it becomes easy for the user of this similar document determination device to confirm in what order the document data of the document data group is derived from the document data of the derivation source.

  The document analysis unit of the similar document determination apparatus selects document data from a document storage unit that stores one or more document data, and the document title of the selected document data and the page title for each page in the document data. And the page text are extracted, and the determination unit performs similarity determination between the selected document data and the document data indicated by the document analysis information already stored in the storage unit, and the similar document information generation unit performs similar document information. Create The document analysis unit creates document analysis information for the document data for which similar document information has been created, and stores the document analysis information in the storage unit. By doing in this way, the similar document determination apparatus can create document analysis information while performing similarity determination of document data.

  In addition, the document analysis unit of the similar document determination device selects the document data stored in the document storage unit in order from the oldest editing date and time, and selects the document title of the selected document data and each page in the document data. The page title and page text are extracted. Therefore, the determination unit creates similar document information in order from the document data with the oldest editing date and time in comparison with the document analysis information already stored in the storage unit. In the similar document determination device, since a large amount of document analysis information is not yet stored in the storage unit at the stage of processing old document data, the determination unit creates similar document information in comparison with a small number of document analysis information. . On the other hand, when new document data is processed, since a large amount of document analysis information is stored in the storage unit, the determination unit creates similar document information in contrast to the large number of document analysis information. Here, if the document data group stored in the document storage unit is a document data group derived from a certain document data as a derivation source, similar document data of the document data (oldest document data) that is the origin The number of itself is relatively small, and the number of similar document data is considered to be relatively large for new document data. Therefore, the determination unit can efficiently perform similarity determination between document data that are actually considered to be derived. The rearrangement processing unit rearranges the document data retrieved from the similar document information and the set of the similar document data in the order of the last edit date of the similar document data in the set from the oldest, and the retrieved document data and the similar document data If there are a plurality of different sets of document data to be paired with respect to the same similar document data, the sets are rearranged in order from the oldest edit date of the document data in the set. Therefore, the user of the similar document determination device can easily confirm in what order the document data of the document data group is derived from the document data of the derivation source.

  According to the present invention, it is possible to easily determine similarity between document data of document data such as PowerPoint (registered trademark), and to easily search for document data that is the basis of the document data.

<Overview>
Hereinafter, the best mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described. First, an outline of processing of the similar document determination apparatus according to the present embodiment will be described with reference to FIGS. 1 and 2 are conceptual diagrams showing an outline of processing of the similar document determination apparatus of the present embodiment.

  Here, the document data handled by the similar document determination device is, for example, document data created by PowerPoint (registered trademark) or the like, and includes contents such as a document title, a page title of each page, and page text. Shall. Although there is a possibility that this content includes a graphic or the like, the similarity determination is performed noting the content of the graphic but paying attention to the content including text information. Whether these contents belong to any attribute of the document title, the page title, and the page text is determined based on attribute information attached to each content.

  Then, the similar document determination device extracts the document title, the page title, the page text, and the like of the document data and the document data subjected to similarity determination with the document data. Then, the similar document determination device determines how similar the respective document data are by paying attention to the document title, page title, and page text. For example, the similar document determination apparatus determines whether or not the document title of the document data A in FIG. 1 is the same as the document title of the document data B. If the document data B includes the same page title as the page title of the document data A, and if the same page title is included, it is determined whether the ratio is equal to or higher than a predetermined threshold. Further, it is determined whether or not the document data B includes the same page text as the page text of the document data A, and if the same page text is included, the ratio is equal to or greater than a predetermined threshold value. In these similarity determinations, the pages to be compared do not have to be the same pages of the respective document data. That is, it is only necessary that the same page title or page text is included in any page of the entire document data. The similarity determination result here is recorded in similar document information (described later) together with identification information (similar pattern identifier) of the similar pattern. Although details of the similar document information will be described later, the similar document information is information used for similar document data search processing using a similar pattern identifier as a key.

  The similar document determination device extracts the document title of the document data, the page title of each page, the page text, and the like as follows. That is, as shown in FIG. 2, the similar document determination device extracts the first page title of the first page (page 0) of the document data as the document title. Further, when extracting a page title from document data, if there is a page title on the page, the page title is extracted as it is as a page title. However, if the page has no page title, the first page text of the page is extracted as the page title. In this way, information that can be estimated as the document title is extracted from the document data, and information that can be regarded as the page title is extracted for pages that do not include the page title or the like, so the document title and page title of the document data are surely confirmed. Can be extracted. Therefore, the similar document determination device can easily perform similarity determination between document data.

<Configuration>
Next, the configuration of such a similar document determination apparatus will be described with reference to FIG. FIG. 3 is a block diagram showing the configuration of the similar document determination apparatus according to the present embodiment.

  The similar document determination apparatus 10 is roughly divided into an input / output unit 11, a processing unit 12, and a storage unit 13. The input / output unit 11 accepts input of document data to be processed by the processing unit 12, and outputs a display screen or the like indicating a processing result by the processing unit 12 to the display device 20 or the like. The processing unit 12 extracts a document title, page title, and page text from the document data, or makes similarity determination between the document data with reference to similar pattern information (described later). The storage unit 13 stores information that is referred to when the processing unit 12 performs similarity determination of document data, similar document information (described later) that is a result of the similarity determination, and the like. When the similar document determination device 10 is realized by program execution processing, the storage unit 13 stores a program for realizing the function of the similar document determination device 10.

  The input / output unit 11 includes an input / output interface. The processing unit 12 is realized by a program execution process by a CPU (Central Processing Unit) included in the similar document determination apparatus 10 or a dedicated circuit. Further, the storage unit 13 includes a storage medium such as a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), and a flash memory.

  The input / output unit 11 receives input of document data to be processed in the similar document determination apparatus 10 and outputs a processing result by the processing unit 12 to the outside. For example, the display screen displaying the processing result by the processing unit 12 is output and displayed on the display device 20.

  The processing unit 12 includes a document analysis unit 121, a determination unit 122, a similar document information creation unit 123, a similar document search unit 124, a rearrangement processing unit 125, and a display processing unit 126.

  The document analysis unit 121 creates document analysis information of the input document data. That is, the document analysis unit 121 determines whether each content indicates a document title, a page title, or a page text from the attribute information of each content included in the document data. Then, based on the determination result, the content of the document data is extracted as one of the document title, the page title, and the page text. As described with reference to FIG. 2, the document analysis unit 121 extracts, as the document title, content whose attribute is the page title from the content of the first page (page 0) of the document data. Further, when a page title is extracted from the second page (first page) and subsequent pages of the document data, if there is a content whose attribute is the page title, the document analysis unit 121 displays the content. Extracted as a page title. If there is no content whose page attribute is a page title, the first page text of that page (content whose attribute is page text) is extracted as a page title. Details of the processing procedure at this time will be described later. In addition to the contents, the document analysis unit 121 extracts information related to the last edit date and time of the document data.

  The determination unit 122 refers to similar pattern information stored in the similar pattern information storage unit 133 (described later), and compares certain document data (selected document data) with document data (comparison target document) to be compared with the document data. Data) similarity pattern is determined. Specifically, the determination unit 122 determines whether or not the document data indicated in the document analysis information is similar document data that applies to any of the similar patterns indicated in the similar pattern information. When the document data is determined to be document data (similar document data) corresponding to any similar pattern, a determination result including the similar pattern is output. Note that the determination unit 122 does not determine that the document data that does not match any of the similar patterns indicated in the similar pattern information is similar document data of the selected document data.

  For example, the determination unit 122 determines whether the document title of the comparison target document data is the same as the document title of the selected document data. Further, it is determined how much the comparison target document data has the same number of pages as the page title of the selected document data. If the document data and the document title of the comparison target document data are the same, the similarity pattern of the comparison target document data is determined to be “1”. Further, if the ratio between the selected document data and the page number of the same page title as the page title of the comparison target document data is equal to or greater than a predetermined threshold, the similar pattern of the comparison target document data is determined to be “2”. If the comparison target document data does not correspond to any of the similar patterns indicated in the similar pattern information, it is not determined as similar document data of the selected document data. Then, the determination unit 122 outputs the determination result.

  Based on the determination result determined by the determination unit 122, the similar document information creation unit 123 determines the document identifier (identification information) of the comparison target document data, the determined similar pattern, and the document data (similar document data) corresponding to the similar pattern. ) Is created in association with the document identifier (similar document identifier). Details of the similar document information will be described later.

  When the similar document search unit 124 receives input of identification information of a similar pattern via the input / output unit 11, the similar document search unit 124 selects one or more document data corresponding to the similar pattern of the input identification information from the similar document information and the similarity thereof. Retrieve a set of identifiers for document data. For example, the similar document search unit 124 searches the document data corresponding to the similar pattern “1” and the document identifier of the similar document data of the document data from the similar document information.

  The rearrangement processing unit 125 rearranges the document data retrieved from the similar document information and the set of the similar document data in a predetermined order. For example, the rearrangement processing unit 125 rearranges the sets of the document data and the similar document data in order from the oldest edit date of the similar document data of the set.

  The display processing unit 126 displays the search results searched by the similar document search unit 124 and the display screen showing the search results sorted by the sorting processing unit 125 on the display device 20 or the like.

  The storage unit 13 includes a document storage unit 131, a document analysis information storage unit 132, a similar pattern information storage unit 133, and a similar document information storage unit 134 in a predetermined area.

  The document storage unit 131 stores one or more document data input from the input / output unit 11.

  The document analysis information storage unit 132 stores the document analysis information created by the document analysis unit 121. This document analysis information is information indicating the document identifier, document title, page title, and page text of the document data. FIG. 4 is a diagram illustrating the document analysis information of FIG. As shown in FIG. 4, the document analysis information includes, for example, document file information indicating the document title of the document data, page title information indicating the page title of each page of the document data, and each page of the document data. And page text information indicating the page text. Among these, the document file information is information indicating the document title extracted from the document data, the date and time of last editing, and the like for each document identifier of the document data. The document file information may further include information such as the document file name and storage folder name of the document data. The page title information includes a document identifier, a page identifier, and a page title of the page. The page text information indicates a document identifier, a page identifier, and a page text of the page. The document file information, page title information, and page text information may be combined into a single piece of information.

  The similar pattern information storage unit 133 in FIG. 3 stores similar pattern information. As illustrated in Table 1 below, the similar pattern information is information indicating a condition for determining that the comparison target document data is applicable to the similar pattern for each similar pattern identifier. This similar pattern information also includes information relating to the threshold value when the threshold value of the page number or the threshold value of the page text is used when determining whether or not the comparison target document data applies to the similar pattern. For example, in the similar pattern information shown in Table 1, the similar pattern identifier “1” indicates a condition that the document titles of the document data are the same. The similar pattern identifier “2” indicates that the ratio of the number of pages in which the comparison target document data includes the same page title as the page title of the selected document data is 75% or more. Further, the similar pattern identifier “3” counts the number of pages in which the ratio of the page text that is the same as the page text of the selected document data is 50% or more in the comparison target document data, and the number of pages is the comparison number. A condition of 75% or more of the total number of pages of the target document data is shown. This similar pattern information is set via the input / output unit 11.

  The similar document information storage unit 134 stores similar document information. As shown in Table 2, the similar document information includes, for each document identifier of document data (selected document data), a similar pattern identifier and a document identifier (similar document identifier) of similar document data that applies to the similar pattern of the identifier. It is information to show. The similar document information may further include the number of pages of selected document data that applies to the similar pattern. This similar document information is used as an index when the similar document search unit 124 searches for similar document data.

<Processing procedure>
Next, the processing procedure of the similar document determination apparatus 10 will be described using a flowchart. First, the procedure in which the similar document determination apparatus 10 creates document analysis information and similar document information will be described with reference to FIG. FIG. 5 is a flowchart showing a processing procedure of the similar document determination apparatus of FIG.

  First, when the similar document determination apparatus 10 receives input of document data via the input / output unit 11, the similar document determination apparatus 10 stores the received document data in the document storage unit 131 (S1). Then, the document analysis unit 121 rearranges the document data group stored in the document storage unit 131 in order of the last editing date and time of the document data (S2), and sets the document data with the oldest last editing date and time of the document data. (Select) (S3).

  Next, the document analysis unit 121 extracts a page title and page text from the set document data (S4). Then, the document analysis unit 121 uses the page title and page text extracted in S4 to create document analysis information (see FIG. 4) regarding this document data (S5). Then, the document analysis information is stored in the document analysis information storage unit 132. Details of the document analysis information creation procedure will be described later.

  Next, the determination unit 122 determines similarity of the selected document data with document data whose document analysis information is already registered in the document analysis information storage unit 132 (S6). Details of the similarity determination process will be described later. The document analysis unit 121 stores the document analysis information of the document data in the document analysis information storage unit 132 after the similarity determination of the document data set in S3.

  When the document analysis unit 121 sets the document data with the oldest editing date and time in S3, the document analysis information is not stored in the document analysis information storage unit 132, so the processing of S6 is not performed. The document analysis information thus stored is stored in the document analysis information storage unit 132 as it is.

  After S6, if the processing of all the document data in the document storage unit 131 is not completed (No in S7), the document analysis unit 121 selects the next document with the oldest edit date from the document data group rearranged in S2. Data is set (S8), and the process returns to S4. On the other hand, if all the document data in the document storage unit 131 has been processed (Yes in S7), the process ends.

  As described above, the similarity determination device 10 selects the document data stored in the document storage unit 131 in order from the oldest editing date and performs the similarity determination with the document data indicated in the document analysis information. It is possible to efficiently create similar document information of document data that is highly likely to be derived. For example, in the case where a plurality of document data derived from certain document data is stored in the document storage unit 131, the last edit date and time of the document data that is the source is the oldest. it is conceivable that. The order of derivation of the document data from the document data is close to the order in which the last editing date and time are arranged in chronological order. Therefore, it is more likely that the combination of all the document data stored in the document storage unit 131 actually has a derivation relationship than when the similarity determination is performed and the similar document information is created (similarity). ) It is possible to efficiently create similar document information related to document data.

  Next, the document analysis information creation process shown in S5 will be described with reference to FIGS. 3, 5, and 6 and FIG. FIG. 6 is a diagram showing an example of the structure of document data according to the present embodiment. FIG. 7 is a flowchart showing details of the document analysis information creation processing of FIG. Here, the document analysis unit 121 of the similar document determination apparatus 10 generates a document title from the document data. In S4 of FIG. 5, when the document analysis unit 121 cannot extract the page title, the page title is generated. In the following description, page numbers in document data are numbers assigned in the order of “0,..., N” from the first page of document data as shown in FIG. The page text number is assigned to each page of the document data in the order of “0, 1,..., N” in order from, for example, the content (text box) located at the top of the page. And However, the first page text number in the first page of the document data is a number starting from “−1” to distinguish it from the first page text of other pages.

  First, the document analysis unit 121 of the similar document determination apparatus 10 in FIG. 3 sets the start page number of the document data set in S3 or S8 in FIG. 5 to 0 (S11). The start page text number (page text number of the start page) is set to -1 (S12).

  Then, if there is a page title on this page (Yes in S13) and the page number is 0 (Yes in S16), the document analysis unit 121 extracts the content to be processed as the document title of this document data. (S17). Further, this content is extracted as the page title of this page (S18). That is, the page title of the first page of the document data is extracted as the document title.

  On the other hand, the document analysis unit 121 has a page title on this page (Yes in S13), but if the page number is not 0 (No in S16), that is, if it is not the first page, The page title is extracted (S18). The document title and page title extracted here are stored in the document file information of the document analysis information, page title information, and the like.

  If the page text number of this content = 0 (Yes in S19), the document analysis unit 121 extracts this content as the page text of the page (S20). The extracted page text is accumulated by the document analysis unit 121 in page text information (see FIG. 4) of document analysis information related to the document data.

  On the other hand, when there is no page title on this page (No in S13), the document analysis unit 121 sets the start page text number to 0 (S14) when the page text number is not 0 in S19 (No in S19). Then, it is confirmed that the page text number of the content to be processed is 0 and there is no page title on the page (Yes in S15), and the process proceeds to S16. On the other hand, when the page text number of the content to be processed is 1 or more in S15, or when the page has a page title (No in S15), the process proceeds to S20, and the content to be processed is extracted as page text. That is, in S15, the document analysis unit 121 determines whether to extract the content to be processed as a document title or a page title or as a page text.

  After S20, the document analysis unit 121 ends the processing of all the page texts of the processing target page (Yes in S21), and ends the processing of all the pages of the processing target document data (S21). The process is terminated. On the other hand, after S20, if there is an unprocessed content on the page to be processed (No in S21), the start page text number is incremented by 1 (S24). Then, the process returns to S15. That is, the process proceeds to the next content on the same page. If all the page texts on the page to be processed have been processed (Yes in S21), but there is an unprocessed page in the document data to be processed (No in S22), the document analysis unit 121 starts the start page. The number is incremented by 1 (S23). Then, the process returns to S13. That is, the process proceeds to the next page in the same document data.

  By doing in this way, the document analysis part 121 of the similar document determination apparatus 10 produces | generates a document title from document data, extracts a page title and page text, and produces document analysis information. The created document analysis information is stored in the document analysis information storage unit 132.

  Next, the process of S6 in FIG. 5 will be described with reference to FIG. FIG. 8 is a flowchart showing details of the similarity determination process of FIG. Here, the similarity determination is performed on the document data (selected document data) selected by the similar document determination apparatus 10 in FIG. 3 with the document data in which the document analysis information is already registered in the document analysis information storage unit 132. A process for creating similar document information based on the determination result will be described.

  First, the determination unit 122 in FIG. 3 searches the document file information of the document analysis information for the document identifier of the document data having the same document title as the selected document data (S31). If there is a corresponding document identifier (Yes in S32), the similar document information creating unit 123 creates similar document information indicating the document identifier. That is, the similar document information creating unit 123 includes the document identifier of the selected document data, the similar pattern identifier “1 (the document title is the same)”, and the document identifier of the similar document data corresponding to the similar pattern identifier in the similar document information. Are registered (S33). Then, the determination unit 122 sets the first page of the selected document data (S34).

  On the other hand, if the document analysis information does not include the document identifier of the document data having the same document title as the selected document data (No in S32), S33 is skipped and the process proceeds to S34. After S34, the determination unit 122 searches the page identifier information of the document analysis information for the document identifier of the document data having the same page title as the page title of the currently set page (S35). If there is a corresponding document identifier (Yes in S36), the similar document information creating unit 123 registers this document identifier in a page title similarity list (not shown) (S37). The page title similarity list is a list for counting the number of document data having the same page title and the number of pages having the same page title in the document data. This list is stored in a predetermined area of the storage unit 13.

  On the other hand, if the document analysis information does not include the document identifier of the document data having the same page title as the currently set page in the selected document data (No in S36), the determination unit 122 skips S37 and the determination unit 122 The first page text of data is set (S38). Then, the determination unit 122 retrieves the document identifier of the document data having the same page text as the page text of the currently set page from the page text information of the document analysis information (S39). If there is a corresponding document identifier (Yes in S40), the similar document information creating unit 123 registers this document identifier in a similar list (not shown) (S41). This similarity list is a list for counting document identifiers of document data having the same page text and the number of the same page text in the document data. This list is also stored in a predetermined area of the storage unit 13.

  There is no document identifier of document data having the same page text as the page text of the currently set page in the selected document data in the document analysis information (No in S40), and all pages in the currently set page If the text processing is completed (Yes in S42), the process proceeds to S44. S44 will be described later. On the other hand, if there is an unprocessed page text in the currently set page in the selected document data (No in S42), the next page text is set (S43), and the process returns to S39. By such processing, the document identifier of the document data having the same page text as the page text in the page to be processed among the pages of the selected document data is registered in the similarity list.

  Here, when the processing of all the page texts in the currently set page is completed in S42 (Yes in S42), the same page text as each page text in the currently set page is displayed in the similar list. As many document identifiers of document data as the number of documents having the same page text in the document data are registered. For example, when page texts A and B are included in page A of the selected document data, N page texts that are the same as page text A are included in the document data of document identifier “A” to be compared. If the same page text as text B is included, (N + M) document identifiers “A002” are registered in the similarity list. As a result, it is understood that the document identifier “A002” has (N + M) page text numbers that are the same as the page text of page A of the selected document data.

  When the processing for one page is completed for the selected document data, the determination unit 122 acquires the number of page texts for each document identifier registered in the similarity list in S44, and the number of page texts is the number of similar pattern information. Document identifiers that are equal to or greater than the threshold for page text are registered in the page text similarity list. For example, when the number of page texts of page A of the selected document data is 6 and the threshold value is 50%, the determination unit 122 selects the page text of the page A among the document identifiers registered in the similarity list. Those having the same page text number of 3 or more are registered in the page text similarity list. As the threshold value, the value indicated by the page text threshold value of the similar pattern related to the page text (similar pattern identifier “3”) among the similar patterns indicated by the similar pattern information (see Table 1) is used. By executing such processing for each page of the selected document data, the determination unit 122 searches for a document identifier of document data similar to the selected document data in units of pages.

  After S44, if there is an unprocessed page for the selected document data (No in S45), the determination unit 122 sets the next page (S46) and returns to S35. On the other hand, if all pages have been processed for the selected document data (Yes in S45), the determination unit 122 determines the number of similar pages for each document identifier (the number of pages having the same page title) from the page title similarity list. ) And the document identifier whose number of similar pages is equal to or greater than the threshold indicated in the similar pattern information and the number of similar pages are registered in the similar document information (S47).

  For example, consider a case where the number of pages of selected document data is 10 and the page threshold value for a similar pattern related to the page title (similar pattern identifier “2” in Table 1) is 80%. In this case, when there is a document identifier having 8 or more similar pages (number of pages having the same page title) among the document identifiers registered in the page title similarity list, the determination unit 122 determines that the document identifier Is determined to be similar document data with the similar pattern identifier “2”. Therefore, the similar document information creation unit 123 uses the document identifier (for example, “A002”) of the document data corresponding to the similar pattern, the document identifier (for example, “A001”) of the selected document data, and the similar pattern identifier “2”. And the number of similar pages “8 pages” are registered in the similar document information (see Table 2).

  Thereafter, the determination unit 122 obtains the number of similar pages for each document identifier from the page text similarity list, and compares the number of similar pages with a document identifier that is equal to or greater than the threshold indicated by the similar pattern information. The document information is registered (S48).

  For example, the selected document data has 10 pages. In this case, among the document identifiers registered in the page text similarity list, the number of similar pages (the number of pages having the same page title) is 8 or more. When there is a document identifier, the determination unit 122 determines that the document data with this document identifier is similar document data with the similar pattern identifier “3”. Therefore, the similar document information creation unit 123 uses the document identifier (for example, “A003”) of the document data corresponding to the similar pattern, the document identifier (for example, “A001”) of the selected document data, and the similar pattern identifier “3”. And the number of similar pages “8 pages” are registered in the similar document information (see Table 2). Then, the process ends.

  In this manner, the similar document determination apparatus 10 in FIG. 3 performs similarity determination on the selected document data with document data in which the document analysis information is already registered in the document analysis information storage unit 132, and the similarity determination result is obtained. Based on this, similar document information is created. The created similar document information is used for search processing of similar document data.

  Here, the determination unit 122 determines that the document data is one of the three similar patterns indicated in the similar document information. However, the present invention is not limited to this. For example, various similar patterns are defined in the similar pattern information by combining the page threshold value of the page title, the page threshold value of the page text, and the value of the page text threshold value, and the determination unit 122 assigns any similar pattern to the document data. You may make it determine whether it is applicable.

  Next, similar document data search processing and display processing using similar document information will be described. FIG. 9 is a flowchart showing similar document data search processing and display processing by the similar document determination apparatus of FIG. Here, when the similar document determination apparatus 10 receives the input of the similar pattern identifier, the similar document information is searched for a set of document data and the similar document data corresponding to the similar pattern identifier, and the document data and the similar document are retrieved. A case where data sets are displayed in time series will be described as an example.

  First, the similar document search unit 124 of the similar document determination apparatus 10 in FIG. 3 receives an input of a similar pattern identifier via the input / output unit 11 (S51). Then, the similar document search unit 124 searches the similar document information (Table 2) for a combination of the document identifier and the similar document identifier that matches the input similar pattern identifier (S52). That is, the similar document search unit 124 acquires, from the similar document information, a set of the document identifier of the document data corresponding to the similar pattern identifier and the similar document identifier of the similar document data using the input similar pattern identifier as a key. .

  Then, the similar document search unit 124 searches the document file information of the document analysis information for the last edit date and time of the document data corresponding to each of the acquired document identifier and the similar document identifier (S53). Then, out of the set of the retrieved document identifier and similar document identifier, the similar document identifier with the oldest edit date and time of the document data corresponding to the similar document identifier is set (S54). Thereafter, the display processing unit 126 displays the set similar document identifier on the display screen (S55). Next, the rearrangement processing unit 125 rearranges the document identifier group that is a pair of similar document identifiers set in S54 by the last edit date and time of the document data corresponding to the document identifier (S56), The document identifier of the oldest document data is set (S57). Then, the display processing unit 126 displays the similar document identifier set in S54 and the document identifier set in S57 (S58). That is, the similar document identifier of the oldest similar document data applicable to the similar pattern in the set of similar document data registered in the similar document information and the document identifier of the document data paired with the similar document data are displayed. . At this time, the number of similar pages registered in the similar document information may also be displayed.

  Thereafter, the similar document search unit 124 sets the document identifier with the next oldest editing date and time next to the document identifiers rearranged in S56 (S59). If there is an unprocessed document identifier for the document identifiers rearranged in S56 (No in S60), the process returns to S58. On the other hand, when all the document identifiers have been processed for the document identifiers rearranged in S56 (Yes in S60), the similar document search unit 124 determines in S54 from the set of document identifiers and similar document identifiers searched in S52. A set of document identifiers (similar document identifiers) of similar document data having the last edit date and time next to the set similar document identifiers is set (S61). In this way, if there is an unprocessed set of the document identifier and the similar document identifier searched in S52 (No in S62), the process returns to S55. On the other hand, when the processing is completed for all pairs of document identifiers and similar document identifiers searched in S52 (Yes in S62), the processing ends.

  In this way, the similar document determination apparatus 10 can search for document identifiers of document data that match a predetermined similar pattern with respect to the document data stored in the document storage unit 131. For example, a document data group having the same document title and a document data group having the same page title at a predetermined ratio or more can be searched. Further, the similar document determination apparatus 10 refers to the last edit date and time of the retrieved document data, arranges the similar document data in order of oldest edit date, and further includes a plurality of document data having the same document data as the similar document data. In some cases, the document data are displayed in order from the oldest. In this way, the document data group in the derivation relationship is displayed in order from the oldest document data (document data that is likely to be the derivation source in the document data group), and is further derived from this document data. It is possible to display the document data having a high possibility of being close to the order of derivation.

  Here, a display screen example displayed by the display processing unit 126 will be described. FIG. 10 is a diagram showing an example of a display screen displayed by the display processing unit of FIG. As shown in FIG. 10, the display screen displays an input similar pattern identifier, a similar document identifier searched using the similar pattern identifier as a key, a paired document identifier, the number of similar pages, similar document data, and document data. It is configured to include the last edit date and time. Here, the similar document identifiers are displayed in order from the oldest last edited date. The document identifiers paired with the similar document identifiers are also displayed in order from the oldest last edited date.

  For example, in the example of the screen shown in FIG. 10, the similar document identifier of the pair having the oldest last edit date / time of the similar document data is “A002” among the pair of the document identifier and the similar document identifier corresponding to the similar pattern identifier “1”. This indicates that the last editing date is “2008.08.21.20:00:00”. Among the document identifiers that form a pair of similar document identifiers, the document identifier of the document data with the oldest last edit date is “A001”, the number of similar pages is “10 (pages)”, and the last edit date is It indicates that it is “2008.08.02.10:05:30”. Of the pair of the document identifier and the similar document identifier corresponding to the similar pattern identifier “1”, the pair having the similar document data with the next oldest edit date is the group having the similar document data “B001”. Yes, it indicates that the last edit date is “2008.08.03.16:04:00”. In addition, among the document identifiers that form a pair of similar document identifiers, the document identifier of the document data with the oldest last edit date is “B003”, the number of similar pages is “5 (page)”, and the last edit Indicates that the date and time is “2008.08.04.13: 13:00”.

  By displaying such a display screen by the display processing unit 126, it is possible to check document data corresponding to the similar pattern designated by the user in time series. That is, the document data can be confirmed in a state close to the document data derivation order.

  It should be noted that a document identifier or similar document identifier on the display screen may be linked to the document data or similar document data stored in the document storage unit 131. When the selection input of the document identifier or similar document identifier is received on the display screen, the display processing unit 126 may display the document data corresponding to the document identifier or similar document identifier. In this way, the user can easily confirm document data that is in a similar relationship (estimated to have a derivation relationship).

  The present invention is not limited to the embodiment described above. For example, in the similarity determination process in S6 of FIG. 5 described above, similar document information is created in order from the document data stored in the document storage unit 131 in order from the oldest edit date. You may create in order from the newest date. Furthermore, in the similarity determination process shown in FIG. 8, the document data stored in the document storage unit 131 is used as the selected document data. However, other document data may be used. That is, when the similar document determination apparatus 10 receives input of new document data, the document analysis unit 121 extracts the document title, page title, and page text of the document data by the above-described procedure, and the determination unit 122 Similarity determination with the document analysis information stored in the analysis information storage unit 132 may be performed. Thereafter, the display processing unit 126 may display the similarity determination result on the display screen. By doing in this way, the similar document determination apparatus 10 can perform the similarity determination of various document data.

  Further, when the document analysis unit 121 extracts the document title from the document data, if there is no page title on the first page, the first page text on this page may be extracted as the document title. By doing in this way, the similar document determination apparatus 10 can extract a document title reliably from document data.

  Further, the similar document information creation unit 123 may create similar document information for all the document data stored in the document storage unit 131. That is, the determination unit 122 may perform similarity determination for any two combinations of document data in the document storage unit 131 and create similar document information. By doing in this way, the similar document determination apparatus 10 can prevent occurrence of omission of generation of similar document information for document data actually having a similar relationship.

  Further, when searching for similar document information, the similar document search unit 124 not only searches for a similar pattern identifier but also a document identifier, a similar document identifier, document data, or the last edit date / time of similar document data, or a combination thereof. You may search using as a key. By doing in this way, the similar document determination apparatus 10 can search various similar document data.

  The similar document determination apparatus 10 according to the present embodiment can be realized by a program for executing the processing as described above, and the program is stored in a computer-readable storage medium (CD-ROM or the like) and provided. Is possible.

<Experimental result>
Next, an evaluation experiment for evaluating how much derived document data can be detected by similarity determination of document data using the similar document determination apparatus 10 will be described with reference to FIGS. 11 and 12. FIG. 11A is a diagram showing the file names of the 18 document data used in this experiment, and FIG. 11B is a diagram showing the correct answer of the document data derivation relationship of FIG. is there. FIG. 12 is a diagram showing evaluation experiment data of the similar document determination device of the present embodiment.

  Here, 18 PowerPoint (registered trademark) files shown in FIG. 11A are used as document data. Further, 15 similar patterns shown in FIG. 12 were used as similar patterns. Note that # 16 in FIG. 12 is a method using a tense index as a comparative example. Then, the similar document determination device 10 applies the fifteen similar patterns, and performs similarity determination between the document data (determination as to whether or not it is possible to estimate that the document data has a derivation relationship). The actual derivation relationship between document data (correct derivation relationship) is as shown in FIG. In FIG. 11B, the value in the circle indicates the ID of the file shown in FIG. Of these, files (document data) connected by “→” indicate that they are actually in a derivation relationship. Of the 18 document data, the number of derivation pairs was 22.

  In the following description, the number of correct answers is the number of pairs of document data that are actually in a derivation relationship, and the extracted number of derivations is the amount of document data that the similar document determination device 10 determines as having a derivation relationship. Is the number of pairs. The number of correct answers extracted is the number of pairs of document data that are actually in a derivation relationship among the extracted derivation numbers. Further, the total number of correct answers is the number of document data pairs that are actually derived in the entire document data to be processed. In addition, here, the calculation was performed with the relevance ratio = the number of extracted correct answers / the number of derived derivatives and the recall ratio = the number of extracted correct answers / the total number of correct answers.

  Similar patterns used here are roughly divided into categories of determination rules 1, 2, 3, and 4, as shown in FIG. The determination rule 1 is a rule that, when the document data to be compared includes the same page as the selected document data, the document data is determined as a derivation relationship (determined as similar document data). The determination rule 2 is a rule that, when there is the same text as the selected document data, the document data is determined as a derivation relationship (determined as similar document data). In this determination rule 2, a page text threshold value and a page threshold value (see Table 1) are set, and nine values are obtained by combining the page text threshold value (25% to 75%) and the page threshold value (25% to 75%). The variation of was set. Further, the determination rule 3 is a rule that, when there is the same page title as the selected document data, the document data is determined as derived document data (similar document data). In this determination rule 3, three variations are set according to the page threshold value (25% to 75%) for the page title. The determination rule 4 is a rule that, when there is the same document title as the selected document data, the document data is determined as derived document data (similar document data). Note that “combination” of # 15 is a derived document that satisfies both the above-described determination rule 1 (# 1) and page rule 25% or more match and page 25% or more match (# 2) of the determination rule 2. It is a rule that it is determined as data (similar document data). The “temporal index” of # 16 is a rule that divides a file name into a character string and a number, and determines an old document with a matching character string as derived document data (similar document data).

  FIG. 12 shows the precision, the recall, the total number of correct answers, the number of derived derivations, and the number of extracted correct answers when using similar patterns in this experiment. As shown in FIG. 12, the recall rate is higher than that of the # 16 tense index, indicating the effectiveness of the similar document determination apparatus 10 of the present embodiment. If # 10 (matching page text 75% or more, matching page 75% or more) of decision rule 2 is used, a high matching rate (0.773) is obtained, and # 2 of decision rule 2 (matching page text 25% or more, page 25). % Reproducibility (0.800) was shown to be high. Therefore, it is effective to use # 10 of the determination rule 2 when it is desired to make a determination with a high precision for the 18 pieces of document data, and use # 2 of the determination rule 2 when it is desired to make a determination with a high reproduction rate. It was shown that there is. Also, determination rule 2 # 5 (matching page text 25% or more, matching page 50% or more) and determination rule 3 # 12 (matching page title 50% or more) are values that balance the precision and recall. Therefore, it was confirmed that it is effective to use one of these similar patterns when it is desired to perform a determination process in which the precision ratio and the recall ratio are balanced.

It is the conceptual diagram which showed the process outline | summary of the similar document determination apparatus of this Embodiment. It is the conceptual diagram which showed the process outline | summary of the similar document determination apparatus of this Embodiment. It is the block diagram which showed the structure of the similar document determination apparatus of this Embodiment. It is the figure which illustrated the document analysis information of FIG. It is the flowchart which showed the process sequence of the similar document determination apparatus of FIG. It is the figure which showed the structural example of the document data of this Embodiment. It is the flowchart which showed the detail of the document analysis information creation process of FIG. It is the flowchart which showed the detail of the similarity determination process of FIG. 4 is a flowchart showing search processing and display processing of similar document data by the similar document determination device of FIG. 3. It is the figure which showed the example of a display screen which the display process part of FIG. 3 displays. (A) is the figure which showed the file name of 18 document data used by this experiment, (b) is the figure which showed the correct answer of the derivation relation of the document data of (a). It is the figure which showed the evaluation experiment data of the similar document determination apparatus of this Embodiment.

Explanation of symbols

10 Similar Document Determination Device 11 Input / Output Unit (Input Unit)
DESCRIPTION OF SYMBOLS 12 Processing part 13 Memory | storage part 20 Display apparatus 121 Document analysis part 122 Judgment part 123 Similar document information creation part 124 Similar document search part 125 Rearrangement process part 126 Display processing part 131 Document storage part 132 Document analysis information storage part 133 Similar pattern information Storage unit 134 Similar document information storage unit

Claims (10)

An input unit for receiving input of one or more document data;
Extract the document title of the document data and the page title and page text for each page in the document data from the input document data structure, and extract the extracted document title, page title and page text. Creating document analysis information associated with the identification information of the original document data and storing it in the storage unit;
The storage unit for storing the document analysis information;
When new document data is received via the input unit, the document analysis unit extracts a document title of the document data, a page title and a page text for each page in the document data,
For each document data indicated in the document analysis information, (1) whether the document title is the same as the document title of the new document data, or (2) the number of pages having the same page title as the page title of the new document data. (3) Based on a similar pattern composed of any one or a combination of the same page text ratio as the page text of the new document data, the document data indicated in the document analysis information is converted into any similar pattern. A similar document determination comprising: a determination unit that determines whether or not the corresponding similar document data is applicable, and outputs a determination result including the similar pattern when it is determined to be similar document data corresponding to any one of the similar patterns apparatus.   2. The similar document determination apparatus according to claim 1, wherein the document analysis unit extracts a page title of the first page of the document data as a document title of the document data, and creates the document analysis information. .   When there is no page title of the first page of the document data, the document analysis unit extracts the page text of the first page of the document data as the document title of the document data, and creates the document analysis information. The similar document determination device according to claim 1, wherein the similar document determination device is provided.   The document analysis unit, when there is a page without a page title in the document data, extracts the page text of the page as a page title and creates the document analysis information. Item 4. The similar document determination device according to any one of items 3 to 3. The similar document determination device includes:
Similar document information in which identification information of the determined similar pattern and identification information of similar document data applicable to the similar pattern are associated with each other based on the determination result by the determination unit. A similar document information creating unit that stores the information in the storage unit;
When an input of a search request indicating a search condition including at least one of the identification information of the similar pattern and the identification information of the document data is received via the input unit, the document data satisfying the search condition and its A similar document search unit for searching a set of similar document data from the similar document information;
The similar document determination apparatus according to claim 1, further comprising: a display processing unit that displays the search result. The document analysis unit further extracts the last edit date and time of each of the document data and includes it in the document analysis information,
The similar document determination device includes:
Referring to the last edit date and time of the document data indicated in the document analysis information, the set of the document data and similar document data retrieved from the similar document information is the oldest of the last edit date and time of the similar document data in the set. When there are a plurality of different pairs of document data for the same similar document data among the sets of the document data searched and the similar document data sorted in order or new order, A sort processing unit for sorting the document data in the oldest order or the newest order of the last edit date and time,
6. The similar document determination apparatus according to claim 5, wherein the display processing unit displays the rearranged document data and a set of similar document data. The storage unit includes a document storage unit that stores the one or more document data,
The document analysis unit selects the document data stored in the document storage unit, extracts the document title of the selected document data, and the page title and page text for each page in the document data,
The determination unit determines, for each document data indicated in the document analysis information already stored in the storage unit, whether (1) the document title is the same as the document title of the selected document data, or (2) the selected document Based on a similar pattern consisting of any one or a combination of the ratio of the number of pages having the same page title as the page title of data, and (3) the ratio of the same page text as the page text of the selected document data, the document analysis information Determining a similar pattern of the displayed document data with respect to the selected document data, and outputting the determination result;
The document analysis unit creates the document analysis information in which the document title, page title, and page text extracted from the selected document data are associated with identification information of the selected document data, and stores the document analysis information in the storage unit The similar document determination apparatus according to claim 6. The document analysis unit selects the document data stored in the document storage unit in order from the oldest editing date and time, and selects the document title of the selected document data and the page title and page for each page in the document data. Extract the text and
The rearrangement processing unit rearranges the document data retrieved from the similar document information and a set of similar document data in descending order of the last edit date of the similar document data in the set, and the retrieved document data and the similar When there are a plurality of different pairs of document data as a pair with respect to the same similar document data among the sets of document data, the sets are rearranged in order from the oldest edit date of the document data in the set. The similar document determination device according to claim 7. A similar document determination device that performs similarity determination of document data,
Accepting input of one or more document data;
Extract the document title of the document data and the page title and page text for each page in the document data from the input document data structure, and extract the extracted document title, page title and page text. Creating document analysis information shown in association with identification information of the original document data;
Storing the created document analysis information in a storage unit;
A step of extracting a document title of the document data, a page title and a page text for each page in the document data when new document data is received;
For each document data indicated in the document analysis information, (1) whether the document title is the same as the document title of the new document data, or (2) the number of pages having the same page title as the page title of the new document data. (3) Based on a similar pattern composed of any one or a combination of the same page text ratio as the page text of the new document data, the document data indicated in the document analysis information is converted into any similar pattern. Determining whether or not it is similar document data to be applied, and executing a step of outputting a determination result including the similar pattern when it is determined to be similar document data corresponding to any one of the similar patterns Method. A program for causing a computer, which is the similar document determination device, to execute the similar document determination method according to claim 9.

Images