JP2012018510A - Document processor, document processing method, document processing program, and computer readable recording medium recorded with document processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract an original document including a document that is properly cited from a document on the Internet.
A partial character string generation unit that generates a partial character string that forms a part of the character string for each document from character strings included in a plurality of documents, and the portion generated by the partial character string generation unit Unique partial character string determination means for determining a partial character string that is not included in a document other than the document in which it is generated as a unique partial character string, the total number of partial character strings for each document, and the unique partial character Unnecessary document detecting means for detecting, as an unnecessary document, a document whose ratio with the number of unique partial character strings determined by the column determining means is within a predetermined range.
[Selection] Figure 1

Description

  The present invention relates to a document processing apparatus that extracts a target document to be analyzed from a document existing on the Internet, for example, a document such as a blog.

With the development of the Internet, the amount of available document data has increased dramatically. In these document data, there are now many cases where individuals voluntarily stated their interests and opinions on social events through blogs. Therefore, by collecting and analyzing document data describing such opinions, it is possible to comprehensively grasp the social trends and consumer trends that previously required recruiting respondents and conducting questionnaires. It is expected to be possible to implement in real time.
Digital data, on the other hand, is easy to obtain, and at the same time, it is easy to quote, edit, modify, and retransmit, and it is said that Internet documents contain a lot of such secondary information. ing. If the original primary information and secondary information by diversion are mixed, storage efficiency decreases due to duplicate storage of similar data, and similar results are repeatedly presented for search queries. Problems such as a drop in the listability due to. In view of this, a system has been proposed in which a partial character string is extracted from each document data and a list of appearing documents is managed for each partial character string so that a document including an overlapping portion can be presented (for example, Patent Document 1). And 2).

JP 2008-33728 A Special table 2008-511081 gazette

  Some blogs, etc., focus on sending advertisements attached to articles rather than the article itself. In many cases, the creator of such an advertisement article automatically generates an advertisement article using a computer program that obtains and quotes other blog articles in order to create an article with as little effort as possible. If there are a large number of such blog articles called spam blogs, the frequency of appearance of automatically quoted information when statistical processing of all document data is performed for the purpose of grasping the social trends and consumer trends described above. A problem arises in that the frequency increases, and the actual frequency of the topic is greatly deviated.

  In a conventional document processing apparatus, for example, in Patent Document 1, the detection of a document with a possibility of duplication is stopped, and a final action such as deletion of a document is performed by a user of the document processing apparatus. It is necessary to carry out after judging. This is because, in blogs, it is not appropriate to delete all duplicated documents uniformly, because legitimate citations are sometimes used when expressing feelings about primary information such as news articles. This is because one has to scrutinize each article. The technique of Patent Document 1 is intended to detect duplication that occurs unintentionally when managing a database, and cannot deal with a situation where data is intentionally cited or copied by another person.

  Japanese Patent Application Laid-Open No. 2004-228561 assumes a document on the Internet, but is a technique for determining whether or not there is duplication in document units for a set of two documents. For this reason, it is not possible to detect duplication of a part of two or more documents cut out from a document set as a unit. Cannot be excluded.

  The present invention has been made to solve the above-described problems, and a document including a certain percentage or more of a partial character string appearing in another document with respect to a document set is a document generated by automatic citation. By excluding it, it is possible to extract original document data including a document that is properly cited.

  In order to solve the problems described above, the document processing apparatus according to the present invention generates a partial character string that generates a partial character string that forms a part of the character string for each document from character strings included in a plurality of documents. And a partial character string determination unit that determines, among the partial character strings generated by the partial character string generation unit, a partial character string that is not included in a document other than the document in which the partial character string is generated as a unique partial character string And unnecessary document detection means for detecting a document whose ratio between the total number of partial character strings for each document and the number of unique partial character strings determined by the unique partial character string determination means is within a predetermined range as an unnecessary document. It is to be prepared.

  In the document processing method according to the present invention, the partial character string generation unit generates a partial character string that forms a part of the character string for each document from the character strings included in the plurality of documents. And the unique partial character string determination means determines, as the unique partial character string, a partial character string that is not included in a document other than the document in which the partial character string is generated by the partial character string generation means. The unique partial character string determination step and the unnecessary document detection means have a ratio between the total number of partial character strings for each document and the number of the unique partial character strings determined by the unique partial character string determination means within a predetermined range. And an unnecessary document detection step of detecting a document as an unnecessary document.

  Further, the document processing program according to the present invention includes a partial character string generation unit that generates, for each document, a partial character string that forms a part of the character string from character strings included in a plurality of documents. Of the partial character strings generated by the character string generating means, a unique partial character string determining means for determining a partial character string that is not included in a document other than the document generated by itself as a unique partial character string, and for each document The document is made to function as an unnecessary document detection unit that detects a document whose ratio between the total number of partial character strings and the number of unique partial character strings determined by the unique partial character string determination unit is within a predetermined range as an unnecessary document. It is.

  The computer-readable recording medium recording the document processing program according to the present invention generates, for each document, a partial character string that forms part of the character string from character strings included in a plurality of documents. A unique part for determining, as a unique partial character string, a partial character string that is not included in a document other than a document in which the partial character string is generated by the partial character string generating unit and the partial character string generated by the partial character string generating unit An unnecessary document for detecting a document having a ratio between the total number of partial character strings for each document and the number of the unique partial character strings determined by the unique partial character string determining unit as a unnecessary document. A document processing program that functions as detection means is recorded.

  According to the present invention, a partial character string is generated for each document from character strings included in a plurality of documents, a ratio of the unique partial character string unique to the document is obtained for each document, and a ratio of the unique partial character string is low. Is detected as a less useful document centered on the citation of one or more other documents, and these documents can be removed, thereby obtaining a document set suitable for statistical processing. effective.

It is a block diagram which shows an example of the document processing apparatus concerning Embodiment 1 of this invention. It is a figure which shows an example of the detailed storage format of the acquisition document data. It is a figure which shows the detailed storage format of the partial character string table. It is a figure which shows the detailed storage format of the document attribute table. It is a figure explaining the detail of the storage method in the computer system of the partial character string table. 5 is a schematic flowchart showing an operation of an unnecessary document removing unit 3. It is a flowchart which shows the detail of step S1. It is a flowchart which shows the detail of operation | movement of step S16. It is a flowchart which shows the detail of operation | movement of step S19. It is a flowchart which shows the detail of step S2. It is a block diagram of the document processing apparatus which concerns on 2nd Embodiment of this invention. 5 is a flowchart showing the operation of unnecessary URL removal means 14. It is a block diagram of the document processing apparatus which concerns on 3rd Embodiment of this invention. The frequent q word sequence table 90 which stores the q word sequence in the case of q = 3 is shown. 7 is a flowchart showing a part of the operation of the unnecessary document removing unit 3 added in the third embodiment with respect to the first embodiment. It is a flowchart which shows the detail of operation | movement corresponding to specific q (q> 1) among operation | movement of step S3.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an example of a document processing apparatus according to Embodiment 1 of the present invention.
In FIG. 1, a document processing apparatus 1 includes a document acquisition unit 2 that acquires documents on external servers such as servers A to C, and an unnecessary document removal unit 3 that removes unnecessary documents. It consists of document data 5, a partial character string table 6, and a document attribute table 7. The unnecessary document removing unit 3 includes a partial character string generating unit 8 that generates a partial character string from a character string in the document, and a partial character that is uniquely determined without appearing in other documents among the generated partial character strings. A unique partial character string determination unit 9 that determines a column (unique partial character string) and an unnecessary document detection unit 10 that detects an unnecessary document based on a ratio between the number of unique partial character strings and the total number of partial character strings. The document URL list 4 holds a list of addresses, for example, URLs (Uniform Resource Locators) for specifying documents on the external server to be acquired by the document acquisition unit 2. The acquired document data 5 stores a plurality of document data acquired by the document acquisition unit 2. The partial character string table 6 stores an intermediate state of processing by the unnecessary document removing unit 3. The document attribute table 7 stores a processing result by the unnecessary document removing unit 3.

  The document processing apparatus 1 uses a general computer system including hardware such as a CPU, a RAM, a magnetic disk device, and a network interface, and operating system software that controls the hardware, and a program that defines the operation of the CPU. Can be realized. In this case, the document acquisition unit 2 and the unnecessary document removal unit 3 are realized as programs that are read from the magnetic disk device into the RAM and executed by the CPU, and include a document URL list 4, acquired document data 5, and partial character string table 6. The document attribute table 7 is realized as a unique storage area on the RAM or the magnetic disk device.

FIG. 2 is a diagram illustrating an example of a detailed storage format of the acquired document data 5.
The acquired document data 5 includes a plurality of entries, and each entry stores a document URL 51, an acquisition date 52, and a document content 53 of one document. The document URL 51 is one of the addresses of the acquisition target documents stored in the document URL list 4, and the acquisition date / time 52 is the date / time when the document acquisition unit 2 actually acquires the document data. The document content 53 is content data of the document acquired by the document acquisition unit 2. Here, the address of the document on the external server is uniquely identified by the document URL 51, but the document contents acquired from the same document URL 51 at different times may be different. Therefore, in the acquired document data 5, there may be a plurality of entries having the same document URL 51 and different acquisition dates 52. Further, the storage position of each entry in the acquired document data 5 is arbitrary without any particular restriction.

FIG. 3 is a diagram showing a detailed storage format of the partial character string table 6.
The partial character string table 6 includes a plurality of entries, and each entry stores a hash value 61, a document URL 62, an acquisition date and time 63, and a duplication flag 64 related to one partial character string. The partial character string here is a character string of a fixed number of words extracted from the document content 53 of the acquired document data 5 shown in FIG. Here, the number of words is k (for example, k = 5).

  The hash value 61 is a hash value calculated by a known one-way hash function such as CRC (Cyclic Redundancy Code) or SHA-256 (Secure Hash Algorithm 256-bit) for the partial character string corresponding to the entry. Store. Since such a hash value can be considered to have a one-to-one correspondence with the original partial character string, it is used as a key for uniquely identifying an entry in the partial character string table 6.

  Here, in order to maintain the uniqueness of the hash value 61, the storage position of each entry in the partial character string table 6 needs to be uniquely determined based on the hash value 61. However, even for the same partial character string, the acquired document data 5 includes a plurality of entries having different document URLs 51 and acquisition dates 52, and therefore the combination of the document URL 62 and the acquisition date 63 is not uniquely determined. Therefore, a duplicate flag 64 is provided for each entry in the partial character string table 6, and the uniqueness of each entry in the partial character string table 6 is maintained as follows.

  In the document URL 62 and the acquisition date and time 63, one combination of the document URL 52 and the acquisition date and time 53 of the entry of the acquired document data 5 from which the partial character string corresponding to the entry is extracted is designated. As the combination of the document URL 52 and the acquisition date / time 53 to be specified, the document URL 52 and the acquisition date / time 53 of the document having the oldest acquisition date / time among the documents including the partial character string corresponding to the entry are used. Alternatively, when a collection of documents collected without using the document acquisition unit 2 is to be analyzed, identification of the document in which the partial character string corresponding to the entry is first generated is used instead of the document URL 52 and the acquisition date / time 53. Information and the date and time when the document was created / updated shall be used.

  For example, 0 is stored in the duplicate flag 64 when the partial character string corresponding to the entry is included only in the document specified by the combination of the document URL 62 and the acquisition date and time 63. Otherwise, 1 is stored. Is stored.

FIG. 4 is a diagram showing a detailed storage format of the document attribute table 7.
The document attribute table 7 includes a plurality of entries, and each entry stores a document URL 71 of one document, an acquisition date and time 72, a partial character string number 73, a unique partial character string number 74, and a removal flag 75. Each entry in the document attribute table 7 has a one-to-one correspondence with the entry of the acquired document data 5, and the document URL 71 and the acquisition date / time 72 correspond to the document URL 51 and the acquisition date / time 52 of the acquired document data 5, respectively. The storage location of each entry in the document attribute table 7 needs to be uniquely determined by the document URL 71 and the acquisition date 72. The partial character string number 73 represents the total number of partial character strings included in the document content 53 of the acquired document data 5 corresponding to the entry, and the unique partial character string number 74 represents the other partial character strings. This represents the number of partial character strings not included in the document content 53 of the entry. In the removal flag 75, 1 is stored when the document corresponding to the entry is generated by automatic citation and is deemed unnecessary, and 0 is stored otherwise.

FIG. 5 is a diagram for explaining the details of the method of storing the partial character string table 6 in the computer system.
In FIG. 5, the document processing apparatus 1 is shown as a computer system including a CPU 11, a RAM 12, and a magnetic disk device 13.

  Since the partial character string table 6 is randomly accessed based on the hash value 61, it is desirable to store the partial character string table 6 in the RAM 12. However, the number of entries in the partial character string table 6, that is, the types of partial character strings is very large. May not have a capacity for storing the entire partial character string table 6. Therefore, the partial character string table 6 is divided into a plurality of fragments, and the partial character string table piece (0) stored in the RAM 12 and the partial character string table piece (1) stored in the magnetic disk device 13 as necessary. , Partial character string table piece (2), partial character string table piece (3),. . . Is provided. The partial character string table piece (0) on the RAM 12 is realized by a known search structure such as a red-black tree using the hash value 61 as a key or a hash table. In the partial character string table piece (j) (j = 1, 2,...) On the magnetic disk device 13, the entries are arranged in the order of the hash value 61.

  Further, the document attribute table 7 is randomly accessed based on the document URL 71 and the acquisition date / time 72. However, in many cases, it is assumed that the document attribute table 7 can be stored in the RAM 12. This can be realized as a red / black tree or a hash table, and the contents can be written to the magnetic disk device 13 at the end of processing.

  The acquired document data 5 is stored on the magnetic disk device 13 because each entry is written and read only once and has a large capacity.

Next, the operation of the document processing apparatus 1 will be described.
The document acquisition means 2 reads the document URL from the document URL list 4, connects to an external server according to the server name in the URL, and sends the URL according to the well-known HTTP (Hyper-Text Transfer Protocol) to send document data. Request. Next, the document acquisition means 2 receives a response from the external server, and appends the URL as the document URL 51, the current time as the acquisition date 52, and the received content as the document content 53 at the end of the acquired document data 5. This process is repeated for all URLs in the document URL list 4. Acquisition of each document does not need to be performed sequentially, and a plurality of documents may be simultaneously acquired in parallel to shorten the required time.

Next, the operation of the unnecessary document removing unit 3 will be described using a flowchart.
FIG. 6 is a schematic flowchart showing the operation of the unnecessary document removing unit 3.
First, in step S 1, the unnecessary document removing unit 3 processes all entries stored in the acquired document data 5 and sets the partial character string table 6 and the document attribute table 7.
Next, in step S2, the unnecessary document removing unit 3 updates the document attribute table 7 based on each entry in the partial character string table 6, and stores the final processing result for each document.

Hereinafter, details of the operation in step S1 of FIG. 6 will be described using a flowchart.
FIG. 7 is a flowchart showing details of step S1.

  First, in step S <b> 11, the unnecessary document removal unit 3 initializes a variable f representing the number of partial character string table pieces on the magnetic disk device 13 to 0 by the partial character string generation unit 8.

  Next, in step S12, the partial character string generation means 8 selects one unprocessed entry from the acquired document data 5 to be processed, and performs a known morphological analysis process on the document content 53 to obtain a single character string. Is divided into a plurality of word strings. Here, the document data identified by the set of the document URL 51 and the acquisition date 52 of the entry is d, the number of elements of the word string obtained by morphological analysis of the document content 53 of the document d is nd, the word string is W1, W2,. . . , Wnd. The processing order of each entry of the acquired document data 5 is arbitrary. For example, the processing order may be set in order from the first entry.

  Next, in step S13, the partial character string generation unit 8 generates an entry of the document attribute table 7 corresponding to the document URL 51 and the acquisition date 52 of the document d, and the document URL 71 and the acquisition date 72 are respectively set to the document URL 51 and the acquisition date and time. 52, the partial character string number 73 is set to nd−k + 1 (k is the number of words in the partial character string), the unique partial character string number 74 is set to 0, and the removal flag is set to 1.

  Next, in step S14, the partial character string generation means 8 sets a variable i for identifying a partial character string in the document d to 0, and adds 1 to the variable i in a subsequent step S15.

Next, in step S16, the partial character string generation means 8 performs the i-th and subsequent consecutive k word strings (Wi, Wi + 1,...) Among the word strings of the morphological analysis result of the document d according to the procedure described later. ., Wi + k−1), the partial character string table piece is updated based on the hash value Si of the i-th partial character string.
In step S17, the variable i is compared with nd−k + 1. If both are equal, the process proceeds to step S18, and if not, the process returns to step S15.

  Next, in step S18, the partial character string generation means 8 determines whether or not all entries of the acquired document data 5 have been processed. If there are any unprocessed entries remaining, the process returns to step S12 and all of them have been processed. If so, the process proceeds to step S19.

  Finally, in step S19, the partial character string generation means 8 integrates f + 1 partial character string table pieces according to the procedure described later, generates a single partial character string table 6, and ends the process.

Next, details of the operation in step S16 in FIG. 7 will be described using a flowchart.
FIG. 8 is a flowchart showing details of the operation in step S16.
First, in step S101, the partial character string generation means 8 calculates a hash value Si of the i-th partial character string (Wi, Wi + 1,..., Wi + k−1).

  Next, in step S102, the partial character string generation means 8 searches the partial character string table piece (0) and determines whether there is an entry having the hash value 61 equal to Si. If there is an entry, the process proceeds to step S107, and if not, the process proceeds to step S103.

  Next, in step S103, the partial character string generation means 8 checks the current number of entries in the partial character string table piece (0) and determines whether or not the value is less than a predetermined value. If it is less than the predetermined value, the process proceeds to step S106, and if not, the process proceeds to step S104. Here, the predetermined value means an upper limit value of the number of entries of the partial character string table piece (0) that can be stored in the capacity of the RAM 12, for example.

  Next, in step S104, the partial character string generation means 8 cannot generate a new entry in the partial character string table fragment (0), so the partial character string table fragment (0) on the RAM 12 is stored on the magnetic disk device 13. A new partial character string table piece (f + 1) is written, and the partial character string table piece (0) is emptied. At this time, the entries of the partial character string table piece (0) are output in the order of the hash value 61. If the partial character string table piece (0) is realized as a tree structure, this is easy. If the partial character string table piece (0) is realized as a hash table, all the entries are sorted. good.

  Subsequently, in step S105, the partial character string generating means 8 adds 1 to the variable f representing the number of partial character string table pieces on the magnetic disk device 13.

  Next, in step S106, the partial character string generating means 8 assigns a new entry to the partial character string table piece (0), the hash value 61 of the entry is set to Si, and the document URL 62 and the acquisition date and time 63 are currently being processed. The duplication flag 64 is set to 0 for the document URL 51 and the acquisition date 52 corresponding to the document d, and the process is terminated.

  On the other hand, in step S107, the unnecessary document removing unit 3 uses the unique partial character string determining unit 9 to check the document URL 62 and the acquisition date 63 of the existing entry on the partial character string table piece (0), and the document d document It is determined whether the URL 51 and the acquisition date / time 52 match each other. If they match, it indicates that a partial character string is included a plurality of times in the same document, and citation between the documents is not suggested, so the processing ends. Otherwise, the process proceeds to step S108.

  Next, in step S108, the unique partial character string determination means 9 sets the duplication flag of the existing entry to 1, and records that the partial character string exists in a plurality of documents.

  Subsequently, in step S109, the unique partial character string determination unit 9 determines whether or not the acquisition date 63 of the existing entry is newer than the acquisition date 52 of the document d. If it is new, the process proceeds to step S110. Exit.

  Finally, in step S110, the unique partial character string determination unit 9 sets the document URL 62 and acquisition date 63 of the existing entry as the document URL 51 and acquisition date 52 of the document d, and ends the process.

Next, details of the operation in step S19 in FIG. 7 will be described using a flowchart.
FIG. 9 is a flowchart showing details of the operation in step S19.
First, in step S21, the unnecessary document removing unit 3 uses the partial character string generating unit 8 to check each head entry of the partial character string table pieces to be integrated, and obtain the minimum hash value. Next, the first entry is acquired from all the partial character string table pieces having the minimum hash value of the first entry, and the first entry is removed from the partial character string table piece.

  Next, in step S22, the partial character string generation means 8 determines whether or not the acquired entry is single. If it is single, the process proceeds to step S26, and if not, the process proceeds to step S23.

  Next, in step S23, the unnecessary document removal unit 3 determines whether the document URL 62 and the acquisition date 63 are all the same in the acquired plurality of entries by the unique partial character string determination unit 9, and if they are the same, the process proceeds to step S25. Otherwise, go to step S24.

Next, in step S24, the unique partial character string determination means 9 clearly shows that a plurality of documents correspond to the same hash value, so the duplication flag 64 is set to 1, a hash value 61, a document URL 62, And the acquisition date and time 63 are set and output as the hash value 61, the document URL 62, and the acquisition date and time 63 of the entry with the smallest acquisition date and time 63 in the acquired entries, and the process proceeds to step S27.
If so, the process proceeds to step S24.

  On the other hand, in step S25, the unique partial character string determination means 9 determines that the duplicate entry 64 has a duplicate flag 64 if all of the obtained entries correspond to the same document. 64 is set to 1; otherwise, the duplication flag 64 is set to 0, and the hash value 61, document URL 62, and acquisition date 63 are set to the hash value 61, document URL 62, and acquisition date 63 of the acquired arbitrary entry, respectively. Output, and the process proceeds to step S27.

  In step S26, the partial character string generation means 8 outputs the content of the acquired single entry itself, and proceeds to step S27.

  Finally, in step S27, the partial character string generation means 8 determines whether all the partial character string table pieces to be integrated are empty. If empty, the process ends. If not, the process returns to step S21. Repeat the process.

In the process of FIG. 9, when the number of partial character string table pieces (f) is very large, a predetermined number (for example, 16 pieces) of partial character string table pieces are integrated into an intermediate partial character string table piece. Next, it is also possible to apply by gradually reducing the number of partial character string table pieces, such as integration of intermediate partial character string table pieces. Thereby, it is possible to suppress a decrease in input / output performance with respect to the magnetic disk device 13.
The above is the detailed operation of step S1 in FIG.

Next, the detailed operation of step S2 in FIG. 6 will be described using a flowchart.
FIG. 10 is a flowchart showing details of step S2.

First, by repeating step S31 to step S32, all entries in the partial character string table 6 are processed in order.
First, in step S31, the unnecessary document removal unit 3 selects one of the unprocessed entries in the partial character string table 6 as a processing target by the unnecessary document detection unit 10, and uses the document URL 62 and the acquisition date and time 63 of the entry. 1 is added to the number of unique partial character strings 74 for the entry of the designated document attribute table 7. Here, even when the duplication flag of the entry is set to 1, in the document having the oldest acquisition date in the duplicate document, the partial character string corresponding to the entry is treated as a unique partial character string. This is because if all duplicate documents are to be removed, the citation document is also removed, and it is considered that the content of the document does not exist.

  Note that, when a collection of documents collected without using the document acquisition unit 2 is to be analyzed, since there is no information on the acquisition date and time described above, in the duplicate document based on the date and time when the document was created / updated A partial character string corresponding to the entry may be treated as a unique partial character string in a document in which an overlapping partial character string is first generated.

  Next, in step S32, the unnecessary document detection unit 10 checks whether there is an unprocessed entry. If all entries in the partial character string table 6 have been processed, the process proceeds to step S33. If not, the process returns to step S31. .

Subsequently, by repeating steps S33 to S34, all entries in the document attribute table 7 are processed in order.
First, in step S33, the unnecessary document detection unit 10 selects one of the unprocessed entries in the document attribute table 7 as a processing target, and the ratio of the number of unique partial character strings 74 to the number of partial character strings 73 is equal to or greater than a predetermined value. If it is (for example, 60% or more), the removal flag 75 of the entry is set to 0.

  Next, in step S34, the unnecessary document detection unit 10 checks whether there is an unprocessed entry. If all entries in the document attribute table 7 have been processed, the process ends. If not, the process returns to step S33.

  In the above description, it is assumed that the document attribute table 7 exists on the RAM 12 and can be efficiently accessed randomly. However, when the data amount of the document attribute table 7 exceeds the capacity of the RAM 12, You can do that. That is, in step S13 in FIG. 7, a new entry in the document attribute table 7 is added to the end of the document attribute table 7 on the magnetic disk device 13, and the entry in the document attribute table 7 is added to the document prior to the processing in FIG. It sorts in order of URL71 and acquisition date 72, and it is set as the input side document attribute table. Similarly, the entries in the partial character string table 6 are also sorted in the order of the document URL 62 and the acquisition date 63.

  In step S31, the entry on the input side document attribute table is processed from the beginning, and as long as the entry having the same document URL and acquisition date / time is read from the partial character string table 6, the number of unique partial character strings in the document attribute table entry is set to 74. 1 is added, and when an entry whose document URL and acquisition date and time do not match is read from the partial character string table 6, the document attribute table entry is added to the output document attribute table.

  In the above description, the partial character string is a string of k words. However, the partial character string may be a string of k ′ characters without performing word division by morphological analysis processing.

  As described above, according to the first embodiment of the present invention, partial character strings are generated for each document from character strings included in a plurality of documents, and the ratio of unique partial character strings unique to the document is obtained for each document. Detecting a low percentage of unique substrings as less useful documents centered on citations of one or more other documents, and enabling these documents to be removed for statistical processing There is an effect that a suitable document set can be obtained.

  In addition, by treating partial character strings appearing in a plurality of documents as unique partial character strings in the document with the oldest acquisition date and time, there is an effect of preventing the removal of a document that is highly likely to be a citation source. .

  Further, by allowing the presence of partial character strings that overlap with other documents up to a certain ratio, there is an effect of preventing the removal of documents that have been properly quoted and added a unique description.

Embodiment 2. FIG.
In the first embodiment described above, it is suitable for statistical processing by removing a document with a low ratio of unique partial character strings as a less useful document centered on citation of one or more other documents. The document processing apparatus capable of obtaining a document set has been described. Next, when iteratively obtaining a document from the same document URL, a document URL that is highly likely not to be statistically useful is obtained from the acquisition target. A second embodiment relating to a document processing apparatus that eliminates unnecessary document acquisition and improves the efficiency of document acquisition from an external server will be described.

FIG. 11 is a block diagram of a document processing apparatus according to the second embodiment of the present invention.
In FIG. 11, the document processing apparatus 1 to the unnecessary document detecting means 10 correspond to the components having the same numbers in FIG. 1, and the unnecessary URL removing means 14 is added to the first embodiment in the second embodiment. It is the part which was done. The unnecessary URL removing unit 14 operates following the operation of the unnecessary document removing unit 3.

FIG. 12 is a flowchart showing the operation of the unnecessary URL removing unit 14.
The unnecessary URL removing unit 14 processes the entries having the same document URL 71 together in order for all the entries in the document attribute table 7.

  First, in step S41, the unnecessary URL removing unit 14 selects one unprocessed document URL 71 in the document attribute table 7, and acquires all entries corresponding to the document URL 71. If the number of acquired entries exceeds a predetermined value and the ratio (removal rate) of entries in which the removal flag 75 is set to 1 in the acquired entries exceeds a predetermined value, only duplicate documents can be obtained from the document URL. Since the possibility is high, the document URL is deleted from the document URL list 4 so that the document is not acquired from the next time.

  Subsequently, in step S42, the unnecessary URL removing unit 14 determines whether all document URLs in the document attribute table 7 have been processed. If not, the process returns to step S41.

  Here, the removal rate is obtained for a plurality of versions of the same document URL, but for each host name in the URL or an upper domain name (for example, http://blog.foo.com/) as the URL The removal rate may be obtained for each of the foo.com, etc.), and URL groups in which many duplicate documents are generated may be collectively removed from the document URL list 4.

  As described above, according to the second embodiment of the present invention, when a document is repeatedly acquired from the same document URL, a document URL that is highly unlikely to be a statistically useful document is removed from the acquisition target. Thus, there is an effect that it is possible to avoid the acquisition of unnecessary documents and to improve the efficiency of document acquisition from an external server.

Embodiment 3 FIG.
In the second embodiment described above, a document URL that avoids unnecessary document acquisition and improves document acquisition efficiency from an external server by removing a document URL that is highly unlikely to obtain a statistically useful document from the acquisition target. The processing apparatus has been described. Next, a document that can detect a document including an unnaturally high number of frequently occurring words and phrases and remove the document can obtain a document set suitable for statistical processing. Embodiment 3 regarding a processing apparatus is shown.

FIG. 13 is a block diagram of a document processing apparatus according to the third embodiment of the present invention.
In FIG. 13, the document processing apparatus 1 to the unnecessary document detection means 10 correspond to the components having the same numbers in FIG. 1, and the longest frequent partial character string table 15 is the third embodiment compared to the first embodiment. It is a part added by.

  First, the storage format of the longest frequent partial character string table 15 will be described in detail. In the partial character string table 6, a word string of k fixed words is a partial character string. However, the longest frequent partial character string table 15 includes a partial character string from one word to a Q word string. (For example, Q = 10). Here, if a partial character string appearing in a predetermined number or more (for example, 500 or more) of partial character strings included in the document content 53 of the acquired document data 5 is defined as a frequent partial character string, the longest frequent part is defined. The character string table 15 is a table for storing the longest frequent partial character string among the frequent partial character strings.

  Hereinafter, the longest frequent partial character string is expressed as the longest frequent q word string (q is the number of words). The longest frequent q word string indicates a string of q words (q word string) appearing in a predetermined number or more of documents, and the longest frequent q + 1 word string including the q word string does not exist.

In order to generate such a longest frequent partial character string table 15, a frequent q word string table that stores candidates for the longest frequent partial character string as a q word string is used.
FIG. 14 shows a frequent q word string table 90 for storing q word strings when q = 3.
The frequent q word string table 90 includes a word string 91 and the number of appearances 92. The frequent q word string table 90 is arranged on the RAM 12 and has a structure that can be searched using the word string 91 as a unique key. The appearance count 92 holds the number of documents in which the corresponding word string 91 appears.

Next, the operation of the unnecessary document removing unit 3 using the longest frequent partial character string table 15 will be described using a flowchart.
FIG. 15 is a flowchart showing a part of the operation of the unnecessary document removing unit 3 that is added to the first embodiment in the third embodiment.
Steps S3 to S5 are executed subsequent to steps S1 and S2 in FIG.

  First, in step S3, the unnecessary document removing unit 3 causes the unnecessary document detecting unit 10 to perform q = 1, 2, 3,. . . , Q, the number of appearing documents of q word strings (words if q = 1) included in the document content 53 of the acquired document data 5 is obtained, and q word strings appearing in a predetermined number or more of documents are Store in the word string table 90.

  Subsequently, in step S4, the unnecessary document detection means 10 determines that q = Q, Q-1,. . . 2, 1, partial character string groups included in the q word string of each entry of the frequent q word string table, that is, q-1 word string, q-2 word string,. . . For a word, the corresponding frequent word string table entry is deleted. The entries remaining in each frequent q word string are the longest frequent q word strings, and these are the longest frequent partial character string table.

  Finally, in step S5, the unnecessary document detection means 10 refers to the longest frequent partial character string table generated in step S4, and the type of the longest frequent partial character string simultaneously included in the sentence content 53 of the acquired document data 5 And the removal flag 75 is set to 1 for an entry in the document attribute table 7 corresponding to a document including a predetermined type or more (for example, 7 types or more) of the longest frequent partial character string.

Next, the operation in step S3 in FIG. 15 will be described using a flowchart.
FIG. 16 is a flowchart showing details of an operation corresponding to a specific q (q> 1) in the operation of step S3.

  First, in step S 51, the unnecessary document removing unit 3 selects an unprocessed document d from the acquired document data 5, and the word strings W 1, W 2,. . . , Wnd. This process can be realized by repeating the same morphological analysis process as in step S12 of FIG. 7 or by saving the previous execution result of step S12 and reusing it.

  Next, in step S52, the unnecessary document detection means 10 picks up unprocessed q word strings (Wi, Wi + 1,..., Wi + q-1) among the word strings as processing targets.

  Next, in step S <b> 53, the unnecessary document detection unit 10 includes all the q words Wj (j = i, i + 1,..., I + q−1) in the frequent word table (frequently used word string table) 90. Determine whether or not. If it exists, the process proceeds to step S54. If not, the q word string cannot occur frequently, and the process is skipped to step S56.

  Subsequently, in step S54, the unnecessary document detection unit 10 determines the q-1 word string (Wi, Wi + 1,..., Wi + q-2) excluding the right end of the q word strings and the q− without the left end. It is determined whether any one word string (Wi + 1,..., Wi + q−1) exists in the frequent q−1 word string table 90. If it exists, the process proceeds to step S55. If not, the q word string cannot occur frequently, and the process is skipped to step S56.

  Next, in step S55, the unnecessary document detection means 10 adds 1 to the appearance count 92 in the frequent q word string table entry corresponding to the q word string (Wi, Wi + 1,..., Wi + q−1). .

  Next, in step S56, the unnecessary document detection unit 10 determines whether all the q word strings of the document d have been processed. If they have been processed, the process proceeds to step S57. Otherwise, the process returns to step S52. Repeat the process.

  Subsequently, in step S57, the unnecessary document detection unit 10 determines whether all the documents of the acquired document data 7 have been processed. If all the documents have been processed, the process proceeds to step S58. If not, the process returns to step S51. Repeat the process.

  Finally, in step S58, the unnecessary document detection unit 10 sequentially examines each entry in the frequent q word string table 90, and deletes an entry whose appearance count 92 is less than a predetermined value. Entries that remain without being deleted are frequent q word strings.

  As described above, according to the third embodiment of the present invention, a document set suitable for statistical processing is detected by detecting a document that contains an unnaturally high number of frequently occurring words and phrases and removing this document. There is an effect that can be obtained.

  Authors who automatically generate spam blogs can easily identify these frequently occurring phrases as topic words using search query rankings of search engines published on the Internet. In order to automatically generate a Web page that can be easily searched, such a topic word may be used as a search target keyword set in the Web page. In this case, a random character string may be used for the text of the Web page instead of automatically quoting another document. Even in such a case, according to the third embodiment of the present invention, it is possible to remove the automatically generated document as described above by using a frequent word (topic word) set as a search target keyword. It becomes possible.

DESCRIPTION OF SYMBOLS 1 Document processing apparatus, 2 Document acquisition means, 3 Unnecessary document removal means, 4 Document URL list, 5 Acquisition document data, 6 Partial character string table, 7 Document attribute table, 8 Partial character string production | generation means, 9 Unique partial character string determination Means, 10 Unnecessary document detection means, 11 CPU, 12 RAM, 13 Magnetic disk device, 14 Unnecessary URL removal means, 15 Longest frequent character string table.

Claims (11)

Partial character string generating means for generating, for each document, a partial character string forming a part of the character string from character strings included in a plurality of documents;
Unique partial character string determination means for determining a partial character string that is not included in a document other than the document in which the partial character string is generated among the partial character strings generated by the partial character string generation means;
A document comprising unnecessary document detection means for detecting, as an unnecessary document, a document whose ratio between the total number of partial character strings for each document and the number of unique partial character strings determined by the unique partial character string determination means is within a predetermined range. Processing equipment. The partial character string generated by the partial character string generating means, the identification information of the document in which the partial character string is first generated, and the duplication indicating whether or not the partial character string is duplicated between different documents A partial character string table that stores flags in association with each other,
In the case where the unique partial character string judging means indicates that the duplication flag stored in the partial character string table is duplicated, the partial character string is regarded as a unique partial character string in the first generated document. The document processing apparatus according to claim 1, wherein the document processing apparatus determines that the document is not a unique partial character string in other documents. A document acquisition means for acquiring a document on an external server specified by a URL (Uniform Resource Locator) via a network, and storing the acquired document data together with the URL and the acquisition date;
The partial character string generated by the partial character string generating means, the URL of the document with the oldest acquisition date and time among the documents including the partial character string, and the document in which the partial character string is different from the acquisition date and time A partial character string table that stores a duplication flag indicating whether or not there is a duplication flag,
When the unique partial character string determination means indicates that the duplication flag stored in the partial character string table is duplicated, the partial character string is determined as a unique partial character string in the document with the oldest acquisition date and time. The document processing apparatus according to claim 1, wherein the document processing apparatus determines that the document is not a unique partial character string in other documents. A document URL list for storing the URL for acquiring a document by the document acquisition unit as an acquisition target URL;
The document processing apparatus according to claim 3, further comprising: an unnecessary URL removing unit that deletes the acquisition target URL stored in the document URL list based on the URL of the unnecessary document detected by the unnecessary document detecting unit. The unnecessary URL removing unit stores the URL of the unnecessary document in the document URL list when the unnecessary document detected by the unnecessary document detecting unit is repeatedly detected as an unnecessary document at a predetermined number or more of acquisition dates and times. The document processing apparatus according to claim 4, wherein the document processing apparatus is deleted from the acquired URL to be acquired. When the unnecessary URL removing unit detects a predetermined ratio or more as unnecessary documents among the documents having URLs whose host names or higher domain names of the URLs of the unnecessary documents detected by the unnecessary document detecting unit match. 5. The document processing apparatus according to claim 4, wherein all URLs having the same host name or higher domain name are deleted from the acquisition target URLs stored in the document URL list. The partial character string generation unit stores the partial character string table in a RAM (Random Access Memory). When the partial character string table reaches a predetermined capacity corresponding to the capacity of the RAM, the partial character string table is stored in the RAM. After writing a fragment of a partial character string table to a magnetic disk device, emptying the partial character string table stored in the RAM, and detecting the unnecessary document for all of the acquired document data, the RAM and the magnetic 7. The document processing apparatus according to claim 2, wherein fragments of the partial character string table stored in a disk device are integrated based on the partial character string. The unnecessary document detection means is based on the number of occurrences of partial character strings within a predetermined length, and the longest frequent character string not included in a longer frequent partial character string among frequent partial character strings appearing in a predetermined number or more of documents. 8. The document processing apparatus according to claim 1, wherein a longest frequent partial character string which is a frequent partial character string is obtained, and a document including a predetermined type or more of the longest frequent partial character string is detected as an unnecessary document. A partial character string generation means for generating, for each document, a partial character string that forms a part of the character string from character strings included in a plurality of documents;
A unique partial character string determination unit is configured to determine, as a unique partial character string, a partial character string that is not included in a document other than the document in which the partial character string is generated by the partial character string generation unit. A substring determination step;
Unnecessary document detecting means for detecting as an unnecessary document a document whose ratio between the total number of partial character strings for each document and the number of unique partial character strings determined by the unique partial character string determining means is within a predetermined range A document processing method comprising a detection step. Computer
Partial character string generating means for generating, for each document, a partial character string forming a part of the character string from character strings included in a plurality of documents;
Unique partial character string determination means for determining a partial character string that is not included in a document other than the document in which the partial character string is generated among the partial character strings generated by the partial character string generation means;
To function as an unnecessary document detection unit that detects a document in which the ratio between the total number of partial character strings for each document and the number of unique partial character strings determined by the unique partial character string determination unit is within a predetermined range as an unnecessary document Document processing program. Computer
Partial character string generating means for generating, for each document, a partial character string forming a part of the character string from character strings included in a plurality of documents;
Unique partial character string determination means for determining a partial character string that is not included in a document other than the document in which the partial character string is generated among the partial character strings generated by the partial character string generation means;
To function as an unnecessary document detection unit that detects a document in which the ratio between the total number of partial character strings for each document and the number of unique partial character strings determined by the unique partial character string determination unit is within a predetermined range as an unnecessary document A computer-readable recording medium on which a document processing program is recorded.

Images