JP2000066945A - Document collection system, device and method and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To collect updated documents without checking each document disclosed on an Internet. SOLUTION: A Web server 100 newly prepares a page catalog 104 for correspondingly registering the URL and changing time of a changed document. In a document collection device 110, the URLs and changing time of documents collected up to now are registered in an URL data base 114. In the case of newly collecting a document, a page catalog is requested to the Web server 100 and the page catalog 115 received by the request is managed by a page catalog management part 116. Then a changing time comparing part 117 successively compares the changing time in the page catalog 115 with the changing time in the data base 114 in each same URL, and when the updating time in the page catalog 115 is later, its corresponding document is requested from the Web server 100.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to World Wi
The present invention relates to a document collection system, an apparatus, and a method for collecting documents required when creating a document index in a search service such as de Web, and a recording medium on which a program for recording the documents is recorded.

[0002]

2. Description of the Related Art As a means for transmitting information on the Internet, World, a distributed hypermedia system, is used.
Wide Web (hereinafter, simply referred to as “Web”) is widely used. A plurality of Web servers existing in the Web hold documents and publish the held documents. Here, on the Web, the location of each document is represented by a format identifier called URL (Uniform Resource Locators), and the transfer of each document is performed according to a procedure called HTTP (HyperText Transfer Protocol).

[0003] The number of documents published on the Web is enormous. So, in the Web,
In addition, there are a large number of search services called search engines that allow users to search for information they need from a huge number of documents. In these search services, documents necessary for creating an index used for search are collected from a Web server. For example, as described in JP-A-8-77057, "Hypermedia Search Method" A simple document collection device is used.

FIG. 5 is a block diagram showing a configuration of a conventional document collection device. Document collection device 30
0 is W connected via the Internet 330
A document is collected from the web server 310. The communication processing unit 301, the document processing unit 302,
RL database 303 and collected documents 30
And 4.

[0005] In the document collection device 300, the communication processing unit 301 operates according to HTTP to the Web server 31.
0 requests the transmission of the document, and receives a response (including the document according to the request). The received document 304 is sent to the document processing unit 302.
From which the modification time of the document and the link destination URL of the hyperlink described in the document are extracted. Then, the modification time and the link destination URL of these documents are registered in the URL database 303.

When the contents of a document published on the Web are changed, it is necessary to update the index of the search service. Therefore, the document collection device 300 needs to re-collect the changed documents. Here, according to the HTTP, the Web server 3
It is possible to specify a time when requesting a document from the server 10, and to specify that only documents changed after that time are actually transmitted.

[0007]

However, conventionally,
In the document collection device 300, there is no means to know in advance which document held by the Web server 310 has been changed prior to a document transmission request. For this reason, the conventional document collection device 300 cannot re-collect the documents without requesting the Web server 310 to transmit all the documents including those that have not been changed one by one. .

This will be specifically described with reference to the sequence diagram of FIG. For example, 200 documents from URL-1 to URL-200 are published on the Web server 310, and among these 200 documents,
Assume that only three of URL-17, URL-125, and URL-144 have been changed since the previous collection. In such a case, only the collection of URL-17, URL-125, and URL-144, that is, only the communication of the sequences 402, 404, and 406 in FIG.
Indispensable communication.

However, since the conventional document collection device 300 has no means for checking the change of the document, all the documents from the URL-1 to the URL-200 are sequentially converted into the document including the unchanged one. Had to request transmission. Therefore, conventionally, a request for transmitting a document that has not been changed, that is, communication in the range indicated by the sequences 401, 403, 405, and 407 in the figure, is also performed, and the resources of the Web server 310 and the Internet 330 are wasted. Was.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems of the prior art, and it is an object of the present invention to collect changed documents without examining documents published on a network one by one. It is an object of the present invention to provide a document collection system, an apparatus and a method capable of knowing the collection time of those documents and a recording medium recording a program therefor.

[0011]

In order to achieve the above object, a document collection system according to a first aspect of the present invention is a server which is connected to each other via a network and holds a document to be published via the network. And a document collection device that collects the document from the server via the network, wherein the server changes the document to be held or creates a document to be newly held when the document to be held is changed. Catalog creation means for creating a catalog for registering identification information in association with a change time indicating a time of change or creation, and in response to a request from the document collection device, the document or the catalog via the network. Server transmitting means for transmitting to the server. The document collection device,
A catalog requesting means for requesting the catalog created by the catalog creating means to the server via the network and transmitting the catalog to the server transmitting means; and transmitting the catalog from the server transmitting means in response to the request by the catalog requesting means. Catalog receiving means for receiving the received catalog, catalog managing means for managing the catalog received by the catalog receiving means, and requesting the document held by the server to the server via the network by designating its identification information. A document requesting means for transmitting the document to the server transmitting means; and a document receiving means for receiving, in response to the request by the document requesting means, the document transmitted from the server transmitting means together with the identification information of the document and the modification time. And the document receiving means A document information registration unit for registering the received document identification information and the modification time in association with each other; a modification time of the document registered in the document information registration unit; Document change time, the same identification information, the same, the change time comparing means for sequentially comparing, the document request means,
As a result of the comparison by the change time comparing means, the server requests the server to transmit a document in which the change time of the document in the catalog is determined to be newer.

[0012] In the above document collection system, the document requesting means transmits to the server only those documents whose change time of the document in the catalog is determined to be newer than the change time of the document in the document information registration means. If requested, all documents that have changed since the last collection can be collected. On the other hand, in order to obtain a catalog to be performed prior to collection of documents, it is only necessary to request the server to transmit the catalog once. Therefore, in general, the number of requests from the document collection device to the server can be significantly reduced as compared with the conventional case, so that the load on the network and the server can be reduced, and the processing time for collecting the document can be reduced. Can also be shortened.

[0013] To achieve the above object, a document collection device according to a second aspect of the present invention is a document collection device that collects a document from a server that publishes the document via a network via the network. Requesting the server via the network a catalog for registering the identification information of the document held by the server and its modification time in association with each other,
Catalog requesting means for transmitting from the server, catalog receiving means for receiving a catalog transmitted from the server in response to a request by the catalog requesting means, and catalog managing means for managing the catalog received by the catalog receiving means Requesting the server via the network by designating the identification information of the document held by the server, and requesting the server to transmit the document; and responding to the request by the document requesting means, Documents sent from
Document receiving means for receiving the document identification information and the modification time together with the document information, document information registration means for registering the document identification information and the modification time received by the document receiving means in association with each other, and registration in the document information registration means And a change time comparing means for sequentially comparing the change time of the document being changed and the change time of the document in the catalog managed by the catalog management means with those having the same identification information. The request means requests the server to transmit a document determined as a result of the comparison by the change time comparison means that the change time of the document in the catalog is newer.

In the above-mentioned document collection apparatus, when the identification information and the change time of the document are registered in the document information registration means, the catalog management means deletes the information corresponding to the registered information from the catalog. It may be provided with a deletion means.

In this case, the document information registering means, when the deleting means deletes all of the identification information and the change time of the document to be deleted, the identification information of the document remaining in the catalog and the change time. May be further registered.

In the document collection device, the identification information of the document may be, for example, a URL.

To achieve the above object, a document collection method according to a third aspect of the present invention is a document collection method for collecting a document from a server that publishes the document via a network via the network. Requesting the server via the network a catalog for registering the identification information of the document held by the server and its modification time in association with each other,
A catalog requesting step for transmitting from the server, a catalog receiving step for receiving a catalog transmitted from the server in response to the request in the catalog requesting step, and a catalog management for managing the catalog received in the catalog receiving step Requesting the document held by the server from the server via the network by specifying the identification information of the document, and requesting the server to transmit the document; and responding to the request in the document requesting step, A document receiving step of receiving the document transmitted from the server together with the identification information and the modification time of the document;
A document information registration step of registering the identification information and the modification time of the document received in the document reception step in association with each other; a modification time of the document registered in the document information registration step; And a change time comparison step of sequentially comparing the change times of the documents in the catalog with those having the same identification information,
The document requesting step requests the server to transmit a document whose change time is determined to be newer in the catalog as a result of the comparison in the change time comparison step.

In order to achieve the above object, a recording medium according to a fourth aspect of the present invention is a recording medium for recording a program for collecting a document via a network from a server which publishes the document via a network. A catalog requesting step of requesting the server to register a catalog in which the identification information of a document held by the server and its modification time are associated with each other via the network and transmitting the catalog from the server; A catalog receiving step of receiving a catalog transmitted from the server in response to the request in step, a catalog managing step of managing the catalog received in the catalog receiving step, and identification information of a document held by the server. To specify the service via the network. Requesting a document to be transmitted from the server; and a document receiving step of receiving, in response to the request in the document requesting step, a document transmitted from the server together with the identification information of the document and a modification time. A document information registration step of registering the identification information and the modification time of the document received in the document reception step in association with each other; a modification time of the document registered in the document information registration step; And performing a change time comparison step of sequentially comparing the change times of the documents in the managed catalog with those having the same identification information, wherein the document requesting step includes the step of comparing in the change time comparison step. Result, the catalog The transmission of the document who is determined to new modification time documents characterized by recording a program for requesting the server.

[0019]

Embodiments of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is a block diagram showing the configuration of a document collection system according to this embodiment. As illustrated, the document collection system includes a Web server 100 and a document collection device 110 connected to each other via the Internet 130. We
b server 100 and document collection device 110
The browsing software 120 operates and the Internet 1
30 can be accessed from a connected personal computer.

The Web server 100 includes a server communication processing unit 101, a document 102, a page catalog creation processing unit 103, and a page catalog 104.

The server communication processing unit 101 includes a processor that operates under program control and a communication device that performs communication according to HTTP, receives a request from the document collection device 110 or the viewing software 120, and transmits a response to the request. I do. The response transmitted by the server communication processing unit 101 includes a document 1 according to the content of the corresponding request.
02 and the page catalog 104 are included.

When there is a change in the document 102, the page catalog creation processing unit 103 creates a new page catalog 104 in which the change time corresponding to the URL of the document is updated.

Note that the document 102 and the page catalog 104 are stored in a magnetic disk device or the like in the Web server 100 and become components of the Web server 100.

The document collection device 110 includes a communication processing unit 111, a document 112, a document processing unit 113, a URL database 114, a page catalog 115, a page catalog management unit 116, and a change time comparison unit 117. Have been.

The communication processing unit 111 includes a processor that operates under program control and a communication device that performs communication according to HTTP, and transmits the page catalog 104 or the document 102 to the document processing device 110 by W.
Request to the web server 100. The communication processing unit 111 also receives a response (including the page catalog 104 or the document 102) transmitted from the server communication processing unit 101 to the request, and sets the document collection device 110 as the page catalog 115 or the document 112.
Hold in.

The document processing unit 113 receives the document 112 received by the communication processing unit 111 and stores the document 112 therein.
, And the link destination URL of the hyperlink described in the document 112 is extracted and registered in the URL database 114. Document processing unit 113
Is the U remaining in the page catalog 115 as described later.
A process for registering the RL in the database is also performed. The URL database 114 stores and manages the modification time of the document 112 and the link destination URL.

The page catalog management unit 116 manages the page catalog 115 and reads from the page catalog 115 the change time of the document corresponding to the URL requested by the change time comparison unit 117. Change time comparison unit 11
Reference numeral 7 compares the change time of the document read by the page catalog management unit 116 with the change time of the document registered in the URL database 114.

The document 112 and the page catalog 115 are stored in a magnetic disk device or the like in the document collection device 110, and the document collection device 11
0.

FIG. 2 shows an example of page catalogs 104 and 115 in the document collection system of FIG. As shown, the page catalogs 104, 11
Reference numeral 5 stores a document URL 501 and its modification time (including date) 502 in association with each other in a list format, and is created by the page catalog creation unit 103 of the Web server 100 as described above. 110 is managed by the page catalog management unit 116.

The operation of the document collection system according to this embodiment will be described below. Here, in the Web server 100, when the held document 102 is changed, the page catalog creation processing unit 103 updates the change time 502 associated with the URL 501 of the document, and sets a new page catalog. Create 104. Also, when a new document 102 is created, the page catalog creation processing unit 103 changes the URL 501 and the change (creation) time 502.
Create a new page catalog in which is registered.

FIG. 3 shows the document collection device 11 of FIG.
0 is a flowchart showing the processing executed. The processing of this flowchart is executed for each Web server 100 that is connected via the Internet 130 and that is to collect documents, periodically by a timer or the like of the document collection device 110 or irregularly in accordance with an instruction of an operator. Things.

When the processing starts, the communication processing unit 111
A page catalog request for requesting transmission of the page catalog 104 to the document processing device 110 is transmitted to the Web server 100 (step 201). And
The communication processing unit 111 receives the page catalog response (referred to as a response including the page catalog 104) sent from the Web server 100, and causes the page catalog management unit 116 to manage the response as a page catalog 115 (step 20).
2).

Next, the change time comparison unit 117 stores the URL of the document to be processed this time in the URL database 114 (the URL held in the Web server 100 with which the document collection device 110 is communicating).
Then, it is determined whether or not unprocessed data remains (step 203).

An unprocessed UR is stored in the URL database 114.
When it is determined that L remains, the change time comparison unit 11
7 reads one URL from the unprocessed URLs remaining in the URL database 114 and sets it as URL-n (step 204). Next, the change time comparison unit 117
Sends an instruction to the page catalog management unit 116 to check whether the URL-n exists in the page catalog 115 (step 205).

As a result of the inspection, U
If RL-n does not exist, the process proceeds to step 207. On the other hand, as a result of the inspection,
If L-n exists, the change time comparison unit 117
The change time in the page catalog 115 corresponding to the URL-n is compared with the change time in the URL database 114 to determine whether the change time in the page catalog 115 is newer (step 206).

The change time in the page catalog 115 is not newer, that is, both change times are the same (the change time in the URL database 114 is newer except when a failure occurs). When the determination is made, the process returns to step 203. On the other hand, when it is determined that the change time in the page catalog 115 is newer,
Proceeding to step 207, the communication processing unit 111 transmits a request for transmitting the document 102 corresponding to the URL-n to the Web server 100 (hereinafter, referred to as a request for URL-n) (step 207).

Then, the communication processing unit 111 transmits the URL-n
Transmitted from the Web server 100 in response to the request
Response including a document corresponding to RL-n (hereinafter, U
RL-n response) and the received URL-n
Is stored in the document collection device 110 as the document 112 (step 208).

When the new document 112 is held, next, the document processing unit 113 extracts the modification time of the document 112 and the link destination of the hyperlink described in the document 112, and
It is registered in the L database 114 (step 209).
Then, the completed URL recorded in the page catalog 115 by the page catalog management unit 115.
-N and the corresponding change time are deleted (step 210). Then, returning to the processing of step 203, U
It is again determined whether or not any unprocessed URL remains in the RL database 114 as the URL of the document to be processed this time.

In step 203, the URL database 11
When it is determined that there is no unprocessed URL remaining in 4,
The control is transferred to the document processing unit 113, and the document processing unit 113 causes the page catalog management unit 116 to check whether there is a URL remaining in the page catalog 115 without being deleted (step 211).

If no URL remains in the page catalog 115 as a result of the inspection by the page catalog management unit 116, the process of this flowchart is terminated as it is. On the other hand, if the URL remains in the page catalog 115, the document processing unit 113 determines that the URL is a URL newly created in the Web server 100 between the previous processing and the current processing, and The URLs registered are all registered in the URL database 114 (step 212). Then, the process of this flowchart ends.

With the above, the collection of the document 112 by the data collection device 110 from the Web server 100 is completed. In practice, a document index for a search service is created from the document 112 thus collected. .

Hereinafter, the operation of the document collection system according to this embodiment will be described in detail with reference to a specific example. Here, URL-1 is used as the Web server 100.
Assuming that 200 documents are published from to 200, and only three of these 200 documents, URL-17, URL-125, and URL-144, are the contents since the last collection. Has been changed.

FIG. 4 is a sequence diagram for explaining the operation in this example. First, the document collection device 1
10 transmits a page catalog request to the Web server 100. Then, document collection device 110 receives a response including the page catalog transmitted from Web server 100 in response thereto (sequence 701).

Next, the document collection device 110 compares the change time in the page catalog with the URL database, and sends the Web server 100 a request for a URL-17 with the newer change time in the page catalog. Then, document collection apparatus 110 receives a response including the document of URL-17 transmitted from Web server 100 thereto (sequence 702).

Subsequently, the document of URL-125 and the document of URL-140 are collected by the request of URL-125 and the response thereto (sequence 703) and the request of URL-144 and the response thereto (sequence 704). Collected in device 110. As described above, the changed documents can be collected again by the communication of the number of documents changed after the previous collection time + 1 (transmission of the request and reception of the response).

As described above, in the document collection system according to this embodiment, the Web server 10
0 creates a page catalog 104, which can be collected by the document collection device 110. Therefore, the document collection device 110 can know the modification time of each document without requesting each document one by one. .

When collecting a document, the document collection device 110 first requests the Web server 100 to transmit the page catalog 104 and, based on the change time of the document in the page catalog 115 acquired in response to the request. Requesting the Web server 100 to transmit only the document 102 that has changed since the previous collection. That is, the document collection device 110
Only needs to request the Web server 100 for documents that have changed since the previous collection.

On the other hand, the page catalog 104 needs to be transmitted only once, which is usually much smaller than the number of the documents 102 that have not been changed. Therefore, it is possible to quickly collect documents from the Web server 100 without wasting resources of the Web server 100 or the Internet 130.

The present invention is not limited to the above embodiment,
Various modifications and applications are possible. Hereinafter, modifications of the above-described embodiment applicable to the present invention will be described.

In the above embodiment, the Web server 10
0, the page catalog 104 is stored in the document 10
It was treated as something different from 2. On the other hand, the page catalog 104 may be treated as a type of document published on the Web. In this case, the communication processing unit 111 of the document collection device 110 may specify the URL of the page catalog 104 and request the transmission to the Web server 100.

In the above-described embodiment, a case has been described where the document collection device 110 collects the documents 102 held by the Web server 100 connected via the Internet 130. However, the present invention can be applied not only to an open network system such as the Internet, but also to a closed network system such as an intranet in a company.

In the above embodiment, in the document collection device 110, the communication processing unit 111, the document processing unit 113, the page catalog management unit 116, and the change time comparison unit 117 operate under program control. Acquires the page catalog 104 and, based on the acquired page catalog 115, the Web
It has requested the server 100 to transmit the changed document 102. On the other hand, the functions of the document collection device 110 are realized by storing and distributing a program for performing such processing on a computer-readable recording medium such as a CD-ROM and installing the program on a hard disk of a general-purpose computer. You may do it.

[0054]

As described above, according to the present invention,
All documents that have changed since the previous collection can be collected without requiring documents that have not changed since the last collection.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a document collection system according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of a page catalog of FIG. 1;

FIG. 3 is a flowchart illustrating a process executed by the document collection device in the embodiment of the present invention.

FIG. 4 is a sequence diagram specifically describing an operation in the document collection system according to the exemplary embodiment of the present invention.

FIG. 5 is a block diagram showing a configuration of a document collection system according to a conventional example.

FIG. 6 is a sequence diagram specifically describing an operation in a document collection system according to a conventional example.

[Explanation of symbols]

 Reference Signs List 100 Web server 101 Server communication processing unit 102 Document 103 Page catalog creation processing unit 104 Page catalog 110 Document collection device 111 Communication processing unit 112 Document 113 Document processing unit 114 URL database 115 Page catalog 116 Page catalog management unit 117 Change time comparison unit 120 Browsing software 130 Internet

Claims (7)

[Claims] 1. A server connected to each other via a network and holding a document to be published via the network, and a document collection device for collecting the document from the server via the network. Is used to create a catalog in which, when a document to be held is changed or a document to be newly held is created, the identification information of the document is associated with a change time indicating the time of the change or creation, and a catalog is created. Means, and a server transmitting means for transmitting the document or the catalog to the server via the network in response to a request from the document collecting apparatus, wherein the document collecting apparatus is created by the catalog creating means. Catalogs from the network Catalog requesting means for requesting the server via the server and transmitting the request to the server transmitting means; catalog receiving means for receiving a catalog transmitted from the server transmitting means in response to a request by the catalog requesting means; A catalog management unit that manages a catalog received by the catalog reception unit; and a request held by the server via the network by specifying a document held by the server through the network.
A document requesting means for causing the server transmitting means to transmit; a document receiving means for receiving, in response to a request by the document requesting means, a document transmitted from the server transmitting means together with the identification information and the modification time of the document; Document information registration means for registering the identification information of the document received by the document reception means and the modification time in association with each other; modification time of the document registered in the document information registration means; And a change time comparing unit for sequentially comparing the change times of the documents in the catalog with those having the same identification information. The document requesting unit includes: a result of the comparison by the change time comparing unit; Document has a newer modification time A document collection system requesting the server to transmit a document determined to be valid. 2. A document collection device for collecting a document from a server that publishes the document via a network via the network, wherein the identification information of the document held by the server is associated with a modification time. Requesting a catalog to be registered by the server via the network and transmitting the catalog from the server; and catalog receiving means for receiving a catalog transmitted from the server in response to a request from the catalog requesting means Requesting the server via the network by specifying the identification information of the document held by the server, the catalog management means for managing the catalog received by the catalog receiving means,
Document requesting means for transmitting from the server; document receiving means for receiving, in response to a request by the document requesting means, a document transmitted from the server together with identification information of the document and a modification time; A document information registration unit for registering the identification information of the received document and the modification time in association with each other, a modification time of the document registered in the document information registration unit, and a catalog managed by the catalog management unit. And a change time comparing means for sequentially comparing the change times of the documents with those having the same identification information, and the document requesting means, as a result of the comparison by the change time comparing means, Changed time is determined to be newer A document collection device requesting the server to transmit a document. 3. The catalog management means includes a deletion means for deleting information corresponding to the registered information from the catalog when the identification information and the modification time of the document are registered in the document information registration means. The document collection device according to claim 2, wherein: 4. The document information registering means according to claim 1, wherein when the deleting means deletes all the identification information of the document to be deleted and the modification time, the identification information of the document remaining in the catalog and the modification time are changed. The document collection device according to claim 3, wherein registration is further performed. 5. The document collection apparatus according to claim 2, wherein the identification information of the document is a URL. 6. A document collection method for collecting a document from a server that publishes the document via a network via the network, wherein the identification information of the document held by the server is associated with a modification time. Requesting a catalog to be registered by the server via the network and transmitting the catalog from the server; and receiving a catalog transmitted from the server in response to the request in the catalog requesting step. And a catalog management step of managing the catalog received in the catalog receiving step; requesting the document held by the server to the server via the network by specifying its identification information;
A document requesting step of transmitting the document from the server; a document receiving step of receiving, in response to the request in the document requesting step, a document transmitted from the server together with the identification information of the document and a modification time; A document information registration step of registering the identification information and the change time of the document received in the step in association with each other, a change time of the document registered in the document information registration step, and a catalog managed in the catalog management step And a change time comparison step for sequentially comparing the change times of the documents in the same document with those having the same identification information, and the document requesting step includes: a result of the comparison in the change time comparison step; Document A document transmission method for requesting the server to transmit a document determined to be newer at the change time of the document. 7. A recording medium for recording a program for collecting a document from a server which publishes the document via a network via the network, comprising: identification information of the document held by the server; Requesting a catalog to be registered in association with the server via the network and transmitting the catalog from the server; and receiving a catalog transmitted from the server in response to the request in the catalog requesting step. A catalog receiving step to perform, a catalog managing step to manage a catalog received in the catalog receiving step, and requesting the document held by the server to the server via the network by specifying its identification information,
A document requesting step of transmitting the document from the server; a document receiving step of receiving, in response to the request in the document requesting step, a document transmitted from the server together with the identification information of the document and a modification time; A document information registration step of registering the identification information and the change time of the document received in the step in association with each other, a change time of the document registered in the document information registration step, and a catalog managed in the catalog management step And performing a change time comparison step of sequentially comparing the change times of the documents in the document with those having the same identification information, and the document requesting step includes, as a result of the comparison in the change time comparison step, Document of A computer-readable recording medium for recording a program for requesting the server to transmit a document whose change time is determined to be newer.