JP2004070957A - Search system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a search system for efficiently acquiring a huge amount of search target data scattered on a network and making it a database.
SOLUTION: In a search system for creating a database based on data collected by using a robot on a network and performing a database search, data is collected using a robot and data obtained in response to an external reference request. And a cache unit for holding the obtained data, and when a reference request is given from the outside, if the data corresponding to the cache unit is held, the data is provided from the cache unit, and And data providing means for acquiring and providing the data from the original server which holds the data if the data to be stored is not held.
[Selection] Fig. 10

Description

The present invention relates to a data retrieval system distributed on a network.

@Altavista (http://www.altavista.com/), Lycos (http://www.lycos.com/), Yahoo! There are many search systems on a network using a robot such as (http://www.yahoo.com/). These use software called robots that mechanically collect information on a network. Then, the collected data is made into a database so that the user can search.

(4) The robot searches for a text described in HTML (Hyper \ Text \ Markup \ Language) on the network, and collects data existing on the network by following the link destination described therein. Regarding the creation of a database, there are those that perform a full-text search, and those that search only parts such as titles and URLs.

デ ー タ ベ ー ス The above database is large and may be decentralized. However, this is merely a division because of the large amount, and is not divided in any way.

キ ー ワ ー ド A keyword search is performed for the above search. In other words, a search is performed by inputting a word that would be included in a sentence to be searched.

On the other hand, mirror sites may be provided to distribute the concentration of access to popular sites and reduce traffic. For example, Point Cast Network (PCN) 's I-Server (http://www.pointcast.com/products/isserver.html) periodically prefetches information to the PCN head office and manages mirror sites. .
S. Shimizu, Construction of Search System on WWW Server, Interface, Japan, CQ Publishing Co., Ltd., July 1, 1996, Vol. 22, No. 6, pp. 130-139.

(4) Conventionally, in a data retrieval system distributed on a network, there are the following problems.

(1) It is becoming difficult to handle increasing data.
For example, it is said that there are over 40 million page data on the WWW in 1996 in the world, and it is expected to increase exponentially in the future. At present, the number of pages and the amount of data per page tend to increase rapidly.
It is extremely difficult to manage the database simply by dividing the rapidly increasing data by the amount.

(2) It is difficult to handle frequently updated information.
Data that is updated many times a day is excluded from robot search targets in current search systems. The reason is that even if frequently updated data is collected by a robot and made into a database, the data is often updated before being searched. In such a case, even if you look at the page that appears in the search results, it is already gone, or something different from what the user intended is displayed because the content has been changed to something completely different Inconvenience occurs.

The present invention has been made in consideration of the above circumstances, and has as its object to provide a search system that efficiently obtains a large amount of search target data scattered on a network and creates a database.

Another object of the present invention is to provide a search system that effectively creates a database even for data with extremely high update frequency.

The present invention relates to a search system for creating a database based on data collected by using a robot on a network and performing a database search, and using a data collected in response to a reference request from the outside and a robot to collect the data. And a cache unit for holding the obtained data, and when a reference request is given from the outside, if the data corresponding to the cache unit is held, the data is provided from the cache unit, and And a data providing means for acquiring and providing the data from an original server which holds the data if the data to be stored is not held.

This search system also functions as a proxy, so that if the data requested by the user is in the system, even if it is obtained by the request from the user, it is collected by the robot. Even if it is done, it can be presented to the user. As a result, the search can be applied to data that is updated frequently.

According to the present invention, in the above-described search system, a prediction unit for performing statistical processing on data requested to be referred to from the outside to predict data to be referred to in the future, the predicted data and the data explicitly specified in advance And a prefetch means for acquiring the data by using a robot and prefetching the data in the cache means.

In the present invention, the robot does not collect all the data that can be obtained in advance, but prefetches the data by the robot with respect to the data specified in advance and the data that is considered to have a reference request from a statistical viewpoint from the user. So that the appropriate data is effectively mirrored.

According to the present invention, in the above retrieval system, the prefetch means reacquires data to be acquired at a frequency corresponding to an update frequency of the data.

According to the present invention, in the above-described search system, as a constraint condition of a range of data to be searched in response to the search request, a condition limited to data collected by a robot, a condition obtained in response to an external reference request is obtained. Conditions, limited to the latest data for data with the same name or address, conditions other than data generated dynamically or interactively, and specified sites or data groups At least one of the following conditions.

According to the present invention, in the above retrieval system, the cache unit adds and retains at least one of update time information and collection time information to the acquired data. As a result, it is possible to manage data that has the same name as the acquisition source data but differs depending on the time.

The inventions relating to the respective devices described above also hold as descriptions relating to methods.

The invention described above is also realized as a machine-readable medium storing a program for causing a computer to execute a corresponding procedure or means.

According to the present invention, data can be managed in different databases according to the data update frequency. As a result, for example, the processing capacity of a computer or the like can be set according to the frequency of update of data managed by the database, and the huge amount of data distributed on the network can be effectively managed. it can.

According to the present invention, since the search system also has a built-in proxy function, data stored in the proxy can be searched and presented. As a result, for example, a search service and a reference service can be provided even for data that is updated frequently.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

First, define the words and phrases.

A proxy is a proxy that enters between a client and a server at the application level when a resource is accessed from a client (for example, a user terminal) to a server (for example, a WWW site), and sends a resource access request from the client to the server. Server that relays the response from the server to the client.

Page means a hypertext page. In the WWW world, one page has a unique URL.

$ URL (Uniform Resource Resource Location) is information necessary to access page data. The URL includes information on a protocol, a domain name, a port number, and a path name.

CGI (Common Gateway Interface) is an interface for starting a program from a server to create an interactive page or a dynamic page.

A robot refers to a document described in a hypertext such as HyperText Markup Language (HTML) or Standard Generalized Markup Language (SGML), and mechanically traces the document while linking the network. This is collected by the above, and is realized by software. Instead of a robot, it may be called a spider or a wanderer.

The basic operation of the robot is as follows.

(Procedure 1) Register the root of the specified URL in the search list.
(Procedure 2) The robot acquires a page according to the search list.
(Step 3) The acquired page is analyzed to extract a URL.
(Procedure 4) Add the extracted URL to the search list (however, do not duplicately register the URL).
Thereafter, steps 2 to 4 are repeated. The page acquisition frequency may be determined according to the page update frequency.

Next, the present embodiment will be schematically described.

In this embodiment, a page is treated as an example of data distributed in a network.

As described above, for example, it is said that the number of pages (types of pages) on the World Wide Web (WWW) exceeds 40 million. This number is expected to continue growing exponentially. It is extremely difficult to manage such a huge number of pages in a single database.

The simplest way to divide a database is to divide the database by site (domain), but all databases must be equally fast. If the database can be split, but everything must be fast, the burden of building the database is still high.

Therefore, in the first embodiment, the contents of the database are divided according to the degree of popularity. Then, the popular database is put on a high-speed system (for example, a machine having a large memory), and the less popular database is put on a low-speed system. In this way, it is sufficient to use a high-speed machine only as a machine on which a popular database is mounted, and the burden of database construction can be effectively reduced.

In order to know the popularity of a page, strictly speaking, it is necessary to conduct a survey on network ratings, etc., but such a task is not realistic with great difficulty. Thus, in the present embodiment, the following well-approximate approximation is used. First, he said, "In order to keep the page popular and keep the page from getting tired, it is necessary to constantly update the content." The reverse is approximated as "a page with a high data update frequency is a popular page". That is, in the present embodiment, the data update frequency is used as a popular barometer, and the contents of the database are divided according to the data update frequency. The page update frequency is information that can be obtained by running the robot.

By the way, some pages that are frequently updated are updated many times a day. When a method of accessing such a page only occasionally is adopted, a state occurs in which actual page data does not match a database in the search system. In particular, when referring to a page based on the results of a database search, the relevant page may already be gone, or the content of the page itself may have been changed to something else. The problem occurs in the case.

On the other hand, in order to reduce inconsistencies caused by database obsolescence, robots need to access pages very frequently. However, frequent access to keep up with the latest information on pages that change frequently and irregularly increases unnecessary traffic, and disadvantages both the information storage site and the search system side. .

Therefore, in the second embodiment, the original data in a database is saved and presented to the user. In this way, the actual page change is slightly delayed, but the traffic does not needlessly increase and the original page corresponding to the search result can always be viewed.

It is also possible to combine the first embodiment and the second embodiment. In this case, both effects can be obtained.

Hereinafter, embodiments of the present invention will be described in detail.

(1st Embodiment)
First, a first embodiment will be described.

例 Examples of the system configuration of the present embodiment are shown in FIGS.

In the present embodiment, a plurality of databases are facilitated, and the databases are selectively used according to the data update frequency. That is, a range of the update frequency of the target page data is assigned to each database. When performing a search for a keyword requested by the user, a search is performed in cooperation with a plurality of databases, and the results are collectively presented to the user.

For example, the following method can be considered as a method of assigning pages to each database.
(A) Sharing by statistical update frequency information
(B) Sharing by last update time
(C) Sharing by comprehensive information of statistical update frequency information and last update time Here, a method of sharing by the last update time of (b) will be described.

It is considered that a certain page is frequently accessed immediately after being updated (that is, popular), and is accessed less frequently (that is, not popular) as time elapses since the last update. Therefore, for example, as shown in FIG. 3, the database to be stored is shared according to the range of the last update time.

For example, the following method is conceivable as a method of determining a database that stores information on a certain page.
(1) Determine a database to be stored for each site. In this case, the average value of the update frequency of the data in the site is used as the evaluation value.
(2) Determine a database to be stored for each directory in the site. In this case, the average value of the update frequency of the data in the directory is used as the evaluation value.
(3) Determine a database to be stored in data units. In this case, the update frequency of the data is used as the evaluation value.

更新 Here, the update frequency is the above-described statistical update frequency information, the last update time, or the like. The above methods (1) to (3) can be used together. For example, the site A may be stored in the database for each site, and the site B may be stored in the database for each data. In the site C, the directory a can be stored in the database in directory units, and the directory b can be stored in the database in data units.

デ ー タ Furthermore, it is conceivable that the more frequently updated data is placed on a server connected to the internal network. For example, data with a higher update frequency is placed on an intranet within the organization, and data with a lower update frequency is managed in a place directly connected to the Internet.

In the present embodiment, it is assumed that not the page itself but the keyword and the URL are stored in the database. Also, it is assumed that a keyword extracted by performing a full-text search of a page is added to a URL and stored, and the URL is searched using the keyword.

Also, in the present embodiment, a database in units of words or keywords is described, but a database in units of characters may be used.

Next, an example of each system configuration shown in FIGS. 1, 4, and 6 will be described.

In the configuration example of FIG. 1, search devices 100, 110, and 120 composed of pairs of robots and databases (101 and 102, 111 and 112, 121 and 122), and a plurality of WWW sites (131 and 132) ), User terminal (133) is connected.

デ ー タ ベ ー ス Assign the target update frequency to each database by the page sharing method described above.

{Circle around (1)} The first robot 102 collects a site group or a data group that changes at a high frequency (for example, from the WWW sites 131 and 132), converts it into a database, and stores it in the first database 101.

(3) The third robot 122 collects a group of sites or data that change at a low frequency, creates a database, and stores the database in the third database 121.

{Circle around (2)} The second robot 112 collects a group of sites or data that change at a medium frequency other than that, converts it into a database, and stores it in the second database 111.

(4) The range of the actual statistical update frequency information (or the last update time, etc.) corresponding to the high frequency, the low frequency, and the other medium frequencies is appropriately set.

(4) Next, we will discuss dynamic changes in database allocation.

In the present embodiment, the URL of the corresponding page is entered into each database divided according to the update frequency information obtained from the statistics. However, the page update frequency と と も に (or the average update frequency of the site to which the page belongs, etc.) over time. May change, so that the update frequency of a certain page (or the average update frequency of a site to which the page belongs) may deviate from the range of the update frequency of the initial database assigned to the page. Therefore, it is desirable to request a database having an appropriate update frequency range from a database that is sharing a page to take charge of the page data or site. This request is realized by negotiation between databases.

For example, in FIG. 1, the first robot 102 obtains a statistically high-frequency data group and stores it in the first database 101. However, if the update frequency of the data that has been updated frequently at first becomes lower than the range covered by the user, the data is received by the second robot 112 and the database 111. If the update frequency drops significantly, the third robot 122 and the database 121 are requested to change their responsibilities.

FIG. 2 shows an example of a processing procedure of each search device when there are a plurality of robots according to the update frequency as shown in FIG. 1 and each has a database.

(4) In step S21, it is checked whether or not another search device requests sharing of a page. If there is a request, step S27 is performed. If not, step S22 is performed.

In step S22, each robot selects one designated page and acquires the page. The page is scheduled to be acquired at a frequency proportional to the statistical update frequency of the page at this time. If there is no information on the statistical update frequency for the page, an average value or a default value of the statistical update frequency obtained from the pages of the site including the page may be used instead.

In step S23, the statistical update information of the acquired page is updated based on whether or not the acquired page has changed from the previous time. If the acquisition of the page fails due to a trouble of the network or the partner server, the process returns to step S22, leaving a record that the acquisition of the page failed.

(4) In step S24, it is checked whether the new update frequency is within the range in which it is in charge.

{Circle around (5)} In step S25, if the user is out of his / her area of responsibility, a request is made to a search apparatus that includes the area within the area in charge of the subsequent processing. At this time, the data of the page is erased.

In step S26, if it is within its own area, the acquired page is converted into a database and stored. For example, page data is subjected to morphological analysis, decomposed into word levels, and compiled into a database in the form of pages containing words. At this time, the data before the page is erased.

In step S27, if there is a request from another search device, the page is registered and the statistical update frequency information of the page is set so that the page can be handled by the own robot.

In the present embodiment, when a search user performs a database search, a method of issuing a search request from the user terminal 133 to all of the plurality of databases 101, 111, and 121, and a method of issuing a search request to any one of the databases 1 Can be considered. When a search request is issued to one of the latter databases, only the database that receives the search request returns a result, and the database also queries other databases and merges and returns the result. Such a mode can be considered.

Next, the configuration example of FIG. 4 will be described. FIG. 4 is basically the same as FIG. 1, and a plurality of databases 201 to 203 are prepared according to the data update frequency, but the configuration example of FIG. Is different from

FIG. 5 shows an example of the processing procedure of the search device when there is one robot and there are a plurality of databases as shown in FIG.

In step S11, one designated page is selected, and the page is acquired using the robot 204. The page is scheduled to be acquired at a frequency proportional to the statistical update frequency of the page at this time. If there is no information on the statistical update frequency for the page, an average value or a default value of the statistical update frequency obtained from the pages of the site including the page may be used instead.

In step S12, the statistical update information of the acquired page is updated based on whether or not the acquired page is different from the previous page. If the acquisition of the page has failed due to a trouble of the network or the partner server, the process returns to step S11 with a record that the acquisition of the page failed.

In step S13, it is determined which database is to be assigned to the page based on the new statistical update probability of the page obtained in step S11.

In step S14, the page information is made into a database. For example, page data is subjected to morphological analysis, decomposed into word levels, and compiled into a database in the form of pages containing words. This data is stored in the database determined in step S13. At this time, the data before the page is erased. Here, if the data is stored in a database different from the previously stored database, it is also deleted. If the acquired page has not been changed from the previous time, the database is not created, but if the database to be stored is changed by that, only the data movement is performed.

As mentioned above, the number of robots does not need to match the number of databases. For example, in the case of FIG. 4, the number of robots may be two or four or more. The correspondence between each robot and the database may be set as appropriate.

The database search by the search user is the same as in FIG. 1 described above.

Next, the configuration example of FIG. 6 will be described. The search device 300 of FIG. 6 differs from the search device 200 of FIG. 4 in that a database front end (DBF) 301 for integrating the entire database is provided.

In this configuration example, the DBF 301 receives a search request from the user terminal 133, inquires of an appropriate database, and presents the result to the user.

Next, the specification of the search target range in the database search will be described.

In the first embodiment, it is preferable that, in addition to a search condition using a keyword, a range of target update frequency and / or a range of update time can be specified in the search request. If the update frequency is not explicitly specified in the search request, the database or DBF may perform the search with a default value (for example, an update frequency range such as only the database with the highest update frequency).

FIG. 7 shows an example of a search procedure in the search device of FIG.

When the user sends a search request from the user terminal 133 to the database front end 301, the database front end 301 receives the search request from the user terminal 308 in step S31.

In step S32, it is determined whether the search request has an update frequency range designation.

If so, in step S33, a search is performed only in a database within an appropriate range according to the target range of the user's search request.

If not, search is performed in all databases in step S34.

(4) In step S35, the result is merged and returned to the user terminal 308.

Next, the hardware configuration of the system will be described.

In the first embodiment, a computer that configures a database (or a database and a robot) that handles a higher update frequency (for example, a higher statistical update frequency information or a latest update time) is included in the computer. , The lower the update frequency (for example, the statistical update frequency information is lower, or the last update time is older, etc.). A system in which a computer in charge of a database that searches for frequently updated data using the same or more than the same number of computers has the same or higher processing capacity than a computer in charge of a database that does not update frequently Is preferable.

In other words, the database responsible for the data with the higher update frequency is used more frequently than the database responsible for the data with the lower update frequency, so only the database responsible for the data with the higher update frequency By simply increasing the processing capability of the above, the overall processing capability can be effectively improved.

Therefore, by dividing the database according to the update frequency as in the present embodiment, it is only necessary to use a high-speed computer as the computer on which the frequently updated database is mounted, and it is possible to effectively reduce the burden of constructing the database. it can.

For example, as shown in FIG. 8, a computer group forming the first search device 410 is in charge of a data group with a high update frequency, and a computer group forming the second search device 401 is in charge of a data group with a low update frequency. In this case, in the first computer group 410, the database is duplicated in terms of hardware to increase the speed. As means for increasing the speed, in addition to multiplexing hardware, there are methods such as using a computer using a fast element or increasing the memory capacity.

In the above, the present embodiment has been described with one network, but an environment in which a plurality of networks 500 to 504 are coupled as shown in FIG. 9 may be used. Further, the networks 500 to 504 may connect physically physically separated places such as organizations and countries.

(Second embodiment)
Next, a second embodiment will be described.

In this embodiment, the search system is also provided with a proxy function, and if the user already has page data to be referred to as a search result, the data is not newly obtained via the network, Returns the data you have.

This makes it possible to address the problem of frequently changing pages described above. That is, in a page that changes frequently, when a link shown as a search result is followed, the page may already be missing or may have been updated and useless. On the other hand, if the data used in the search database is presented, such a problem does not occur.

That is, frequently changing pages are acquired in a sampling manner as shown in FIG. 13, and the contents are held until the next acquisition. Thus, even if the page is lost or the content is shifted to another at t1 in FIG. 13, the content at the last sampling time t0 can be presented.

FIG. 10 shows a system configuration example of the present embodiment.

As shown in FIG. 10, a search device 601 according to the present embodiment is connected to a network 600, and includes a robot 602, a cache 603, a database unit 604, a database 605, a database front end (DBF) 607, and a WWW front end. 606. Although not shown in FIG. 10, it is assumed that each WWW site and user terminal are connected via the network 600. Further, in FIG. 10, the database is represented as one, but may be divided into a plurality. Further, the invention described in the first embodiment may be applied to a plurality of databases, and the storage of information may be shared among the databases according to the frequency of data update.

で は In the present embodiment, the URL of the page is stored in the database. It is also assumed that a keyword extracted by performing a full-text search of a page is added to a URL and stored, and the URL is searched by the keyword.

First, we will explain up to the creation of a database, and then explain how to use it.

An example of the procedure up to database creation is shown below.
First, the robot 602 collects data from another WWW site via the network 600 according to the search list. If it is a WWW site that has its own contents, it also collects data from itself.
The collected data is stored in the cache 603.
From the data stored in the cache 603, the search database 605 is created by the database conversion unit 604. For example, when performing a keyword search in word units, the database conversion unit 604 performs morphological analysis on the data in the cache 603 and converts the data into word-based databases. Thus, when a user requests information including a specific word, a database search can be immediately performed. Here, in the present search apparatus, the address of the data stored in the cache 603 is used instead of the address (URL) on the network from which the data was obtained, as the location of the data when creating the database.

On the other hand, a page accessed and obtained by the WWW front-end 606 in response to a reference request from the user is stored in the cache 603 and is also made into a database as described above.

<< An example of the procedure for the next use is shown below >>.

The user accesses the WWW front end 606 of the search device 601 via the network 600 and issues a search request.
The request is passed to the database front end (DBF) 607, and if there are multiple databases, an appropriate database is selected and a search request is issued to it.
When a search request is issued to a plurality of databases, the database front end (DBF) 607 collects the results and presents the search results to the user via the WWW front end 606.
The user issues a reference request to the WWW front end 606 of the search device 601 if there is a search result that the user wants to see further.
In the WWW front end 606, if the page requested to be referenced is stored in its own cache 603, the page is retrieved from the cache 603 and returned to the reference requester. If it is not in its own cache 603, the fact is returned to the reference requester.

Here, the search device does not collect all the obtainable data using the robot, but prefetches the data by the robot in addition to the data specified in advance and the data that seems to have a reference request from a statistical viewpoint. You may do it. This is effective when not searching all data on the WWW or when updating data based on a user's request instead of the actual page update frequency.

That is, if all data on the WWW is not to be searched, it becomes a problem which range is collected by the robot. Therefore, pages and sites appearing in the request to the search server / proxy are statistically processed, and data with high frequency or data of the sites are prefetched in advance by using a robot in advance. At this time, the higher the update information of the actual page is, the better the robot will visit the page, and the higher the probability of the reference request to the page is, the better the robot will visit the page. As a result, appropriate data is mirrored without any special designation by the system administrator.

FIG. 11 shows an example of the configuration of the above search device.

The search device 701 in FIG. 11 is obtained by adding a user request recording unit 708 to the search device 601 in FIG. Therefore, the description of the corresponding parts will be omitted, and the description will focus on the different parts.

FIG. 12 shows a processing procedure of information collection by the search device 701.

In step S41, the access log of the user is analyzed to obtain information on pages and sites frequently referred to on the site.

{Circle around (2)} In step S42, the page and site information explicitly specified by the system administrator or the like is merged with the information obtained in step S41.

で In step S43, the data obtained above is obtained using a robot according to the statistical update probability. If the statistical update probability information has not been obtained for the page, the average of the known statistical update probability information of the pages of the site including the page is substituted. Further, if the statistical update probability information of the site is not known, the statistical update probability information of all known sites or the default value is used. Data is repeatedly acquired at a frequency proportional to the statistical update probability information. Also, if you find that a site is likely to be updated at a certain time, try to get information a little later than that time.

Since the search device 701 also serves as a proxy, the user issues a reference request to the search device 701 when the user simply wants information on the network, not a search request. The reference request is output to the user request recording unit 708 via the WWW front end 706, where the request data is recorded. If the requested data is in the cache 703, it is returned as it is. If not, the data is fetched via the network 700, and the data is temporarily stored in the cache 703, and then used via the WWW front end 707. To the person.

As described above, in the search device of FIG. 11, information indicating which data the user is interested in is stored in the user request recording unit 708. Therefore, when the robot collects data in advance, it does not try to acquire all the data that can be acquired by the robot. Instead, the data stored in the user request recording unit 708 and the data to be acquired that are explicitly specified are acquired. To get.

Note that data groups that should not be acquired may be specified so that they may not be acquired even if they are in the user request recording unit 708.

By the way, with regard to frequently updated data, it is considered that it is not effective even if the record in the user request recording unit 708 is seen. This is because the data is likely to be lost when you return. Therefore, for such data, only the path to the site or the data is regarded as valid information, and if it is not the same data, the data of the same site is acquired by the robot.

For example, there is a high possibility that a URL having the following number as a name exists only temporarily.
http: // www. tsb. co. jp / foo / 1246639. html
In such a case, instead of reacquiring this file, a file linked to this file is acquired, and a file following the link is acquired from the file.

検 索 The search device in FIG. 11 assumes that the prefetched data will be used in the future. Here, it is assumed that a target such as character information, an image, a sound, or a moving image can be arbitrarily selected as a prefetch target. For example, if you specified to prefetch only character information due to storage capacity restrictions, but the page contains a video, if the user refers to the video, go to the network to get it, Will not be displayed either.

Next, a description will be given of a page acquisition frequency in the search device in FIGS. 10 and 11.

(4) The robot goes to acquire the page of the same URL periodically, and it is preferable that the robot retake the page at a frequency corresponding to the update frequency of the target page. That is, the target page is obtained by the number of times that is proportional to the number of times that the target page is statistically changed in one day. However, if the specified data disappears, do not go to get that data again. If the acquired data is a hyperlink, it is also possible to get information on the link destination.

(4) Regarding the data of the designated site group and URL group, even if the user issues a reload request, the user does not respond to the request. This ensures that the search server does not request the same URL more than a certain number of times.

Next, a search target in the search device of FIGS. 10 and 11 will be described.

In this embodiment, the data collected by the robot is also stored in the cache of the proxy, and is managed in the same place as the data directly requested by the user.

Here, since the referred content may be unencrypted paid data and there is a problem of the privacy of the user, the data to be searched by the search system may be restricted.

The way of giving the restriction is a combination of one or more of the following conditions.
(1) limited to those collected by robots,
(2) Only those that hold data as a proxy,
(3) Information with the same name or address is limited to the latest information.
(4) Excluding information generated dynamically or interactively by CGI, etc.
(5) Limited to specified sites and URLs.

For example, in FIG. 10, when data is put into the cache 604, the acquisition status of the data is also recorded. That is, whether the data is collected by a robot, requested directly by the user, or generated dynamically or interactively by CGI (CGI or BIN is added to the URL path name. Information is also recorded together with the data, such as whether the group is a designated site group or a URL group. The administrator can specify which type of data can be searched for in the cache. According to this specification, the search system creates only a database that meets the conditions in a database.

Next, a description will be given of the replacement of the address of the collected data in the search device in FIG. 10 or FIG.

In this embodiment, when storing the collected data in the cache of the search device, the address or URL of the collected data may be changed and stored. That is, since the position of the data has been moved from a location on the network to the cache in the search device, the domain name is changed to the domain name of the search device. Next, the original domain name is added to the head of the path name. For example,
Original URL http: // www. foo. co. jp / bar / index. html
Domain name of search device www. search. co. jp
New URL http: // www. search. co. jp / www. foo. co. jp / bar / index. html
By doing so, mirroring of data can be realized.

Next, a description will be given of time management of collected data in the search device of FIGS.

In the present embodiment, the collected data may be managed by adding update time data. For the same address (URL) as a normal proxy, not only the latest data but also the past data is managed and stored. Here, the time has a time when the data becomes valid or a time when the data becomes invalid.

When the content becomes updated at the same URL, the update time notified from the server changes. Therefore, the time becomes invalid when the data itself is lost. Is the time at which it has been determined that it has disappeared due to access.

The address (URL name) is changed and managed for time management.

First, since the position of the data has been moved from a certain place on the network to the cache in the search device, the domain name is changed to the domain name of the search device. Next, the original domain name is added to the head of the path name. For example,

Original URL http: // www. foo. co. jp / bar / index. html
Domain name of search device www. search. co. jp
New URL http: // www. search. co. jp / www. foo. co. jp / bar / index. html
Further, time information is also added to this. For example, if the data is valid from 16:39 on March 23, 1996 to 10:23 on April 30, 1996, the following is performed.

http: // www. search. co. jp / www. foo. co. jp / bar / index. html / 199603232339-199604301023
In addition, the following modifications are also conceivable.

http: // www. search. co. jp / www. foo. co. jp / bar / index. html / 1996.3.23.26.16.39-1996.4.30.10.23
Each configuration in the embodiment of the present invention described above can be realized by creating a program for causing a computer to execute a corresponding procedure or means, and causing the computer to execute the program.

It is also possible to record the above-mentioned program on a machine-readable medium so that a computer reads and executes the program from this medium.

The present invention is not limited to the above-described embodiment, and can be implemented with various modifications within the technical scope.

FIG. 2 is a diagram illustrating a configuration example of a search device according to the first embodiment of the present invention. A flowchart showing an example of a processing procedure of the search device Diagram for explaining how to share the database according to the last update time FIG. 5 is a diagram showing another configuration example of the search device according to the embodiment. A flowchart showing an example of a processing procedure of the search device FIG. 6 is a diagram showing still another configuration example of the search device according to the embodiment. A flowchart showing an example of a processing procedure of the search device FIG. 6 is a diagram showing still another configuration example of the search device according to the embodiment. Diagram showing an example of a system configuration when a plurality of networks are connected FIG. 6 is a diagram illustrating a configuration example of a search device according to a second embodiment of the present invention. FIG. 9 is a diagram illustrating a configuration example of another search device according to the second embodiment of the present invention. A flowchart showing an example of a processing procedure of the search device Illustration to illustrate sampling of frequently changing pages

Explanation of reference numerals

100, 500 to 504, 600 Network 100, 110, 120, 200, 300, 401, 410, 601 Search device 102, 112, 122, 204, 602 Robot 101, 101-1, 101-2, 111, 121, 605: Database 131, 132: WWW site 133: User terminal 301, 301-1, 301-2, 607: Database front end (DBF) 603: Cache 604: Database conversion unit 606: WWW front end 708 ... User request recording section

Claims (5)

In a search system that creates a database based on data collected using robots on the network and performs database search,
A cache unit for holding data acquired in response to a reference request from outside and data collected using a robot,
When a reference request is given from the outside, if data corresponding to the cache means is held, data is provided from the cache means, and if data corresponding to the cache means is not held, A data providing means for acquiring and providing the data from an original server holding the data. Forecasting means for performing statistical processing on data requested to be referred from the outside and predicting data requested to be referred in the future;
2. The retrieval system according to claim 1, further comprising: a prefetch unit that acquires predicted data and data explicitly specified in advance using a robot, and prefetches the data into the cache unit. 2. The retrieval system according to claim 1, wherein the prefetch unit refetches the data to be acquired at a frequency corresponding to an update frequency of the data. As constraints on the range of data to be searched in response to the search request, conditions limited to data collected by the robot, conditions limited to data acquired in response to an external reference request, same names Or, at least one of the following conditions is limited to only the latest data having addresses, conditions other than data generated dynamically or interactively, and conditions limited to designated sites or data groups. The search system according to claim 1, wherein the search system is charged. 2. The search system according to claim 1, wherein the cache unit adds and retains at least one of update time information and collection time information to the acquired data.

Images