CN103116580A - Providing method, system and device of website content information - Google Patents

Abstract

The invention discloses a method, system and device for providing website content information. The method includes: performing a crawling search according to the acquired initial link information of the imported website, obtaining the link information included in the imported website, and obtaining the link information link objects and their attribute information; according to the obtained link objects and their attribute information of the link information, establish the link object index corresponding to the link information; according to the association relationship between the link object indexes of the link information, Establishing a website resource view of each imported website; the website resource view includes link object indexes corresponding to the link information of each imported website arranged according to set rules; and providing website content information to the website information requester according to the established website resource view. The content information imported into the website can be accurately and timely obtained, and the accurate scheduling of the content by the information requester is supported.

Description

Method, system and device for providing website content information

technical field

The invention relates to the field of data services, in particular to a method, system and device for providing website content information.

Background technique

The Internet Data Center (IDC) can realize the introduction of website content and provide services for users. IDC can provide users with website content services directly or indirectly through Content Delivery Network (CDN).

At present, when IDC imports website content sources, it usually adopts a non-full method, and only imports some channels or content of the website. IDC usually adopts the following methods to obtain, update and manage the content information of hosted websites. First, the website resource provider manually Report, so as to grasp the imported website resource information; second, the IDC administrator manually obtains and configures the imported website resource information.

Among the above methods of IDC importing website content, method 1 is highly dependent on the active operation of the content provider, and cannot guarantee the accuracy, timeliness and precision requirements of website content indexing; method 2 requires a lot of IDC management labor costs, and the efficiency is low. There is no guarantee that the Index of Introduced Content will be kept up-to-date. That is to say, the current control over the introduction of IDC website content only implements device-level control, and the content-level control is relatively extensive. It is difficult to obtain the content index information of IDC imported websites accurately and in a timely manner at a relatively low cost. Accurate management poses certain difficulties.

When the existing IDC content introduction and control mechanism is used in a CDN network, in addition to the website content service mainly oriented to the IDC introduction in the CDN network, there is also a cache control (WebCache) system for caching hot website content, and the CDN resource scheduling center It will uniformly coordinate the user's access schedule to the website content introduced by IDC and cached by the WebCache system, so that users can reasonably access the website content introduced by IDC and WebCache system.

In general, since IDC directly introduces website content to content providers, the website content it introduces should be more timely than the website content cached by the WebCache system. Therefore, it is generally hoped to prioritize the scheduling of website content introduced by IDC for users. There is also a WebCache system in the network. It is very likely that the content requested by the user is scheduled to the WebCache system when it has been introduced into the IDC, which greatly wastes the IDC system resources and WebCache cache resources. In order to avoid conflicts, it is necessary to have a detailed understanding of the websites introduced by the IDC. content information.

Under the existing IDC content introduction and control mechanism, the website content information introduced by IDC mainly depends on the provider or IDC administrator to manually update, which is slow in update speed, complicated in operation, low in accuracy, and poor in real-time performance. After the content source of the IDC website is updated, the CDN bus system cannot know the changes of the IDC content in a timely manner. Therefore, when the user visits, there may be an access conflict between the website content introduced by the IDC and the website content cached by the WebCache, and the CDN bus cannot be informed. Whether the user schedules the IDC to import website resources or the website resources cached by WebCache.

Contents of the invention

Embodiments of the present invention provide a method, system and device for providing website content information to solve the problem in the prior art that website content imported by IDC cannot be accurately known, resulting in inability to accurately schedule user content access requests and wasting system resources.

A method for providing website content information, comprising:

performing a crawl search according to the obtained initial link information of the imported website, obtaining the link information included in the imported website, and obtaining the link object and its attribute information of the linked information;

Establishing a link object index corresponding to the link information according to the acquired link object and attribute information of the link information;

According to the association relationship between the link object indexes of each of the link information, establish the website resource view of each imported website; the website resource view includes the link object indexes corresponding to the link information of each imported website arranged according to the set rules;

Provide website content information to the website information requester based on the established website resource view.

A website content information providing device, comprising:

A search module, configured to crawl and search according to the obtained initial link information of the imported website, obtain the link information included in the imported website, and obtain the link object and its attribute information of the linked information;

An index module, configured to establish a link object index corresponding to the link information according to the acquired link object and its attribute information;

The view resource generation module is used to establish the website resource view of each imported website according to the association relationship between the link object indexes of each of the link information; the website resource view includes the links of each imported website arranged according to the set rules The link object index corresponding to the information;

An access retrieval module is used to provide website content information to the website information requester according to the established website resource view.

A system for providing website content information, comprising the above-mentioned apparatus for providing website content information and at least one website information requesting device.

The beneficial effects of the present invention are as follows:

The method, system and device for providing website content information provided by the embodiments of the present invention will include the initial link information of the website and all the link information associated with it step by step, as well as the corresponding link objects, perform an association index, and establish a website resource view, so that the website resource view can be established Accurately know the website content information introduced by IDC and provide it to users, and at the same time save query time and reduce the utilization of system resources; even if there are website resources introduced by IDC and website resources cached by WebCache at the same time, it can prioritize scheduling for users. Website resources, avoid conflicts during content access scheduling, and save system resources.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention, and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute improper limitations to the present invention. In the attached picture:

1 is a flowchart of a method for providing website content information in an embodiment of the present invention;

2 is a schematic structural diagram of a system for providing website content information in an embodiment of the present invention;

3 is a schematic structural diagram of a device for providing website content information in an embodiment of the present invention;

4 is a schematic structural diagram of a system for providing website content information in an embodiment of the present invention;

FIG. 5 is a flow chart of generating a resource view by a website content information providing device in an embodiment of the present invention;

6 is a flowchart of a method for providing website content information in Embodiment 1 of the present invention;

FIG. 7 is a flowchart of a method for providing website content information in Embodiment 2 of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer and clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

An embodiment of the present invention provides a method for providing website content information. By establishing a website resource view to obtain an updated website content index, the link objects corresponding to the initial link information of the website and all associated link information are indexed, and the website is established. The resource view is used to realize the provision of website content information according to the website resource view. The flow of the method is shown in Figure 1, including the following steps:

Step S11: Perform a crawling search according to the obtained initial link information of the imported website, obtain the link information contained in the imported website, and obtain the link object and its attribute information of the obtained link information.

When the crawling search is performed according to the obtained initial link information of the incoming website, specifically, the crawling search is performed according to the obtained initial link information of the incoming website and a pre-configured search strategy. Wherein, the search strategy includes one or a combination of the following strategies: a depth-first strategy, a breadth-first strategy, and a focused search strategy.

According to the obtained initial link information of the imported website, all associated link information can be crawled and searched, that is, the initial link object and its attributes can be obtained according to the initial link information, and the crawl search can be performed from the current link object to obtain new link information. Get the corresponding object and attribute of the new link information. Wherein, the initial link information may be the top-level domain name of the initial web page, such as a uniform resource locator (Uniform/Universal Resource Locator, URL), and the associated link information may be each URL found by crawling and searching on the web page. According to all link information including initial link information and associated link information found through crawling, crawling searches for a link object of each link information.

The above-mentioned link object includes the web page and/or file corresponding to the link information; the attribute information of the above-mentioned link object includes one or a combination of the following information: link value, link type, web page title, number of crawls, crawl time, crawl depth, Whether to first fetch, default encoding, page snapshot, file object name, and object type.

For example: crawling search can be realized through a website content information providing device, and the URL on the initial web page is crawled from one or several top-level domain name links of the imported website provided by the IDC service providing device. For each URL, the crawler saves the URL corresponding to the link. Attribute information of link objects such as webpages or files, including but not limited to link value, link type, webpage Title, number of crawls, crawling time, crawling depth, whether it is the first crawl, default encoding, webpage snapshot, file object name , object type and other information. At the same time, the crawler continuously extracts new URLs from the current page and puts them into the queue. After analyzing the current page, it extracts new URLs from the queue and continues to crawl webpage or object information until the preset search stop condition is met.

Step S12: According to the link object and its attribute information of the obtained link information, establish a link object index corresponding to the obtained link information.

Construct the content index of each link object according to the link object and its attribute information included in the obtained link information, and determine the relationship between the link information according to the path information of the obtained link information; after analysis and filtering, establish a link information including each The link object index of the content index of the link object included in the association relationship and each link information.

The link information obtained by crawling search, link objects of the link information and their attribute information are processed, including building a content index of each link object and data association of each link object. According to the information obtained by the crawler, such as link value, link type, webpage Title, number of times crawled, crawling time, crawling depth, whether it is crawled for the first time, default encoding, web page snapshot, file object name, object type, etc., indexing and construction The content index of each link object; record the URL path captured by the crawler, judge the parent-child relationship between different URLs, form the association relationship between content indexes, obtain the link object index of each link information, and provide data support for generating a global view of website resources .

Preferably, before establishing the link object index corresponding to the link information, it also includes performing data deduplication processing on the link information searched by crawling, as well as the link objects of the link information and their attribute information. It can perform MD5 (message digest algorithm fifth edition) calculation on the link information found by crawling search, as well as the link object and its attribute information of the link information, and judge whether the calculated MD5 value is the same as the link information that has established the link object index , when it is the same, no link object index will be established. Of course, other methods can also be used to determine whether the link information found by crawling is the same as the link information for which the link object index has been established. For example: After a URL is successfully crawled, it does not need to be crawled again within the update time period, but other web pages may contain this URL, so the URL needs to be deduplicated. This system uses MD5 calculations on URLs that have been captured, and compares the MD5 values of the URLs to ensure the uniqueness of the captured URLs, that is, URLs with the same MD5 value will not be repeatedly captured.

By performing indexing, associating, cleaning, deduplication and other processing on the relevant data of the link information obtained by crawling search, the standard IDC content index data is generated, and the link object index of each link information is obtained.

Step S13: According to the association relationship between the link object indexes of each link information, a website resource view of each imported website is established. The website resource view established therein includes link object indexes corresponding to link information of each imported website arranged according to set rules.

According to the association relationship between the link object indexes of each link information established by the above index, for example, the parent-child relationship of each link information, the website resource view of each imported website can be established.

Step S14: Provide website content information to the website information requester according to the established website resource view.

When providing website content information to the website information requester based on the established website resource view, the website content information is generally provided to the website information requester through the website information requesting device, either by providing the resource view to the website information requesting device, or by An open query interface is adopted for the website information requesting device to query the resource view, and the website content information requesting device provides the website content information to the website information requester according to the website resource view. Wherein, the IDC website information requesting device may be a CDN resource bus, or an IDC service platform or other IDC website information requesting devices.

Of course, it is optional, and the website content information may be directly provided to the website information requester according to the established website resource view without using the website content information requesting device.

Among them, the method of providing the resource view to the website information requesting device specifically includes: according to the view resource acquisition request sent by the website information requesting device, providing the established website resource view to the website information requesting device or according to the configuration in the view resource obtaining request It is required to configure and adjust the established website resource view and provide it to the website information requesting device, and the website information requesting device will provide the requested website content information to the website information requester according to the obtained website resource view; including providing information based on the provided website resource view Website access scheduling service or IDC site management service.

Among them, the method of opening the query interface for the website information requesting device to query the resource view specifically includes: opening the query interface to the website information requesting device according to the view resource query request sent by the website information requesting device, and providing the website information requesting device through the query interface The established website resource view or provide a website resource view after configuring and adjusting the established website resource view according to the configuration requirements in the view resource acquisition request; the website information requesting device provides the website information requester with all Requested website content information; including providing website access scheduling services or IDC site management services based on provided website resource views.

Based on the above-mentioned website content information providing method provided by the embodiment of the present invention, the embodiment of the present invention also provides a website content information providing system, its structure is shown in Figure 2, including the above-mentioned website content information providing device and at least one website information request The two devices can be connected through interfaces such as IF1 and IF2. For example: the device for providing website content information may be an IDC content information synchronization device, and the device for requesting website information may be an IDC service platform and a CDN resource bus. Realize information interaction between the website content information providing device and several IDC website information requesting devices such as the IDC business platform and CDN resource bus, and provide the resource view download or query of the established imported website for the website information requesting devices such as the IDC business platform and CDN resource bus .

The above-mentioned CDN resource bus can implement functions such as resource management, content management, and user scheduling. After obtaining the resource view of the website content introduced by IDC through the website content information providing device, it reasonably schedules user access requests, and distributes the IDC website content to appropriate CDN content nodes and service nodes according to the content distribution strategy. The above-mentioned IDC service platform can realize hardware management and software management. Among them, the software management can provide basic content configuration and management functions for the website content provider, and provide basic domain name information for the IDC content information synchronization device.

Based on the above method for providing website content information provided by the embodiment of the present invention, the embodiment of the present invention also provides a device for providing website content information, and its structure is shown in Figure 3, including: a search module 10, an index module 20, and a view resource generation module 30 and access retrieval module 40.

The search module 10 is configured to crawl and search according to the obtained initial link information of the imported website, obtain the link information included in the imported website, and obtain the link object and its attribute information of the link information.

The indexing module 20 is configured to establish a link object index corresponding to the obtained link information according to the obtained attribute information of the link object.

The view resource generation module 30 is used to establish the website resource view of each imported website according to the association relationship between the link object indexes for obtaining link information; wherein, the website resource view includes the links of each imported website arranged according to the set rules The link object index corresponding to the information.

The access retrieval module 40 is configured to provide website content information to the website information requester according to the established website resource view.

Preferably, the above-mentioned website content information providing device further includes: a search strategy management module 50; wherein:

The search strategy management module 50 is configured to configure a search strategy, and the configured search strategy includes one or a combination of the following strategies: a depth-first strategy, a breadth-first strategy and a focused search strategy.

Correspondingly, the above-mentioned search module 10 is specifically configured to perform crawling search according to the obtained initial link information of the imported website and the pre-configured search strategy.

Preferably, the above-mentioned index module 20 is specifically configured to construct a content index of each link object according to the obtained link objects of the link information and their attribute information, and determine the association relationship between each link information according to the obtained path information of the link information; A link object index including the association relationship of each link information and the content index of the link objects included in each link information is established.

Preferably, the above-mentioned indexing module 20 is also used for: before establishing the link object index corresponding to the link information searched by crawling, data deduplication is performed on the link information searched by crawling, as well as the link objects of the link information and their attribute information deal with.

Preferably, the above-mentioned access retrieval module 40 is specifically configured to provide the established website resource view to the website information requesting device according to the view resource acquisition request sent by the website information requesting device, or to provide the established website resource view according to the configuration requirements in the view resource obtaining request. The website resource view is configured and adjusted and provided to the website information requesting device, and the website information requesting device provides the requested website content information to the website information requester according to the provided website resource view; or according to the view resource query request sent by the website information requesting device , open the query interface to the website information requesting device, provide the established website resource view to the website information requesting device through the query interface or provide the website resource view after configuring and adjusting the established website resource view according to the configuration requirements in the view resource query request, The website information requesting device provides the requested website content information to the website information requester according to the queried website resource view.

Preferably, the above-mentioned website content information providing device further includes a local domain control module 60, configured to control the search scope of the crawling search by the search module.

The specific structure of the above-mentioned website content information providing system is shown in FIG. domain control module 60 , system management module 70 and portal (Portal) module 80 . in:

The access retrieval module 40 of the website content information providing device implements communication with the website information requesting device, so as to provide website content information to the website information requester according to the established website resource view. The access retrieval module 40 implements data interaction with external devices such as CDN resource bus, IDC service platform, etc. based on the interface protocol, provides authentication function as a server role, and performs user authentication on the account number and password of the external device, so as to realize the introduction of IDC into the website The import of initial link information and the sending function of IDC website content resource information data; it belongs to the technical realization of shielding different bottom access methods for the upper layer.

Portal module 80 provides the portal for administrators to manage, maintain, and access website content information providing devices. The system is based on the B/S architecture and provides necessary page (Web) operation and management display interfaces for functions such as user login, log query, and statistical reports. It belongs to the technical realization of the user interaction level.

The search module 10 realizes the crawling search of the imported website, utilizes the standard http protocol, according to the initial link information of the imported website provided by the website information request device and the Portal module 80, and the search strategy formulated by the search strategy management module, within the scope of this domain, Retrieve the content of the website introduced by the IDC, and traverse all the link information in the domain of the IDC website, as well as the corresponding link objects and their attribute information.

The indexing module 20 implements indexing of data related to link information obtained through crawling search, and establishes a link object index. By analyzing the link information retrieved by the search module 10 and related information data of link objects such as webpages and files, after multiple processing such as extraction, association, cleaning, and deduplication, the function of generating standard IDC content index data is realized, and each link is obtained. The link object index corresponding to the information.

The view resource generation module 30, based on the IDC content index data generated by the index module 20, generates a website resource view of the imported website, so as to provide users with scheduling.

The access retrieval module 40 can also provide a retrieval function on the basis of the IDC content index data established by the index module 20 for the Portal module and the website information requesting device to query the content of the website imported by the IDC.

The search strategy management module 50 is used to allow administrators to configure and manage search strategies, such as rules such as depth-first search, breadth-first search, focused search, etc., for calling by the search module 10 .

The local domain control module 60 configures and manages the local domain search strategy, controls the search range of the search module 10, and limits the search operation to be carried out inside the imported website of this domain, or to be linked to IDC computer rooms or servers of other domains.

The system management module 70 provides the local network management function, and this module belongs to an optional module, which monitors and manages system availability, device performance, and network indicators in real time, such as: obtaining the resource usage and health status of the website content information provision system in real time; The alarm information generated in the system is collected uniformly, and the corresponding strategy is called according to the alarm level for processing; the connection with the upper-level network management system and data collection and transmission are realized through the network management interface; various data generated by monitoring are recorded and analyzed, and the user is responsible for recording The operation log when using the system realizes the statistical function of query records of the upper and lower systems; automatically generates regular reports and various personalized reports to support the analysis of various management needs; configures related configuration information of external network elements, and displays them through the Portal to provide classification The authority management function ensures that users with different roles can only use authorized functions, and can only view and maintain authorized data.

The above-mentioned website content information providing device supports the configuration of IDC website content resource subscription requirements required by different website information requesting devices through the IDC business platform, Portal interface, and CDN resource bus, and can generate different resource view files for different website information requesting devices. In order to ensure the security and independence of the files, the website content information providing device should store the website content resource view files for different website information requesting devices in different paths, and control them through different access user names and permissions.

The interaction process between the modules of the above-mentioned website content information providing device is shown in Figure 5, which specifically includes the following steps:

Step S21: The search module requests a search policy from the search policy management module.

The search module in the website content information providing device requests the search strategy of the crawler from the search strategy management module.

Step S22: The search strategy management module returns the configured search strategy to the search module.

For example: the search strategy management module returns the crawler search strategy to the search module.

Step S23: The search module requests the local domain control policy from the local domain control module.

Step S24: the local domain control module returns the configured local domain control policy to the search module.

As described in the method section above, the crawling search scope of the search module can be determined according to the domain control policy.

Step S25: The search module crawls and searches according to the configured search strategy and the domain control strategy.

The search module crawler obtains the link information of the specified website, the corresponding link object and the attribute information of the link object within the range specified by the domain control policy according to the configured search policy.

Refer to step S11 for the specific implementation process.

Step S26: The search module sends data such as the link object and its attribute information of each link information searched to the index module.

Step S27: The index module processes the data searched by the search module and generates a link object index corresponding to the obtained link information.

Refer to step S12 for the specific implementation process.

Step S28: The index module sends index information such as the generated link object index to the resource view generation module.

Step S29: The resource view generation module processes the index data to generate a website resource view.

Refer to step S13 for the specific implementation process.

The foregoing describes the process of each module in the website content information providing device interacting to realize the generation of the website resource view.

The above-mentioned website content information providing device is used to provide the website content introduced by the IDC and mainly supports two types of data transmission methods: one is to provide the File Transfer Protocol (File Transfer Protocol, FTP) service function, and the website information request device first initiates a request for a specific imported website. In response to an acquisition request for a website resource view, the website content information providing device parses the content and generates resource view information of a corresponding range for downloading by the business platform. The second is to support the interaction with the website information request device through HyperText Transfer Protocol (HyperTextTransfer Protocol, HTTP) + web service (WebService). The website information request device initiates a query request for the website resource view of a specific imported website. The content information providing apparatus returns corresponding resource view information to the website information requesting device. The implementation process of the method for providing website content information in the above two different data transmission modes is described below through specific embodiments:

Embodiment one

The method for providing website content information provided by Embodiment 1 of the present invention implements the website resource view based on the file interface to provide the download of the website resource view, and its flow is shown in Figure 6, including the following steps:

Step S101: The website information requesting device transmits the initial link information of the website to the website content information providing device.

For example: the operator transmits the original information of the IDC website, including the domain name, the initial crawling link, etc., to the website content information providing device through the IDC business platform or Portal interface.

Step S102: The website content information providing device crawls and searches on the IDC website server through the interface of the IDC service platform.

Step S103: Obtain each link information corresponding to the initial link information from the IDC website server, as well as the link object and the attribute information of the link object corresponding to each link information.

Step S104: The website content information providing device builds a link object index based on the link objects corresponding to each link information obtained by crawling and the attribute information of the link objects.

The website content information providing device establishes a link object index of each link information included in the imported website through data processing operations.

Step S105: the website content information providing device generates a standard IDC website resource view.

Step S106: the website information requesting device sends a view resource acquisition request to the website content information providing apparatus.

The website information requesting device uploads configuration requirements of view resources and downloads website resource views through the interface with the website content information providing device. As shown in FIG. 6, the website information requesting device may be an IDC service platform or a CDN resource bus.

Step S107: The website content information providing device adjusts the configuration of the established website resource view according to the configuration requirements in the view resource acquisition request.

This step is optional, and is not performed when the view resource acquisition request does not carry configuration requirements. When the configuration requirements are carried in the view resource acquisition request, the website content information providing device outputs the IDC website resource view files meeting the configuration requirements according to the configuration requirements of the website information requesting device and stores them in the corresponding path for downloading by the website information requesting device.

Step S108: The website information requesting device downloads the requested website resource view from the website content information providing device.

The website information requesting device establishes a connection with the website content information providing device according to its own needs, and downloads the website resource view file from the website content information providing device.

Embodiment two

The method for providing website content information provided by Embodiment 2 of the present invention implements website resource view query based on a real-time query interface, and its process is shown in FIG.

Step S201: The website information requesting device transmits the initial link information of the website to the website content information providing device.

The website content information providing device provides WebService or other real-time message interfaces to the outside. The operator transmits the original information of the IDC website, including domain name, initial crawl link, etc., to the website content information providing device through the IDC business platform or Portal interface.

Step S202: The website content information providing device crawls and searches on the IDC website server through the interface of the IDC service platform.

Step S203: Obtain each link information corresponding to the initial link information from the IDC website server, as well as the link object and the attribute information of the link object corresponding to each link information.

Step S204: The website content information providing device establishes a link object index based on the link objects corresponding to each link information obtained by crawling and the attribute information of the link objects.

The website content information providing device establishes a link object index of each link information included in the imported website through data processing operations.

Step S205: the website content information providing device generates a standard IDC website resource view.

Step S206: the website information requesting device requests to log in to the website content information providing device.

When the website information requesting device needs to obtain the website resource view, it sends a login request to the website content information providing device.

Step S207: The website content information providing device responds to the login request of the website information requesting device.

Optionally, the website content information providing device may allow the service platform to log in after authenticating the website information requesting device.

Step S208: The view resource query request sent by the website information requesting device to the website content information providing device.

The website information requesting device uploads configuration requirements of view resources and queries website resource views through the interface with the website content information providing device. As shown in FIG. 7 , the website information requesting device may be an IDC service platform or a CDN resource bus.

Step S209: The website content information providing device adjusts the configuration of the established website resource view according to the configuration requirements in the view resource query request.

This step is optional, and is not performed when the view resource query request does not carry configuration requirements. When the configuration requirement is carried in the view resource query request, the website content information providing device outputs the IDC website resource view that meets the configuration requirements according to the configuration requirements of the website information requesting device and stores it in a corresponding path for the website information requesting device to query.

Step S210: the website content information providing device responds to the view resource query request of the website information requesting device.

The website information requesting device establishes a connection with the website content information providing device according to its own needs, and queries the website resource view from the website content information providing device.

Step S211: The website information requesting device website sends a logout request to the content information providing device.

When the website information requesting device no longer needs to obtain the website resource view, it sends a logout request to the website content information providing device.

Step S212: The website content information providing device responds to the logout request of the website information requesting device.

The website content information providing device logs out the login information of the website information requesting device.

The method and device for providing website content information provided by the embodiments of the present invention can automatically access, collect, and obtain website content information from the IDC website server in HTTP mode, control the range of URLs obtained by crawlers, and only obtain website resources that have been introduced into a specific IDC domain information; for the acquired URL information, it supports processing such as URL association and deduplication, and generates link object index information up to the link object level; and supports flexible generation of different website resource view information according to the requirements of different website information request devices, so as to Provided to the website information request device; it can support the generation of website resource view files according to the demand through the file method, and provide it to the website information request device; it can also support the real-time query method based on the message, and the website information request device can communicate with the website content through the interface The information providing device interacts and actively initiates an IDC website resource view query request, and the website content information providing device returns the queried website resource view to the website information requesting device.

The above method effectively solves the defects of low information synchronization efficiency, poor accuracy, slow speed, and untimely synchronization caused by manual configuration or collection of IDC website information at this stage, and has the advantages of automatic collection and integration processing, high efficiency, and strong real-time performance , can further optimize the timeliness and accuracy of IDC website information, so as to strengthen the ability of CDN network to intelligently dispatch website resources.

The above method centralizes the processing of IDC website resource information in the newly added website content information providing device, avoids the operation of all website information requesting devices to perform integrated processing of IDC website information, and effectively reduces the complexity of implementing IDC website content management. It reduces the construction and investment costs of business-side equipment, and provides a good solution for website information request equipment to quickly and efficiently obtain IDC website resource information.

The foregoing description shows and describes preferred embodiments of the present invention, but as previously stated, it should be understood that the present invention is not limited to the form disclosed herein, and should not be viewed as excluding other embodiments, but can be used in various Other combinations, modifications and circumstances, and can be modified within the scope of the inventive concept described herein, by the above teachings or by skill or knowledge in the relevant field. However, changes and changes made by those skilled in the art do not depart from the spirit and scope of the present invention, and should all be within the protection scope of the appended claims of the present invention.

Claims (11)

1. A method for providing website content information, comprising: performing a crawl search according to the obtained initial link information of the imported website, obtaining the link information included in the imported website, and obtaining the link object and its attribute information of the linked information; Establishing a link object index corresponding to the link information according to the acquired link object and attribute information of the link information; According to the association relationship between the link object indexes of each of the link information, establish the website resource view of each imported website; the website resource view includes the link object indexes corresponding to the link information of each imported website arranged according to the set rules; Provide website content information to the website information requester based on the established website resource view. 2. The method according to claim 1, wherein the crawling search is performed according to the obtained initial link information of the introduced website, specifically comprising: The crawling search is performed according to the obtained initial link information of the introduced website and a pre-configured search strategy, wherein the search strategy includes one or a combination of the following strategies: depth-first strategy, breadth-first strategy and focused search strategy. 3. The method according to claim 1, wherein the link object index corresponding to the link information is established according to the acquired link object and its attribute information of the link information, specifically comprising: Construct the content index of each link object according to the link object corresponding to the obtained link information and its attribute information, and determine the association relationship between the link information according to the path information of the link information; establish an association including each link information The link object index of the content index of the link object corresponding to the relation and each link information. 4. The method according to claim 1, wherein the providing website content information to the website information requester according to the established website resource view specifically includes: According to the view resource acquisition request sent by the website information requesting device, provide the established website resource view to the website information requesting device or provide the website information requesting device after configuring and adjusting the established website resource view according to the configuration requirements in the view resource obtaining request , the website information requesting device provides the requested website content information to the website information requester according to the provided website resource view; or Open the query interface to the website information requesting device according to the view resource query request sent by the website information requesting device, and provide the established website resource view to the website information requesting device through the query interface or provide the established website according to the configuration requirements in the view resource query request The resource view is a website resource view after configuration and adjustment; the website information requesting device provides the requested website content information site management service to the website information requester according to the queried website resource view. 5. A website content information providing device, characterized in that it comprises: A search module, configured to crawl and search according to the obtained initial link information of the imported website, obtain the link information included in the imported website, and obtain the link object and its attribute information of the linked information; An index module, configured to establish a link object index corresponding to the link information according to the acquired link object and its attribute information; The view resource generation module is used to establish the website resource view of each imported website according to the association relationship between the link object indexes of each of the link information; the website resource view includes the links of each imported website arranged according to the set rules The link object index corresponding to the information; An access retrieval module is used to provide website content information to the website information requester according to the established website resource view. 6. The device according to claim 5, further comprising: a search policy management module; The search strategy management module is configured to configure a search strategy, the search strategy includes but is not limited to one or a combination of the following strategies: depth-first strategy, breadth-first strategy and focused search strategy The search module is specifically configured to perform crawling search according to the obtained initial link information of the imported website and a pre-configured search strategy. 7. The device according to claim 5, wherein the index module is specifically used for: Construct the content index of each link object according to the link objects and their attribute information included in the obtained link information, and determine the association relationship between the link information according to the path information of the link information; establish an association including each link information The link object index of the content index of the link object corresponding to the relation and each link information. 8. The device according to claim 5, wherein the access retrieval module is specifically used for: According to the view resource acquisition request sent by the website information requesting device, provide the established website resource view to the website information requesting device or provide the website information requesting device after configuring and adjusting the established website resource view according to the configuration requirements in the view resource obtaining request , the website information requesting device provides the requested website content information to the website information requester according to the provided website resource view; or Open the query interface to the website information requesting device according to the view resource query request sent by the website information requesting device, and provide the established website resource view to the website information requesting device through the query interface or provide the established website according to the configuration requirements in the view resource query request The resource view is a website resource view after configuration and adjustment; the website information requesting device provides the requested website content information to the website information requester according to the queried website resource view. 9. The device according to any one of claims 5-8, wherein the index module is further used for: Before establishing the link object index corresponding to the link information searched by crawling, data deduplication processing is performed on the link information searched by crawling, as well as the link objects of the link information and their attribute information. 10. The device according to any one of claims 5-8, further comprising: This domain control module is used to control the search scope of the crawling search of the search module. 11. A system for providing website content information, characterized by comprising the apparatus for providing website content information according to any one of claims 5-10 and at least one website information requesting device.

Images