CN101984429B - Method and device for acquiring destination page, search engine and browser - Google Patents

Abstract

The invention provides a method and a device for acquiring a destination page, a search engine and a browser. The method comprises the following steps of: capturing a foundation page corresponding to a received uniform resource locator (URL) and a script of the foundation page by the search engine; and analyzing the captured foundation page and the captured script to generate over one state path comprising dynamic information and corresponding to the foundation page, and capturing the destination page by using the generated state path, wherein the state path comprises the URL of the foundation page, position information of a document object model (DOM) event for generating the dynamic information in the foundation page and a callback function index corresponding to the DOM event. The search engine can capture dynamic contents in the page when searching the destination page.

Description

Method, device, search engine and browser for acquiring target page

technical field

The invention relates to Internet technology, in particular to a method, device, search engine and browser for acquiring a target page.

Background technique

With the rapid development of the network, the Internet has become the carrier of a large amount of information, how to effectively extract and use this information has become a huge challenge. As a tool to assist people in retrieving information, search engines become the entrance and guide for users to access the Internet. A web crawler (Spider) is a program that automatically extracts web pages and is an important component of search engines.

Traditional web crawlers start from the Uniform Resource Locator (URL) of one or several initial web pages, grab the basic page of the URL, and analyze the content of the current basic page to obtain the URL of the target page, and perform data processing, including establishing After the web page is summarized, snapshotted, indexed and stored, it is returned to the browser for the user to choose.

However, traditional web crawlers can only grab static pages when obtaining the URL of the target page. To meet the needs of this transformation, the dynamic content of the page cannot be crawled.

Contents of the invention

The invention provides a method, device, search engine and browser for acquiring a target page, so that the search engine can grab the dynamic content in the page when searching the target page.

The specific technical scheme is as follows:

A method for obtaining a target page, the method includes the following steps:

A. Grab the base page corresponding to the received uniform resource locator URL and the script of the base page;

B, analyze the DOM node downloaded in the basic page of grabbing, judge whether the script corresponding to the DOM event in the DOM node produces dynamic information, produce more than one corresponding to the basic page according to the analysis result and include the dynamic information A state path, using the generated state path to grab the target page; wherein, the state path includes: the URL of the base page, the location information of the DOM event that generates dynamic information in the base page, and the callback function corresponding to the DOM event index.

Wherein, the step B specifically includes:

Download each DOM node during the crawling process of the basic page and script, and execute steps B11 to B13 to the downloaded DOM nodes in turn, until after the download of all DOM nodes ends, execute step B14;

B11, judging whether the currently downloaded DOM node is a script tag, if yes, go to step B11 for the next downloaded DOM node, otherwise, execute step B12;

B12, judging whether the currently downloaded DOM node contains a DOM event and a callback function, if not, proceed to step B11 for the next downloaded DOM node, if yes, execute step B13;

B13. Utilize the DOM events contained in the currently downloaded DOM node to generate a state path, and save the generated state path in the state path queue, and go to step B11 for the next downloaded DOM node;

B14. Obtain the target pages corresponding to each state path in the state queue one by one, judge whether to generate new page content or page jump, and determine the state path that generates new page content or page jump as the basic page Corresponding state path.

Or, the step B specifically includes:

Download each DOM node during the crawling process of the basic page and script, and execute steps B21 to B23 to the downloaded DOM nodes in turn, until the download of all DOM nodes ends;

B21, judge whether the currently downloaded DOM node is a script tag, if so, go to step B21 for the next downloaded DOM node, otherwise execute step B22;

B22, judge whether the currently downloaded DOM node contains a DOM event and a callback function, if not, go to step B21 for the next downloaded DOM node, if yes, execute step B23;

B23, using the DOM event contained in the currently downloaded DOM node to generate a state path;

B24, obtain the target page corresponding to the state path, judge whether to generate new page content or generate a page jump, if so, determine that the state path is the state path corresponding to the basic page, and download to the next DOM node Go to step B21; otherwise, go to step B21 for the next downloaded DOM node.

In the above manner, judging whether a page jump occurs includes: if the acquired target page is different from the URL of the basic page, determining that a page jump occurs.

Specifically, judging whether to generate new page content includes: comparing the acquired target page and the base page with sentence signatures or character strings, and if the comparison result shows that the target page and the base page have different page content, then determine to generate new page content; or,

Calculate the similarity between the acquired target page and the base page, and determine to generate new page content if the calculation result shows that the target page and the base page have different page content.

Wherein, the location information of the DOM event includes: a DOM node identifier, a path Xpath of the DOM node, and a DOM event identifier.

Further, after the step B, the method also includes:

C. Store the state path corresponding to the basic page generated in step B and the snapshot of the captured target page, and establish and store an index of the target page.

A method for obtaining a target page, based on the above method, includes:

After receiving the search request from the browser, match the keywords contained in the search request with the index of the stored target page, include the state path corresponding to the matched target page in the search result and return it to the browser for the browser to Use the state path selected by the user to obtain the corresponding target page.

In addition, the search result may also include: snapshot information of the matched target page;

After receiving the snapshot information of the target page selected by the user returned by the browser, return the snapshot of the corresponding target page to the browser.

Furthermore, after the state path corresponding to the matched target page is included in the search result and returned to the browser, the method further includes:

After receiving the state path selected by the user sent by the browser, a target page request is sent to the target page site according to the state path selected by the user, so that the target page site pushes the target page to the browser.

A method of obtaining a target page, the method comprising:

After the browser sends a search request to the search engine, it receives the search result including the state path returned by the search engine;

Send a target page request to the target page site according to the state path selected by the user;

receiving the target page pushed by the target page site;

Wherein, the search result including the state path is returned by the search engine by using the method described in claim 8 .

A device for acquiring a target page, the device comprising:

The first grabbing unit is used to grab the basic page corresponding to the received uniform resource locator URL and the script of the basic page;

The analysis unit is used to analyze the DOM node downloaded in the basic page captured by the first grabbing unit, determine whether the script corresponding to the DOM event in the DOM node generates dynamic information, and generate dynamic information corresponding to the basic page according to the analysis result Corresponding to more than one state path containing the dynamic information; wherein, the state path includes: the URL of the base page, the location information of the Document Object Model DOM event that generates the dynamic information in the base page, and the callback function corresponding to the DOM event index;

The second grabbing unit is used to grab the target page by using the state path generated by the analysis unit.

Wherein, the analysis unit specifically includes: a first judging module, a second judging module, a first path generating module and a first path determining module;

The first grabbing unit downloads each DOM node during the grabbing process of the basic page and its script, and sends the currently downloaded DOM node to the first judging module until the download of all DOM nodes ends , sending a determination notification to the first path determination module;

The first judging module is used to judge whether the currently downloaded DOM node is a script tag, if so, trigger the first grabbing unit to download the next DOM node, otherwise, send a judgment notification to the second judging module ;

The second judging module is used to judge whether the currently downloaded DOM node contains a DOM event and a callback function, if not, trigger the first grabbing unit to download the next DOM node, and if so, send to the first path The generation module sends an execution notification;

The first path generation module is configured to use the currently downloaded DOM node to generate a state path after receiving the execution notification, and save the generated state path in the state path queue, and trigger the first grabbing unit Download the next DOM node;

The first path determination module is configured to trigger the second grabbing unit to acquire the target pages corresponding to each status path in the status queue one by one when receiving the determination notification, and according to the acquisition result of the second grabbing unit Judging whether new page content is generated or page jump occurs, and the generated new page content or the state path where page jump occurs is determined as the state path corresponding to the basic page.

Specifically, the analysis unit may include: a third judging module, a fourth judging module, a second path generating module, and a second path determining module;

The first grabbing unit downloads each DOM node during the grabbing process of the basic page and its script, and sends the currently downloaded DOM node to the third judging module until the download of all DOM nodes ends;

The third judging module is used to judge whether the currently downloaded DOM node is a script tag, if so, trigger the first grabbing unit to download the next DOM node, otherwise, send a judgment notification to the fourth judging module;

The fourth judging module is used to judge whether the currently downloaded DOM node contains a DOM event and a callback function, if not, trigger the first grabbing unit to download the next DOM node, and if so, send to the second path The generation module sends an execution notification;

The second path generation module is configured to use the DOM event contained in the currently downloaded DOM node to generate a state path when receiving the execution notification, and send the generated state path to the second path determination module;

The second path determination module is configured to trigger the second grabbing unit to acquire the target page corresponding to the status path when receiving the status path, and judge whether to generate a new page according to the acquisition result of the second grabbing unit Content or generate a page jump, if so, determine that the state path is the state path corresponding to the basic page, trigger the first grabbing unit to download the next DOM node, otherwise trigger the first grabbing unit to download the next DOM node DOM node.

Wherein, judging whether a page jump occurs includes: if the acquired target page is different from the URL of the basic page, determining that a page jump occurs.

Judging whether to generate new page content includes: comparing the acquired target page and the base page with sentence signatures or character strings, and if the comparison result shows that the target page and the base page have different page content, then determine to generate a new page content; or,

Calculate the similarity between the acquired target page and the base page, and determine to generate new page content if the calculation result shows that the target page and the base page have different page content.

Specifically, the location information of the DOM event includes: a DOM node identifier, a path Xpath of the DOM node, and a DOM event identifier.

Furthermore, the device also includes:

The storage unit is configured to store the state path corresponding to the basic page generated by the analysis unit and the snapshot of the target page captured by the second capture unit, and to establish and store an index of the target page.

A search engine, the search engine comprising: the above-mentioned device for acquiring a target page, a user interface unit and a search processing unit;

The user interface unit is configured to receive a search request from a browser, and send keywords contained in the search request to the search processing unit; return the search results sent by the search processing unit to the browser, For the browser to use the state path selected by the user to obtain the corresponding target page;

The search processing unit is configured to match the keyword with the index of the target page stored in the storage unit in the device, include the state path corresponding to the matched target page in the search result and send it to the user interface unit.

Furthermore, the search result also includes: snapshot information of the matched target page;

The user interface unit is further configured to send the snapshot information of the target page selected by the user returned by the browser to the search processing unit; return the snapshot of the target page sent by the search processing unit to the browser;

The search processing unit is further configured to, according to the snapshot information of the target page selected by the user, acquire a snapshot of the corresponding target page from the storage unit and send it to the user interface unit.

Furthermore, the search engine also includes: a path analysis unit and a network interface unit;

The user interface unit is further configured to send the state path to the path analysis unit after receiving the state path selected by the user sent by the browser;

The path parsing unit is configured to generate a target page request according to the received state path;

The network interface unit is configured to send the target page request generated by the path analysis unit to the target page site.

A browser, the browser includes: a network side interface unit, a path analysis unit and a user side interface unit;

The network-side interface unit is configured to receive the search result including the state path sent by the search engine according to claim 19; send the target page request sent by the path analysis unit to the target page site;

The user-side interface unit is configured to display the search results received by the network-side interface unit to the user; send the state path selected by the user to the path analysis unit;

The path analysis unit is configured to generate a target page request according to the status path selected by the user and send it to the network side interface unit.

It can be seen from the above technical solutions that the present invention introduces the concept of state path based on the analysis of the basic page and its script, that is, generates a state path corresponding to the basic page that contains dynamic information, and the target page pointed to by the state path contains the dynamic information of the page. Content, so that subsequent search engines can crawl the dynamic content in the page when searching for the target page.

Description of drawings

Fig. 1 is the main method flowchart that the present invention provides;

FIG. 2 is a detailed method flowchart provided by Embodiment 1 of the present invention;

FIG. 3 is a flow chart of generating a state path provided by Embodiment 2 of the present invention;

FIG. 4 is a flow chart of generating a state path provided by Embodiment 3 of the present invention;

FIG. 5 is a flow chart of a browser obtaining a target page provided by Embodiment 4 of the present invention;

FIG. 6 is a flow chart of the browser obtaining the target page provided by Embodiment 5 of the present invention;

FIG. 7 is a flow chart of the browser obtaining a target snapshot provided by Embodiment 6 of the present invention;

Fig. 8 is a schematic diagram of the structure diagram of the device provided by the present invention;

Fig. 9 is a schematic structural diagram of the analyzing unit in Fig. 8;

Fig. 10 is another schematic structural diagram of the analysis unit in Fig. 8;

Fig. 11 is a schematic structural diagram of a search engine provided by the present invention;

Fig. 12 is a schematic structural diagram of a browser provided by the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

The main method provided by the present invention can be as shown in Figure 1, comprises the following steps:

Step 101: Grab the basic page corresponding to the received URL and the script of the basic page.

Step 102: Analyze the captured basic page and script, and generate more than one status path containing dynamic information corresponding to the basic page; wherein, the status path includes: the URL of the basic page, the document object generating dynamic information in the basic page The location information of the model (DOM) event and the callback function index corresponding to the DOM event that generates dynamic information.

Step 103: Use the generated state path to grab the target page.

The method flow shown in Figure 1 above is the operation performed by the search engine. Further, the search engine will also store the generated state path, so that after receiving the search request from the browser, it will return to the browser containing the state path The search results for the browser to use the state path selected by the user to obtain the corresponding target page.

The above method will be described in detail below through specific examples.

Embodiment one,

Fig. 2 is a detailed method flowchart provided by Embodiment 1 of the present invention. As shown in Fig. 2, the method may specifically include the following steps:

Step 201: The search engine receives the URL.

Search engines can automatically crawl URLs in batches in the background.

Step 202: Grab the basic page corresponding to the received URL and the script of the basic page.

There may be the following two types of correspondence between the basic page and the script: first, the script document exists in the HTML tag included in the source code of the basic page. Second, there is a link to the script document in the HTML tag contained in the original code of the basic page, and the link of the script document points to the script document; that is to say, the basic page and the script document are two different documents, but there is a reference relationship.

Step 203: Analyze the DOM nodes downloaded in the captured basic page, determine whether the script corresponding to the DOM event in the DOM node generates dynamic information, and generate more than one state path corresponding to the basic page containing dynamic information according to the analysis result And using the state path to obtain the target page; wherein, the state path includes: the URL of the base page, the location information of the DOM event that generates dynamic information in the base page, and the callback function index corresponding to the DOM event that generates dynamic information.

The script language involved in the present invention includes but not limited to: javascript, vbscript, perl or python.

Wherein, the location information of the DOM event may include: a DOM node identifier, a path (Xpath) of the DOM node, and a DOM event identifier. Wherein, the DOM node identifier may be: the ID of the DOM node or the name of the DOM node.

The callback function index in the state path is used to reference the callback function corresponding to the DOM event. All callback functions in the script have indexes, and the corresponding relationship between the indexes and specific callback functions can be stored through data structures such as global function tables and mapping functions. The callback function corresponding to the DOM event can be obtained by querying the data structure containing the corresponding relationship between the index and the specific callback function through the callback function index in the state path. The callback function here may include: an anonymous callback function and a non-anonymous callback function.

For the state path, after compiling and executing the callback function corresponding to the DOM event, the corresponding target page can be obtained.

The specific implementation of this step will be described in detail in Embodiment 2 and Embodiment 3.

For a basic page, it may correspond to N state paths and N target pages, where N may be an integer greater than one.

For example, for the base page whose URL is www.baidu.com, the two generated state paths can be:

{base_url: http://www.baidu.com, id: idsample1, xpath: html/body/a/, event: click, type: new_content, callback: fun1}

{base_url: http://www.baidu.com, id: idsample2, xpath: html/body/li/a/, event: click, type: new_link, callback: fun2}

It should be noted that the present invention does not limit the specific format of the state path, and the above is only one example.

Step 204: Store the state path corresponding to the basic page and the snapshot of the target page corresponding to the state path, establish and store an index of the target page, so that it can be found by the search engine and returned to the browser as a search result.

In this embodiment, the basic page captured in step 202 and its script may be stored, the status path generated in step 203 may be stored, and the snapshot of the target page acquired in step 203 may be stored. Wherein, the storage of the basic page may specifically include: the URL of the basic page, a snapshot of the basic page, and the like.

The process for the search engine to obtain the target page can be executed periodically or manually triggered. Each time the state path corresponding to the basic page is generated, if there is a stored state path, the generated state path corresponding to the basic page can be compared with the stored state path corresponding to the basic page, and if they are different, update in time The state path corresponding to the stored base page.

In addition, the search engine can regularly check whether the target page is updated according to the index of the target page, and update the stored index of the target page in time. Likewise, if the target page snapshot acquired each time is different from the stored target page snapshot, the stored target page snapshot may be replaced with a newly acquired target page snapshot.

The above three types of stored content can be stored independently or combined.

The above steps 201 to 204 are all operations of the search engine in the background. If the search engine receives a search request from the browser, it will continue to execute the following steps in the foreground.

Step 205: After receiving the search request from the browser, match the keywords contained in the search request with the indexes of each target page, include the state path corresponding to the matched target page in the search result and return it to the browser for The browser uses the state path selected by the user to obtain the corresponding target page.

When the search engine receives a search request containing keywords, in addition to the index of the target page participating in the matching, the index of the basic page will also participate in the matching, that is to say, the basic page will also be included in the search results. There are technologies that are the same, and will not be described in detail.

Furthermore, the search result may also include snapshot information of the target page, or may also include an index of the target page.

In this step, how the browser obtains the corresponding target page using the state path selected by the user refers to Embodiment 4 and Embodiment 5.

The manner of generating the state path in the above step 203 can adopt the second embodiment and the third embodiment.

Embodiment two,

FIG. 3 is a flow chart of generating a state path provided by Embodiment 2 of the present invention. As shown in FIG. 3 , it may specifically include the following steps:

Step 301: Download each DOM node during the crawling process of the basic page and its script.

Step 302: Determine whether to end the download of the DOM node, if yes, end the crawling process of the basic page, and go to step 306; otherwise, execute step 303 for the currently downloaded DOM node.

Step 303: Determine whether the currently downloaded DOM node is a script tag, if yes, go to step 302 for the next downloaded DOM node; otherwise, execute step 304.

For a node with a script tag, the script corresponding to the script tag can be sent to the script parsing engine for compilation and execution.

Step 304: Determine whether the DOM node contains a DOM event and a callback function, if not, skip the analysis of the DOM node, and go to step 302 for the next downloaded DOM node; if yes, execute step 305.

If the DOM node does not contain DOM events and callback functions, it means that the DOM node will not cause page jumps and new page content, that is, no page dynamic information will be generated, and the DOM node can be skipped. If there is a next DOM node, start the analysis of the next DOM node.

Step 305: Use the DOM event contained in the DOM node to generate a state path, and save the generated state path in the state path queue; go to step 302 for the next downloaded DOM node.

Step 306: Obtain the target pages corresponding to each state path in the state queue one by one, determine whether new page content or page jump occurs, and determine the state path that generates new page content or page jump occurs as the basic page correspondence state path.

Then, the state paths that generate new page content or page jumps and their corresponding target pages can be stored.

A manner of judging whether a page jump occurs may be as follows: if the URLs of the target page and the base page are different, then it is determined that a page jump occurs. The method of judging whether to generate new page content can be: perform sentence signature or character string comparison between the target page and the base page, or calculate the similarity between the target page and the base page, if the comparison result or the similarity calculation result shows that the target page If the page and the base page have different page contents, it is determined to generate new page contents. Wherein, when comparing sentence signatures, existing calculation methods such as MD5 can be used for the calculation of sentence signatures, and there is no specific limitation here.

In the second embodiment, the state paths generated by DOM events are first stored in the state path queue, but because the state paths of DOM events do not necessarily generate page dynamic information, there may also be some invalid state paths, so , and then judge each state path in the state path queue one by one to determine whether the target page corresponding to the state path queue contains dynamic information. The flow from step 303 to step 305 is the process of analyzing each DOM node to initially generate a state path, that is to say, execute step 303 to step 305 for each downloaded DOM node until all DOM nodes are downloaded and then execute step 306 Finally determine the state path corresponding to the basic page.

Embodiment three,

FIG. 4 is a flow chart of generating a state path provided by Embodiment 3 of the present invention. As shown in FIG. 4 , it may specifically include the following steps:

Step 401: Download each DOM node during the crawling process of the basic page and its script.

Step 402: Determine whether to end the download of the DOM node, if yes, end the crawling process of the basic page; otherwise, execute step 403 for the currently downloaded DOM node.

Step 403: Determine whether the currently downloaded DOM node is a script tag, if yes, go to step 402 for the next downloaded DOM node; otherwise, execute step 404.

For a node with a script tag, the script corresponding to the script tag can be sent to the script parsing engine for compilation and execution.

Step 404: Determine whether the DOM node contains a DOM event and a callback function, if not, skip the analysis of the DOM node, and go to step 402 for the next downloaded DOM node; if yes, execute step 405.

Step 405: Utilize the DOM event in the DOM node to generate a state path.

In this step, a state path can be generated for all DOM events, more preferably, a state path can also be generated for DOM events in the preset DOM event list. The DOM events in the preset DOM event list may include: onclick, ondbclick, onmouseover, onmousemove, onmouseout, onblur, onfocus, onchange, onsubmit, onselect, etc. These DOM events are DOM events that may generate page dynamic information.

Step 406: Obtain the target page corresponding to the state path, judge whether to generate new page content or page jump, if so, execute step 407; otherwise, go to step 402 for the next downloaded DOM node.

Step 407: Determine that the state path is the state path corresponding to the basic page, store the state path and its corresponding target page, and go to step 402 for the next downloaded DOM node.

Different from embodiment two, in embodiment three, every time a state path is generated, a judgment is made to determine whether the target page corresponding to the state path queue contains dynamic information (ie step 406), and if so, store the state path and its corresponding of the target page. Steps 403 to 407 are the process of analyzing each downloaded DOM node to generate a state path, that is, execute steps 403 to 407 for each downloaded DOM node until all DOM nodes are downloaded.

So far, the process shown in the third embodiment ends.

In the above-mentioned second and third embodiments, when obtaining the target page corresponding to the status path and judging whether to generate new page content or page jump, the callback function index corresponding to the DOM event will be sent to the script for analysis engine, the script parsing engine obtains the corresponding callback function according to the callback function index, executes the target page corresponding to the state path according to the result of compiling and executing the obtained callback function, and judges whether to generate new page content or generate a page jump turn steps. Among them, for anonymous functions, the script parsing engine will compile and execute the obtained callback functions in real time after obtaining the corresponding callback functions, while for non-anonymous functions, the script parsing engine will compile and execute the obtained callback functions in real time after obtaining the corresponding callback functions , you can use the previous compilation and execution results of the callback function.

The manner in which the browser obtains the target page by using the state path can be divided into two types according to whether the browser has the function of parsing the state path, which will be described through Embodiment 4 and Embodiment 5 respectively.

Embodiment four,

When the browser has the function of parsing the state path, the corresponding flow chart is shown in Figure 5, including the following steps:

Step 501: the browser sends a search request (Query) including keywords to the search engine.

Step 502: The search engine executes step 205 and returns the search result including the state path to the browser.

Step 503: The browser sends a target page request to the target page site according to the state path selected by the user.

When the user clicks the state path of the target page, the browser parses the state path clicked by the user, and sends a target page request to the target page site according to the state path.

Step 504: The target page site pushes the target page to the browser.

Embodiment five,

When the browser does not have the function of parsing the state path, the corresponding flow chart is shown in Figure 6, including the following steps:

Step 601: the browser sends a search request including keywords to the search engine.

Step 602: The search engine executes step 205 and returns the search result including the state path to the browser.

Step 603: the browser sends the state path selected by the user to the search engine.

Step 604: The search engine sends a target page request to the target page site according to the status path selected by the user.

Since the browser does not have the state path parsing function, the browser only sends the state path selected by the user to the search engine, and the search engine parses the state path and sends a target page request to the target page site according to the state path.

Step 605: The target page site pushes the target page to the browser.

The target page request sent by the search engine will include browser information so that the target page site will push the target page to the browser.

So far, the process shown in the fifth embodiment ends.

In another case, if the search engine returns the target page snapshot information in step 205 of the first embodiment, if the user clicks the target page snapshot, the interaction between the browser and the search engine can be performed according to the sixth embodiment .

Embodiment six,

Fig. 7 is the flowchart that the browser that embodiment 6 provides obtains target snapshot, as shown in Fig. 7, may include the following steps:

Step 701: the browser sends a search request including keywords to the search engine.

Step 702: The search engine executes step 205 and returns the search result including the status path and target page snapshot information to the browser.

Step 703: the browser sends the snapshot information of the target page selected by the user to the search engine.

Step 704: The search engine determines the corresponding target page snapshot and returns it to the browser.

Since the search engine has stored the snapshots of the target pages locally, there is no need to interact with the target page site, and the corresponding target page snapshots are directly acquired locally and returned to the browser.

The above is a detailed description of the method provided by the present invention. The following is a detailed description of the device for obtaining the target page provided by the present invention. As shown in FIG. 8 , the device may include: a first grabbing unit 800, an analyzing unit 810 and The second grabbing unit 820 .

The first grabbing unit 800 is configured to grab the basic page corresponding to the received URL and the script of the basic page.

The analysis unit 810 is used to analyze the basic page and script captured by the first grabbing unit 800, and generate more than one state path corresponding to the basic page that contains dynamic information; wherein, the state path includes: the URL of the basic page, the basic page The location information of the DOM event that generates the dynamic information and the callback function index corresponding to the DOM event.

The second grabbing unit 820 is configured to grab the target page using the state path generated by the analysis unit 810 .

Wherein, the analysis unit 810 can adopt two structures, the first structure is as shown in FIG. .

The first grabbing unit 800 downloads each DOM node during the grabbing process of the basic page and its script, and sends the currently downloaded DOM node to the first judging module 811. The path determination module 814 sends a determination notification.

The first judging module 811 is used to judge whether the currently downloaded DOM node is a script tag, if so, trigger the first grabbing unit 800 to download the next DOM node, otherwise, send a judgment notification to the second judging module 812 .

The second judging module 812 is used to judge whether the currently downloaded DOM node contains a DOM event and a callback function, if not, triggers the first grabbing unit 800 to download the next DOM node, and if so, sends it to the first path generation module 813 Execution notification.

The first path generating module 813 is configured to use the currently downloaded DOM node to generate a state path after receiving the execution notification, and save the generated state path in the state path queue, triggering the first grabbing unit 800 to download the next DOM node.

The first path determination module 814 is configured to trigger the second grabbing unit 820 to acquire the target pages corresponding to each status path in the status queue one by one when receiving the confirmation notification, and judge whether to generate a new one according to the acquisition result of the second grabbing unit 820 The page content or the page jump occurs, and the generated new page content or the state path where the page jump occurs is determined as the state path corresponding to the base page.

In addition, the second structure of the analyzing unit 810 is shown in FIG. 10 , which may specifically include: a third judging module 911 , a fourth judging module 912 , a second path generating module 913 and a second path determining module 914 .

The first grabbing unit 800 downloads each DOM node during the grabbing process of the basic page and its script, and sends the currently downloaded DOM node to the third judging module 911 until the download of all DOM nodes ends.

The third judging module 911 is used to judge whether the currently downloaded DOM node is a script tag, if so, trigger the first grabbing unit 800 to download the next DOM node, otherwise, send a judgment notification to the fourth judging module 912 .

The fourth judging module 912 is used to judge whether the currently downloaded DOM node contains a DOM event and a callback function, if not, trigger the first grabbing unit 800 to download the next DOM node, and if so, send it to the second path generation module 913 Execution notification.

The second path generation module 913 is configured to use the DOM events contained in the currently downloaded DOM node to generate a state path when receiving the execution notification, and send the generated state path to the second path determination module 914 .

The second path determination module 914 is configured to trigger the second grabbing unit 820 to acquire the target page corresponding to the status path when receiving the status path, and judge whether to generate new page content according to the acquisition result of the second grabbing unit 820 Or generate a page jump, if yes, determine that the state path is the state path corresponding to the basic page, trigger the first grabbing unit 800 to download the next DOM node, otherwise trigger the first grabbing unit 800 to download the next DOM node.

Specifically, when applied to the above two structures, judging whether a page jump occurs may include: if the obtained URLs of the target page and the base page are different, determining that a page jump occurs.

Judging whether to generate new page content may include: comparing the acquired target page and the base page with sentence signatures or character strings, and if the comparison result shows that the target page and the base page have different page content, then determine to generate new page content ; or, calculate the similarity between the acquired target page and the base page, and if the calculation result shows that the target page and the base page have different page content, determine to generate new page content.

Wherein, the above-mentioned location information of the DOM event in the state path includes: a DOM node identifier, an Xpath of the DOM node, and a DOM event identifier.

Furthermore, the device may also include:

The storage unit 830 is configured to store the state path corresponding to the basic page generated by the analysis unit 810 and the snapshot of the target page captured by the second capture unit 820, and establish and store an index of the target page.

In addition, the storage unit 830 will also store the basic page captured by the first capture unit 800, wherein, the three parts of the basic page, the state path and the snapshot of the target page can be stored independently or in a unified manner. The way.

FIG. 11 is a schematic structural diagram of a search engine provided by the present invention. As shown in FIG. 11 , the search engine includes: the device shown in FIG. 8 , a user interface unit 1101 and a search processing unit 1102 .

The user interface unit 1101 is configured to receive a search request from the browser, and send the keywords contained in the search request to the search processing unit 1102; return the search results sent by the search processing unit 1102 to the browser for the browser to Use the state path selected by the user to obtain the corresponding target page.

The search processing unit 1102 is configured to match the keyword with the index of the target page stored in the storage unit 830 , include the state path corresponding to the matched target page in the search result and send it to the user interface unit 1101 .

More preferably, the search result may further include: snapshot information of the matched target page. at this time,

The user interface unit 1101 is further configured to send the snapshot information of the target page selected by the user returned by the browser to the search processing unit 1102; and return the snapshot of the target page sent by the search processing unit 1102 to the browser.

The search processing unit 1102 is further configured to obtain a snapshot of the corresponding target page from the storage unit 830 according to the snapshot information of the target page selected by the user and send it to the user interface unit 1101 .

Furthermore, when the browser does not have the function of parsing the state path, the search engine needs to have this function to assist in pushing the target page to the browser. At this point, the search engine may further include: a path parsing unit 1103 and a network interface unit 1104 .

The user interface unit 1101 is further configured to send the state path to the path analysis unit 1103 after receiving the state path selected by the user sent by the browser.

The path parsing unit 1103 is configured to generate a target page request according to the received status path.

The network interface unit 1104 is configured to send the target page request generated by the path analysis unit 1103 to the target page site.

FIG. 12 is a structural diagram of a browser provided by the present invention. The browser has a state path analysis function. As shown in FIG.

The network side interface unit 1201 is configured to receive the search result including the state path sent by the search engine shown in FIG. 11 ; and send the target page request sent by the path analysis unit 1202 to the target page site.

The user-side interface unit 1203 is configured to display the search results received by the network-side interface unit 1201 to the user; and send the state path selected by the user to the path analysis unit 1202 .

The path analysis unit 1202 is configured to generate a target page request according to the status path selected by the user and send it to the network side interface unit 1201 .

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (22)

1. A method for obtaining a target page is characterized by comprising the following steps:

A. capturing a basic page corresponding to the received uniform resource locator URL and a script of the basic page;

B. analyzing DOM nodes downloaded from the captured basic pages, judging whether scripts corresponding to DOM events in the DOM nodes generate dynamic information or not, generating more than one state path containing the dynamic information corresponding to the basic pages according to the analysis result, and capturing target pages by using the generated state paths; wherein the state path comprises: the method comprises the steps of obtaining URL of a basic page, position information of a Document Object Model (DOM) event generating dynamic information in the basic page and a callback function index corresponding to the DOM event.

2. The method according to claim 1, wherein step B specifically comprises:

downloading each DOM node in the grabbing process of the basic page and the script, and sequentially executing steps B11 to B13 on the downloaded DOM nodes until the downloading of all the DOM nodes is finished, and then executing step B14;

b11, judging whether the currently downloaded DOM node is a script tag, if so, transferring to the step B11 for the next downloaded DOM node, otherwise, executing the step B12;

b12, judging whether the currently downloaded DOM node contains a DOM event and a call-back function, if not, transferring to the step B11 for the next downloaded DOM node, and if so, executing the step B13;

b13, generating a state path by using the DOM event contained in the currently downloaded DOM node, storing the generated state path in a state path queue, and turning to the step B11 for the next downloaded DOM node;

b14, acquiring the target pages corresponding to each state path in the state queue one by one, judging whether to generate new page content or generate page jump, and determining the state path generating the new page content or generating the page jump as the state path corresponding to the basic page.

3. The method according to claim 1, wherein step B specifically comprises:

downloading each DOM node in the grabbing process of the basic page and the script, and sequentially executing the steps B21 to B23 on the downloaded DOM nodes until the downloading of all the DOM nodes is finished;

b21, judging whether the currently downloaded DOM node is a script tag, if so, transferring to the step B21 for the next downloaded DOM node, otherwise, executing the step B22;

b22, judging whether the currently downloaded DOM node contains a DOM event and a call-back function, if not, transferring to the step B21 for the next downloaded DOM node, and if so, executing the step B23;

b23, generating a state path by using the DOM event contained in the currently downloaded DOM node;

b24, acquiring a target page corresponding to the state path, judging whether to generate new page content or generate page jump, if so, determining that the state path is the state path corresponding to the basic page, and turning to the step B21 for the next downloaded DOM node; otherwise, go to step B21 for the next downloaded DOM node.

4. The method of claim 2 or 3, wherein determining whether a page jump has occurred comprises: and if the obtained URL of the target page is different from that of the basic page, determining that page jump occurs.

5. The method of claim 2 or 3, wherein determining whether to generate new page content comprises: carrying out sentence signature or character string comparison on the acquired target page and the basic page, and if the comparison result shows that the target page and the basic page have different page contents, determining to generate new page contents; or,

and calculating the similarity between the acquired target page and the basic page, and determining to generate new page content if the calculation result shows that the target page and the basic page have different page contents.

6. A method according to any of claims 1 to 3, wherein the location information of DOM events comprises: DOM node identification, path Xpath of DOM node and DOM event identification.

7. A method according to any one of claims 1 to 3, characterized in that after said step B, the method further comprises:

C. and B, storing the state path corresponding to the basic page generated in the step B and the snapshot of the captured target page, and establishing and storing an index of the target page.

8. A method for obtaining a target page, the method according to claim 7 being followed by:

after receiving a search request from a browser, matching keywords contained in the search request with indexes of stored target pages, including a state path corresponding to the matched target page in a search result, and returning the search result to the browser, so that the browser can obtain the corresponding target page by using the state path selected by a user.

9. The method of claim 8, wherein the search results further comprise: snapshot information of the matched target page;

and after receiving snapshot information of the target page selected by the user and returned by the browser, returning a corresponding snapshot of the target page to the browser.

10. The method of claim 8, wherein after including the status path corresponding to the matched target page in the search result and returning the search result to the browser, the method further comprises:

and after receiving the state path selected by the user and sent by the browser, sending a target page request to a target page site according to the state path selected by the user, so that the target page site can push a target page to the browser.

11. A method for obtaining a target page is characterized by comprising the following steps:

the browser receives a search result containing a state path returned by a search engine after sending a search request to the search engine;

sending a target page request to a target page site according to a state path selected by a user;

receiving a target page pushed by the target page site;

wherein the search result containing the state path is returned by the search engine using the method of claim 8.

12. An apparatus for obtaining a target page, the apparatus comprising:

the first grabbing unit is used for grabbing a basic page corresponding to the received uniform resource locator URL and a script of the basic page;

the analysis unit is used for analyzing the DOM nodes downloaded from the basic page captured by the first capture unit, judging whether scripts corresponding to DOM events in the DOM nodes generate dynamic information or not, and generating more than one state path which corresponds to the basic page and contains the dynamic information according to an analysis result; wherein the state path comprises: the method comprises the steps that URL of a basic page, position information of a Document Object Model (DOM) event generating dynamic information in the basic page and a callback function index corresponding to the DOM event are obtained;

and the second grabbing unit is used for grabbing the target page by using the state path generated by the analysis unit.

13. The apparatus according to claim 12, wherein the analysis unit comprises in particular: the device comprises a first judgment module, a second judgment module, a first path generation module and a first path determination module;

the first grabbing unit downloads each DOM node in the grabbing process of the basic page and the script thereof, sends the currently downloaded DOM node to the first judging module, and sends a confirmation notice to the first path determining module after finishing the downloading of all the DOM nodes;

the first judging module is used for judging whether the currently downloaded DOM node is a script tag or not, if so, triggering the first grabbing unit to download the next DOM node, and otherwise, sending a judgment notice to the second judging module;

the second judging module is used for judging whether the currently downloaded DOM node contains a DOM event and a callback function, if not, the first capturing unit is triggered to download the next DOM node, and if so, an execution notice is sent to the first path generating module;

the first path generation module is used for generating a state path by using the currently downloaded DOM node after receiving the execution notification, storing the generated state path in a state path queue and triggering the first capture unit to download the next DOM node;

the first path determining module is configured to, when receiving the determination notification, trigger the second capturing unit to obtain target pages corresponding to each state path in the state queue one by one, determine whether to generate new page content or generate page jump according to an obtaining result of the second capturing unit, and determine the generated new page content or the state path in which the page jump occurs as the state path corresponding to the basic page.

14. The apparatus according to claim 12, wherein the analysis unit comprises in particular: the device comprises a third judging module, a fourth judging module, a second path generating module and a second path determining module;

the first grabbing unit downloads each DOM node in the grabbing process of the basic page and the script thereof, and sends the currently downloaded DOM node to the third judging module until the downloading of all DOM nodes is finished;

the third judging module is used for judging whether the currently downloaded DOM node is a script tag or not, if so, the first grabbing unit is triggered to download the next DOM node, and otherwise, a judgment notice is sent to the fourth judging module;

the fourth judging module is used for judging whether the currently downloaded DOM node contains a DOM event and a callback function, if not, the first capturing unit is triggered to download the next DOM node, and if so, an execution notification is sent to the second path generating module;

the second path generating module is configured to generate a state path by using a DOM event included in a currently downloaded DOM node when receiving the execution notification, and send the generated state path to the second path determining module;

and the second path determining module is used for triggering the second capturing unit to obtain a target page corresponding to the state path when the state path is received, judging whether new page content is generated or page jump is generated according to the obtaining result of the second capturing unit, if so, determining that the state path is the state path corresponding to the basic page, and triggering the first capturing unit to download a next DOM node, otherwise, triggering the first capturing unit to download the next DOM node.

15. The apparatus of claim 13 or 14, wherein determining whether a page jump occurs comprises: and if the obtained URL of the target page is different from that of the basic page, determining that page jump occurs.

16. The apparatus of claim 13 or 14, wherein determining whether to generate new page content comprises: carrying out sentence signature or character string comparison on the acquired target page and the basic page, and if the comparison result shows that the target page and the basic page have different page contents, determining to generate new page contents; or,

and calculating the similarity between the acquired target page and the basic page, and determining to generate new page content if the calculation result shows that the target page and the basic page have different page contents.

17. The apparatus according to any of claims 12 to 14, wherein the location information of the DOM event comprises: DOM node identification, path Xpath of DOM node and DOM event identification.

18. The apparatus of any one of claims 12 to 14, further comprising:

and the storage unit is used for storing the state path corresponding to the basic page generated by the analysis unit and the snapshot of the target page captured by the second capture unit, and establishing and storing the index of the target page.

19. A search engine, comprising: the apparatus, user interface unit, and search processing unit of claim 18;

the user interface unit is used for receiving a search request from a browser and sending a keyword contained in the search request to the search processing unit; returning the search result sent by the search processing unit to the browser, so that the browser can obtain a corresponding target page by using the state path selected by the user;

and the search processing unit is used for matching the keyword with the index of the target page stored in the storage unit of the device, and sending the state path corresponding to the matched target page to the user interface unit by including the state path in the search result.

20. The search engine of claim 19, further comprising, in the search results: snapshot information of the matched target page;

the user interface unit is also used for sending the snapshot information of the target page selected by the user and returned by the browser to the search processing unit; returning the snapshot of the target page sent by the search processing unit to the browser;

the search processing unit is further configured to obtain a snapshot of the corresponding target page from the storage unit according to the snapshot information of the target page selected by the user, and send the snapshot to the user interface unit.

21. The search engine of claim 19, further comprising: a path analysis unit and a network interface unit;

the user interface unit is also used for sending the state path to the path analysis unit after receiving the state path selected by the user and sent by the browser;

the path analysis unit is used for generating a target page request according to the received state path;

and the network interface unit is used for sending the target page request generated by the path analysis unit to a target page site.

22. A browser, comprising: the system comprises a network side interface unit, a path analysis unit and a user side interface unit;

the network side interface unit, configured to receive a search result including a status path sent by the search engine according to claim 19; sending the target page request sent by the path analysis unit to a target page site;

the user side interface unit is used for displaying the search result received by the network side interface unit to a user; sending the state path selected by the user to the path analysis unit;

and the path analysis unit is used for generating a target page request according to the state path selected by the user and sending the target page request to the network side interface unit.

Images