CN106202348A - A kind of web page form information extraction method - Google Patents

Abstract

The invention discloses a method for extracting web page form information, which includes: the user configures the operation file in advance; the user inputs the URL address of the web page to be captured, and the java system captures the web page; the java system captures the captured The webpage is preprocessed; the java system analyzes and locates the webpage according to the information manually input by the user, and at the same time, generates extraction rules and stores them in the rule base for maintenance; the java system extracts the required data from the page according to the extraction rules, and extracts the extracted The data is stored in the MySQL database; the java system uses the JSP page to display the extracted data to the user in the form of a dynamic page according to the configuration of the operation file for user operation. The invention reduces the consumption of system resources in the process of extracting web page form information, speeds up the speed of extracting web page form information, facilitates users to perform secondary processing on the extracted web page form data, and improves the user's secondary processing of web page form data. processing efficiency.

Description

A method for extracting web form information

technical field

The invention relates to the field of web page information extraction, in particular to a method for extracting web page form information.

Background technique

With the rapid development of web technology, web pages can accommodate massive amounts of data, but there are many information that users do not care about, such as advertising images and promotional links, etc., which are even confused with the text content of the web page, making it difficult for users to Quickly get important information from web pages. In addition, if users want to use the target information for other purposes, they can only manually extract the information first, and at the same time remove HTML tags or other noise information, and then reorganize, and finally display the target information according to their wishes. Not only is the accuracy rate low, but it is time-consuming and labor-intensive, and the efficiency is low.

Tables are widely used in web pages in various fields because of their ability to express relational information concisely and effectively. For most Table tags, they are used to display relational data to users, such as train schedules, online Shopping, online banking and management information systems, etc. However, the application of web information extraction technology in tables is limited now. Most of them first process the web pages through the DOM (Document Object Model) tree and then extract the data. If there are many pages to be processed, it will consume a lot of memory; in addition, in the process of processing table data, most applications are limited to extracting the data in the table, and do not perform any possible processing on it. Extract some public operations and make further processing, which often affects the efficiency of secondary processing of data.

In view of this, there is an urgent need to provide a new method for extracting webpage table information, which can solve the problems of large consumption of system resources and low efficiency of secondary processing of webpage data by the existing webpage table information extraction technology.

Contents of the invention

The technical problem to be solved by the present invention is to solve the problems that the existing web page form information extraction technology consumes a lot of system resources and the efficiency of secondary processing of web page data is low.

In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is to provide a method for extracting web page form information, comprising the following steps:

The user configures the operation file in advance;

The user inputs the URL address of the webpage to be captured, and the java system crawls the webpage;

The java system preprocesses the captured web pages;

The java system analyzes and locates the webpage according to the information manually input by the user, and at the same time, generates extraction rules and stores them in the rule base for maintenance;

The java system extracts the required data from the page according to the extraction rules, and stores the extracted data into the MySQL database;

According to the configuration of the operation file, the java system uses the JSP page to display the extracted data to the user in the form of a dynamic page for user operation.

In the above technical solution, the user adds operations to be performed in the operation file according to a pre-defined format, including operations of confirming, modifying and deleting web page form data.

In the above technical solution, the pre-defined format is consistent with the format used when the java system parses the operation file.

In the above technical solution, the preprocessing of the captured webpage by the java system includes removing irrelevant pictures, videos, music, large sections of text, navigation bars and formatting HTML files in the webpage.

In the above technical solution, when the content of the webpage is large, denoising processing is performed on the webpage.

In the above technical solution, the operation file is manipulate.conf.

In the above technical solution, the extraction rule includes information about which table and which column the required data is located on the page.

In the above technical solution, the java system uses the java open source toolkit Jsoup to analyze and locate the webpage.

The invention reduces the consumption of system resources in the process of extracting web page form information, speeds up the speed of web page form information extraction, facilitates users to perform secondary processing on the extracted web page form data, and improves the user's secondary processing of web page form data. processing efficiency.

Description of drawings

Fig. 1 is a flow chart of a method for extracting web page form information provided by an embodiment of the present invention;

FIG. 2 is a data flow chart for reprocessing the extracted webpage form data provided by the embodiment of the present invention.

detailed description

The present invention relates to operation file manipulate.conf configuration, webpage crawling, webpage preprocessing, Jsoup analysis, table positioning based on table sample rows, rule knowledge base maintenance, data extraction, data normalization and data persistence, data and operation display In terms of content, use the user's example input to locate the target form of the web page, and store the data of the target form in the database, and then display it according to the user's wishes, so that the target data can be reused. This reduces the consumption of system resources in the process of web page form information extraction, speeds up the speed of web page form information extraction, facilitates the user's secondary processing of the extracted web page form data, and improves the user's secondary processing of web page form data. s efficiency.

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

The embodiment of the present invention provides a method for extracting web page form information, as shown in Figure 1, comprising the following steps:

S1. The user configures the operation file manipulate.conf in advance according to requirements.

The user configures the operation file manipulate.conf in advance according to the operation he wants to perform on the extracted web page table data, and adds the operation he wants to perform in the operation file manipulate.conf according to the pre-defined format, including the web page table data Confirmation, modification, deletion and other operations, the above-mentioned pre-defined format only needs to be consistent with the format used when the java system parses the operation file manipulate.conf.

S2. The user inputs the URL (Uniform Resource Locators, Uniform Resource Locators) address of the webpage to be captured, and the written java system captures the webpage.

In order to improve the stability of the java system, the URL address entered by the user is required to conform to the URL standard. Otherwise, when the URL address entered does not conform to the URL standard, a corresponding prompt window will pop up when submitting.

S3 and the java system preprocess the captured web pages.

In order to optimize subsequent extraction of webpages, the present invention preprocesses the webpages captured, including removing irrelevant pictures, videos, music, large sections of text, navigation bars and formatting HTML files (i.e. HTML pages XML serialization). At the same time, if the content of the webpage is large, denoising processing may be performed on the webpage.

S4. The java system uses the java open source toolkit Jsoup to analyze and locate the webpage according to the information manually input by the user. At the same time, the extraction rules are generated and stored in the rule base for maintenance.

The user manually enters a line of interest, such as a line of ticket data from the train ticketing network, including voyage number, price, departure point, destination, and remaining number of tickets, etc., so that the java system can quickly locate the data required by the user.

The extraction rules include information such as which table and column the required data is located on the page, and are stored in the rule base to facilitate the extraction of information directly from similar pages on this webpage in the future to save time.

S5. The java system extracts the required data on the page according to the extraction rules, and stores the extracted data into the MySQL database, so as to facilitate secondary processing of the data in the future.

As shown in FIG. 2 , it is a data flow chart for reprocessing the extracted web page form data provided by the embodiment of the present invention.

S6. According to the configuration of the operation file manipulate.conf, the java system uses JSP pages (Java Server Pages, java server pages) to display the extracted data to the user in the form of a dynamic page for user operation.

Example 1.

First define the configuration format of the operation file manipulate.conf as follows:

1: dump browse, 0, none;

2: Single sign-in, 2, sign-in, operation;

3: Double sign-in, 4, employee operation, employee sign-in, supervisor operation, supervisor sign-in.

Assume that a department manager of a company needs everyone in his team to sign in and confirm the team's billing form. However, due to the complex structure of enterprise-level applications and the large number of external resources involved, employees cannot obtain the interface of these data and can only manually copy the form information At the same time, the check-in confirmation cannot be automated, which not only wastes manpower and time, but also cannot ensure that the data is accurate and available. According to the scheme designed in the present invention, the department manager can first configure the operation file manipulate.conf; input the URL address that includes the team account webpage, and grab the webpage; carry out XML serialization processing on the webpage that is grabbed, and at the same time, if the webpage is noisy There are many, which can be denoised; the department manager will input all the data of one of the personal bills, generate extraction rules based on this, and add the extraction rules to the rule base for maintenance; process the web page according to the extraction rules, and extract the required data; store the extracted data in the database; finally, display the finally extracted data and the operations configured in the operation file manipulate. Operation, and the department manager can also see whether the colleagues in the department have checked in or not.

The above method specifically includes the following steps:

S10. The department manager adds the following statement in the operation file manipulate.conf according to the defined format: single check-in, 2, check-in, operation.

S11. The department manager logs in to the service webpage, and enters the URL address of the webpage to be captured in the webpage.

S12. If the webpage captured this time is larger than 4MB, since what is to be extracted this time is bill data, all information such as audio and pictures in the webpage can be removed to speed up table positioning and identification.

S13. The department manager inputs a line of reimbursement data according to the data format of the original web page.

S14. The java system uses the java open source toolkit Jsoup to analyze and locate the webpage according to the bill data input by the department manager, and finally stores the extraction rules obtained this time, such as which table and column are located in the rule base , so that you can directly extract information from similar pages on this website in the future to save time.

S15. The java system stores the data extracted according to the bill data input by the department manager into the MySQL database, so as to facilitate secondary processing of the data in the future.

S16. According to the configuration of the operation file manipulate.conf and the content stored in the MySQL database, the java system uses the JSP page to display the extracted data to the department manager through the browser. On this page, the employees of the department where the department manager belongs can sign in Confirmation, and the manager of the department can also see whether an employee has signed in for confirmation.

The present invention is not limited to the above-mentioned best implementation mode, and any structural changes made by anyone under the inspiration of the present invention, and any technical solutions that are the same as or similar to the present invention, all fall within the protection scope of the present invention.

Claims (8)

1. A webpage form information extraction method, is characterized in that, comprises the following steps: The user configures the operation file in advance; The user inputs the URL address of the webpage to be captured, and the java system crawls the webpage; The java system preprocesses the captured web pages; The java system analyzes and locates the webpage according to the information manually input by the user, and at the same time, generates extraction rules and stores them in the rule base for maintenance; The java system extracts the required data from the page according to the extraction rules, and stores the extracted data into the MySQL database; According to the configuration of the operation file, the java system uses the JSP page to display the extracted data to the user in the form of a dynamic page for user operation. 2. The method according to claim 1, wherein the user adds operations to be performed in the operation file according to a pre-defined format, including operations of confirming, modifying and deleting web page form data. 3. The method according to claim 2, wherein the predefined format is consistent with the format used when the java system parses the operation file. 4. The method according to claim 1, wherein the java system preprocesses the captured webpage including removing irrelevant pictures, videos, music, large sections of text, navigation bars and HTML File formatting. 5. The method according to claim 4, characterized in that, when the content of the webpage is large, denoising processing is performed on the webpage. 6. The method according to claim 1, wherein the operation file is manipulate.conf. 7. The method according to claim 1, wherein the extraction rule includes information about which table and which column the required data is located on the page. 8. The method according to claim 1, wherein the java system uses the java open source toolkit Jsoup to analyze and locate the webpage.