CN111143650B - Method, device, medium and electronic device for obtaining page data - Google Patents

Abstract

The disclosure relates to a method, a device, a medium and an electronic device for acquiring page data, wherein the method comprises the following steps: receiving an original request of page operation; acquiring characteristic parameters of the original request, and storing the characteristic parameters into a target structure; generating a data acquisition request according to the target structure body, wherein the data acquisition request is used for requesting to acquire page data of a response page of the original request; sending the data acquisition request to a server; and receiving the page data of the response page sent by the server. Therefore, when the page data of the response page of the original request is acquired, the data acquisition request is generated according to the characteristic parameters of the original request, so that the data is requested to the server based on the data acquisition request, the page data can be acquired without simulating the request response operation of the browser, the complexity of acquiring the page data can be effectively reduced, and the efficiency and accuracy of acquiring the page data are improved.

Description

Method, device, medium and electronic device for obtaining page data

Technical Field

The present disclosure relates to the field of computer technology, and in particular, to a method, device, medium and electronic device for acquiring page data.

Background technique

With the development of computer technology, the demand for network data is increasing. In some scenarios, users want to obtain data from a certain page. However, if the page does not provide a data export function, it is difficult for users to obtain data. In order to solve the above problem, in the prior art, the page data of such pages is usually obtained based on crawler technology.

However, in the prior art, the implementation of web crawler technology is usually based on a headless browser to simulate page operations to obtain data in the page. In this process, since it is carried out by simulating user operations, it is necessary to ensure that any step in the simulation process is consistent with the user operation and no errors are made. Otherwise, if any step in the middle makes a mistake, the entire process will fail and the page data cannot be obtained.

Summary of the invention

The purpose of the present disclosure is to provide a method, device, medium and electronic device for obtaining page data conveniently and accurately.

In order to achieve the above objective, according to a first aspect of the present disclosure, a method for acquiring page data is provided, the method comprising:

Receive the original request for page operation;

Obtaining characteristic parameters of the original request, and storing the characteristic parameters in a target structure;

Generate a data acquisition request according to the target structure, wherein the data acquisition request is used to request to acquire page data of a response page of the original request;

Sending the data acquisition request to the server;

Receive page data of the response page sent by the server.

Optionally, before the step of requesting the server for data acquisition, the method further includes:

If it is determined that a login operation is required before responding to the original request, obtaining a response token corresponding to the login operation;

The step of generating a data acquisition request according to the target structure includes:

A data acquisition request is generated according to the target structure and the response token.

Optionally, obtaining a response token corresponding to the login operation includes:

If there is a login script corresponding to the original request, the login script is replayed to obtain a response token after the login information used by the login script is authenticated, wherein the login script is pre-recorded and generated based on the login plug-in.

Optionally, during the playback of the login script, if the current operation is a verification code input operation, the verification code image in the current page is obtained, and the verification code image is sent to a verification code recognition module, so that the verification code recognition module recognizes the verification code image and obtains verification code information; the verification code input operation is performed based on the verification code information obtained by the verification code recognition module.

Optionally, obtaining a response token corresponding to the login operation includes:

If a login operation is required before responding to the original request, and there is no login script corresponding to the original request, output a prompt message to prompt the user to perform the login operation;

Detect login status through login plugin;

A response token is received from the login plug-in when a successful login is detected.

Optionally, after the step of receiving the page data of the response page sent by the server, the method further includes:

Determine a text similarity parameter corresponding to the first page text information of the response page and the second page text information of the response sample corresponding to the original request;

Determining whether the response page matches the response sample according to the text similarity parameter;

If it is determined that the response page matches the response sample, the page data of the response page is stored.

Optionally, storing the page data of the response page includes:

The page data of the response page is stored in a structured file, so as to be stored in a database based on the structured file.

According to a second aspect of the present disclosure, a device for acquiring page data is provided, the device comprising:

A first receiving module, used for receiving an original request for a page operation;

A first storage module, used for obtaining characteristic parameters of the original request and storing the characteristic parameters in a target structure;

A generating module, used for generating a data acquisition request according to the target structure, wherein the data acquisition request is used for requesting to acquire page data of a response page of the original request;

A sending module, used for sending the data acquisition request to the server;

The second receiving module is used to receive the page data of the response page sent by the server.

Optionally, the device further comprises:

an acquisition module, configured to acquire a response token corresponding to the login operation if it is determined that a login operation is required before responding to the original request before the sending module sends the data acquisition request to the server;

The generating module is used for:

A data acquisition request is generated according to the target structure and the response token.

Optionally, the acquisition module is used to:

If there is a login script corresponding to the original request, the login script is replayed to obtain a response token after the login information used by the login script is authenticated, wherein the login script is pre-recorded and generated based on the login plug-in.

Optionally, during the playback of the login script, if the current operation is a verification code input operation, the verification code image in the current page is obtained, and the verification code image is sent to a verification code recognition module, so that the verification code recognition module recognizes the verification code image and obtains verification code information; the verification code input operation is performed based on the verification code information obtained by the verification code recognition module.

Optionally, the acquisition module includes:

an output submodule, configured to output prompt information to prompt the user to perform a login operation when a login operation is required before responding to the original request and there is no login script corresponding to the original request;

The detection submodule is used to detect the login status through the login plug-in;

The receiving submodule is used to receive the response token sent by the login plug-in when detecting that the login is successful.

Optionally, the device further comprises:

A first determining module is used to determine a text similarity parameter corresponding to first page text information of the response page and second page text information of the response sample corresponding to the original request after the second receiving module receives the page data of the response page sent by the server;

A second determination module, configured to determine whether the response page matches the response sample according to the text similarity parameter;

The second storage module is used to store the page data of the response page when it is determined that the response page matches the response sample.

Optionally, the second storage module is used for:

The page data of the response page is stored in a structured file, so as to be stored in a database based on the structured file.

According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any method described in the first aspect are implemented.

According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

a memory having a computer program stored thereon;

A processor is used to execute the computer program in the memory to implement the steps of any method described in the first aspect above.

In the above technical solution, the original request for receiving the page operation is determined; the characteristic parameters of the original request are obtained, and the characteristic parameters are stored in the target structure; a data acquisition request is generated according to the target structure; the data acquisition request is sent to the server, and the page data of the response page sent by the server is received. Through the above technical solution, when obtaining the page data of the response page of the original request, a data acquisition request is generated according to the characteristic parameters of the original request, and then data is requested from the server based on the data acquisition request, so that the page data can be obtained without simulating the request response operation of the browser. On the one hand, the complexity of page data acquisition can be effectively reduced, the method of page data acquisition can be simplified, and the reliability and accuracy of page data acquisition can be improved. On the other hand, the scope of use of the method for obtaining page data can be effectively broadened.

Other features and advantages of the present disclosure will be described in detail in the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

FIG1 is a flow chart of a method for acquiring page data according to an embodiment of the present disclosure;

FIG2 is a block diagram of an apparatus for acquiring page data according to an embodiment of the present disclosure;

FIG3 is a block diagram of an electronic device according to an exemplary embodiment;

Fig. 4 is a block diagram of an electronic device according to an exemplary embodiment.

Detailed ways

The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

FIG1 is a flowchart of a method for obtaining page data according to an embodiment of the present disclosure. As shown in FIG1 , the method includes:

In S11, an original request for a page operation is received, wherein the page operation may be a page query operation, a view operation, etc., and the original request is a request initiated to perform the page operation.

In S12, characteristic parameters of the original request are obtained and stored in the target structure.

Optionally, the characteristic parameter may include a path parameter corresponding to the original request, which can be obtained from the URL (Uniform Resource Locator) of the original request. The original request may be used to request the entire page, and the path parameter is the URL of the original request; it may also be used to request a portion of data in the page, such as the original request is used to request the body of the page, in which case the path parameter corresponding to the original request is the URL corresponding to the body.

Optionally, the characteristic parameters may also include query parameters, which are parameters used to constrain the displayed data in the page. For example, if the system is queried for a list of students with scores higher than 80, the query parameter is scores higher than 80. Since the original request corresponds to different systems, in one embodiment, the query parameter may be obtained from the URL of the original request; in another embodiment, the query parameter may be obtained from the request body of the original request, which is not limited in the present disclosure.

After obtaining the above characteristic parameters of the original request, the characteristic parameters can be stored in the target structure. For example, the target structure can be a JSON file with the following format:

URL: http://localhost:8080/api/{user-defined path A}/{user-defined path B}

Path parameters: ? {user-defined parameter a} = parameter value & {user-defined parameter b} = parameter value Query parameters:

Therefore, after extracting the characteristic parameters, the characteristic parameters can be stored as parameter values in the target structure, so that a standardized representation method can be used for original requests in different forms, thereby uniformly representing and managing original requests in different representation methods, improving the compatibility of the method, and broadening the application scenarios and scope of the method.

In S13, a data acquisition request is generated according to the target structure, wherein the data acquisition request is used to request to acquire page data of a response page of the original request.

In S14, a data acquisition request is sent to the server.

That is to say, in the present application, when obtaining the page data of the response page according to the original request, a data acquisition request is directly generated according to the characteristic parameters of the target structure, rather than simulating the browser operation to obtain the page data according to the original request as in the prior art. Therefore, it can be applied to systems with B/S (Browser/Server) architecture, as well as C/S (Client/Server) architecture and APP.

In S15, the page data of the response page sent by the server is received.

In the above technical solution, the original request for receiving the page operation is determined; the characteristic parameters of the original request are obtained, and the characteristic parameters are stored in the target structure; a data acquisition request is generated according to the target structure; the data acquisition request is sent to the server, and the page data of the response page sent by the server is received. Through the above technical solution, when obtaining the page data of the response page of the original request, a data acquisition request is generated according to the characteristic parameters of the original request, and then data is requested from the server based on the data acquisition request, so that the page data can be obtained without simulating the request response operation of the browser. On the one hand, the complexity of page data acquisition can be effectively reduced, the method of page data acquisition can be simplified, and the reliability and accuracy of page data acquisition can be improved. On the other hand, the scope of use of the method for obtaining page data can be effectively broadened.

The following describes in detail a method for determining a data acquisition request corresponding to an original request.

For example, a recording tool may be used to record in advance the process from a user initiating a certain operation request to obtaining a page corresponding to the operation request, thereby obtaining a description file of the recording result.

Afterwards, the description file of the recording result can be parsed, and the parsed result can be converted into a predefined standard format. Among them, since the formats of the description files of the recording results obtained by different recording tools may be different, therefore, by converting the parsed results into a standard format, adaptation to a variety of recording tools can be achieved. Among them, it should be noted that in the prior art, in the process of initiating an operation request to obtain the data of the response page corresponding to the operation request, multiple requests will be interacted with the background server, for example, a request for obtaining the style file of the response page, a request for obtaining the JS file of the response page, a request for obtaining the page data of the response page, etc. In the above-mentioned operation request, the request for obtaining the page data of the response page is the request actually used to obtain data in the process of responding to the operation request, that is, the data acquisition request described in the present disclosure.

Among them, the description file in the standard format (for the convenience of the following description, abbreviated as the formatted description file) contains all the requests in the process of responding to the operation request. For example, the formatted description file includes a total of 7 requests from request a to g. When determining the data acquisition request according to the formatted description file, the page data displayed in the response page can be extracted and analyzed to obtain keyword information. Based on the keyword information, the request containing the keyword information is queried from the formatted description file, and request g is queried for example, and request g is used as the query request. After that, the parameters carried in request g can be used as new keyword information to query the request containing the new keyword information from the formatted description file. For example, the parameters carried in request g are cookieA and ciphertext B, then the request corresponding to cookieA can be queried from the formatted description file based on cookieA, and the request corresponding to ciphertext B can be queried from the formatted description file based on ciphertext B. Among them, it should be noted that if multiple requests are queried, the request with the earliest request time among the multiple requests is used as a new query request. If no new request is found, or the found request is a login request, the analysis process is terminated, and the determined query requests are sorted in order from early to late according to the request time, and the sorting result is used as the data acquisition request, that is, the data acquisition request can contain multiple sub-requests with order correlation.

The recording tool may be any existing recording tool, and the analysis of the description file is a prior art, which is not limited in the present disclosure.

Therefore, through the above technical solution, the data acquisition request actually used to acquire page data in an operation request can be determined. Therefore, when acquiring page data, a corresponding data acquisition request can be generated according to the original request, thereby effectively avoiding the acquisition of unnecessary style files or JS files and improving the efficiency of page data acquisition.

Optionally, before the step of sending the data acquisition request to the server, the method further includes:

If it is determined that a login operation is required before responding to the original request, a response token corresponding to the login operation is obtained. The response token is returned by the server after the user's login information is authenticated, and is used to indicate the user's legal login status.

Wherein, in one embodiment, determining whether a login operation is required before responding to the original request can be determined by an identifier associated with the original request. For example, if a login operation is required before responding to the original request, the identifier associated with the original request (such as identifier 1) is used to indicate that a login operation is required, and if a login operation is not required before responding to the original request, the identifier associated with the original request (such as identifier 2) is used to indicate that a login operation is not required. Wherein, the associated identifier can be associated with the corresponding original request when determining the data acquisition request. Therefore, when the original request is obtained, it can be directly determined whether a login operation is required based on its associated identifier.

The step of generating a data acquisition request according to the target structure includes:

A data acquisition request is generated according to the target structure and the response token.

In the above technical solution, before sending a data acquisition request to the server, it is possible to determine whether the current user is in a legal state by determining that a login operation is required before responding to the original request. If it is determined that a login operation is required before responding to the original request, it is necessary to obtain a response token corresponding to the login operation, thereby ensuring the legal login state of the user. In addition, when generating a data acquisition request, the response token can be carried in the data acquisition request, thereby ensuring the accuracy of the response page, thereby improving the efficiency and accuracy of page data acquisition, and avoiding the influence of the user's login state on page data acquisition.

Optionally, an exemplary implementation of obtaining a response token corresponding to the login operation is as follows, including:

If there is a login script corresponding to the original request, the login script is replayed to obtain a response token after the login information used by the login script is authenticated, wherein the login script is pre-recorded and generated based on the login plug-in.

For example, for a relatively simple login process, such as a login process that does not require a verification code or requires an image verification code, the user can record a login script through a login plug-in, and the login script records the login information entered by the user when logging in, such as account number, password, etc., and associates the login script with the original request. Therefore, when it is determined that a login operation is required before responding to the original request, the login script corresponding to the original request can be obtained, and then the login can be performed by replaying the login script, so as to obtain a response token.

Optionally, for a login scenario that requires an image verification code, during the playback of the login script, if the current operation is a verification code input operation, the verification code image in the current page is obtained, and the verification code image is sent to a verification code recognition module, so that the verification code recognition module recognizes the verification code image and obtains verification code information; the verification code input operation is performed based on the verification code information obtained by the verification code recognition module.

In this embodiment, for the login scenario that requires an image verification code, the image verification code when the user logs in is dynamically changed. Therefore, during the playback of the login script, if the current operation is a verification code input operation, the verification code information in the image verification code displayed in real time on the input page needs to be input. Therefore, the verification code image on the current page can be obtained to be recognized by the verification code recognition module, so that the recognized verification code information is input to complete the login operation.

Through the above technical solution, the automated login operation can be realized through the pre-recorded login script, which can effectively reduce the user's participation and reduce the user's workload. At the same time, by obtaining the verification code image and identifying the verification code information, the login operation can be completed according to the login script and the dynamically changing image verification code in the page, improving the accuracy and convenience of the login operation, further reducing user operations and improving the user experience.

Optionally, another implementation of obtaining a response token corresponding to the login operation is as follows, including:

If a login operation is required before responding to the original request, and there is no login script corresponding to the original request, output a prompt message to prompt the user to perform the login operation;

The login status is detected by the login plug-in, wherein the login status may include successful login and failed login. When the user successfully logs in, the server returns a response token corresponding to the user's login information, at which time the login plug-in can detect the successful login and obtain the response token.

A response token is received from the login plug-in when a successful login is detected.

In this embodiment, in the absence of a login script corresponding to the original request, a response token can be obtained by prompting the user to perform a login operation. Specifically, the user can log in in a browser or client, and the user's login operation can be monitored by a login plug-in, wherein the login plug-in is executed in a hidden manner and the operation is transparent to the user. Therefore, after the user successfully logs in, the response token when the login is successful can be obtained through the login plug-in. For different systems, the storage location of the response token can be different, for example, the response token can be stored in a cookie, local storage or session storage. Therefore, the data in the cookie, local storage and session storage can be obtained through the login plug-in to obtain the response token.

In the above technical solution, if a login operation is required before responding to the original request and there is no login script corresponding to the original request, the user can be prompted to log in. And the user only needs to log in through his browser or client without operating the login plug-in to obtain the response token, avoiding the user's repeated login operation, thereby ensuring the accuracy of the subsequent data acquisition request, and further improving the accuracy of the acquired page data.

Among them, in cases where user login is illegal, network connection times out, etc., the page returned by the server is usually different from the response page corresponding to the original request. In order to ensure the accuracy of the page data obtained in this case, the present disclosure also provides the following embodiments.

Optionally, after the step of receiving the page data of the response page sent by the server, the method further includes:

Determine the text similarity parameter corresponding to the first page text information of the response page and the second page text information of the response sample corresponding to the original request. The response sample corresponding to the original request is a page preset in response to the original request, wherein the page can be a response page used when analyzing and determining the data acquisition request corresponding to the original request, or a page set by the developer based on the response page, which is not limited by the present disclosure.

By way of example, the first page text information of the response page may be a text vector corresponding to the response page, and the second page text information of the response sample corresponding to the original request may be a text vector corresponding to the response sample. Then, the text similarity parameter may be determined by the text vector corresponding to the response page and the text vector corresponding to the response sample. By way of example, the text similarity parameter may be the distance or cosine similarity between the text vector corresponding to the response page and the text vector corresponding to the response sample. The method of calculating the distance or cosine similarity between vectors is a prior art and will not be described in detail herein.

Determine whether the response page matches the response sample according to the text similarity parameter.

As an example, if the text similarity parameter is the distance between the text vector corresponding to the response page and the text vector corresponding to the response sample, the response page is determined to match the response sample when the distance is less than a preset distance threshold.

As an example, if the text similarity parameter is the cosine similarity between the text vector corresponding to the response page and the text vector corresponding to the response sample, the response page is determined to match the response sample when the cosine value is greater than a preset cosine similarity threshold.

Among them, the distance threshold and the cosine similarity threshold can be set according to the actual usage scenario, and the present disclosure does not limit this.

If it is determined that the response page matches the response sample, the page data of the response page is stored. In other words, when it is determined that the response page is an accurate page, its page data is stored.

In the above technical solution, by determining whether the response page and the response sample match, it is determined whether the page data of the response page is the data obtained by the original request; when it is determined that the response page and the response sample match, the page data of the response page is stored, so that the problem of inconsistency between the obtained page data and the data obtained by the original request can be avoided by verifying the response page, thereby improving the accuracy and effectiveness of page data acquisition. At the same time, it can also effectively avoid storing wrong page data and avoid wasting storage resources.

Optionally, storing the page data of the response page includes:

The page data of the response page is stored in a structured file, so as to be stored in a database based on the structured file.

For example, the structured file can be an HTML, JSON, or XML file, and the page data of the response page is stored in the structured file, so that the page data can be stored in the database through ETL technology based on the structured file. ETL (Extract-Transform-Load) is used to extract, transform, and load data from the source end to the destination end.

In the above technical solution, by storing page data in a structured file, the data in the structured file can be conveniently stored in a database, thereby achieving secure and persistent storage of data, facilitating user viewing and improving user experience.

The present disclosure also provides a device for acquiring page data, as shown in FIG2 , the device 10 includes:

The first receiving module 100 is used to receive an original request for a page operation;

A first storage module 200 is used to obtain characteristic parameters of the original request and store the characteristic parameters in a target structure;

A generating module 300, configured to generate a data acquisition request according to the target structure, wherein the data acquisition request is used to request to acquire page data of a response page of the original request;

A sending module 400 is used to send the data acquisition request to the server;

The second receiving module 500 is used to receive the page data of the response page sent by the server.

Optionally, the device further comprises:

an acquisition module, configured to acquire a response token corresponding to the login operation if it is determined that a login operation is required before responding to the original request before the sending module sends the data acquisition request to the server;

The generating module is used for:

A data acquisition request is generated according to the target structure and the response token.

Optionally, the acquisition module is used to:

If there is a login script corresponding to the original request, the login script is replayed to obtain a response token after the login information used by the login script is authenticated, wherein the login script is pre-recorded and generated based on the login plug-in.

Optionally, during the playback of the login script, if the current operation is a verification code input operation, the verification code image in the current page is obtained, and the verification code image is sent to a verification code recognition module, so that the verification code recognition module recognizes the verification code image and obtains verification code information; the verification code input operation is performed based on the verification code information obtained by the verification code recognition module.

Optionally, the acquisition module includes:

an output submodule, configured to output prompt information to prompt the user to perform a login operation when a login operation is required before responding to the original request and there is no login script corresponding to the original request;

The detection submodule is used to detect the login status through the login plug-in;

The receiving submodule is used to receive the response token sent by the login plug-in when detecting that the login is successful.

Optionally, the device further comprises:

A first determining module is used to determine a text similarity parameter corresponding to first page text information of the response page and second page text information of the response sample corresponding to the original request after the second receiving module receives the page data of the response page sent by the server;

A second determination module, configured to determine whether the response page matches the response sample according to the text similarity parameter;

The second storage module is used to store the page data of the response page when it is determined that the response page matches the response sample.

Optionally, the second storage module is used for:

The page data of the response page is stored in a structured file, so as to be stored in a database based on the structured file.

Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

The present disclosure also provides a computer interface, which is used to implement the steps of the method for obtaining page data. By encapsulating the method for obtaining page data into a computer interface, it can be convenient for a third party to call, thereby improving the convenience and compatibility of obtaining page data.

Fig. 3 is a block diagram of an electronic device 700 according to an exemplary embodiment. As shown in Fig. 3, the electronic device 700 may include: a processor 701, a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input/output (I/O) interface 704, and a communication component 705.

The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned method for obtaining page data. The memory 702 is used to store various types of data to support the operation of the electronic device 700. For example, these data may include instructions for any application or method used to operate on the electronic device 700, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and/or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I/O interface 704 provides an interface between the processor 701 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 705 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned method for obtaining page data.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned method for obtaining page data are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 702 including program instructions, and the above-mentioned program instructions can be executed by the processor 701 of the electronic device 700 to complete the above-mentioned method for obtaining page data.

FIG4 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Referring to FIG4 , the electronic device 1900 includes a processor 1922, which may be one or more, and a memory 1932 for storing a computer program executable by the processor 1922. The computer program stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processor 1922 may be configured to execute the computer program to execute the above-mentioned method for obtaining page data.

In addition, the electronic device 1900 may further include a power supply component 1926 and a communication component 1950, wherein the power supply component 1926 may be configured to perform power management of the electronic device 1900, and the communication component 1950 may be configured to implement communication of the electronic device 1900, for example, wired or wireless communication. In addition, the electronic device 1900 may further include an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, etc.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned method for obtaining page data are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 1932 including program instructions, and the above-mentioned program instructions can be executed by the processor 1922 of the electronic device 1900 to complete the above-mentioned method for obtaining page data.

In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above method for acquiring page data when executed by the programmable device.

The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims (9)

1. A method for obtaining page data, characterized in that the method comprises: Receive the original request for page operation; Obtaining characteristic parameters of the original request, and storing the characteristic parameters in a target structure; Generate a data acquisition request according to the target structure, wherein the data acquisition request is used to request to acquire page data of a response page of the original request; Sending the data acquisition request to the server; Receiving page data of the response page sent by the server; Determine a text similarity parameter corresponding to the first page text information of the response page and the second page text information of the response sample corresponding to the original request; Determining whether the response page matches the response sample according to the text similarity parameter; If it is determined that the response page matches the response sample, storing the page data of the response page; Among them, when determining the data acquisition request according to the formatting description file, the page data displayed in the response page is extracted and analyzed to obtain keyword information; based on the keyword information, the request containing the keyword information is queried from the formatting description file; the parameters carried in the queried request are used as new keyword information, and the step of re-executing the step of querying the request containing the keyword information from the formatting description file based on the keyword information is performed; if multiple requests are queried, the request with the earliest request time among the multiple requests is used as the new queried request; if no request is found, or the queried request is a login request, the query process is terminated, and the queried requests are sorted in order from early to late according to the request time, and the sorting result is used as the data acquisition request, and the data acquisition request contains multiple sub-requests with sequence correlation; Among them, a recording tool is used to record in advance the process from the user initiating the original request to obtaining the page corresponding to the original request to obtain a description file of the recording result, and the description file of the recording result is parsed, and the parsed result is converted into a predefined standard format to obtain the formatted description file. 2. The method according to claim 1, characterized in that before the step of requesting the server for data acquisition, the method further comprises: If it is determined that a login operation is required before responding to the original request, obtaining a response token corresponding to the login operation; The step of generating a data acquisition request according to the target structure includes: A data acquisition request is generated according to the target structure and the response token. 3. The method according to claim 2, wherein obtaining a response token corresponding to the login operation comprises: If there is a login script corresponding to the original request, the login script is replayed to obtain a response token after the login information used by the login script is authenticated, wherein the login script is pre-recorded and generated based on the login plug-in. 4. The method according to claim 3 is characterized in that, during the playback of the login script, if the current operation is a verification code input operation, the verification code image in the current page is obtained, and the verification code image is sent to a verification code recognition module, so that the verification code recognition module recognizes the verification code image and obtains verification code information; the verification code input operation is performed according to the verification code information obtained by the verification code recognition module. 5. The method according to claim 2, wherein obtaining a response token corresponding to the login operation comprises: If a login operation is required before responding to the original request, and there is no login script corresponding to the original request, output a prompt message to prompt the user to perform the login operation; Detect login status through login plugin; A response token is received from the login plug-in when a successful login is detected. 6. The method according to claim 1, wherein storing the page data of the response page comprises: The page data of the response page is stored in a structured file, so as to be stored in a database based on the structured file. 7. A device for acquiring page data, characterized in that the device comprises: A first receiving module, used for receiving an original request for a page operation; A first storage module, used for obtaining characteristic parameters of the original request and storing the characteristic parameters in a target structure; A generating module, used for generating a data acquisition request according to the target structure, wherein the data acquisition request is used for requesting to acquire page data of a response page of the original request; A sending module, used for sending the data acquisition request to the server; A second receiving module, used for receiving the page data of the response page sent by the server; A first determining module is used to determine a text similarity parameter corresponding to first page text information of the response page and second page text information of the response sample corresponding to the original request after the second receiving module receives the page data of the response page sent by the server; A second determination module, configured to determine whether the response page matches the response sample according to the text similarity parameter; A second storage module is used to store page data of the response page when it is determined that the response page matches the response sample; Among them, when determining the data acquisition request according to the formatting description file, the page data displayed in the response page is extracted and analyzed to obtain keyword information; based on the keyword information, the request containing the keyword information is queried from the formatting description file; the parameters carried in the queried request are used as new keyword information, and the step of re-executing the step of querying the request containing the keyword information from the formatting description file based on the keyword information is performed; if multiple requests are queried, the request with the earliest request time among the multiple requests is used as the new queried request; if no request is found, or the queried request is a login request, the query process is terminated, and the queried requests are sorted in order from early to late according to the request time, and the sorting result is used as the data acquisition request, and the data acquisition request contains multiple sub-requests with sequence correlation; Among them, a recording tool is used to record in advance the process from the user initiating the original request to obtaining the page corresponding to the original request to obtain a description file of the recording result, and the description file of the recording result is parsed, and the parsed result is converted into a predefined standard format to obtain the formatted description file. 8. A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented. 9. An electronic device, comprising: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 6.