CN107689951A - Web data crawling method, device, user terminal and readable storage medium storing program for executing - Google Patents

Abstract

The present invention relates to a kind of web data crawling method, device, user terminal and readable storage medium storing program for executing, this method includes the website log interface to be crawled being embedded in by client, the account corresponding with website to be crawled and password of input are received, passes through account corresponding with website to be crawled and password login website to be crawled；Detect whether Successful login website to be crawled；When Successful login is when website is crawled, then judge whether the account of client matches with the account of website to be crawled；Upon a match, then the data to be crawled in website to be crawled are crawled；The data to be crawled crawled are sent to server.Above-mentioned web data crawling method, device, user terminal and readable storage medium storing program for executing, the data to be crawled in website to be crawled are crawled by client, the data-triggered air control mechanism to be crawled in server end crawls website to be crawled can be avoided, the generation for situations such as causing user account to be locked.

Description

Web page data crawling method, device, user terminal and readable storage medium

technical field

The present invention relates to the field of computer technology, in particular to a web page data crawling method, device, user terminal and readable storage medium.

Background technique

At present, a large amount of valuable information on the Internet needs to be crawled to the server for analysis, so as to analyze user behavior. Crawl the data stored in the website to be crawled, but because the current security mechanism of each website is very high, crawling too many account information from the same IP address will trigger the website's risk control mechanism, resulting in the user's account being blocked, thus The user cannot use the account.

Contents of the invention

Based on this, it is necessary to provide a webpage data crawling method, device, user terminal and readable storage medium for the problem of account blocking caused by the server crawling the data in the website to be crawled.

A web page data crawling method, the method comprising:

Receive the account and password corresponding to the website to be crawled through the login interface of the website to be crawled embedded in the client, and log in to the website to be crawled through the account and password corresponding to the website to be crawled;

Detecting whether the website to be crawled is successfully logged in;

When successfully logging in to the website to be crawled, it is judged whether the account of the client matches the account of the website to be crawled;

When the account of the client matches the account of the website to be crawled, then crawl the data to be crawled in the website to be crawled;

Send the crawled data to be crawled to the server.

In one of the embodiments, the step of crawling the data to be crawled in the website to be crawled includes:

Send a crawling script acquisition request to the server;

Receiving the crawling script corresponding to the crawling script acquisition request returned by the server;

The data to be crawled in the website to be crawled is crawled through the crawling script.

In one of the embodiments, before the step of sending the crawling script acquisition request to the server, it also includes:

Obtain the time when the crawling script was returned by the receiving server last time;

When the difference between the time when the crawling script was returned by the receiving server last time and the current time is within a preset range, the data to be crawled in the website to be crawled is crawled through the crawling script returned by the server received last time ; When the difference between the last time the server received the crawling script and the current time is not within the preset range, execute the step of sending the crawling script acquisition request to the server.

In one of the embodiments, the step of detecting whether to successfully log in to the website to be crawled includes:

Detect whether the URL address of the current page displayed by the client changes;

When the URL address of the current page displayed by the client changes, the website to be crawled is successfully logged in;

When the URL address of the current page displayed by the client does not change, the login to the website to be crawled is unsuccessful.

In one of the embodiments, the website to be crawled is a mailbox website;

The step of crawling the data to be crawled in the website to be crawled includes:

Selecting emails with titles corresponding to the data to be crawled from the mailbox website;

Crawl the data of the preset fields from the selected emails as the crawled data to be crawled.

In one of the embodiments, the step of sending the crawled data to be crawled to the server includes:

Encrypt the crawled data to be crawled;

Pack the encrypted data to be crawled;

Send the packaged data to be crawled to the server.

A web page data crawling device, said device comprising:

The login module is used to receive the account and password corresponding to the website to be crawled through the login interface of the website to be crawled embedded in the client, and log in to the website to be crawled through the account and password corresponding to the website to be crawled. crawl the website;

A detection module, configured to detect whether the website to be crawled is successfully logged in;

A verification module, configured to determine whether the account of the client matches the account of the website to be crawled when successfully logging in to the website to be crawled;

A crawling module, configured to crawl the data to be crawled in the website to be crawled when the account of the client matches the account of the website to be crawled;

The sending module is used to send the crawled data to be crawled to the server.

In one of the embodiments, the sending module is also used to send a crawling script acquisition request to the server;

The crawling module includes:

A receiving unit, configured to receive the crawling script returned by the server and corresponding to the crawling script acquisition request;

A crawling unit, configured to crawl the data to be crawled in the website to be crawled through the crawling script.

A user terminal includes a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor implements the steps in the above method when executing the computer program.

A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method are realized.

The above web page data crawling method, device, user terminal and readable storage medium use the client to crawl the data to be crawled in the website to be crawled, and log in to the website to be crawled through the login interface of the website to be crawled embedded in the client. After the website, verify whether the account of the website to be crawled corresponds to the account of the client to ensure that the data to be crawled is the data of the client user, and send the data to be crawled to the server for the server The processing and analysis can avoid the risk control mechanism triggered by crawling the data to be crawled in the website to be crawled on the server side, resulting in the lockout of the user account.

Description of drawings

Fig. 1 is the application environment diagram of web page data crawling method in an embodiment;

Fig. 2 is the flowchart of the web page data crawling method in an embodiment;

Fig. 3 is the flowchart of step S208 of the embodiment shown in Fig. 2;

Fig. 4 is the interface figure of qq mailbox login interface in an embodiment;

Fig. 5 is an interface diagram of the bill data crawling process interface in an embodiment;

Fig. 6 is an interface diagram of successful crawling of bill data in an embodiment;

Fig. 7 is another flowchart of step S208 in the embodiment shown in Fig. 2;

FIG. 8 is a flowchart of step S210 in the embodiment shown in FIG. 2;

Fig. 9 is a schematic structural diagram of a webpage data crawling device in an embodiment;

Fig. 10 is a schematic structural diagram of a user terminal in an embodiment.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Before describing the embodiments according to the present invention in detail, it should be noted that the described embodiments mainly lie in the combination of steps and system components related to the web page data crawling method, device, user terminal and readable storage medium. Accordingly, pertinent system components and method steps have been indicated at appropriate places in the drawings by conventional symbols, and only details relevant to the understanding of the embodiments of the present invention have been shown so as not to cause confusion to those who would benefit from the present invention. Details that would be apparent to one of ordinary skill in the art obscure the present disclosure.

In this text, relational terms such as left and right, up and down, front and back, first and second are used merely to distinguish one entity or action from another without necessarily requiring or implying such Any actual such relationship or sequence between entities or actions. The terms "comprising", "comprising" or any other variant are intended to cover a non-exclusive inclusion whereby a process, method, article or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed elements, or elements inherent in such a process, method, article, or apparatus.

Referring to Fig. 1, Fig. 1 is an application environment diagram of a webpage data crawling method in an embodiment, in this embodiment, including a server and several user terminals, the server can communicate with several user terminals respectively, wherein the user terminal The client APP is installed, and the website to be crawled is embedded in the client APP. The user terminal can be a terminal such as a mobile phone, a tablet, or a computer, and the client APP installed in the user terminal can be an APP of any APP provider, which embeds a website to be crawled. For example, a client APP such as WeChat can embed an email login interface, etc.

Please refer to FIG. 2 . In one embodiment, a webpage data crawling method is provided. This embodiment is illustrated by applying the method to the server in FIG. 1 above. A webpage data crawling program runs on the server, and the webpage data crawling method is implemented through the webpage data crawling program. The method specifically includes the following steps:

S202: Receive an input account and password corresponding to the website to be crawled through the login interface of the website to be crawled embedded in the client, and log in to the website to be crawled through the account and password corresponding to the website to be crawled.

Specifically, the client refers to an application such as an APP installed on a user terminal, in which a login interface of a website to be crawled is embedded. The website interface to be crawled may be an email login interface or an e-commerce login interface, such as a qq mailbox login interface, 126 mailbox login interface, 163 mailbox login interface, Taobao login interface, Alipay login interface, Jingdong login interface, Vipshop login interface, etc.

After the user logs in to the client through the account of the client, open the login interface of the website to be crawled, and enter the account and password of the website to be crawled, so that the website to be crawled can be logged in through the login interface of the website to be crawled embedded in the client . For example, there is a qq mailbox login interface embedded in the "Ping An One Account APP". Users can first log in to the "Ping An One Account APP" through the "Ping An One Account APP", and then open the On the qq mailbox login interface, log in to the qq mailbox by entering the qq mailbox account and password into the qq mailbox login interface.

S204: Detect whether the website to be crawled is successfully logged in.

Specifically, before crawling the data to be crawled in the website to be crawled, it is necessary to successfully log in to the website to be crawled, so before crawling the data to be crawled in the website to be crawled, it is necessary to detect whether the login is successful If you do not log in successfully, you cannot crawl the data to be crawled in the website to be crawled.

S206: When the website to be crawled is successfully logged in, it is determined whether the account of the client matches the account of the website to be crawled.

Specifically, in some cases, users may log in other users’ accounts of websites to be crawled through their own clients. If they also crawl accounts of other users’ websites to be crawled at this time, it will eventually lead to The data is not the user's own, resulting in data errors. Therefore, before crawling, in order to ensure that the account of the website to be crawled is the user's own, it is judged whether the account of the client matches the account of the website to be crawled. For example, the user's unique identifier, such as the user's ID number, can be set in the user account of "Ping An One Account", and the user's unique identifier, such as the ID number, can also be set in the account of qq mailbox. Only when the account unique identifier of "One Account Tong APP" matches the account unique identifier of qq mailbox, will the next step be performed to crawl the data to be crawled in the website to be crawled.

S208: When the account of the client matches the account of the website to be crawled, crawl the data to be crawled in the website to be crawled.

Specifically, when the account of the client matches the account of the website to be crawled, it is proved that the data to be crawled in the website to be crawled is the user's data, and the website to be crawled is directly crawled through the crawling program in the client The data to be crawled in is enough, so that each user's data to be crawled is crawled on each user terminal, but not all users' data to be crawled is crawled on the server, which can effectively avoid due to crawling The risk control mechanism of the crawling website locks the account of the user's website to be crawled.

S210: Send the crawled data to be crawled to the server.

Specifically, when the user terminal crawls the corresponding data to be crawled, the data to be crawled can be sent to the server, so that the server can provide the user with corresponding services according to the data. For example, when the user terminal crawls the credit card bill in the qq mailbox, the server can remind the user when to repay according to the bill data, or provide the user with a repayment red envelope, for example, when the user needs to repay 1,000 yuan, then Provide users with services such as 5 yuan deduction of red envelopes.

The above web page data crawling method uses the client to crawl the data to be crawled in the website to be crawled, and after logging in to the website to be crawled through the login interface of the website to be crawled embedded in the client, verify the account of the website to be crawled Whether it corresponds to the account of the client to ensure that the crawled data to be crawled is the data of the client user, and the crawled data to be crawled is sent to the server for processing and analysis by the server, which can avoid crawling on the server side Getting the data to be crawled from the website to be crawled triggers the risk control mechanism, resulting in the lockout of the user account.

In one of the embodiments, referring to FIG. 3, FIG. 3 is a flow chart of step S208 of the embodiment shown in FIG. 2. The step S208, that is, the step of crawling the data to be crawled in the website to be crawled may include:

S302: Send a crawling script acquisition request to the server.

Specifically, the crawling script refers to a script that can be used in a user terminal to crawl data to be crawled in a website to be crawled. The crawling script is stored on the server, so that the crawling script can only be modified on the server side, and before the crawling data in the website to be crawled is crawled next time, a new crawling script is directly downloaded from the server. Yes, since the crawling script is in the form of a script, it occupies a small space and has a fast transmission speed. When the account on the client side matches the account of the website to be crawled, the user terminal sends a crawling script acquisition request to the server, and the server queries the crawling script after receiving the crawling script acquisition request, and then crawls the Take the script and send it to the corresponding client after packaging, so as to reduce the amount of data transmission.

S304: Receive the crawling script corresponding to the crawling script acquisition request returned by the server.

Specifically, when the server inquires about the crawling script corresponding to the crawling script acquisition instruction, it sends the crawling script to the user terminal, and the user terminal can use the crawling script to crawl the pages to be crawled in the website to be crawled. fetch data.

S306: Crawl the data to be crawled in the website to be crawled by using the crawling script.

Specifically, the user terminal crawls the corresponding data to be crawled through the crawling script downloaded from the server. Please refer to FIGS. 4 to 6. FIG. 4 is an interface diagram of the qq mailbox login interface in an embodiment, and FIG. The interface diagram of the billing data crawling process interface in the example, as shown in 6, is the interface diagram of the successful crawling of the billing data in an embodiment. Among them, the qq mailbox interface is embedded in the client terminal of the user terminal. The user logs in to the qq mailbox by entering the account and password in the qq mailbox interface, as shown in Figure 4. When the qq mailbox is successfully logged in, the user terminal detects the client account and the qq mailbox After the accounts are matched, the crawling script is downloaded from the server, and then the billing information in the qq mailbox is crawled through the crawling script, as shown in Figure 5, which can display the progress of the user terminal crawling the data to be crawled. Figure 5 shows qq The email verification was successful, and the corresponding bill was also searched, and the bill has been crawled by 64%. When the user terminal crawls the data to be crawled, that is, the bill, the user can be prompted to complete the crawling, as shown in Figure 6.

In the above embodiment, when the account of the client matches the account of the website to be crawled, the information of obtaining the script is sent to the server. After receiving the information, the server packages the latest script and transmits it to the user terminal. In this way, first, the script is stored on the server, and the crawling script can only be modified on the server, but if the crawling script is delivered together with the client installation package, when the crawling script is modified, a new one needs to be issued The installation package will increase the frequency of client updates. Secondly, when sending the crawling script, it will be packaged before sending, which can reduce the amount of data transmission.

In one of the embodiments, before the step of sending the crawling script acquisition request to the server, it may also include: obtaining the time when the last receiving server returned the crawling script; the difference between the time when the last receiving server returned the crawling script and the current time When the value is within the preset range, the data to be crawled in the website to be crawled is crawled through the crawling script received last time and returned by the server. When the difference between the last time the server received the crawling script and the current time is not within the preset range, the step of sending a crawling script acquisition request to the server is performed, that is, step S302.

Specifically, in order to prevent the user terminal from crawling the crawling script multiple times from the server in a short period of time, a preset range is set, as long as the difference between the time when the user obtained the crawling script from the server last time and the current time is within the preset range , the user terminal does not need to download the crawling script from the server again. The preset range may be 1 hour, 30 minutes, 2 hours, 1 day, 1 week, etc., which is not limited here. For example, when crawling last time, the crawling script was downloaded from the server at 9:30 am, and the default range is 2 hours, then the time of crawling again is 10:30 am, because it is different from 9:30 am The difference is 1 hour, which is less than the preset range of 2 hours. Therefore, when crawling at 10:30, the crawling script downloaded from the server last time can be used, and there is no need to re-download the crawling script, but if you crawl again The time taken is 2:30 pm, the difference between 9:30 am and 9:30 am is 5 hours, which is 2 hours longer than the preset range, so when crawling at 2:30 pm, you need to download the crawling script from the server again.

In the above embodiment, after the client account matches the account of the website to be crawled, the time when the last crawled script was acquired can be obtained first, if the difference between the last crawled script and the current time is within the preset range, then Directly call the crawling script stored in the user terminal instead of downloading it from the server, which can avoid, for example, the user frequently logs in to the qq mailbox to synchronize bills within a day, which leads to downloading the script every time, resulting in waste of data traffic, etc.

In one of the embodiments, the step of detecting whether the website to be crawled is successfully logged in, that is, step S204 in the embodiment shown in FIG. 2 may include: detecting whether the URL address of the current page displayed by the client has changed; If the URL address of the displayed current page changes, the website to be crawled is successfully logged in; when the URL address of the current page displayed by the client does not change, the website to be crawled is unsuccessfully logged in.

Specifically, since the URL (Uniform Resource Locator, Uniform Resource Locator) addresses of different webpages are different, it can be determined whether the website to be crawled is successfully logged in by detecting whether the URL address of the webpage changes. For example, the URL address of the login interface of the website to be crawled may be A, and the URL address may become B after successful login. If the login fails, it will still stay at the current login interface of the website to be crawled, that is, its URL address has not yet been A, so that by judging whether the URL address has changed, you can judge whether you have successfully logged in to the website to be crawled, and the operation is simple.

In the above embodiment, detecting whether the website to be crawled is successfully logged in can be performed by detecting whether the URL address of the current interface of the client changes. Only when the login is successful, the URL address of the current interface of the client will change. When the login fails, the URL address of the client's current interface remains unchanged, and a corresponding prompt message of login failure will be provided.

In one of the embodiments, the website to be crawled is a mailbox website; referring to FIG. 7, FIG. 7 is another flowchart of step S208 in the embodiment shown in FIG. The steps to crawl data can include:

S702: Select emails with titles corresponding to the data to be crawled from the mailbox website.

Specifically, because there may be a large amount of data stored in the mailbox, and the server only cares about the emails corresponding to the data to be crawled, firstly, the emails corresponding to the data to be crawled can be selected from the mailbox according to the properties of the data to be crawled. For example, when billing data needs to be crawled, emails whose titles are related to bills are crawled first.

S704: Crawl the data of the preset field from the selected email as the crawled data to be crawled.

Specifically, because a large amount of billing information may be stored in the billing email, for example, some billing information may include name, date, consumption amount, payee and other information, but the server only needs to crawl the name and consumption amount information. Then the user terminal crawls the data in the fields of the name and consumption amount from the selected email as the crawled data, without the need to crawl other additional data.

In the above embodiment, firstly, the mail is locked according to the title of the mail, for example, the title of the mail in the inbox can be traversed, or the title of the mail in the inbox in a certain period of time can be traversed to determine the mail associated with the credit card bill . When the user is using the client APP for the first time, it is necessary to traverse the mails in the entire inbox, but if the user is not using the client APP for the first time, you can obtain the time when the server obtained the bill last time, and only need to traverse the emails after that time Emails in your inbox will do. After you have locked the emails whose titles are related to bills, you can get the contents of the preset fields, for example, you can only get the date, summary, income, expenditure and other information, that is, filter out useless information, or you can also get all the information, for example Balance, expenditure object information, etc.

In one of the embodiments, referring to FIG. 8, FIG. 8 is a flowchart of step S210 in the embodiment shown in FIG. 2. The step S210, the step of sending the crawled data to be crawled to the server may include:

S802: Encrypt the crawled data to be crawled.

Specifically, since the crawled data involves the user's private information, it needs to be encrypted during the transmission process, which may adopt a symmetric encryption method or an asymmetric encryption method, which is not limited here. When the user terminal crawls the data to be crawled, it encrypts the data to be crawled, and then sends it to the server. After receiving the data, the server performs corresponding decryption operations to obtain the crawled data to be crawled .

S804: Pack the encrypted data to be crawled.

Specifically, in order to reduce the amount of data transmission, the crawled data can be packaged, and the packaged data can be sent to the server, thereby reducing the usage of user traffic.

S806: Send the packaged data to be crawled to the server.

Specifically, when the data to be crawled is packaged, the packaged data is sent to the server. At this time, the user terminal can detect the current network environment. When the network is a wifi network, the packaged data to be crawled is Get the data and send it to the server. When the network is a mobile network, the packaged data to be crawled will not be sent temporarily until the network of the user terminal is programmed as a wifi network, then the packaged data to be crawled will be sent to the server. This reduces the usage of user traffic.

In the above embodiment, when sending the crawled data to be crawled, the data to be crawled is first encrypted, and then the encrypted data to be crawled is packaged. In this way, the transmission of the data to be crawled can be ensured. In-process security, there is the possibility to reduce the amount of data transfer.

Referring to FIG. 9, FIG. 9 is a schematic structural diagram of a webpage data crawling device in an embodiment, and the webpage data crawling device includes:

The login module 100 is used to receive the input account and password corresponding to the website to be crawled through the login interface of the website to be crawled embedded in the client, and log in to the website to be crawled through the account and password corresponding to the website to be crawled.

The detection module 200 is used to detect whether the website to be crawled is successfully logged in.

The verification module 300 is configured to determine whether the account of the client matches the account of the website to be crawled when successfully logging in to the website to be crawled.

The crawling module 400 is configured to crawl the data to be crawled in the website to be crawled when the account of the client matches the account of the website to be crawled.

The sending module 500 is configured to send the crawled data to be crawled to the server.

In one of the embodiments, the sending module may be further configured to send a crawling script acquisition request to the server.

Crawling modules can include:

The receiving unit is configured to receive the crawling script returned by the server and corresponding to the crawling script acquisition request.

The crawling unit is used to crawl the data to be crawled in the website to be crawled through a crawling script.

In one of the embodiments, the web page data crawling device may also include:

The time acquisition module is used to acquire the time when the crawling script was returned by the receiving server last time.

The comparison module is used to crawl the website to be crawled through the crawling script returned by the server received last time when the difference between the time when the last receiving server returned the crawling script and the current time is within a preset range. Crawling data; when the difference between the last time the server received the crawling script and the current time is not within the preset range, the step of sending the crawling script acquisition request to the server is executed.

In one of the embodiments, the detection module can also be used to detect whether the URL address of the current page displayed by the client changes; when the URL address of the current page displayed by the client changes, the website to be crawled is successfully logged in; when the client If the URL address of the current page displayed on the terminal has not changed, the website to be crawled has not been successfully logged in.

In one of the embodiments, the website to be crawled is a mailbox website. The crawling module can also be used to select emails whose titles correspond to the data to be crawled from the mailbox website; crawl data in preset fields from the selected emails as the crawled data to be crawled.

In one of the embodiments, the sending module may include:

An encryption unit, configured to encrypt the crawled data to be crawled.

The packaging unit is used to package the encrypted data to be crawled.

The sending unit is used to send the packaged data to be crawled to the server.

The modules and units involved in the webpage data crawling device can be program segments divided according to functions. In addition, the above-mentioned limitations on the webpage data crawling device can be added to the above-mentioned limitations on the webpage data crawling method, which will not be repeated here. repeat.

Please refer to FIG. 8. FIG. 8 is a schematic structural diagram of a user terminal in an embodiment. The user terminal may be a conventional server or any other computer equipment, including a memory, a processor, and a computer stored in the memory and operable on the processor. A computer program, wherein the memory may include a non-volatile storage medium and an internal memory, the computer program may be stored in the non-volatile storage medium, and the following steps are implemented when the processor executes the program: the crawler embedded in the client The website login interface receives the input account and password corresponding to the website to be crawled, and logs in to the website to be crawled through the account and password corresponding to the website to be crawled; detects whether the website to be crawled is successfully logged in; when successfully logged in to the website to be crawled website, it is determined whether the account of the client matches the account of the website to be crawled; when the account of the client matches the account of the website to be crawled, the data to be crawled in the website to be crawled is crawled; The data to be crawled is sent to the server.

In one of the embodiments, when the processor executes the program, the following steps can also be implemented: sending a crawling script acquisition request to the server; receiving the crawling script corresponding to the crawling script acquisition request returned by the server; crawling the waiting script through the crawling script Crawl the data to be crawled in the website.

In one of the embodiments, when the processor executes the program, the following steps can also be implemented: obtaining the time when the receiving server returned the crawling script last time; when the last receiving server returned the crawling script time and the current time When it is in the range, crawl the data to be crawled in the website to be crawled through the crawling script returned by the server received last time; when the difference between the time when the crawling script was returned by the receiving server last time and the current time is not within the preset range , execute the step of sending the crawling script acquisition request to the server.

In one of the embodiments, the following steps can also be implemented when the processor executes the program: the step of obtaining the input application information includes: detecting whether the URL address of the current page displayed by the client changes; If the URL address changes, the website to be crawled is successfully logged in; when the URL address of the current page displayed by the client has not changed, the website to be crawled is unsuccessfully logged in.

In one of the embodiments, the website to be crawled is a mailbox website; when the processor executes the program, the following steps can also be implemented: selecting the email whose title corresponds to the data to be crawled from the mailbox website; Set the data of the field as the crawled data to be crawled.

In one of the embodiments, when the processor executes the program, the following steps can also be implemented: encrypting the crawled data to be crawled; packaging the encrypted data to be crawled; packaging the packed data to be crawled sent to the server.

For the above limitations on crawler terminals, please refer to the specific limitations on web page data crawling methods above, which will not be repeated here.

Please continue to refer to FIG. 8 , there is also provided a computer-readable storage medium on which a computer program is stored, such as the non-volatile storage medium shown in FIG. 8 , wherein, when the program is executed by the processor, the following steps are implemented: Receive the input account and password corresponding to the website to be crawled through the login interface of the website to be crawled embedded in the client, and log in to the website to be crawled through the account and password corresponding to the website to be crawled; check whether the login to be crawled is successful website; when successfully logging in to the website to be crawled, it is judged whether the account of the client matches the account of the website to be crawled; when the account of the client matches the account of the website to be crawled, crawl the Crawling data; sending the crawled data to be crawled to the server.

In one of the embodiments, when the program is executed by the processor, the following steps can also be implemented: sending a crawling script acquisition request to the server; receiving the crawling script corresponding to the crawling script acquisition request returned by the server; Get the data to be crawled from the website to be crawled.

In one of the embodiments, when the program is executed by the processor, the following steps can also be implemented: obtaining the time when the receiving server returned the crawling script last time; when the time when the receiving server returned the crawling script last time and the current time When the scope is preset, crawl the data to be crawled in the website to be crawled through the crawling script returned by the server received last time; If it is within the set range, the step of sending the crawling script acquisition request to the server is executed.

In one of the embodiments, when the program is executed by the processor, the following steps can also be implemented: the step of obtaining the input application information includes: detecting whether the URL address of the current page displayed by the client changes; If the URL address of the page changes, the website to be crawled is successfully logged in; when the URL address of the current page displayed by the client has not changed, the website to be crawled is unsuccessfully logged in.

In one of the embodiments, the website to be crawled is a mailbox website; when the program is executed by the processor, the following steps can also be implemented: selecting the mail with the title corresponding to the data to be crawled from the mailbox website; crawling from the selected mail Take the data in the preset field as the crawled data to be crawled.

In one of the embodiments, when the program is executed by the processor, the following steps can also be implemented: encrypting the crawled data to be crawled; packaging the encrypted data to be crawled; packing the packed data to be crawled Get the data and send it to the server.

For the above limitations on the computer-readable storage medium, please refer to the above specific limitations on the web page data crawling method, and details will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the programs can be stored in a non-volatile computer-readable storage medium When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) and the like.

The various technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the various technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, should be considered as within the scope of this specification.

The above-mentioned embodiments only express several implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but should not be construed as limiting the patent scope of the invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (10)

1. A webpage data crawling method, is characterized in that, described method comprises: Receive the account and password corresponding to the website to be crawled through the login interface of the website to be crawled embedded in the client, and log in to the website to be crawled through the account and password corresponding to the website to be crawled; Detecting whether the website to be crawled is successfully logged in; When successfully logging in to the website to be crawled, it is judged whether the account of the client matches the account of the website to be crawled; When the account of the client matches the account of the website to be crawled, then crawl the data to be crawled in the website to be crawled; Send the crawled data to be crawled to the server. 2. The method according to claim 1, wherein the step of crawling the data to be crawled in the website to be crawled comprises: Send a crawling script acquisition request to the server; Receiving the crawling script corresponding to the crawling script acquisition request returned by the server; The data to be crawled in the website to be crawled is crawled through the crawling script. 3. The method according to claim 2, characterized in that, before the step of sending the crawling script acquisition request to the server, further comprising: Obtain the time when the crawling script was returned by the receiving server last time; When the difference between the time when the crawling script was returned by the receiving server last time and the current time is within a preset range, the data to be crawled in the website to be crawled is crawled through the crawling script returned by the server received last time ; When the difference between the last time the server received the crawling script and the current time is not within the preset range, execute the step of sending the crawling script acquisition request to the server. 4. The method according to claim 1, wherein the step of detecting whether the website to be crawled is successfully logged in comprises: Detect whether the URL address of the current page displayed by the client changes; When the URL address of the current page displayed by the client changes, the website to be crawled is successfully logged in; When the URL address of the current page displayed by the client does not change, the login to the website to be crawled is unsuccessful. 5. The method according to claim 1, wherein the website to be crawled is a mailbox website; The step of crawling the data to be crawled in the website to be crawled includes: Selecting emails with titles corresponding to the data to be crawled from the mailbox website; Crawl the data of the preset fields from the selected emails as the crawled data to be crawled. 6. The method according to claim 1, wherein the step of sending the crawled data to be crawled to the server includes: Encrypt the crawled data to be crawled; Pack the encrypted data to be crawled; Send the packaged data to be crawled to the server. 7. A web page data crawling device, characterized in that the device comprises: The login module is used to receive the account and password corresponding to the website to be crawled through the login interface of the website to be crawled embedded in the client, and log in to the website to be crawled through the account and password corresponding to the website to be crawled. crawl the website; A detection module, configured to detect whether the website to be crawled is successfully logged in; A verification module, configured to determine whether the account of the client matches the account of the website to be crawled when successfully logging in to the website to be crawled; A crawling module, configured to crawl the data to be crawled in the website to be crawled when the account of the client matches the account of the website to be crawled; The sending module is used to send the crawled data to be crawled to the server. 8. The device according to claim 7, wherein the sending module is further configured to send a crawling script acquisition request to the server; The crawling module includes: A receiving unit, configured to receive the crawling script returned by the server and corresponding to the crawling script acquisition request; A crawling unit, configured to crawl the data to be crawled in the website to be crawled through the crawling script. 9. A user terminal, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, characterized in that, when the processor executes the computer program, any of claims 1 to 6 is realized. A step in said method. 10. A computer-readable storage medium, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.