CN102006174B - Data processing method and device based on online behavior of mobile phone user - Google Patents

Abstract

The present invention relates to a data processing method and device based on the online behavior of mobile phone users, wherein the method includes: generating a first call list including the user's access URL according to the user's online surfing data; and preprocessing the data in the first call list according to predetermined rules , generating a second bill; performing statistical analysis on the data in the second bill. Before the call bill data is put into the database, the invention preprocesses the call bill data firstly, generates new call bill data after a series of preprocessing and puts it into the database, and then the system database performs statistical analysis on the second call bill data. This eliminates the need for the database to analyze a large number of URL data, thereby greatly improving the efficiency of dialogue list data processing, and solving the performance bottleneck problem of mobile phone users' online behavior analysis.

Description

Data processing method and device based on mobile phone user's online behavior

technical field

The invention relates to the technical field of mobile networks, in particular to a data processing method and device based on mobile phone users' online behavior.

Background technique

At present, in mobile network services, analyzing and mining user surfing data has become a popular trend. With the continuous growth of service providers and the number of users using mobile phones to access the Internet, the bills generated by mobile service systems continue to increase. In service systems with a large number of bills, the service volume TPS (Tip-Per-Second) has even reached 5000 Articles/second, the amount of data per day is about 100 million to 200 million.

As an operator that needs to understand the online behavior of mobile phone users, it is usually necessary to analyze the online behavior of mobile phone users as follows:

a) Internet type analysis: the types of websites with high user visit frequency;

b) Designated website traffic analysis: visit traffic of the website or specific content within the website;

c) Advertisement access flow analysis: the access flow of specific classifications of advertisement URLs.

In the traditional technology, the method adopted for analyzing the mobile phone user's online data is: analyzing the URL (Uniform/Universal Resource Locator, also known as the web page address) field in the bill generated by the mobile service system. in:

The process of online type analysis includes: entering bill data into the database, maintaining a comparison table of HOST and type, analyzing HOST for a single URL, querying the type from the comparison table, and analyzing all URLs;

The process of specifying website traffic analysis includes: bill data storage, maintenance of URL conversion rule comparison table, conversion of a single URL and analysis of all URLs;

The process of advertising access traffic analysis includes: storing bill data, maintaining the correspondence table between URLs and advertisements, querying which advertisement a single URL belongs to, and analyzing all URLs.

In the case of a large data flow, using the above method to process dialog data will cause a bottleneck phenomenon in system performance. Because the URLs in the call list are all encrypted and stored, the URL must be decrypted before parsing the URL, and complex calculations must be performed on the decrypted string of the URL, so the data processing time is long. The test data of user online behavior analysis is shown in Table 1 below:

Bill generation speed 3000W every day, 10W every 5 minutes 10W bill processing speed It takes 8 minutes for 10W The processing speed of a day's call list It takes about 1.5 days

Table 1

It can be seen from the above table 1 that the speed of generating bills is faster than the speed of processing bills, so that bills will pile up more and more and cannot be processed in time, which not only causes serious delays in data processing, but also increases the system database processing burden.

Contents of the invention

The main purpose of the present invention is to provide a data processing method and device based on mobile phone users' online behavior, aiming at improving the processing speed of mobile phone users' online data and improving system performance.

The present invention proposes a data processing method based on the online behavior of mobile phone users, the method comprising:

Generate the first call list containing the URL of the user's web page address according to the user's Internet access data;

Preprocessing the data in the first bill according to predetermined rules to generate a second bill;

Statistical analysis is performed on the data in the second bill.

Preferably, the step of preprocessing the data in the first bill according to predetermined rules includes:

The data in the first call list is analyzed and processed by URL type of Internet access and/or specified website traffic analysis and/or advertisement access traffic analysis and processing.

Preferably, the step of analyzing and processing the data in the first call list for the URL type of Internet access includes:

Adding a field of the URL type in the first bill to store the category to which the URL belongs;

Analyzing the source URL in the first bill;

Find the category corresponding to the source URL from the preset URL category comparison table, and write it into the field of the URL type corresponding to the source URL in the second bill.

Preferably, the step of performing specified website traffic analysis on the data in the first call list includes:

Adding a new URL field in the first bill for storing the converted new URL;

Convert the source URL in the first bill according to a predetermined conversion rule;

Write the converted source URL into the new URL field corresponding to the source URL in the second bill.

Preferably, the step of analyzing and processing the data in the first bill includes:

Adding an advertisement URL field in the first bill to store the advertisement URL;

separating the advertising URL according to the predetermined identifier carried by the source URL in the first bill;

Write the advertising URL into the advertising URL field corresponding to the source URL in the second bill.

Preferably, the step of analyzing and processing the data in the first bill includes:

Adding an advertisement URL field in the first bill to store the advertisement URL;

separating the advertising URL according to the predetermined identifier carried by the source URL in the first bill;

Write the advertising URL into the advertising URL field corresponding to the source URL in the second bill.

The present invention also proposes a data processing device based on the online behavior of mobile phone users, including:

The original bill generation module is used to generate the first bill that includes the user's access URL according to the user's online data;

A new bill generating module, configured to preprocess the data in the first bill according to predetermined rules to generate a second bill;

The new bill data processing module is used to perform statistical analysis on the data in the second bill.

Preferably, the new bill generation module is further configured to analyze the data in the first bill for URL analysis of the online type, designated website traffic analysis and/or advertisement access traffic analysis.

Preferably, the new bill generation module includes:

A field adding unit, configured to add a field for storing the URL type of the category to which the URL belongs in the first bill;

A parsing unit, configured to parse the source URL in the first bill;

A writing unit, configured to search for the category corresponding to the source URL from the preset URL category comparison table, and write the field of the URL type corresponding to the source URL in the second bill.

Preferably, the field adding unit is further configured to add a new URL field for storing the converted new URL in the first bill;

The parsing unit is further configured to convert the source URL in the first bill according to a predetermined conversion rule;

The writing unit is also used to write the converted source URL into the new URL field corresponding to the source URL in the second bill; or

The field adding unit is also used to add an advertisement URL field for storing the advertisement URL in the first bill;

The parsing unit is further configured to separate the advertisement-type URL according to the predetermined identifier carried by the source URL in the first bill;

The writing unit is further configured to write the advertising URL into the advertising URL field corresponding to the source URL in the second bill.

A data processing method and device based on the online behavior of mobile phone users proposed by the present invention uses a preprocessing device such as an interface machine to preprocess the dialogue ticket data before the bill data is stored in the database. Classify and summarize URLs, convert URLs according to certain rules, etc., and generate new bill data into the database after a series of preprocessing. Afterwards, the system database performs statistical analysis and processing on the second bill data. Thus, the URL parsing process is handed over to the interface machine for processing, and a new bill is generated from the parsed result data, and the system database performs statistical analysis directly according to the result data, eliminating the process of analyzing a large amount of URL data. Thereby, the efficiency of dialogue list data processing is greatly improved, and the performance bottleneck problem of mobile phone users' online behavior analysis is solved.

Description of drawings

Fig. 1 is a schematic flow chart of an embodiment of a data processing method based on mobile phone user online behavior of the present invention;

Fig. 2 is the schematic flow chart of carrying out the Internet type URL analysis processing to the data in the first bill in an embodiment of the data processing method based on the mobile phone user's online behavior of the present invention;

Fig. 3 is the schematic flow chart of carrying out the specified website traffic analysis process to the data in the first call list in an embodiment of the data processing method based on the mobile phone user's online behavior in the present invention;

Fig. 4 is a schematic flow diagram of performing advertisement access traffic analysis processing on the data in the first call list in an embodiment of the data processing method based on the mobile phone user's online behavior in the present invention;

Fig. 5 is a schematic structural diagram of an embodiment of a data processing device based on mobile phone users' online behavior in the present invention;

FIG. 6 is a schematic structural diagram of a new bill generation module in an embodiment of the data processing device based on mobile phone users' online behavior of the present invention.

In order to make the technical solution of the present invention clearer and clearer, it will be further described below in conjunction with the accompanying drawings.

Detailed ways

The solution of the embodiment of the present invention is mainly to preprocess the call list data before the call list data is stored in the database. After the preprocessing of the system, new bill data is generated and put into the database. Afterwards, the system database performs statistical analysis and processing on the second bill data, so as to improve the efficiency of the data processing of the conversation bill, and solve the performance bottleneck problem of mobile phone users' online behavior analysis.

As shown in Figure 1, an embodiment of the present invention proposes a data processing method based on mobile phone users' online behavior, including:

Step S101, generating a first bill containing the user's access URL according to the user's online data;

In this embodiment, the user can surf the Internet through the mobile phone and visit various websites to obtain corresponding network information. When a user surfs the Internet through a mobile phone, the mobile service system obtains network data according to the mobile phone user's access website, and generates an original call list, which is the first call list in this embodiment. Increase.

Wherein, the bill includes the URL accessed by the user. URL is an identification method used to completely describe the addresses of web pages and other resources on the Internet. Every web page on the Internet has a unique name identifier, usually called a URL address. This address can be a local disk, or a certain computer on the LAN, and more often it is a site on the Internet. Simply put, a URL is a web address, commonly known as a "web address".

After the mobile service system obtains the first bill, it needs to analyze and statistically process the data of the first bill, so as to understand the online behavior of mobile phone users based on the processing results, such as: which types of websites users often like, and certain designated websites The visit flow of mobile phone users and the advertisement visit traffic that merchants care about, so as to take corresponding commercial measures according to the online behavior of mobile phone users.

Step S102, preprocessing the data in the first bill according to predetermined rules to generate a second bill;

In this embodiment, the predetermined rule is formulated for the main issues of the operator concerned, such as the type of website visited, website visit traffic, and advertisement visit traffic, wherein the preprocessing of the data in the first bill according to the predetermined rule includes: The data in a call list is analyzed and processed by Internet type URL and/or specified website traffic analysis and/or advertisement access traffic analysis and processing. Specifically, for example, it can classify and summarize URLs generated by users surfing the Internet, and convert URLs according to certain rules. wait.

According to the requirement of obtaining data processing information, the above predetermined rules may also be other similar rules.

In this embodiment, the data in the first call list is preprocessed, independent equipment can be used, such as an interface machine, and the interface machine is used to preprocess the dialogue list data first, such as classifying and summarizing URLs generated by users surfing the Internet, URLs according to Certain rules are converted etc., after a series of preprocessing, generate a new bill, which is the second bill in this embodiment, and then put the new bill data into the database, so that in the subsequent processing process, the system database will update the second bill data Perform statistical analysis. In this embodiment, the storage of the second bill data may use the IMP storage program to input the preprocessed data into the database table specified by the system.

Step S103, performing statistical analysis on the data in the second bill.

As mentioned above, the newly generated bill data is submitted to the system database for statistical analysis and processing. For example, according to the category of the URL in the second bill, the summary data of a certain type of URL that the user expects to obtain can be counted. Thus, the analysis process of the URL in the first call list is handed over to the interface machine for processing, and the result data after analysis generates a new call list, and the system database directly conducts statistical analysis based on the result data, eliminating the need for a large number of URL data The analysis process greatly improves the efficiency of dialogue list data processing and solves the performance bottleneck problem of mobile phone users' online behavior analysis.

As shown in Figure 2, in step S102, the step that the data in the first call list is carried out to surf the Internet type URL analysis process includes:

Step S1021, adding a URL type field for storing the category to which the URL belongs in the first bill;

Step S1022, parsing the source URL in the first bill;

Step S1023: Find the category corresponding to the source URL from the preset URL category comparison table, and write it into the field of the URL type corresponding to the source URL in the second bill.

The process of carrying out the Internet type URL analysis process to the data in the first call list is illustrated below with specific examples, if there is the first call list data shown in Table 2 below:

serial number URL 1 http://www.sina.com/sport/1001.htm 2 http://www.sina.com/sport/1002.htm 3 http://www.sina.com/sport/1003.htm 4 http://www.sina.com/news/1004.htm 5 http://www.sina.com/news/1005.htm 6 http://www.sina.com/movie/1006.htm

Table 2

Among them, the URL classification standard is the preset URL category comparison table as shown in Table 3 below:

Category URL sports http://www.sina.com/sport/* News http://www.sina.com/news/* Movies http://www.sina.com/movie/

table 3

Table 4 below shows the results of preprocessing through the URL analysis of Internet access types:

serial number URL Category 1 http://www.sina.com/sport/1001.htm sports 2 http://www.sina.com/sport/1002.htm sports 3 http://www.sina.com/sport/1003.htm sports 4 http://www.sina.com/news/1004.htm News 5 http://www.sina.com/news/1005.htm News 6 http://www.sina.com/movie/1006.htm Movies

Table 4

It can thus be drawn that, according to the category of the URL in the second call list, the summary data of a certain class such as the URL of the news class that the user expects to obtain can be counted. There are two URLs of the news class shown in Table 4, http ://www.sina.com/news/1004.htm and http://www.sina.com/news/1005.htm.

As shown in Figure 3, in step S102, the step of carrying out the designated website flow analysis processing to the data in the first bill includes:

Step S1024, adding a new URL field for storing the converted new URL in the first bill;

Step S1025, converting the source URL in the first bill according to a predetermined conversion rule;

Step S1026, write the converted source URL into the new URL field corresponding to the source URL in the second bill.

Wherein, the predetermined conversion rule can be a conversion rule table formulated according to the system HOST file setting rules. For example, for a certain HOST, the following rules are arranged, as shown in Table 5, wherein each URL is set with "whether to process extension", "Ignore parameters" option.

table 5

According to the conversion rule table above, the source URL in the first bill can be converted into a new URL, and written into the corresponding new URL field in the second bill. According to the information in the new URL field in the second bill, the access traffic of the designated website or specific content in the designated website can be counted.

It should be noted that when preprocessing the first bill data, the three preprocessing methods described in this embodiment can be used, namely: performing Internet-type URL analysis and processing on the data in the first bill, and specifying website traffic analysis. The processing and the analysis and processing of the advertisement access flow are carried out in combination, so that, according to the finally generated second bill, the type of website visited by the user, the visit flow of the specified website, and the advertisement visit flow can be simultaneously counted.

Through data test, obtain the solution that the embodiment of the present invention analyzes mobile phone user's online behavior and the comparative situation of traditional solution as shown in table 6 below:

Table 6

It can be seen from Table 6 that, compared with the traditional technology, the solution of the embodiment of the present invention can analyze the user's online data more quickly, greatly improves the processing speed of bill data, reduces the processing burden of the system database, and solves the problem of mobile phone users' online behavior. Analyze performance bottlenecks.

In this embodiment, before the bill data is stored in the database, a preprocessing device such as an interface machine is used to preprocess the dialog bill data. The preprocessing process includes classifying and summarizing the URLs generated by users surfing the Internet, converting the URLs according to certain rules, etc., through After a series of preprocessing, new bill data is generated and stored in the database. Afterwards, the system database performs statistical analysis and processing on the second bill data. Thus, the URL parsing process is handed over to the interface machine for processing, and a new bill is generated from the parsed result data, and the system database performs statistical analysis directly according to the result data, eliminating the process of analyzing a large amount of URL data. Thereby, the efficiency of dialogue list data processing is greatly improved, and the performance bottleneck problem of mobile phone users' online behavior analysis is solved.

As shown in Figure 4, the step of performing advertisement access traffic analysis processing on the data in the first bill in step S102 includes:

Step S1027, adding an advertisement URL field for storing the advertisement URL in the first bill;

Step S1028, separating the advertising URL according to the predetermined identifier carried by the source URL in the first bill;

Step S1029, write the advertising URL into the advertising URL field corresponding to the source URL in the second bill.

As shown in Figure 5, an embodiment of the present invention proposes a data processing device based on mobile phone users' online behavior, including: an original bill generation module 501, a new bill generation module 502, and a new bill data processing module 503, wherein:

The original bill generation module 501 is used to generate the first bill that includes the user's access URL according to the user's online data;

In this embodiment, the user can surf the Internet through the mobile phone and visit various websites to obtain corresponding network information. When a user surfs the Internet through a mobile phone, the original call list generation module 501 in the mobile service system obtains network data according to the mobile phone user's access URL, and generates the original call list, which is the first call list in this embodiment. The amount of bills generated by the business system increases accordingly.

Wherein, the bill includes the URL accessed by the user. URL is an identification method used to completely describe the addresses of web pages and other resources on the Internet. Every web page on the Internet has a unique name identifier, usually called a URL address. This address can be a local disk, or a certain computer on the LAN, and more often it is a site on the Internet. Simply put, a URL is a web address, commonly known as a "web address".

After the mobile service system obtains the first bill, it needs to analyze and statistically process the data of the first bill, so as to understand the online behavior of mobile phone users based on the processing results, such as: which types of websites users often like, and certain designated websites The visit flow of mobile phone users and the advertisement visit traffic that merchants care about, so as to take corresponding commercial measures according to the online behavior of mobile phone users.

The new bill generation module 502 is used to preprocess the data in the first bill according to predetermined rules to generate a second bill;

In this embodiment, the new bill generation module 501 performs preprocessing on the data in the first bill according to predetermined rules, specifically including performing URL analysis processing on the Internet type, traffic analysis processing on a specified website and/or advertisement processing on the data in the first bill. Access traffic analysis processing.

Among them, the predetermined rules are formulated for the main issues of the operator concerned, such as the type of website visited, the traffic of website visits, and the traffic of advertisements. The preprocessing of the data in the first bill according to the predetermined rules includes: The data is analyzed and processed for Internet access type URLs and/or specified website traffic analysis and/or advertising access traffic analysis processing. Specifically, for example, URLs generated by users can be classified and summarized, and URLs can be converted according to certain rules.

According to the requirement of obtaining data processing information, the above predetermined rules may also be other similar rules.

In this embodiment, the data in the first call list is preprocessed, independent equipment can be used, such as an interface machine, and the interface machine is used to preprocess the dialogue list data first, such as classifying and summarizing URLs generated by users surfing the Internet, URLs according to Certain rules are used for conversion, etc. After a series of preprocessing, the second bill generation module 502 generates new bill data for storage, so that in the subsequent processing, the system database performs statistical analysis on the second bill data. In this embodiment, the second bill data storage may use the IMP storage program to input the preprocessed data into the database table specified by the system.

The new bill data processing module 503 is configured to perform statistical analysis on the data in the second bill.

As mentioned above, the newly generated call list data is handed over to the new call list data processing module 503 of the system database for statistical analysis and processing. Aggregated data.

Thus, the analysis process of the URL in the first call list is handed over to the interface machine for processing, and the result data after analysis generates a new call list, and the system database directly conducts statistical analysis based on the result data, eliminating the need for a large number of URL data The analysis process greatly improves the efficiency of dialogue list data processing and solves the performance bottleneck problem of mobile phone users' online behavior analysis.

As shown in Figure 6, the new bill generation module 502 includes: a field adding unit 5021, a parsing unit 5022 and a writing unit 5023, wherein:

A field adding unit 5021, configured to add a field for storing the URL type of the category to which the URL belongs in the first bill;

A parsing unit 5022, configured to parse the source URL in the first bill;

The writing unit 5023 is configured to search the category corresponding to the source URL from the preset URL category comparison table, and write the field of the URL type corresponding to the source URL in the second bill.

Further, the field adding unit 5021 is also used to add a new URL field for storing the converted new URL in the first bill;

The parsing unit 5022 is further configured to convert the source URL in the first bill according to a predetermined conversion rule;

The writing unit 5023 is further configured to write the converted source URL into the new URL field corresponding to the source URL in the second bill.

Furthermore, the field adding unit 5021 is also used to add an advertisement URL field for storing the advertisement URL in the first bill;

The parsing unit 5022 is further configured to separate the advertisement-type URL according to the predetermined identifier carried by the source URL in the first bill;

The writing unit 5023 is further configured to write the advertising URL into the advertising URL field corresponding to the source URL in the second bill.

The data processing method and device based on the mobile phone user's online behavior in the embodiment of the present invention use a preprocessing device such as an interface machine to preprocess the dialog ticket data before the bill data is stored in the database. The preprocessing process includes URL generated by the user surfing the Internet Sorting and summarizing, converting URLs according to certain rules, etc., after a series of preprocessing, new bill data is generated and stored in the database. Afterwards, the system database performs statistical analysis and processing on the second bill data. Thus, the URL parsing process is handed over to the interface machine for processing, and a new bill is generated from the parsed result data, and the system database performs statistical analysis directly according to the result data, eliminating the process of analyzing a large amount of URL data. Thereby, the efficiency of dialogue list data processing is greatly improved, and the performance bottleneck problem of mobile phone users' online behavior analysis is solved.

The above is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or process transformation made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (7)

1. A data processing method based on mobile phone user surfing behavior, said method comprising: Generate the first call list containing the URL of the user's web page address according to the user's Internet access data; Perform preprocessing on the data in the first call list according to predetermined rules to generate a second call list; specifically include: performing Internet access type URL analysis processing and/or specifying website traffic analysis processing and/or analyzing and processing the data in the first call list or advertisement access traffic analysis and processing; adopt independent equipment to preprocess the data in the first call list; said preprocessing includes classifying and summarizing the URLs generated by users surfing the Internet, and converting the URLs according to certain rules; Statistical analysis is performed on the data in the second bill. 2. method according to claim 1, is characterized in that, described the step that the data in the first call list is carried out to surf the Internet type URL analysis process comprises: Adding a field of the URL type in the first bill to store the category to which the URL belongs; Analyzing the source URL in the first bill; Find the category corresponding to the source URL from the preset URL category comparison table, and write it into the field of the URL type corresponding to the source URL in the second bill. 3. The method according to claim 1 or 2, characterized in that, the step of carrying out the designated website traffic analysis process to the data in the first bill comprises: Adding a new URL field in the first bill for storing the converted new URL; Convert the source URL in the first bill according to a predetermined conversion rule; Write the converted source URL into the new URL field corresponding to the source URL in the second bill. 4. The method according to claim 1, characterized in that, the step of analyzing and processing the data in the first call list for advertising access traffic comprises: Adding an advertisement URL field in the first bill to store the advertisement URL; separating the advertising URL according to the predetermined identifier carried by the source URL in the first bill; Write the advertising URL into the advertising URL field corresponding to the source URL in the second bill. 5. A data processing device based on the online behavior of mobile phone users, characterized in that it comprises: The original bill generation module is used to generate the first bill that includes the user's access URL according to the user's online data; The new bill generation module is used to preprocess the data in the first bill according to predetermined rules to generate a second bill; specifically, it is used to analyze and process the data in the first bill for online type URL analysis and specify Website traffic analysis and/or advertisement access traffic analysis and processing; independent equipment is used to pre-process the data in the first call list; the pre-processing includes classifying and summarizing URLs generated by users when surfing the Internet, and performing URLs according to certain rules conversion; The new bill data processing module is used to perform statistical analysis on the data in the second bill. 6. The device according to claim 5, wherein the new bill generation module comprises: A field adding unit, configured to add a field for storing the URL type of the category to which the URL belongs in the first bill; A parsing unit, configured to parse the source URL in the first bill; A writing unit, configured to search for the category corresponding to the source URL from the preset URL category comparison table, and write the field of the URL type corresponding to the source URL in the second bill. 7. The device of claim 6, wherein: The field adding unit is also used to add a new URL field for storing the converted new URL in the first bill; The parsing unit is further configured to convert the source URL in the first bill according to a predetermined conversion rule; The writing unit is also used to write the converted source URL into the new URL field corresponding to the source URL in the second bill; or The field adding unit is also used to add an advertisement URL field for storing the advertisement URL in the first bill; The parsing unit is further configured to separate the advertisement-type URL according to the predetermined identifier carried by the source URL in the first bill; The writing unit is further configured to write the advertising URL into the advertising URL field corresponding to the source URL in the second bill.