CN106815274A - Daily record data method for digging and system based on Hadoop - Google Patents

Abstract

The invention discloses a log data mining method based on Hadoop, which saves the acquired first log data set in the current time period into the Hadoop database; if the number of the first log data sets already stored in the Hadoop database satisfies the preset value, then use the preset parallel operation model to carry out parallel aggregation processing on the first log data set in the Hadoop database to obtain the second log data set; according to the dimension of the log data in the second log data set, the second log data The log data in the set is divided into dimensions, and the obtained third log data sets corresponding to different dimensions are saved in the Hadoop database. The invention also discloses a log data mining system based on Hadoop. The invention can quickly and effectively realize massive data mining, and meets the storage and calculation requirements for massive data mining.

Description

Hadoop-based log data mining method and system

technical field

The invention relates to the field of computer data processing, in particular to a Hadoop-based log data mining method and system.

Background technique

Since entering the Internet era, how to quickly find more suitable, quantifiable, predictable and precise marketing strategies in the ever-increasing mass of user information has become the core demand of many companies, including operators.

However, traditional databases have limited data computing capabilities and high storage costs, which cannot meet the needs of massive data mining.

The above content is only used to assist in understanding the technical solution of the present invention, and does not mean that the above content is admitted as prior art.

Contents of the invention

The main purpose of the present invention is to provide a log data mining method and system based on Hadoop, aiming to solve the technical problems that traditional databases have limited data computing capabilities, high storage costs, and cannot provide massive data mining.

In order to achieve the above object, a kind of log data mining method based on Hadoop provided by the present invention comprises:

The first log data set in the current time period obtained is saved in the Hadoop database;

If the number of the first log data sets stored in the Hadoop database satisfies the preset value, then utilize the preset parallel operation model to carry out parallel aggregation processing on the first log data sets in the Hadoop database to obtain the second collection of log data;

Dimensionally divide the log data in the second log data set according to the dimensions of the log data in the second log data set, and save the obtained third log data sets corresponding to different dimensions into the Hadoop database.

Preferably, the method also includes:

Obtain log data in the current time period from the network side;

Aggregating the log data within the current time period is performed to obtain a first set of log data within the current time period.

Preferably, after the step of obtaining log data in the current time period from the network side, the step further includes:

Perform data cleaning on the log data in the current time period to obtain the cleaned log data in the current time period;

Then the step of aggregating the log data in the current time period to obtain the first log data set in the current time period includes:

Perform aggregation processing on the cleaned log data in the current time period to obtain a first set of log data in the current time period.

Preferably, the method also includes:

If a data query instruction is received, the third log data set corresponding to the query dimension is read from the Hadoop database according to the query dimension included in the data query instruction;

Perform data analysis on the third log data set, and display the results of the data analysis on the display interface.

Preferably, said performing data analysis on said third log data set includes:

performing user grouping on the users in the third log data set according to a preset clustering algorithm to obtain a user grouping list;

Obtain a level configuration table corresponding to at least two user dimensions according to the log data of users in the user grouping list, the user dimension is preset, and the level configuration table includes users in the user grouping list according to the user The level at which the dimension is hierarchically determined.

To achieve the above object, the present invention also provides a Hadoop-based log data mining system, comprising:

The first saving module is used to save the first log data collection in the current time period obtained in the Hadoop database;

Parallel aggregation module, for if the number of the first log data collection that described Hadoop database has preserved meets preset numerical value, then utilize the preset parallel computing model to carry out parallel to the first log data collection in described Hadoop database Aggregating and processing to obtain a second log data set;

The division and storage module divides the log data in the second log data set according to the dimensions of the log data in the second log data set, and saves the obtained third log data sets corresponding to different dimensions in the Hadoop database.

Preferably, the system also includes:

The acquisition module is used to acquire log data in the current time period from the network side;

The first aggregation module is configured to perform aggregation processing on the log data within the current time period to obtain a first set of log data within the current time period.

Preferably, the system also includes a cleaning module;

The cleaning module is used to perform data cleaning on the log data in the current time period after the acquisition module acquires the log data in the current time period, so as to obtain the cleaned log data in the current time period;

And the first aggregation module is specifically configured to aggregate the cleaned log data in the current time period to obtain the first log data set in the current time period.

Preferably, the system also includes:

The reading module is used to read the third log data set corresponding to the query dimension from the Hadoop database according to the query dimension contained in the data query instruction if the data query instruction is received;

An analysis module, configured to perform data analysis on the third log data set, and display the result of the data analysis on the display interface.

Preferably, the analysis module includes:

A clustering module, configured to perform user grouping on users in the third log data set according to a preset clustering algorithm to obtain a user grouping list;

An acquisition and display module, configured to obtain a level configuration table corresponding to at least two user dimensions according to the log data of users in the user group list, the user dimension is preset, and the level configuration table includes the user group list of users are graded and determined according to the user dimension

The present invention provides a log data mining method based on Hadoop, which saves the acquired first log data collection in the current time period into the Hadoop database, if the number of the first log data collections stored in the Hadoop database satisfies the preset numerical value, then use the preset parallel operation model to perform parallel aggregation processing on the first log data set in the Hadoop database to obtain the second log data set. According to the dimension of the log data in the second log data set, the second The log data in the log data set is maintained and divided, and the obtained third log data sets corresponding to different dimensions are stored in the Hadoop database to complete the mining of the log data. Since the Hadoop database has good distributed storage and parallel computing capabilities, using the Hadoop database to store log data in a distributed manner and use the parallel computing model for parallel computing can quickly and effectively realize massive data mining and meet the needs of massive data mining. Storage and computing requirements for mining.

Description of drawings

Fig. 1 is the schematic flow sheet of the log data mining method based on Hadoop of the first embodiment of the present invention;

Fig. 2 is a schematic flow chart of additional steps before step 101 of the first embodiment in Fig. 1;

FIG. 3 is a schematic flow chart of additional steps after step 103 of the first embodiment in FIG. 1;

Fig. 4 is the schematic diagram of the functional module of the log data mining system based on Hadoop in the second embodiment of the present invention;

FIG. 5 is a schematic diagram of additional functional modules in the second embodiment of FIG. 4;

FIG. 6 is a schematic diagram of additional functional modules in the second embodiment of FIG. 4 .

The realization of the purpose of the present invention, functional characteristics and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

detailed description

It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The present invention provides a log data mining method based on Hadoop, which saves the acquired first log data collection in the current time period into the Hadoop database, if the number of the first log data collections stored in the Hadoop database satisfies the preset numerical value, then use the preset parallel operation model to perform parallel aggregation processing on the first log data set in the Hadoop database to obtain the second log data set. According to the dimension of the log data in the second log data set, the second The log data in the log data set is maintained and divided, and the obtained third log data sets corresponding to different dimensions are stored in the Hadoop database to complete the mining of the log data. Since the Hadoop database has good distributed storage capability and parallel computing capability, using the Hadoop database to store log data in a distributed manner and using the preset parallel computing model in Hadoop to perform parallel computing can quickly and effectively realize massive data processing. Mining to meet the storage and computing needs of massive data mining.

Referring to Fig. 1, it is a schematic flow chart of the Hadoop-based log data mining method in the first embodiment of the present invention, including:

Step 101, saving the first log data collection obtained in the current time period into the Hadoop database;

In the embodiment of the present invention, the Hadoop-based log data mining method can be applied in a Hadoop-based log data mining system (hereinafter referred to as: mining system), and the mining system saves the first log data set in the current time period obtained to the Hadoop database.

Wherein, the mining system obtains the first log data set according to the time period. For example, if the time period is 15 minutes or 30 minutes, the mining system obtains the first log data set within the current 15-minute time period or obtains the current The first log data set in a 30-minute time period.

Wherein, the time period is a cycle for acquiring data, and the length of the time period may be determined according to the amount of data.

Among them, Hadoop can implement Distributed File System (Hadoop Distributed File System, HDFS). The framework core of Hadoop is Hadoop database and parallel operation model. Among them, Hadoop database can provide distributed storage for massive data, and the parallel operation model can provide massive The data provides parallel operation.

Preferably, the parallel computing model is a mapreduce computing model.

Step 102, if the number of the first log data set stored in the Hadoop database satisfies the preset value, then use the preset parallel operation model to carry out parallel aggregation processing on the first log data set in the Hadoop database to obtain the second log data collection;

In the embodiment of the present invention, the mining system saves the obtained first log data set in the Hadoop database in each time period, if the number of the first log data sets that the Hadoop database has saved meets the preset value, the first log data set in the Hadoop database can be aggregated using the preset parallel operation model in the Hadoop framework to obtain the second log data set.

Wherein, in practical applications, this value can be preset according to specific needs. For example, if the above-mentioned time period is 15 minutes, and the first log data set within one hour needs to be aggregated, the preset value is 4. If the above time period is 30 minutes, and the first log data set within 1 day needs to be aggregated, the preset value is 48.

It can be understood that, based on the above-mentioned aggregation processing, the mining system can also use a similar method to obtain log data collections in different time periods, for example: the first log data collection with 4 time periods of 15 minutes can be used to obtain log data collections within an hour The log data set of , you can use 24 log data sets within one hour to get the log data set within one day, you can use 30 log data sets within one day to get the log data set within one month, and so on, you can get Collection of log data at different times to meet different needs.

In the embodiment of the present invention, when the mining system uses a preset parallel operation model to perform parallel aggregation processing, it accumulates count values of the same log data.

Step 103: Dimensionally divide the log data in the second log data set according to the dimensions of the log data in the second log data set, and save the obtained third log data sets corresponding to different dimensions into the Hadoop database.

In the embodiment of the present invention, after the mining system obtains the second log data set, it divides the dimensions of the log data in the second log data set according to the dimensions of the log data in the second log data set, and divides the obtained The third log data collections corresponding to different dimensions are saved in the Hadoop database to realize the mining of massive log data, and the saved third log data collections can be used as the data source for user data query, and support icon and graphic query on the display interface and multiple Dimension query enables data to be displayed from multiple angles to achieve the display effect of data mining.

Among them, there are many dimensions of log data, including but not limited to online content, online location, and online time. Among them, online content refers to the browsing location of the user, and the browsing location can be a specific location, such as Baidu, Sohu, Sina Weibo, etc., can also be a type of website, such as: music, movies, etc. The online location refers to the geographical range of the IP address used by the user, and the online time refers to the time when the log data is generated. And the division of dimensions is based on the requirements of the system, and the further description of the overall behavior of users is completed through the data on the dimensions. It should be noted that for different types of log data, the dimensions of the log data are also different. For example, when the user traffic data in the log data is mined using the technical solution in the embodiment of the present invention, the dimension In addition to the above-mentioned online content, online location, and online time, it can also include online frequency, user age, monthly consumption, etc. Therefore, in practical applications, dimensions can be divided according to specific needs, which is not limited here.

Preferably, in the embodiment of the present invention, after the mining system saves the third log data sets corresponding to different dimensions into the Hadoop database, it can also save the third log data sets corresponding to the different dimensions in the column storage array, It enables the collaborative work of the Hadoop database and the column storage array to meet the data requirements of different application scenarios.

Preferably, because the mining system executes the above-mentioned parallel aggregation processing and dimension division operations when the number of the first log data collections stored in the Hadoop database satisfies the preset value, the obtained third In fact, the log data set also corresponds to a time period. When saving, the mining system can save the corresponding relationship between the dimension, the time period and the third log data set.

In the embodiment of the present invention, the mining system saves the first log data set obtained in the current time period into the Hadoop database, and if the number of the first log data set saved by the Hadoop database meets the preset value, then use The preset parallel operation model performs parallel aggregation processing on the first log data set in the Hadoop database to obtain the second log data set, and according to the dimension of the log data in the second log data set, the log data in the second log data set The log data is maintained and divided, and the obtained third log data sets corresponding to different dimensions are stored in the Hadoop database to complete the mining of log data. Since the Hadoop database has good distributed storage and parallel computing capabilities, using the Hadoop database to store log data in a distributed manner and using the parallel computing model in Hadoop to perform parallel computing can quickly and effectively realize massive data mining, satisfying Storage and computing requirements for mining massive amounts of data.

Please refer to FIG. 2, which is a schematic flow chart of additional steps before step 101 in the first embodiment shown in FIG. 1 of the present invention, including:

Step 201, obtaining log data in the current time period from the network side;

In the embodiment of the present invention, the mining system obtains the log data in the current time period from the network side, specifically: the mining system can obtain the log data in the current time period from the network side by extracting the log data, or can Use web crawler technology to obtain log data in the current time period from the network side, or obtain log data in the current time period from the BOSS tent database on the network side, or accept current data provided by third-party vendors on the network side log data within a time period, or obtain log data within a current time period in combination with at least two of the above methods.

Step 202, aggregate the log data in the current time period to obtain a first set of log data in the current time period.

In the embodiment of the present invention, after the mining system acquires the log data in the current time period, it aggregates the log data in the current time period to obtain the first log data set in the current time period.

Among them, the aggregation in step 202 can be classified according to the content of the log data, and the log data of the same content or content belonging to the same category are accumulated as a piece of data, and the order of magnitude of the first log data set obtained after the aggregation will be far The order of magnitude is far lower than the log data obtained in the current time period, and the meaning of the data at that time is completely preserved.

In the embodiment of the present invention, the mining system acquires the first log data set through the additional steps shown in FIG. The order of magnitude of the log data reduces the storage space required in the Hadoop database and saves storage space.

Preferably, in the embodiment of the present invention, the mining system may also perform the following steps before performing step 202:

Perform data cleaning on the log data in the current time period to obtain the cleaned log data in the current time period;

In the embodiment of the present invention, before the mining system aggregates the acquired log data in the current time period, it may also perform data cleaning on the log data in the current time period to obtain the cleaned log data in the current time period.

And if the mining system executes the above steps, it is also necessary to make adaptive adjustments to step 202, and the adaptive adjustments to step 202 are:

Aggregate the cleaned log data in the current time period to obtain the first log data set in the current time period.

Wherein, cleaning the log data may be removing some log data that does not meet the preset data type, and/or discovering and correcting identifiable errors in the log data, and correcting or deleting identifiable log data.

In the embodiment of the present invention, the mining system can remove some useless or erroneous log data by cleaning the log data in the current time period, reduce the amount of log data processing, and facilitate better data mining.

Please refer to FIG. 3, which is a schematic flow chart of additional steps after step 103 in the first embodiment shown in FIG. 1 of the present invention, including:

Step 301, if the data query instruction is received, read the third log data set corresponding to the query dimension from the Hadoop database according to the query dimension contained in the data query instruction;

In the embodiment of the present invention, after the mining system saves the obtained third log data into the Hadoop database, the user can request to query the data by inputting a data query instruction, and if the mining system receives the data query instruction, it will follow the data query instruction. The query dimension included in the query instruction reads the third log data set corresponding to the dimension from the Hadoop database.

Preferably, the data query instruction may also include a certain time period, and the mining system will read the third log data set corresponding to the query dimension within the time period.

Step 302, perform data analysis on the third log data set, and display the result of the data analysis on the display interface.

In the embodiment of the present invention, the mining system will also perform data analysis on the third log data set, and display the results of the data analysis on the display interface. Specifically: the mining system will analyze the third log data set according to a preset clustering algorithm. The users in the user group are grouped to obtain the user group list; according to the log data of the users in the user group list, the level configuration table corresponding to at least two user dimensions is obtained, and the level configuration table is displayed on the display interface; the user dimension is preset , the level configuration table contains the levels determined by the users in the user group list according to the user dimension.

Among them, the user dimension can be divided into a horizontal dimension and a vertical dimension, and users are rated in different dimensions. For example: the user groups obtained by the mining system include: all user groups and Weibo user groups. For all user groups, all users in the group are ranked according to the amount of traffic used, and the top 20% are five-star users, the top 20% to 40% are four-star users, and by analogy, determine the star rating of each user in all user groups. This is the horizontal dimension rating. For the users in the Weibo user portfolio, rankings are made according to the amount of traffic generated after users start Weibo. The top 20% are five-star users, and the top 20% to 40% are four-star users. By analogy, the star rating of each user in the microblog user group is determined. This is the vertical dimension rating. Through the horizontal dimension rating and the vertical dimension rating, it is possible to realize the portrait display of user groups, so that business experts can get targeted solutions for specific group portraits.

Preferably, the preset clustering algorithm may be K-means algorithm.

Wherein, the query dimension is set based on the dimension corresponding to the third log data set stored in the Hadoop database. For example, the query dimension can be any one or any several of online content, online time, online location, etc.

In the embodiment of the present invention, the mining system reads the third log data set corresponding to the query dimension from the Hadoop database according to the query dimension contained in the data query instruction, and performs data analysis on the third log data set, and The result of data analysis is displayed on the display interface, so that the result of data mining can be effectively displayed to the user.

It should be noted that, in the embodiment of the present invention, the log data mining method based on the Hadoop database can be applied in the precision marketing system of traffic data, for example, through the technical solutions described in the embodiments shown in Figures 1 to 3 Realize the mining of target users and marketing site selection, etc., and provide operators with a data basis for targeted and refined marketing of target users or target base station communities.

Wherein, if it is necessary to determine the target user, then in step 301 in the embodiment shown in FIG. 3 , the query dimension may be Internet content or Internet traffic, and if it is necessary to determine the target base station cell, the query dimension may be Internet location.

In practical applications, users can select query dimensions according to specific needs, which is not limited here.

Referring to Fig. 4, it is a schematic diagram of the functional modules of the Hadoop-based log data mining system in the second embodiment of the present invention, including:

The first saving module 401 is used to save the first log data set in the obtained current time period into the Hadoop database;

Wherein, the mining system obtains the first log data set according to the time period. For example, if the time period is 15 minutes or 30 minutes, the mining system obtains the first log data set within the current 15-minute time period or obtains the current The first log data set in a 30-minute time period.

Wherein, the time period is a cycle for acquiring data, and the length of the time period may be determined according to the amount of data.

Among them, Hadoop can implement Distributed File System (Hadoop Distributed File System, HDFS). The framework core of Hadoop is Hadoop database and parallel operation model. Among them, Hadoop database can provide distributed storage for massive data, and the parallel operation model can provide massive The data provides parallel operation.

Preferably, the parallel operation model is a mapreduce operation model.

Parallel aggregation module 402, for if the number of the first log data set that described Hadoop database has preserved meets preset numerical value, then utilize the preset parallel computing model to carry out the first log data set in described Hadoop database Parallel aggregation processing to obtain a second log data set;

Wherein, in practical applications, this value can be preset according to specific needs. For example, if the above-mentioned time period is 15 minutes, and the first log data set within one hour needs to be aggregated, the preset value is 4. If the above time period is 30 minutes, and the first log data set within 1 day needs to be aggregated, the preset value is 48.

It can be understood that, based on the above aggregation processing, the parallel aggregation module 402 can also use a similar method to obtain log data sets in different time periods, for example: four first log data sets with a time period of 15 minutes can be used to obtain a For the log data collection within an hour, you can use 24 log data collections within an hour to get the log data collection within a day, you can use 30 log data collections within a day to get the log data collection within a month, and so on. Log data collections at different times can be obtained to meet different needs.

The division and storage module 403 divides the log data in the second log data set into dimensions according to the dimensions of the log data in the second log data set, and saves the obtained third log data sets corresponding to different dimensions in the set described Hadoop database.

Among them, there are many dimensions of log data, including but not limited to online content, online location, and online time. Among them, online content refers to the browsing location of the user, and the browsing location can be a specific location, such as Baidu, Sohu, Sina Weibo, etc., can also be a type of website, such as: music, movies, etc. The online location refers to the geographical range of the IP address used by the user, and the online time refers to the time when the log data is generated. And the division of dimensions is based on the requirements of the system, and the further description of the overall behavior of users is completed through the data on the dimensions. It should be noted that for different types of log data, the dimensions of the log data are also different. For example, when the user traffic data in the log data is mined using the technical solution in the embodiment of the present invention, the dimension In addition to the above-mentioned online content, online location, and online time, it can also include online frequency, user age, monthly consumption, etc. Therefore, in practical applications, dimensions can be divided according to specific needs, which is not limited here.

Preferably, in the embodiment of the present invention, after the mining system saves the third log data sets corresponding to different dimensions into the Hadoop database, it can also save the third log data sets corresponding to the different dimensions in the column storage array, It enables the collaborative work of the Hadoop database and the column storage array to meet the data requirements of different application scenarios.

In the embodiment of the present invention, the first saving module 401 saves the acquired first log data collection in the current time period into the Hadoop database, if the number of the first log data collections saved by the Hadoop database satisfies the preset value, then the parallel aggregation module 402 uses the preset parallel operation model to carry out parallel aggregation processing on the first log data set in the Hadoop database to obtain the second log data set, and finally divides and saves the module 403 according to the second log data set Dimensions of the log data in the data set divide the log data in the second log data set into dimensions, and save the obtained third log data sets corresponding to different dimensions into the Hadoop database.

In the embodiment of the present invention, the mining system saves the first log data set obtained in the current time period into the Hadoop database, and if the number of the first log data set saved by the Hadoop database meets the preset value, then use The parallel computing model in the Hadoop database performs parallel aggregation processing on the first log data set in the Hadoop database to obtain a second log data set, and the second log data set is obtained according to the dimension of the log data in the second log data set The log data in is maintained and divided, and the obtained third log data sets corresponding to different dimensions are stored in the Hadoop database to complete the mining of log data. Since the Hadoop database has good distributed storage and parallel computing capabilities, using the Hadoop database to store log data in a distributed manner and using the parallel computing model in Hadoop to perform parallel computing can quickly and effectively realize massive data mining, satisfying Storage and computing requirements for mining massive amounts of data.

Please refer to Fig. 5, which is a schematic diagram of additional functional modules in the second embodiment shown in Fig. 4, including:

An acquisition module 501, configured to acquire log data in the current time period from the network side;

In the embodiment of the present invention, the acquisition module 501 acquires the log data in the current time period from the network side, specifically: the acquisition module 501 can acquire the log data in the current time period from the network side by extracting the log data, or , you can use the web crawler technology to obtain the log data in the current time period from the network side, or you can obtain the log data in the current time period from the BOSS tent database on the network side, or you can accept the third-party vendors on the network side. log data in the current time period, or combine at least two of the above methods to obtain log data in the current time period.

The first aggregation module 502 is configured to perform aggregation processing on the log data within the current time period to obtain a first set of log data within the current time period.

Among them, the first aggregation module 502 can classify according to the content of the log data, and accumulate the log data of the same content or content belonging to the same category as a piece of data, and the order of magnitude of the first log data set obtained after aggregation will be The order of magnitude is far lower than the log data obtained in the current time period, and the meaning of the data at that time is completely preserved.

In the embodiment of the present invention, the mining system will start to execute the first saving module 401 in the embodiment shown in FIG. 4 after executing the first gathering module 502 .

In the embodiment of the present invention, the system further includes a cleaning module 503;

The cleaning module 503 is used to perform data cleaning on the log data in the current time period after the acquisition module 501 acquires the log data in the current time period, to obtain the cleaned log data in the current time period;

And if the mining system executes the cleaning module 503, the first aggregation module 502 is specifically configured to aggregate the cleaned log data in the current time period to obtain the first log data set in the current time period.

In the embodiment of the present invention, the mining system acquires the first log data set through the additional steps shown in FIG. The order of magnitude of the log data reduces the storage space required in the Hadoop database and saves storage space. Moreover, the mining system can also perform data cleaning on the log data in the current time period, so as to remove some useless or erroneous log data, reduce the amount of log data processing, and facilitate better data mining.

Please refer to FIG. 6, which is a schematic diagram of the additional functional modules of the second embodiment shown in FIG. 4, including:

The reading module 601 is used to read the third log data set corresponding to the query dimension from the Hadoop database according to the query dimension included in the data query instruction if a data query instruction is received;

The analysis module 602 is configured to perform data analysis on the third log data set, and display a result of the data analysis on a display interface.

Wherein, the analysis module 602 includes:

A clustering module 603, configured to perform user grouping on the users in the third log data set according to a preset clustering algorithm to obtain a user grouping list;

The acquisition and display module 604 is used to obtain at least two level configuration tables corresponding to user dimensions according to the log data of users in the user grouping list, and display the level configuration tables on the display interface; the user dimensions are preset, The level configuration table includes levels determined according to the user dimension of the users in the user grouping list.

Among them, the user dimension can be divided into a horizontal dimension and a vertical dimension, and users are rated in different dimensions. For example: the user groups obtained by the mining system include: all user groups and Weibo user groups. For all user groups, all users in the group are ranked according to the amount of traffic used, and the top 20% are five-star users, the top 20% to 40% are four-star users, and by analogy, determine the star rating of each user in all user groups. This is the horizontal dimension rating. For the users in the Weibo user portfolio, rankings are made according to the amount of traffic generated after users start Weibo. The top 20% are five-star users, and the top 20% to 40% are four-star users. By analogy, the star rating of each user in the microblog user group is determined. This is the vertical dimension rating. Through the horizontal dimension rating and the vertical dimension rating, it is possible to realize the portrait display of user groups, so that business experts can get targeted solutions for specific group portraits.

Preferably, the preset clustering algorithm may be K-means algorithm.

Wherein, the query dimension is set based on the dimension corresponding to the third log data set stored in the Hadoop database. For example, the query dimension can be any one or any several of online content, online time, online location, etc.

In the embodiment of the present invention, the mining system reads the third log data set corresponding to the query dimension from the Hadoop database according to the query dimension contained in the data query instruction, and performs data analysis on the third log data set, and The result of data analysis is displayed on the display interface, so that the result of data mining can be effectively displayed to the user.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods of various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process conversion made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (10)

1. A log data mining method based on Hadoop, characterized in that, comprising: The first log data set in the current time period obtained is saved in the Hadoop database; If the number of the first log data sets stored in the Hadoop database satisfies the preset value, then utilize the preset parallel operation model to carry out parallel aggregation processing on the first log data sets in the Hadoop database to obtain the second collection of log data; Dimensionally divide the log data in the second log data set according to the dimensions of the log data in the second log data set, and save the obtained third log data sets corresponding to different dimensions into the Hadoop database. 2. The method according to claim 1, characterized in that the method further comprises: Obtain log data in the current time period from the network side; Aggregating the log data within the current time period is performed to obtain a first set of log data within the current time period. 3. The method according to claim 2, further comprising: Perform data cleaning on the log data in the current time period to obtain the cleaned log data in the current time period; Then the step of aggregating the log data in the current time period to obtain the first log data set in the current time period includes: Perform aggregation processing on the cleaned log data in the current time period to obtain a first set of log data in the current time period. 4. The method according to any one of claims 1 to 3, wherein the method further comprises: If a data query instruction is received, the third log data set corresponding to the query dimension is read from the Hadoop database according to the query dimension included in the data query instruction; Perform data analysis on the third log data set, and display the results of the data analysis on the display interface. 5. The method according to claim 4, wherein said performing data analysis on said third log data set, and displaying a result of the data analysis on a display interface comprises: performing user grouping on the users in the third log data set according to a preset clustering algorithm to obtain a user grouping list; Obtain at least two level configuration tables corresponding to user dimensions according to the log data of users in the user grouping list, and display the level configuration tables on the display interface; the user dimensions are preset, and the level configuration tables include The users in the user grouping list are graded and determined according to the user dimension. 6. A log data mining system based on Hadoop, characterized in that, comprising: The first saving module is used to save the first log data collection in the current time period obtained in the Hadoop database; Parallel aggregation module, for if the number of the first log data collection that described Hadoop database has preserved meets preset numerical value, then utilize the preset parallel computing model to carry out parallel to the first log data collection in described Hadoop database Gathering and processing to obtain a second log data set; A partitioning and saving module, configured to divide the log data in the second log data set into dimensions according to the dimensions of the log data in the second log data set, and save the obtained third log data sets corresponding to different dimensions in The Hadoop database. 7. The system according to claim 6, further comprising: The acquisition module is used to acquire log data in the current time period from the network side; The first aggregation module is configured to perform aggregation processing on the log data within the current time period to obtain a first set of log data within the current time period. 8. The system according to claim 7, further comprising a cleaning module; The cleaning module is used to perform data cleaning on the log data in the current time period after the acquisition module acquires the log data in the current time period, so as to obtain the cleaned log data in the current time period; And the first aggregation module is specifically configured to aggregate the cleaned log data in the current time period to obtain the first log data set in the current time period. 9. The system according to any one of claims 6 to 8, wherein the system further comprises: The reading module is used to read the third log data set corresponding to the query dimension from the Hadoop database according to the query dimension contained in the data query instruction if the data query instruction is received; An analysis module, configured to perform data analysis on the third log data set, and display the result of the data analysis on the display interface. 10. The system according to claim 9, wherein the analysis module comprises: A clustering module, configured to perform user grouping on users in the third log data set according to a preset clustering algorithm to obtain a user grouping list; Obtaining a display module, used to obtain at least two level configuration tables corresponding to user dimensions according to the log data of users in the user grouping list, and display the level configuration tables on the display interface; the user dimensions are preset, so The level configuration table includes levels determined according to the user dimension of the users in the user grouping list.