CN105447081A - A cloud platform-oriented government public opinion monitoring method - Google Patents

Abstract

The present invention relates to the field of cloud computing technology, and in particular to a government affairs public opinion monitoring method for a cloud platform. The method of the present invention includes data collection, data preprocessing, data analysis and early warning; the system is mounted on a distributed cluster, and is composed of a crawler server as a master node and multiple crawler clients as slave nodes. The master node is responsible for task allocation, and the child node is responsible for task execution. The master and slave nodes communicate using encrypted heartbeat packets; the slave nodes include data collection, preprocessing, analysis and early warning modules; the collection module captures forum, news, post bar, blog and other data according to user configuration and knowledge base, and automatically filters duplicate data to build a subject database; the data preprocessing module extracts text data based on rules and automatic mixing; the data analysis and early warning module uses machine learning methods to cluster, sentiment analysis, and hot spot analysis on the cleaned text, and warns the analysis results. The present invention solves the problems of users' network public opinion monitoring and can be used for government affairs public opinion monitoring.

Description

A cloud platform-oriented government public opinion monitoring method

technical field

The invention relates to the technical field of cloud computing, in particular to a cloud platform-oriented government public opinion monitoring method.

Background technique

The distributed real-time intelligent monitoring method based on the cloud database integrates Internet information collection technology and information intelligent processing technology, and realizes user network public opinion monitoring and news through automatic capture of massive Internet information, automatic classification and clustering, topic detection, and topic focus Information needs such as topic tracking, form analysis results such as briefings, reports, charts, etc., provide customers with a comprehensive grasp of the ideological trends of the masses, make correct public opinion guidance, and provide analysis basis.

Contents of the invention

The technical problem solved by the present invention is to provide a cloud platform-oriented government public opinion monitoring method.

The technical scheme that the present invention solves the problems of the technologies described above is:

The method includes data collection, data preprocessing, data analysis and early warning; the system is carried on a distributed cluster and consists of a crawler server as a master node and a plurality of crawler clients as slave nodes, the master node Responsible for task allocation, sub-nodes are responsible for task execution, and encrypted heartbeat packets are used for communication between master and slave nodes; slave nodes include data acquisition, preprocessing, analysis and early warning modules; the acquisition module is based on user configuration and knowledge base Grab data from forums, news, post bars, blogs, etc., and automatically filter duplicate data to build a theme database; the data preprocessing module extracts text data based on rules and automatic mixing methods; the data analysis and early warning module uses machine learning methods to clustering, sentiment analysis, hotspot analysis, and early warning of the analysis results.

The communication between the master-slave nodes includes the following steps:

In the first step, the user starts the collection task;

In the second step, the master node saves the task information to the metadata repository;

In the third step, the master node performs task initialization according to the user configuration information;

In the fourth step, the master node allocates tasks according to the CPU, memory, current number of tasks and other indicators of the cluster nodes;

The fifth step is to receive the task from the node;

Step 6, the slave node sends a message of successfully receiving the task to the master node;

The seventh step, the master node writes the task information to the metadata database;

The eighth step is to execute the task from the node;

In the ninth step, if the master node does not receive the heartbeat packet from the slave node for N times, it is considered that the plex node is down and recorded in the log system, and the task is reassigned to other nodes.

The specific processing flow of the acquisition module is:

The first step is to obtain the URL to be collected;

The second step is to filter the URL through the data router;

The third step is to grab the page data;

The fourth step is to perform text extraction and link extraction on the captured data, and add the extracted links to the collection of URLs to be collected;

The fifth step is automatic text feature extraction to generate web page fingerprints;

The sixth step is to detect whether there are identical articles;

The seventh step, if the same article already exists, give up the crawling and return to the first step, otherwise perform word segmentation on the body text;

The eighth step, use the TF_IDF algorithm to extract N keywords;

The ninth step is to find the m articles with the highest degree of overlap;

In the tenth step, if the coincidence degree is greater than c, it is classified into the corresponding subject database;

In the eleventh step, create an inverted index for use by other modules.

The specific processing flow of the data analysis and early warning module is:

The first step is to reconstruct the subject database and select representative data;

The second step is to perform sentiment analysis on each document and calculate the score Tendency ∈ [-1, 1];

The third step is to record the above-mentioned analysis results into the early warning database;

The fourth step is to calculate the warning level, Among them, degree _i represents the popularity of the i-th document, and its calculation formula is:

degree _i ＝(praise _i ×0.3+comment _i ×0.7)/(hour _i +2)

Among them: praise _i represents the number of likes, comment _i represents the number of comments, and hour _i represents the time difference from the posting time to the present;

The fifth step is to give corresponding early warning information such as email or SMS according to the early warning strategy and early warning level.

In the automatic text feature extraction, the steps of generating web page fingerprints are: .

The first step is to extract the keywords of the first sentence of each paragraph of the text (remove the stop words) as the main feature of the article;

The second step is to extract the punctuation marks of each paragraph of the text as sub-features;

The third step is to use SimHash for the main feature and the secondary feature respectively, and then splicing the two feature codes to get the fingerprint of the entire article;

The fourth step is to store in the cache database.

The present invention adopts the distributed multi-threading method to improve the crawling speed, and improves the timeliness of news; through URL deduplication and text similarity algorithm to detect text repetition, thereby saving disk space and improving the crawling speed at the same time ; Improve the speed and accuracy of web page repetition detection through the web page fingerprint algorithm.

Description of drawings

The present invention is further described below in conjunction with accompanying drawing:

Fig. 1 is a framework diagram of the present invention;

Figure 2 is a master-slave node architecture diagram;

Fig. 3 is a heart data capture flow chart;

Figure 4 is a flow chart of data analysis.

detailed description

As shown in Figures 1 to 4, the method of the present invention includes data collection, data preprocessing, data analysis and early warning; the system is carried on a distributed cluster, and consists of a crawler server as a master node and a plurality of crawler servers as slave nodes. Composed of crawler clients, the master node is responsible for task allocation, and the sub-nodes are responsible for task execution. The encrypted heartbeat packets are used for communication between the master and slave nodes; the slave nodes include data collection, preprocessing, analysis and early warning modules; the acquisition module According to user configuration and knowledge base, data such as forums, news, post bars, and blogs are captured, and duplicate data is automatically filtered to build a theme database; the data preprocessing module extracts text data based on rules and automatic mixing; the data analysis and early warning module utilizes The method of machine learning performs clustering, sentiment analysis, and hotspot analysis on the cleaned text, and provides early warning of the analysis results.

As shown in Figure 2: the above-mentioned one master node and multiple slave nodes are composed, the master node is responsible for task distribution, and the child nodes are responsible for task execution, and the encrypted heartbeat packets are used for communication between the master and slave nodes, including the following steps:

In the first step, the user starts the collection task;

In the second step, the master node saves the task information to the metadata repository;

In the third step, the master node performs task initialization according to the user configuration information;

In the fourth step, the master node allocates tasks according to the CPU, memory, current number of tasks and other indicators of the cluster nodes;

The fifth step is to receive the task from the node;

Step 6, the slave node sends a message of successfully receiving the task to the master node;

The seventh step, the master node writes the task information to the metadata database;

The eighth step is to execute the task from the node;

In the ninth step, if the master node does not receive the heartbeat packet from the slave node for N times, it is considered that the plex node is down and recorded in the log system, and the task is reassigned to other nodes.

As shown in Figure 3: the acquisition module captures data such as forums, news, post bars, and blogs according to user configurations and knowledge bases, and filters duplicate data to build a theme database, including the following processes:

The first step is to obtain the URL to be collected;

The second step is to filter the URL through the data router;

The third step is to grab the page data;

The fourth step is to perform text extraction and link extraction on the captured data, and add the extracted links to the collection of URLs to be collected;

The fifth step is automatic text feature extraction to generate web page fingerprints;

The sixth step is to detect whether there are identical articles;

The seventh step, if the same article already exists, give up the crawling and return to the first step, otherwise perform word segmentation on the body text;

The eighth step, use the TF_IDF algorithm to extract N keywords;

The ninth step is to find the m articles with the highest degree of overlap;

In the tenth step, if the coincidence degree is greater than c, it is classified into the corresponding subject database;

In the eleventh step, create an inverted index for use by other modules.

As shown in Figure 4, the data analysis module uses machine learning methods to perform clustering, sentiment analysis, and hotspot analysis on the cleaned text, and provides early warning of the analysis results, including the following steps:

The first step is to reconstruct the subject database and select representative data;

The second step is to perform sentiment analysis on each document and calculate the score Tendency ∈ [-1, 1];

The third step is to record the above-mentioned analysis results into the early warning database;

The fourth step is to calculate the warning level, Among them, degree _i represents the popularity of the i-th document, and its calculation formula is:

degree _i ＝(praise _i ×0.3+comment _i ×0.7)/(hour _i +2)

Among them: praise _i represents the number of likes, comment _i represents the number of comments, and hour _i represents the time difference from the posting time to the present;

The fifth step is to give corresponding early warning information such as email or SMS according to the early warning strategy and early warning level.

As shown in Figure 1, the information obtained by using the method of the present invention can be displayed on the front end of the WEB.

Claims (7)

1. A kind of public opinion real-time monitoring method for government affairs, it is characterized in that: described method comprises data acquisition, data preprocessing, data analysis and early warning; Described system is carried on the distributed cluster, by one as master node The crawler server is composed of multiple crawler clients as slave nodes. The master node is responsible for task allocation, and the sub-nodes are responsible for task execution. The master-slave nodes use encrypted heartbeat packets for communication; the slave nodes include data collection, preprocessing, and analysis. and early warning module; the acquisition module captures data such as forums, news, post bars, and blogs according to user configuration and knowledge base, and automatically filters duplicate data to build a theme database; the data preprocessing module extracts based on rules and automatic mixing Text data; data analysis and early warning module uses machine learning methods to perform clustering, sentiment analysis, and hotspot analysis on the cleaned text, and early warning of the analysis results. 2. a kind of public opinion real-time monitoring method facing government affairs according to claim 1, is characterized in that: the communication between described master-slave node, comprises the steps: In the first step, the user starts the collection task; In the second step, the master node saves the task information to the metadata repository; In the third step, the master node performs task initialization according to the user configuration information; In the fourth step, the master node allocates tasks according to the CPU, memory, current number of tasks and other indicators of the cluster nodes; The fifth step is to receive the task from the node; Step 6, the slave node sends a message of successfully receiving the task to the master node; The seventh step, the master node writes the task information to the metadata database; The eighth step is to execute the task from the node; In the ninth step, if the master node does not receive the heartbeat packet from the slave node for N times, it is considered that the plex node is down and recorded in the log system, and the task is reassigned to other nodes. 3. a kind of public opinion real-time monitoring method facing government affairs according to claim 1, is characterized in that: the specific processing flow of described collection module is: The first step is to obtain the URL to be collected; The second step is to filter the URL through the data router; The third step is to grab the page data; The fourth step is to perform text extraction and link extraction on the captured data, and add the extracted links to the collection of URLs to be collected; The fifth step is automatic text feature extraction to generate web page fingerprints; The sixth step is to detect whether there are identical articles; The seventh step, if the same article already exists, give up the crawling and return to the first step, otherwise perform word segmentation on the body text; The eighth step, use the TF_IDF algorithm to extract N keywords; The ninth step is to find the m articles with the highest degree of overlap; In the tenth step, if the coincidence degree is greater than c, it is classified into the corresponding subject database; In the eleventh step, create an inverted index for use by other modules. 4. a kind of public opinion real-time monitoring method facing government affairs according to claim 2, is characterized in that: the specific processing flow of described collection module is: The first step is to obtain the URL to be collected; The second step is to filter the URL through the data router; The third step is to grab the page data; The fourth step is to perform text extraction and link extraction on the captured data, and add the extracted links to the collection of URLs to be collected; The fifth step is automatic text feature extraction to generate web page fingerprints; The sixth step is to detect whether there are identical articles; The seventh step, if the same article already exists, give up the crawling and return to the first step, otherwise perform word segmentation on the body text; The eighth step, use the TF_IDF algorithm to extract N keywords; The ninth step is to find the m articles with the highest degree of overlap; In the tenth step, if the coincidence degree is greater than c, it is classified into the corresponding subject database; In the eleventh step, create an inverted index for use by other modules. 5. A method for real-time monitoring of public opinion facing government affairs according to any one of claims 1 to 4, characterized in that: the specific processing flow of the data analysis and early warning module is: The first step is to reconstruct the subject database and select representative data; The second step is to perform sentiment analysis on each document and calculate the score Tendency∈[-1,1]; The third step is to record the above-mentioned analysis results into the early warning database; The fourth step is to calculate the warning level, Among them, degree _i represents the popularity of the i-th document, and its calculation formula is: degree _i ＝(praise _i ×0.3+comment _i ×0.7)/(hour _i +2) Among them: praise _i represents the number of likes, comment _i represents the number of comments, and hour _i represents the time difference from the posting time to the present; The fifth step is to give corresponding early warning information such as email or SMS according to the early warning strategy and early warning level. 6. according to a kind of public opinion real-time monitoring method facing government affairs described in claim 3 or 4, it is characterized in that: described automatic text feature extraction, the step of generating webpage fingerprint is: The first step is to extract the keywords of the first sentence of each paragraph of the text (remove the stop words) as the main feature of the article; The second step is to extract the punctuation marks of each paragraph of the text as sub-features; The third step is to use SimHash for the main feature and the secondary feature respectively, and then splicing the two feature codes to get the fingerprint of the entire article; The fourth step is to store in the cache database. 7. a kind of public opinion real-time monitoring method facing government affairs according to claim 5, is characterized in that: described automatic text feature extraction, the step of generating webpage fingerprint is: The first step is to extract the keywords of the first sentence of each paragraph of the text (remove the stop words) as the main feature of the article; The second step is to extract the punctuation marks of each paragraph of the text as sub-features; The third step is to use SimHash for the main feature and the secondary feature respectively, and then splicing the two feature codes to get the fingerprint of the entire article; The fourth step is to store in the cache database.