CN103345514B - Streaming data processing method under big data environment - Google Patents

Abstract

The present invention discloses the streaming data processing method under a kind of big data environment, relate generally to the improvement of MapReduce computation module, specifically comprise: the localized irredundant of data is deposited and treatment mechanism, allow each computing node only store and process corresponding interval in data; Dispatch Map and Reduce related linear program in pipelined fashion with speed up processing; The internal memory of intermediate result deposits mechanism, in order to ensure data localization and effective enforcement of streamline, it is provided that at a high speed, memory access patterns easily. By above three modules, ensure data to flow to reliability and the high efficiency of row relax, meets the demand of data processing in practical application under big data environment.

Description

Streaming data processing method in big data environment

technical field

The present invention relates to cloud computing, mainly relates to big data processing, streaming data processing, and specifically to a streaming data processing method in a big data environment in practical applications.

Background technique

In recent years, with the rapid development of network applications such as portal websites, social networks, and e-commerce, as well as the continuous growth and extension of business, a large amount of business data has been generated and accumulated. These data have a large amount of data and diverse data structures. The characteristics of high data growth rate are typical big data.

On the other hand, users continuously access and use these network applications to obtain required services, forming a series of real-time streaming data. In order to meet the real-time service requirements of users, network applications not only need to analyze and process a large amount of historical data, but also need to further quickly process real-time streaming data. Such an application scenario of fast processing of streaming data based on big data is a typical big data service.

Compared with general data services, this type of big data service has special properties: first, the business data is big data, and the newly arrived streaming data is small in scale and simple in structure; second, the data flow continues to arrive, business data continues to grow, and Update; Finally, fast processing of streaming data must be done on top of big data to provide efficient services.

For example, in the e-commerce real-time recommendation system, there are a large number of commodity information stored in the system, and the user's registration, search, collection, purchase and other record information are called historical data; at the same time, with the real-time access of a large number of users, a Continuously arriving real-time service request data. In order to achieve real-time recommendations for users, not only historical data needs to be analyzed, but also real-time streaming data needs to be processed quickly. Only the combination of the two can achieve effective real-time recommendations. Therefore, it is necessary to study the real-time streaming data processing technology in the big data environment to provide support for such big data application services.

In order to realize the processing of big data, Google first proposed the MapReduce batch processing method. This type of data processing method based on the MapReduce framework is mainly for batch data processing and does not support the processing of streaming data.

In terms of streaming data processing, it is biased towards specific application environments. For example, S4 (SimpleScalableStreamingSystem) is a distributed streaming data processing system inspired by MapReduce, which is mainly used to solve practical applications such as search, error detection, and online dating. In order to avoid the complexity of the system, S4 is only for streaming data processing.

At present, there is a lack of effective methods for streaming data processing in the big data environment.

Contents of the invention

Technical problem: It can be seen from the above that there is currently a lack of an effective streaming data processing system in a big data environment. The present invention is based on MapReduce to build an efficient and real-time streaming data processing framework, which mainly includes three modules: data localization without redundancy The storage and processing mechanism schedules Map and Reduce related threads in a pipelined manner. Memory storage mechanism for data/intermediate results.

Technical solution: a streaming data processing method in a big data environment of the present invention comprises the following steps:

1): Process the accumulated big data, that is, historical data to generate an intermediate result set, divide the result set and distribute the cache to each computing node;

2): Each computing node regularly receives all streaming data, and obtains intermediate results through Map processing;

3): The intermediate results of the node are filtered through the intermediate result division method, cached on the local node, and a fragment is formed after reaching the threshold of 10,000, and the fragment is sent;

4): When the intermediate result fragments arrive, according to the pipeline scheduling algorithm, the historical data intermediate results and the intermediate result fragments are used as Reduce input;

5): Output calculation results, which are partial outputs of a task in different periods, and merge all these results into the same file to form the final output result.

The accumulated big data in step 1), that is, historical data, is backed up on the distributed file system. Before the system starts or starts computing tasks, it is necessary to read this part of the data for preprocessing and distribute them Stored in each computing node, ready for the following calculations.

In step 2), each computing node regularly receives all streaming data, all computing nodes cache this part of data, and generate intermediate results through processing; at the same time, according to the specific historical data backup method, this part of streaming data needs to be periodically Update to historical dataset.

In the intermediate result division method in step 3), each node corresponds to a certain range of key values, and each node only processes data within this range, that is, for all stream data that has been accepted and has generated intermediate results, filter Output the intermediate results within the interval; and, when the specified threshold is reached, an intermediate result fragment is formed and the data fragment is sent.

Step 4) After the intermediate result fragments arrive, combine the intermediate result groups of step 1) as the input data of the next Reduce task; at the same time, since a computing task generally generates multiple fragments in step 3), it is necessary to A thread is dispatched for each such Reduce task to perform calculations asynchronously.

The step 5) outputs the calculation result, which is the calculation result generated by performing a Reduce calculation; after all the Reduce tasks are completed, these Reduce calculation results are combined to output a final result file.

Beneficial effect: the effectiveness of the present invention is:

By adding three modules to the existing hadoop platform to support fast processing of streaming data in a big data environment, it can support real-time recommendation of such applications in e-commerce,

Compared with the prior art, the invention has the following advantages:

1. The present invention provides a reference for data processing methods between two types of applications by integrating static big data processing technology and real-time streaming data processing technology;

2. The data set is divided by the probability model, which makes the system load tend to be balanced, effectively increases the throughput of the system, and provides a new idea for optimizing the parallel processing of big data;

3. Through pipeline, the data is processed in batches, the granularity of data processing is refined, the calculation speed is accelerated, and it meets the task requirements of high response ratio;

4. By adding memory management, it provides support for the effective operation of the above two modules, and enhances the adaptability of the system to data of different scales.

Description of drawings

Figure 1 is a processing flow chart,

Figure 2 is a system architecture diagram,

Figure 3 is a memory structure diagram.

detailed description

The streaming data processing in the big data environment is based on the existing Hadoop—a framework that implements MapReduce—and adds three modules to the existing functions to support the processing of streaming data. It is especially suitable for real-time streaming data processing based on a large amount of complex historical data, including data localization module, pipeline scheduling module and memory management module. The specific implementation methods are as follows:

In the data localization module, the key value interval is generally divided by the Hash function, so that each node can store data without redundancy. In order to ensure the balance of data division, the method based on probability statistics is used to divide the data, so that the data basically obeys the uniform distribution. Perform the following steps:

Step 1: Randomly collect some historical data or streaming data as samples, sort by keywords, if the keywords are strings, sort by their encoded values;

Step 2: count the frequency of occurrence of all keywords after sorting;

Step 3: According to the calculation of the number of nodes N, the ideal load factor of each node is obtained, which is generally the reciprocal 1/N of the number of nodes;

Step 4: According to the keyword frequency, continuously distribute keywords from the keyword list to N collections, try to make the frequency sum of each collection closest to the load factor;

Step 5: Each computing node receives keywords in a set to form its key range (set).

Step 6: Each node only receives or processes data within the key range.

In the pipeline scheduling module, the calculation speed is accelerated by distributing intermediate data (map output) and calculation results (reduce output) asynchronously, and this asynchronous process needs to be controlled according to the system load. Here, the distribution speed and the distribution of computing tasks are adjusted by monitoring the system runtime parameters. It mainly includes the following steps:

Step 1: Monitor the number of intermediate result fragments of the streaming data, that is, the number of intermediate result caches after the streaming data is processed by map each time;

Step 2: Monitor the number of intermediate results that have been distributed, that is, the number of intermediate results that have not been processed in the reduce stage and are in the cache;

Step 3: Count the number of map threads allocated by the system, the number of reduce threads, and the maximum number of threads allocated by the platform to the system;

After obtaining these parameters, the system adjusts the execution speed of each task through the following steps to maximize the task execution speed while ensuring that resources do not overflow:

Step 1: In the case of a limited number of threads allocated by the system, if the arrival speed of the data flow is small, so that the map task is light, you can follow step 2, otherwise go to step 3;

Step 2: If the consumption of intermediate results in the reduce stage is faster than the production speed of the map, reduce the map intermediate result cache; otherwise, increase the reduce stage cache, or increase the number of reduce threads appropriately;

Step 3: Increase the number of map threads as much as possible to ensure that data is not lost, and increase the map cache;

Step 4: Go back to Step 1.

Generally, when completing light jobs, the system load is light and the number of threads is relatively sufficient. These caches and map and reduce task distribution restrictions can be canceled to enable the system to achieve fast response data processing.

In the memory management module, the storage space is expanded by combining the memory and external memory (mainly disk), to ensure the scalability of the intermediate data cache, and to ensure the fast search and read of the data. This is mainly aimed at the situation where the system task is heavy and the amount of data is large, and it mainly includes the following steps:

Step 1: Create an intermediate result index area through the keyword Hash index, and store it in memory, and the hash pointer points to an information header structure;

Step 2: Create an intermediate result buffer in the memory, which can be configured according to the size of the memory; the information header in step 1 searches for data in the buffer and returns the result;

Step 3: When the intermediate result is large and the memory space is insufficient, create a storage area in the external memory. The external storage area also has a retrieval area and a data area, but no cache area;

Step 4: When the information header cannot find the corresponding data in the cache area, go to the external storage area to search.

In the actual system, the intermediate results generated are generally key-value pairs. If there is still space in the cache space, directly store the intermediate result, update the entry in the retrieval area, and allocate the information header; otherwise, store the data in the external storage, and the information header points to the external storage.

In general applications, the historical data set is relatively large, and it can be considered to store historical data in external storage and establish an internal and external storage swap area; for real-time data streams, data is stored in memory, and these specific technologies already have ready-made algorithms and data structures.

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

The basic processing flow of the present invention is shown in FIG. 1 , and the present invention has an obvious hierarchical structure, as shown in FIG. 2 . There are mainly three new modules: localization, pipeline and memory management. Memory management mainly serves the other two modules to ensure the reliability of its data storage. By adding these three modules under the Hadoop framework, it supports big data On the basis of processing, it provides support for data flow, and the specific implementation method is as follows:

In the data localization module, the generally processed data are <key, value> key-value pairs, which are divided into value intervals by the bytecode of the key, and the probability statistics model is used specifically: samples are randomly selected from historical data and part of the data stream , analyze the occurrence frequency of the key-value pair key, divide the interval so that the key value is basically evenly distributed; after the division is completed, the node N _i corresponds to the key-value interval S _i , assuming that the number of nodes is N, the interval division algorithm is as follows:

After the interval is divided, each node only processes <key, value> in the corresponding interval. In this case, when node k processes the item <key(x), value(y)> in the data stream, it can judge key( x) Determine whether to filter the data item in the interval array[k].

In the pipeline scheduling module, it is first necessary to collect relevant job runtime parameter information, and then judge the system health status according to these parameters, and then change the job running status by controlling these parameters.

The information to be collected is:

V _in (split/s) average arrival rate of streaming data,

V _mc (split/s) map thread average consumption rate; V _mp (piece/s) map thread average production rate,

V _rc (piece/s) reduce thread average consumption rate,

The number of threads assigned by _TN jobs; the number of _TM map threads; the number of T _R reduce threads.

First judge whether T _N is greater than the number of threads required by general jobs. If it is greater, it means that the system resources are abundant. After the map generates an intermediate result to form a fragment, it can be directly sent to the reduce stage for processing; otherwise, the map needs to be controlled according to the situation. Or the number of reduce threads and the cache of intermediate results, the specific algorithm is as follows:

In the memory management module, by reorganizing the storage and access methods of the intermediate results, the high reliability and transparency of the storage of the intermediate results are ensured. As shown in Figure 3, firstly, the intermediate results are quickly addressed through the hash tree table, and the After the information header, you can directly access the memory data or external storage data through the information header. The information header and the hash table are stored in the memory, and the intermediate data is partially stored in the internal memory and partially stored in the external storage. The pages of the internal and external storage (key-value pair records ) exchange can be done by least recently used algorithm LRU algorithm.

Claims (6)

1. A streaming data processing method under a big data environment, characterized in that the method comprises the following steps: 1): Process the accumulated big data, that is, historical data to generate an intermediate result set, divide the result set and distribute the cache to each computing node; 2): Each computing node regularly receives all streaming data, and obtains intermediate results through Map processing; 3): The intermediate results of the node are filtered through the intermediate result division method, cached on the local node, and a fragment is formed after reaching the threshold of 10,000, and the fragment is sent; 4): When the intermediate result fragments arrive, according to the pipeline scheduling algorithm, the historical data intermediate results and the intermediate result fragments are used as Reduce input; 5): Output calculation results, which are partial outputs of a task in different periods, and merge all these results into the same file to form the final output result. 2. the streaming data processing method under the big data environment according to claim 1, is characterized in that: the accumulated big data in the described step 1), i.e. historical data, is all backed up on the distributed file system, in Before the system starts or starts computing tasks, it needs to read this part of the data for preprocessing, and store them distributedly to each computing node to prepare for the following calculations. 3. The streaming data processing method under the big data environment according to claim 1, characterized in that: in the step 2), each computing node regularly accepts all streaming data, and all computing nodes cache all the streaming data The streaming data is processed to generate intermediate results; at the same time, according to the specific historical data backup method, all these streaming data need to be regularly updated to the historical data set. 4. the streaming data processing method under the big data environment according to claim 1, is characterized in that: in the intermediate result division method in described step 3), each node corresponds to a certain key value interval, each The node only processes the data in this range, that is, for all the streaming data that has been accepted and has generated intermediate results, filter out the intermediate results in this range; and, when the specified threshold is reached, an intermediate result fragment is formed and the intermediate result is sent The result is sliced. 5. The streaming data processing method under the big data environment according to claim 1, characterized in that: after the arrival of the intermediate result fragments in the step 4), the intermediate result grouping in combination with the step 1) is used as the next step of the Reduce task At the same time, since a calculation task generally generates multiple fragments in step 3), it is necessary to dispatch a thread for each such Reduce task to perform calculations asynchronously. 6. the streaming data processing method under the big data environment according to claim 1, is characterized in that: the output calculation result in described step 5), this result is the calculation result that carries out a Reduce calculation to produce; When all After the Reduce task is completed, these Reduce calculation results are combined to output a final result file.