CN112667614A - A data processing method, device and computer equipment - Google Patents

Abstract

The embodiment of the invention discloses a data processing method, a data processing device and computer equipment. The method comprises the following steps: acquiring data to be processed, and adding the data to be processed to a first data message queue; processing each data to be processed in the first data message queue by adopting a streaming data processing mode based on a Flink real-time processing frame to obtain queue processing data; adding the queue processing data to a second data message queue; and performing real-time data processing on each queue processing data in the second data message queue based on the Flink real-time processing framework. The technical scheme can ensure the consistency and the real-time performance of the data processing process.

Description

A data processing method, device and computer equipment

technical field

Embodiments of the present invention relate to the technical field of data processing, and in particular, to a data processing method, apparatus, and computer equipment.

Background technique

With the rapid development of the Internet, there are more and more diverse data, and these data are often real-time. When processing big data, it is necessary to rely on technologies such as distributed processing or distributed database, and ensuring data consistency and real-time in the process of data processing is always an important issue in data processing.

At present, in the field of data processing, there are generally two types of tasks: batch computing and real-time stream computing. Flink is an open source data platform for both distributed real-time stream processing and batch data processing. It can support both stream processing and batch processing when running on the same Flink real-time processing framework. When ensuring data consistency in a real-time processing system, it is usually necessary to perform idempotent write operations or transactional write operations on the data. Among them, the idempotent write operation only affects the target system once when data is written to a system any number of times, but this operation requires the data to be idempotent. The transactional write operation combines Flink's consistency checkpoint Checkpoint mechanism to ensure that it only affects the external output once, but only the data confirmed by the Checkpoint can be written to the outside. Reduce the real-time nature of data. Therefore, how to keep data consistent and real-time during processing based on the Flink real-time processing framework is an urgent problem to be solved.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a data processing method, apparatus, and computer equipment to ensure the consistency and real-time performance of the data processing process.

In a first aspect, an embodiment of the present invention provides a data processing method, including:

obtaining data to be processed, and adding the data to be processed to the first data message queue;

Based on the Flink real-time processing framework, the data to be processed in the first data message queue is processed by a streaming data processing method, and the queue processing data is obtained;

adding the queue processing data to the second data message queue;

Real-time data processing is performed on each of the queue processing data in the second data message queue based on the Flink real-time processing framework.

In a second aspect, an embodiment of the present invention further provides a data processing device, including:

a first data message queue generation module, configured to obtain the data to be processed, and add the to-be-processed data to the first data message queue;

The queue processing data generation module is configured to process each of the to-be-processed data in the first data message queue based on the Flink real-time processing framework using a streaming data processing method to obtain queue processing data;

The second data message queue generation module is configured to add the queue processing data to the second data message queue;

The real-time data processing module is configured to perform real-time data processing on each of the queue processing data in the second data message queue based on the Flink real-time processing framework.

In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implementing the program as described in the present invention when the processor executes the program The data processing method described in any embodiment.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the data processing method according to any embodiment of the present invention.

In the technical solution provided by the embodiment of the present invention, after the acquired data to be processed is added to the first data message queue, each to-be-processed data in the first data message queue is processed based on the Flink real-time processing framework and the streaming data processing method is adopted. The data is processed to obtain the queue processing data, and then the queue processing data is added to the second data message queue, so that real-time data processing can be performed on the queue processing data in the second data message queue based on the Flink real-time processing framework. The real-time processing framework uses two data message queues to process data in real time, with no special requirements for data, which solves the problem that the existing data processing process is difficult to effectively ensure consistency and real-time performance, and ensures the consistency and real-time performance of the data processing process. sex.

Description of drawings

1 is a schematic flowchart of a data processing method in Embodiment 1 of the present invention;

2 is a schematic flowchart of a data processing method in Embodiment 2 of the present invention;

3 is a schematic flowchart of a data breakpoint resuming transmission in Embodiment 2 of the present invention;

4 is a schematic flowchart of a data cleaning and processing in Embodiment 2 of the present invention;

5 is a schematic structural diagram of a data processing apparatus in Embodiment 3 of the present invention;

FIG. 6 is a schematic diagram of a hardware structure of a computer device in Embodiment 4 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all structures related to the present invention.

Before discussing the exemplary embodiments in greater detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts various operations (or steps) as a sequential process, many of the operations may be performed in parallel, concurrently, or concurrently. Additionally, the order of operations can be rearranged. The process may be terminated when its operation is complete, but may also have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, subroutines, and the like.

Example 1

1 is a flowchart of a data processing method provided in Embodiment 1 of the present invention. The embodiment of the present invention is applicable to processing any type of data based on the Flink real-time processing framework to ensure the consistency and real-time performance of the data processing process. In some cases, the method may be executed by the data processing apparatus provided in the embodiment of the present invention, and the apparatus may be implemented in the form of software and/or hardware, and may generally be integrated in computer equipment.

As shown in Figure 1, the data processing method provided by this embodiment specifically includes:

S110. Acquire data to be processed, and add the data to be processed to a first data message queue.

The data to be processed may be log data of various sources, types and formats, such as buried point data, log files, or external data, and so on. That is, the data to be processed in this embodiment of the present invention may be any type of data, and it is not required that it must satisfy idempotency.

In this embodiment of the present invention, the data to be processed may be acquired by collecting data by a data collector. The data collector may apply any technology with a data collection function, which is not specifically limited in this embodiment. Optionally, a data collector using Flume technology can collect a large amount of various data to be processed in a centralized manner. The collection, aggregation and transmission of massive logs and other data to ensure the stability and fault tolerance of the system.

A data message queue is a sequence of messages that can write data and read data from it. Optionally, the first data message queue can be a distributed Kafka message queue, which writes the data to be processed into the Kafka message queue. When processing the data, the data can be read from the Kafka message queue to avoid obtaining The problem that the speed of data to be processed is inconsistent with the speed of processing data further ensures the stability of the system.

After acquiring the to-be-processed data, the acquired to-be-processed data is added to the first data message queue for subsequent processing of the to-be-processed data in the first data message queue.

S120. Based on the Flink real-time processing framework, the data to be processed in the first data message queue is processed in a streaming data processing manner to obtain queue processing data.

The Flink real-time processing framework is an open-source data platform that can simultaneously face distributed real-time stream processing and batch data processing. When running the same Flink real-time processing framework, it can simultaneously provide functions that support both stream processing and batch processing applications.

Streaming data processing refers to processing one or more event streams in real time, that is, processing a piece of data in real time after each piece of data is read, ensuring the real-time nature of the data.

Queue processing data refers to the data obtained by processing the data to be processed in the first data message queue based on the Flink real-time processing framework and using the streaming data processing method.

Based on the Flink real-time processing framework and adopting the streaming data processing method to process the data to be processed in the first data message queue, the queue processing data can be obtained, which ensures the consistency and real-time performance of the data. Wherein, when each data to be processed is processed, an external service can be invoked to assist in the processing of the data. For example, when cleaning data, it is necessary to convert the latitude and longitude in the acquired data to be processed into a specific location. At this time, an external service (such as an electronic map) can be called to accurately locate and process the data in the map. The specific location corresponding to the latitude and longitude in . The so-called cleaning process is also the process of converting the data to be processed into directional data. For example, after the latitude and longitude data is cleaned and processed, it can be converted into specific location information.

Optionally, before processing each to-be-processed data in the first data message queue based on the Flink real-time processing framework, it may further include: reading each to-be-processed data in the first data message queue; storing each to-be-processed data in a distributed format file system.

Distributed file system refers to a file system for storing and managing data. The data stored in it can communicate and transmit data between nodes through a computer network, where nodes can be distributed at any position in the data communication link. Storing each data to be processed in a distributed file system can extend the data to be processed to the entire network, thereby enhancing the scalability, stability and execution efficiency of the system.

Before processing the data to be processed in the first data message queue based on the Flink real-time processing framework, read the data to be processed in the first data message queue, store the data to be processed in the distributed file system, and retain the obtained data. The raw data that has not been processed in any way ensures the originality and integrity of the data, and the retained raw data can be regularly compared with the real-time streaming data and batch data processed by the Flink real-time processing framework to verify the data processing results. accuracy.

S130. Add the queue processing data to the second data message queue.

Optionally, the second data message queue may be of the same type as the first data message queue, and is also a distributed Kafka message queue. The queue processing data obtained based on the Flink real-time processing framework and processed by the streaming data processing method is added to the second data message queue for subsequent application processing, such as real-time streaming data analysis and calculation, batch data analysis and calculation, etc. At this time, the data in the second data message queue is the data obtained by processing the obtained data to be processed.

S140. Perform real-time data processing on each queue processing data in the second data message queue based on the Flink real-time processing framework.

Real-time data processing refers to the analysis and calculation of data or data volume that requires real-time characteristics, for example, statistics of the unique visitor (UV) data volume of the website at the current moment, and statistics of the page views (Page View) at the current moment. , PV), statistics of the total number of current visits and other real-time indicators.

After the queue processing data is added to the second data message queue, real-time data processing is performed on each queue processing data in the second data message queue based on the Flink real-time processing framework, and corresponding real-time processing results can be obtained.

In the process of data processing, two message queues, the first data message queue and the second data message queue, are used, which not only avoids the problem of inconsistency between the speed of acquiring the data to be processed and the speed of processing the data, but also enables the It is ensured that the system processes each piece of data only once, and the data will not be repeatedly processed or processed multiple times, so that a piece of data can only affect the final result after one processing, ensuring data consistency.

As an optional implementation manner, after performing real-time data processing on the processing data of each queue in the second data message queue, it may further include:

Read the processing data of each queue from the second data message queue and store it in a static database; the static database is used to perform batch data processing on the processing data of each queue;

Store real-time data processing results and batch data processing results in a results database; or

The processing data of each queue is read from the second data message queue, and sent to the target application through a preset interface.

A static database refers to a database that can store offline data and other non-real-time data. Optionally, the static database can be a Hive data warehouse, which can be used to process, query, and analyze data. Due to the limited amount of data that the data message queue can carry, if the amount of data in the current data message queue has reached the maximum value, when data is written again, the first existing data in the current data message queue will be automatically deleted and cannot be retained. Complete data, and further in the batch data processing of each queue processing data, due to the lack of data, the accuracy of batch data processing is reduced, so storing each queue processing data in the second data message queue to the static database can be The integrity of the processing data of each queue in the second data message queue is preserved, the batch data processing of the processing data of each queue is realized, the accuracy of batch data processing is improved, and it can also be used for other data analysis and calculation in the future.

Batch data processing refers to analyzing, calculating, and other processing of data or data volume within a certain period of time. For example, data can be processed based on each queue in the second data message queue stored in the static database, and statistics can be made within a specific period of time. Batch data such as the total page views of the webpage and the total daily transaction volume in the past month.

The result database refers to a database that can store real-time data processing results and batch data processing results so as to use the real-time data processing results and batch data processing results as caches. For example, the result database can be a mysql database, etc. When it is necessary to re-access a real-time data processing result or batch data processing result obtained from previous processing, the corresponding processing result can be directly read in the result database, without re-processing real-time data processing or batch data processing, so that the data result has a reproducible value. Reusability reduces the repeated processing of data and improves the efficiency of data processing.

After the real-time data processing is performed on the processing data of each queue in the second data message queue, the processing data of each queue can also be read from the second data message queue, and the processing data of each queue can be stored to the data that can be used for processing each queue. A static database for batch data processing, so that the real-time data processing results and batch data processing results can be stored in the result database as a cache, so that the real-time data processing results and batch data processing results can be directly read in the subsequent data processing process, avoiding It realizes the decoupling of the two functions of real-time stream processing and batch data processing in the Flink real-time processing framework.

The preset interface refers to a preset interface that can transmit the processing data of each queue in the second data message queue to a third-party application program. The preset interface may be an application programming interface (Application Programming Interface, API). When a third-party application needs to obtain processing data of each queue in the second data message queue, it can process data through each queue in the second data message queue through a preset interface.

The target application refers to a third-party application that needs to obtain the processing data of each queue in the second data message queue. For example, if a map APP needs to obtain processed real-time data, the map APP is the target application .

After real-time data processing is performed on the processing data of each queue in the second data message queue, the processing data of each queue can also be read from the second data message queue and sent to the target application through a preset interface, realizing Flink real-time data processing. The architecture extension of the processing framework satisfies the data requirements corresponding to different applications and improves the flexibility of the architecture.

According to the technical solution provided by the embodiment of the present invention, after the acquired data to be processed is added to the first data message queue, each data to be processed in the first data message queue is processed based on the Flink real-time processing framework and the streaming data processing method is adopted. Perform processing to obtain the queue processing data, and then add the queue processing data to the second data message queue, so that real-time data processing can be performed on the queue processing data in the second data message queue based on the Flink real-time processing framework. Real-time data processing based on Flink The processing framework uses two data message queues to process data in real time, with no special requirements for data, which solves the problem that the existing data processing process is difficult to effectively ensure consistency and real-time performance, and ensures the consistency and real-time performance of the data processing process. .

Embodiment 2

FIG. 2 is a flowchart of a data processing method according to Embodiment 2 of the present invention. This embodiment is embodied on the basis of the above-mentioned embodiment, wherein the data to be processed can be added to the first data message queue, specifically:

Determine the first target partition sequence number and the first target partition position corresponding to the data to be processed;

The data to be processed is added to the first target data partition in the first data message queue according to the sequence number of the first target partition and the position of the first target partition.

Further, using the streaming data processing method based on the Flink real-time processing framework to process the data to be processed in the first data message queue may include:

Determine the current partition serial number and data processing progress identifier corresponding to the current data partition;

Determine the current data to be processed according to the current partition serial number and the data processing progress identifier;

Based on the Flink real-time processing framework, the current data to be processed is processed in real time.

As shown in Figure 2, a data processing method provided by this embodiment specifically includes:

S210: Acquire the data to be processed, and determine the sequence number of the first target partition and the position of the first target partition corresponding to the data to be processed.

S220. Add the data to be processed to the first target data partition in the first data message queue according to the sequence number of the first target partition and the position of the first target partition.

The sequence number of the target partition refers to distributing all data to different data partitions, and the sequence number corresponding to each data partition, for example, partition 0 (job0), partition 1 (job1), and so on. When processing big data, since big data belongs to distributed data, data sets can be distributed to different data partitions by data partitioning, so that data can be processed or queried in multiple data partitions at the same time, which improves data management. and query efficiency. The data partition may be to distribute data to different areas of the same disk, or to distribute data to different disks or machines, which is not specifically limited in the present invention. The sequence number of the first target partition may be the sequence number of the first target data partition where the current data to be processed is located in the first data message queue. The first target data partition, that is, the current data to be processed is added to the corresponding partition in the first data message queue.

The target partition location refers to the specific storage location of each piece of data in the target data partition corresponding to the serial number. The data can be placed at a specific target partition location corresponding to the target data partition, or the corresponding data can be queried according to the target partition location. According to the location of the target partition corresponding to the data, the specific location of the data in the target data partition in the data message queue can be determined. The target partition position may be represented by an offset (offset). The first target partition location may be a specific queue location of the current data to be processed in the first target data partition in the first data message queue.

After acquiring the data to be processed, it is necessary to determine the first target partition serial number and the first target partition position corresponding to the to-be-processed data, and then add the to-be-processed data to the first data message queue according to the first target partition serial number and the first target partition position The first target data partition in .

S230 , processing each data to be processed in the first data message queue by adopting a streaming data processing method based on the Flink real-time processing framework.

It is worth pointing out that in data processing, the system may often fail due to various unexpected factors, such as traffic surges, network jitters, problems with cloud service resource allocation, and system pressures that lead to downtime or crashes. In this case, the Flink real-time processing framework will restart the job. In order to ensure the consistency of the data processing process, that is, the data processing results obtained after the failure is successfully handled and recovered are compared with the processing results obtained without any failure. The processing results are correct, that is to say, the occurrence of system failure will not affect the obtained data processing results. Usually, Flink's Checkpoint (consistency checkpoint) mechanism is used to ensure that only one impact on external output occurs, but Checkpoint There is a certain time interval between, which will reduce the real-time performance of the data. Therefore, when the Flink real-time processing framework normally starts to process data or restarts the Flink real-time processing framework due to a system failure, in order to ensure the consistency and real-time performance of the data processing process at the same time, and to realize the continuous transmission of data at breakpoints, the embodiment of the present invention adopts A closed-loop approach to data processing.

Wherein, S230 may specifically include the following operations S231-S233:

S231. Determine the current partition serial number and the first data processing progress identifier corresponding to the current data partition.

The current data partition may be the data partition where the to-be-processed data currently being processed is located.

Data processing progress identifier, which refers to a flag that can be used to indicate the progress of data processing. In this embodiment of the present invention, the first data processing identifier may be represented by the position of the last partition in the first data message queue, and is used to identify the data processing progress of the first data message queue. The position of the last partition in the first data message queue refers to the last data to be processed based on the Flink real-time processing framework to complete real-time processing before the Flink real-time processing framework starts normally or before the system fails to restart the Flink real-time processing framework. The position of the first target partition corresponding to the currently processed data partition in the data message queue.

The current partition serial number and the data processing progress identifier corresponding to the current data partition are determined, so as to be used for determining the current data to be processed.

S232: Determine the current data to be processed according to the current partition serial number and the first data processing progress identifier.

According to the current partition serial number and the first data processing progress identifier, the current pending data in the currently processed data partition is determined in the first data message queue, so as to perform real-time processing on the current pending data based on the Flink real-time processing framework.

S233 , process the current data to be processed in real time based on the Flink real-time processing framework, and obtain queue processing data.

Based on the Flink real-time processing framework, the current to-be-processed data in the first data message queue is processed in real time to obtain queue-processed data, in which the current to-be-processed data can be processed by using a stream data processing method.

Optionally, processing the current data to be processed in real time based on the Flink real-time processing framework may include: converting the current data to be processed in real time into target directional data based on the Flink real-time processing framework.

Target-oriented data refers to data containing certain information that can be intuitively understood by humans, for example, information about a specific location or a certain product.

Exemplarily, when the current to-be-processed data is processed in real time based on the Flink real-time processing framework, the current to-be-processed data may be cleaned to convert the current to-be-processed data into target directional data in real time. For example, the current data to be processed is the latitude and longitude. After real-time processing based on the Flink real-time processing framework, the longitude and latitude can be converted into specific location information that can be intuitively seen by people; for another example, the current data to be processed is the ID of a product, which is processed in real time based on Flink. After the framework is processed in real time, the product ID can be converted into specific commodity information. Based on the Flink real-time processing framework, the current data to be processed is converted into target-oriented data in real time, and the cleaning of the current data to be processed is completed.

S240. Add the queue processing data to the second data message queue.

The queue processing data obtained after processing based on the Flink real-time processing framework and using the streaming data processing method is added to the second data message queue.

Optionally, adding the queue processing data to the second data message queue may include: determining the second target partition sequence number and the second data processing progress identifier corresponding to the queue processing data; according to the second target partition serial number and the second data processing progress The identification determines the second target partition position of the queue processing data in the second target partition sequence number; the queue processing data is added to the second target data partition in the second data message queue according to the second target partition sequence number and the second target partition position.

The sequence number of the second target partition may be the sequence number of the second target data partition where the to-be-processed data corresponding to the queue processing data obtained after processing is located in the second data message queue. The second target partition location may be a specific queue location in the second target data partition in the second data message queue of the to-be-processed data corresponding to the queue processing data obtained after processing. The second target data partition, that is, the queue processing data obtained after processing, is added to the corresponding partition in the second data message queue. When adding the queue processing data to the second data message queue, the second target partition sequence number and the second data processing progress identifier corresponding to the queue processing data may be determined first. In this embodiment of the present invention, the second data processing identifier may be passed through the The position representation of the last partition in the second data message queue is used to identify the data addition progress of the second data message queue. Among them, the position of the last partition in the second data message queue means that before the Flink real-time processing framework starts normally or the system fails to restart the Flink real-time processing framework, the last queue processing data is added to the second data message queue corresponding to the first partition 2. Target partition location. Then determine the second target subregion position of the queue processing data in the second target subregion sequence number according to the second target subregion sequence number and the second data processing progress mark, wherein the second target subregion position corresponds to the second data processing progress indicator; The second target partition sequence number and the second target partition location add the queue processing data to the second target data partition in the second data message queue.

It should be noted that, when each of the data to be processed in the first data message queue is processed by the streaming data processing method based on the Flink real-time processing framework, the determined first data processing progress identifier is used to lock the first data message queue. Data processing progress. Similarly, when the queue processing data is added to the second data message queue, the determined second data processing progress identifier is used to lock the data adding progress of the second data message queue. That is, the first data processing progress identifier and the second data processing progress identifier are two different progress identifiers.

For example, it is assumed that the first target partition sequence number of the first target data partition in the first data message queue where the current data to be processed is located is 2, and the corresponding first data processing progress identifier is 04, indicating that the first data message queue is currently being processed. The fourth piece of data in the data partition 2 of the first data message queue has been added to the first target data partition 2 due to the inconsistency between the data processing speed and the speed of obtaining the data to be processed. At the position 08 of the first target partition, it may even appear that the pending data in the first data message queue has been added to the first target data partition 3. Restart the Flink real-time processing framework. When re-determining the current data to be processed, it is necessary to determine the current data to be processed according to the current partition sequence number corresponding to the current data partition (ie partition 2) and the first data processing progress identifier (ie 04), and then based on The Flink real-time processing framework processes the currently pending data.

It should be pointed out that if the speed of obtaining the data to be processed is the same as the data processing speed, that is to say, when a piece of data to be processed is obtained, the processing of the piece of data to be processed will be completed immediately, and then the next piece of data just obtained will be processed immediately. A piece of data to be processed, in this case, the current partition serial number corresponding to the current data partition corresponds to the first target partition serial number of the to-be-processed data in the first target data partition, and the first data processing progress identifier corresponds to the first target The positions of the partitions correspond to each other, so the current data to be processed can be determined only according to the sequence number of the first target partition and the position of the first target partition in the first data message queue.

For another example, when adding the queue processing data obtained by processing the current data to be processed based on the Flink real-time processing framework to the second data message queue, considering the inconsistency between the data processing speed and the speed at which data is added to the second data message queue, It is necessary to determine the second target partition sequence number and the second data processing progress identifier corresponding to the queue processing data. Assuming that the corresponding second target partition serial number is 2 and the second data processing progress identifier is 04, it can be determined that the queue processing data should be added to the first partition. At the second target partition position 04 in the two target partitions 2, the queue processing data is added to the second target data partition 2 in the second data message queue according to the second target partition sequence number 2 and the second target partition position 04.

It should be pointed out that if the data processing speed is the same as the speed at which the queue processing data is added to the second data message queue, that is, when a piece of pending data is processed, the obtained queue processing data can be immediately added to the second data message queue. In the message queue, and then immediately add the obtained next queue processing data, in this case, the second target partition serial number corresponds to the first target partition serial number, and the second data processing progress identifier and the first data processing progress The identifiers correspond, so the obtained queue processing data can be added to the second data message queue according to the sequence number of the first target partition and the first data processing progress identifier.

It can be understood that the storage modes of the data in the first data message queue and the second data message queue are consistent, that is, the storage locations of the same data in the two data message queues are consistent. For example, if the sequence number of the first target partition corresponding to the data to be processed in the first data message queue is 0 and the position of the first target partition is 2, the queue processing data obtained after real-time processing of the data to be processed is added to the second data When the message queue is used, the sequence number of the corresponding second target partition is 0 and the position of the second target partition is 2.

FIG. 3 is a schematic flowchart of a breakpoint resuming transmission provided by an embodiment of the present invention. In a specific example, as shown in FIG. 3 , when the speed of acquiring the data to be processed, the data processing speed and the queue processing data are added to the first When the speeds of the two data message queues are inconsistent with each other, during normal processing and transmission of data, after processing based on the Flink real-time processing framework, the obtained queue processing data can be added to the corresponding data according to the second target sequence number and the second target partition location in the second target data partition in the second data message queue. When the system fails during data processing and restarts the Flink real-time processing framework, it can determine the current partition sequence number corresponding to the current data partition and the data processing progress of the first data message queue (ie, the first data processing progress identifier), and then according to the first The current partition serial number in the data message queue and the first data processing progress identifier determine the current pending data, and continue to process the current pending data based on the Flink real-time processing framework, so as to ensure that the pending data processing continues at the location where the data processing was performed before the system failure. Process data and realize data resuming. And then can determine the second target partition position of the queue processing data in the second target partition serial number according to the second target partition sequence number and the data addition progress of the second data message queue (that is, the second data processing progress mark), and continue to queue the processing data. is added to the second target data partition in the second data message queue.

S250. Perform real-time data processing on each queue processing data in the second data message queue based on the Flink real-time processing framework.

For details that are not explained in this embodiment, please refer to the foregoing embodiments, which will not be repeated here.

FIG. 4 is a schematic flowchart of data cleaning and processing provided by an embodiment of the present invention. In a specific example, as shown in FIG. 4 , the data to be cleaned is collected by a data collector using the Flume technology. The cleaning data can include log files, buried point data or external data. The collected data to be cleaned is added to the first data message queue, and then each data to be cleaned in the first data message queue can be read and the data to be cleaned can be read. The data is stored in the distributed file system to retain the original data for comparison and verification. It is also possible to use streaming processing to clean the data to be cleaned in the first data message queue based on the Flink real-time processing framework, and process the resulting queue. The data is added to the second data message queue to perform real-time data processing on the processing data of each queue in the second data message queue based on the Flink real-time processing framework. In addition, the processing data of each queue can also be read from the second data message queue , and store it in the static database for batch data processing of each queue processing data, and can store the batch data processing results and real-time data processing results in the result database, or read each queue from the second data message queue. The queue processes data and sends it to the target application through the preset interface, which realizes the architecture expansion of the Flink real-time processing framework, meets the data requirements corresponding to different applications, and improves the flexibility of the architecture.

In the above technical solution, after the acquired data to be processed is added to the first target data partition in the first data message queue according to the first target partition serial number and the first target partition position, the streaming data is used based on the Flink real-time processing framework. When processing the data to be processed in the first data message queue, a closed-loop method is used to process the data, that is: first determine the current partition serial number corresponding to the current data partition and the data processing progress identifier, and according to the current partition serial number And the data processing progress indicator determines the current data to be processed, and then processes the current data to be processed in real time based on the Flink real-time processing framework to obtain the queue processing data, thus avoiding data inconsistency and reduction when the Flink real-time processing framework starts normally or the system restarts due to failure. The problem of real-time data, and then add the queue processing data to the second data message queue, so that real-time data processing can be performed on the processing data of each queue in the second data message queue based on the Flink real-time processing framework, realizing real-time processing based on Flink. The framework also uses two data message queues to process data in real time, and adopts a closed-loop method in the data processing process, with no special requirements for data, which solves the problem that the existing data processing process is difficult to effectively ensure consistency and real-time performance, and ensures The data processing process is consistent and real-time, and the function of continuous data transmission is realized.

Embodiment 3

FIG. 5 is a schematic structural diagram of a data processing apparatus provided in Embodiment 3 of the present invention. The embodiment of the present invention is applicable to processing any type of data based on the Flink real-time processing framework to ensure the consistency and real-time performance of the data processing process. In some cases, the apparatus can be implemented in software and/or hardware, and can generally be integrated in computer equipment.

As shown in FIG. 5 , the data query apparatus specifically includes: a first data message queue generation module 510 , a queue processing data generation module 520 , a second data message queue generation module 530 and a real-time data processing module 540 . in,

The first data message queue generating module 510 is configured to obtain data to be processed, and add the data to be processed to the first data message queue;

The queue processing data generation module 520 is configured to process each of the to-be-processed data in the first data message queue based on the Flink real-time processing framework using a streaming data processing method to obtain queue processing data;

The second data message queue generating module 530 is configured to add the queue processing data to the second data message queue;

The real-time data processing module 540 is configured to perform real-time data processing on each of the queue processing data in the second data message queue based on the Flink real-time processing framework.

According to the technical solution provided by the embodiment of the present invention, after the acquired data to be processed is added to the first data message queue, each data to be processed in the first data message queue is processed based on the Flink real-time processing framework and the streaming data processing method is adopted. Perform processing to obtain the queue processing data, and then add the queue processing data to the second data message queue, so that real-time data processing can be performed on the queue processing data in the second data message queue based on the Flink real-time processing framework. Real-time data processing based on Flink The processing framework uses two data message queues to process data in real time, with no special requirements for data, which solves the problem that the existing data processing process is difficult to effectively ensure consistency and real-time performance, and ensures the consistency and real-time performance of the data processing process. .

Optionally, the first data message queue generating module 510 is specifically set as:

Determine the first target partition sequence number and the first target partition position corresponding to the data to be processed;

The data to be processed is added to the first target data partition in the first data message queue according to the sequence number of the first target partition and the position of the first target partition.

Optionally, the queue processing data generation module 520 further includes: a current partition serial number and a processing progress identification unit, a current data determination unit to be processed, and a current data processing unit to be processed, wherein,

The current partition serial number and the processing progress identification determining unit is set to determine the current partition serial number corresponding to the current data partition and the first data processing progress identification;

A current to-be-processed data determination unit, configured to determine current to-be-processed data according to the current partition serial number and the first data processing progress identifier;

The current to-be-processed data processing unit is set to process the current to-be-processed data in real time based on the Flink real-time processing framework.

Optionally, the current data processing unit to be processed, specifically set as:

Based on the Flink real-time processing framework, the current data to be processed is converted into target directional data in real time.

Optionally, the second data message queue generation module 530 is specifically set as:

Determine the second target partition sequence number and the second data processing progress identifier corresponding to the queue processing data;

Determine the second target partition position of the queue processing data in the second target partition serial number according to the second target partition serial number and the second data processing progress identifier;

The queue processing data is added to the second target data partition in the second data message queue according to the second target partition sequence number and the second target partition position.

Optionally, the above-mentioned device further includes: a data storage module to be processed, wherein the data storage module to be processed is specifically configured to process each of the data to be processed in the first data message queue based on the Flink real-time processing framework. , read each of the data to be processed in the first data message queue;

Each of the data to be processed is stored in a distributed file system.

Optionally, the above-mentioned device further includes: a data processing result storage module or a processing data sending module, wherein the data processing result storage module is specifically configured to: process data for each of the queues in the second data message queue. After the real-time data processing is performed, each of the queue processing data is read from the second data message queue and stored in a static database; the static database is used to perform batch data processing on each of the queue processing data;

Store real-time data processing results and batch data processing results in a results database; or

The processing data sending module is specifically configured to: after performing real-time data processing on each of the queue processing data in the second data message queue, read each of the queue processing data from the second data message queue , and sent to the target application through the preset interface.

The above data processing apparatus can execute the data processing method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the data processing method.

Embodiment 4

FIG. 6 is a schematic diagram of a hardware structure of a computer device according to Embodiment 4 of the present invention. Figure 6 shows a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 6 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present invention.

As shown in FIG. 6, computer device 12 takes the form of a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , system memory 28 , and a bus 18 connecting various system components including system memory 28 and processing unit 16 .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect ( PCI) bus.

Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including both volatile and nonvolatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . Computer device 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 6, commonly referred to as a "hard drive"). Although not shown in Figure 6, a disk drive may be provided for reading and writing to removable non-volatile magnetic disks (eg "floppy disks"), as well as removable non-volatile optical disks (eg CD-ROM, DVD-ROM) or other optical media) to read and write optical drives. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. System memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present invention.

A program/utility 40 having a set (at least one) of program modules 42, which may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and programs Data, each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the described embodiments of the present invention.

Computer device 12 may also communicate with one or more external devices 14 (eg, keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable a user to interact with computer device 12, and/or communicate with Any device (eg, network card, modem, etc.) that enables the computer device 12 to communicate with one or more other computing devices. Such communication may take place through input/output (I/O) interface 22 . Also, the computer device 12 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter 20 . As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18 . It should be understood that, although not shown in FIG. 6, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tapes drives and data backup storage systems.

The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implements a data processing method provided by the embodiment of the present invention. That is, when the processing unit executes the program, it realizes:

obtaining data to be processed, and adding the data to be processed to the first data message queue;

Based on the Flink real-time processing framework, the data to be processed in the first data message queue is processed by a streaming data processing method, and the queue processing data is obtained;

adding the queue processing data to the second data message queue;

Real-time data processing is performed on each of the queue processing data in the second data message queue based on the Flink real-time processing framework.

Embodiment 5

Embodiment 5 of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements a data processing method as provided by all inventive embodiments of the present application: that is, the program Implemented when executed by the processor:

obtaining data to be processed, and adding the data to be processed to the first data message queue;

Based on the Flink real-time processing framework, the data to be processed in the first data message queue is processed by a streaming data processing method, and the queue processing data is obtained;

adding the queue processing data to the second data message queue;

Real-time data processing is performed on each of the queue processing data in the second data message queue based on the Flink real-time processing framework.

Any combination of one or more computer-readable media may be employed. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (a non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .

Program code embodied on a computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including object-oriented programming languages (such as Java, Smalltalk, C++), and conventional procedural programming language (such as the "C" language or similar programming language). The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention. The scope is determined by the scope of the appended claims.

Claims (10)

1. a data processing method, is characterized in that, comprises: obtaining data to be processed, and adding the data to be processed to the first data message queue; Based on the Flink real-time processing framework, the data to be processed in the first data message queue is processed by a streaming data processing method, and the queue processing data is obtained; adding the queue processing data to the second data message queue; Real-time data processing is performed on each of the queue processing data in the second data message queue based on the Flink real-time processing framework. 2. The method according to claim 1, wherein the adding the to-be-processed data to the first data message queue comprises: Determine the first target partition sequence number and the first target partition position corresponding to the data to be processed; The data to be processed is added to the first target data partition in the first data message queue according to the sequence number of the first target partition and the position of the first target partition. 3. The method according to claim 2, wherein the processing of each of the data to be processed in the first data message queue based on the Flink real-time processing framework using a streaming data processing method comprises: Determine the current partition sequence number corresponding to the current data partition and the first data processing progress identifier; Determine the current data to be processed according to the current partition serial number and the first data processing progress identifier; The current data to be processed is processed in real time based on the Flink real-time processing framework. 4. The method according to claim 3, wherein the real-time processing of the current data to be processed based on the Flink real-time processing framework comprises: Based on the Flink real-time processing framework, the current data to be processed is converted into target directional data in real time. 5. The method according to any one of claims 2-4, wherein the adding the queue processing data to the second data message queue comprises: Determine the second target partition sequence number and the second data processing progress identifier corresponding to the queue processing data; Determine the second target partition position of the queue processing data in the second target partition serial number according to the second target partition serial number and the second data processing progress identifier; The queue processing data is added to the second target data partition in the second data message queue according to the second target partition sequence number and the second target partition position. 6 . The method according to claim 1 , wherein before processing each of the data to be processed in the first data message queue based on the Flink real-time processing framework, the method further comprises: 6 . reading each of the data to be processed in the first data message queue; Each of the data to be processed is stored in a distributed file system. 7. The method according to claim 1, wherein after performing real-time data processing on each of the queue processing data in the second data message queue, the method further comprises: Read each of the queue processing data from the second data message queue and store it in a static database; the static database is used to perform batch data processing on each of the queue processing data; Store real-time data processing results and batch data processing results in a results database; or Each of the queue processing data is read from the second data message queue, and sent to the target application through a preset interface. 8. A data processing device, comprising: a first data message queue generation module, configured to obtain the data to be processed, and add the to-be-processed data to the first data message queue; The queue processing data generation module is configured to process each of the to-be-processed data in the first data message queue based on the Flink real-time processing framework using a streaming data processing method to obtain queue processing data; The second data message queue generation module is configured to add the queue processing data to the second data message queue; The real-time data processing module is configured to perform real-time data processing on each of the queue processing data in the second data message queue based on the Flink real-time processing framework. 9. The device according to claim 8, wherein the first data message queue generation module is specifically set as: Determine the first target partition sequence number and the first target partition position corresponding to the data to be processed; The data to be processed is added to the first target data partition in the first data message queue according to the sequence number of the first target partition and the position of the first target partition. 10. A computer device, comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements any of claims 1-7 when the processor executes the program. A described data processing method.

Images