CN117009139A - Data stream computing anomaly recovery method, device, equipment and storage medium - Google Patents

Abstract

The present invention relates to the field of data processing technology, and in particular to a data stream computing exception recovery method, device, equipment and storage medium. This invention determines the abnormal operator and adjacent operators when the flow calculation is abnormal, calls the target operator corresponding to the abnormal operator, and replaces the abnormal operator with the target operator to establish a connection with the adjacent operator to ensure the integrity Data processing executability, while reading the replay point data in the pre-written log through adjacent numbers, and delivering the replay point data to the target operator, so that the target operator can perform flow calculations based on the replay point data to complete Recalculation of abnormal flow calculation results and recalculation of all data avoids the technical problem of low recalculation efficiency in the existing technology when the real-time data flow calculation based on flink is abnormal, and reduces the cycle of abnormal recovery of flow calculation.

Description

Data flow computing exception recovery method, device, equipment and storage medium

Technical field

The present invention relates to the field of data processing technology, and in particular to a data stream computing exception recovery method, device, equipment and storage medium.

Background technique

In cloud computing jobs or large-scale data computing jobs, since real-time computing jobs are executed in a larger cluster server, hardware failures or abnormalities will cause the computing jobs to fail. In traditional technology, distributed stream processing is generally used. And the batch processing framework (Apache Flink) adopts the form of periodically saving snapshots, so that when an exception occurs during data processing, the calculation process can be reflowed based on the saved data. However, for large-scale real-time computing jobs, the saved job status It may be very large, the cycle of state backup and reprocessing is very long, the fault tolerance cost is high, and because the data state is real-time, the results of re-stream calculation are inconsistent with the normal calculation results, affecting the data processing efficiency.

The above content is only used to assist in understanding the technical solution of the present invention, and does not represent an admission that the above content is prior art.

Contents of the invention

The main purpose of the present invention is to provide a data flow calculation abnormality recovery method, device, equipment and storage medium, aiming to solve the technical problem in the prior art of low recalculation efficiency when real-time data flow calculation based on flink is abnormal.

In order to achieve the above purpose, the present invention provides a data stream computing exception recovery method, which includes the following steps:

When an exception occurs in the data flow calculation, determine the abnormal operator and the corresponding adjacent operator;

Call the target operator corresponding to the abnormal operator, and establish a connection between the target operator and adjacent operators;

Read the replay point data in the write-ahead log based on the adjacent operator, and deliver the replay point data to the target operator;

Data flow calculation is performed on the replay point data based on the target operator.

Optionally, performing data flow calculation on the replay point data based on the target operator includes:

Read the status information corresponding to the replay point data in the external storage database based on the target operator;

Data flow calculation is performed based on the replay point data and the status information based on the target operator.

Optionally, the data flow computing exception recovery method also includes:

Write the flow calculation results into the write-ahead log;

Call an external storage database to update the status information corresponding to the replay point data.

Optionally, the adjacent operators include superior operators;

The data flow computing exception recovery method also includes:

Send confirmation information to the superior operator based on the target operator;

Based on the upper-level operator, the data offset of the replay point data in the write-ahead log is adjusted according to the confirmation information, and a normal data flow calculation mode is executed.

Optionally, perform flow calculation on the replay point data based on the target operator, including:

Use the target operator to determine whether the replay point data is duplicate data based on the data offset in the write-ahead log;

When the replay point data is not repeated data, the step of performing data flow calculation on the replay point data based on the target operator is performed.

Optionally, the data flow computing exception recovery method also includes:

When the replay point data is repeated data, generate reminder information based on the target operator, and feed back the reminder information to the superior operator;

Adjust the data offset of the replay point data in the write-ahead log;

The superior operator reads the data to be processed corresponding to the adjusted data offset in the external storage database according to the reminder information, and sends the data to be processed to the target operator to execute the data Stream computing.

Optionally, reading the replay point data in the write-ahead log based on the adjacent operator includes:

Query the historical data offset in the write-ahead log based on the adjacent operator;

Read replay point data in the write-ahead log according to the historical data offset.

In addition, in order to achieve the above object, the present invention also proposes a data flow computing abnormality recovery device. The data flow computing abnormality recovery device includes:

The acquisition module is used to determine the abnormal operator and the corresponding adjacent operator when an exception occurs in data processing;

A calling module, used to call the target operator corresponding to the abnormal operator and establish a connection between the target operator and adjacent operators;

A reading module, configured to read the replay point data in the write-ahead log based on the adjacent operator, and send the replay point data to the target operator;

A calculation module, configured to perform data flow calculation on the replay point data based on the target operator.

In addition, in order to achieve the above object, the present invention also proposes a data flow computing exception recovery device. The data flow computing exception recovery device includes: a memory, a processor, and a device that is stored on the memory and can run on the processor. The data flow computing exception recovery program is configured to implement the steps of the data flow computing exception recovery method as described above.

In addition, in order to achieve the above object, the present invention also proposes a storage medium, a data flow calculation exception recovery program is stored on the storage medium, and when the data flow calculation exception recovery program is executed by the processor, the data as described above is realized. Steps of the streaming computing exception recovery method.

In the present invention, when an abnormality occurs in data flow calculation, the abnormal operator and the corresponding adjacent operator are determined; the target operator corresponding to the abnormal operator is called, and the relationship between the target operator and the adjacent operator is established. Connect; read the replay point data in the write-ahead log based on the adjacent operator, and send the replay point data to the target operator; perform the replay based on the target operator Point data is used for data flow calculation. This invention determines the abnormal operator and adjacent operator when the flow calculation is abnormal, and calls the target operator corresponding to the abnormal operator, and replaces the abnormal operator with the target operator to establish a relationship with the adjacent operator. The connection between them ensures the executability of the overall data processing. At the same time, the replay point data in the pre-written log is read through adjacent numbers, and the replay point data is sent to the target operator, so that the target operator can be based on the replay Perform flow calculation on point data to complete the recalculation of abnormal flow calculation results and recalculate all data. This avoids the technical problem of low recalculation efficiency in the existing technology when the real-time data flow calculation based on flink is abnormal and reduces the flow rate. Calculate the abnormal recovery period.

Description of the drawings

Figure 1 is a schematic structural diagram of a data flow computing exception recovery device for a hardware operating environment involved in an embodiment of the present invention;

Figure 2 is a schematic flowchart of the first embodiment of the data flow calculation exception recovery method of the present invention;

Figure 3 is a schematic diagram of the flink architecture according to one embodiment of the data stream computing exception recovery method of the present invention;

Figure 4 is a schematic diagram of an existing flink single job according to an embodiment of the data stream computing exception recovery method of the present invention;

Figure 5 is a schematic diagram of a single flink job after the computing node status is stripped according to one embodiment of the data flow computing exception recovery method of the present invention;

Figure 6 is a schematic flowchart of the second embodiment of the data flow calculation exception recovery method of the present invention;

Figure 7 is a schematic diagram of the normal data processing flow of one embodiment of the data flow calculation abnormality recovery method of the present invention;

Figure 8 is a schematic diagram of an exception recovery process according to an embodiment of the data flow calculation exception recovery method of the present invention;

Figure 9 is a structural block diagram of the first embodiment of the device for recovering data flow calculation exceptions according to the present invention.

The realization of the purpose, functional features and advantages of the present invention will be further described with reference to the embodiments and the accompanying drawings.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

Referring to FIG. 1 , FIG. 1 is a schematic structural diagram of the data flow computing exception recovery device of the hardware operating environment involved in the embodiment of the present invention.

As shown in Figure 1, the data flow computing exception recovery device may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize connection communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). The optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

Those skilled in the art can understand that the structure shown in Figure 1 does not constitute a limitation on the data stream computing exception recovery device, and may include more or less components than shown in the figure, or combine certain components, or different components. layout.

As shown in Figure 1, memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a data flow calculation exception recovery program.

In the data flow computing abnormality recovery equipment shown in Figure 1, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processing in the data flow computing abnormality recovery equipment of the present invention The processor 1001 and the memory 1005 can be arranged in a data flow computing exception recovery device. The data flow computing exception recovery device calls the data flow computing exception recovery program stored in the memory 1005 through the processor 1001, and executes the data provided by the embodiment of the present invention. Stream computing exception recovery method.

Embodiments of the present invention provide a data flow computing exception recovery method. Refer to Figure 2. Figure 2 is a flow chart of a data flow computing exception recovery method according to the first embodiment of the present invention.

In this embodiment, the data flow computing exception recovery method includes the following steps:

Step S10: When an abnormality occurs in the data flow calculation, determine the abnormal operator and the corresponding adjacent operator.

It should be noted that the execution subject of the method of this embodiment can be a device with data processing, program running, and network communication, such as a cloud server or a computer cluster, or other devices that can implement the same or similar functions. This implementation This example does not specifically limit this. In this embodiment and the following embodiments, a cloud server will be used as an example for explanation.

It is worth noting that the application scenario of the method in this embodiment is based on the framework and distributed processing engine (Apache Flink), hereinafter referred to as flink. Flink is used to perform stateful calculations on unbounded and bounded data streams, and can be used in all common It runs in a cluster environment and can perform calculations at memory speed and at any scale, with high throughput and low latency.

In the specific implementation, refer to Figure 3, the Flink cluster contains the job manager (Job Manager) and the working node (Task Manager). The job manager is the job management process, the working node is the working process, and the management process regularly triggers state saving (Trigger Checkpoint) and confirm the completion of state saving (Ack.Checkpint). The operator in TaskManager is responsible for saving the state snapshot to an external distributed storage (snapshot store), such as Hadoop Distributed File System (Hadoop Distributed File System, HDFS) or resis, etc. Distributed file system, this embodiment does not impose specific restrictions on this.

Referring to Figure 4, in traditional technology, when executing data processing jobs based on flink, each operator uses memory or memory combined with the file system to store status on its own computing node, and regularly backs up the data status information to the distributed file system. This At the same time, if an abnormality occurs in a certain computing node or operator, the backup data status information can be read from the distributed file system to facilitate re-calculation of the data flow. At this time, there is no need to recalculate the entire data processing job. Once, the data processing efficiency is improved.

Among them, the data status information is mainly for data in an intermediate state during multiple data processing processes. For example: in the data processing job of "calculating five times the tenth power of 2", you can first calculate the tenth power of 2, and then Multiply 2 to the tenth power by 5. If there is an error in the step of "multiplying 2 to the tenth power by 5", you can recalculate the value of multiplying 2 to the tenth power by 5 by calling the previously saved "2 to the tenth power". Instead of calculating the entire data processing process, this process of saving intermediate data calculation results to facilitate "breakpoint recovery" in the event of a failure is data status information.

In the specific implementation, due to the large data throughput of flink, the bytes occupied by the calculation of a batch of data can be from KB level to TB level. When the data status information is regularly backed up to the distributed file system, for smaller Data status information can be backed up in a shorter time period, such as 30 seconds/time, to reduce the need to recalculate the amount of data when stream computing exceptions occur and improve the efficiency of stream computing exception recovery. For larger data status information, a longer period can be used. The time period for backing up is once, for example: 2 minutes. This embodiment does not impose specific restrictions on this.

It can be understood that an operator refers to the mapping of a function calculation space to another function calculation space, such as: source operator, window operator, sink operator, etc. This embodiment does not impose specific restrictions on this. Exception operators refer to When performing stream calculations, the mapping relationship of incorrect calculation results appears. In this embodiment, referring to Figure 5, Flink's processing of real-time data streams is divided into obtaining data, processing data and outputting processing results. Operators are used to process data. , the data source for obtaining data can be the Kafka business system. After the operator obtains the data in the Kafka business system, it can output the calculation results to the Kafka business system to complete a complete data flow calculation process.

In this embodiment, the adjacent operators of the abnormal operator refer to the upper-level operator and the lower-level operator connected to the abnormal operator. For example: when the window operator in Figure 5 is an abnormal operator, the adjacent operators include the upper-level operator The operator source operator and the subordinate operator sink operator.

In addition, traditional flow computing exception recovery will obtain the data status information from the distributed storage engine, apply it locally, and then re-execute the flow computing; this embodiment directly reads the data status information of the distributed storage engine. , no application process is needed to improve the efficiency of stream computing exception recovery.

Step S20: Call the target operator corresponding to the abnormal operator, and establish a connection between the target operator and adjacent operators.

It can be understood that the target operator can be an operator that belongs to a different server than the abnormal operator but has the same function. This process is performed by the job manager. Establishing a connection between the target operator and adjacent operators means to The information of the upstream and downstream operators connected to the abnormal operator, the storage path of the abnormal operator's prewritten log, and the state storage information are sent to the target operator, thereby replacing the adjacent operator with the target operator to establish a complete connection. , which can further realize the transmission of data and messages between upstream and downstream operators.

Step S30: Read the replay point data in the write-ahead log based on the adjacent operator, and deliver the replay point data to the target operator.

It should be noted that the Write Ahead Log (WAL) database system is a means to ensure the atomicity and durability of data operations. For non-memory databases, disk I/O operations are database A major bottleneck in efficiency. Under the same data volume, when a transaction is submitted in a database system using write-ahead logs, the disk write operations are only about half of those of traditional rollback logs, which greatly improves the efficiency of database disk I/O operations and thus improves the database's performance. performance.

Among them, when using a write-ahead log system, all modifications must be written to the log file before submission. The log file usually includes redo and undo information. For example, a program loses power while the machine is performing certain operations. When restarting, the program may need to know whether the operation performed at that time was successful, partially successful, or failed. If a write-ahead log is used, The program can check the log file and compare the contents of the operations planned to be performed when the power is suddenly lost with the contents of the operations actually performed. Based on this comparison, the program can decide whether to undo the operations that have been done or to continue to complete the operations that have been done. operation, or leave it as is.

The replay point data refers to the data after the offset corresponding to the last data calculation that was successful and correct. For example: when executing three data operations of ABC, AB is calculated successfully and the calculation results are correct, but the calculation of C is abnormal. Then the replay point data is C, which is the data after the last successfully calculated data.

In the specific implementation, since different operators perform different data processing procedures, during the data processing process, the data that the operator needs to process is issued by the operator at the upper level. Therefore, when performing exception recovery, the same The adjacent operator reads the replay point data in the pre-written log and sends the replay point data to the target operator. Only then can the target operator perform data flow calculations based on the replay point data and complete the data. abnormal recovery.

Further, reading the replay point data in the write-ahead log based on the adjacent operator includes:

Query the historical data offset in the write-ahead log based on the adjacent operator;

Read replay point data in the write-ahead log according to the historical data offset.

In a specific implementation, when reading the replay point data in the write-ahead log, the location of the replay point data can be determined based on the offset set in the write-ahead log, for example:

Step S40: Perform data flow calculation on the playback point data based on the target operator.

It is worth mentioning that when performing data flow calculation, the received data can be calculated according to the business logic corresponding to the function space mapping relationship of the target operator.

In the specific implementation, each upstream operator finds the data replay point from the pre-written log and replays the data. The state is stored in the external high-concurrency distributed KV storage. The state does not need special processing, and all states do not need to be processed. Redistribution achieves immutability by replaying data. Even if the external dimension table changes, it will not cause the data processing results after error recovery to be inconsistent with those before error recovery.

This embodiment determines the abnormal operator and adjacent operators when the stream calculation is abnormal, calls the target operator corresponding to the abnormal operator, and uses the target operator to replace the abnormal operator to establish a connection with the adjacent operator, ensuring that Overall data processing executability, while reading the replay point data in the pre-written log through adjacent numbers, and delivering the replay point data to the target operator, so that the target operator can perform stream calculations based on the replay point data. Complete the recalculation of abnormal flow calculation results and recalculate all data, which avoids the technical problem of low recalculation efficiency in the existing technology when the real-time data flow calculation based on flink is abnormal, and reduces the cycle of abnormal recovery of flow calculation.

Referring to Figure 6, Figure 6 is a schematic flowchart of a second embodiment of a data flow calculation exception recovery method according to the present invention.

Based on the above first embodiment, in this embodiment, step S40 includes:

Step S401: Read the status information corresponding to the playback point data in the external storage database based on the target operator.

It should be noted that when calculating the replay point data, if there is a time difference in the replay point data, for example: using a stream computing job to determine the user's online status, if no error occurs, the associated result is (xx user, online ), if the user goes offline during the abnormal time period, then after the job is resumed, the user's status becomes (XX user, offline). It can be found that the results of the two calculations are completely different. At this time, it cannot be determined because Whether the fault leads to misjudgment that the user is offline or the user actively goes offline, idempotence cannot be guaranteed.

The external storage database may be a distributed high-availability, high-performance KV type state storage, such as Redis, HBASE and other databases. This embodiment does not impose specific restrictions on this.

Step S402: Perform data flow calculation according to the playback point data and the status information based on the target operator.

In the specific implementation, refer to Figure 7, which is a schematic flow chart of stream computing exception recovery in this embodiment. The new operator is the target operator. When a stream computing exception occurs, the job manager pulls the Start a new operator 3, and inform the new operator 3 of its upstream and downstream information, the storage path of the old operator 3's WAL log, and the old operator's status storage information. The new operator 3 instance is initialized and re-established with the upstream operator 2. , the connection relationship of downstream operator 4, thereby realizing data transfer and event notification between upstream and downstream, and then the upstream operator finds the replay point data from the prewritten log and sends the replay point data to the new downstream operator 3 , the new operator reads the state information from the distributed state memory, and then the flow calculation can be re-calculated based on the function mapping relationship corresponding to the target operator.

Further, after the flow calculation is completed, in order to ensure that abnormal results are avoided again, the data flow calculation exception recovery method also includes:

Write the flow calculation results into the write-ahead log;

Call an external storage database to update the status information corresponding to the replay point data.

It can be understood that after the flow calculation is completed, the flow calculation results can be first written into the write-ahead log. At the same time, if the calculation of data status is involved, the status in the distributed state memory can also be updated.

Further, in order to ensure that subsequent data flow calculations can be performed normally, the data flow calculation exception recovery method also includes:

Send confirmation information to the superior operator based on the target operator;

Based on the upper-level operator, the data offset of the replay point data in the write-ahead log is adjusted according to the confirmation information, and a normal data flow calculation mode is executed.

After completing the exception recovery, you can notify the superior that the data processing is completed; then start the normal data processing process. For example: after operator 3 completes exception recovery, it notifies operator 2 that the recovery is complete. At this time, operator 2 can read the preset data. Write the data in the log, and adjust the data offset of the replay point data in the write-ahead log, so that the data offset is shifted backward or increased by 1. The specific adjustment method depends on the setting method of the data offset. For example: There are four data calculation tasks ABCD in the streaming data computing job. When there is an exception in C, the offset value of the streaming data computing job is 2. After the exception recovery is completed, the offset value of the streaming data computing job is 3. At this time The data calculation task of D will be calculated, and this embodiment does not impose specific limitations on this.

In the specific implementation, referring to Figure 8, the normal data flow calculation mode means that the upstream operator (operator 2) sends data to the downstream; the downstream operator is responsible for data deduplication to prevent repeated data processing. If the data is sent repeatedly, then There is no need to process this data, otherwise the downstream operator will process the data according to business logic after receiving the data, and first write the processing results to the WAL log. At this time, no matter how many times the job is started or resumed, idempotence can be guaranteed; after the log is written successfully, if it is a stateful calculation, the external distributed state storage is called to update the status; updating the status and writing the log are atomic The operations either succeed or fail together; the data status is updated, and a message is sent to the upstream operator to confirm that the data sent by the operator has been processed; the upstream operator records the offset of the position of the successfully processed data in the WAL log. quantity.

Further, performing flow calculation on the replay point data based on the target operator includes:

Use the target operator to determine whether the replay point data is duplicate data based on the data offset in the write-ahead log;

When the replay point data is not repeated data, the step of performing data flow calculation on the replay point data based on the target operator is performed.

In a specific implementation, determining whether the replay point data is duplicate data may be to determine whether the position of the replay point data in the pre-written log corresponds to the data offset. For example: there are ABCD four in the stream data calculation job. data calculation task, the offset value of the stream data calculation job is 2, and the position of the replay point data is the position of C, which means that the replay point data is not the replay point data, the offset value of the stream data calculation job is 2. If the position of the replay point data is the position of B or A, it means that the replay point data is the replay point data.

Further, when the replay point data is repeated data, reminder information is generated based on the target operator, and the reminder information is fed back to the superior operator;

Adjust the data offset of the replay point data in the write-ahead log;

The superior operator reads the data to be processed corresponding to the adjusted data offset in the external storage database according to the reminder information, and sends the data to be processed to the target operator to execute the data Stream computing.

It should be noted that in this embodiment, in addition to receiving the reminder information from the target operator and resending the data, the superior operator can also detect whether the flow calculation completion information from the target operator is received within a certain period of time. If not, , it means that the superior operator delivers data, connects the operator, writes the flow calculation results to the write-ahead log, updates the status, and notifies the superior operator that there is an exception in an item. The occurrence of any abnormality will affect the data flow calculation. the process of.

Therefore, if the upstream does not receive the confirmation message that the downstream processing is completed, and it exceeds a certain time, the upstream operator 2 will resend the data, and the downstream operator 3 is responsible for judging whether it is duplicate data. If it is duplicate data, it will notify the upstream data The processing has been completed. If the downstream operator considers the data to be non-duplicate, the data will be processed according to the normal data processing flow.

This embodiment performs data flow calculations based on the replay point data and the state information corresponding to the replay point data, to avoid inconsistent calculation results caused by state changes, affecting the idempotence of the data processing job, and improving the accuracy of the data processing job. .

In addition, an embodiment of the present invention also proposes a storage medium on which a data flow calculation exception recovery program is stored. When the data flow calculation exception recovery program is executed by the processor, the data flow calculation exception recovery program is implemented as described above. Steps of the recovery method.

Since this storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described again here.

Referring to FIG. 9 , FIG. 9 is a structural block diagram of a first embodiment of a data flow calculation exception recovery device according to the present invention.

As shown in Figure 9, the data stream computing exception recovery device proposed by the embodiment of the present invention includes:

The acquisition module 10 is used to determine the abnormal operator and the corresponding adjacent operator when an abnormality occurs in data processing.

The calling module 20 is used to call the target operator corresponding to the abnormal operator, and establish a connection between the target operator and adjacent operators.

The reading module 30 is configured to read the replay point data in the write-ahead log based on the adjacent operator, and deliver the replay point data to the target operator.

The calculation module 40 is configured to perform data flow calculation on the replay point data based on the target operator.

In an embodiment, the calculation module 40 is also configured to read the status information corresponding to the replay point data in the external storage database based on the target operator; based on the target operator according to the replay Point data and the state information are used for data flow calculations.

In one embodiment, the calculation module 40 is also configured to write flow calculation results into the write-ahead log; and call an external storage database to update the status information corresponding to the replay point data.

In one embodiment, the calculation module 40 is further configured to send confirmation information to the upper-level operator based on the target operator; and adjust the pre-written log according to the confirmation information based on the upper-level operator. The data offset of the replay point data and performs normal data flow calculation mode.

In one embodiment, the calculation module 40 is further configured to use the target operator to determine whether the replay point data is duplicate data based on the data offset in the write-ahead log; When the playback point data is not repeated data, the step of performing data flow calculation on the playback point data based on the target operator is performed.

In one embodiment, the calculation module 40 is also configured to generate reminder information based on the target operator when the replay point data is repeated data, and feed back the reminder information to the superior operator; adjust the The data offset of the replay point data in the write-ahead log; the upper-level operator reads the data to be processed corresponding to the adjusted data offset in the external storage database according to the reminder information, and The data to be processed is sent to the target operator to perform data flow calculation.

In one embodiment, the calculation module 40 is also used to query the historical data offset in the write-ahead log based on the adjacent operator; and read the write-ahead log based on the historical data offset. Replay point data in the log.

This embodiment determines the abnormal operator and adjacent operators when the stream calculation is abnormal, calls the target operator corresponding to the abnormal operator, and uses the target operator to replace the abnormal operator to establish a connection with the adjacent operator, ensuring that Overall data processing executability, while reading the replay point data in the pre-written log through adjacent numbers, and delivering the replay point data to the target operator, so that the target operator can perform stream calculations based on the replay point data. Complete the recalculation of abnormal flow calculation results and recalculate all data, which avoids the technical problem of low recalculation efficiency in the existing technology when the real-time data flow calculation based on flink is abnormal, and reduces the cycle of abnormal recovery of flow calculation.

It should be understood that although each step in the flow chart in the embodiment of the present application is displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least some of the steps in the figure may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and their execution order is not necessarily sequential. may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of stages.

It should be understood that the above are only examples and do not constitute any limitation on the technical solution of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any limitations on this.

It should be noted that the workflow described above is only illustrative and does not limit the scope of the present invention. In practical applications, those skilled in the art can select some or all of them for implementation according to actual needs. The purpose of this embodiment is not limited here.

In addition, for technical details that are not described in detail in this embodiment, please refer to the data stream computing exception recovery method provided by any embodiment of the present invention, and will not be described again here.

Furthermore, it should be noted that, as used herein, the terms "include", "comprises" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or system that includes a list of elements includes not only those elements, but also other elements not expressly listed or elements inherent to the process, method, article or system. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

The above serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product that is essentially or contributes to the existing technology. The computer software product is stored in a storage medium (such as a read-only memory). , ROM)/RAM, magnetic disk, optical disk), including several instructions to cause a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the method described in various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the description and drawings of the present invention may be directly or indirectly used in other related technical fields. , are all similarly included in the scope of patent protection of the present invention.

Claims (10)

1. A data stream computation anomaly recovery method, characterized in that the data stream computation anomaly recovery method comprises:

when the data stream calculation is abnormal, determining an abnormal operator and a corresponding adjacent operator;

invoking a target operator corresponding to the abnormal operator, and establishing connection between the target operator and an adjacent operator;

reading replay point data in a pre-written log based on the adjacent operator, and issuing the replay point data to the target operator;

and carrying out data stream calculation on the replay point data based on the target operator.

2. The data stream computation anomaly recovery method of claim 1, wherein said performing data stream computation on the playback point data based on the target operator comprises:

reading state information corresponding to the replay point data in an external storage database based on the target operator;

and carrying out data stream calculation according to the replay point data and the state information based on the target operator.

3. The data stream computation anomaly recovery method of claim 2, wherein the data stream computation anomaly recovery method further comprises:

writing a stream calculation result into the pre-written log;

and calling an external storage database to update the state information corresponding to the replay point data.

4. The data stream computation anomaly recovery method of claim 3, wherein the neighbor operator comprises a superior operator;

the data stream computing anomaly recovery method further comprises the following steps:

transmitting confirmation information to the superior operator based on the target operator;

and adjusting the data offset of the replay point data in the pre-written log according to the confirmation information based on the superior operator, and executing a normal data stream calculation mode.

5. The data stream computation anomaly recovery method according to any one of claims 1 or 2, wherein stream computation of the playback point data based on the target operator comprises:

judging whether the replay point data is repeated data or not based on the data offset in the pre-written log through the target operator;

and executing the step of performing data stream calculation on the replay point data based on the target operator when the replay point data is not the duplicate data.

6. The data stream computation anomaly recovery method of claim 5, wherein the data stream computation anomaly recovery method further comprises:

when the replay point data are repeated data, generating reminding information based on the target operator, and feeding back the reminding information to a superior operator;

adjusting a data offset of the playback point data in the pre-written log;

and reading the data to be processed corresponding to the adjusted data offset in the external storage database according to the reminding information through the superior operator, and issuing the data to be processed to the target operator so as to execute data stream calculation.

7. The data stream computation anomaly recovery method of claim 1, wherein the reading replay point data in a pre-written log based on the neighbor operator comprises:

querying historical data offset in a pre-written log based on the adjacent operator;

and reading the replay point data in the pre-written log according to the historical data offset.

8. A data stream computation anomaly recovery device, characterized in that the data stream computation anomaly recovery device comprises:

the acquisition module is used for determining an abnormal operator and a corresponding adjacent operator when the data processing is abnormal;

the calling module is used for calling a target operator corresponding to the abnormal operator and establishing connection between the target operator and an adjacent operator;

the reading module is used for reading the replay point data in the pre-written log based on the adjacent operator and issuing the replay point data to the target operator;

and the calculation module is used for carrying out data stream calculation on the replay point data based on the target operator.

9. A data stream computation anomaly recovery device, characterized in that the data stream computation anomaly recovery device comprises: a memory, a processor, and a data stream computation anomaly recovery program stored on the memory and executable on the processor, the data stream computation anomaly recovery program configured to implement the data stream computation anomaly recovery method of any one of claims 1 to 7.

10. A storage medium having stored thereon a data stream computation anomaly recovery program which, when executed by a processor, implements the data stream computation anomaly recovery method of any one of claims 1 to 7.