CN111190892B - A method and device for processing abnormal data in data backfilling - Google Patents

Abstract

The invention discloses a method and a device for processing abnormal data in data backfill, and relates to the technical field of computers. One embodiment of the method comprises: in the case of abnormal data, determining metadata and time field information of the abnormal data; based on the stored blood relationship data, determining a target task flow and an execution sequence of the target task flow according to the metadata of the abnormal data; wherein the blood relationship data is at least used for expressing the association relationship between the data; and executing the target task flow according to the execution sequence and the time field information of the abnormal data. According to the method, in a data backfill scene, abnormal data is automatically processed based on the blood relationship between data and tasks, manual intervention is not needed, the processing efficiency is greatly improved, and the labor cost is reduced.

Description

A method and device for processing abnormal data in data backfilling

technical field

The invention relates to the field of computer technology, and in particular, to a method and device for processing abnormal data in data backfilling.

Background technique

Airflow is a programmable, scheduling and monitoring workflow platform. Based on the directed acyclic graph dag, airflow can define a set of dependent tasks and execute them in sequence according to the dependencies. Airflow provides a wealth of command-line tools for system management and control, and its web management interface can also easily manage and control scheduling tasks, and monitor the running status of tasks in real time, which facilitates system operation, maintenance and management. Unlike other schedulers that use XML or text files to define workflows, airflow uses python files for flow, and users can fully customize their own workflows through code. The main functions of airflow: workflow definition, task scheduling, task dependencies, variables, pools, distributed execution tasks, etc. The backfill function is that airflow can backfill the data of multiple tasks in the dag according to the dependencies according to the time period. The Clear function is that airflow can clear the status of some tasks according to the time period, so that a task in the dag can be rerun.

As a task scheduling system, Aiflow cannot know the abnormal situation of data collection, and can only re-run the data after the data development engineer finds the abnormal situation and writes scripts to trigger the backfill and clear functions. Time-consuming from finding the problem to solving it. In addition, the scope of the Backfill and Clear functions is within the scope of a dag, and tasks across dag cannot be re-run according to dependencies. In the actual data collection scenario, once the data is abnormal and the data is backfilled, it is necessary to rely on the experience of the data developer to manually compile the script processing, which wastes human resources.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention provide a method and apparatus for processing abnormal data in data backfilling, which can automatically process abnormal data problems in a data backfilling scenario without human intervention, greatly improving processing efficiency and reducing labor costs.

To achieve the above object, according to an aspect of the embodiments of the present invention, a method for processing abnormal data in data backfilling is provided.

The method for processing abnormal data in data backfilling according to the embodiment of the present invention includes: in the case of abnormal data, determining metadata and time field information of the abnormal data; based on the stored blood relationship data, according to the metadata of the abnormal data data, determine the target task flow and the execution order of the target task flow; wherein, the blood relationship data is at least used to express the association relationship between the data; according to the execution order and the time field information of the abnormal data, execute the target task flow.

Optionally, in the presence of abnormal data, before determining the metadata and time field information of the abnormal data, the method further includes: parsing the original data stored in the distributed file system to obtain the time of the original data. field information; determine whether the time field information of the original data conforms to the normal range condition; when the time field information of the original data does not conform to the normal range condition, it is determined that abnormal data exists in the original data.

Optionally, based on the stored blood relationship data and according to the metadata of the abnormal data, the step of determining the target task flow and the execution order of the target task flow includes: according to the time field information of the original data, The historical data is backwashed into the corresponding time partition table, and according to the time field information of the abnormal data, the target table name of the time partition table of the abnormal data is determined; wherein, the metadata includes at least the table name; based on storage According to the blood relationship data of the abnormal data, the target task flow and the execution order of the target task flow are determined according to the target table name of the abnormal data.

Optionally, the time partition table is a hive table; and/or,

After executing the target task flow according to the execution order and the time field information of the abnormal data, the method further includes: updating the data in the hive table to a corresponding database.

Optionally, after executing the target task flow according to the execution order and the time field information of the abnormal data, the method further includes: determining that there are no tasks that have not been successfully executed in the target task flow, and judging whether there are subsequent tasks. Delay data; if present, confirm the presence of abnormal data.

Optionally, the method further includes: determining that the abnormal data is duplicate data; and filtering the duplicate data.

To achieve the above object, according to another aspect of the embodiments of the present invention, an apparatus for processing abnormal data in data backfilling is provided.

The apparatus for processing abnormal data in data backfilling according to the embodiment of the present invention includes:

a metadata determination module, configured to determine metadata and time field information of the abnormal data in the presence of abnormal data;

The task flow determination module is used to determine the target task flow and the execution order of the target task flow based on the stored blood relationship data and according to the metadata of the abnormal data; wherein, the blood relationship data is at least used for expressing data. relationship between;

An execution module, configured to execute the target task flow according to the execution sequence and the time field information of the abnormal data.

Optionally, it also includes an abnormality detection module, which is used to parse the original data stored in the distributed file system to obtain the time field information of the original data; judge whether the time field information of the original data meets the normal range conditions ; When the time field information of the original data does not meet the normal range condition, it is determined that abnormal data exists in the original data.

Optionally, the task flow determination module is further configured to, according to the time field information of the original data, backwash the historical data into a corresponding time partition table, and according to the time field information of the abnormal data, Determine the target table name of the time partition table of the abnormal data; wherein, the metadata includes at least a table name; based on the stored blood relationship data, according to the target table name of the abnormal data, determine the target task flow and the target The execution order of the task flow.

Optionally, an execution result determination module is further included, configured to determine that there is no unsuccessfully executed task in the target task flow, and determine whether there is subsequent delayed data; if so, confirm the existence of abnormal data.

Optionally, a duplicate exception processing module is further included, configured to determine that the abnormal data is duplicate data; and filter the duplicate data.

To achieve the above object, according to yet another aspect of the embodiments of the present invention, an electronic device is provided.

An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more programs cause the One or more processors implement any one of the above-mentioned methods for processing abnormal data in data backfilling.

In order to achieve the above object, according to another aspect of the embodiments of the present invention, a computer-readable medium is provided on which a computer program is stored, wherein the program is executed by a processor to realize the data of any one of the above Method for handling abnormal data in backfill.

An embodiment of the above invention has the following advantages or beneficial effects: in the scenario of data backfilling, based on the blood relationship between data and tasks, determine the execution sequence of task flow and track the running status of tasks, and automatically handle abnormal data problems , without human intervention, greatly improving the processing efficiency and reducing labor costs.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic diagram of the main flow of a method for processing abnormal data in data backfilling according to an embodiment of the present invention;

2 is a schematic diagram of a method for processing abnormal data in data backfilling according to an embodiment of the present invention;

3 is a schematic diagram of main modules of an apparatus for processing abnormal data in data backfilling according to an embodiment of the present invention;

4 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 5 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

1 is a schematic diagram of the main flow of a method for processing abnormal data in data backfilling according to an embodiment of the present invention. As shown in FIG. 1 , the method for processing abnormal data in data backfilling according to an embodiment of the present invention mainly includes:

Step S101: In the case of abnormal data, determine metadata and time field information of the abnormal data.

Step S102: Determine the target task flow and the execution sequence of the target task flow according to the metadata of the abnormal data based on the stored blood relationship data; wherein the blood relationship data is at least used to express the association relationship between the data.

Step S103: Execute the target task flow according to the execution sequence and the time field information of the abnormal data.

In the embodiment of the present invention, in the scenario of data backfilling, based on the blood relationship between data and tasks, the execution order of the task flow is determined and the running status of the tasks is tracked, and abnormal data problems are automatically processed without human intervention, which greatly improves the processing efficiency. efficiency and reduce labor costs.

Preferably, in the presence of abnormal data, before determining the metadata and time field information of the abnormal data, the original data stored in the distributed file system is parsed to obtain the time field information of the original data. It is judged whether the time field information of the original data conforms to the normal range condition; when the time field information of the original data does not conform to the normal range condition, it is determined that abnormal data exists in the original data.

Preferably, in the process of determining the target task flow and the execution sequence of the target task flow based on the stored blood relationship data and the metadata of the abnormal data, the historical data is backwashed to the corresponding time partition according to the time field information of the original data In the table, and according to the time field information of the abnormal data, the target table name of the time partition table of the abnormal data is determined; wherein, the metadata includes at least the table name. Based on the stored blood relationship data, the target task flow and the execution order of the target task flow are determined according to the target table name of the abnormal data.

Preferably, the time partition table is a hive table. After executing the target task flow according to the execution order and the time field information of the abnormal data, the method further includes: updating the data in the hive table to the corresponding database.

Preferably, after executing the target task flow according to the execution order and the time field information of the abnormal data, it is determined that there are no tasks that have not been successfully executed in the target task flow, and it is determined whether there is subsequent delayed data. If so, confirm the presence of abnormal data.

Preferably, in this embodiment of the present invention, after it is determined that the abnormal data is duplicate data, the duplicate data is filtered.

In the prior art, Aiflow, as a task scheduling system, cannot actively detect the abnormal situation of data collection, and can only re-run the data after the data development engineer finds the abnormal situation and writes a script to trigger the backfill and clear functions. time and inefficiency. And, since Aiflow does not provide dependencies between dags, and the scope of the backfill and clear functions is within the scope of a dag, it is impossible to rerun tasks across dags based on dependencies. Therefore, in the data collection scenario, once the data is delayed and the data is backfilled, the correct execution order of the data backwashing task cannot be obtained according to the dependency relationship. It needs to rely on the experience of the data developer for processing, which is time-consuming and labor-intensive. And, for a specific scenario, a script for determining the sequence of the corresponding data cleaning, and one type of script is used for processing each scenario. This method also needs to write specific scripts for different cleaning scenarios, and call different processing scripts for processing. The abnormal situation of data collection is generally based on the alarm, and then the data developer is notified for processing.

In view of the above problems in the prior art, the embodiment of the present invention uses the unified data storage technology hive, and also uses the same data processing technology, so that the data flow relationship can be obtained more conveniently. Hive is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provides a complete SQL query function, which can convert SQL statements into MapReduce tasks for operation. Secondly, the model of the data warehouse is also unified. In the same basic data model for each business module, the relationship between the data flow is very clear. Finally, the same set of data cleaning templates is also used at the script level, and the same macro parameter is used for the incoming time parameter, which lays the foundation for automation when rerunning the task. On this basis, the embodiment of the present invention re-executes the historical task according to the trend of the data flow, that is, the blood relationship of the data obtained from the metadata. And update the data in the latest hive table to the OLAP engine kylin. Olap is an online online analysis application, which is a solution for multi-dimensional data analysis and query. Kylin is a MOLAP system under the Hadoop ecosystem, supporting a distributed Olap analysis engine with data volumes ranging from TB to PB. And, in the embodiment of the present invention, metadata atlas is used to associate the data flow with the tasks on the airflow, and the sequence of the data cleaning and aggregation tasks is determined according to the trend of the data flow. Among them, atlas is a scalable and extensible collection of core foundational governance services that enables enterprises to effectively and efficiently meet compliance requirements in Hadoop and allows integration with the entire enterprise data ecosystem. Apache Atlas provides open metadata management and governance capabilities for organizations to build their catalog of data assets, categorize and manage those assets, and provide collaboration capabilities around these data assets.

FIG. 2 is a schematic diagram of a method for processing abnormal data in data backfilling according to an embodiment of the present invention.

As shown in Figure 2, the original json (JavaScript ObjectNotation) file of the distributed file system hdfs is first parsed to obtain the time field information. According to the time field information, it is judged whether the time is within a reasonable range. If it is not within a reasonable range, it is abnormal. The opposite is normal. If it is abnormal, determine whether the abnormal data situation is due to the duplication of collected historical data or the delay of historical data. If the data is collected repeatedly, filter the data, and then perform the normal data cleaning process. If there is a historical data delay in the collected data, the affected time range is obtained according to the time distribution. The affected time range is the time period of the data affected by the delay, that is, the value range determined by the time field information of the delayed data, and the historical data Backwash into the corresponding time-partitioned table and get an indication of the hive table directly affected and the time range affected.

According to the affected table name, go to the blood relationship of the metadata atlas to query the corresponding task name. Re-execute the data cleaning task according to the affected time range, and wait for all tasks to be successfully executed. If some tasks fail to execute, resubmit the failed tasks for execution and wait for the successful execution. If all tasks are successfully executed, it is judged whether there are subsequent affected hive tables. If there is an affected hive table, obtain the corresponding task name from the blood relationship of the metadata atlas according to the affected table name again, and repeat the above operation until there is no next affected hive table. Update the corresponding cube in kylin based on the previous list and data of all affected tables.

In the embodiment of the present invention, the original log file in hdfs can be analyzed, and data backwashing can be performed according to the data collection situation. In use, it can be connected according to the data quality monitoring system. Once there is a problem with the data acquisition link, the data problem can be processed automatically without human intervention. This step is mainly realized by the judgment of abnormal data in the first two steps. At the same time, if the statistical logic of the data is modified, it is necessary to re-execute the tasks involved in the entire data model, and it can also be automatically tracked and run through this program. Through the use of blood relationship, it can correctly judge the execution order of tasks and track the running status of tasks, improve the processing efficiency of data problems, and liberate manpower.

3 is a schematic diagram of main modules of an apparatus for processing abnormal data in data backfilling according to an embodiment of the present invention. As shown in FIG. 3 , the apparatus 300 for processing abnormal data in data backfilling according to an embodiment of the present invention includes a metadata determination module 301, a task Flow determination module 302 and execution module 303 .

The metadata determination module 301 is configured to determine metadata and time field information of the abnormal data in the presence of abnormal data.

The task flow determination module 302 is configured to, based on the stored blood relationship data and according to the metadata of the abnormal data, determine the target task flow and the execution sequence of the target task flow; wherein the blood relationship data is at least used to express the association relationship between the data. The task flow determination module is also used to, according to the time field information of the original data, backwash the historical data into the corresponding time partition table, and determine the target table name of the time partition table of the abnormal data according to the time field information of the abnormal data; The metadata includes at least a table name; based on the stored blood relationship data, the target task flow and the execution order of the target task flow are determined according to the target table name of the abnormal data.

The execution module 303 is configured to execute the target task flow according to the execution sequence and the time field information of the exception data.

The apparatus for processing abnormal data in data backfilling according to the embodiment of the present invention further includes an abnormality detection module for analyzing the original data stored in the distributed file system to obtain time field information of the original data. And, it is judged whether the time field information of the original data conforms to the normal range condition. When the time field information of the original data does not meet the normal range condition, it is determined that abnormal data exists in the original data.

The apparatus for processing abnormal data in data backfilling according to the embodiment of the present invention further includes an execution result determination module and a repeated exception processing module. The execution result determination module is used to determine that there is no task that has not been successfully executed in the target task flow, and to determine whether there is subsequent delayed data; if so, to confirm that there is abnormal data. The repeated exception processing module is used to determine the abnormal data as repeated data; filter the repeated data.

In the embodiment of the present invention, in the scenario of data backfilling, based on the blood relationship between data and tasks, the execution order of the task flow is determined and the running status of the tasks is tracked, and abnormal data problems are automatically processed without human intervention, which greatly improves the processing efficiency. efficiency and reduce labor costs.

FIG. 4 shows an exemplary system architecture 400 of a method for processing abnormal data in data backfilling or an apparatus for processing abnormal data in data backfilling according to an embodiment of the present invention.

As shown in FIG. 4 , the system architecture 400 may include terminal devices 401 , 402 , and 403 , a network 404 and a server 405 . The network 404 is a medium used to provide a communication link between the terminal devices 401 , 402 , 403 and the server 405 . The network 404 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 401, 402, 403 to interact with the server 405 through the network 404 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 401 , 402 and 403 , such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 405 may be a server that provides various services, for example, a background management server that provides support for shopping websites browsed by the terminal devices 401 , 402 , and 403 (just an example). The background management server can analyze and process the received product information query request and other data, and feed back the processing result to the terminal device.

It should be noted that the method for processing abnormal data in data backfill provided by the embodiment of the present invention is generally performed by the server 405 , and accordingly, the apparatus for processing abnormal data in data backfill is generally set in the server 405 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 4 are merely illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Referring to FIG. 5 below, it shows a schematic structural diagram of a computer system 500 suitable for implementing a terminal device according to an embodiment of the present invention. The terminal device shown in FIG. 5 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 5, a computer system 500 includes a central processing unit (CPU) 501 which can be loaded into a random access memory (RAM) 503 according to a program stored in a read only memory (ROM) 502 or a program from a storage section 508 Instead, various appropriate actions and processes are performed. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501 , the ROM 502 , and the RAM 503 are connected to each other through a bus 504 . An input/output (I/O) interface 505 is also connected to bus 504 .

The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc. ; and a communication section 509 including a network interface card such as a LAN card, a modem, and the like. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 509 and/or installed from the removable medium 511 . When the computer program is executed by the central processing unit (CPU) 501, the above-described functions defined in the system of the present invention are performed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be provided in the processor, for example, it can be described as: a processor includes a metadata determination module, a task flow determination module and an execution module. Among them, the names of these modules do not constitute a limitation of the module itself under certain circumstances. For example, the metadata determination module can also be described as "in the presence of abnormal data, determine the metadata and time fields of abnormal data. Information Module".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes: in the presence of abnormal data, metadata and time field information for determining abnormal data ; Based on the stored blood relationship data, according to the metadata of the abnormal data, determine the execution order of the target task flow and the target task flow; wherein, the blood relationship data is at least used to express the relationship between the data; according to the execution order and the abnormal data Time field information to execute the target task flow.

In the embodiment of the present invention, in the scenario of data backfilling, based on the blood relationship between data and tasks, the execution order of the task flow is determined and the running status of the tasks is tracked, and abnormal data problems are automatically processed without human intervention, which greatly improves the processing efficiency. efficiency and reduce labor costs.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (8)

1. A method for processing abnormal data in data backfilling is characterized by comprising the following steps:

in the case of abnormal data, determining metadata and time field information of the abnormal data;

based on the stored blood relationship data, determining a target task flow and an execution sequence of the target task flow according to the metadata of the abnormal data; wherein the blood relationship data is at least used for expressing the association relationship between the data;

executing the target task flow according to the execution sequence and the time field information of the abnormal data,

the step of determining a target task flow and an execution sequence of the target task flow according to the metadata of the abnormal data based on the stored blood relationship data comprises the following steps:

according to time field information of original data stored in a distributed file system, backwashing historical data into a corresponding time partition table, and according to the time field information of abnormal data, determining a target table name of the time partition table of the abnormal data; wherein the metadata includes at least a table name;

and determining a target task flow and an execution sequence of the target task flow according to the target table name of the abnormal data based on the stored blood relationship data.

2. The method of claim 1, wherein in the presence of anomalous data, prior to determining metadata and time field information for the anomalous data, further comprising:

analyzing original data stored in a distributed file system to obtain time field information of the original data;

judging whether the time field information of the original data meets the condition of a normal range or not; and when the time field information of the original data does not meet the normal range condition, determining that abnormal data exists in the original data.

3. The method of claim 1, wherein the partition schedule is a hive schedule; and/or the presence of a gas in the gas,

after executing the target task flow according to the execution sequence and the time field information of the abnormal data, the method further comprises the following steps: and updating the data in the hive table to a corresponding database.

4. An apparatus for handling exception data during data backfill, comprising:

the metadata determining module is used for determining the metadata and the time field information of the abnormal data under the condition that the abnormal data exists;

the task flow determining module is used for determining a target task flow and an execution sequence of the target task flow according to the metadata of the abnormal data based on the stored blood relationship data; wherein the blood relationship data is at least used for expressing the association relationship between the data;

an execution module for executing the target task flow according to the execution sequence and the time field information of the abnormal data,

the task flow determining module is further used for washing back historical data into a corresponding time partition table according to time field information of original data stored in the distributed file system, and determining a target table name of the time partition table of the abnormal data according to the time field information of the abnormal data; wherein the metadata includes at least a table name; and determining a target task flow and an execution sequence of the target task flow according to the target table name of the abnormal data based on the stored blood relationship data.

5. The apparatus of claim 4, further comprising an anomaly detection module, configured to parse raw data stored in the distributed file system to obtain time field information of the raw data; judging whether the time field information of the original data meets the condition of a normal range or not; and when the time field information of the original data does not meet the normal range condition, determining that abnormal data exists in the original data.

6. The apparatus according to claim 4, further comprising an execution result determining module, configured to determine that there is no task that is not successfully executed in the target task stream, and determine whether there is subsequent delay data; if so, confirming the existence of abnormal data.

7. An electronic device, comprising:

one or more processors;

a storage device to store one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-3.

8. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-3.

Images