CN102541942B - Data bulk transfer system and method thereof - Google Patents

Abstract

The invention provides a data batch transfer system and method thereof. The system includes a trigger module, which is used to judge whether the trigger condition is satisfied, and if it is satisfied, then generate and send the task descriptor to the intermediate conversion module, otherwise continue to read the indicators in the source database; The database reads the source data and processes it to generate target data, determines the target data parameters according to the task descriptor, and sends the target data and target data parameters to the writing module; the writing module writes the target data in a predetermined data format target database. The invention not only realizes loosely coupled batch data transfer, but also has the characteristics of being fast, supporting multi-thread concurrent processing, and the like.

Description

A data batch transfer system and method thereof

technical field

The invention relates to a data transfer system and method, in particular to a data batch transfer system and method.

Background technique

At present, the general method of ETL (Extraction-Transform-Load, that is, extraction-transformation-loading) data extraction is to target the target data requirements, extract the available data resources that meet the transformation rules in the source database, and analyze the available data resources in the existing source system. The relationship between data resources and target data requirements, and formulate the whole process of data extraction, write ETL code according to the fixed extraction process, and complete the extraction.

The advantage of this method is that it is easy to implement and develop quickly. The disadvantage is that the code is dedicated and only developed for a certain data requirement. Once the data requirement changes, the dedicated code cannot continue to provide the changed target data and requires developers. Recompile the entire program. In the stage of system change where data extraction and requirements change frequently, developers have a lot of development work. Since the mode of each data transfer is basically the same, developers repeat work and inefficiency.

Taking the batch transfer of public parameters of a financial institution as an example, the daily public parameter library needs to deliver more than 60 public parameter tables to 16 subsystems, and basically all of them have certain business rules to copy the data source to the target database Although there are differences in these processes, there are many overlaps, and the core differences before and after the transfer are mainly reflected in: 1. Trigger conditions; 2. Business caliber; 3. Extract follow-up processing. At the same time, the deployment and scheduling of so many data interfaces has become a difficult problem and is very complicated.

Contents of the invention

In view of this, the purpose of this invention is to provide a system and method for bulk transfer of data, which will analyze the seven basic elements of data extraction, namely transfer trigger conditions, preprocessing, input, output, conversion rules, processing procedures, and Subsequent processing requests from subsystems are modularized, so as to apply the idea of batch processing in the process of data extraction and data transfer, and carry out data-driven rather than process-driven; establish a loosely coupled and reusable process configuration mechanism to facilitate Manage and control task scheduling, cope with frequently changing business standards and data structures, and effectively solve the existing problems of complex data interfaces, inconvenient deployment and scheduling, and frequent changes in data transfer requirements.

To achieve the above object, the present invention provides a batch data transfer system, comprising:

The trigger module is used to judge whether the trigger condition is satisfied, if it is satisfied, generate and send the task descriptor to the intermediate conversion module, otherwise continue to read the indicators in the source database;

A conversion module, configured to read source data from the source database according to the task descriptor and process it to generate target data, determine target data parameters according to the task descriptor, and send the target data and the target data parameters to the writing module;

The writing module writes the target data into the target database according to the predetermined data format.

Preferably, in the system of the present invention, the intermediate conversion module includes:

The preprocessing module is used to detect whether the system satisfies the conditions for executing data batch transfer; if it is satisfied, data processing is performed, otherwise an alarm signal is sent;

The source data interface module determines the source data parameters according to the task descriptor and sends the source data parameters to the processor;

The target data interface module determines the target data parameters according to the task descriptor and sends the target data parameters to the writing module;

A conversion rule module, which determines the conversion rule according to the task descriptor and sends it to the processor;

The processor reads the source data from the source database according to the source data parameters; processes the source data according to the conversion rules to generate target data, and sends the target data to the writing module.

Preferably, in the system of the present invention, the processor is further configured to send target data to the subsystem in response to a data request from the subsystem.

Preferably, in the system of the present invention, the source data parameters include the owner of the source data table, the name of the source data table, and the number of sets of the source data table.

Preferably, in the system of the present invention, the target data parameters include the owner of the target data table, the name of the target data table, and the number of sets of the target data table.

Preferably, in the system of the present invention, the conversion rules are selected from the group consisting of: data normalization, filling in default values, data integration, data grouping, data association, data merging, sorting and function operations.

The present invention also provides a batch data transfer method, comprising the following steps:

A. Determine whether the trigger condition is met, if so, generate and send the task descriptor, otherwise continue to read the indicators in the source database;

B. Read the source data from the source database according to the task descriptor and process it to generate target data, and determine the target data parameters according to the task descriptor;

C. Write the target data into the target database according to the predetermined data format according to the target data parameters.

Preferably, in the method of the present invention, step B further comprises the following steps:

Detect whether the system meets the initialization conditions for performing batch data transfer; if so, proceed to the next steps, otherwise send an alarm signal

determining source data parameters according to the task descriptor;

determining target data parameters according to the task descriptor;

Determine transformation rules according to task descriptors;

Read the source data from the source database according to the source data parameters;

According to the transformation rules, the source data is processed to generate the target data.

Preferably, in the method of the present invention, the method also includes the following steps:

In response to the data request sent by the subsystem, the target data is sent to the subsystem.

Preferably, in the method of the present invention, the source data parameters include the owner of the source data table, the name of the source data table, and the number of sets of the source data table.

Preferably, in the method of the present invention, the target data parameters include the owner of the target data table, the name of the target data table, and the number of sets of the target data table.

Preferably, in the method of the present invention, the conversion rules are selected from the group consisting of: data normalization, filling in default values, data integration, data grouping, data association, data merging, sorting and function operations.

The technical effect of the present invention is that the workload of coding is effectively reduced, so that the impact on the batch transfer itself is reduced every time the subsystem is changed, and at the same time, the performance of the subsystem itself is not greatly affected. This design not only realizes loosely coupled batch data transfer, but also has the characteristics of fast speed and support for multi-threaded concurrent processing.

Description of drawings

1 is a schematic diagram of a data batch transfer system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a data bulk transfer method according to an embodiment of the present invention.

Detailed ways

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, in which the same reference numerals denote the same elements.

FIG. 1 is a schematic diagram of a data batch transfer system according to an embodiment of the present invention. As shown in the figure, the system includes a trigger module 1 , a conversion module 2 and a writing module 3 .

Trigger module 1, reads the indicators in the source database and judges whether the trigger conditions are met, and if so, generates and sends a task descriptor to the intermediate conversion module, otherwise continues to read the indicators in the source database. The index is, for example but not limited to, the data volume of the source system and a certain flag in the database of the source system. Correspondingly, the trigger condition is, for example but not limited to, whether the data volume of the source system reaches a predetermined value, and whether the flag bit is valid. Those skilled in the art can define indicators and trigger conditions to be read according to actual needs.

The conversion module 2 is used to read the source data from the source database 4 according to the task descriptor and process it to generate target data, determine the target data parameters according to the task descriptor and send the target data and the target data parameters to the writing module 3 ;

Further, the conversion module 2 includes a preprocessing module 20 , a source data interface module 21 , a conversion rule module 22 , a target data interface module 23 and a processor 24 .

The preprocessing module 20, when receiving the task descriptor, is used to detect whether the system satisfies the conditions for executing data batch transfer; if so, data processing is performed; otherwise, an alarm signal is sent. The preprocessing module 20, for example, checks whether the system satisfies the conditions for performing batch transfer of data through the initialization settings of the extraction tasks, including but not limited to self-status check and monitoring and cleaning of the operating environment. Those skilled in the art can make settings according to actual needs.

The source data interface module 21 receives the task descriptor from the trigger module 1, parses the task descriptor to determine the source data parameters, and sends the source data parameters to the processor. The source data parameters include the owner of the source data table, the name of the source data table, and the number of sets of the source data table.

The target data interface module 23 receives the task descriptor from the trigger module 1, parses the task descriptor to determine the target data parameter, and sends the target data parameter to the writing module. The target data parameters include the owner of the target data table, the name of the target data table, and the number of sets of the target data table.

The conversion rule module 22 receives the task descriptor from the trigger module 1 , parses the task descriptor to determine a conversion rule and sends it to the processor 24 . Data conversion rules generally follow the following basic rules:

Target data = Transformation (source data input field 1, source data input field 2, . . . ). The conversion rules are, for example but not limited to, data normalization, filling in default values, data integration, data grouping, data association, data merging, sorting and function operations. The meanings of the conversion rules are as follows:

Data normalization (Expression): convert data definitions in the source system that have the same essential meaning but different descriptions into a common, unified form that can be recognized by downstream subsystems.

Fill in the default default value: When a definition does not exist in the source system, new data will be generated according to the agreement. For example: generate sequence (Sequence Generator):

Data Integration (Aggregator): To achieve the aggregation of data. When the source system is based on business flow, coarser-grained summary statistics can be generated through data integration.

Data grouping (Router): realize the distribution of data. The source data can be read once and distributed to multiple subsystems, reducing the pressure on the source system and saving transmission costs.

Data association (Joiner): to achieve data merging.

Sorting (Sorter): To achieve the specified sorting of data.

Function operation: Some special function operations, such as numerical modulo, a specific time, a subset of characters and so on.

The processor 24 reads the source data from the source data 4 library according to the source data parameters; processes the source data according to the conversion rules to generate target data, and sends the target data to the writing module 3 .

The writing module 3 writes the target data into the target database 5 according to a predetermined data format according to the target data parameters. The writing module 3 is, for example but not limited to, an ETL extraction transformation engine.

Preferably, in the present invention, the processor 24 is further configured to send the target data to the subsystem 6 in response to the data request from the subsystem 6 .

FIG. 2 is a schematic diagram of a data bulk transfer method according to an embodiment of the present invention. The method includes the following steps:

A. Read the source system indicators and judge whether the trigger conditions are met. If so, generate and send a task descriptor, otherwise continue to read the source system indicators;

B. Read the source data from the source database according to the task descriptor and process it to generate target data, and determine the target data parameters according to the task descriptor;

C. Write the target data into the target database according to the predetermined data format according to the target data parameters.

Wherein, step B further comprises the following steps:

Detect whether the system meets the initialization conditions for performing batch data transfer; if so, proceed to the next steps, otherwise send an alarm signal

Determine the source data parameters according to the task descriptor, where the source data parameters include the owner of the source data table, the name of the source data table, and the number of sets of the source data table;

Determine the target data parameters according to the task descriptor, wherein the target data parameters include the owner of the target data table, the name of the target data table, and the number of sets of the target data table;

determining transformation rules based on the task descriptor, wherein the transformation rules are selected from the group consisting of: data normalization, filling default default values, data integration, data grouping, data association, data merging, sorting, and function operations;

Read the source data from the source database according to the source data parameters;

According to the transformation rules, the source data is processed to generate the target data.

Preferably, the batch data transfer method of the present invention further includes the step of sending the target data to the subsystem in response to the data request sent by the subsystem.

In view of these teachings, other embodiments, combinations and modifications of the invention will readily occur to those skilled in the art. Accordingly, the invention is only limited by the appended claims when read in conjunction with the foregoing description and accompanying drawings.

Claims (1)

1. a batch data transfer system, is characterized in that, comprising:

Trigger module, for reading the index of source database and judging whether to meet trigger condition, if met, generates and sends task descriptor to modular converter, otherwise continues to read the index in source database;

Modular converter, for reading source data and it is processed to generate target data from source database according to task descriptor, determines target data parameter and target data and target data parameter is sent to writing module according to task descriptor;

Writing module, writes target database by target data according to tentation data form, and wherein, described modular converter comprises:

Whether pretreatment module, meet for detection of system the condition that executing data shifts in batches; If met, carry out data processing, otherwise send alerting signal;

Source data interface module, determines source data parameter and source data parameter is sent to processing module according to task descriptor;

Target data interface module, determines target data parameter and target data parameter is sent to writing module according to task descriptor;

Transformation rule module, determines transformation rule and is sent to processing module according to task descriptor;

Processing module, reads source data according to source data parameter from source database; According to transformation rule, source data is processed and generated target data, target data is sent to writing module.

2. the system as claimed in claim 1, is characterized in that, described processing module is further configured to the request of data of response from subsystem, and target data is sent to this subsystem.

3. system as claimed in claim 1 or 2, is characterized in that, described source data parameter comprises source data table owner, source data table name, source data table tricks.

4. system as claimed in claim 1 or 2, is characterized in that, described target data parameter comprises target matrix owner, target data table name, target matrix tricks.

5. system as claimed in claim 1 or 2, is characterized in that, described transformation rule selects the group of free the following composition: data normalization, fill in that default default value, data integration, packet, data correlation, data merge, Scheduling Sum function computing.

6. a batch data transfer method, is characterized in that, comprises the following steps:

A, read the index in source database and judge whether to meet trigger condition, if met, generate and send task descriptor, otherwise continue to read the index in source database;

B, read source data and it is processed to generate target data from source database according to task descriptor, determine target data parameter according to task descriptor;

C, according to target data parameter, target data is write to target database according to tentation data form, wherein, step B is further comprising the steps:

Whether detection system meets the initialization condition that executing data shifts in batches; If met, carry out subsequent step, otherwise send alerting signal

Determine source data parameter according to task descriptor;

Determine target data parameter according to task descriptor;

Determine transformation rule according to task descriptor;

Read source data according to source data parameter from source database;

According to transformation rule, source data is processed and generated target data.

7. method as claimed in claim 6, is characterized in that, described method is further comprising the steps of:

The request of data that response subsystem sends, is sent to described subsystem by target data.

8. method as claimed in claim 6, is characterized in that, described source data parameter comprises source data table owner, source data table name, source data table tricks.

9. method as claimed in claim 6, is characterized in that, described target data parameter comprises target matrix owner, target data table name, target matrix tricks.

10. method as claimed in claim 6, is characterized in that, described transformation rule selects the group of free the following composition: data normalization, fill in that default default value, data integration, packet, data correlation, data merge, Scheduling Sum function computing.