CN111897877A - High-performance and high-reliability data sharing system and method based on distributed idea - Google Patents

Abstract

The invention discloses a high-performance and high-reliability data sharing system and method based on a distributed idea, belonging to the field of information technology. The intuitive and shared content is difficult to monitor. The technical scheme is as follows: the system includes a database connection management unit, a timed task management unit, a task log management unit, a task statistics unit, a task monitoring unit and a failure reporting unit. The method is as follows: S1, configure and start the data sharing task, and the task regularly extracts data from the source database and through the defined rules to form a data text file, compresses and encrypts the file, uploads it to the document center and records the access link; S2, provides data to the outside world Sharing interface, the user obtains the download link through the data sharing interface; S3, call the download link to go to the document center to download the encrypted file, and decrypt it through the pre-allocated key to obtain the data.

Description

High-performance and high-reliability data sharing system and method based on distributed idea

technical field

The invention relates to the field of information technology, mainly relates to efficient and safe data extraction and data sharing technology among multiple information systems, in particular to a high-performance and high-reliability data sharing system and method based on distributed thinking.

Background technique

At present, the informatization construction of various enterprises has reached a certain scale, and a large amount of data with extremely high value has been accumulated. Data is an important asset of an enterprise and the soul of the enterprise's survival and sustainable development. Usually, various system developers ( There are various reasons among the hereinafter referred to as developers) that cause these data resources to be monopolized by each developer. However, the analysis of data within the enterprise requires the openness of some data among different developers to cooperate. In order to enable various developers to share data conveniently and quickly, companies generally use three ways to share data:

The first method: Enterprises usually need to coordinate various developers to share data in the form of interfaces. This method can solve a small amount of data sharing problem, but it is helpless for data with a relatively large amount of data. For example, it is necessary to synchronize the sales order data of retail terminals in the national tobacco industry (about 10 million orders per day) to the tobacco marketing analysis system every day, and ordinary methods are stretched. Moreover, the interface is generally developed and held by each developer, and enterprises generally cannot manage and monitor the shared data in real time.

The second way: Enterprises need to coordinate with developers to open the database read-only permissions to achieve the purpose of data sharing. In this way, the detailed design of the database table structure needs to be disclosed so that other manufacturers can extract the data they need. However, this method is easy to leak the underlying design of the developer database, which is not only insecure but also unfair to the developers involved in data sharing.

The third way: Enterprises need to coordinate with developers and use existing data extraction tools such as ETL to extract the data that needs to be shared to a separate shared database. This method is labor-intensive, consumes a lot of human resources, and is unsafe and unintuitive.

To sum up, the existing technology has the following problems: for a long time, it is difficult to share data within enterprises, large-scale data is difficult to share, traditional sharing methods are inefficient and insecure, and at the same time, data sharing is difficult to manage, unintuitive in sharing methods, and difficult to share content. monitor.

SUMMARY OF THE INVENTION

The technical task of the present invention is to provide a high-performance and high-reliability data sharing system and method based on the distributed idea, to solve the problem of low efficiency and insecurity of large-scale data sharing, and at the same time, the problems of difficult management of data sharing, unintuitive sharing methods and sharing content difficult to monitor.

The technical task of the present invention is achieved in the following manner, a high-performance and high-reliability data sharing system based on distributed thinking, the data sharing system includes,

The database connection management unit is used to add the source database to be shared and modify or delete the database connection;

The timing task management unit is used to add, modify, start, stop, delete and manually compensate for timing tasks;

The task log management unit is used to view the detailed execution process of each historical task, the execution process of each historical task and the storage location of the generated encrypted data files in the document center, and also provides viewing SQL, delete logs and advanced operations;

The task statistics unit is used to perform statistics on the execution process of each task;

The task monitoring unit is used to monitor any task in real time, view and analyze the task running status in the form of a visual chart, and provide administrators with a full range of data sharing task monitoring information;

The failure reporting unit is used to generate a report on the failed task, and the administrator can intuitively see the link where the failed task has a problem.

Preferably, the database connection management unit includes:

The new module of database connection is used to add the source database to be shared through the database connection management function. At the same time, it provides a test connection button to test the database connectivity at any time and manage various database connections;

The database connection modification and deletion module is used to modify or delete the existing database connection at any time.

More preferably, the newly added content of the source database to be shared includes connection name, connection address, connection port number, database name, database user name, database password and database type.

Preferably, the timed task management unit includes:

A new module for timed tasks is used to add timed tasks through the new function of timed task management;

The timing task modification module is used to modify the existing data sharing timing tasks;

The timing task start, stop or delete module is used to start, stop or delete the existing data sharing timing tasks accordingly;

The manual compensation task module is used to quickly and freely perform manual compensation on historical data.

More preferably, the information to be maintained for the newly added scheduled task includes task name, agreed table name, connection name, cron expression, task step size, extraction format, extraction start time, time offset, and key type. , SQL type, task SQL and full name of timestamp field;

Among them, the agreed table name is used to protect the strategy designed by the developer's underlying design, and the developer can fill in this field at will;

The cron expression refers to the time instruction expression commonly used in the Linux operating system;

The task step length refers to how long the data needs to be fetched from the source database when the task is executed, measured in minutes;

Time offset refers to how long the task step needs to be shifted forward, which is used to solve the problem of data storage delay when using database transactions to process complex services;

Key type means that different public and private key pairs can be selected for encryption when encrypting files;

The SQL type allows the system administrator to tell the system the parameter filling strategy when executing the SQL task, including automatic splicing of parameters and manual definition of parameters;

Task SQL refers to a structured query language for data sharing tasks;

The full name of the timestamp field is used to solve the problem of various developers' timestamp field names, and it is also used to indicate which field in the task sql represents the timestamp.

Preferably, the task log management unit includes:

The task log clearing module is used to clear all the history of the selected task and clear the unnecessary history records;

The SQL viewing module is used to view the specific execution SQL of the selected task and assist in analyzing the problems in the data sharing process;

The log deletion module is used to delete task logs one by one;

The advanced operation module is used to provide more advanced operations for log management.

Preferably, the content counted by the task statistics unit includes the number of shared data rows, the size of the shared data text file, the size of the shared data encrypted file, and the duration of connection to the database.

Preferably, the data sharing system provides two working modes, as follows:

1. Single Master mode: The data sharing system works on one server, and will automatically simulate multiple slave sub-nodes through multi-threading;

2. Master-Slave mode: manually split the data sharing system into Master and Slave, in which the Master node works on one server, the Slave node works on other servers, and the MQ message intermediate is used between the Master node and the Slave node files to communicate.

A high-performance and high-reliability data sharing method based on distributed thinking, the method is as follows:

S1. The system administrator configures and starts the data sharing task, and the task regularly extracts data from the source database and through the defined rules to form a data text file, compresses and encrypts the file and uploads it to the document center (internal unstructured database management system of the enterprise) and record access links;

S2. The data sharing system provides a data sharing interface externally, and the user obtains the download link through the data sharing interface;

S3. Go to the document center to download the encrypted file by calling the download link, and decrypt the encrypted file through the pre-allocated key to obtain the data.

Preferably, the data sharing system can automatically split a shared task with a huge amount of data according to the time dimension according to the task step size set by the administrator, and perform decentralized execution to reduce the pressure on the source database;

The data obtained by the data sharing system from the source database forms a text file, which is encrypted by the RSA asymmetric encryption algorithm to form an encrypted file, and the encrypted file is sent to the document center for storage. Users who need to share data directly download the data from the document center through the link, thereby Pressure to shift data sharing systems;

Users who need to share data need to obtain the target data in two steps, as follows:

①, first call the interface provided by the data sharing system;

②. Call the interface of the document center to obtain the encrypted file according to the result obtained in the first step.

The high-performance and high-reliability data sharing system and method based on the distributed idea of the present invention has the following advantages:

(1) The present invention provides a brand-new solution for internal data sharing of enterprises, breaks the barriers of traditional data sharing difficulties, the sharing method is very intuitive, the sharing process is efficient, safe and reliable, it can realize convenient and efficient management and sharing tasks, and can achieve Real-time monitoring of data sharing tasks can save a lot of human resource costs and provide strong support for data sharing within enterprises, especially the rapid sharing of ultra-large-scale data;

(2) The present invention can effectively solve the problem of data sharing within enterprises, especially the problem of large-scale data sharing, and can manage data sharing tasks conveniently and quickly, efficiently and safely achieve large-scale data sharing, Provide safe and efficient solutions for data sharing within enterprises, especially the sharing of large amounts of data;

(3) In order to reduce the access pressure on the source database in the process of data aggregation and avoid affecting the normal operation of other business systems, the present invention divides a large amount of data sharing tasks into N small data sharing tasks, which can be connected through a database connection pool. Database, perform a data extraction task every corresponding time, extract data of step size (time length) once, the extracted data are compressed and encrypted and uploaded to the document center for storage;

(4) Go to Maning has the following features in addition to the features of common data extraction and sharing tools:

①Security: The invocation of the data sharing interface requires the platform to grant the developer permission to call, which ensures the security of the interface; the data files are stored in the document center in the form of RSA encryption, and the developer must apply for the RSA private key to obtain the RSA private key. Decrypt data files;

②High performance: Using the distributed idea, based on the Master-Slave mode (one master and multiple slaves), the master node is only used to generate and dispatch tasks, and the slave nodes are used to process these tasks. There can be multiple slave nodes, which can handle large-scale data extraction and sharing tasks at the same time;

③High availability: The task dispatched by the master node is stored in the message middleware (hereinafter referred to as MQ). At the same time, the slave node has multiple nodes and is deployed on different machines to ensure the high availability of the system;

④Convenient management: through the present invention, the system administrator can add data sharing tasks at any time, modify data extraction rules at any time, start data sharing tasks at any time, stop data sharing tasks at any time, and delete data sharing tasks at any time;

⑤Monitorable statistics: the present invention can view the running status of tasks in real time, monitor the success and failure of tasks; meanwhile, provide various dimension task statistical reports and visual charts, and grasp the running status of tasks in real time;

⑥ Compensable: For failed tasks, the present invention has both automatic compensation and manual compensation functions. When a task failure is detected, the compensation mechanism will be activated immediately to compensate for the failed task. At the same time, the administrator can also perform compensation manually. Easy and fast operation;

⑦ Depressurization of the source database: The present invention provides the ability to split the same data sharing task, divides a data sharing task into n multiple small tasks, realizes time sharing, and reads data from the database in sections, so as to avoid pressure on the source database , while improving the speed of data extraction and sharing;

⑧Depressurization of the data sharing system: The data obtained from the source database is processed and sent to the document center for storage. The system developers who need to share the data directly download the data from the document center through the link, thereby transferring the pressure of the data sharing system.

Description of drawings

The present invention will be further described below with reference to the accompanying drawings.

Accompanying drawing 1 is the structural block diagram of the high-performance and high-reliability data sharing system based on distributed thought;

Accompanying drawing 2 is the framework diagram of the working principle of data sharing system in accompanying drawing 1;

Accompanying drawing 3 is the interface screenshot of new database connection;

Accompanying drawing 4 is the interface screenshot one of newly added timing task module;

Accompanying drawing 5 is the interface screenshot two of the newly added timing task module;

Accompanying drawing 6 is the interface screenshot one of task log management unit;

Accompanying drawing 7 is the interface screenshot two of task log management unit;

Accompanying drawing 8 is the interface screenshot of task statistics unit;

Accompanying drawing 9 is the interface screenshot of the line graph of task monitoring unit;

Accompanying drawing 10 is the interface screenshot of the pie chart of task monitoring unit;

Figure 11 is a screenshot of the interface of the failure report.

Detailed ways

The high-performance and high-reliability data sharing system and method based on the distributed idea of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Example 1:

As shown in FIG. 1, the high-performance and high-reliability data sharing system based on the distributed idea of the present invention includes:

The database connection management unit is used to add the source database to be shared and modify or delete the database connection; the database connection management unit includes,

The database connection new module is used to add the source database to be shared through the database connection management function, and provides a test connection button at the same time to test the database connectivity and manage various database connections at any time, as shown in Figure 3;

The database connection modification and deletion module is used to modify or delete the existing database connection at any time.

The newly added content of the source database to be shared includes connection name, connection address, connection port number, database name, database user name, database password, and database type.

The timed task management unit is used to add, modify, start, stop, delete and manually compensate timed tasks; the timed task management unit includes,

A new module for timed tasks is used to add timed tasks through the new functions of timed task management; the information to be maintained for new timed tasks includes task name, agreed table name, connection name, cron expression, task step size, and extraction time. Format, extraction start time, time offset, key type, SQL type, task SQL, and full name of the timestamp field, as shown in Figures 4 and 5;

Among them, the agreed table name is used to protect the strategy designed by the developer's underlying design, and the developer can fill in this field at will;

The cron expression refers to the time instruction expression commonly used in the Linux operating system;

The task step length refers to how long the data needs to be fetched from the source database when the task is executed, measured in minutes;

Time offset refers to how long the task step needs to be shifted forward, which is used to solve the problem of data storage delay when using database transactions to process complex services;

Key type means that different public and private key pairs can be selected for encryption when encrypting files;

The SQL type allows the system administrator to tell the system the parameter filling strategy when executing the SQL task, including automatic splicing of parameters and manual definition of parameters;

Task SQL refers to a structured query language for data sharing tasks;

The full name of the timestamp field is used to solve the problem of various developers' timestamp field names, and it is also used to indicate which field in the task sql represents the timestamp.

The timing task modification module is used to modify the existing data sharing timing tasks;

The timing task start, stop or delete module is used to start, stop or delete the existing data sharing timing tasks accordingly;

The manual compensation task module is used to quickly and freely perform manual compensation on historical data.

The task log management unit is used to view the detailed execution process of each historical task, the execution process of each historical task, and the storage location of the generated encrypted data files in the document center. As shown in Figures 6 and 7; the task log management unit includes,

The task log clearing module is used to clear all the history of the selected task and clear the unnecessary history records;

The SQL viewing module is used to view the specific execution SQL of the selected task and assist in analyzing the problems in the data sharing process;

The log deletion module is used to delete task logs one by one;

The advanced operation module is used to provide more advanced operations for log management.

The task statistics unit is used to perform statistics on each task execution process, as shown in Figure 8; the content of the task statistics unit statistics includes the number of shared data lines, the size of the shared data text file, the size of the shared data encrypted file and the length of the connection database .

The task monitoring unit is used to monitor any task in real time, view and analyze the task running status in the form of a visual chart, and provide administrators with all-round data sharing task monitoring information, as shown in Figures 9 and 10;

The failure reporting unit is used to generate a report on the failed task, and the administrator can intuitively see the link where the failed task has a problem, as shown in FIG. 11 .

As shown in Figure 2, the data sharing system provides two working modes, as follows:

1. Single Master mode: The data sharing system works on one server, and will automatically simulate multiple slave sub-nodes through multi-threading;

2. Master-Slave mode: manually split the data sharing system into Master and Slave, in which the Master node works on one server, the Slave node works on multiple other servers, and the MQ message intermediate is used between the Master node and the Slave node files to communicate.

The working process of the data sharing system is as follows:

(1), the system administrator logs in, and adds the database to be shared data through the database management function;

(2), configure timed tasks and data sharing rules and start the task;

(3) The data sharing system will regularly extract data from the source database and form a data text file through the defined rules according to the configuration of the system administrator;

(4) After the data text files are formed, the data sharing system will archive and compress these text files and encrypt them through the RSA asymmetric algorithm to form encrypted files, and then the system will upload the encrypted files to the internal unstructured database management system of the enterprise (referred to as document for short). Center) and record access links;

(4) The data sharing system provides a data sharing interface externally, and developers can obtain the download link through the data sharing system interface;

(5) Call the link to download the encrypted file from the document center, and decrypt it with the pre-allocated key to obtain the data.

The present invention is applicable to the retail industry.

Example 2:

The high-performance and high-reliability data sharing method based on the distributed idea of the present invention is specifically as follows:

S1. The system administrator configures and starts the data sharing task, and the task regularly extracts data from the source database and through the defined rules to form a data text file, compresses and encrypts the file and uploads it to the document center (internal unstructured database management system of the enterprise) and record access links;

S2. The data sharing system provides a data sharing interface externally, and the user obtains the download link through the data sharing interface;

S3. Go to the document center to download the encrypted file by calling the download link, and decrypt the encrypted file through the pre-allocated key to obtain the data.

Among them, the data sharing system can automatically split a shared task with a huge amount of data according to the time dimension according to the task step set by the administrator, and execute it in a decentralized manner, reducing the pressure on the source database;

The data obtained by the data sharing system from the source database forms a text file, which is encrypted by the RSA asymmetric encryption algorithm to form an encrypted file, and then the encrypted file is sent to the document center for storage. Users who need to share data directly download the data from the document center through the link, thereby Pressure to shift data sharing systems;

Users who need to share data need to obtain the target data in two steps, as follows:

①, first call the interface provided by the data sharing system;

②. Call the interface of the document center to obtain the encrypted file according to the result obtained in the first step.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (10)

1. a high-performance and high-reliability data sharing system based on distributed thinking, is characterized in that, this data sharing system comprises, The database connection management unit is used to add the source database to be shared and modify or delete the database connection; The timing task management unit is used to add, modify, start, stop, delete and manually compensate for timing tasks; The task log management unit is used to view the detailed execution process of each historical task, the execution process of each historical task and the storage location of the generated encrypted data files in the document center, and also provides viewing SQL, delete logs and advanced operations; The task statistics unit is used to perform statistics on the execution process of each task; The task monitoring unit is used to monitor any task in real time, view and analyze the task running status in the form of a visual chart, and provide administrators with a full range of data sharing task monitoring information; The failure reporting unit is used to generate a report on the failed task, and the administrator can intuitively see the link where the failed task has a problem. 2. The high-performance and high-reliability data sharing system based on distributed thinking according to claim 1, wherein the database connection management unit comprises: The new module of database connection is used to add the source database to be shared through the database connection management function. At the same time, it provides a test connection button to test the database connectivity at any time and manage various database connections; The database connection modification and deletion module is used to modify or delete the existing database connection at any time. 3. The high-performance and high-reliability data sharing system based on distributed thinking according to claim 2, wherein the newly added content of the source database to be shared comprises connection name, connection address, connection port number, database name , database username, database password, and database type. 4. The high-performance and high-reliability data sharing system based on distributed thinking according to claim 1, wherein the timing task management unit comprises: A new module for timed tasks is used to add timed tasks through the new function of timed task management; The timing task modification module is used to modify the existing data sharing timing tasks; The timing task start, stop or delete module is used to start, stop or delete the existing data sharing timing tasks accordingly; The manual compensation task module is used to quickly and freely perform manual compensation on historical data. 5. The high-performance and high-reliability data sharing system based on distributed thinking according to claim 4, characterized in that, the information that the newly added scheduled task needs to maintain includes task name, convention table name, connection name, cron expression , task step size, extraction format, extraction start time, time offset, key type, SQL type, task SQL, and full name of the timestamp field; Among them, the agreed table name is used to protect the strategy designed by the developer's underlying design, and the developer can fill in this field at will; The cron expression refers to the time instruction expression commonly used in the Linux operating system; The task step length refers to how long the data needs to be fetched from the source database when the task is executed, measured in minutes; Time offset refers to how long the task step needs to be shifted forward, which is used to solve the problem of data storage delay when using database transactions to process complex services; Key type means that different public and private key pairs can be selected for encryption when encrypting files; The SQL type allows the system administrator to tell the system the parameter filling strategy when executing the SQL task, including automatic splicing of parameters and manual definition of parameters; Task SQL refers to a structured query language for data sharing tasks; The full name of the timestamp field is used to solve the problem of various developers' timestamp field names, and it is also used to indicate which field in the task sql represents the timestamp. 6. The high-performance and high-reliability data sharing system based on distributed thinking according to claim 1, wherein the task log management unit comprises: The task log clearing module is used to clear all the history of the selected task and clear the unnecessary history records; The SQL viewing module is used to view the specific execution SQL of the selected task and assist in analyzing the problems in the data sharing process; The log deletion module is used to delete task logs one by one; The advanced operation module is used to provide more advanced operations for log management. 7. The high-performance and high-reliability data sharing system based on distributed thinking according to claim 1, wherein the content of the statistics of the task statistics unit includes the number of shared data lines, the size of the shared data text file, and the shared data encryption. File size and time to connect to the database. 8. The high-performance and high-reliability data sharing system based on distributed thinking according to claim 1, is characterized in that, this data sharing system provides two kinds of working modes, and is specifically as follows: 1. Single Master mode: The data sharing system works on one server, and will automatically simulate multiple slave sub-nodes through multi-threading; 2. Master-Slave mode: manually split the data sharing system into Master and Slave, in which the Master node works on one server, the Slave node works on other servers, and the MQ message intermediate is used between the Master node and the Slave node files to communicate. 9. A high-performance and high-reliability data sharing method based on distributed thinking, characterized in that the method is specifically as follows: S1. The system administrator configures and starts the data sharing task, and the task regularly extracts data from the source database and through the defined rules to form a data text file, compresses and encrypts the file, uploads it to the document center and records the access link; S2. The data sharing system provides a data sharing interface externally, and the user obtains the download link through the data sharing interface; S3. Go to the document center to download the encrypted file by calling the download link, and decrypt the encrypted file through the pre-allocated key to obtain the data. 10. The high-performance and high-reliability data sharing method based on distributed thinking according to claim 9, wherein the data sharing system can divide a shared task with a huge amount of data by time according to the task step size set by the administrator. The dimensions are automatically split and executed in a decentralized manner, reducing the pressure on the source database; The data obtained by the data sharing system from the source database forms a text file, which is encrypted by the RSA asymmetric encryption algorithm to form an encrypted file, and the encrypted file is sent to the document center for storage. Users who need to share data directly download the data from the document center through the link, thereby Pressure to shift data sharing systems; Users who need to share data need to obtain the target data in two steps, as follows: ①, first call the interface provided by the data sharing system; ②. Call the interface of the document center to obtain the encrypted file according to the result obtained in the first step.

Images