CN115718690A - Data accuracy monitoring system and method - Google Patents

Abstract

The invention discloses a data accuracy monitoring system and method, comprising: a data statistics module, used to determine the first data information of the first table to be counted in the tenant database to be counted from the database cluster included in the production system; Determine the second data information of the second table to be counted corresponding to the first table to be counted from the real-time data warehouse included in the production system; the log collection module is used to collect log information included in each node on the data link in real time ; The data analysis module is used to determine the data statistics result information by comparing the first data information and the second data information determined by the data statistics module; by comparing the log information of each node collected by the log collection module, determine the log statistics result information . The data analysis module compares the first data information and the second data information to determine the data statistical result information, and determines the log statistical result information by comparing the log information of each node, thereby improving the accuracy of data monitoring.

Description

A data accuracy monitoring system and method

technical field

The embodiments of the present invention relate to the technical field of data processing, and in particular to a data accuracy monitoring system and method.

Background technique

With the development of the field of data processing technology, the demand for data processing of large amounts of data is constantly increasing. When performing data processing on a large amount of data, it is necessary to perform data monitoring on the large amount of data in order to find data loss or data inconsistency in the large amount of data.

In the existing technology, the accuracy monitoring of data usually pays more attention to the accuracy of real-time data and the stability of the data link, etc., and the method of automatic expansion and active and standby links is adopted to improve the stability of the data link, and when the main data link Data loss may occur when the standby data link is activated due to road failure. However, in the existing technology, more attention is paid to the accuracy of real-time data, and the lost data cannot be located, which reduces the accuracy of data monitoring. Therefore, how to improve the accuracy of data monitoring is an urgent technical problem to be solved at present.

Contents of the invention

The invention provides a data accuracy monitoring system and method, which improves the accuracy of data monitoring.

In a first aspect, an embodiment of the present invention provides a data accuracy monitoring system, including:

The data statistics module is used to determine the first data information of the first statistics table in the tenant library to be counted from the database cluster included in the production system; The second data information of the second table to be counted corresponding to the table to be counted;

The log collection module is used to collect log information included in each node on the data link in real time;

The data analysis module is used to connect the data statistics module and the log collection module respectively; by comparing the first data information and the second data information determined by the data statistics module, determine the data statistics result information; By comparing the log information of each node collected by the log collection module, determine the log statistical result information;

Wherein, the number of the database clusters is one or more, and each of the database clusters includes one or more tenant databases, and each tenant database includes one or more first tables to be counted; the second table to be counted The first table to be counted is transmitted to the real-time data warehouse through a data link.

In the second aspect, an embodiment of the present invention provides a data accuracy monitoring method, which is applied to the data accuracy monitoring system described in the first aspect, including:

Determining the first data information of the first table to be counted in the tenant database to be counted from the database cluster included in the production system through the data statistics module;

Determining the second data information of the second table to be counted corresponding to the first table to be counted from the real-time data warehouse included in the production system through the data statistics module;

The log information included in each node on the data link is collected in real time through the log collection module;

comparing the first data information and the second data information determined by the data statistics module through the data analysis module to determine the data statistics result information;

The log information of each node collected by the log collection module is compared by the data analysis module to determine the log statistical result information.

In the technical scheme of the embodiment of the present invention, the first data information of the first table to be counted is determined by the data statistics module, and the second data information of the second table to be counted corresponding to the first table to be counted; real-time collection by the log collection module The log information included in each node on the data link; compare the first data information and the second data information through the data analysis module to determine the data statistical result information, and compare the log information of each node through the data analysis module to determine the log statistical result information , improving the accuracy of data monitoring.

It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will be easily understood from the following description.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a schematic structural diagram of a data accuracy monitoring system provided according to Embodiment 1 of the present invention;

Fig. 2 is a schematic structural diagram of a production system provided according to Embodiment 1 of the present invention;

3 is a schematic structural diagram of a data accuracy monitoring system provided according to Embodiment 2 of the present invention;

4 is a schematic structural diagram of a multi-tenant database data accuracy monitoring system provided according to Embodiment 3 of the present invention;

Fig. 5 is a schematic structural diagram of a data statistics sub-module provided according to Embodiment 3 of the present invention;

Fig. 6 is a flowchart of a data accuracy monitoring method according to Embodiment 4 of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

It can be understood that before using the technical solutions disclosed in the embodiments of the present invention, the user should be informed of the type, scope of use, and use scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and the authorization of the user should be obtained. .

Embodiment one

FIG. 1 is a schematic structural diagram of a data accuracy monitoring system according to Embodiment 1 of the present invention. This embodiment is applicable to the situation of implementing data accuracy monitoring. As shown in Figure 1, a data accuracy monitoring system 10 includes:

The data statistics module 11 is used to determine the first data information of the first table to be counted in the tenant library to be counted from the database cluster included in the production system; The second data information of the second table to be counted corresponding to the statistical table;

The log collection module 12 is used to collect log information included in each node on the data link in real time;

The data analysis module 13 is used to connect the data statistics module 11 and the log collection module 12 respectively; by comparing the first data information and the second data information determined by the data statistics module 11, determine the data statistics result information; by comparing the log collection module 12 The collected log information of each node determines the log statistical result information;

Wherein, the number of database clusters is one or more, and each database cluster includes one or more tenant libraries, and each tenant library includes one or more first tables to be counted; the second table to be counted is the first table to be counted The table transmitted to the real-time data warehouse through the data link.

Among them, the production system may refer to the information system that supports daily business operations under normal circumstances. Fig. 2 is a schematic structural diagram of a production system provided according to Embodiment 1 of the present invention. The data accuracy monitoring system 10 in the embodiment of the present invention can monitor the accuracy of the data in the production system, so that it is found that there are Tenant library or table for data loss issues.

As shown in Figure 2, the production system may include one or more database clusters (each leftmost rectangular box in Figure 2 may represent a database cluster), and each database cluster may include one or more tenant libraries ( Each cylindrical box in any leftmost rectangular box in FIG. 2 may represent a tenant library), and each tenant library may include one or more database tables (Table).

The production system can also include a data link, which can include a relational database management system binary log consumer (Mysql Binlog Consumer), a message queue middleware (Message Queue) for storing binary logs, and messages for processing binary logs on the data link Consumer (Message Consumer), the nodes corresponding to these components can be equipped with corresponding logs and record the data records processed by themselves. Among them, Mysql Binlog Consumer can be used to indicate the Binlog corresponding to the tenant library, and each tenant library can have a corresponding Binlog; Message Queue can be used to store Binlog, and the tenant library that needs to be analyzed for data accuracy can be selected by configuring the Message Queue And/or table; Message Consumer can be used to process Binlog, such as format conversion of Binlog, etc. Message Consumer can indicate information such as the time or message type generated by Binlog.

The production system can also include a real-time data warehouse. The real-time data warehouse pays more attention to the real-time nature of the data. The tenant library and/or table in the original database cluster (that is, the tenant library shown in any leftmost rectangle in Figure 2 and/or table) can be transmitted to the real-time data warehouse through the data link, the real-time data warehouse can have the original tenant library and/or table in the database cluster, and there can also be processed by the nodes in the data link, and the original The tenant library and/or table corresponding to the tenant library and/or table in the database cluster.

The data statistics module 11 is configured to determine the first data information of the first table to be counted in the tenant database to be counted from the database cluster included in the production system.

Wherein, the tenant database to be counted may refer to one or more tenant databases to be counted in the database cluster of the production system. The first table to be counted may refer to any database table to be counted in any tenant database, and each tenant database may include one or more first tables to be counted. The number of the first tables to be counted and the tenant database in which the first tables to be counted are not limited. Specifically, the first table to be counted can be selected from the tenant database to be counted according to actual application needs.

The first data information may refer to the information indicated by the data in the first table to be counted. For example, the first data information may include data volume and service indicators. The amount of data may be the total amount of data in the first table to be counted, such as the total number of rows of data recorded in the first table to be counted. The business indicator may be an indicator related to a business scenario selected according to actual application needs, such as an indicator such as sales or sales quantity.

The manner in which the data statistics module 11 determines the first data information of the first table to be counted in the tenant database to be counted is not limited, as long as the first data information of the first table to be counted in the tenant database to be counted can be determined. For any first table to be counted, the data statistics module 11 can provide the interface of the first table to be counted to access the data accuracy monitoring system 10, so that the first table to be counted can be connected to the data through the interface provided by the data statistics module 11. Accuracy monitoring system 10; when the first table to be counted is connected to the data accuracy monitoring system 10, the data statistics module 11 can abstract the total number of rows of data included in the first table to be counted into the amount of data through the count() function , through the sum() function, the accumulated value of the data included in the first table to be counted is abstracted into a business index; then the abstracted data is processed through programming statements, and the data statistics module 11 executes the programming statement to determine the first table to be counted The data amount and service index corresponding to the statistical table is to determine the first data information of the first table to be counted. Wherein, the programming statement can be set according to actual application needs, such as Structured Query Language (Structured Query Language, SQL).

The data statistics module 11 is configured to determine the second data information of the second table to be counted corresponding to the first table to be counted from the real-time data warehouse included in the production system.

The second table to be counted is located in the real-time data warehouse included in the production system, and the second table to be counted may refer to the table that the first table to be counted is transmitted to the real-time data warehouse through a data link. During the process of transmitting the first table to be counted to the real-time data warehouse through the data link, the nodes corresponding to the components in the data link can be equipped with corresponding logs and record the data records processed by themselves.

When the user has operations such as modifying the first table to be counted, the first table to be counted can be processed by nodes in the data link, such as recording the user's modification operation on the first table to be counted, and then the processed first table to be counted The table to be counted is transmitted to the real-time data warehouse, which is the second table to be counted in the real-time data warehouse, and the second table to be counted is the database table after the modification operation on the first table to be counted at the current moment.

The second data information may refer to the information indicated by the data in the second table to be counted. For example, the second data information may include data volume and service indicators. The amount of data may be the total amount of data in the second table to be counted, such as the total number of rows of data recorded in the second table to be counted. The business indicator may be an indicator related to a business scenario selected according to actual application needs, such as an indicator such as sales or sales quantity.

The data statistics module 11 determines the second data information of the second table to be counted corresponding to the first table to be counted from the real-time data warehouse included in the production system is not limited, as long as the second data of the second table to be counted can be determined Information is enough. For example, the manner of determining the second data information of the second table to be counted by the data statistics module 11 may be the same as the manner of determining the first data information of the first table to be counted, and will not be repeated here.

The log collection module 12 is configured to collect log information included in each node on the data link in real time.

Wherein, each node on the data link may refer to the nodes corresponding to components such as Mysql Binlog Consumer, Message Queue, and Message Consumer, such as the first node corresponding to the binary log consumer, and the first node corresponding to the message queue middleware storing the binary log. The second node and the third node that handles the message consumer of the binary log, these nodes can be equipped with corresponding logs and record the data records processed by themselves.

The log information included in each node may refer to the information indicated by the log provided by each node. The log information can include key information of the data processed or stored by each node, such as the primary key value recorded in the table. The logs indicated by the log information can maintain a unified format and directory structure.

Optionally, the log collection module 12 may also collect log information of the fourth node corresponding to the real-time data warehouse in real time.

The log collection module 12 collects the log information included in each node on the data link in real time, or collects the log information of the fourth node corresponding to the real-time data warehouse in real time. The method is not limited. For example, the log file can be collected through the log file consignment tool Filebeat information, Filebeat can monitor the logs of each node, and track and read the files corresponding to the logs of each node.

The data analysis module 13 is used to connect the data statistics module 11 and the log collection module 12 respectively.

The way that data analysis module 13 connects data statistics module 11 and log collection module 12 respectively is not limited, as long as the first data information and the second data information determined by data statistics module 11 and each node collected by log collection module 12 can be made The log information can be transmitted to the data analysis module 13. For example, it may be through electrical connection, or through network connection, etc.

The data statistical result information may refer to the information indicated by the result of comparing the first data information and the second data information, and the data statistical result information may indicate that no data loss or data loss occurs from the first table to be counted to the second table to be counted .

The data analysis module 13 is used to determine the data statistics result information by comparing the first data information and the second data information determined by the data statistics module 11 . For example, compare the amount of data included in the first data information determined at the current moment with the amount of data included in the second data information through Structured Query Language (Structured Query Language, SQL), or compare the amount of data included in the first data information determined at the current moment. Included business indicators and business indicators included in the second data information, when the data volume and business indicators included in the first data information are the same as the data volume and business indicators included in the second data information, it is determined that the data statistics result information indicates that there is no Data loss has occurred; otherwise, the determined data statistics result message indicates that data loss has occurred.

As another example, the data analysis module 13 compares the first data information and the second data information determined by the data statistics module 11 through a trend analysis method to determine the data statistics result information. The trend analysis method can refer to the method of finding the problem of data loss through the analysis of the change trend of each time segment of the first data information or the second data information. The trend analysis method can deal with the statistical error caused by the inability of the real-time system to maintain a static state analysis problem.

The trend analysis method divides the data into segments according to the length of time as needed. For example, the latest week can be divided into 7 time segments according to the number of days. The duration of each time segment is 1 day. The first data information and The second data information, when statistics are performed on the 8th day, can be based on the first data information and the second data information determined on the 8th day, combined with the first data information and the second data information determined in the time segment of the previous 7 days, Determine the data statistics result information corresponding to the 8th day. And when the data accuracy monitoring system 10 is started for the first time, if a consistent baseline is determined so that the first data information and the second data information are consistent, the trend analysis method can be used to trace back to the time segment when the data is abnormal. The fragment's data statistics result message indicates that data loss has occurred.

The log statistical result information may refer to the information indicated by the result of comparing the log information of each node, and the log statistical result information may indicate a node having a data inconsistency problem on the data link.

The data analysis module 13 determines the log statistical result information by comparing the log information of each node collected by the log collection module 12 . For example, the data analysis module 13 compares the business primary key, business occurrence time, and binlog type to find nodes with data inconsistencies on the data link, and determine the nodes with data inconsistencies as log statistical result information.

It should be noted that the frequency of the data analysis module 13 determining the data statistical result information and the log statistical result information is not limited. For example, the data analysis module 13 determines the data statistical result information once a day, or the data analysis module 13 determines the log statistical result information once a day. Or the data analysis module 13 determines the log statistical result information every hour.

In the technical scheme of the embodiment of the present invention, the first data information of the first table to be counted is determined by the data statistics module, and the second data information of the second table to be counted corresponding to the first table to be counted; real-time collection by the log collection module The log information included in each node on the data link; compare the first data information and the second data information through the data analysis module to determine the data statistical result information, and compare the log information of each node through the data analysis module to determine the log statistical result information . Through the statistical result information of the data, it can indicate that no data loss has occurred or that data loss has occurred. When data loss occurs, the time segment of data loss can be determined through the trend analysis method; through the statistical result information of the log, it can be indicated that there is a The node of data inconsistency; that is, when data loss occurs, the lost data can be located, which improves the accuracy of data monitoring.

Further, the data accuracy monitoring system 10 also includes:

The visualization processing module is used for connecting to the data analysis module 13 ; and visually processing the data statistical result information and/or log statistical result information determined by the data analysis module 13 .

The way that the visualization processing module is connected to the data analysis module 13 can be through an electrical connection or a network connection, etc., and the data analysis module 13 can transmit the data statistical result information and/or the log statistical result information to the visualization processing module, so that the visualization processing module can count the data. Visual processing of result information and/or log statistics result information.

The visual processing module does not limit the way of visual processing of data statistical result information and/or log statistical result information. For example, the visual processing module generates analysis reports according to data statistical result information and/or log statistical result information, and according to database clusters, tenant libraries and The database table presents the analysis report on the display. Among them, the analysis report can indicate the database clusters, tenant libraries and database tables that have data loss problems. The display screen can be a monitoring panel.

At the same time, the visualization processing module can combine the data statistical result information and/or log statistical result information determined in different time segments, and draw a data trend curve according to the time dimension to show the accuracy of the data. Wherein, the data trend curve may refer to a curve formed by data corresponding to different time segments. For example, the data trend curve may indicate the number of database clusters, tenant libraries, and database tables where data loss occurs in each time segment.

Further, the data accuracy monitoring system 10 may also include:

Abnormal alarm module, the abnormal alarm module can be used to send the database cluster, tenant library and database table with data loss problem to the operation and maintenance and development personnel for processing by means of alarm.

Embodiment two

Fig. 3 is a schematic structural diagram of a data accuracy monitoring system provided according to Embodiment 2 of the present invention. This embodiment is based on the above-mentioned Embodiment 1, and describes the specific structures of the data statistics module 11 and the log collection module 12. For technical details not described in detail in this embodiment, refer to the first embodiment above.

In the embodiment of the present invention, the data statistics module 11 includes:

Data statistics sub-module 111 and data storage sub-module 112;

The data statistics sub-module 111 is used to determine the first data information of the first table to be counted in the tenant database to be counted from the database cluster included in the production system; determine the first data information from the real-time data warehouse included in the production system The second data information of the second table to be counted corresponding to the table to be counted;

The data storage sub-module 112 is used to respectively connect the data statistics sub-module 111 and the data analysis module 13; store the first data information and the second data information determined by the data statistics sub-module 111; store the first data information and the second data information Sent to the data analysis module 13.

The manner in which the data storage sub-module 112 is respectively connected to the data statistics sub-module 111 and the data analysis module 13 is not limited, for example, it may be through an electrical connection, or through a network connection.

The data storage submodule 112 is connected to the data statistics submodule 111. After the data statistics submodule 111 determines the first data information and the second data information, the data statistics submodule 111 can transmit the first data information and the second data information to the data storage submodule. Module 112, the data storage sub-module 112 receives and stores the first data information and the second data information.

The data storage submodule 112 can be an online analysis processing database (OLAP DB), wherein, OLAP is an online analysis processing, and OLAP can be used for data analysis, and the information from multiple database clusters can be analyzed simultaneously through OLAP, that is, the information can be extracted through OLAP The required data and query data for data analysis from different angles.

The data storage sub-module 112 is connected to the data analysis module 13, and the data storage sub-module 112 can send the first data information and the second data information to the data analysis module 13, so that the data analysis module 13 can determine according to the first data information and the second data information Data statistics result information.

Further, the data statistics sub-module 111 includes:

The service interface unit 1111 is configured to provide an interface for accessing the system for the first table to be counted and the second table to be counted.

The service interface unit 1111 can provide an interface for accessing the first table to be counted and the second table to be counted to be counted, that is, the first table to be counted and the second table to be counted to be counted in the production system can be accessed through the service interface unit 1111 To the data accuracy monitoring system 10, so that the data accuracy monitoring system 10 can obtain the first table to be counted and the second table to be counted, and perform subsequent processing on the first table to be counted and the second table to be counted, such as determining the first The first data information of the table to be counted, determining the second data information of the second table to be counted, or comparing the first data information with the second data information, etc. Correspondingly, the tenant library or database cluster where the first table to be counted or the second table to be counted is located may also be connected to the data accuracy monitoring system 10 through the interface provided by the service interface unit 1111 .

The task queue unit 1112 is used to store statistical demand tasks waiting to be processed. The statistical demand task is to determine the first data information of the first table to be counted and/or to determine the second data information of the second table to be counted.

Statistical demand task can refer to the task that needs to carry out statistics to the data in any database table, as statistical demand task can be to determine the first data information of the first table to be counted, and/or determine the second data information of the second table to be counted. Data information. There is no limit to the number of statistical demand tasks, which can be determined according to actual application needs.

When the first table to be counted or the second table to be counted is connected to the data accuracy monitoring system 10 through the service interface unit 1111, it indicates that the first data information of the first table to be counted needs to be determined, or the information of the second table to be counted needs to be determined. The second data information, each first table to be counted can correspond to a statistical demand task, and each second counted table can also correspond to a statistical demand task, each statistical demand task can be stored in the task queue unit 1112, in the task queue Unit 1112 is waiting to be processed.

The tenant library management unit 1113 is configured to synchronize the configuration of each tenant library in each database cluster from the production system, and classify the tenant libraries in the same database cluster into the same group.

Each tenant library in the production system is organized in physical distribution, and at the same time is continuously adjusted according to the development of the software as a service (Software as a Service, SaaS) business. The tenant library management unit 1113 can synchronize each database from the production system The configuration of each tenant library in the cluster, and divide each tenant library in the same database cluster into the same group, that is, the tenant library management unit 1113 can identify the tenant library in each database cluster in the production system, and accurately access the data All tenant databases belonging to the same database cluster of the performance monitoring system 10 are classified into the same group.

The business abstraction unit 1114 is configured to abstract the data in the first table to be counted and/or the second table to be counted.

The way of abstracting the data in the first to-be-statistics table and/or the second to-be-statistics table is not limited, for example, the task execution unit 1116 needs to determine the first data information and/or the second data information later, and the first When the data information and/or the second data information include data volume and service indicators, the service abstraction unit 1114 may abstract the data in the first table to be counted and/or the second table to be counted into data volume and business indicators.

In the first table to be counted and/or the second table to be counted, the total number of rows of data included in the first table to be counted and/or the second table to be counted can be used as the amount of data through the count() function; The () function takes the cumulative value of the data included in the first table to be counted and/or the second table to be counted as a business index, such as all sales recorded in the first table to be counted and/or the second table to be counted The cumulative value of is used as a business indicator.

The symbolization of data in the first table to be counted and/or the second table to be counted to be counted can be realized through the business abstraction unit 1114, which is required when providing the first table to be counted and/or the second table to be counted to be counted and statistical data The function can automatically realize the statistics of data volume and business indicators.

The task execution management unit 1115 is configured to manage concurrently processed statistical demand tasks, and/or manage the number of tenant databases connected to the system.

The task execution management unit 1115 is used to manage the statistically required tasks of concurrent processing, that is, the task execution management unit 1115 can manage the number of concurrently processed statistically required tasks. The data statistics of more tenant databases can be processed concurrently, that is, more statistical demand tasks can be processed concurrently, so that when the data in the production system continues to change, errors in the statistical data will not be caused by too long statistical time, ensuring Timeliness of statistics.

The task execution management unit 1115 is used to manage the number of tenant databases connected to the system. When it is necessary to perform statistics on the data in the first to-be-statistics table or the second to-be-statistics table in the tenant database, it indicates that the tenant database has corresponding statistics requirements The task is to connect the tenant library where the first table to be counted or the second table to be counted to be counted to the data accuracy monitoring system 10 . Correspondingly, when the first table to be counted or the second table to be counted does not exist in the tenant database, it indicates that there is no database table that requires data statistics in the tenant database, that is, there is no corresponding statistical demand task, and the tenant database is made Disconnecting the connection with the data accuracy monitoring system 10 can save environment resources where the data accuracy monitoring system 10 runs, and save the connection overhead of the tenant database.

The task execution unit 1116 is configured to concretize the data abstracted and processed by the business abstraction unit 1114 into executable statements, and determine the first data information and/or the second data information.

The task execution unit 1116 is configured to embody the data abstracted and processed by the business abstraction unit 1114 into executable statements, and determine the first data information and/or the second data information. It can be understood that the task execution unit 1116 can be programmed The statement processes the data abstracted by the business abstraction unit 1114, and the first data information and/or the second data information can be determined by executing the programming statement through the task execution unit 1116. Wherein, the programming statement can be set according to actual application needs, such as Structured Query Language (Structured Query Language, SQL).

The result output unit 1117 is configured to send the first data information and/or the second data information to the data storage sub-module 112 .

In the result output unit 1117, the first data information and/or the second data information can be stored in the buffer queue, and the first data information and/or the second data information stored in the buffer queue can be sent to the data storage submodule when needed 112. For example, the first data information and/or the second data information stored in the buffer queue may be written into the data storage sub-module 112 in batches through a timing script, wherein the timing script may be a script set according to actual application needs, and is not specifically described. It is limited as long as the first data information and/or the second data information can be sent to the data storage sub-module 112 .

Further, the task execution management unit 1115 is specifically used for:

The concurrently processed statistical demand tasks are centrally managed through the thread manager, and the number of concurrently processed statistical demand tasks is linearly related to the number of tenant libraries;

The database connection manager is combined with the tenant database management unit 1113 to manage the number of tenant databases connected to the system.

The concurrently processed statistical demand tasks are centrally managed through the thread manager, and the number of concurrently processed statistical demand tasks is linearly related to the number of tenant libraries. It can be understood that when the number of tenant libraries increases, the corresponding threads are expanded through the thread manager so that more tasks with statistical requirements can be processed concurrently. Among them, concurrent processing can comply with the linear expansion of the tenant library in the production system, and the corresponding expansion of the number of threads can ensure the timeliness of statistical data, and will not cause errors due to continuous changes in the data in the production system when the statistical time is too long.

Correspondingly, when the number of tenant libraries decreases, the number of corresponding threads is reduced through the thread manager, so that fewer statistical demand tasks can be processed concurrently, thereby saving the environment resources run by the data accuracy monitoring system 10, and saving tenants at the same time. The connection overhead of the library.

Through the database connection manager combined with the tenant database management unit 1113, the number of tenant databases connected to the system can be managed, which can be understood as limiting the number of tenant databases connected to the system, and can record each statistical requirement that needs to be executed through the HashMap data structure The tenant library of the task, and through the configuration of each tenant library in each database cluster in the production system recorded by the tenant library management unit 1113, the tenant libraries that need to perform statistical demand tasks in the same database cluster are identified as the same group. For any database cluster, the key (Key) corresponding to the HashMap data structure can be used as the identifier of the database cluster, and the value (Value) can be used as the number of tenant libraries connected to the data accuracy monitoring system 10 in the database cluster. can determine the number of tenant libraries connected to the data accuracy monitoring system 10 in the database cluster. After the identification of the database cluster is determined, the database cluster can be connected to the data accuracy monitoring system 10 through the database connection manager, and the number of tenant libraries corresponding to the value (Value) can be connected to the data accuracy monitoring system 10. The management unit 1113 manages the number of tenant databases connected to the system.

Wherein, the identification of the database cluster is not limited, for example, it may be a digital number, and each database cluster may have a unique corresponding digital number, so as to distinguish different database clusters. The current limiting process can make the execution of statistical demand tasks not affect the performance of the production system.

Further, the task execution management unit 1115 is specifically used for:

Use the database connection manager to release the connection between the tenant library and the system without statistical demand tasks;

When there is no statistical demand task, the thread manager is used to release the thread.

When there is no statistical requirement task in the tenant library, the thread manager is used to release the thread, and the connection between the tenant library and the data accuracy monitoring system 10 is released by the database connection manager; when there is a statistical requirement task in the tenant library, the thread manager is used The thread is kept active, and the tenant library is kept connected to the data accuracy monitoring system 10 through the database connection manager, thereby saving the environment resources in which the data accuracy monitoring system 10 operates and saving the connection overhead of the tenant library.

Further, the data analysis module 13 is specifically used for:

In the historical time period, the absolute value of the difference between the value corresponding to the first data information and the second data information corresponding to each historical time segment in the historical time period is determined as the first error corresponding to the historical time segment ;

According to the first error corresponding to each historical time segment, determine the first mean value and the first mean square error corresponding to the historical time segment;

determining the absolute value of the difference between the value corresponding to the first data information corresponding to the current time segment and the value corresponding to the second data information as the second error corresponding to the current time segment;

The absolute value of the difference between the second error and the first mean value is determined as the third error of the current time segment relative to the historical time segment;

If the third error does not exceed the first mean square error, then determine that the data statistical result information indicates that no data loss has occurred; if the third error exceeds the first mean square error, then determine that the data statistical result information indicates that data loss occurs;

Wherein, the historical time segment includes multiple historical time segments of the same duration, and the current time segment and the historical time segment have the same duration.

When the data analysis module 13 compares the first data information and the second data information, and determines the statistical result information of the data, it can be realized by a trend analysis method. The first data information and the second data information determined by the current time segment are combined with the historical time segment The first data information and the second data information determined for each historical time segment in the current time segment are determined to determine the data statistics result information corresponding to the current time segment.

Taking the historical time period as 1 week as an example, a week can be divided into 7 historical time segments according to the number of days, each historical time segment is 1 day long, and the historical time segment includes 7 historical time segments of the same duration. The duration of the current time segment is also 1 day, and the current time segment may be a time segment after the historical time segment.

Specifically, within the historical time period, the absolute value of the difference between the value corresponding to the first data information and the second data information corresponding to each historical time segment in the historical time period is determined as the value corresponding to the historical time segment first error.

The value corresponding to the first data information corresponding to each historical time segment in the historical time period can be expressed in order of time as:

r ₁ ,r ₂ ,r ₃ ,r ₄ ,r ₅ ,r ₆ ,r ₇

The value corresponding to the second data information corresponding to each historical time segment in the historical time period can be expressed in order of time as:

w ₁ ,w ₃ ,w ₃ ,w ₄ ,w ₅ ,w ₆ ,w ₇

Then the first error corresponding to each historical time segment can be expressed in order of time as:

δ ₁ =|w ₁ -r ₁ |, δ ₂ =|w ₂ -r ₂ |, δ ₃ =|w ₃ -r ₃ |,...,δ ₇ =|w ₇ -r ₇ |

According to the first error corresponding to each historical time segment, a first mean value and a first mean square error corresponding to the historical time segment are determined.

The formula for calculating the first mean value corresponding to the historical time period is:

The formula for calculating the first mean square error corresponding to the historical time period is:

The absolute value of the difference between the value corresponding to the first data information corresponding to the current time segment and the value corresponding to the second data information is determined as the second error corresponding to the current time segment. The value corresponding to the first data information corresponding to the current time segment can be expressed as r ₈ , and the value corresponding to the second data information corresponding to the current time segment can be expressed as w ₈ , then the calculation formula of the second error is:

δ ₈ ＝|w ₈ -r ₈ |

The absolute value of the difference between the second error and the first mean value is determined as the third error of the current time segment relative to the historical time segment, then the calculation formula of the third error is:

If the third error does not exceed the first mean square error, it is determined that the statistical result information of the data indicates that no data loss occurs, wherein the third error does not exceed the first mean square error can be expressed as:

σ ₈ ≤ σ

If the third error exceeds the first mean square error, it is determined that the data statistical result information indicates that data loss occurs, wherein the third error exceeds the first mean square error and can be expressed as:

σ ₈ >σ

In the embodiment of the present invention, the log collection module 12 includes:

The log collection tool 121 is used to collect the log information corresponding to the first node of the binary log consumer; collect the log information corresponding to the second node of the message queue middleware storing the binary log; collect the corresponding log information of the message consumer processing the binary log The log information of the three nodes; collect the log information of the fourth node corresponding to the real-time data warehouse;

The log storage tool 122 is used to connect the log collection tool 121 and the data analysis module 13 respectively; store the log information included in each node collected by the log collection tool 121 ; and send the log information to the data analysis module 13 .

The log collection tool 121 may be a log file consignment tool Filebeat, through which log information of the first node, the second node, the third node and the fourth node can be collected.

The manner in which the log storage tool 122 is connected to the log collection tool 121 and the data analysis module 13 may be electrical connection or network connection.

The log storage tool 122 is connected to the log collection tool 121, and can acquire and store the log information of each node transmitted by the log collection tool 121. The log storage tool 122 may support a distributed search and analysis engine, such as ElasticSearch, to simultaneously support accuracy comparison and problem location capabilities. The storage of log information can be automatically managed in the lifecycle, such as the index template and lifecycle function of Elasticsearch, thereby reducing storage costs and operation and maintenance costs.

The log storage tool 122 is connected to the data analysis module 13, and the log storage tool 122 can send the log information included in each node to the data analysis module 13, so that the data analysis module 13 can compare the log information of each node and determine the log statistical result information.

Further, the data analysis module 13 is specifically used for:

Contrast the log information included in each node collected by the log collection module 12 with the set frequency, and determine the problem node where the data inconsistency problem occurs on the data link;

Determine the problem node as log statistics result information.

The set frequency may refer to the frequency of determining the log statistical result information, and the set frequency may be determined according to actual application needs, for example, the set frequency may be every hour. The problem node may refer to a node where data inconsistency occurs on the data link.

The data analysis module 13 can compare the log information included in each node every hour, such as comparing the business primary key indicated by the log information, the time of business occurrence and the binlog type, etc., to find problem nodes with data inconsistencies on the data link, Determine the problem node with data inconsistency as the log statistics result information.

The technical solutions of the embodiments of the present invention describe the specific structures of the data statistics module and the log collection module, and the specific functions of the data analysis module. The data analysis module can determine the data statistics corresponding to the current time segment through the first data information and the second data information corresponding to each historical time segment in the historical time segment, and the first data information and the second data information corresponding to the current time segment Result information; By selecting different current time segments and historical time segments, you can further determine the time segment where data loss occurs; the data analysis module can compare the log information included in each node by setting the frequency, determine the log statistical result information, log statistics The result information indicates the node where data inconsistency occurs on the data link; that is, when data loss occurs, the lost data can be located, which improves the accuracy of data monitoring.

Embodiment Three

The embodiments of the present invention are exemplary descriptions of the above-mentioned embodiments. The embodiment of the present invention proposes a multi-tenant database data accuracy monitoring system, which can monitor data in a production system including multiple servers (ie, multi-database clusters) and multiple tenants, and supports simultaneous OLAP and real-time analysis.

In a real-time production system, when faced with terabytes of data, it is difficult to find data loss and locate problems. This system comprehensively considers cost and efficiency, and uses statistical methods and data link tracking methods to analyze macro, micro and time Dimensions, combined with analysis charts to automatically check, comprehensively analyze and alert data accuracy, thereby solving the problem of real-time data accuracy checking in multi-tenant scenarios, and can find data accuracy problems in the T+1 cycle, and in H+1 The operating status of tens of thousands of data sources can be found within a cycle, with low operating cost, high efficiency, and easy to use.

Among them, the cycle is the frequency of data monitoring; there are two strategies for monitoring: full amount and sampling; in the execution of the full amount strategy, it is executed once a day on average, and correspondingly we define this cycle as T+1; in the execution of the sampling strategy, the average Executed once an hour, correspondingly we define this cycle as H+1.

Fig. 4 is a schematic structural diagram of a multi-tenant database data accuracy monitoring system provided according to Embodiment 3 of the present invention. The monitoring system shown in Fig. 4 is a multi-tenant database data accuracy monitoring system provided in Embodiment 3 of the present invention. As for the monitoring system, the production system is also shown in FIG. 4 , and the monitoring system can monitor the data accuracy of the production system.

As shown in Figure 4, the multi-tenant database data accuracy monitoring system includes: statistics (that is, data statistics sub-module), collecting logs (that is, log collection tools), distributed search and analysis engines (that is, log storage tools), online analysis Processing database (ie, data storage sub-module), analysis (ie, data analysis module), monitoring panel (ie, visual processing module) and abnormal alarm (ie, abnormal alarm module).

Among them, statistics (that is, the data statistics sub-module) are mainly used to meet various needs of the data statistics step; data are distributed in different database servers (that is, database clusters), and SaaS services are running on them, so it is necessary to solve the problems of multiple tenant databases, production resource constraints.

Statistics (that is, the data statistics sub-module) is designed as a service task, which can meet various statistical needs, thus providing the possibility for automatic integration; in the implementation, multi-threading is used to improve concurrency, and the ability of concurrency depends on the size of the demand. That is, how many database servers are there, so as to achieve a linear expansion capability equivalent to the SaaS business; a strict database operation framework is implemented at the bottom layer to strictly control database connections and request management; the number of database connections is determined by the production system operation and maintenance The party evaluates and gives the value allowed to be used by the monitoring system according to the tenant database capacity and business volume; the database does not maintain a long connection, but only connects when needed, and releases the request immediately after processing the request; the database request only supports query requests, avoiding production data. Impact; all requests only support sequential execution, avoiding excessive occupation of production resources and causing production request blocking.

FIG. 5 is a schematic structural diagram of a data statistics submodule provided according to Embodiment 3 of the present invention. Specifically, the data statistics submodule includes:

The service interface (ie, the service interface unit) is used to provide the application for statistical requirements, and can specify the cluster (ie, database cluster) or table (the first table to be counted or the second table to be counted) that needs to be counted on time and flexibly;

Task queue (i.e. task queue unit), statistical requirements (i.e. statistical demand tasks) enter the system and enter the task queue uniformly to wait for processing;

Tenant library management module (that is, tenant library management unit), the tenant library of the production system is organized in physical distribution, and at the same time is continuously adjusted according to the development of the SaaS business. Databases are configured and identified, and tenant libraries of the same database cluster are grouped (that is, each tenant library in the same database cluster is divided into the same group);

The business abstraction module (that is, the business abstraction unit), the business includes data volume statistics and business index statistics. The business abstraction module abstracts the two types of statistics into count and sum, and realizes the symbolization of the description of statistical requirements, that is, it provides tables and tables for corresponding storage of business data. Statistics type, the system can automatically realize the statistics;

The task execution management module (i.e., the task execution management unit), the task execution management module sets the rules of concurrent and current limiting double guarantees for the execution of tasks:

The concurrency guarantee can adapt to the linear expansion of the production system tenant library and expand the number of threads accordingly, so as to ensure the timeliness of statistical data, and will not cause errors due to continuous changes in production data when the statistical time is too long; a thread management is established in the program The thread manager is used to centrally manage concurrent tasks, and the number of concurrent tasks is linearly related to the number of tenant libraries (that is, the number of concurrently processed statistical demand tasks is centrally managed through the thread manager, and the number of concurrently processed statistical demand tasks is linearly related to the number of tenant libraries) ;

Current limiting can ensure that statistical tasks will not affect the performance of the production system; the program manages a HashMap data structure to record each tenant library that is running statistically (that is, performing statistical demand tasks), and combines the tenant management module to identify the same database cluster. Tenant library, the key value of HashMap is the name of the cluster server (that is, the identity of the database cluster), and the value value is the number of connections of the database cluster where the current tenant library is located. At the same time, a database connection manager is implemented to provide the database connection required for task execution In this way, the number of connections to the tenant database is managed (that is, the number of tenant libraries connected to the system is managed through the database connection manager combined with the tenant library management unit);

When tasks (that is, statistical demand tasks) continue to come in, both the thread and the database connection remain active. Once it is found that there is no task coming in, the thread manager actively releases the thread, and the database connection manager actively releases the database connection (that is, through the database connection) The manager releases the connection between the tenant library and the system that has no statistical demand task; when there is no statistical demand task, the thread manager is used to release the thread), which can save the environment resources of the statistical system and save the connection of the production system tenant library overhead;

The task execution module (that is, the task execution unit) is used to concrete the business abstraction into executable SQL, and execute the SQL to obtain the result (that is, to materialize the abstracted data processed by the business abstraction unit into an executable statement, and to determine the first data information and/or second data information);

The result output module (that is, the result output unit), the statistical results executed by the concurrent threads will be stored in a buffer queue, and the statistical results in the buffer queue will be written to the external storage in batches by a cycle storage thread job timing (that is, the first data information and/or second data information is sent to the data storage sub-module).

Through analysis (that is, the data analysis module), first compare the data volume of the tenant database (that is, the data source) with the data volume of the real-time data warehouse (that is, the target database). database, the algorithm in this step is implemented with SQL, which is simple, fast, and highly efficient; the comparison of business indicators is also implemented using the same method;

The data source is usually a MySQL-type database cluster, with multiple servers and multiple tenant libraries. The data of the tables in each tenant library is counted, including the data volume and business indicators, and written to the database that supports OLAP analysis. In the database, it is necessary to compare the tables of multiple tenant libraries to find tenant libraries and tables with data loss problems;

When comparing business indicators, you can select core business indicators, such as sales and sales volume, and compare the total of business indicators. When comparing, you can perform statistical comparisons according to the storage tables corresponding to each tenant database and business indicators, so that Tenant libraries and tables with inaccurate data issues were found.

The results based on the above processing reflect static data differences, which need to be further analyzed by using the trend analysis method combined with historical data; the data accuracy problems caused by real-time system errors are reflected in the data, which is the reason for the analysis of the difference results; by comparing historical data, we can Distinguishing whether this difference is a historical problem or a recent problem requires a statistical approach.

The trend analysis method is superimposed on the back of the statistical analysis method to deal with the analysis of statistical errors caused by the inability of the real-time system to maintain a static state; the trend analysis method divides the data into segments according to the length of time as needed, such as the last week (that is, the historical time period) It can be divided into 7 time segments (that is, historical time segments) by day, and data statistics will be performed once in each time segment and the statistics and analysis results (that is, first data information, second data information and data statistical result information) will be saved. When performing statistics on the 8th day, you can combine the execution results in the time segment of the previous 7 days to determine the corresponding data statistics result information on the 8th day.

It should be noted that the trend analysis method is to compare the difference value (that is, the first error) in each time period. For example, from the 1st to the 7th day, each statistics finds a difference of about 100 pieces of data, then when the 8th day is counted If there is still a difference of about 100, it means that no data is lost today. There are two sources of error: one is that there is a time difference in statistics, because the data is constantly changing, so there will be errors; the other is that there is a backlog of dirty data in history, which needs to be traced back to an earlier time segment; when the monitoring system is started for the first time If we ensure a consistent baseline, that is, the data in the tenant database is consistent with the data in the real-time data warehouse, then we can always go back to the time when the exception occurred, or in fact, we can find it at that time. The problem of data loss is gone.

Collecting logs (that is, log collection tools) is designed as a general collection framework, and the logs maintain a unified format and directory structure; the logs are uniformly collected on storage engines that support full-text retrieval (that is, log storage tools), such as ElasticSearch, to simultaneously support accurate Sexual comparison and problem location capabilities; log collection uses open source tools, such as Filebeat; log storage is automatically managed in the life cycle, such as the index template and lifecycle function of Elasticsearch, thereby reducing storage costs and operation and maintenance costs.

Collect the log information of each node on the data link in real time by collecting logs (that is, log collection tools). These nodes include Mysql Binlog Consumer (that is, the first node), the MessageQueue that stores Binlog (that is, the second node), and the Message Consumer that processes binlog (that is, the third node), and the final data warehouse node (that is, the fourth node), the content indicated by the log information is the key information of the data processed or stored by each node, that is, the primary key value recorded in the table.

By analyzing (that is, the data analysis module) and comparing the log information, it is possible to find nodes (that is, log statistical result information) where data inconsistency occurs on the link. Among them, the sampling is aimed at the data field; usually the corresponding data field of the core business is selected, and the real-time collection link is tracked in this data field. The logs collected by Elasticsearch can be used to compare the data logs between the nodes on the link. In the implementation of the algorithm, it mainly compares the business primary key, business occurrence time and binlog type; through sampling inspection, it can be found whether the real-time data collection link is running stably and whether the data processing is complete, especially for the problems that cannot be found by supplementary statistical methods for changed data.

Generate report data from the analysis results, and display them on a large scale according to the database instance, tenant library, and table, and draw the data trend curve according to the time dimension in combination with the historical statistical results to display the data accuracy report (that is, the data determined by the visual processing module to the data analysis module Visual processing of data statistical result information and/or log statistical result information); for tenant libraries and tables with abnormal data, send them to operation and maintenance and developers for processing through the abnormal alarm module alarm.

In the automated process, abnormal problems are mainly found, and there is a lack of a general description of the overall data. This involves the complexity of description when multi-tenant libraries, multi-tables and time dimensions are considered together, which is not what the monitoring system is good at handling; at this time, you can Combined with analysis charts, analysis is performed through visual chart tools such as tables and graphs, and configurable analysis of various dimensions and data ranges can be performed on demand, such as a table on a MySQL Server (that is, a database cluster) or a tenant library , or the historical data trend graph of a table, etc.; in the specific implementation, open source visual analysis tools can be used.

A multi-tenant library data accuracy monitoring system proposed by the embodiment of the present invention requires a machine responsible for collecting data to run the collection task (that is, a statistical demand task), a database system supporting tens of millions of data, and a Elasticsearch system; all of these are conventional requirements, usually two small-scale machines can meet; the occupation of production resources on operating resources can be ignored, and there is no need to expand production;

In terms of architecture design, statistics and analysis are separated, and most of them can be realized through the SQL level, the system complexity is low, and the development cost is low; the log collection is realized through an open source framework. Currently, there are various mature components that do not require investment. Additional cost; Compared with the method of re-logging the business data details and then calculating the indicator logic, this system directly utilizes production resources, avoiding the complexity of the business data detail storage and hardware maintenance costs.

A multi-tenant library data accuracy monitoring system proposed by the embodiment of the present invention, for a large-scale (10 billion level), multi-tenant (10,000 level) real-time system, the system can complete complete data verification at the hour level, improving Improve the efficiency of data accuracy monitoring. Through system integration, it can meet the requirements of regular inspection and on-demand calling at the same time; the whole process is fully automated, which greatly reduces the complexity of use and the requirements for users.

Embodiment four

Fig. 6 is a flow chart of a data accuracy monitoring method provided according to Embodiment 4 of the present invention. This embodiment is applicable to the realization of data accuracy monitoring, and the data accuracy monitoring method can be applied to any embodiment of the present invention Data accuracy monitoring system provided.

As shown in Figure 6, the method includes:

S110. Determine the first data information of the first table to be counted in the tenant database to be counted from the database cluster included in the production system through the data counting module.

S120. Determine the second data information of the second table to be counted corresponding to the first table to be counted from the real-time data warehouse included in the production system through the data statistics module.

S130. Collect log information included in each node on the data link in real time through the log collection module.

S140. Using the data analysis module to compare the first data information and the second data information determined by the data statistics module, determine data statistics result information.

S150. Determine log statistical result information by comparing the log information of each node collected by the log collection module by the data analysis module.

In the method, the data analysis module compares the first data information and the second data information to determine the data statistical result information, and determines the log statistical result information by comparing the log information of each node of the data analysis module, thereby improving the accuracy of data monitoring.

Further, the first data information of the first statistics table in the tenant library to be counted is determined from the database cluster included in the production system through the data statistics module, and determined from the real-time data warehouse included in the production system through the data statistics module The second data information of the second table to be counted corresponding to the first table to be counted includes:

Through the data statistics sub-module included in the data statistics module, from the database cluster included in the production system, determine the first data information of the first table to be counted in the tenant database to be counted, and determine from the real-time data warehouse included in the production system. The second data information of the second table to be counted corresponding to the first table to be counted;

The data storage sub-module included in the data statistics module stores the first data information and the second data information determined by the data statistics sub-module; and sends the first data information and the second data information to the data analysis module.

Further, through the data statistics sub-module included in the data statistics module, from the database cluster included in the production system, determine the first data information of the first table to be counted in the tenant database to be counted, and from the real-time data warehouse included in the production system Determine the second data information of the second table to be counted corresponding to the first table to be counted, including:

Through the service interface unit included in the data statistics sub-module, an interface for accessing the system is provided for the first table to be counted and the second table to be counted;

The task queue unit included in the data statistics submodule stores the statistical demand tasks waiting to be processed, and the statistical demand task is to determine the first data information of the first table to be counted, and/or determine the second data information of the second table to be counted;

Through the tenant library management unit included in the data statistics sub-module, the configuration of each tenant library in each database cluster is synchronized from the production system, and each tenant library in the same database cluster is divided into the same group;

Abstract processing of data in the first table to be counted and/or the second table to be counted through the business abstraction unit included in the data statistics submodule;

Through the task execution management unit included in the data statistics sub-module, manage the statistical demand tasks for concurrent processing, and/or manage the number of tenant libraries connected to the system;

Through the task execution unit included in the data statistics sub-module, the data abstracted and processed by the business abstraction unit is embodied into an executable statement, and the first data information and/or the second data information are determined;

The first data information and/or the second data information are sent to the data storage sub-module through the result output unit included in the data statistics sub-module.

Further, through the task execution management unit included in the data statistics sub-module, manage the concurrently processed statistical demand tasks, and/or manage the number of tenant libraries connected to the system, including:

The concurrently processed statistical demand tasks are centrally managed through the thread manager, and the number of concurrently processed statistical demand tasks is linearly related to the number of tenant libraries;

The number of tenant databases connected to the system is managed through the database connection manager combined with the tenant database management unit.

Further, the number of tenant libraries connected to the system is managed through the database connection manager combined with the tenant library management unit, including:

Use the database connection manager to release the connection between the tenant library and the system without statistical demand tasks;

When there is no statistical demand task, the thread manager is used to release the thread.

Further, the data statistics result information is determined by comparing the first data information and the second data information determined by the data statistics module through the data analysis module, including:

In the historical time period, the absolute value of the difference between the value corresponding to the first data information and the second data information corresponding to each historical time segment in the historical time period is determined as the first error corresponding to the historical time segment ;

According to the first error corresponding to each historical time segment, determine the first mean value and the first mean square error corresponding to the historical time segment;

determining the absolute value of the difference between the value corresponding to the first data information corresponding to the current time segment and the value corresponding to the second data information as the second error corresponding to the current time segment;

The absolute value of the difference between the second error and the first mean value is determined as the third error of the current time segment relative to the historical time segment;

If the third error does not exceed the first mean square error, then determine that the data statistical result information indicates that no data loss has occurred; if the third error exceeds the first mean square error, then determine that the data statistical result information indicates that data loss occurs;

Wherein, the historical time segment includes multiple historical time segments of the same duration, and the current time segment and the historical time segment have the same duration.

Further, the log information included in each node on the data link is collected in real time through the log collection module, including:

Through the log collection tool included in the log collection module, collect the log information corresponding to the first node of the binary log consumer; collect the log information corresponding to the second node of the message queue middleware storing the binary log; collect the message consumption corresponding to the binary log The log information of the third node of the user; the log information of the fourth node corresponding to the real-time data warehouse is collected;

The log information included in each node collected by the log collection tool is stored through the log storage tool included in the log collection module; and the log information is sent to the data analysis module.

Further, the log information of each node collected by the log collection module is compared by the data analysis module to determine the log statistical result information, including:

Compare the log information included in each node collected by the log collection module at a set frequency, and determine the problem node with data inconsistency on the data link;

Determine the problem node as log statistics result information.

Further, the data accuracy monitoring method also includes:

Through the visual processing module, the data statistical result information and/or the log statistical result information determined by the data analysis module are visually processed.

The data accuracy monitoring method provided by the embodiments of the present invention can be implemented based on the data accuracy monitoring system provided by any of the above embodiments, belongs to the same inventive concept, and has corresponding functions and beneficial effects.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present invention may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution of the present invention can be achieved, there is no limitation herein.

The above specific implementation methods do not constitute a limitation to the protection scope of the present invention. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A data accuracy monitoring system, characterized in that, comprising: The data statistics module is used to determine the first data information of the first statistics table in the tenant library to be counted from the database cluster included in the production system; The second data information of the second table to be counted corresponding to the table to be counted; The log collection module is used to collect log information included in each node on the data link in real time; The data analysis module is used to connect the data statistics module and the log collection module respectively; by comparing the first data information and the second data information determined by the data statistics module, determine the data statistics result information; By comparing the log information of each node collected by the log collection module, determine the log statistical result information; Wherein, the number of the database clusters is one or more, and each of the database clusters includes one or more tenant databases, and each tenant database includes one or more first tables to be counted; the second table to be counted The first table to be counted is transmitted to the real-time data warehouse through a data link. 2. system according to claim 1, is characterized in that, described data statistics module comprises: Data statistics sub-module and data storage sub-module; The data statistics sub-module is used to, from the database cluster included in the production system, determine the first data information of the first statistics table in the tenant library to be counted; The second data information of the second table to be counted corresponding to the first table to be counted; The data storage sub-module is used to respectively connect the data statistics sub-module and the data analysis module; store the first data information and the second data information determined by the data statistics sub-module; Send the first data information and the second data information to the data analysis module. 3. system according to claim 2, is characterized in that, described data statistics submodule comprises: a service interface unit, configured to provide an interface for accessing the system for the first table to be counted and the second table to be counted; The task queue unit is used to store statistical demand tasks waiting to be processed, and the statistical demand task is to determine the first data information of the first table to be counted, and/or to determine the second data information of the second table to be counted; The tenant library management unit is used to synchronize the configuration of each tenant library in each database cluster from the production system, and divide each tenant library in the same database cluster into the same group; A business abstraction unit, configured to abstract the data in the first table to be counted and/or the second table to be counted; A task execution management unit, used to manage concurrently processed statistical demand tasks, and/or manage the number of tenant libraries connected to the system; A task execution unit, configured to embody the data abstracted and processed by the business abstraction unit into executable statements, and determine the first data information and/or the second data information; A result output unit, configured to send the first data information and/or the second data information to the data storage sub-module. 4. The system according to claim 3, wherein the task execution management unit is specifically configured to: The concurrently processed statistical demand tasks are centrally managed through the thread manager, and the number of concurrently processed statistical demand tasks is linearly related to the number of tenant libraries; The number of tenant databases connected to the system is managed through the database connection manager in combination with the tenant database management unit. 5. The system according to claim 4, wherein the task execution management unit is specifically used for: Release the connection between the tenant library and the system without statistical demand tasks through the database connection manager; When there is no statistical demand task, the thread manager is used to release the thread. 6. The system according to claim 1, wherein the data analysis module is specifically used for: In the historical time segment, the absolute value of the difference between the value corresponding to the first data information and the value corresponding to the second data information corresponding to each historical time segment in the historical time segment is determined as the corresponding value of the historical time segment the first error of determining a first mean value and a first mean square error corresponding to the historical time segment according to the first error corresponding to each of the historical time segments; determining the absolute value of the difference between the value corresponding to the first data information corresponding to the current time segment and the value corresponding to the second data information as the second error corresponding to the current time segment; determining the absolute value of the difference between the second error and the first mean value as a third error of the current time segment relative to the historical time segment; If the third error does not exceed the first mean square error, then determine that the data statistical result information indicates that no data loss has occurred; if the third error exceeds the first mean square error, then determine the data statistical result A message indicating that data loss has occurred; Wherein, the historical time segment includes a plurality of historical time segments of the same duration, and the current time segment is of the same duration as the historical time segment. 7. The system according to claim 1, wherein the log collection module comprises: The log collection tool is used to collect the log information corresponding to the first node of the binary log consumer; collect the log information corresponding to the second node of the message queue middleware storing the binary log; collect the third node corresponding to the message consumer processing the binary log The log information of the node; collect the log information of the fourth node corresponding to the real-time data warehouse; The log storage tool is used to respectively connect the log collection tool and the data analysis module; store the log information included in each node collected by the log collection tool; and send the log information to the data analysis module. 8. The system according to claim 1, wherein the data analysis module is specifically used for: Comparing the log information included in each node collected by the log collection module with a set frequency to determine the problem node with data inconsistency on the data link; The problem node is determined as the log statistics result information. 9. The system according to claim 1, further comprising: The visualization processing module is configured to connect to the data analysis module; and visually process the data statistical result information and/or the log statistical result information determined by the data analysis module. 10. A data accuracy monitoring method, characterized in that it is applied to the data accuracy monitoring system according to any one of claims 1-9, comprising: Determining the first data information of the first table to be counted in the tenant database to be counted from the database cluster included in the production system through the data statistics module; Determining the second data information of the second table to be counted corresponding to the first table to be counted from the real-time data warehouse included in the production system through the data statistics module; The log information included in each node on the data link is collected in real time through the log collection module; comparing the first data information and the second data information determined by the data statistics module through the data analysis module to determine the data statistics result information; The log information of each node collected by the log collection module is compared by the data analysis module to determine the log statistical result information.

Images