TWI888283B - Processing method of data quality dynamic information and data processing device - Google Patents

Abstract

A data processing device is provided, which includes a storage device and a data quality assessment measurement calculation module. The storage device is configured to obtain and store change data of data tables from a data center. The data quality assessment measurement calculation module is configured to calculate an updated data quality assessment measurement value and an updated data quality characteristic data according to the change data and a data quality assessment measurement reference value, record the updated data quality assessment measurement value and the updated data quality characteristic data into a data quality assessment measurement value table, and perform a notification function according to the updated data quality assessment measurement value and a data quality threshold. The data quality assessment measurement calculation module is configured to compare the updated data quality assessment measurement value with the data quality threshold to generate a comparison result. The data quality assessment measurement calculation module is configured to generate and send a notification signal to perform a notification function when the comparison result indicates that the updated data quality assessment measurement value is greater than or equal to the data quality threshold.

Description

Material quality dynamic information processing method and data processing device

The present invention relates to a data quality dynamic information processing method and a data processing device, and in particular to a data quality dynamic information processing method and a data processing device that are beneficial to digital transformation research and development.

Data quality is a concept that measures the quality of data in a data form. At the same time, it is also an important basis for evaluating whether a data form is worth the cost of data analysis, data mining, or digital application development. However, there is no uniform standard definition of data quality, and it usually depends on the purpose of the data. For example, many times the amount of data in a data form is not enough to train an artificial intelligence model. Generally speaking, before conducting data analysis, mining, or digital application development, in order to avoid the R&D costs of digital transformation going to waste, data users must constantly confirm whether the data quality is good enough to conduct research and development. However, there are usually thousands of data forms with large amounts of data in the data center, and they are constantly being updated. In order to start research and development work as quickly as possible, data users must download all the data in the huge data spreadsheets to their local machines every day to calculate the data quality values. However, this will consume a lot of computing resources of data users and data centers, and it will easily endanger system information security. Therefore, the existing technology really needs to be improved.

In order to solve the above problems, the present invention provides a data quality dynamic information processing method and data processing device that are beneficial to digital transformation research and development to solve the above problems.

The present invention provides a data processing device, comprising: a storage device for obtaining and storing a change data of a data table of a data center from a data center; and a data quality evaluation measurement calculation module for calculating an updated data quality evaluation measurement value and an updated data quality feature data according to the change data and a data quality evaluation measurement reference value, and storing the updated data quality evaluation measurement value and the updated data quality feature data. The data is recorded in a data quality assessment measurement value table, and a notification function is executed according to the updated data quality assessment measurement value and a data quality valve value; wherein the data quality assessment measurement calculation module compares the updated data quality assessment measurement value with the data quality valve value to generate a comparison result, and when the comparison result shows that the updated data quality assessment measurement value is greater than or equal to the data quality valve value, a notification signal is generated and sent to execute the notification function.

The present invention also provides a data quality dynamic information processing method, comprising: obtaining and storing changed data from a data center; calculating a data quality evaluation measurement value and a data quality characteristic data of the changed data according to the changed data; calculating an updated data quality evaluation measurement value and an updated data quality characteristic data according to the data quality evaluation measurement value and the data quality characteristic data of the changed data and a data quality evaluation measurement reference value; and updating the updated data quality evaluation measurement value and the updated data quality characteristic data. The evaluation measurement value and the updated data quality characteristic data are recorded in a data quality evaluation measurement value table; and a notification function is executed according to the updated data quality evaluation measurement value and a data quality valve value, including comparing the updated data quality evaluation measurement value with the data quality valve value to generate a comparison result and generating and sending a notification signal when the comparison result shows that the updated data quality evaluation measurement value is greater than or equal to the data quality valve value to execute the notification function.

1:Data processing system

10: Data Center

2: Process

20: Data processing device

200: Data quality assessment measurement calculation module

202: Storage device

204: Data quality assessment measurement value form

206: User information form

208: User Interface

S200, S202, S204, S206, S208, S210, S212: Steps

Figure 1 is a schematic diagram of a data processing system according to the first embodiment of the present invention.

Figure 2 is a schematic diagram of a process of one embodiment of the present invention.

Figure 3 is a schematic diagram of the user interface of an embodiment of the present invention visually displaying data quality information.

Figure 4 is a schematic diagram of the automatic notification of data quality information in an embodiment of the present invention.

Please refer to Figure 1, which is a schematic diagram of a data processing system 1 of an embodiment of the present invention. The data processing system 1 includes a data center 10 and a data processing device 20. The data center 10 stores a plurality of data forms. The data center 10 may include a database to store the data forms. The data center 10 may include a plurality of networked computer devices that work, process, store and share data in collaboration with each other. The embodiment of the present invention provides a data processing device 20, which allows data users of the data center 10 to automatically track and evaluate the data quality of data forms of any amount of data without occupying the computing resources of the data center 10, which will be very helpful in promoting digital transformation research and development.

The data processing device 20 includes a data quality assessment measurement calculation module 200, a storage device 202, a data quality assessment measurement value form 204, and a user information form 206. The storage device 202 can be used to store data. For example, the storage device 202 can store change data related to the data center 10, and the change data can include new data to be added to the data form of the data center 10 or deleted data to be deleted from the data form of the data center 10. The data quality assessment measurement calculation module 200 can access the data stored in the storage device 202, and the data quality assessment measurement calculation module 200 can also receive and process data from an external device. The data quality assessment measurement calculation module 200 can be a main controller or processing device, such as a central processing unit (CPU), a microprocessor, or a micro controller unit (MCU), but is not limited thereto. The data quality assessment measurement calculation module 200 can access the data content of the data quality assessment measurement value table 204 and the user information table 206. The data quality assessment measurement calculation module 200 can record the relevant data in the data quality assessment measurement value table 204 and the user information table. The data quality assessment measurement calculation module 200 can calculate the updated data quality assessment measurement value and the updated data quality characteristic data according to the changed data and the data quality assessment measurement reference value, and record the updated data quality assessment measurement value and the updated data quality characteristic data into the data quality assessment measurement value form 206. The data quality assessment measurement value form 204 includes all forms of data quality related information used to calculate the data tables of the data center 10. The data quality assessment measurement value form 204 includes information such as the data table name, the data table field name, the data change behavior, the data quality assessment measurement value, the data quality assessment characteristic data and the update time. The user information form 206 includes a form for user setting information. The user information form 206 includes information such as the user account, table name, data field name, data quality measurement name, data quality threshold, and the terminal device (user device) to be notified.

The embodiment of the present invention can automatically and dynamically track and evaluate the data quality changes of the data table of the data center 10 through the data processing device 20, without having to re-acquire the overall system data of the data center to evaluate the overall data quality. Please refer to Figure 2, which is a schematic diagram of process 2 of one embodiment of the present invention. Process 2 includes the following steps:

Step S200: Start.

Step S202: The storage device stores the changed data.

Step S204: Calculate a data quality evaluation measurement value and a data quality feature data of the changed data based on the changed data.

Step S206: Calculate an updated data quality assessment measurement value and an updated data quality feature data based on the changed data and a data quality assessment measurement reference value.

Step S208: Record the updated data quality assessment measurement value and the updated data quality feature data into a data quality assessment measurement value table.

Step S210: Execute a notification function based on the updated data quality evaluation measurement value and a data quality threshold.

Step S212: End.

According to process 2, in step S202, the data center 10 may notify the data processing device 20 of data change information (including addition, deletion and update) related to all its data forms. After the data processing device 20 receives the data change information of the data form from the data center 10, the storage device 202 may store the change data and the data change information of the data form. The change data includes the added data to be added to the multiple data forms of the data center 10 or the deleted data to be deleted from the multiple data forms of the data center 10. For example, the data center 10 can transmit the data change information of the data form and the change data including the new data to be added to the data form or the deleted data to be deleted from the data form to the storage device 202 for storage. For example, the storage device 202 can access the data center 10 according to the received data change information of the data form to obtain the change data including the new data to be added to the data form or the deleted data to be deleted from the data form from the data center 10 and store the obtained change data.

In step S204, the data quality assessment measure calculation module 200 can read the change data stored in the storage device 202, and calculate a data quality assessment measure value and a data quality feature data of the change data according to the change data. For example, when the data quality assessment measure value includes a mean measure value, the data quality assessment measure calculation module 200 can calculate the mean of the change data as the data quality assessment measure value of the change data. For example, when the data quality assessment measure value includes a parent variance measure value, the data quality assessment measure calculation module 200 can calculate the parent variance of the change data as the data quality assessment measure value of the change data. In addition, after the data quality assessment measurement calculation module 200 reads the changed data stored in the storage device 202, the storage device 202 can also release the space for storing data to save storage space.

In step S206, the data quality assessment measurement calculation module 200 calculates an updated data quality assessment measurement value and an updated data quality characteristic data according to the changed data and a data quality assessment measurement reference value. The data quality assessment measurement reference value includes a pre-update data quality assessment measurement value and a pre-update data quality characteristic data. The data quality assessment measurement value table 204 stores the latest data quality assessment measurement reference value. The data quality assessment measurement value table 204 records the data quality assessment measurement reference value before the changed data is updated to the data center 10 (i.e., the pre-update data quality assessment measurement value and the pre-update data quality characteristic data). The data quality evaluation measure calculation module 200 can access the data quality evaluation measure value table 204 to obtain the pre-update data quality evaluation measure value and the pre-update data quality characteristic data. The data quality evaluation measure calculation module 200 can calculate the post-update data quality evaluation measure value and the post-update data quality characteristic data based on the function related to the pre-update data quality evaluation measure value, the pre-update data quality characteristic data, the data quality evaluation measure value of the changed data, and the data quality characteristic data of the changed data. The post-update data quality evaluation measure value and the post-update data quality characteristic data can be expressed as follows: M _new = _FM (M _Ori , M _change , C _Ori , C _change ) C _new = _FC (C _Ori , C _change ) (1)

Among them, M _new represents the data quality evaluation measurement value after the update, M _Ori represents the data quality evaluation measurement value before the update, M _change represents the data quality evaluation measurement value of the changed data, FM(.) and FC(.) represent functions, C _new represents the data quality feature data after the update, C _Ori represents the data quality feature data before the update, and C _change represents the data quality feature data of the changed data.

In one embodiment, the data quality evaluation measurement value before updating includes a missing value ratio (MR) measurement value before updating, the data quality evaluation measurement value of the changed data includes a missing value ratio measurement value of the changed data, and the data quality evaluation measurement value after updating includes a missing value ratio measurement value after updating. The data quality evaluation measurement calculation module 200 calculates the missing value ratio measurement value after updating and the data quality feature data after updating according to the missing value ratio measurement value before updating, the missing value ratio measurement value of the changed data, and the data quality feature data of the changed data. When the changed data is new data to be added to multiple data tables, the data quality evaluation measurement calculation module 200 calculates the updated null value ratio measurement value and the updated data quality feature data according to the following formula:

Among them, MR _new represents the measurement value of the null value ratio after updating, MR _Ori represents the measurement value of the null value ratio before updating, MR _change represents the measurement value of the null value ratio of the changed data, C _new represents the data quality feature data after updating, C _Ori represents the data quality feature data before updating, C _change represents the data quality feature data of the changed data, N represents the amount of data used to calculate the data quality evaluation measurement value before updating, that is, the data amount N is the amount of data used to calculate the data quality evaluation measurement value before updating before the changed data are changed in the data table of the data center 10, wherein C _Ori =N, k represents the amount of data used to calculate the data quality evaluation measurement value of the changed data, wherein C _change =k.

When the changed data is deleted data to be deleted from the plurality of data tables, the data quality evaluation measurement calculation module 200 calculates the updated null value ratio measurement value and the updated data quality characteristic data according to the following formula:

Where, C _Ori =N, and C _change =K.

In one embodiment, the data quality assessment measurement value before updating includes a mean measurement value before updating and a variance measurement value before updating, the data quality assessment measurement value of the changed data includes a mean measurement value and a variance measurement value of the changed data, and the data quality assessment measurement value after updating includes a mean measurement value after updating and a variance measurement value after updating. The data quality evaluation measurement calculation module 200 calculates the updated mean measurement value, the updated parent variation measurement value, and the updated data quality feature data according to the pre-updated mean measurement value, the pre-updated parent variation measurement value, the pre-updated data quality feature data, the mean measurement value of the abnormal data, the parent variation measurement value of the abnormal data, and the data quality feature data of the abnormal data. When the abnormal data is new data to be added to a plurality of data tables, the data quality evaluation measurement calculation module 200 calculates the updated mean measurement value, the updated parent variation measurement value, and the updated data quality feature data according to the following formula:

Among them, μ _new represents the mean measurement value after updating, μ _Ori represents the mean measurement value before updating, μ _change represents the mean measurement value of the changed data, σ ² _new represents the matrix variation measurement value after updating, σ ² _Ori represents the matrix variation measurement value before updating, σ ² _change represents the matrix variation measurement value of the changed data, C _new represents the data quality feature data after updating, C _Ori represents the data quality feature data before updating, C _change represents the data quality feature data of the changed data, N represents the data amount used to calculate the mean measurement value and the matrix variation measurement value before updating, wherein C _Ori =N, k represents the data amount used to calculate the mean measurement value and the matrix variation measurement value of the changed data, wherein C _change =k.

When the changed data is deleted data to be deleted from multiple data tables, the data quality assessment measurement calculation module 200 calculates the updated mean measurement value, the updated parent variance measurement value, and the updated data quality feature data according to the following formula:

Where C _change =k.

In one embodiment, the pre-update data quality assessment measure value includes a pre-update matrix correlation coefficient (correlation coefficient) measurement value, the data quality evaluation measurement value of the changed data includes a matrix correlation coefficient measurement value of the changed data, and the updated data quality evaluation measurement value includes an updated matrix correlation coefficient measurement value. The data quality evaluation measurement calculation module 200 calculates the updated matrix correlation coefficient measurement value and the updated data quality feature data according to the matrix correlation coefficient measurement value before updating, the data quality feature data before updating, the matrix correlation coefficient measurement value of the changed data, and the data quality feature data of the changed data. When the changed data is new data to be added to a plurality of data tables, the data quality evaluation measurement calculation module 200 calculates the updated matrix correlation coefficient measurement value and the updated data quality feature data according to the following formula:

Among them, ρ _new represents the measurement value of the matrix correlation coefficient after update, ρ _Ori represents the measurement value of the matrix correlation coefficient before update, ρ _change represents the measurement value of the matrix correlation coefficient of the changed data, C _new represents the data quality characteristic data after update, x, y represent variables (representing different data groups), σ _{x_new} , σ _{y_new} represent the standard deviation of the variable after update, σ _{x_Ori} , σ _{y_Ori} represent the standard deviation of the variable before update, σ _{x_change} , σ _{y_change} represent the standard deviation of the variable of the changed data, μ _{y_new} represents the mean measurement value of the variable after update, μ _{x_Ori} , μ _{y_Ori} represent the mean measurement value of the variable before update, μ _{x_change} , μ _{y_change} represent the mean measurement value of the variable of the changed data, among them, C _new =(N+k,μ _new ,σ ² _new ).

When the changed data is deleted data to be deleted from multiple data tables, the data quality evaluation measurement calculation module 200 calculates the updated matrix correlation coefficient and the updated data quality characteristic data according to the following formula:

Among them, C _new =(N+k,μ _new ,σ ² _new ).

In one embodiment, the data quality assessment measurement value before updating includes a unique value ratio (UR) measurement value before updating, the data quality assessment measurement value of the changed data includes a unique value ratio measurement value of the changed data, and the data quality assessment measurement value after updating includes a unique value ratio measurement value after updating. The data quality evaluation measurement calculation module 200 calculates the updated unique value ratio measurement value and the updated data quality feature data according to the unique value ratio measurement value before the update, the data quality feature data before the update, the unique value ratio measurement value of the changed data, and the data quality feature data of the changed data. When the changed data is new data to be added to a plurality of data tables, the data quality evaluation measurement calculation module 200 calculates the updated unique value ratio measurement value and the updated data quality feature data according to the following formula:

Wherein, UR _new represents the unique value ratio measurement value after update, ∪ represents the union operation, key(.) represents the key value, # represents the number calculation, MR _Ori represents the unique value ratio measurement value before update, MR _change represents the unique value ratio measurement value of the changed data, C _new represents the data quality feature data after update, C _Ori represents the data quality feature data before update and the unique value ratio measurement value before update, C _change represents the data quality feature data of the changed data and the unique value ratio measurement value, and N represents the amount of data used to calculate the data quality evaluation measurement value before update: When the changed data is the deleted data to be deleted from a plurality of data tables, the data quality evaluation measurement operation module 200 calculates the unique value ratio measurement value after update and the data quality feature data after update according to the following formula;

For example, the unique value ratio measure before update C _ori ={"Ivy":2,"Ben":3}, which means that the data has two unique values, Ivy and Ben, of which Ivy has 2 and Ben has 3. In addition, if the unique value ratio measure of the changed data C _change ={"Ivy":2,"Ben":1}, it means that the data has two unique values, Ivy and Ben, of which Ivy has 2 and Ben has 1. In this case, C _ori ∪ C _change ={"Ivy":4,"Ben":4} in formula (8), and C _ori -C _change ={"Ben":2} in formula (9).

In other words, the embodiment of the present invention only needs to obtain the changed data and then calculate the updated data quality evaluation measurement value and feature data based on the data quality evaluation measurement reference value before the update, so that the current data quality can be understood in real time without having to download and obtain the entire system data again. Furthermore, for the data center, the data center only needs to provide the changed data information of the added data added to the data form or the deleted data deleted from the data form to the data processing device 20, which can effectively save the computing resources of the data center

In step S208, the data quality assessment measurement calculation module 200 can record the updated data quality assessment measurement value and the updated data quality characteristic data into the data quality assessment measurement value table 204, so that when there are other data changes in the data table of the data center 10, it can be used as the data quality assessment measurement value information before the data change. On the other hand, the data processing system 1 also includes a user interface, which can be used to provide a visualization service for querying the data quality assessment measurement value table 204. For example, please refer to Figure 3, which is a schematic diagram of the user interface visually displaying data quality information in an embodiment of the present invention. As shown in FIG. 3 , the user interface visually displays the row count, consistency, outlier, missing value, and redundant information of the data table related to the data center 10. The user interface may also provide a setting interface for related functions (such as automatic notification function) so that the user can input the required settings.

In step S210, the data quality assessment measurement calculation module 200 can determine whether to execute a notification function based on the updated data quality assessment measurement value and a data quality valve value. The data quality assessment measurement calculation module 200 can compare the updated data quality assessment measurement value with the data quality valve value to generate a comparison result. When the comparison result shows that the updated data quality assessment measurement value is greater than or equal to the data quality valve value, the data quality assessment measurement calculation module 200 can generate and send a notification signal to a user device (the device object can be pre-set) to remind the user that the data quality has reached the expected level. For example, please refer to Figure 4, which is a schematic diagram of automatically notifying data quality information in an embodiment of the present invention. As shown in FIG. 4 , the data processing system 1 provides a user interface 208 for the user to input the data quality valve value required for setting. After the user inputs the data quality valve value into the user information form 206 through the user interface 208, the data quality assessment measurement calculation module 200 can access the user information form 206 to obtain a data quality valve value and the user device information to be notified. The data quality assessment measurement calculation module 200 can access the data quality assessment measurement value form 204 to obtain the updated data quality assessment measurement value. Then, the data quality assessment measurement calculation module 200 compares the updated data quality assessment measurement value with the data quality valve value to generate a comparison result. When the comparison result shows that the updated data quality assessment measurement value is greater than or equal to the data quality threshold, the data quality assessment measurement calculation module 200 can generate and send a notification signal to a user device UE to remind the administrator, engineer or user of the data center that the data quality of the data center has reached the expected level and the relevant research and development work can be started immediately to realize the execution notification function. For example, the data quality assessment measurement calculation module 200 can use communication software such as E-mail, Teams, etc. to notify the data user of the user device UE that the current data quality assessment measurement value has exceeded the data quality threshold. In this way, the data user can start the research and development of data analysis, exploration or digital application in the fastest time.

Therefore, after the user inputs the data quality threshold into the user information form 206, the data processing device 20 of the embodiment of the present invention will calculate the latest data quality evaluation measurement value based on the data change in the data center 10, and simultaneously connect the user information form 206 to obtain the data quality threshold value to evaluate whether the updated data quality evaluation measurement value reaches the data quality threshold value. Once the data quality reaches the level set by the user, the notification function will be executed to notify the preset user device, thereby realizing a customized automatic notification function, so that the user does not need to monitor the changes in data quality all the time.

A person of ordinary skill in the art may combine, modify or change the above-described embodiments according to the spirit of the present invention, but is not limited thereto. All of the above statements, steps, and/or processes (including recommended steps) may be implemented through hardware, software, firmware (i.e., a combination of hardware devices and computer instructions, where the data in the hardware devices are read-only software data), electronic systems, or a combination of the above devices. The device may be a data processing system 1. The hardware may include analog, digital and hybrid circuits (i.e., microcircuits, microchips or silicon chips). For example, the hardware may be an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic element, a coupled hardware element, or a combination of the above hardware. In other embodiments, the hardware may include a general purpose processor, a microprocessor, a controller, a digital signal processor (DSP), or a combination of the above hardware. The software may be a combination of program codes, a combination of instructions, and/or a combination of functions (functionality) stored in a storage device, such as a computer-readable recording medium or a non-transitory computer-readable medium. For example, the computer-readable recording medium may include read-only memory (ROM), flash memory, random-access memory (RAM), subscriber identity module (SIM), hard disk, floppy disk or CD-ROM/DVD-ROM/BD-ROM, but is not limited thereto. The data processing system 1 of the embodiment of the present invention may include a processing circuit and a storage device 202. The process steps and embodiments of the present invention may be compiled into a program code or instruction form and stored in the storage device 202. The processing circuit can be used to read and execute the program code or instructions stored in the storage device 202 to implement all the aforementioned steps and functions.

In summary, the embodiment of the present invention provides a data processing device that enables data users in a data center to automatically track and evaluate data quality for data forms of any amount of data. The embodiment of the present invention only needs to obtain the changed data to calculate the current data quality evaluation measurement value without having to re-download and obtain the entire system data. In addition, once the data quality reaches the level set by the user, the notification function will be executed to notify the preset user device, thereby realizing a customized automatic notification function, so that the user does not need to monitor the changes in data quality all the time. At the same time, for the data center, the data center only needs to provide the changed data information of the new data added to the data form or the deleted data deleted from the data form to the data processing device, which can effectively save the computing resources of the data center, and will be very helpful to promote the research and development of digital transformation.

The above is only the preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the patent application of the present invention shall fall within the scope of the present invention.

2: Process

S200, S202, S204, S206, S208, S210, S212: Steps

Claims (8)

A data processing device includes: a storage device for obtaining and storing a change data of a data table of a data center from a data center; and a data quality evaluation measurement calculation module for calculating a post-update data quality evaluation value based on a data quality evaluation measurement value of the change data, a data quality characteristic data of the change data, a data quality evaluation measurement value of a data quality evaluation measurement reference value before updating, and a data quality characteristic data of the data quality evaluation measurement reference value before updating. The method comprises: collecting the updated data quality evaluation measurement value and an updated data quality characteristic data, recording the updated data quality evaluation measurement value and the updated data quality characteristic data in a data quality evaluation measurement value table, and executing a notification function according to the updated data quality evaluation measurement value and a data quality valve value; wherein the data quality evaluation measurement calculation module compares the updated data quality evaluation measurement value with the data quality valve value to generate a comparison result, and displays in the comparison result that the updated data quality evaluation measurement value is greater than or equal to When the data quality threshold is reached, a notification signal is generated and sent to execute the notification function; wherein the changed data includes the newly added data to be added to the plurality of data tables or the deleted data to be deleted from the plurality of data tables, the data quality evaluation measurement value before the update includes a null value ratio measurement value before the update, the data quality evaluation measurement value of the changed data includes a null value ratio measurement value of the changed data, the data quality evaluation measurement value after the update includes a null value ratio measurement value after the update, the data quality evaluation measurement value The estimation measurement calculation module calculates the post-update null value ratio measurement value and the post-update data quality characteristic data according to the pre-update null value ratio measurement value, the pre-update data quality characteristic data, the null value ratio measurement value of the changed data, and the data quality characteristic data of the changed data, wherein when the changed data is new data to be added to the plurality of data tables, the data quality assessment measurement calculation module calculates the post-update null value ratio measurement value and the post-update data quality characteristic data according to the following formula; in, Indicates the null value ratio measurement value after the update. Indicates the null value ratio measurement value before the update. Indicates the null value ratio measurement value of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the changed data, N represents the amount of data used to calculate the data quality evaluation measurement value before the update, where = N, k represents the amount of data used to calculate the data quality evaluation measure value of the change data, where = k; wherein when the changed data is deleted data to be deleted from the plurality of data tables, the data quality evaluation measurement calculation module calculates the updated null value ratio measurement value and the updated data quality characteristic data according to the following formula; in, Indicates the null value ratio measurement value after the update. Indicates the null value ratio measurement value before the update. Indicates the null value ratio measurement value of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the changed data, N represents the amount of data used to calculate the data quality evaluation measurement value before the update, where = N, K represents the amount of data used to calculate the data quality evaluation measure value of the change data, where =K. A data processing device as described in claim 1, wherein the changed data includes new data to be added to the plurality of data tables or deleted data to be deleted from the plurality of data tables, the data quality assessment measure value before the update includes a mean measure value before the update and a matrix variance measure value before the update, the data quality assessment measure value of the changed data includes a mean measure value and a matrix variance measure value of the changed data, the data quality assessment measure value after the update includes a mean measure value after the update and a matrix variance measure value after the update, and the data quality assessment measure calculation module calculates the data quality assessment measure value according to the data quality assessment measure value before the update. The updated mean measurement value, the updated matrix variation measurement value, and the updated data quality characteristic data are calculated by the mean measurement value, the matrix variation measurement value before the update, the data quality characteristic data before the update, the mean measurement value of the abnormal data, the matrix variation measurement value of the abnormal data, and the data quality characteristic data of the abnormal data, wherein when the abnormal data is the newly added data to be added to the plurality of data tables, the data quality assessment measurement calculation module calculates the updated mean measurement value, the updated matrix variation measurement value, and the updated data quality characteristic data according to the following formula; in, represents the updated mean measurement value, represents the mean measurement value before the update, represents the mean measurement value of the change data. represents the updated measure of the population variance, represents the measurement value of the parent population variance before the update, represents the measure of the parent population variance of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the mean measurement value before the update and the matrix variance measurement value before the update, where = N, k represents the amount of data used to calculate the mean measurement value and the parent variance measurement value of the change data, where = k; wherein when the changed data is deleted data to be deleted from the plurality of data tables, the data quality assessment measurement calculation module calculates the updated mean measurement value, the updated parent population variance measurement value and the updated data quality characteristic data according to the following formula; in, represents the updated mean measurement value, represents the mean measurement value before the update, represents the mean measurement value of the change data. represents the updated measure of the population variance, represents the measurement value of the parent population variance before the update, represents the measure of the parent population variance of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the mean measurement value before the update and the matrix variance measurement value before the update, where = N, k represents the amount of data used to calculate the mean measurement value and the parent variance measurement value of the change data, where =k. A data processing device as described in claim 1, wherein the changed data includes new data to be added to the plurality of data tables or deleted data to be deleted from the plurality of data tables, the pre-update data quality assessment measure value includes a pre-update matrix correlation coefficient measure value, the data quality assessment measure value of the changed data includes a matrix correlation coefficient measure value of the changed data, the post-update data quality assessment measure value includes a post-update matrix correlation coefficient measure value, and the data quality assessment measure calculation module The updated matrix correlation coefficient measurement value and the updated data quality characteristic data are calculated according to the matrix correlation coefficient measurement value before the update, the data quality characteristic data before the update, the matrix correlation coefficient measurement value of the changed data, and the data quality characteristic data of the changed data, wherein when the changed data is new data to be added to the plurality of data tables, the data quality evaluation measurement calculation module calculates the updated matrix correlation coefficient measurement value and the updated data quality characteristic data according to the following formula; in, represents the updated matrix correlation coefficient measure value, represents the value of the matrix correlation coefficient before the update, Indicates the value of the parent correlation coefficient of the change data. represents the updated data quality feature data, x, y represent variables, represents the updated standard deviation of the variable, represents the standard deviation of the variable before updating, Indicates the standard deviation of the variable of the change data. represents the updated mean measurement value of the variable, represents the mean measurement value of the variable before updating, Represents the mean measurement value of the variable of the change data, where, ; Wherein when the changed data is deleted data to be deleted from the plurality of data tables, the data quality assessment measurement calculation module calculates the updated matrix correlation coefficient and the updated data quality characteristic data according to the following formula; in, represents the updated matrix correlation coefficient measure value, represents the value of the matrix correlation coefficient before the update, Indicates the value of the parent correlation coefficient of the change data. represents the updated data quality feature data, x, y represent variables, represents the updated standard deviation of the variable, represents the standard deviation of the variable before updating, Indicates the standard deviation of the variable of the change data. represents the updated mean measurement value of the variable, represents the mean measurement value of the variable before updating, Represents the mean measurement value of the variable of the change data, where, . A data processing device as described in claim 1, wherein the changed data includes new data to be added to the plurality of data tables or deleted data to be deleted from the plurality of data tables, the pre-update data quality assessment measure value includes a pre-update unique value ratio measure value, the data quality assessment measure value of the changed data includes a unique value ratio measure value of the changed data, the post-update data quality assessment measure value includes a post-update unique value ratio measure value, and the data quality assessment measure calculation module The updated unique value ratio measurement value and the updated data quality characteristic data are calculated based on the unique value ratio measurement value before the update, the data quality characteristic data before the update, the unique value ratio measurement value of the changed data, and the data quality characteristic data of the changed data, wherein when the changed data is new data to be added to the plurality of data tables, the data quality assessment measurement calculation module calculates the updated unique value ratio measurement value and the updated data quality characteristic data according to the following formula; in, Represents the unique value ratio measurement value after the update. represents a union operation, key(．) represents a key value, and # represents a count. It represents the unique value ratio measurement value before the update. The unique value ratio measure of the change data. Indicates the updated data quality feature data. represents the data quality feature data before the update and the unique value ratio measurement value before the update, represents the data quality characteristic data and the unique value ratio measurement value of the changed data, and N represents the amount of data used to calculate the data quality assessment measurement value before the update; wherein when the changed data is deleted data to be deleted from the plurality of data tables, the data quality assessment measurement calculation module calculates the unique value ratio measurement value after the update and the data quality characteristic data after the update according to the following formula; in, Represents the unique value ratio measurement value after the update. represents a union operation, key(．) represents a key value, and # represents a count. It represents the unique value ratio measurement value before the update. The unique value ratio measure of the change data. Indicates the updated data quality feature data. represents the data quality feature data before the update and the unique value ratio measurement value before the update, represents the data quality characteristic data and the unique value ratio measurement value of the changed data, and N represents the amount of data used to calculate the data quality assessment measurement value before the update. A data quality dynamic information processing method includes: obtaining and storing changed data from a data center; calculating a data quality evaluation measurement value and a data quality characteristic data of the changed data according to the changed data; calculating an updated data quality evaluation measurement value according to the data quality evaluation measurement value of the changed data, the data quality characteristic data of the changed data, a data quality evaluation measurement value before updating of a data quality evaluation measurement reference value, and a data quality characteristic data before updating of the data quality evaluation measurement reference value. and an updated data quality characteristic data; recording the updated data quality evaluation measurement value and the updated data quality characteristic data into a data quality evaluation measurement value table; and executing a notification function according to the updated data quality evaluation measurement value and a data quality valve value, including comparing the updated data quality evaluation measurement value with the data quality valve value to generate a comparison result and generating and sending a notification signal when the comparison result shows that the updated data quality evaluation measurement value is greater than or equal to the data quality valve value, so as to Execute the notification function; wherein the changed data includes new data to be added to a plurality of data tables or deleted data to be deleted from the plurality of data tables, wherein the step of calculating the updated data quality assessment measurement value and the updated data quality characteristic data based on the data quality assessment measurement value and the data quality characteristic data before the update of the data quality assessment measurement reference value includes: calculating the updated data quality assessment measurement value and the updated data quality characteristic data based on the data quality assessment measurement value and the data quality characteristic data before the update of the data quality assessment measurement reference value; The method further comprises calculating a post-update null value ratio measurement value of the updated data quality assessment measurement value and the post-update data quality feature data according to a null value ratio measurement value before the update of the measurement value, the data quality feature data before the update, a null value ratio measurement value of the data quality assessment measurement value of the changed data, and the data quality feature data of the changed data, wherein when the changed data is new data to be added to the plurality of data tables, the post-update null value ratio measurement value and the post-update data quality feature data are calculated according to the following formula: in, Indicates the null value ratio measurement value after the update. Indicates the null value ratio measurement value before the update. Indicates the null value ratio measurement value of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the changed data, N represents the amount of data used to calculate the data quality evaluation measurement value before the update, where = N, k represents the amount of data used to calculate the data quality evaluation measure value of the change data, where = k; wherein when the changed data is deleted data to be deleted from the plurality of data tables, the updated null value ratio measurement value and the updated data quality characteristic data are calculated according to the following formula; in, Indicates the null value ratio measurement value after the update. Indicates the null value ratio measurement value before the update. Indicates the null value ratio measurement value of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the changed data, N represents the amount of data used to calculate the data quality evaluation measurement value before the update, where = N, K represents the amount of data used to calculate the data quality evaluation measure value of the change data, where =K. A data quality dynamic information processing method as described in claim 5, wherein the changed data includes new data to be added to the plurality of data tables or deleted data to be deleted from the plurality of data tables, the data quality assessment measurement value before updating includes a mean measurement value before updating and a matrix variance measurement value before updating, the data quality assessment measurement value of the changed data includes a mean measurement value and a matrix variance measurement value of the changed data, the data quality assessment measurement value after updating includes a mean measurement value after updating and a matrix variance measurement value after updating, and the data quality dynamic information processing method The method further comprises: calculating the updated mean measurement value, the updated matrix variation measurement value and the updated data quality characteristic data according to the pre-update mean measurement value, the pre-update matrix variation measurement value, the pre-update data quality characteristic data, the mean measurement value of the abnormal data, the matrix variation measurement value of the abnormal data and the data quality characteristic data of the abnormal data, wherein when the abnormal data is new data to be added to the plurality of data tables, the updated mean measurement value, the updated matrix variation measurement value and the updated data quality characteristic data are calculated according to the following formula; in, represents the updated mean measurement value, represents the mean measurement value before the update, represents the mean measurement value of the change data. represents the updated measure of the population variance, represents the measurement value of the parent population variance before the update, represents the measure of the parent population variance of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the mean measurement value before the update and the matrix variance measurement value before the update, where = N, k represents the amount of data used to calculate the mean measurement value and the parent variance measurement value of the change data, where = k; wherein when the changed data is deleted data to be deleted from the plurality of data tables, the updated mean measurement value, the updated parent population variance measurement value and the updated data quality characteristic data are calculated according to the following formula; in, represents the updated mean measurement value, represents the mean measurement value before the update, represents the mean measurement value of the change data. represents the updated measure of the population variance, represents the measurement value of the parent population variance before the update, represents the measure of the parent population variance of the change data. Indicates the updated data quality feature data. Indicates the data quality characteristics before the update. represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the mean measurement value before the update and the matrix variance measurement value before the update, where = N, k represents the amount of data used to calculate the mean measurement value and the parent variance measurement value of the change data, where =k. The data quality dynamic information processing method as described in claim 5, wherein the changed data includes new data to be added to the plurality of data tables or deleted data to be deleted from the plurality of data tables, the data quality evaluation measure value before updating includes a matrix correlation coefficient measure value before updating, the data quality evaluation measure value of the changed data includes a matrix correlation coefficient measure value of the changed data, the data quality evaluation measure value after updating includes a matrix correlation coefficient measure value after updating, and the data quality dynamic information processing method includes new data to be added to the plurality of data tables or deleted data to be deleted from the plurality of data tables, the data quality evaluation measure value before updating includes a matrix correlation coefficient measure value before updating, and the data quality dynamic information processing method includes new data to be added to the plurality of data tables or deleted data ... or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data from the plurality of data tables or deleted data The state information processing method further includes: calculating the updated matrix correlation coefficient measurement value and the updated data quality characteristic data according to the matrix correlation coefficient measurement value before the update, the data quality characteristic data before the update, the matrix correlation coefficient measurement value of the changed data, and the data quality characteristic data of the changed data, wherein when the changed data is new data to be added to the plurality of data tables, the updated matrix correlation coefficient measurement value and the updated data quality characteristic data are calculated according to the following formula; in, represents the updated matrix correlation coefficient measure value, represents the value of the matrix correlation coefficient before the update, Indicates the value of the parent correlation coefficient of the change data. represents the updated data quality feature data, x, y represent variables, represents the updated standard deviation of the variable, represents the standard deviation of the variable before updating, Indicates the standard deviation of the variable of the change data. represents the updated mean measurement value of the variable, represents the mean measurement value of the variable before updating, Represents the mean measurement value of the variable of the change data, where, ; Wherein when the changed data is deleted data to be deleted from the plurality of data tables, the updated matrix correlation coefficient and the updated data quality characteristic data are calculated according to the following formula; in, represents the updated matrix correlation coefficient measure value, represents the value of the matrix correlation coefficient before the update, Indicates the value of the parent correlation coefficient of the change data. represents the updated data quality feature data, x, y represent variables, represents the updated standard deviation of the variable, represents the standard deviation of the variable before updating, Indicates the standard deviation of the variable of the change data. represents the updated mean measurement value of the variable, represents the mean measurement value of the variable before updating, Represents the mean measurement value of the variable of the change data, where, . A data quality dynamic information processing method as described in claim 5, wherein the changed data includes new data to be added to the plurality of data tables or deleted data to be deleted from the plurality of data tables, the data quality evaluation measure value before the update includes a unique value ratio measure value before the update, the data quality evaluation measure value of the changed data includes a unique value ratio measure value of the changed data, the data quality evaluation measure value after the update includes a unique value ratio measure value after the update, and the data quality The dynamic information processing method further includes: calculating the updated unique value ratio measurement value and the updated data quality feature data according to the unique value ratio measurement value before the update, the data quality feature data before the update, the unique value ratio measurement value of the changed data, and the data quality feature data of the changed data, wherein when the changed data is new data to be added to the plurality of data tables, the updated unique value ratio measurement value and the updated data quality feature data are calculated according to the following formula; in, Represents the unique value ratio measurement value after the update. represents a union operation, key(．) represents a key value, and # represents a count. It represents the unique value ratio measurement value before the update. The unique value ratio measure of the change data. Indicates the updated data quality feature data. represents the data quality feature data before the update and the unique value ratio measurement value before the update, represents the data quality characteristic data and the unique value ratio measurement value of the changed data, and N represents the amount of data used to calculate the data quality assessment measurement value before the update; wherein when the changed data is deleted data to be deleted from the plurality of data tables, the unique value ratio measurement value after the update and the data quality characteristic data after the update are calculated according to the following formula; in, Represents the unique value ratio measurement value after the update. represents a union operation, key(．) represents a key value, and # represents a count. It represents the unique value ratio measurement value before the update. The unique value ratio measure of the change data. Indicates the updated data quality feature data. represents the data quality feature data before the update and the unique value ratio measurement value before the update, represents the data quality characteristic data and the unique value ratio measurement value of the changed data, and N represents the amount of data used to calculate the data quality assessment measurement value before the update.

Images