CN117762914A - Data quality detection method and system - Google Patents

Abstract

The invention discloses a data quality detection method and a data quality detection system, and relates to the technical field of data quality detection. Comprising the following steps: acquiring at least one piece of data to be detected in a data source, and disassembling the data to be detected into a state of data characters; constructing a screening rule and summarizing data characters of the similar character information; and constructing a data analysis model internally loaded with different systems. Compared with the traditional data quality detection method, the method can detect the data quality problem more finely by disassembling the data to be detected into the state of the data character, can extract information in the data character more accurately by constructing the screening rule and the data analysis model, and obtain a plurality of data character items, can obtain more accurate item difference scores by comparing historical data character indexes with the data character items, and can judge whether the data quality meets the standard more accurately by constructing the judging rule and setting the judging threshold.

Description

A data quality detection method and system

Technical field

The invention relates to the technical field of data quality detection, specifically a data quality detection method and system.

Background technique

In the process of analyzing and managing data, the emergence of abnormal data will directly lead to significant changes in the analysis results, causing the results to exceed expectations, thus making the managers' inference, control and prediction of the entire project process inaccurate, resulting in Wrong judgment will bring huge risks to the entire project, so data quality testing is required.

After searching, the Chinese invention patent with the authorization announcement number "CN112395280B" disclosed "a data quality detection method and its system". This application established a data model based on the original integrated historical data, and simultaneously performed prediction and identification on new sample data to obtain the data. Quality fluctuations, so that adjustment measures can be taken in advance and data quality management can be carried out in a targeted manner.

In addition, he applied for a Chinese invention patent with publication number "CN109933581A", which disclosed "a data quality detection method and system". This application uses distributed file storage to store data in equal parts on the disks and memories of multiple servers. The computing terminal calculates the data on the distributed memory at the same time to realize distributed memory parallel computing, and then summarizes the operation results on multiple computing terminals to complete the quality inspection.

However, when the above two patents are actually used, there are the following problems:

First, although the first patent can establish a data model based on historical data and predict and identify new sample data, it cannot detect and process abnormal data in real time. Adjustment measures can only be taken after data quality fluctuations occur, which cannot be done Deal with it promptly.

Second, although the second patent uses distributed memory parallel computing to improve the efficiency of data quality detection, it does not improve the accuracy and reliability of data quality detection, and the detection and detection of abnormal data are not improved. Processing capacity is limited.

Therefore, based on the above two shortcomings, the applicant proposed a new data quality detection method and system.

Contents of the invention

The purpose of the present invention is to provide a data quality detection method and system to solve the problems raised in the above background technology.

In order to achieve the above objects, the present invention provides the following technical solutions:

In the first aspect, a data quality detection method is designed, which includes:

The methods include:

Obtain at least one piece of data to be detected from the data source, and disassemble the data to be detected into the state of data characters;

Construct filtering rules and group data characters of similar character information together;

Build an internal data analysis model equipped with different systems, extract character information within data characters, obtain multiple data character items, compare historical data character indicators and data character items, and obtain the item difference score of data characters;

Construct a discrimination rule, set a discrimination threshold, and compare the item difference score of each data character with the preset threshold to determine whether a single data character "qualifies", counts the number of data character items that "qualify", and calculates the Quantity determines the overall quality of the data source;

Establish impact indicators, based on the application scenarios of the data, divide the weights of the project difference scores of multiple data characters, and finally judge the overall quality of the data to be detected based on the number of "standard" data character projects after the weight division.

As a further preference of this technical solution, comparison is made based on the item difference score and a preset threshold;

When the difference score is higher than the threshold, the project is judged to be high-quality data and recorded as "standard";

When the difference score is lower than the threshold, the item is judged to be low-quality data and recorded as "substandard".

As a further preference of this technical solution, the quality level of the data is further judged based on the number of data character items that "qualify";

When at least 80% of the data character items are "up to standard", the data quality is "80 points - 90 points";

When 50%-79% of the data character items "reach the standard", the data quality is "60 points-79 points";

When less than 50% of the data character items are "up to standard", the data quality is "40 points - 59 points".

As a further preference of this technical solution, the construction standards of the impact indicators include:

a. Accuracy: For projects that "qualify", when there are differences between their character information and historical data, they will be marked as "possible problems" and assigned a weight of 20%-35%;

b. Consistency: For data characters from the same data source or the same batch, when multiple items have discrepancies between character information and historical data, mark them as "requiring focus" and allocate 40%-55% Weights;

C. Stability: For a project that "qualifies", when there is no difference between its character information and historical data, it will be marked as "good stability" and assigned a weight of 10%-40%.

As a further optimization of this technical solution, the item difference score of the data character is compared with the preset threshold, and the difference is obtained based on the construction standard of the impact indicator, and is reconstructed. The constructed value is equal to the original value multiplied by the weight value plus the original value.

As a further preference of this technical solution, the method of constructing screening rules includes:

Check the data characters for invalid, ambiguous, wrong and repeated characters and mark these characters;

Replace or delete marked characters;

Construct a classification standard based on character information to group data characters of similar characters together.

As a further preference of this technical solution, the analysis method of the data analysis model includes:

Extract useful character features from filtered data characters;

Based on the extracted character features, associate and generate multiple data character items in the same field;

Store and manage the generated data character items into the database.

Secondly, in order to improve this technical solution, the applicant has proposed a data quality detection system based on the above-mentioned data quality detection method. The system includes:

Data acquisition module: used to obtain at least one piece of data to be detected in the data source;

Data disassembly module: used to disassemble the data to be detected into the state of data characters;

Data induction module: used to build filtering rules and summarize data characters of similar character information together;

Data extraction module: used to build data analysis models equipped with different systems internally, extract character information within data characters, and obtain multiple data character items;

Indicator comparison module: used to compare historical data character indicators and data character items to obtain the item difference score of data characters;

Discrimination module: used to build discrimination rules, set discrimination thresholds, compare the project difference score of each data character with the preset threshold to determine whether a single data character "qualifies", and counts the data characters that "qualify" Number of projects;

Quality judgment module: used to further judge the quality level of data based on the number of data character items that "qualify";

Impact indicator module: used to establish impact indicators, data-based application scenarios, and weight division of project difference scores of multiple data characters;

Final judgment module: used to judge the overall quality of the data to be detected based on the number of "standard" data character items divided by weights.

Compared with the prior art, the beneficial effects of the present invention are:

Compared with traditional data quality detection methods, this data quality detection method and system can more accurately detect data quality problems by disassembling the data to be detected into the state of data characters. Secondly, by building filtering rules and data analysis models, it can Extract the information within the data characters more accurately and obtain multiple data character items. Thirdly, by comparing the historical data character indicators and data character items, a more accurate item difference score can be obtained. Finally, by constructing a discriminant Rules and setting of discrimination thresholds can more accurately determine whether the data quality meets the standards.

In addition, the data quality detection method and system of the present invention can quickly and effectively detect data quality and improve detection efficiency by disassembling the data to be detected into data character states, and using filtering rules and data analysis models to perform data extraction and analysis. ;

In addition, by constructing modules such as screening rules, data analysis models, discrimination rules, and impact indicators, the present invention can realize automatic detection and discrimination of data quality, thereby reducing the cost and error rate of manual intervention.

In addition, it should be noted that the data quality detection method provided by the present invention realizes comprehensive detection and evaluation of data quality by constructing screening rules, data analysis models, discrimination rules and impact indicators, and the method has strong versatility. , can be applied to the detection of various data sources, such as databases, files, network data, etc. At the same time, this method also considers the application scenarios of the data, and makes the detection results more consistent by weighting the project difference scores of data characters. Actual business needs.

It should be added that the present invention also provides specific methods for constructing screening rules, data analysis models, discrimination rules and impact indicators, making the data quality detection process clearer and more orderly. By implementing the present invention, the data quality detection can be effectively improved. The accuracy and efficiency provide reliable data quality assurance for various enterprises and institutions.

Description of the drawings

Figure 1 is a content analysis diagram of the data quality detection method of the present invention;

Figure 2 is a module composition diagram of the data quality detection system of the present invention;

Figure 3 is a flow chart of steps for obtaining data source detection data according to the present invention;

Figure 4 is a flow chart of the operation steps of the data analysis model of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

The present invention provides a technical solution: a data quality detection method includes: obtaining at least one piece of data to be detected in the data source, disassembling the data to be detected into the state of data characters; constructing filtering rules, and summarizing the data characters of similar character information Together; build an internal data analysis model equipped with different systems, extract character information within data characters, obtain multiple data character items, compare historical data character indicators and data character items, and obtain the item difference score of data characters; build a discriminant Rules, set the judgment threshold, compare the item difference score of each data character with the preset threshold to determine whether a single data character "qualifies", counts the number of data character items that "qualify", and makes a judgment based on the number The overall quality of the data source; establishing impact indicators, based on data application scenarios, weighting the project difference scores of multiple data characters, and finally judging the data to be detected based on the number of "standard" data character projects after weight division overall quality.

As a specific implementation, obtaining the detection data of the data source includes the following steps:

Step 1. Determine the data source: First, you need to determine the data source to be detected, which can be a database, file, network data, etc.;

Step 2. Obtain data: Obtain the data to be detected from the determined data source;

Step 3. Data cleaning: Clean the data to be detected to remove invalid, fuzzy, wrong and repeated characters to ensure the accuracy of the data;

Step 4. Data disassembly: disassemble the cleaned data into the state of data characters, that is, convert the data into a recognizable character sequence according to specific rules or formats;

Step 5. Data storage: Store and manage the disassembled data characters into the database to facilitate subsequent data quality detection and analysis.

It should be noted that through the above steps, the detection data in the data source to be detected can be obtained, and subsequent data quality detection and analysis can be performed.

As a specific implementation, the present invention compares the project difference score with a preset threshold. When the difference score is higher than the threshold, the project is determined to be high-quality data and is recorded as "standard"; If the difference score is lower than the threshold, the item is judged to be low-quality data and recorded as "not up to standard". The quality level of the data is further judged based on the number of "up to standard" data character items.

When at least 80% of the data character items are "up to standard", the data quality is "80 points - 90 points";

When 50%-79% of the data character items "reach the standard", the data quality is "60 points-79 points";

When less than 50% of the data character items are "up to standard", the data quality is "40 points - 59 points".

It should be noted that the construction standards of the impact indicators in this invention include:

a. Accuracy: For projects that "qualify", when there are differences between their character information and historical data, they will be marked as "possible problems" and assigned a weight of 20%-35%;

b. Consistency: For data characters from the same data source or the same batch, when multiple items have discrepancies between character information and historical data, mark the data source or the same batch of data characters as "needs focus" Follow” and assign 40%-55% weight;

C. Stability: For a project that "qualifies", when there is no difference between its character information and historical data, it will be marked as "good stability" and assigned a weight of 10%-40%.

It should be emphasized that the project difference score of the data character in the present invention is compared with the preset threshold, and the difference obtained is reconstructed based on the construction standard of the impact index. The constructed value is equal to the original value multiplied by the weight value plus the original value. value.

In addition, it should be added that the detection logic of the data quality detection method of the present invention is: first, obtain the data to be detected in the data source and disassemble it into the state of data characters; then, according to the constructed filtering rules, similar characters are The data characters of the information are summarized together; then, a data analysis model equipped with different systems is built internally, and the character information within the data characters is extracted to obtain multiple data character items; then, historical data character indicators and data character items are compared, Obtain the item difference score of the data characters; then, according to the constructed discrimination rules and set discrimination thresholds, compare the item difference score of each data character with the preset threshold to determine whether a single data character "qualifies" , count the number of "standard" data character items; finally, judge the overall quality of the data source based on the number of "standard" data character items. In addition, impact indicators can also be established. Based on data application scenarios, the weights of the project difference scores of multiple data characters can be divided. Based on the number of "standard" data character projects after weight division, the overall data to be detected can finally be judged. quality.

As a specific implementation, the method of constructing filtering rules includes:

Check the invalid, ambiguous, wrong and repeated characters in the data characters and mark these characters; replace or delete the marked characters; build a classification standard, based on the character information, the data characters of similar characters can be grouped together, and what needs to be added Yes, the classification methods of similar characters are divided into the following types:

First, shape similarity classification: This classification method is mainly based on the shape characteristics of characters. For example, in Chinese characters, the shapes of characters such as "日" and "月" are very similar, and they can be classified into the same category.

Second, functional similarity classification: This classification method is mainly based on the function or use of characters. For example, in Chinese, pronouns such as "you", "I" and "he" have similar functions and can be classified into the same category; in English, the functions of conjunctions such as "and", "or" and "but" Also similar, they can be classified into the same category.

Third, semantic similarity classification: This classification method is mainly based on the semantics of characters. For example, words such as "apple", "banana" and "orange" all refer to fruits and can be classified into the same category; in English, articles such as "the", "a" and "an" have similar semantics and can be Put them in the same category.

Fourth, frequency and occurrence probability classification: This classification method is mainly based on the frequency and occurrence probability of characters in the text. For example, in Chinese texts, auxiliary words such as "的", "乐", and "Ah" appear very frequently, and they can be classified into high-frequency character categories; in English texts, "the", "and", " The probability of occurrence of common words such as "or" is also very high, and they can be classified into the high-frequency character category.

Methods for constructing discrimination rules include:

a. Set the discrimination threshold: Determine an appropriate discrimination threshold based on the item difference score of historical data characters. The setting of the discrimination threshold can be based on the overall distribution of the data, such as the mean, median, etc. It can also be set according to the business characteristics of the data, such as the fluctuation range of the data, abnormal values, etc.

b. Compare the item difference score of the data character with the discrimination threshold: For each data character, compare its item difference score with the discrimination threshold. If the item difference score is greater than the discrimination threshold, the data character is judged to be high-quality data; conversely, if the item difference score is less than the discrimination threshold, the data character is judged to be low-quality data.

c. Count the number of data character items that "reach the standard": among all data characters, count the number of data characters whose item difference score is greater than the discrimination threshold.

d. Judge the overall quality of the data source: Judge the overall quality of the data source based on the number of "standard" data character items. The higher the proportion of data character items that "qualify", the higher the overall quality of the data source.

It should be noted that the data quality detection method provided by the present invention achieves comprehensive detection and evaluation of data quality by constructing screening rules, data analysis models, discrimination rules and impact indicators. The method has strong versatility. It can be applied to the detection of various data sources, such as databases, files, network data, etc. At the same time, this method also considers the application scenarios of the data, and makes the detection results more consistent with reality by weighting the project difference scores of data characters. Business needs.

In addition, the present invention also provides specific methods for constructing screening rules, data analysis models, discrimination rules and impact indicators, making the data quality detection process clearer and more orderly. By implementing the present invention, the accuracy of data quality detection can be effectively improved. and efficiency, providing reliable data quality assurance for various enterprises and institutions.

As a specific implementation, the steps of running the data analysis model include:

Step 1. Data preprocessing: Before in-depth analysis of the filtered data characters, it needs to be preprocessed. The preprocessing process mainly includes noise removal and standardization. Noise is removed to ensure the accuracy of the analysis and to eliminate errors and outliers in the data. Standardization is the process of converting data to the same scale to facilitate subsequent analysis. Both steps are to improve the quality of data and lay a solid foundation for subsequent analysis.

Step 2. Application of machine learning algorithm: The preprocessed data will be used to train the machine learning algorithm. This process mainly includes two parts: feature extraction and model construction. Feature extraction is to convert data characters into feature vectors that can be understood by computers, including the syntax, semantics and other information of the text. Model construction uses training data to learn the relationship between feature vectors and target variables through machine learning algorithms (such as decision trees, support vector machines, neural networks, etc.) to build prediction and classification models.

Step 3. Data storage and management: The built model can predict and classify new data characters. In order to facilitate subsequent query and use, we need to store and manage the generated data character items. Storage methods can include relational databases, non-relational databases, or distributed storage systems. Management mainly includes data addition, deletion, modification and other operations, as well as model update and optimization.

Step 4. Model update and optimization: The data analysis model is scalable and maintainable. As business develops and data changes, models need to be constantly updated and optimized. This includes the introduction of new data, adjustment of algorithm parameters, improvement of model structure, etc. Additionally, we need to evaluate the model to ensure its accuracy and effectiveness in real-world applications.

Step 5. Result output and application: Data characters predicted and classified by the model can be used in various business scenarios. For example, in the financial field, it can be used for credit risk assessment, investment decisions, etc.; in the medical field, it can be used for disease prediction, medication recommendations, etc.; in the education field, it can be used for student performance prediction, teaching effect evaluation, etc. These applications all help improve business efficiency and decision-making quality.

As a specific implementation, the application also proposes a detection system based on the above-mentioned data quality detection method. The system includes the following modules:

Data acquisition module: used to obtain data to be detected from various data sources. These data sources can be obtained from databases, files, network connections, or other data storage and transmission methods.

Data disassembly module: Preprocess the data to be detected and disassemble it into the state of data characters. This process generally includes data cleaning, format conversion and normalization operations.

Data induction module: summarize and classify the disassembled data characters, and build filtering rules to summarize data characters of similar character information together.

Data extraction module: Based on the induction results, build a data analysis model equipped with different systems internally, extract character information within data characters, and obtain multiple data character items.

Indicator comparison module: Compare historical data character indicators with current data character items, and calculate the item difference score. This score can reflect the degree of variation and abnormality in the data.

Discrimination module: Based on the discrimination rules and discrimination thresholds, the project difference score of each data character is discriminated to determine whether a single data character "qualifies". At the same time, count the number of data character items that "qualify".

Quality judgment module: Judge the quality level of the data based on the number of data character items that "qualify". This grade reflects the accuracy and reliability of the data.

Impact indicator module: Based on data application scenarios, impact indicators are established to weight the project difference scores of multiple data characters. This weight division can reflect the impact of different project difference scores on the overall data quality.

Final judgment module: Based on the number of "standard" data character items after weight division, the overall quality of the data to be detected is finally judged. This discrimination result can provide a reference for data users to help them understand the quality of the data.

As a specific implementation method, the following modules can also be added to the data quality detection system, such as: anomaly detection and early warning module: during the data quality detection process, if abnormal data characters are found, this module can mark them out in time , and sends early warning information to data managers, which helps ensure that data managers can take timely measures to reduce the impact of low-quality data on business systems. Data visualization module: In order to display data quality detection results more intuitively, this module provides data visualization functions. Through charts, reports, etc., data users can clearly understand the status of data quality, thereby optimizing data quality in a targeted manner. User interaction module: This module is designed to improve the humanity of the system and allow data users to operate and use the system more conveniently. It should be noted that this module allows users to set filtering rules, view detection results, and adjust models through the interface. Parameters, etc., to meet the needs of different business scenarios. System optimization and upgrade module: In order to continuously improve system performance and adapt to the changing data environment, this module is responsible for optimizing and upgrading the system. It should be noted that this includes improvements to data analysis models, algorithm strategies, etc., to improve Accuracy and efficiency of data quality testing.

Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art will understand that various changes, modifications, and substitutions can be made to these embodiments without departing from the principles and spirit of the invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (8)

1. A data quality detection method, characterized in that the method includes: Obtain at least one piece of data to be detected from the data source, and disassemble the data to be detected into the state of data characters; Construct filtering rules and group data characters of similar character information together; Build an internal data analysis model equipped with different systems, extract character information within data characters, obtain multiple data character items, compare historical data character indicators and data character items, and obtain the item difference score of data characters; Construct a discrimination rule, set a discrimination threshold, and compare the item difference score of each data character with the preset threshold to determine whether a single data character "qualifies", counts the number of data character items that "qualify", and calculates the Quantity determines the overall quality of the data source; Establish impact indicators, based on the application scenarios of the data, divide the weights of the project difference scores of multiple data characters, and finally judge the overall quality of the data to be detected based on the number of "standard" data character projects after the weight division. 2. A data quality detection method according to claim 1, characterized in that: comparison is made according to the item difference score and a preset threshold; When the difference score is higher than the threshold, the project is judged to be high-quality data and recorded as "standard"; When the difference score is lower than the threshold, the item is judged to be low-quality data and recorded as "substandard". 3. A data quality detection method according to claim 2, characterized in that: further judging the quality level of the data based on the number of "standard" data character items; When at least 80% of the data character items are "up to standard", the data quality is "80 points - 90 points"; When 50%-79% of the data character items "reach the standard", the data quality is "60 points-79 points"; When less than 50% of the data character items are "up to standard", the data quality is "40 points - 59 points". 4. A data quality detection method according to claim 3, characterized in that: the construction standards of the impact indicators include: a. Accuracy: For projects that "qualify", when there are differences between their character information and historical data, they will be marked as "possible problems" and assigned a weight of 20%-35%; b. Consistency: For data characters from the same data source or the same batch, when multiple items have discrepancies between character information and historical data, mark them as "requiring focus" and allocate 40%-55% Weights; C. Stability: For a project that "qualifies", when there is no difference between its character information and historical data, it will be marked as "good stability" and assigned a weight of 10%-40%. 5. A data quality detection method according to claim 4, characterized in that: the item difference score of the data character is compared with the preset threshold, and the difference obtained is reconstructed based on the construction standard of the impact index, and the constructed The value is equal to the original value multiplied by the weight value plus the original value. 6. A data quality detection method according to claim 1, characterized in that: the method of constructing screening rules includes: Check the data characters for invalid, ambiguous, wrong and repeated characters and mark these characters; Replace or delete marked characters; Construct a classification standard based on character information to group data characters of similar characters together. 7. A data quality detection method according to claim 1, characterized in that: the analysis method of the data analysis model includes: Extract useful character features from filtered data characters; Based on the extracted character features, associate and generate multiple data character items in the same field; Store and manage the generated data character items into the database. 8. A data quality detection system, characterized by including: Data acquisition module: used to obtain at least one piece of data to be detected in the data source; Data disassembly module: used to disassemble the data to be detected into the state of data characters; Data induction module: used to build filtering rules and summarize data characters of similar character information together; Data extraction module: used to build data analysis models equipped with different systems internally, extract character information within data characters, and obtain multiple data character items; Indicator comparison module: used to compare historical data character indicators and data character items to obtain the item difference score of data characters; Discrimination module: used to build discrimination rules, set discrimination thresholds, compare the project difference score of each data character with the preset threshold to determine whether a single data character "qualifies", and counts the data characters that "qualify" Number of projects; Quality judgment module: used to further judge the quality level of data based on the number of data character items that "qualify"; Impact indicator module: used to establish impact indicators, data-based application scenarios, and weight division of project difference scores of multiple data characters; Final judgment module: used to judge the overall quality of the data to be detected based on the number of "standard" data character items divided by weights.