CN113591485B - Intelligent data quality auditing system and method based on data science - Google Patents

Abstract

The present invention discloses an intelligent data quality audit system and method based on data science, the method includes: data collection: metadata collection of detection objects and log data collection and analysis; data feature extraction: identifying and eliminating invalid tables and invalid fields, and automatically revising field types according to data content through revision algorithms, and extracting features according to field types; anomaly detection: presetting a data anomaly detection method library, matching with data features to select corresponding anomaly detection methods and detect; task scheduling and orchestration: setting an orchestration server and a node server, and the orchestration server splits the above tasks into several sub-steps according to task requests and distributes them to different node servers for processing. The present invention reduces the threshold of data asset management and data quality governance, realizes the versatility, scale, automation and intelligence of data quality audits, and improves the efficiency and work quality of data quality audits as a whole.

Description

An intelligent data quality audit system and method based on data science

Technical Field

The present invention relates to the field of data processing, and in particular to an intelligent data quality audit system and method based on data science.

Background technique

With the development of the digital economy, all walks of life are no longer blindly pursuing the scale of data volume. In the process of data application, the requirements for data quality are getting higher and higher. Faced with massive data resources, how to discover and locate data quality problems faster, more accurately and more intelligently, and carry out corresponding governance work, is the focus and core of current enterprise-level data asset management.

In the prior art, for example, the invention with publication number CN105554152A discloses a method and device for data feature extraction. In more detailed technical content, for example, the invention with publication number CN108256074A discloses a method for verification processing, including obtaining a model of a data warehouse to be verified, each model including multiple field information, the field information including field definition and field type; verifying the field information according to a pre-stored data dictionary, the data dictionary including multiple standard terms, each standard term including a standard definition and a standard type; if the field definition matches the standard definition and the field type does not match the standard type, then modifying the field type to be consistent with the standard type. The method verifies the model of the data warehouse according to the standard terminology, and when the field definition matches the standard definition and the field type does not match the standard type, the field type is modified to be consistent with the standard type in a targeted manner, thereby obtaining a standard and consistent model.

The existing technologies have their own advantages and disadvantages in solving related problems. Under the traditional data quality governance model, the selection of problem detection objects requires business experts to specify specific data tables and fields based on business specifications and experience knowledge. It is necessary to indicate what characteristics each field has and what rules apply. Such methods and results require extremely high experience and professional skills of business experts. The scope of detection objects for data quality problems is relatively limited and highly dependent on business experts. For large-scale massive data, business experts are required to specify the corresponding detection objects and scopes one by one. In addition, the universality of data features is weak and maintenance is time-consuming and labor-intensive. It is impossible to achieve large-scale, automated data quality detection objects and extraction of corresponding data features. The efficiency of data quality audits is low and seriously affected by manual experience.

Summary of the invention

In view of the problems of low audit efficiency and low detection accuracy in the existing technology when facing large-scale data, the present invention provides an intelligent data quality audit system and method based on data science. Through data feature extraction, anomaly detection and task scheduling and orchestration management, the threshold of data asset management and data quality governance is lowered, and the universality, scale, automation and intelligence of data quality audit are achieved, thereby improving the efficiency and work quality of data quality audit as a whole.

The following is the technical solution of the present invention.

An intelligent data quality audit system based on data science, including:

Data collection module: collects metadata of detection objects and collects and analyzes log data; data feature extraction module: extracts features according to field types; anomaly detection module: matches data features to select corresponding anomaly detection methods and detects; task scheduling and orchestration module: includes orchestration servers and node servers. The orchestration server splits the above tasks into several sub-steps according to task requests and distributes them to different node servers for processing.

The present invention uses feature extraction and anomaly detection as the basis for implementing data auditing, and uses a task scheduling and orchestration module to reasonably allocate server resources, ultimately achieving large-scale data auditing.

In addition, the present invention also provides an intelligent data quality audit method based on data science, which is used in the above system and includes the following steps:

S1: Data collection: collect metadata of detection objects and collect and analyze log data; S2: Data feature extraction: identify and eliminate invalid tables and invalid fields, and automatically revise field types according to data content through revision algorithms, and extract features according to field types; S3: Anomaly detection: preset data anomaly detection method library, match with data features to select corresponding anomaly detection methods and detect; S4: Task scheduling and orchestration: set up orchestration servers and node servers. The orchestration server splits the above tasks into several sub-steps according to task requests and distributes them to different node servers for processing.

Preferably, the process of data feature extraction includes: preliminary identification of field types of data, and elimination of invalid tables and invalid fields; determining the Chinese description and field type of the data, sampling unmatched data, calculating the proportion of each field type in the sample, and revising the field type according to the proportion result; extracting features according to the field type; the field type includes at least one of numeric, text and date types.

Preferably, the process of preliminary identification includes: preliminary identification of the data to be identified based on the existing field type database, or introduction of a recognition model trained by a neural network for preliminary identification, to obtain preliminary identification results of the field type; the process of eliminating invalid tables and invalid fields includes: defining invalid tables and invalid fields, judging empty tables, zombie tables, log tables, backup tables, temporary tables, single-field tables, and low-heat tables as invalid tables through metadata information and data content of the tables; uniformly judging empty fields and single-value fields as invalid fields; identifying and eliminating invalid tables and fields. Different field types have their own characteristics. In the prior art, databases and training models are usually used for comparison and identification, which helps to reduce implementation costs and has a certain basic accuracy guarantee; in addition, invalid tables and invalid fields cover various common invalid data, and elimination can reduce the processing pressure of subsequent data extraction and analysis.

Preferably, the process of revising the field type includes: using the NLP natural language processing module to segment and semantically identify the Chinese description of the data, and after parsing, using the type decision tree to identify the path of similar words or similar characters. If the semantics of the Chinese description does not match the field type, it is marked as a suspected revised field type; then the data content with the same or similar semantics in the Chinese description is sampled multiple times, and the proportion of different field types in the sampled data is counted, and the type with a proportion exceeding the threshold is used as the recommended revised field type, and finally revised to the field type to which the real data is stored. Natural language processing technology can segment and semantically identify the Chinese description, and the decision tree can identify paths with similar meanings to help determine whether it belongs to a suspected revised field type. Finally, the result is determined by setting a threshold and using the proportion as the judgment standard. The revision process is a supplement to the preliminary identification and further improves the identification accuracy.

Preferably, the process of extracting features according to field type includes: for numeric fields, features and feature values are extracted using mean, maximum, minimum, median, variance, quartile, interquartile range, numerical clustering and length clustering; for text fields, statistical attribute features are analyzed from length clustering and structural distribution, and content features are extracted through word segmentation and semantic recognition of data content; for date fields, structural analysis is performed, and features are extracted from date format and length.

As a preferred method, after the revision field type is completed, a verification step is also included: converting the date data into text data and copying it into a verification group and an interference group, wherein the verification group inserts the year, month, and day description according to the original date format, and the interference group adds the counting unit description according to the number of digits of the original date data, inserting the verification group and the interference group into the text data adjacent to itself, and semantically identifying the spliced text data through the NLP natural language processing module, recording the recognition speed of each pair of interference group and verification group, if the recognition speed of the verification group is faster than that of the interference group and exceeds the amplitude threshold, the verification is passed, otherwise the corresponding original date data is listed as a suspected error type. Since both date data and numerical data are often related to the adjacent text data, when the original recognition is correct, the text spliced by the verification group is easier to recognize, so the recognition speed is faster, and if the original recognition is wrong, the text spliced by the verification group is wrong, so compared with the interference group, there is no recognition speed advantage, or even slower, so it will be listed as a suspected error type.

Preferably, the anomaly detection process includes:

Construct a data anomaly detection method library, set the corresponding detection method according to each data feature, and summarize it to form a data anomaly detection method library. The data anomaly detection method library is stored in a dictionary type, and the tuple consisting of the data feature name and its feature parameters is used as the dictionary key, and the anomaly detection method corresponding to the data feature is used as the dictionary value; match the data features with anomaly detection methods, and detect according to the anomaly detection method in the matching results; traverse large-scale data features to match and detect each data feature. The setting of the method library is to design corresponding anomaly detection methods for different data features from the perspectives of statistics, common sense, natural laws, and professional general knowledge. For example, the data value feature design reports anomalies when the field value has an extreme value, and the date feature reports anomalies for field content that does not conform to the date format. The setting of the method library is specifically determined according to actual usage needs, and targeted detection is performed after matching. The Python dictionary type is a key-value pair. We use the Python dictionary type to store data features and their anomaly detection methods. The dictionary key stores a tuple consisting of the data feature name and its feature parameters. The dictionary value stores the anomaly detection method corresponding to the data feature. The threshold of each anomaly detection method is given by the feature parameter. By storing it in a dictionary, we can clearly divide the keys and values, which is convenient for subsequent matching.

Preferably, the matching includes the following process: embedding the name of the data feature to be processed and the key in the anomaly detection method library into the word vector obtained by NLP, calculating the cosine similarity between the word vectors, the keys with similarity greater than the threshold are the potential keys corresponding to the data feature, and the anomaly detection methods corresponding to these keys are the matching results; the calculation formula of the cosine similarity is as follows:

Where u and v represent two word vectors. Word vectors contain multi-dimensional values, and with the help of cosine similarity, they can be judged and compared more accurately.

As a preferred embodiment, the large-scale data feature traversal process includes: scaling the value of each dimension in the word vector to be matched to a range of 0 to 255, and representing each word vector with an array of n pixels arranged in sequence, wherein n is the dimension of the word vector, and the value of each dimension of the word vector is the gray value of each pixel, so as to copy the image represented by the pixel array to a white background image of m pixels to obtain a replica, wherein m is x^2 times of n, and x is a natural number greater than or equal to 2, reduce the pixels of the replica to n, read the gray value of each pixel, form a new special word vector, and use the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data. When facing large-scale data, if it is still completely consistent with the way of processing single data, the amount of calculation will be very large and the overall efficiency is low. Therefore, the above method is used to fuzzify the vector. Although there will be deviations between the fuzzy word vector and the original word vector, the originally similar word vectors still retain a suitable similarity, so the calculation results of the similarity are relatively small. In this way, the computational pressure under massive data can be coped with.

There is also an alternative solution, that is, the large-scale data feature traversal process includes: scaling the value of each dimension in the word vector to be matched to the range of 0 to 255, and dividing 0 to 225 into several orders, modifying the value of each dimension to the middle number in the order corresponding to the value, generating a new special word vector, and using the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data. This solution is still based on fuzzy word vectors to reduce the computational intensity under large-scale data.

Preferably, the task scheduling process includes:

The orchestration server splits the tasks into different nodes and deploys them on multiple servers, allocating server computing resources to reduce the computing pressure of each server; the orchestration server uniformly collects and distributes cluster requests, adopts the producer-consumer model, distributes tasks to the node server cluster in an orderly manner, allocates task execution strategies according to the cluster configuration, and provides real-time feedback on task execution; it schedules tasks according to the node server cluster. When a node in the cluster fails, the tasks on it are transferred to other normal nodes to ensure that the task operation is not affected by the downtime of some node servers. Reasonable planning of server computing power can provide efficiency bonus for the entire audit method, further expanding the efficiency advantages of feature extraction and anomaly detection in this solution.

The substantial effects of the present invention include: providing solutions and system function services for data science methods and artificial intelligence technologies in data quality verification, lowering the threshold for data asset management and data quality governance, achieving the universality, scale, automation and intelligence of data quality audits, and improving the efficiency and work quality of data quality audits as a whole.

Detailed ways

The technical scheme of the present application will be described below in conjunction with the embodiments. In addition, in order to better illustrate the present invention, numerous specific details are provided below. It should be understood by those skilled in the art that the present invention can be implemented without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the gist of the present invention.

Example:

An intelligent data quality audit system and method based on data science, the system comprising:

Data collection module: collects metadata of detection objects and collects and analyzes log data; data feature extraction module: extracts features according to field types; anomaly detection module: matches data features to select corresponding anomaly detection methods and detects; task scheduling and orchestration module: includes orchestration servers and node servers. The orchestration server splits the above tasks into several sub-steps according to task requests and distributes them to different node servers for processing.

This embodiment uses feature extraction and anomaly detection as the basis for implementing data auditing, and uses the task scheduling and orchestration module to reasonably allocate server resources, ultimately achieving large-scale data auditing.

The corresponding audit method in this embodiment includes the following steps:

S1: Data collection:

Collect metadata of detection objects and analyze log data.

S2: Data feature extraction:

Perform preliminary identification of the field type of the data and eliminate invalid tables and invalid fields; determine the Chinese description and field type of the data, sample the unmatched data, calculate the proportion of each field type in the sample, and revise the field type based on the proportion results; extract features based on the field type; the field type includes at least one of numeric, text, and date types.

The preliminary identification process includes: preliminary identification of the data to be identified based on the existing field type database, or introduction of a recognition model trained by a neural network for preliminary identification, and obtaining preliminary identification results of the field type; the process of eliminating invalid tables and invalid fields includes: defining invalid tables and invalid fields, judging empty tables, zombie tables, log tables, backup tables, temporary tables, single-field tables, and low-heat tables as invalid tables through metadata information and data content of the tables; uniformly judging empty fields and single-value fields as invalid fields; identifying and eliminating invalid tables and fields. Different field types have their own characteristics. In the prior art, databases and training models are usually used for comparison and identification, which helps to reduce implementation costs and has a certain basic accuracy guarantee; in addition, invalid tables and invalid fields cover various common invalid data, and elimination can reduce the processing pressure of subsequent data extraction and analysis.

The process of revising the field type includes: using the NLP natural language processing module to segment and semantically identify the Chinese description of the data, and after parsing, using the type decision tree to identify the path of similar words or similar characters. If the semantics of the Chinese description does not match the field type, it is marked as a suspected revised field type; then the data content with the same or similar semantics in the Chinese description is sampled multiple times, and the proportion of different field types in the sampled data is counted, and the type with a proportion exceeding the threshold is used as the recommended revised field type, and finally revised to the field type to which the real data belongs. Natural language processing technology can segment and semantically identify the Chinese description, while the decision tree can identify paths with similar meanings to help determine whether it belongs to the suspected revised field type. Finally, the result is determined by setting a threshold and using the proportion as the judgment standard. The revision process is a supplement to the preliminary identification and further improves the identification accuracy.

The process of extracting features according to field types includes: for numeric fields, features and feature values are extracted using mean, maximum, minimum, median, variance, quartile, interquartile range, numeric clustering, and length clustering; for text fields, length clustering and structural distribution statistical attribute features are used, and content features are extracted through word segmentation and semantic recognition of data content; for date fields, structural analysis is performed, and features are extracted from date format and length.

More specifically, the data features and feature extraction methods applicable to the field type can be found in the data feature library, and all applicable data feature extraction methods for the field type can be traversed based on the dependency and mutually exclusive relationship network of the corresponding data features. For example, after determining that a data field is numeric, the feature extraction algorithm will load methods for extracting attribute features such as length, integer, positive number, negative number, decimal, and other attribute features, as well as methods for extracting business features such as mobile phone number and zip code. By continuously identifying and extracting the data content, features such as length concentration, integer, and mobile phone number can be obtained. At the same time, the two opposing and mutually exclusive features of "positive-negative" can be distinguished, thereby obtaining multi-angle features and feature values for the field.

In addition, after the revision of the field type is completed, a verification step is also included: converting the date data into text data and copying it into a verification group and an interference group. The verification group inserts the year, month, and day description according to the original date format, and the interference group adds the counting unit description according to the number of digits of the original date data. The verification group and the interference group are inserted into the adjacent text data, and the spliced text data is semantically recognized through the NLP natural language processing module, and the recognition speed of each pair of interference groups and verification groups is recorded. If the recognition speed of the verification group is faster than that of the interference group and exceeds the amplitude threshold, the verification is passed, otherwise the corresponding original date data is listed as a suspected error type. Since both date data and numerical data are often related to the adjacent text data, when the original recognition is correct, the text spliced by the verification group is easier to recognize, so the recognition speed is faster, and if the original recognition is wrong, the text spliced by the verification group is wrong, so compared with the interference group, there is no recognition speed advantage, or even slower, so it will be listed as a suspected error type.

S3: Anomaly Detection:

Construct a data anomaly detection method library, set the corresponding detection method according to each data feature, and summarize it to form a data anomaly detection method library. The data anomaly detection method library is stored in a dictionary type, and the tuple consisting of the data feature name and its feature parameters is used as the dictionary key, and the anomaly detection method corresponding to the data feature is used as the dictionary value; match the data features with anomaly detection methods, and detect according to the anomaly detection method in the matching results; traverse large-scale data features to match and detect each data feature. The setting of the method library is to design corresponding anomaly detection methods for different data features from the perspectives of statistics, common sense, natural laws, and professional general knowledge. For example, the data value feature design reports anomalies when the field value has an extreme value, and the date feature reports anomalies for field content that does not conform to the date format. The setting of the method library is determined according to the actual use needs, and targeted detection is performed after matching. The Python dictionary type is a key-value pair. We use the Python dictionary type to store data features and their anomaly detection methods. The dictionary key stores a tuple consisting of the data feature name and its feature parameters. The dictionary value stores the anomaly detection method corresponding to the data feature. The threshold of each anomaly detection method is given by the feature parameter. By storing it in a dictionary, we can clearly divide the keys and values, which is convenient for subsequent matching.

The matching process includes the following steps: embed the name of the data feature to be processed and the key in the anomaly detection method library into the word vector obtained by NLP, calculate the cosine similarity between the word vectors, and the keys with similarity greater than the threshold are the potential keys corresponding to the data feature. The anomaly detection methods corresponding to these keys are the matching results. The calculation formula of cosine similarity is as follows:

Where u and v represent two word vectors. Word vectors contain multi-dimensional values, and with the help of cosine similarity, they can be judged and compared more accurately.

The large-scale data feature traversal process includes: scaling the value of each dimension in the word vector to be matched to the range of 0 to 255, representing each word vector with an array of n pixels arranged in sequence, where n is the dimension of the word vector, and the value of each dimension of the word vector is the gray value of each pixel, so as to copy the image represented by the pixel array to a white background image of m pixels to obtain a replica, where m is x^2 times of n, and x is a natural number greater than or equal to 2, reduce the pixels of the replica to n, read the gray value of each pixel, form a new special word vector, and use the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data. When facing large-scale data, if it is still exactly the same as the way of processing single data, the amount of calculation will be very large and the overall efficiency is low. Therefore, the above method is used to fuzzy the vector. Although there will be deviations between the fuzzy word vector and the original word vector, the originally similar word vectors still retain a suitable similarity, so the calculation results of the similarity are relatively small. This method can cope with the computational pressure under massive data.

There is also an alternative solution, that is, the large-scale data feature traversal process includes: scaling the value of each dimension in the word vector to be matched to the range of 0 to 255, and dividing 0 to 225 into several orders, modifying the value of each dimension to the middle number in the order corresponding to the value, generating a new special word vector, and using the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data. This solution is still based on fuzzy word vectors to reduce the computational intensity under large-scale data.

S4: Task scheduling and orchestration:

Set up an orchestration server and a node server. The orchestration server splits the above task into several sub-steps according to the task request and distributes them to different node servers for processing.

The orchestration server splits the tasks into different nodes and deploys them on multiple servers, allocating server computing resources to reduce the computing pressure of each server; the orchestration server uniformly collects and distributes cluster requests, adopts the producer-consumer model, distributes tasks to the node server cluster in an orderly manner, allocates task execution strategies according to the cluster configuration, and provides real-time feedback on task execution; it schedules tasks according to the node server cluster. When a node in the cluster fails, the tasks on it are transferred to other normal nodes to ensure that the task operation is not affected by the downtime of some node servers. Reasonable planning of server computing power can provide efficiency bonus for the entire audit method, further expanding the efficiency advantages of feature extraction and anomaly detection in this solution.

This embodiment uses feature extraction and anomaly detection as the basis for data auditing, and uses task scheduling and orchestration to reasonably allocate server resources, ultimately achieving large-scale data auditing. The substantial effects of this embodiment include: providing solutions and system function services for data science methods and artificial intelligence technologies in data quality verification, lowering the threshold for data asset management and data quality governance, achieving the universality, scale, automation and intelligence of data quality auditing, and overall improving the efficiency and work quality of data quality auditing.

Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

In the embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. For example, it can be implemented in the form of hardware or in the form of a software functional unit. If it is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (8)

1. An intelligent data quality auditing system based on data science, which is characterized by comprising:

and a data acquisition module: collecting and analyzing metadata of the detection object and collecting and analyzing log data;

and the data characteristic extraction module is used for: extracting features according to the field types;

an abnormality detection module: matching with the data characteristics to select a corresponding abnormality detection method and detect;

a task scheduling module: the system comprises an arrangement server and a node server, wherein the arrangement server divides the task into a plurality of sub-steps according to a task request and distributes the sub-steps to different node servers for processing;

The data feature extraction module performs:

performing preliminary identification of field types on the data, and eliminating an invalid table and an invalid field;

Judging Chinese description and field types of the data, sampling the unmatched data, calculating the duty ratio of each field type in the sample, and revising the field types according to the duty ratio result;

Extracting features according to the field types;

the field type comprises at least one of a numerical type, a text type and a date type;

the process of revising the field type includes: the Chinese description of the data is subjected to word segmentation and semantic recognition by utilizing an NLP natural language processing module, the path recognition of the approximate word or the approximate word is carried out through a type decision tree after analysis, the semantics of the Chinese description are not matched with the field type, and the Chinese description is marked as a suspected revised field type; and then sampling the data content with the same or similar semantic meaning for multiple times, counting the duty ratio conditions of different field types in the sampled data, taking the type with the duty ratio exceeding a threshold value as a recommended revised field type, and finally revising the recommended revised field type into the field type to which the real storage data belongs.

2. An intelligent data quality auditing method based on data science, which is used for an auditing system as claimed in claim 1, and is characterized by comprising the following steps:

S1: and (3) data acquisition: collecting and analyzing metadata of the detection object and collecting and analyzing log data;

S2: and (3) data characteristic extraction: identifying and removing an invalid table and an invalid field, automatically revising the field type according to the data content through a revising algorithm, and extracting features according to the field type;

S3: abnormality detection: presetting a data anomaly detection method library, and matching with data characteristics to select a corresponding anomaly detection method and detect;

s4: task scheduling: setting an arranging server and a node server, dividing the task into a plurality of sub-steps according to the task request by the arranging server, and then distributing the sub-steps to different node servers for processing;

The process of data feature extraction comprises the following steps:

performing preliminary identification of field types on the data, and eliminating an invalid table and an invalid field;

Judging Chinese description and field types of the data, sampling the unmatched data, calculating the duty ratio of each field type in the sample, and revising the field types according to the duty ratio result;

Extracting features according to the field types;

the field type comprises at least one of a numerical type, a text type and a date type;

the process of revising the field type includes: the Chinese description of the data is subjected to word segmentation and semantic recognition by utilizing an NLP natural language processing module, the path recognition of the approximate word or the approximate word is carried out through a type decision tree after analysis, the semantics of the Chinese description are not matched with the field type, and the Chinese description is marked as a suspected revised field type; and then sampling the data content with the same or similar semantic meaning for multiple times, counting the duty ratio conditions of different field types in the sampled data, taking the type with the duty ratio exceeding a threshold value as a recommended revised field type, and finally revising the recommended revised field type into the field type to which the real storage data belongs.

3. The intelligent data quality auditing method based on data science according to claim 2, wherein the preliminary identification process includes: performing preliminary identification on data to be identified according to the existing field type database, or introducing an identification model trained by a neural network to perform preliminary identification, so as to obtain a preliminary identification result of the field type; the process of eliminating the invalid table and the invalid field comprises the following steps: defining an invalid table and an invalid field, and uniformly judging an empty table, a zombie table, a log table, a backup table, a temporary table, a single-field table and a low-heat table as invalid tables through metadata information and data content judgment of the tables; uniformly judging the null field and the single value field as invalid fields; and identifying and rejecting invalid tables and fields.

4. The intelligent data quality auditing method based on data science according to claim 2, wherein the process of extracting features according to field types includes: extracting features and feature values of the numerical value type fields by means of average value, maximum value, minimum value, median, variance, quartile range, numerical value cluster and length cluster; for text type fields, statistical attribute features are distributed from length clusters and structures, and extraction on content features is performed through word segmentation and semantic recognition of data content; and carrying out structural analysis on the date type field, and carrying out feature extraction on the date format and the length.

5. The intelligent data quality auditing method based on data science according to claim 2, wherein the anomaly detection process includes:

Constructing a data anomaly detection method library, setting corresponding detection methods according to each data feature, and summarizing to form the data anomaly detection method library, wherein the data anomaly detection method library is stored in a dictionary type, a tuple consisting of a data feature name and a feature parameter thereof is used as a key of the dictionary, and an anomaly detection method corresponding to the data feature is used as a value of the dictionary;

Performing anomaly detection method matching on the data characteristics, and detecting according to the anomaly detection method in the matching result;

And traversing the large-scale data features, and matching and detecting each data feature.

6. The intelligent data quality auditing method based on data science according to claim 5, in which the matching includes the following processes: the method comprises the steps that a word vector obtained through NLP is respectively embedded into a data feature name to be processed and keys in an anomaly detection method library, cosine similarity between the word vectors is calculated, keys with similarity being in a threshold value are potential keys corresponding to the data feature, and anomaly detection methods corresponding to the keys are matching results;

The cosine similarity is calculated as follows:

where u and v represent two word vectors, respectively.

7. The intelligent data quality auditing method according to claim 6, wherein the large-scale data feature traversal process includes: scaling each dimension value in the word vectors to be matched to be in a range of 0 to 255, sequentially expanding and arranging n pixel point arrays to represent each word vector, wherein n is the dimension of the word vector, the value of each dimension of the word vector is the gray value of each pixel point, so that an image represented by the pixel point arrays is copied to white background pictures of m pixel points to obtain a reproduction graph, wherein m is x-2 times of n, x is a natural number greater than or equal to 2, the pixels of the reproduction graph are reduced to n, the gray value of each pixel is read, a new special word vector is formed, and cosine similarity calculation is performed by using the special word vector to reduce the calculation intensity under large-scale data quantity.

8. The intelligent data quality auditing method based on data science according to claim 2, wherein the task scheduling process includes:

the arrangement server splits the tasks into different nodes, deploys the different nodes on a plurality of servers respectively, and distributes the operation resources of the servers to reduce the calculation pressure of each server;

The arrangement server uniformly collects and distributes cluster requests, orderly distributes tasks to the node server clusters by adopting a producer consumer mode, distributes task execution strategies according to cluster configuration conditions, and feeds back task execution conditions in real time;

and performing task scheduling according to the condition of the node server cluster, and transferring the tasks on the node server cluster to other normal nodes under the condition that a certain node in the cluster fails, so as to ensure that the task operation is not influenced by downtime of part of the node servers.