TWI758725B - Data analysis system and data analysis method - Google Patents

Abstract

A data analysis method includes following steps: obtaining a data table including a plurality of fields, each of which stores field data; classifying a field type based on these field data; determining the field type of each of these fields; calculating a similarity between the fields, and determining a correlation between the fields based on the similarities; generating a field data description file according to these field types, field types, and correlations, and by the field data description file to determine whether there is abnormal data in the data to evaluate the data quality.

Description

Data analysis system and data analysis method

Embodiments of the present invention mainly relate to an analysis method, and particularly relate to a data analysis system and data analysis method.

As data collection has become more convenient, the amount of available data has increased rapidly, and data analysis technology has also flourished. Effective big data data analysis results depend on good data quality, so data quality is an important issue in data analysis. The current data quality diagnosis methods can be divided into data analysis experts using programming language to analyze, or using market analysis package software.

However, in the data analysis process, the data quality must be confirmed first, and then the data preprocessing is carried out. However, in practice, the data quality is often observed simultaneously in the data preprocessing stage, which requires a lot of manpower at this stage, resulting in huge communication and Time costs.

Therefore, how to establish an automatic auxiliary mechanism to reduce the manpower and time cost required in the data preprocessing stage has become one of the problems to be solved in this field.

In view of the above-mentioned problems of the prior art, embodiments of the present invention provide a data analysis system and method.

According to an embodiment of the present invention, a data analysis system is provided. The above data analysis system includes a processor, a storage device, a column type analysis device, a column classification device and a column association device. The processor is used for obtaining at least one data table, and the data table includes a plurality of fields, each of which stores a field of data. The storage device is used to store the data table. The column type analysis device is used for analyzing a column type according to the column data. The field classification device is used for judging a field type of each of the fields. The field association device is used for calculating a similarity between the fields in the cross-data table, and judging a correlation between the fields according to the similarity. The processor generates a field data description file according to the field types, field types and these associations, and the processor determines whether the field data description file is abnormal.

According to an embodiment of the present invention, a data analysis method is provided. The steps of the above data analysis method include: obtaining a data table, the data table includes a plurality of fields, each of which stores a field data; analyzing a field type according to the field data; judging these fields a field category for each of the fields; calculating a similarity between the fields in the cross data table, judging a correlation between the fields according to the similarity; and according to the fields A field data description file is generated based on the category, field type and association, and then it is judged whether the field data description file is abnormal.

According to the data analysis method and data analysis system proposed by the present invention, in the stage of data preprocessing, an automated mechanism can be established to generate the data of the field by analyzing information such as field type, field type, correlation, etc. The description file can help users to quickly understand the data, reduce the manpower required in the data preprocessing stage, and improve the data analysis efficiency in the data preprocessing stage.

The following descriptions are preferred implementations for completing the invention, and are intended to describe the basic spirit of the invention, but are not intended to limit the invention. Reference must be made to the scope of the following claims for the actual inventive content.

It must be understood that the words "including" and "including" used in this specification are used to indicate the existence of specific technical features, values, method steps, operation processes, elements and/or components, but do not exclude the possibility of Plus more technical features, values, method steps, job processes, elements, components, or any combination of the above.

The use of words such as "first", "second", "third", etc. in the claims is used to modify the elements in the claims, and is not used to indicate that there is a priority order, an antecedent relationship between them, or an element Prior to another element, or chronological order in which method steps are performed, is only used to distinguish elements with the same name.

FIG. 1 shows a block diagram of a data analysis system 100 according to an embodiment of the present invention. As shown in FIG. 1 , the data analysis system 100 may include a processor 10 , a storage device 20 , a column type analysis device 30 , a column classification device 40 and a column association device 50 . It should be noted here that the block diagram shown in FIG. 1 is only for the convenience of explaining the embodiment of the present invention, but the present invention is not limited to FIG. 1, and the data analysis system 100 may also include other components.

In one embodiment, the processor 10 is, for example, a microcontroller, a microprocessor, a digital signal processor, an application specific integrated circuit (ASIC) or an logic circuit.

In one embodiment, the column type analysis device 30, the column classification device 40 and the column association device 50 may be implemented individually or in combination as, for example, a microcontroller, a microprocessor, a digital signal A digital signal processor, an application specific integrated circuit (ASIC) or a logic circuit.

In one embodiment, the field type analysis device 30 , the field classification device 40 and the field association device 50 may be software running on electronic devices (eg, including circuits, processors, or logic circuits).

In one embodiment, the storage device 20 is, for example, a read-only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a pen drive, a magnetic tape, a database accessible through a network, or a person skilled in the art can easily contemplate and storage media with the same function. The storage device 20 can be used to store one or more data tables.

FIG. 2 is a schematic diagram showing a data analysis method 200 according to an embodiment of the present invention. The data analysis method 200 of FIG. 2 can be implemented by the data analysis system 100 of FIG. 1 .

In step 210, the processor 10 obtains a data table.

In one embodiment, the data table includes a plurality of fields, each of which stores a field of data. For example, the data table includes a machine model field, a machine identification (ID) field, a machine multi-task field, a manufacturing time field, a shipping time field, etc., and these fields store different For example, the machine model field stores "NB1" (this is a string), the machine identification field stores "3" (this is an integer), and the machine multi-task field stores "0" (this is the Boolean value ), the manufacturing time field stores "2020/03/16" (this is the date), and the shipping time field stores "2020/09/16" (this is the date). However, this is just an example, and the fields and field data of the present invention are not limited thereto.

In one embodiment, the processor 10 may obtain multiple data tables.

In step 220, the processor 10 triggers the column type analysis device 30, the column classification device 40 and the column association device 50 to generate a column data description file.

In one embodiment, step 220 includes any one or a combination of a plurality of sub-steps 220(a)-220(c). In sub-step 220(a), the processor 10 analyzes and obtains the field type, and in sub-step 220(b), the processor 10 analyzes and obtains the field identification. In sub-step 220(c), the processor 10 analyzes and obtains the field association.

In one embodiment, the field type analysis device 30 analyzes a field type according to the field data. Field type refers to the data type of the content stored in each field (for example, there are 500 data in a row). In the number of data records, the type of data that accounts for more is regarded as the main type of the field. For example, there are 500 records in a field in the data table, 499 of which are numerical values, then this field is defined as a value field type state.

In one embodiment, the field classification device 40 determines a field type of each of the fields. Field category refers to the category to which the field name itself belongs, such as human, machine, material, method, measurement, other...etc. For example, if the field name includes the keyword machine, the field category is divided into the field of the machine category.

In one embodiment, the field association device 50 calculates a similarity between two fields in different data tables (cross-data tables), and determines whether a correlation between the fields is related according to the similarity. exist. Relevance refers to the degree of correlation between at least two fields in a cross data table, such as the manufacturing time field in the product manufacturing table and the shipping time field in the product shipping table, these two columns from different data tables Bits are correlated in time.

In one embodiment, the processor 10 generates a field data description file according to the field types, field types and these associations, and then determines whether the field data description file is abnormal.

In one embodiment, the field data description file includes information such as the field types, field types, field associations, and so on.

The detailed flow of the column type analysis device 30 , the column classification device 40 , and the column association device 50 will be described correspondingly in the following Figures 3 to 5 .

In step 230, the processor 10 determines whether the field data description file is abnormal. In one embodiment, the processor 10 determines whether the field data description file is complete or correct. In one embodiment, if the processor 10 determines that the field data description file is incomplete or incorrect, it proceeds to step 240 . If the processor 10 determines that the field data description file is complete and correct, the process ends.

In one embodiment, the situation that the field data description file is judged to be abnormal includes: the field data description file is incomplete, or the field data description file has errors.

For example, there are 500 pieces of data in a field in the data table, 499 of which are numeric values and 1 is a string. This field should be defined as a numeric field type. If the field type analysis device 30 If the analysis is other field types (such as string, boolean value, time), the processor 10 determines that the field data description file is abnormal, and proceeds to step 240 .

For example, there are 500 pieces of data in a field in the data table, 499 of the field data are numerical values, and 1 is blank data. If the field type analysis device 30 fails to analyze the field type due to the blank data , the processor 10 determines that the field data description file is incomplete or incorrect, and proceeds to step 240 .

In step 240, when the processor 10 determines that the field data description file is abnormal, it automatically corrects the content of the field data description file.

In one embodiment, the processor 10 recalculates the missing data from the storage device 20 based on the missing part in the field data description file to automatically correct the content in the field data description file, for example, step 240 includes: Sub-steps 241 to 243 : amend the column data category 241 , amend the column data type 242 and/or amend the related column 243 in other data tables. In one embodiment, the user may enter new data description file content based on the missing portion of the data description file. For example, the user inputs new or updated data based on the missing part in the description file through an input device (such as a mouse cursor, a touch screen, a keyboard), and the processor 10 receives the new or updated data from the input device After the data is stored, the processor 10 completes the content in the field data description file through the added or updated data. For example, the automatic correction includes: adding/updating field data description, adding or updating field data Number of groups (group), new or updated fields allow nullable values (nullable), new or updated field data upper and lower bounds (value range), whether exception data is allowed to be ignored, and/or new or updated identical data tables There is a relation column.

In one embodiment, the processor 10 performs the missing part in the data description file according to a preset rule (for example, filling the blank field with "0" or calculating an average value according to the data of two adjacent fields of the blank field. value, fill in the average value in the blank field) to correct the missing part.

In one embodiment, the processor 10 determines that the field data can be a null value according to a preset rule, then the processor 10 is set to allow the field data in the field data description to be a null value, and the subsequent data analysis system will ignore this. Unusual data.

In one embodiment, when the processor 10 determines that the field data description file is abnormal, the processor 10 modifies the field data description file (for example, converts the value into a string), adds a field data description file (for example, Through user input or processor 10 retrieves missing data from storage device 20), edits the field data description file (eg changing the value size), ignores abnormal data or displays the field data description file exception through a display.

FIGS. 3A-3B show a flowchart of a field type analysis method 300 according to an embodiment of the present invention. In step 310, the processor 10 obtains one or more data tables. In step 320, the column type analysis device 30 analyzes the column type.

In one embodiment, the field type analysis device 30 regards the data type with the largest number in a single field as the field type of the field. For example, there are 500 pieces of data in a field in the data table, and there are 499 pieces of data. is a numeric value, this field type is defined as the numeric field type. For example, there are 500 pieces of data in a column in the data table, and 480 of them are strings, so this field type is defined as the string field type.

In step 330, the field type analysis device 30 determines whether the field type is a numerical field type. If the field type analysis device 30 determines that the field type is a numerical field type, the process proceeds to step 340 . If the field type analysis device 30 determines that the field type is not the numeric field type, then the process proceeds to step 350 .

In step 340, the field type analysis device 30 determines whether the field data are integers or floating-point numbers. If the field type analysis device 30 determines that the field data are integers or floating-point numbers, the process proceeds to step 343 . If the field type analysis device 30 determines that the field data are not integers or floating-point numbers, the process proceeds to step 345 .

In one embodiment, integers and floating point are collectively referred to as numeric values.

In step 343, the data type analysis device 30 confirms that the field type in the field data description file is the numeric field type.

In one embodiment, the numeric field types include integers and floating-point numbers.

In one embodiment, if the field type analysis device 30 finds that there is an abnormality in the field data, it adds a field data description file, edits the field data description file, ignores the abnormal field data, or displays the abnormality through a display. material. For example, if there are some null values in the field data, the null value data in this field will be ignored.

In step 345, the field type analysis device 30 corrects the field type to a non-numerical field type.

In one embodiment, when the field type analysis device 30 further determines that only 0 or 1 is stored in the field data, it is regarded as a boolean value field type, so the corrected field type is a non-numerical field type state. This is just an example, not limited to this.

In step 350, the field type analysis device 30 determines whether the field data includes a numerical value. If the field type analysis device 30 determines that the field data includes numerical values, the process proceeds to step 353 . If the field type analysis device 30 determines that the field data are not numerical values, the process proceeds to step 355 .

In one embodiment, the field type analysis device 30 further determines that "12" of the string type stored in the field data is regarded as including a numerical value, so step 353 is entered. However, this is only an example, and the present invention is not limited thereto.

In step 353, the field type analysis device 30 modifies the field type in the field data description file to the value field type.

In one embodiment, if the field type analysis device 30 finds that there is an abnormality in the field data, it adds a field data description file, edits the field data description file, ignores the abnormal field data, or displays the abnormality through a display. material. For example, if there are many null values in the field data (resulting in determining that the field type is a non-numeric field type in step 320), these null value field data may be ignored for this field, thereby modifying the field In the data description file, if all the non-null values in the field data are numeric data, the field type in the field data description file is modified to the numeric field type.

In step 355, the field type analysis device 30 determines whether the field data is one of the data types of date, time, time and date. If the field type analysis device 30 determines that the field data is one of the data types of date, time, time and date, the process proceeds to step 360 . If the field type analysis device 30 determines that the field data is not one of the data types of date, time, time and date, the process proceeds to step 370 .

In one embodiment, the data types of date, time, time and date are collectively referred to as time data types.

In step 360, the field type analysis device 30 modifies the field type in the field data description file to the time field type.

In one embodiment, the field type analysis device 30 subdivides the time field type. For example, the field type analysis device 30 subdivides the time field type into time or date. For another example, the field type analysis device 30 subdivides the time field type into date and time.

In step 370, the field type analysis device 30 determines whether the field data can be divided into other field types. If the column type analysis device 30 determines that the column data can be divided into other column types (for example, the column type analysis device 30 can still analyze that the specific column data accounts for a larger proportion), then go to step 380 . If the field data in the field type analysis device 30 cannot be divided into other field types, the process ends.

In step 380, the field type analysis device 30 determines whether the field data is a text data or a Boolean value data, when the field type analysis device 30 determines that the field data is a text data or a Boolean value data, corresponding to these field data, modify the field type in the field data description file to a text type or a boolean value type.

FIG. 4 is a flowchart showing a field classification method 400 according to an embodiment of the present invention. In step 410, the field classification apparatus performs word segmentation on each of the field names. For example, if the field name in Chinese is "machine number", the hyphenated words are "machine" and "number". For another example, if the field name in English is "functionId", the hyphenated words are "function" and "Id". The method of word segmentation of Chinese field names is usually to map the field name to a known corpus. If a matching word is found, the word is separated. In addition, known word segmentation algorithms such as CKIP, HanLP, Ansj, Jieba...etc. The word segmentation method for English field names can be to find out the capitalization rules, radicals, underscores, blanks, or naming rules according to field names to separate words.

In step 420 , the field classification device 40 converts each of the plurality of words after word segmentation into a word feature, and inputs these word features into a classification model.

In one embodiment, the field classification device 40 compares all segmented words with a pre-built corpus. For example, if the word "machine" exists in the pre-built corpus, mark "machine" as 1; for example, if the word "ice cream" does not exist in the pre-built corpus, mark "ice cream" is 0. After the field classification device 40 compares the pre-built corpus with all the segmented words, there will be many word features composed of 0s and 1s.

In one embodiment, the word features may be feature vectors, feature matrices, or a sequence of values. The field classification device 40 inputs these word features into a classification model, such as a decision tree model. Decision tree models are often used in decision analysis to help determine a strategy that is most likely to achieve a goal. Decision trees can be used as a descriptive means of calculating conditional probabilities. In other words, decision trees can analyze the most likely category of fields based on word characteristics. The decision tree model is a known technology, so it will not be repeated here.

In step 430, the classification model outputs field categories according to the word features. In one embodiment, the field category is, for example, human, machine, material, method, measurement, or others. However, this is only an example, and the present invention is not limited thereto.

For example, if the word feature corresponding to "machine" is input into the decision tree model, the decision tree model will map "machine" to the column category of the machine.

For example, if the word feature corresponding to "centimeter" is input into the decision tree model, the decision tree model will map "centimeter" to the measured field category.

In one embodiment, the column classification device 40 uses a Decision Tree algorithm, a Bayes Classification algorithm, a k-Nearest Neighbors algorithm, a Support Vector Machine (SVM) Vector Machine) algorithm to determine the field type of each of the fields

Thereby, the column classification device 40 can apply the column classification method 400 to analyze the column type according to the table and the column name.

FIG. 5 is a flowchart illustrating a field association method 500 according to an embodiment of the present invention. In one embodiment, the processor 10 obtains a plurality of data tables. In step 510, the field association device 50 selects two data tables from a plurality of different data tables as a first data table and a second data table, and selects a first field from the first data table , select a second column from the second data table, the first column includes a first word segmentation data, and the second column includes a second word segmentation data.

In one embodiment, the field association device 50 performs word segmentation on the field data in the first field and the second field to obtain the first word segmentation data and the second word segmentation data.

In one embodiment, the language of the first segmented data and the second segmented data is the same. For example, in the Chinese example, the first word segmentation data is "machine", and the second word segmentation data is "machine". For example, in the English example, the first segmented data is "wire", and the second segmented data is "wireless".

In step 520, the field association device 50 calculates the similarity between the first segmented data and the second segmented data. In one embodiment, the minimum edit distance is selected, and the similarity is calculated according to the minimum edit distance. However, the present invention is not limited to this.

In one embodiment, the field association device 50 uses the minimum edit distance as the similarity implementation method, and the minimum edit distance refers to the number of different characters between the first word segmentation data and the second word segmentation data, for example, in the Chinese example , when the first word segmentation data is "machine" and the second word segmentation data is "machine", the difference between the two is 1, and the minimum edit distance is regarded as 1. For example, in the English example, when the first segmented data is "wire" and the second segmented data is "wireless", the number of characters (number of English letters) that are different between the two is 4, and the minimum edit distance is regarded as 4 .

In one embodiment, the field association device 50 calculates the similarity according to the minimum edit distance. For example, in the above Chinese example, the longest character string has two Chinese characters, in other words, the longest character string is 2, and 2 is used as the denominator. , take the longest string minus the minimum edit distance (2-1=1) as the numerator, so the similarity is 1/2 (ie 50%).

Another example is in the Chinese example, when the first word segmentation data is "number" and the second word segmentation data is "number", the longest word has two Chinese characters, in other words, the longest word string is 2, the 2 is used as the denominator, and the number of different words between the two is 0. The longest character string minus the minimum edit distance (2-0=2) is used as the numerator. Therefore, the similarity is 2/2 (ie 100%).

For example, in the above English example, the longest word has eight English letters, in other words, the longest character string is 8, and 8 is used as the denominator, and the longest character string minus the minimum edit distance (8-4=4) is used as the Molecules, therefore, are 4/8 (i.e. 50%) similar.

In step 530, the field association device 50 determines whether the data is greater than a similarity threshold. When the field association device 50 determines that the similarity is not greater than the similarity threshold, the process proceeds to step 550 . When the field association device 50 determines that the similarity is greater than the similarity threshold, the process proceeds to step 540 .

For example, the similarity threshold can be preset to 80%, which means that when the similarity is greater than 80%, the two fields are considered to be related. In the above example, when the first segmented data is "number" and the second segmented data is "number", the similarity is 100%, and the similarity of 100% is greater than the similarity threshold of 80%, representing the first column There is an association between the bit and the second field.

In one embodiment, the field classification device 40 calculates the Euclidean Distance, Manhattan Distance, Hamming Distance, Minkowski distance, Cosine Similarity, Jaccard Similarity, Edit Distance or Pearson Correlation Coefficient to generate similarity.

In step 540, the field association device 50 establishes the association between the first field and the second field. In one embodiment, for example, a flag may be added to the first field and the second field or the association may be recorded in a file.

In this way, the first field can be associated with the second field to facilitate subsequent use, for example, the parameters of a specific experiment are recorded in the first field, and the results of a specific experiment are recorded in the second field. The association between the field and the second field can associate the parameter with the result. In other words, establishing associations helps to centralize relevant fields in complex and massive data tables and their field data, and can also be used for other applications in data characteristics.

In step 550, the field association device 50 determines whether all the field combinations in the first data table and the second data table have calculated the similarity. If the field association device 50 determines that the similarity has been calculated for all the field combinations in the first data table and the second data table, the process ends. If the field association device 50 determines that the data similarity has not been calculated for all the field combinations in the first data table and the second data table, the process returns to step 510 .

In one embodiment, the processor 10 or the user selects the database data of a department within the enterprise as the data source, there are 2 different data tables, 30 fields, and nearly 36,000 data transactions (maybe one field data Including multiple records), the data needs to be cleaned and merged for subsequent analysis and use. In this experiment, an experimental group and a control group are designed. The experimental group uses the data analysis system 100 of this case to analyze the data. The control group invites experts in the field to check the field type, field type and field correlation by manual process, and evaluate the field. The criterion is the time it takes to evaluate each project. The experimental results are shown in Table 1: project type control group test group Analysis Field Type Experts in the field manually check the content of the field data and determine the data type of the field, which takes a total of 198 seconds Apply the data analysis method and data analysis system proposed by the present invention, which lasts 15 seconds Analysis Field Type Experts in the field manually mark fields, and it takes about 10~15 seconds for each field to determine its field type. This time, 30 fields are marked, and it takes 300~450 seconds in total Using the data analysis method and data analysis system proposed by the present invention, it lasted 0.3 seconds, and the accuracy rate reached 95.3% (in order to confirm the correctness of the automatic analysis, the automatically judged column type was compared with the manually judged column category, the resulting accuracy.) Analyze Field Associations Experts in the field manually determine whether there is a relationship between fields in multiple data tables, which takes a total of 165 seconds Applying the data analysis method and data analysis system proposed by the present invention, the comparison of every pair of columns lasts 0.2 seconds Among the performances of the three items in Table 1, the time spent in the experimental group is far better than that in the control group. Therefore, the data analysis method and data analysis system proposed by the present invention improve the efficiency of data analysis for a large amount of data and can analyze the data in real time. Huge and complex data.

According to the data analysis method and data analysis system proposed by the present invention, in the stage of data preprocessing, an automated mechanism can be established to generate the data of the field by analyzing information such as field type, field type, correlation, etc. Description file to help users quickly understand the data. It can reduce the manpower required in the data preprocessing stage and improve the data analysis efficiency in the data preprocessing stage.

The steps of the method and algorithm disclosed in the description of the present invention can be directly applied to hardware and software modules or a combination of the two by executing a processor. A software module (including execution instructions and associated data) and other data can be stored in data memory, such as random access memory (RAM), flash memory, read only memory (ROM) , Erasable Programmable Read-Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), Scratchpad, Hard Disk, Portable Hard Disk, CD-ROM ), DVD, or any other computer-readable storage media format known in the art. A storage medium can be coupled to a machine device, such as a computer/processor (for convenience of description, it is referred to as a processor in this specification), and the processor can read information (such as a program) through code), and write information to the storage medium. A storage medium can integrate a processor. An application specific integrated circuit (ASIC) includes a processor and a storage medium. A user equipment includes an application-specific integrated circuit. In other words, the processor and storage medium are included in the user equipment in a manner that is not directly connected to the user equipment. Furthermore, in some embodiments, any product suitable for a computer program includes a readable storage medium, wherein the readable storage medium includes code associated with one or more of the disclosed embodiments. In some embodiments, the product of the computer program may include packaging material.

The above paragraphs use multiple levels of description. Obviously, the teachings herein can be implemented in a variety of ways, and any particular architecture or functionality disclosed in the examples is merely a representative case. Based on the teachings herein, anyone skilled in the art should understand that each aspect disclosed herein may be implemented independently or two or more aspects may be implemented in combination.

Although the present disclosure has been disclosed above with examples, it is not intended to limit the present disclosure. Anyone who is familiar with the art can make some changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the invention is The scope of the patent application attached herewith shall prevail.

100: Data Analysis System 10: Processor 20: Storage device 30: Column type analysis device 40: Column sorting device 50: Field Association Device 200: Data Analysis Methods 300: Field Pattern Analysis Methods 400: Field Classification Method 500: Field Association Method 210~243, 310~380, 410~430, 510~550: Steps

FIG. 1 shows a block diagram of a data analysis system according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing a data analysis method according to an embodiment of the present invention. FIGS. 3A-3B are flowcharts showing a method for analyzing a field type according to an embodiment of the present invention. FIG. 4 is a flowchart showing a field classification method according to an embodiment of the present invention. FIG. 5 is a flowchart showing a field association method according to an embodiment of the present invention.

200: Data Analysis Methods 210~250: Steps

Claims (14)

A data analysis system, comprising: a processor for obtaining at least one data table, the data table including a plurality of fields, each of which stores a field data; a storage device for storing the data table ; a field type analysis device for analyzing a field type according to the field data; a field classification device for judging a field type of each of the fields; and a field association a device for calculating a similarity between the fields in the cross-data table, and judging a correlation between the fields according to the similarity; wherein the processor is based on the field types , the field type and these associations generate a field data description file, and the processor determines whether the field data description file is abnormal; wherein the field classification device is based on a first segmentation data and a second Word segmentation data to calculate Euclidean Distance, Manhattan Distance, Hamming Distance, Minkowski Distance, Cosine Similarity, Jaccard Similarity Jaccard Similarity, Edit Distance or Pearson Correlation Coefficient to generate this similarity. The data analysis system of claim 1, wherein when the processor generates the field data description file and determines whether the field data description file is abnormal, a display is used to display whether the field data description file is abnormal or not. According to the data analysis system of claim 1, the situation in which the field data description file is judged to be abnormal includes: the field data description file is incomplete, or the field data description file has errors. The data analysis system of claim 1, wherein when the processor determines that the field data description file is abnormal, it automatically corrects the content of the field data description file. According to the data analysis system of claim 1, the automatic correction includes adding/updating field data description (description), adding or updating field data group number (group), adding or updating field allowable null value (nullable) ), add or update the upper and lower bounds of the column data (value range), whether to allow ignoring abnormal data, and/or add or update a relation column in the same data table. The data analysis system of claim 1, wherein the data type analysis device determines whether the field type is a numeric field type, and if the data type analysis device determines that the field type is the numeric field type If the data type analysis device determines whether the field data is a numerical value, the data type analysis device confirms that the field data description file is in the The field type is the numeric field type. If the data type analysis device determines that the field data is not a numeric value, the data type analysis device modifies the field type to a non-numeric field type. The data analysis system of claim 1, wherein the data type analysis device determines whether the field type is a numeric field type, and if the data type device determines that the field type is not the numeric field type If the data type analysis device determines whether the field data is a numerical value, the data type analysis device determines whether the field data is a numerical value. The field type of is modified to the value field type. The data analysis system of claim 5, wherein if the data type analysis device determines that the field data is not a numerical value, the data type analysis device determines whether the field data is a plurality of time data, if the data The type analysis device determines that the field data are the time data, and then corrects the field type in the field data description file to the time field type. According to the data analysis system of claim 8, if the data type analysis device determines that the field data is not the time data, it determines whether the field data is a text data or a Boolean value data, if the data is not the time data The type analysis device determines that the field data is the text data or the Boolean value data, and corresponding to the field data, corrects the field type in the field data description file to a text type or a Boolean value type. According to the data analysis system of claim 1, wherein the field classification device performs word segmentation on each of the field data, converts each of the multiple words after the segmentation into a word feature, and inputs the word features into a classification model, the classification model outputs the field category according to the word features. The data analysis system of claim 1, wherein the processor obtains a plurality of data tables, and the field association device selects two data tables from different data tables As a first data table and a second data table, a first field is selected from the first data table, and a second field is selected from the second data table, and the first field includes the first field a word segmentation data, the second field includes the second word segmentation data, and generates a similarity between the first word segmentation data and the second word segmentation data, when the field association device determines that the similarity is greater than one When the similarity threshold is set, the correlation between the first field and the second field is established. The data analysis system of claim 11, wherein the similarity is calculated by calculating a minimum edit distance (Minimum Edit Distance) between the first word segmentation data and the second word segmentation data, and is calculated and generated according to the minimum edit distance the similarity. The data analysis system of claim 9, wherein the field classification device uses a Decision Tree algorithm, a Bayes Classification algorithm, a k-Nearest Neighbors algorithm, supports A Vector Machine (Support Vector Machine) algorithm is used to determine the field type of each of the fields. A data analysis method, comprising: obtaining a data table, the data table including a plurality of fields, each of which stores a field data; analyzing a field type according to the field data; judging the a field category for each of the fields; calculating a similarity between the fields in the cross-data table, and judging a correlation between the fields according to the similarity; and Generate a field data description file according to the field types, the field type and the associations, and then determine whether the field data description file is abnormal; according to a first segmentation data and a second segmentation data Calculate Euclidean Distance, Manhattan Distance, Hamming Distance, Minkowski Distance, Cosine Similarity, Jaccard Similarity Similarity), Edit Distance or Pearson Correlation Coefficient to generate this similarity.

Images