CN110502552B - Classification data conversion method based on fine-tuning conditional probability - Google Patents

Abstract

The invention relates to the field of data mining or machine learning of data preprocessing, and provides a classification data conversion method based on fine-tuning conditional probability, which comprises the following steps: s1, collecting classified data; s2, preprocessing data, and cleaning missing data, noise data and invalid data in the classified data; s3, calculating conditional probability, and converting the cleaned classification data into numerical vectors; s4, fine-tuning the conditional probability, and performing numerical fine-tuning on the numerical vector converted in the step S3; and S5, embedding numerical values of the classification data, namely embedding or mapping the original classification data into numerical value data for the numerical value vector subjected to numerical value fine adjustment in the step S4. The invention can convert the classification value in the classification data set into a high-quality numerical vector, and the converted numerical data can keep the real distribution of the original data, thereby ensuring the reliability of the data mining task.

Description

A Categorical Data Transformation Method Based on Fine-tuning Conditional Probability

technical field

The invention relates to the field of data mining or machine learning for data preprocessing, in particular to a classification data conversion method based on fine-tuning conditional probability.

Background technique

In a data mining or machine learning task, the collected data usually contains two types of data, numerical and categorical. However, most machine learning algorithms (such as neural networks, support vector machines, logistic regression, etc.) can only directly process numerical data, and only a few algorithms such as decision trees and Bayesian can directly process classified data; Algorithms for categorical data often have more efficient performance than algorithms that process categorical data directly. In order to widely use machine learning algorithms with numerical input, categorical data needs to be converted to numerical data. At present, a variety of categorical data conversion methods have been proposed at home and abroad. However, one of the defects of most of these methods is that the categorical data is converted into low-quality numerical data, which deviates from the true distribution of the original data, so that the next step is reduced. Performance and reliability of stage machine learning algorithms. Therefore, it is extremely important to study an efficient and reasonable classification data conversion method.

Among the many methods for converting categorical data into numerical data, the most commonly used method is One-hot Encoding, which converts each categorical value in a categorical attribute into a high-dimensional 0-1 vector; when the classification When the categorical value cardinality of attributes is large, this method is prone to the curse of dimensionality, which increases the overhead of data storage and the time overhead of subsequent machine learning algorithms. For this reason, patent CN109740680A discloses a classification method and system for mixed-value attribute approval data. After one-hot encoding is converted into high-dimensional numerical data, neural network is used for deep encoding to reduce the attribute dimension, but it takes a lot of money Time to find a good neural network structure; patent US20190164083A1 discloses a classification data conversion and clustering method for machine learning in the field of natural language processing. This method first uses one-hot encoding conversion, and then uses a clustering algorithm to Reduce attribute dimensionality. In addition to one-hot encoding and its improved method, patent CN109255373A discloses a data processing method for digitizing classified data, but this method is only applicable to environmental issues such as land use and soil type, and is not universal. The authorized patent US9619757B2 discloses a nominal attribute conversion method using the possibility of results, which converts each categorical value into the possibility (or probability) of the categorical value appearing in the data set. This method does not consider the class label information. Therefore some information may be lost.

Kasif et al. proposed a conversion method based on memory inference after considering the class label information, converting each categorical value within a categorical attribute into a conditional probability vector. However, they did not apply the transformed conditional probability vectors to machine learning algorithms for numerical inputs, but only for computing distances between categorical values. Hernández-Pereira et al. applied the conditional probability of the above conversion method to the neural network algorithm of numerical input, and achieved good experimental results in the intrusion detection problem. The conversion method based on memory reasoning has obtained higher-quality numerical data due to the consideration of class label information. However, we have found through in-depth analysis of this conversion method that it relies on the attribute independence assumption, assuming that the attributes in the data set are mutually independant. This assumption is violated when there is a certain dependency between attributes (note: attributes are usually interdependent), so the converted conditional probability is not very reliable, and slightly deviates from the true distribution of the original data.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a classification data conversion method based on fine-tuning conditional probability, which can convert the classification values in the classification data set into high-quality numerical vectors, so that the converted numerical data can still maintain the true distribution of the original data, thereby It improves the classification performance of the next-stage machine learning algorithms and ensures the reliability of data mining tasks.

A classification data conversion method based on fine-tuning conditional probability proposed by the present invention includes:

S1. Data collection of classified data;

S2. Data preprocessing, cleaning missing data, noisy data, and invalid data in the classified data;

S3. Conditional probability calculation, converting the cleaned classified data into numerical vectors;

S4, fine-tuning the conditional probability, performing numerical fine-tuning on the converted numerical vector in step S3;

S5. Numerical embedding of categorical data, embedding or mapping the original categorical data into numerical data for the numerical vector after numerical fine-tuning in step S4.

Beneficial effects of the classification data conversion method based on fine-tuning conditional probability of the present invention: reliability: the classification values in the classification data set can be converted into high-quality numerical vectors, and the converted numerical data can maintain the true distribution of the original data, ensuring Improve the reliability of data mining tasks;

High performance: After the converted data is applied to the next stage of machine learning algorithms, high performance indicators (high accuracy, recall, F score, etc.) can be obtained;

Efficiency: The converted data dimension is much lower than the one-hot encoding method, and has less running time than the one-hot encoding and its improved methods;

Convenience: The number of preset parameters is small, which reduces the trouble caused by user setting parameters, and is more conducive to actual application scenarios;

Universality: It is a data-driven transformation method that can be adaptively applied to various classification data sets.

Description of drawings

FIG. 1 is an algorithm flow chart of a classification data conversion method based on fine-tuning conditional probability according to an embodiment of the present invention;

FIG. 2 is an actual application environment diagram of a classification data conversion method based on fine-tuning conditional probability according to an embodiment of the present invention;

3 is a sample diagram of a classification data matrix based on a fine-tuning conditional probability classification data conversion method according to an embodiment of the present invention;

FIG. 4 is an application system architecture diagram of a classified data conversion method based on fine-tuning conditional probability according to an embodiment of the present invention;

FIG. 5 is an example of a classification data conversion method based on fine-tuning conditional probability to realize classification data conversion according to an embodiment of the present invention;

Among them: 101. Data collection, 102. Network, 103. Database, 104. Service system, 105. User equipment, 200. Classification data sample, 301. Data conversion module, 302. Classifier module, 303. Analysis report, 401 . Calculation of conditional probability, 402. Estimated effective range, 403. Fine-tuning conditional probability, 404. Verification after fine-tuning, 405. Conditional judgment, 501. Classification data set, 502. Classification attribute, 505. Numerical data set.

Detailed ways

A classification data conversion method based on fine-tuning conditional probability of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. The following examples are only used to illustrate the technical solution of the present invention more clearly, but not to limit the protection scope of the present invention.

As shown in Figure 1, the present invention is a classification data conversion method based on fine-tuning conditional probability, comprising:

S1. Data collection of classified data 101;

S2. Data preprocessing, cleaning missing data, noisy data, and invalid data in the classified data;

S3. Conditional probability calculation 401, converting the cleaned classified data into a numerical vector;

S4, fine-tuning the conditional probability, performing numerical fine-tuning on the converted numerical vector in step S3;

S5. Numerical embedding of categorical data, embedding or mapping the original categorical data into numerical data for the numerical vector after numerical fine-tuning in step S4.

Beneficial effects of a classification data conversion method based on fine-tuning conditional probability of the present invention: reliability: the classification value in the classification data set 501 can be converted into a high-quality numerical vector, and the converted numerical data set 505 can maintain the original data True distribution ensures the reliability of data mining tasks;

High performance: After the converted data is applied to the next stage of machine learning algorithms, high performance indicators (high accuracy, recall, F score, etc.) can be obtained;

Efficiency: The converted data dimension is much lower than the one-hot encoding method, and has less running time than the one-hot encoding and its improved methods;

Convenience: The number of preset parameters is small, which reduces the trouble caused by user setting parameters, and is more conducive to actual application scenarios;

Universality: It is a data-driven transformation method that can be adaptively applied to various classification data sets501.

在算法的流程图中，条件概率计算401首先提取每个分类属性502A_i和类标签C中的数据，然后计算属性A_i内每个分类值a_i(x)的条件概率，并生成如下的l维数值向量：Suppose X is a classification data set 501 containing N samples, each sample is represented by an m-dimensional vector [a ₁ (x),...,am _{(x)], where a i (} x) is the sample x The categorical value of the i-th attribute, in addition, the class label of X is

In the flow chart of the algorithm, the conditional probability calculation 401 first extracts the data in each classification attribute 502A _i and class label C, and then calculates the conditional probability of each classification value a _i (x) in the attribute A _i , and generates the following l-dimensional numeric vector:

a _i (x)→[P(c ₁ |a _i (x)),…,P(c _j |a _i (x)),…,P(c _l |a _i (x))] (1)

The conditional probability item P(c _j |a _i (x)) in formula (1) is calculated by Bayesian Estimation of Laplace Smoothing, which is:

I(x,y) in formula (2) is an indicator function, that is, I(x,y)=1 when x=y, otherwise I(x,y)=0; λ(≥0) is a Laplace smoothing factor.

Estimate the valid range 402, using the valid range algorithm (ValidRanges Algorithm) to calculate the valid range of each classification value a _i (x) in the attribute A _i [P _min (c _j |a _i ), P _max (c _j |a _i )], where 0≤P _min (c _j |a _i )≤P _max (c _j |a _i )≤1.

S01. If the conditional probability item P(c _j |a _i (x)) is used to correctly classify the number of samples greater than the number of wrongly classified samples, that is, Neg_ratio(a _i ,c _j )＞pos_ratio(a _i ,c _j ) , fine-tune the probability item P(c _j |a _i (x)), otherwise exit the fine-tuning process;

与条件概率

的绝对值，

其中

S02, calculate the average effective range of the classification value a _i (x)

with conditional probability

the absolute value of

in

用

进行更新，即

S03, the conditional probability

use

to update, i.e.

即

S04. Conditional probability of normalized update

which is

Verification after fine-tuning 404 , using a machine learning classifier to verify whether the performance of the conditional probability after fine-tuning is improved, that is, verifying whether the fine-tuning algorithm can more realistically fit the distribution of the original data.

Condition judgment 405, judge whether the performance of the conditional probability in verification 404 after fine-tuning is improved, if it is improved, it shows that this fine-tuning is effective, go to fine-tuning conditional probability 403, and continue fine-tuning; otherwise, terminate the fine-tuning process and exit the program; in addition, for To prevent the fine-tuning process from entering an infinite loop, the number of fine-tuning is limited to the preset 1000 times.

The computing environment diagram includes four functional blocks of data acquisition 101 , storage database 103 , data mining service system 104 and user equipment 105 coupled by communication network 102 . The data collection 101 terminal may automatically collect useful classified data (such as e-commerce web page data, medical monitoring data, etc.) online by desktop computers, notebook computers or mobile devices, or it may be manually collected and then entered into the classified data set 501 of the system (such as market access data, census data, etc.). The data collection 101 terminal sends the collected classified data set 501 to the database 103 through the network 102 for storage. The database storing the classified data set 501 may be a local workstation or a remote server, or a cloud data server. The user sends a request to the service system 104 through the user equipment 105 to analyze a certain data mining task (such as a credit card fraud detection task). After receiving the request, the service system 104 calls the corresponding classified data set 501 from the database 103, and sends the analysis report 303 back to the user device 105 after data mining and analysis for the user to view and make decisions.

The data acquisition 101 stores the collected classified data sets 501 in the database 103 , an example of these classified data sets 501 is shown in FIG. 3 . The classification data sample 200 is a data matrix set for credit card fraud detection, each row of the matrix represents a credit loan customer, and each column describes the basic information (or attributes, such as gender, marriage status, income, credit history) of the customer. These attributes are categorical data (such as the value of gender is "male", "female") rather than numerical data (such as 0.12, 1.85, etc.).

When the user device 105 requests the service system 104 to analyze a certain data mining task, a service system 104 applied to the user is shown in FIG. 4 . The service system 104 first calls the corresponding classification data set 501 from the database 103 , and then runs the data conversion module 301 and the classifier model 302 in the system, and compiles the analysis report 303 . The data conversion module 301 of the present invention can convert the classified data in the database 103 into high-quality numerical data, and it includes a data preprocessing submodule with a data cleaning function (such as cleaning missing data, noise data, etc.), a conditional probability calculator module, a fine-tuning conditional probability submodule, and an embedded numerical submodule (data embedding or data mapping). The converted numerical data is sent into the classifier module 302, and the classifier module 302 selects a suitable machine learning model (such as learning models such as neural network, support vector machine, and logistic regression) and a loss function (square loss, 0-1 loss , cross-entropy loss, log loss, etc.) to train a classifier. Then, the classifier in the classifier module 302 evaluates the converted data in the data conversion module 301 and generates an analysis report 303 . The analysis report 303 mainly includes the labels of the predicted samples, and the evaluation of the performance and efficiency of the classifier.

An embodiment of the data conversion module 301:

The classification data in the data set can be converted into numerical data by using the data conversion module 301 , and the following uses the data set of credit card fraud detection as an example for illustration. The credit card fraud detection data set comes from the credit card department of a certain bank in a certain city. A total of 284,807 data records were collected in 2013, and each record contains 28 classification attributes 502. An example of this data set is shown in the sample categorical data matrix 200 .

The operation steps are as follows:

Step1: the data preprocessing sub-module in the data conversion module 301 obtains the processed classification data set 501 after operations such as cleaning missing data and noise data on the original data;

Step2: extract every N classification attribute 502 and class label from classification data set 501;

Step3: Calculate the conditional probability 401 through the formulas (1) and (2), for example: the conditional probability corresponding to the classification value "married" is [0.15, 0.51, ...], and the conditional probability corresponding to the classification value "single" is [0.33 , 0.12, ...] and so on;

Step4: The fine-tuning conditional probability 403 is carried out according to the accompanying drawing Fig. 5 of the present invention, for example, the conditional probability corresponding to the classification value "marriage" is [0.15, 0.51, ...] After fine-tuning, its corresponding fine-tuning conditional probability 403 is [0.13, 0.47, ...];

Step5: The categorical data of the categorical data set 501 is converted with the fine-tuning conditional probability 403 , and the converted numerical data is saved in the numerical data set 505 .

Claims (2)

1. A classification data conversion method based on fine-tuning conditional probability, characterized in that, comprising: S1. Data collection of classified data; S2. Data preprocessing, cleaning missing data, noisy data, and invalid data in the classified data; S3. Conditional probability calculation, converting the cleaned classified data into numerical vectors; S4, fine-tuning the conditional probability, performing numerical fine-tuning on the converted numerical vector in step S3; S5. Numerical embedding of classified data, embedding or mapping the original classified data into numerical data for the numerical vector after numerical fine-tuning in step S4; Wherein steps S2 and S3 specifically include: X is a classification data set containing N samples, each sample is represented by an m-dimensional vector [a ₁ (x),...,am (x)], where a _i (x) is the i- _th sample x The categorical value of the attribute, in addition, the class label of X is

The conditional probability calculation first extracts the data in each classification attribute A _i and class label C, and then calculates the conditional probability of each classification value a _i (x) in the attribute A _i , and generates the following l-dimensional numerical vector: a _i (x)→[P(c ₁ |a _i (x)),…,P(c _j |a _i (x)),…,P(c _l |a _i (x))] (1) The conditional probability item P(c _j |a _i (x)) in formula (1) is calculated by Bayesian Estimation of Laplace Smoothing, which is:

I(x,y) in formula (2) is an indicator function, that is, I(x,y)=1 when x=y, otherwise I(x,y)=0; λ≥0 is a Lapp Lass smoothing factor; Wherein step S3 also includes: To estimate the valid range, use the valid range algorithm (Valid Ranges Algorithm) to calculate the valid range of each classification value a _i (x) in the attribute A _i [P _min (c _j |a _i ), P _max (c _j |a _i )], where 0≤P _min (c _j |a _i )≤P _max (c _j |a _i )≤1; Wherein step S4 comprises: S01. If the conditional probability item P(c _j |a _i (x)) is used to correctly classify the number of samples greater than the number of wrongly classified samples, that is, Neg_ratio(a _i ,c _j )＞pos_ratio(a _i ,c _j ) , fine-tune the probability item P(c _j |a _i (x)), otherwise exit the fine-tuning process; S02, calculate the average effective range of the classification value a _i (x)

with conditional probability

The absolute value of the difference,

in

S03, the conditional probability

use

to update, i.e.

S04. Conditional probability of normalized update

which is

Wherein step S4 also includes: Verification after fine-tuning, use the machine learning classifier to verify whether the performance of the conditional probability after fine-tuning is improved, that is, verify whether the fine-tuning algorithm can more realistically fit the distribution of the original data; Wherein step S5 comprises, Conditional judgment, judge whether the performance of conditional probability in verification after fine-tuning is improved, if it is improved, it means that this fine-tuning is effective, go to fine-tuning conditional probability, and continue fine-tuning; otherwise, terminate the fine-tuning process and exit the program. 2. A classification data conversion method based on fine-tuning conditional probability as claimed in claim 1, characterized in that, the number of fine-tuning is limited within the preset 1000 times.

Images