TWM660529U - System for screening learning sample data and removing abnormal data for machine learning - Google Patents

Abstract

A system for screening learning sample data and filtering abnormal data for machine learning, the system comprising a data analysis device and a machine learning model device, the data analysis device further comprising a data receiving module, a feature conversion module, a clustering module, a similarity calculation module, a clustering merging module, an abnormal data filtering module and a sample generation module, the machine learning model device further comprising a sample receiving module The data receiving module is connected to the feature conversion module, the feature conversion module is connected to the clustering module, the clustering module is connected to the similarity calculation module, the similarity calculation module is connected to the clustering merging module, the clustering merging module is connected to the abnormal data filtering module, the abnormal data filtering module is connected to the sample generation module, and the sample generation module is connected to the sample receiving module. The sample receiving module is connected to the model training module. The data analysis device converts the original data into feature data and then converts it into transformed data based on the feature space. The transformed data is classified into multiple data clusters using a cluster analysis algorithm. The similarity index array of each data cluster is calculated and then the data clusters are clustered and merged based on the first threshold. When the similarity index is less than the second threshold and When the number of data in a cluster is less than the total number of transformed data divided by the total number of clusters in the data cluster, the corresponding cluster is deleted. According to the specified ratio or the ratio of the number of samples to the number of data, the transformed data that meets the requirements in the data cluster is selected as sample data, and then provided to the machine learning model for learning and training, thereby achieving the technical effect of providing the machine learning model with an appropriate amount of accurate sample data.

Description

System for screening learning sample data and removing abnormal data for machine learning

A data screening and abnormal data filtering system, in particular, a data screening and abnormal data filtering system that provides a machine learning model with an appropriate amount of accurate sample data for learning and training.

Machine learning generally involves building a machine learning model and providing sample data to enable the machine to achieve learning effects. The accuracy and effectiveness of the machine learning model are often affected by the amount and precision of the sample data. Therefore, in order to improve the accuracy and effectiveness of the machine learning model, providing an appropriate amount of accurate sample data will be the development goal.

In order to achieve the above technical objectives, the existing technology has proposed to classify the data to be processed into multiple data clusters, calculate the similarity index between the data centers of different data clusters, set a threshold and then merge the data clusters according to the similarity index, and then select the data that meets the requirements in the data cluster as sample data by specifying a ratio or a ratio of the number of samples to the number of data. However, the existing technology ignores that if a data cluster is an abnormal data cluster, the accuracy of the sample data finally provided will be problematic. Therefore, this work will propose the technical content of abnormal data filtering to solve the technical problems that the existing technology cannot solve.

In summary, it can be seen that the prior art has long had the problem that the data in the existing data clustering and merging process has not considered the abnormal situation of filtering out the data clustering, resulting in reduced accuracy of the sample data. Therefore, it is necessary to propose improved technical means to solve this problem.

In view of the fact that the existing data clustering and merging process in the prior art does not consider filtering out data clusters as abnormal situations, which leads to reduced accuracy of sample data, this invention discloses a system for learning sample data screening and abnormal data screening for machine learning, wherein:

The system disclosed in this invention for machine learning for learning sample data screening and abnormal data screening includes: a data analysis device and a machine learning model device. The data analysis device further includes: a data receiving module, a feature conversion module, a clustering module, a similarity calculation module, a clustering merging module, an abnormal data screening module and a sample generation module. The machine learning model device further includes: a sample receiving module and a model training module.

The data receiving module of the data analysis device receives the original data; the feature conversion module of the data analysis device uses the feature analysis algorithm to convert the original data into feature data based on the data features; the clustering module of the data analysis device converts the feature data into converted data based on the feature space, and uses the clustering analysis algorithm to classify the converted data into multiple data clusters with a set number of clusters, and sets a corresponding cluster label for each data cluster; The similarity calculation module calculates the similarity index from the data center of each data cluster to the data center of other data clusters according to the cluster label to generate a similarity index array, and calculates the similarity index from the data center of the merged cluster to the data center of other data clusters according to the cluster label to generate a merged similarity index array; the cluster merging module of the data analysis device repeatedly selects the largest similarity index in the similarity index array and the selected similarity index is greater than the pre-set similarity index. The first threshold is set, and the conversion data of the two data clusters corresponding to the selected similarity index are merged into a merged cluster until all similarity indexes in the merged similarity index array are less than the threshold. The cluster label of the merged cluster is the cluster label with more conversion data of the two data clusters. The abnormal data filtering module of the data analysis device is when the similarity index in the merged similarity index array is less than the second threshold set in advance and the similarity index corresponds to the merged cluster. When the number of data items in a cluster is less than the total number of transformed data items divided by the total number of clusters in the data clustering, the cluster corresponding to the similarity index in the merged similarity index array is deleted as the abnormal screening similarity index array; and the sample generation module of the data analysis device selects the transformed data of the merged cluster or the data cluster close to the data center in the merged cluster or the data cluster as sample data by using the specified ratio of the merged cluster or the data cluster in the abnormal screening similarity index array or the ratio of the sample number to the data number.

The sample receiving module of the machine learning model device receives sample data from the data analysis device; and the model training module of the machine learning model device learns and trains the machine learning model according to the sample data.

The system disclosed in the present invention is as described above. The data analysis device converts the original data into feature data and then converts it into transformed data based on the feature space. The transformed data is classified into multiple data clusters using a cluster analysis algorithm. The similarity index array of each data cluster is calculated and then the data clusters are clustered and merged based on a first threshold. When the similarity index is less than a second threshold and the number of data items in the cluster is less than the total number of transformed data items divided by the total number of clusters of the data clusters, the corresponding cluster is deleted. The transformed data that meets the requirements in the data cluster is screened out as sample data according to a specified ratio or a ratio of the number of samples to the number of data, and then provided to the machine learning model for learning and training.

Through the above-mentioned technical means, this work can achieve the technical effect of providing an appropriate amount of accurate sample data for the machine learning model.

The following will be used in conjunction with diagrams and embodiments to explain in detail the implementation of this invention, so that the invention can fully understand how to apply technical means to solve technical problems and achieve technical effects and implement them accordingly.

The following first explains the system for filtering learning sample data and removing abnormal data for machine learning disclosed in the present invention, and please refer to "Figure 1", which is a system block diagram of the system for filtering learning sample data and removing abnormal data for machine learning disclosed in the present invention.

The learning sample data screening system for the machine learning model disclosed in this invention includes: a data analysis device 10 and a machine learning model device 20. The data analysis device further includes: a data receiving module 11, a feature conversion module 12, a clustering module 13, a similarity calculation module 14, a clustering merging module 15, an abnormal data screening module 16 and a sample generation module 17. The machine learning model device 20 further includes: a sample receiving module 21 and a model training module 22.

The data analysis device 10 obtains the original data from the data receiving module 11. It is worth noting that the data types of the original data include but are not limited to time series data, categorical data, image data, etc. These are only given as examples and do not limit the scope of application of this invention.

The feature conversion module 12 captures the original data with a fixed unit data capture area, and then uses a feature analysis algorithm to convert the original data into feature data based on data features. It is worth noting that the aforementioned feature analysis algorithm includes integral transform algorithms such as Fast Fourier Transform (FFT), wavelet transform, etc., which are only used as examples to illustrate, and are not intended to limit the scope of application of this creation.

The clustering module 13 converts the feature data into transformed data based on the feature space, and classifies the transformed data into multiple data clusters using a clustering analysis algorithm with a set number of clusters (for example, 50, 100, etc., which is only used as an example here and does not limit the scope of application of the present invention). Each data cluster is set with a corresponding cluster label, and the cluster label is, for example: 0, 1, 2, 3, etc., which is only used as an example here and does not limit the scope of application of the present invention.

It is worth noting that the aforementioned clustering analysis algorithms include K-means clustering, quality threshold clustering, etc. They are only used as examples here and are not intended to limit the scope of application of this work.

Then, the similarity calculation module 14 calculates the similarity index (which can also be a homogeneity index) from the data center of each data cluster to the data center of other data clusters according to the cluster label to generate a similarity index array. It is worth noting that the measurement methods used for the similarity index include Euclidean distance, Manhattan distance, Mahalanobis distance, cosine similarity, Hamming distance, etc., which are only used as examples here and are not intended to limit the scope of application of this creation.

Then, the cluster merging module 15 selects the largest similarity index in the similarity index array and the selected similarity index is greater than a preset first threshold, and merges the transformed data of the two data clusters corresponding to the selected similarity index into a merged cluster. The cluster label of the merged cluster is the cluster label of the two data clusters with more transformed data. 14 calculates the similarity index from the data center of the merged cluster to the data center of other data clusters and generates a merged similarity index array according to the cluster label, and repeatedly uses the similarity calculation module 14 and the cluster merging module 15 to merge clusters and calculate and generate a merged similarity index array until all similarity indicators in the merged similarity index array are less than a threshold value.

Specifically, if the preset first threshold is "0.85", the number of transformed data of the data cluster with cluster label "2" is "55", and the number of transformed data of the data cluster with cluster label "9" is "70", the cluster merging module 15 selects the largest similarity index "0.95" in the similarity index array, and the two data clusters corresponding to the similarity index "0.95" are the data cluster with cluster label "2" and the data cluster with cluster label "9". Clustering, because the similarity index of "0.95" is greater than the first threshold of "0.85", the data cluster with cluster label "2" is merged into the data cluster with cluster label "9", the cluster label of the merged cluster is set to "9" and the number of transformed data of the merged cluster is "125", and then the similarity calculation module 14 calculates the similarity index from the data center of the merged cluster to the data center of other unselected data clusters according to the cluster labels to generate a merged similarity index array.

Then, when the similarity index in the merged similarity index array is less than a preset second threshold and the number of data items in the cluster corresponding to the similarity index is less than the total number of converted data items divided by the total number of clusters of the data clustering, the abnormal data filtering module 16 deletes the cluster corresponding to the similarity index in the merged similarity index array as the abnormal filtering similarity index array.

Specifically, assume that the similarity index from cluster label "11" to cluster label "6" in the merged similarity index array is "0.16" and the similarity index from cluster label "11" to cluster label "12" is "0.12", the total number of transformed data is "300", the total number of clusters of data clustering is "30", the number of data with cluster label "11" is "3", the preset second threshold is "0.25", and the similarity index is " 0.16" and the similarity index "0.12" are both less than the second threshold "0.25", and the number of data with the cluster label "11" is "3" which is less than "10 (i.e., 300/30, calculated by dividing the total number of transformed data "300" by the total number of clusters of data clustering "30")", the abnormal data filtering module 16 deletes the cluster corresponding to the cluster label "11" in the merged similarity index array as the abnormal filtering similarity index array.

Next, the sample generation module 17 selects the transformed data of the merged cluster or data cluster close to the data center in the merged cluster or data cluster as sample data by using the specified ratio of the merged cluster or data cluster in the abnormal filtering similarity index array or the ratio of the sample quantity to the data quantity.

Specifically, if only cluster label "6", cluster label "7" and cluster label "9" remain in the anomaly filtering similarity index array, the number of transformed data for the merged cluster with cluster label "6" is "303", the number of transformed data for the data cluster with cluster label "7" is "26", and the number of transformed data for the data cluster with cluster label "9" is "314".

When the specified ratio is "70%", 212 conversion data with cluster label "6" closest to the data center of the merged cluster will be screened out (i.e. 70% of 303 is 212), 18 conversion data with cluster label "7" closest to the data center of the data cluster will be screened out (i.e. 70% of 26 is 18), and 219 conversion data with cluster label "9" closest to the data center of the merged cluster will be screened out (i.e. 70% of 314 is 219). All the selected conversion data are sample data.

When the sample size is "200", the proportion of data with cluster label "6" is calculated to be "47%" (i.e. 303/(303+26+314)), the proportion of data with cluster label "7" is calculated to be "4%" (i.e. 26/(303+26+314)), and the proportion of data with cluster label "9" is calculated to be "49%" (i.e. 314/(303+26+314)).

According to the ratio of the number of data with cluster label "6" to "47%", 94 conversion data closest to the data center of the merged cluster are selected (that is, 47% of 200 is 94); according to the ratio of the number of data with cluster label "7" to "4%", 8 conversion data closest to the data center of the data cluster are selected (that is, 4% of 200 is 8); according to the ratio of the number of data with cluster label "9" to "49%", 98 conversion data closest to the data center of the merged cluster are selected (that is, 49% of 200 is 98). All the selected conversion data are sample data.

The sample receiving module 21 of the machine learning model device 20 receives sample data from the data analysis device 10, and then the model training module 22 of the machine learning model device 20 learns and trains the machine learning model according to the sample data.

The first and second embodiments below use the k-means clustering method to cluster data into corresponding cluster labels. The data center of the data cluster is calculated by using the weighted average method or the simple average method to calculate the transformed data of the data cluster. This is only used as an example to illustrate, and does not limit the scope of application of this invention. Based on the k-means clustering method, the similarity index between the data center of each data cluster and the data center of other data clusters can be calculated using the following formula:

Similarity index =

in, For each transformed data of a data cluster, For each transformed data in the other data cluster, is the data center of a data cluster. The center of the data where the other data are clustered.

The following uses the first embodiment to illustrate the accuracy of the model prediction. Please refer to "Figure 2A" and "Figure 2B". "Figure 2A" shows the cluster scatter diagram of the first embodiment of the present invention without filtering out the abnormal data clustering; "Figure 2B" shows the confusion matrix diagram of the first embodiment of the present invention without filtering out the abnormal data clustering.

After clustering 300 normal original data, 3 abnormal data were randomly placed in the cluster, and the decision tree model was used. Then, 75 normal test data were clustered. The accuracy of the predicted clustering using the decision tree model was calculated to be 92%. In "Figure 2A", the overlap of the original data 31 and the predicted data 32 indicates that the prediction is correct. In "Figure 2A", the non-overlap of the original data 31 and the predicted data 32 indicates that the prediction is incorrect. The accuracy of the model prediction can be obtained from the confusion matrix 33 (as shown in "Figure 2B").

Please refer to "Figure 3A" and "Figure 3B", "Figure 3A" is a cluster scatter diagram of abnormal data clustering filtering according to the first embodiment of the present invention; "Figure 3B" is a confusion matrix diagram of abnormal data clustering filtering according to the first embodiment of the present invention.

After clustering 300 normal original data, the decision tree model was used to build the model. Then, 75 normal test data were clustered. The accuracy of the predicted clustering using the decision tree model was calculated to be 96%. In "Figure 3A", the overlap of the original data 31 and the predicted data 32 indicates that the prediction is correct. In "Figure 3A", the non-overlap of the original data 31 and the predicted data 32 indicates that the prediction is incorrect. The accuracy of the model prediction can be obtained from the confusion matrix 33 (as shown in "Figure 3B"). It can be clearly seen that the accuracy of the model prediction is improved after the abnormal data is clustered and filtered.

The following uses the second embodiment to illustrate the accuracy of the model prediction. Please refer to "Figure 4A" and "Figure 4B". "Figure 4A" shows the cluster scatter diagram of the unfiltered abnormal data clustering of the second embodiment of the present creation; "Figure 4B" shows the confusion matrix diagram of the unfiltered abnormal data clustering of the second embodiment of the present creation.

After clustering 300 normal original data, 3 abnormal data were randomly placed in the cluster, and the decision tree model was used. Then, 75 normal test data were clustered, and the accuracy of the predicted clustering using the decision tree model was calculated to be 48%. In "Figure 4A", the overlap of the original data 31 and the predicted data 32 indicates that the prediction is correct, and the non-overlap of the original data 31 and the predicted data 32 in "Figure 4A" indicates that the prediction is incorrect. The accuracy of the model prediction can be obtained from the confusion matrix 33 (as shown in "Figure 4B").

Please refer to “Figure 5A” and “Figure 5B”, “Figure 5A” is a cluster scatter diagram of abnormal data clustering filtering according to the second embodiment of the present invention; “Figure 5B” is a confusion matrix diagram of abnormal data clustering filtering according to the second embodiment of the present invention.

After clustering 300 normal original data, the decision tree model was used to build the model. Then, 75 normal test data were clustered. The accuracy of the predicted clustering using the decision tree model was calculated to be 54.7%. In "Figure 3A", the overlap of the original data 31 and the predicted data 32 indicates that the prediction is correct. In "Figure 5A", the non-overlap of the original data 31 and the predicted data 32 indicates that the prediction is incorrect. The accuracy of the model prediction can be obtained from the confusion matrix 33 (as shown in "Figure 5B"). It can be clearly seen that the accuracy of the model prediction is improved after the abnormal data is clustered and filtered.

Next, the operation process of the present invention will be described below, and please refer to "Figure 6A" and "Figure 6B" at the same time. "Figure 6A" and "Figure 6B" are flowcharts of the present invention for learning sample data screening and abnormal data screening for machine learning.

First, the data analysis device obtains the original data (step 401); then, the data analysis device uses the feature analysis algorithm to convert the original data into feature data based on data features (step 402); then, the data analysis device converts the feature data into transformed data based on the feature space, and uses the cluster analysis algorithm to classify the transformed data into multiple data clusters with a set number of clusters, and sets a corresponding cluster label for each data cluster (step 403); then, the data analysis device calculates the similarity index from the data center of each data cluster to the data center of other data clusters according to The cluster label is generated as a similarity index array (step 404); then, the data analysis device selects the largest similarity index in the similarity index array and the selected similarity index is greater than a preset first threshold, and the conversion data of the two data clusters corresponding to the selected similarity index are merged into a merged cluster, and the cluster label of the merged cluster is the cluster label with a larger amount of conversion data of the two data clusters (step 405); then, the data analysis device calculates the similarity index from the data center of the merged cluster to the data center of other unselected data clusters respectively, and generates a merged cluster according to the cluster label. similarity indicator array (step 406); then, the data analysis device repeatedly selects the largest similarity indicator in the merged similarity indicator array and the similarity indicator is greater than the first threshold, merges the merged clusters, and calculates and generates the merged similarity indicator array until all the similarity indicators in the merged similarity indicator array are less than the threshold (step 407); then, when the similarity indicator in the merged similarity indicator array is less than the preset second threshold and the number of data items in the merged cluster corresponding to the similarity indicator is less than the total number of converted data items divided by the total number of clusters in the data cluster, the data analysis device deletes the merged similarity indicator array. The cluster corresponding to the similarity index in the similarity index array is the anomaly-filtered similarity index array (step 408); then, the data analysis device selects the merged cluster or the data cluster in the anomaly-filtered similarity index array by a specified ratio or a ratio of the sample quantity to the data quantity to select the transformed data of the merged cluster or the data cluster close to the data center as sample data (step 409); then, the machine learning model device receives the sample data from the data analysis device (step 410); finally, the machine learning model device learns and trains the machine learning model according to the sample data (step 411).

The data analysis device 10 and the machine learning model device 20 are different forms of computing devices, which are only used as examples for illustration and are not intended to limit the scope of application of the present invention. Please refer to “Figure 7”, which is a schematic diagram of the components of the computing device proposed in the present invention.

The computing device mentioned in the present invention includes but is not limited to hardware components such as one or more processors 501, one or more memory modules 502, and a bus 503, wherein the bus 503 can connect different hardware components. Through the multiple hardware components included, the computing device can load and execute an operating system, so that the operating system runs on the computing device, and can also execute software or programs. The computing device also includes a housing 509, and the above-mentioned hardware components are arranged in the housing.

The bus 503 of the computing device of the present invention may include one or more types, such as a data bus, an address bus, a control bus, an expansion bus, and/or a local bus. The bus of the computing device includes but is not limited to a parallel Industrial Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, a Video Electronics Standards Association (VESA) local bus, a serial Universal Serial Bus (USB), a Peripheral Component Interconnect Express (PCI-E) bus, etc.

The processor 501 of the computing device proposed in the present invention is coupled to a bus 503. The processor 501 includes a register group or a register space, which can be completely set on the processing chip, or completely or partially set outside the processing chip and coupled to the processor via a dedicated electrical connection and/or via a bus. The processor 501 can be a processing unit, a microprocessor, or any suitable processing element. If the computing device is a multi-processor device, that is, the computing device includes multiple processors, the processors included in the computing device are the same or similar, and are coupled and communicated through a bus. The processor 501 can interpret a series of multiple instructions to perform specific calculations or operations, such as mathematical operations, logical operations, data comparison, copying/moving data, etc., so as to run an operating system or execute various programs, modules, and/or components.

The processor 501 of the computing device can be coupled to the chipset or electrically connected to the chipset through the bus 503. The chipset is composed of one or more integrated circuits (ICs), including a memory controller and a peripheral input/output (I/O) controller, that is, the memory controller and the peripheral input/output (I/O) controller can be included in one integrated circuit, or can be implemented using two or more integrated circuits. The chipset usually provides input/output and memory management functions, as well as multiple general and/or dedicated registers, timers, etc., wherein the above-mentioned general and/or dedicated registers and timers can be accessed or used by one or more processors coupled or electrically connected to the chipset.

The processor 501 of the computing device can also access the data in the memory module 502 and the mass storage area installed on the computing device through the memory controller. The above-mentioned memory module 502 includes any type of volatile memory and/or non-volatile memory (NVRAM) memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), flash memory (Flash), read-only memory (ROM), etc. The mass storage area mentioned above may include any type of storage device or storage media, such as a hard drive, an optical disk, a flash memory, a memory card, a solid state disk (SSD), or any other storage device. In other words, the memory controller can access data in static random access memory, dynamic random access memory, flash memory, hard drive, and solid state disk.

The processor 501 of the computing device can also be connected and communicated with peripheral devices or interfaces such as peripheral output devices, peripheral input devices, communication interfaces, and GPS receivers through the peripheral input/output controller via the bus 503. The peripheral input device can be any type of input device, such as a keyboard, a mouse, a trackball, a touchpad, a joystick, etc. The peripheral output device can be any type of output device, such as a display, a printer, etc. The peripheral input device and the peripheral output device can also be the same device, such as a touch screen, etc. The communication interface may include a wireless communication interface and/or a wired communication interface. The wireless communication interface may include an interface supporting wireless local area networks such as Wi-Fi, Zigbee, Bluetooth, infrared, near field communication (NFC), mobile communication networks such as 3G/4G/5G, or other wireless data transmission protocols. The wired communication interface may be an Ethernet device, an ATM device, a DSL modem, a cable modem, etc. The processor 501 may periodically poll various peripheral devices and interfaces so that the computing device can input and output data through various peripheral devices and interfaces, and can also communicate with another computing device having the components described above.

The modules mentioned in the data analysis device 10 and the machine learning model device 20 are usually generated after the processor 501 in the respective computing device executes a specific program loaded into the memory module 502, or are included in the processor 501.

In summary, the data analysis device converts the original data into feature data and then converts it into transformed data based on the feature space. The transformed data is classified into multiple data clusters using a cluster analysis algorithm. The similarity index array of each data cluster is calculated and then the data clusters are clustered and merged based on a first threshold. When the similarity index is less than a second threshold and the number of data items in the cluster is less than the total number of transformed data items divided by the total number of clusters in the data cluster, the corresponding cluster is deleted. The transformed data that meets the requirements in the data cluster is screened out as sample data according to a specified ratio or a ratio of the number of samples to the number of data, and then provided to the machine learning model for learning and training.

This technical means can be used to solve the problem of previous technologies that the accuracy of sample data is reduced due to the failure to consider the abnormal data in the existing data clustering and merging process, thereby achieving the technical effect of providing a proper amount of accurate sample data for the machine learning model.

Although the implementation methods disclosed in this work are as above, the content described is not used to directly limit the scope of patent protection of this work. Anyone with ordinary knowledge in the technical field to which this work belongs can make slight changes in the form and details of implementation without departing from the spirit and scope disclosed in this work. The scope of patent protection of this work shall still be defined by the scope of the attached patent application.

10: Data analysis device 11: Data receiving module 12: Feature conversion module 13: Clustering module 14: Similarity calculation module 15: Clustering merging module 16: Abnormal data filtering module 17: Sample generation module 20: Machine learning model device 21: Sample receiving module 22: Model training module 31: Original data 32: Prediction data 33: Confusion matrix 401-411: Steps 501: Processor 502: Memory module 503: Bus 509: Housing

FIG. 1 shows a system block diagram of the system for filtering learning sample data and filtering abnormal data for machine learning of the present invention. FIG. 2A shows a cluster scatter diagram of abnormal data clustering without filtering of the first embodiment of the present invention. FIG. 2B shows a confusion matrix diagram of abnormal data clustering without filtering of the first embodiment of the present invention. FIG. 3A shows a cluster scatter diagram of abnormal data clustering after filtering of the first embodiment of the present invention. FIG. 3B shows a confusion matrix diagram of abnormal data clustering after filtering of the first embodiment of the present invention. FIG. 4A shows a cluster scatter diagram of abnormal data clustering without filtering of the second embodiment of the present invention. FIG. 4B shows a confusion matrix diagram of the abnormal data clustering without filtering in the second embodiment of the present invention. FIG. 5A shows a cluster scatter diagram of the abnormal data clustering filtering in the second embodiment of the present invention. FIG. 5B shows a confusion matrix diagram of the abnormal data clustering filtering in the second embodiment of the present invention. FIG. 6A and FIG. 6B show a flow chart of the learning sample data filtering and abnormal data filtering for machine learning in the present invention. FIG. 7 shows a schematic diagram of the components of the computing device proposed in the present invention.

10: Data analysis device

11: Data receiving module

12: Feature conversion module

13: Clustering module

14: Similarity calculation module

15: Clustering and merging module

16: Abnormal data filtering module

17: Sample generation module

20: Machine learning model device

21: Sample receiving module

22: Model training module

Claims (5)

A system for filtering learning sample data and filtering abnormal data for machine learning, comprising: A data analysis device, the data analysis device further comprising: A data receiving module, receiving an original data; A feature conversion module, using a feature analysis algorithm to convert the original data into a feature data based on data features; A clustering module, converting the feature data into a converted data based on a feature space, and using a clustering analysis algorithm to classify the converted data into a plurality of data clusters with a set number of clusters, and setting a corresponding cluster label for each data cluster; A similarity calculation module, which calculates the similarity index from the data center of each data cluster to the data center of other data clusters according to the cluster label to generate a similarity index array, and calculates the similarity index from the data center of a merged cluster to the data center of other data clusters according to the cluster label to generate a merged similarity index array; A cluster merging module, which repeatedly selects the largest similarity index in the similarity index array and the selected similarity index is greater than a preset first threshold, and merges the conversion data of the two data clusters corresponding to the selected similarity index into the merged cluster until all similarity indexes in the merged similarity index array are less than the threshold, and the cluster label of the merged cluster is the cluster label of the two data clusters with more conversion data; An abnormal data filtering module, when the similarity index in the merged similarity index array is less than a preset second threshold and the number of data items in the merged cluster corresponding to the similarity index is less than the total number of converted data items divided by the total number of clusters in the data cluster, deletes the cluster corresponding to the similarity index in the merged similarity index array as an abnormal filtered similarity index array; and A sample generation module, selects the merged cluster or the converted data close to the data center in the data cluster as a sample data according to the specified ratio of the merged cluster or the data cluster in the abnormal filtered similarity index array or the ratio of the sample number to the data number; and A machine learning model device, the machine learning model device further comprises: a sample receiving module, receiving the sample data from the data analysis device; and a model training module, training a machine learning model based on the sample data. A system for filtering learning sample data and removing abnormal data for machine learning as described in claim 1, wherein the feature analysis algorithm includes an integral transform algorithm of Fast Fourier Transform (FFT) and wavelet transform. A system for filtering learning sample data and removing abnormal data for machine learning as described in claim 1, wherein the clustering analysis algorithm includes K-means clustering and quality threshold clustering. A system for filtering learning sample data and removing abnormal data for machine learning as described in claim 1, wherein the similarity index is measured by Euclidean distance, Manhattan distance, Mahalanobis distance, cosine similarity and Hamming distance. The system for selecting learning sample data and removing abnormal data for machine learning as described in claim 1, wherein the similarity index is Calculated, among which, For each transformed data of a data cluster, For each transformed data in the other data cluster, is the data center of a data cluster. The center of the data where the other data are clustered.