TWI845355B - System for judging input mode of form data - Google Patents

Abstract

A system for judging input mode of form data is configured to extract a study-use field information and a study-use volume of time stamp from each form data having a ground truth of being manually input or automatically input to according execute a learning calculation to generate a judgment calculation model; further to extract an under-judged field information and a under-judged volume of time stamp from each under-judged form data without the ground truth to accordingly generate a judgment result for predicting that the under-judged form data is manually input or automatically input; and further to define abnormal judgment form data and a trace back ground truth to re-execute the learning calculation to revise the judgment calculation model when the judgment result does not comply with a feedback ground truth.

Description

Form data input method judgment system

The present invention relates to a judgment system, and more particularly to a judgment system for judging whether the input method of form data is manual input or automatic input.

In response to the needs of organizational management, system management, and information integration management, more and more form data needs to be sent to the data storage device (server) of the data management center for centralized management, so as to serve as the data basis for analyzing specific things. Some of these form data are generated by manual input methods such as typing, clicking on options, or handwriting combined with image recognition technology; other parts may be generated by automatic input methods such as reading barcodes, chip sensing, label sensing, image recognition, or automatic system import or import.

In order to accurately judge and analyze the meaning of all the form data, accurate statistical analysis must be performed through big data calculations. The premise of this approach is that the content of the form data itself must have extremely high accuracy to avoid misjudging the true meaning of the form data. However, the error rate when entering data manually by typing, writing or clicking is often much higher than the error rate when entering data automatically by reading barcodes, chip sensing, label sensing, image recognition or system automatic import or import.

Due to the huge amount of form data, it is impractical for the staff of the data management center to check and confirm whether the form data content is correct one by one. It can only be assisted by auxiliary verification tool software. Even so, due to the high diversity of form data, it is impossible to develop corresponding verification tool software for all form data to conduct verification one by one.

If form data can be automatically generated, the workload of data management center staff can be greatly reduced. Therefore, it is imperative to promote the automatic input of form data. However, due to various usage scenarios and usage requirements, not all forms can be converted to automatic data input in a short period of time, so it is inevitable that some form data will still be generated by manual data input.

Since the form data stored in the data storage device (server) of the data management center is mixed with some form data generated by automatic input and some form data generated by manual input, it is necessary to develop a new judgment technology to judge which form data is generated by automatic input and which is generated by manual input. In this way, more inspection resources can be invested in the inspection of form data generated by manual input, so as to improve the overall accuracy of form data.

In view of the fact that in the prior art, there is a lack of judgment technology for judging whether form data is automatically input or manually input, and therefore it is impossible to concentrate inspection resources on the form data generated by manually input data, resulting in the difficulty in improving the overall accuracy of the form data. One of the necessary technical means adopted by the present invention to solve the problems of the prior art is to provide a form data input method judgment system (hereinafter referred to as "judgment system"), and the judgment system includes a data storage device and a judgment device.

The data storage device stores a plurality of reference truth form data and a plurality of to-be-judged form data, and the reference truth form data correspondingly have a plurality of initial reference truths defined as automatically input or manually input. The judgment device is communicatively connected to the data storage device to capture the reference truth form data and the to-be-judged form data, and generates a feature capture module, a supervised learning module, a judgment module, and a verification alarm module after installing and executing a judgment program.

The feature extraction module extracts a learning field information quantity and a learning timestamp quantity reflecting the data disorder from each reference truth form data, so that the reference truth form data has a corresponding plurality of the above-mentioned learning field information quantities and a plurality of the above-mentioned learning timestamp quantities, and extracts a judgment field information quantity and a judgment timestamp quantity reflecting the data disorder from each form data to be judged.

The supervised learning module performs a learning calculation based on the learning field information amount and the learning time stamp amount corresponding to the benchmark truth table data and the initial benchmark truth to generate a judgment calculation model.

The judgment module judges each form data to be judged based on the judgment calculation model and the judgment field information amount and judgment timestamp amount corresponding to each form data to be judged to generate a judgment result for predicting whether each form data to be judged is automatically input or manually input, thereby generating multiple judgment results.

The verification alarm module receives a plurality of feedback reference truths for defining the form data to be judged as automatically input or manually input, and issues a warning message when it is verified that the judgment result corresponding to the form data to be judged is inconsistent with the feedback reference truth, thereby defining a judgment abnormal form data and a confirmed reference truth. The verification alarm module further stores the judgment abnormal form data and the confirmed reference truth in the data storage device as the reference truth form data and the initial reference truth, so as to provide the supervised learning module with a new learning calculation to modify the judgment calculation model.

Among the subsidiary technical means derived from the above necessary technical means, preferably, the judgment system further includes a plurality of data input terminal devices, and the reference truth form data and the form data to be judged can be transmitted from the data input terminal device to the data storage device for storage. The data storage device can be a data storage server, and the judgment device can be a computing server. Each data input terminal device further includes a feedback operation interface, so that an operator of each data input terminal device can input the feedback reference truth accordingly when verifying that the judgment result corresponding to the form data to be judged is wrong.

Preferably, the verification alarm module further includes a verification cycle setting interface for setting a verification cycle, so as to periodically determine abnormal form data and confirm the baseline truth as the baseline truth form data and the initial baseline truth according to the verification cycle, so that the supervised learning module can periodically perform the learning calculation according to the verification cycle.

The feature extraction module may include a field information extraction unit, and the field information extraction unit operates according to a field entropy algorithm to obtain learning field information and judgment field information, and the field entropy algorithm is ,in , k represents the number of data field types, indicating that there are k types of data field types in total. represents the number of the i-th data field type among the k data field types, where i, k and All are natural numbers.

The feature extraction module may further include a timestamp quantity extraction unit, and the timestamp quantity extraction unit allows a user to specify q timestamp fields in p fields, and extracts learning timestamp quantities and judgment timestamp quantities according to a timestamp quantity algorithm, and the timestamp quantity algorithm is ,in Indicates the number of different data content types contained in the r columns of data corresponding to the jth timestamp field among the q timestamp fields, where j, , p, q and r are all natural numbers, and p＞q.

The judgment module may further include a marking unit, and the marking unit assigns an automatic input mark or a manual input mark to each form data to be judged according to the judgment result, and then stores it in the data storage device.

In addition, for the learning algorithm part, preferably, the learning algorithm may include at least one basic training algorithm, and the basic training algorithm includes at least one of the K-nearest neighbor algorithm (KNN), the support vector machine (SVM) algorithm, the decision tree algorithm and the regression algorithm. More preferably, the learning algorithm may further include at least one fitting algorithm, and the fitting algorithm includes at least one of the random forest algorithm and the extreme gradient boosting (XGBoost).

In summary, in the form data input method judgment system provided by the present invention, based on the long-term observation results of the correlation and regularity between the form data automatically input or manually input, the field information amount and timestamp amount related to time and data chaos are specially selected as the important feature basis for subsequent learning training and judgment. Based on this, supervised learning training is carried out to establish a judgment calculation model with a higher confidence level and obtain a judgment result with a higher judgment accuracy in a short time.

Furthermore, by periodically conducting judgment, verification, warning, and generating a basis for verification, the erroneous judgment results can be corrected, and the judgment calculation model can be modified by relearning the calculation. In this way, not only can the effect of automatically judging the input method be achieved, but also the judgment accuracy can be greatly improved in a shorter period of time. After obtaining a judgment result with a higher judgment accuracy rate, the inspection resources (including personnel, equipment, and/or tool software, etc.) can be further concentrated on the inspection of the form data generated by manually input data, so as to further achieve the effect of improving the overall accuracy of the form data.

Since the form data input method determination system provided by the present invention can be widely used to determine whether the form data is generated by automatic input or manual input, its application level is quite broad, so it will not be described one by one here, and only a preferred embodiment is listed to be specifically explained, and this embodiment is only used to conveniently and clearly assist in explaining the purpose and effect of the embodiment of the present invention.

Please refer to the first figure, which is a functional block diagram of the form data input method determination system provided by the preferred embodiment of the present invention. As shown in the first figure, a form data input method determination system (hereinafter referred to as "determination system") 100 includes a data storage device 1, a determination device 2 and data input terminal devices 3a~3c.

The data storage device 1 may be a data storage server. The judgment device 2 may be a computing server. The data input terminal devices 3a~3c may be a terminal device for inputting form data such as a built-in computer of a work device, an industrial computer, a desktop computer, a laptop computer, a tablet computer or a smart phone, and each of them has a data input interface 31a~31c and a feedback operation interface 32a~32c. The data input interface 31a~31 and the feedback operation interface 32a~32 may be a program operation page after the data input terminal device 3a~3c executes a specific program, or may be a network operation page displayed on the data input terminal device 3a~3c after connecting to a web server.

The data storage device 1 stores a plurality of reference truth form data GF and a plurality of form data to be judged JF, and the reference truth form data GF and the form data to be judged JF can be input by at least one operator among the data input terminal devices 3a~3c using the data input interface 31a~31c, and then transmitted to the data storage device 1 for storage. The reference truth form data GF has a plurality of initial reference truths defined as automatically input (i.e., form data generated by automatically inputting data) or manually input (i.e., form data generated by manually inputting data). The so-called initial reference truth refers to the factual truth that has been verified and has a very high credibility before the form data to be judged JF is judged by the judgment system 100. The form data to be judged JF is the form data that the judgment system 100 judges to be automatically input or manually input.

For example, as shown in Table 1, data storage device 1 stores 10 reference truth table data, and the corresponding serial numbers are 0001 to 0010. The initial reference truth of the reference truth table data with serial numbers 0001 and 0005 is input by manual input, and the so-called manual input method can be input by typing, clicking on options, or handwriting combined with text recognition software. In order to facilitate personnel and software recognition and interpretation, the reference truth table data with serial numbers 0001 and 0005 can be given an "M" mark to represent manual input. The initial baseline truths of the remaining 8 baseline truth table data are input by automatic input, and the so-called automatic input method can be input by reading barcodes, chip sensing, label sensing, image recognition, or automatic system input or import. Similarly, the automatically input baseline truth table data can be given an "A" mark.

Table 1: List of benchmark truth table data Baseline truth table data serial number Initial Base Truth Mark 0001 Manual input M 0002 Automatic input A 0003 Automatic input A 0004 Automatic input A 0005 Manual input M 0006 Automatic input A 0007 Automatic input A 0008 Automatic input A 0009 Automatic input A 0010 Automatic input A

The judgment device 2 is communicatively connected to the data storage device 1 to capture the reference truth form data GF and the form data to be judged JF, and is installed with a judgment program JAP. After executing the judgment program JAP, a feature capture module 21, a supervised learning module 22, a judgment module 23 and a verification alarm module 24 are generated.

After long-term observation, the inventors found that there are common correlations and regularities between automatically input form data and manually input form data, including: 1. Automatic input can input a large amount of data in a short time, so the data input in a short time is mostly automatically input; 2. Data with high similarity or high repetition is mostly automatically input, and data with high dissimilarity or low repetition is mostly manually input; 3. Since form data is input based on fields, there is a regularity that multiple rows of data in the same field mostly have the same type of attributes (such as time, text, and numbers, etc.); and 4. The characteristics of data such as similarity, dissimilarity and repetition can be reflected through parameters or indicators related to data noise.

According to the above-mentioned various correlations observed and summarized by the inventors over a long period of time, in order to improve the judgment ability of the judgment device 2, it should be given priority to extract parameters or indicators related to time and data chaos as a basis for learning and training and subsequent judgment. Under the above premise, the feature extraction module 21 includes a field information quantity extraction unit 211 and a timestamp quantity extraction unit 212.

During the learning phase, the field information extraction unit 211 can extract a learning field information quantity reflecting the data chaos from each reference truth form data GF, and when judging whether each form data to be judged JF is automatically input or manually input (i.e., during the judgment phase), extract a judgment field information quantity reflecting the data chaos from the form data to be judged JF.

The field information extraction unit 212 can be operated according to a field entropy algorithm to obtain the learning field information and the judgment field information, and the field entropy algorithm is ,in , k represents the number of data field types, indicating that there are k types of data field types in total. represents the number of the i-th data field type among the k data field types, where i, k and are all natural numbers. Since the field entropy algorithm is a logarithmic function of probability, the probability must be less than 1, and the logarithm of a number less than 1 must be a negative value, so a negative sign must be added to restore it to a positive value.

In addition to the above-mentioned field entropy algorithm, the field information extraction unit 212 can also extract other parameters related to data chaos, such as data repetition rate (the number of fields with the same data or the total number of data or the proportion) or data similarity (the proportion of the same part of the data content) as field information for learning and field information for judgment.

The timestamp quantity acquisition unit 212 may provide a timestamp designation interface (not shown) for a user to designate q timestamp fields among p fields, and to acquire learning timestamp quantities according to a timestamp quantity algorithm during the learning phase, and to acquire judgment timestamp quantities according to the same timestamp quantity algorithm during the judgment phase. The timestamp quantity algorithm may be ,in Indicates the number of different data content types contained in the r columns of data corresponding to the jth timestamp field among the q timestamp fields, where j, , p, q and r are all natural numbers, and p＞q.

Although the data content of the timestamp field does not necessarily have to be the time itself, it is best to be related to time, such as a sequence number or serial number that can reflect the time sequence. In addition, although in this embodiment, the timestamp quantity algorithm takes the maximum value of the number of different data content types in all timestamp fields, in actual application, the arithmetic mean, median or mode of the number of different data content types in all timestamp fields can also be used as the timestamp quantity.

Regarding the extraction of field information (for learning or judgment), for example, as shown in Table 2, a form data contains 6 fields: sales date, brand, model, quantity, unit price and sales amount. Among them, sales date belongs to the time field; brand and model belong to the text field; quantity, unit price and sales amount belong to the number field. Therefore, there are 3 types of fields in total, namely time field, text field and number field, indicating that the above k is 3; the quantity of the time field is 1, indicating that is 1; the number of text fields is 2, indicating is 2; and the number of numeric fields is 3, indicating is 3. The probability of the first type of field (time type field) , which is equal to 1/6; The probability of the second field (text field) , which is equal to 2/6; The probability of the third field (numeric field) , equal to 3/6; Bring in the field entropy algorithm , the amount of field information can be obtained to be 0.4392473. When the form data is the above-mentioned benchmark truth form data GF, the amount of field information for learning is 0.4392473; when the form data is the above-mentioned to-be-judged form data JF, the amount of field information for judgment is 0.4392473.

Regarding the extraction of timestamp quantity (for learning or judgment), for example, as shown in Table 2, as mentioned above, there are 6 fields representing P equal to 6, and among these 6 fields, only the sales date is related to time. Therefore, the user can specify the "sales date" field as the timestamp field through a timestamp designation interface (not shown) provided by the timestamp quantity extraction unit 212. bit, indicating that q is 1. The timestamp field (the "Sales Date" field) in the table has 14 rows of data, indicating that r is 14. The only timestamp field (the "Sales Date" field) has 14 rows of data with only three different data contents: "March 26", "March 27" and "March 28", indicating that the number of different data content types is 3, that is, is 3. Because there is only one timestamp field, q is 1, so That is 3, indicating that the number of timestamps is 3. When this form data is the above-mentioned reference truth form data GF, it means that the number of timestamps for learning is 3; when this form data is the above-mentioned to-be-judged form data JF, it means that the number of timestamps for judgment is 3.

Table 2: Form data Sales Date brand Model quantity Unit Price Sales Amount March 26 Brand A G655 47 1280 60160 March 26 Brand B E222 87 1100 95700 March 26 Brand C V990 35 880 30800 March 26 Brand D R448 25 1430 35750 March 26 Brand B E224 72 1077 77544 March 27 Brand B E222 67 1100 73700 March 27 Brand A G655 90 1280 115200 March 27 Brand A G900 18 880 15840 March 27 Brand C V980 42 999 41958 March 28 Brand B E224 twenty four 1077 25848 March 28 Brand D U922 25 889 22225 March 28 Brand C V980 33 999 32967 March 28 Brand C V350 72 1372 98784 March 28 Brand D R448 36 1430 51480

Although in the above example, the six fields of sales date, brand, model, quantity, unit price and sales amount are arranged horizontally, in practice, the above fields may also be arranged vertically. When the fields are arranged vertically, the above r columns (arranged horizontally) of data can be replaced by r rows (arranged vertically) of data, and the feature extraction method used, including the extraction of field information (for learning or judgment) and the extraction of timestamp (for learning or judgment) is similar to the above description, except that the vertical and horizontal fields and the rows and columns are interchanged, which will not be elaborated below.

The supervised learning module 22 performs a learning calculation to generate a judgment calculation model based on the learning field information and learning time stamp corresponding to the benchmark truth table data GF and the initial benchmark truth (which can be represented by the automatic input label A or the manual input label M). For the learning calculation part, preferably, the learning calculation can include at least one basic training algorithm, and the basic training algorithm includes at least one of the K-nearest neighbor algorithm (KNN), support vector machine (SVM) algorithm, decision tree algorithm and regression algorithm. Preferably, the learning algorithm may include at least one fitting algorithm in addition to the basic training algorithm, and the fitting algorithm includes at least one of the Random Forest algorithm and the Extreme Gradient Boosting (XGBoost). The judgment algorithm model refers to a mathematical algorithm model automatically derived by performing the above-mentioned learning algorithm.

Since the above-mentioned learning algorithm techniques (including basic training algorithms and fitting algorithms) are currently quite mature algorithm techniques, anyone with general knowledge in the relevant fields can use the above-mentioned algorithms or their combinations to construct the above-mentioned judgment algorithm model, and will not be elaborated on below.

The judgment module 23 may include a judgment unit 231 and a marking unit 232. The judgment unit 231 judges each form data to be judged according to the judgment calculation model and the judgment field information amount and judgment timestamp amount corresponding to each form data to be judged to generate a judgment result for predicting whether each form data to be judged JF is automatically input or manually input, thereby generating a plurality of judgment results. The marking unit 232 assigns an automatic input mark A or a manual input mark M to each form data to be judged according to the judgment result, and then stores it in the data storage device 1.

Simply put, the judgment algorithm model is used to learn and train how to derive the initial benchmark truth based on the learning field information and the learning timestamp in the learning stage (which can be represented by the automatic input label A or the manual input label M), so as to derive and predict the judgment result based on the judgment field information and the judgment timestamp in the judgment stage (which can also be represented by the automatic input label A or the manual input label M).

For example, as shown in Table 3, the data storage device 1 also stores 10 pending form data JF, and the corresponding serial numbers are 1001~1010, so 10 judgment results will be generated after the judgment is completed. The judgment results of the pending form data with serial numbers 1001, 1003 and 0005 are input by manual input, so the marking unit 232 will assign the manual input mark M to the pending form data with serial numbers 1001, 1003 and 0005, and the judgment results of the remaining pending form data are automatically input, and the marking unit 232 will assign the automatic input mark A.

Table 3: Judgment results and markings of the form data to be judged The serial number of the form data to be determined Judgment results Mark 1001 Manual input M 1002 Automatic input A 1003 Manual input M 1004 Automatic input A 1005 Manual input M 1006 Automatic input A 1007 Automatic input A 1008 Automatic input A 1009 Automatic input A 1010 Automatic input A

After the judgment result is stored in the data storage device 1, the judgment device 2 can send a push signal to the data input terminal device 3a~3c, asking the operator of the data input terminal device 3a~3c to verify whether the judgment result is correct. In the data input terminal device 3a~3c, when it is verified that the judgment result corresponding to the form data to be judged is wrong, the feedback reference truth is inputted correspondingly using the feedback operation interface 32a~32c. At this time, the corresponding form data to be judged will be defined as a judgment abnormal form data. When it is verified that the judgment result corresponding to the form data to be judged is correct, the judgment correct information can be inputted. Both the feedback reference truth and the correct judgment information will be transmitted to the judgment device 2.

The verification alarm module 24 may include a verification unit 241, an alarm unit 242, a verification cycle setting interface 243, and a judgment accuracy calculation unit 244. When the judgment device 2 receives the feedback reference truth, the verification unit 241 will define the corresponding form data to be judged as a judgment abnormal form data, and determine that the judgment result does not match the feedback reference truth. At this time, the verification unit 241 will list the feedback reference truth as the confirmation reference truth corresponding to the judgment abnormal form data, and the alarm unit 242 will issue a judgment abnormality prompt message.

On the contrary, when the judgment device 2 receives the correct judgment information, the verification unit 241 will define the corresponding form data to be judged as a correct judgment form data, and determine that the judgment result is consistent with the feedback benchmark truth. At this time, the verification unit 241 will directly list the judgment result as the confirmed benchmark truth corresponding to the correct judgment form data. Then, the verification unit 241 can transmit the abnormal judgment form data and the corresponding confirmed benchmark truth to the data storage device 1 for storage, as the newly added benchmark truth form data and the initial benchmark truth, respectively, so as to provide the supervised learning module 22 with a new learning calculation to modify the judgment calculation model.

The verification cycle setting interface 243 can be further used to set a verification cycle, so that according to the verification cycle, the abnormal form data and the confirmed reference truth are periodically used as the reference truth form data and the initial reference truth, so that the supervised learning module can periodically perform learning calculations according to the verification cycle to periodically modify the judgment calculation model. The judgment accuracy calculation unit 244 will count the number of abnormal form data and correct form data defined in the judgment form data, so as to calculate the judgment accuracy of the judgment made during each verification cycle. The verification cycle can be determined based on the statistical cycle requirements for the accuracy of the judgment, the amount of form data, or other requirements. It can be set to verify once a day, once a week, once a month, once a quarter, etc.

For example, according to the judgment result of Table 3, as shown in Table 4, after the operator of the data input terminal device 3a~3c verifies the judgment result, it is found that the judgment result of the form data to be judged with serial number 1003 is manually input, but in fact the form data to be judged with serial number 1003 is automatically input. Therefore, the feedback operation interface 32a~32c can be used to One corresponding input should be the feedback standard truth of "automatic input". At this time, the verification unit 241 will define the form data to be judged with the serial number 1003 as the form data to be judged abnormally, and feedback the standard truth (automatic input) as the confirmation standard truth of the form data to be judged abnormally (i.e. the form data to be judged with the serial number 1003), and store it in the data storage device 1. The warning unit 242 will issue a judgment abnormality prompt message to prompt that the judgment result generated when the form data to be judged with the serial number 1003 is judged by using the currently established mathematical calculation model is wrong.

On the contrary, after the operator of the data input terminal device 3a~3c verifies the judgment result, if it is found that the judgment results of the remaining serial numbers of the form data to be judged are all correct, the correct judgment information can be input through the feedback operation interface 32a~32c. At this time, the verification unit 241 will directly define the remaining serial numbers of the form data to be judged as correct judgment form data, and directly list the corresponding judgment result as the verification benchmark truth, and also store it in the data storage device 1.

During each verification cycle, multiple verified pending form data JF and the corresponding confirmed baseline truths can be accumulated, and when entering the next verification cycle, part or all of the accumulated pending form data JF (such as abnormal form data, or both abnormal form data and correct form data) that have been verified and the corresponding confirmed baseline truths are used as newly added baseline truth form data and the corresponding initial baseline truth respectively.

The judgment accuracy calculation unit 244 will calculate that among the 10 pieces of form data to be judged, only 1 piece (i.e., the form data to be judged with serial number 1003) is judged as abnormal form data, and the remaining 9 pieces of form data to be judged are judged as correct form data. Based on this, the judgment accuracy calculation unit 244 can calculate that during this verification cycle, the judgment accuracy of the judgment system 100 in judging whether the form data to be judged JF is automatically input or manually input is 90%. At the same time, the judgment accuracy calculation unit 244 can also calculate the form data automation rate during this verification cycle. As shown in the results of the verification benchmark in Table 4, during this verification cycle, there are 8 automatically entered form data and 2 manually entered form data, which means that the form data automation rate during this verification cycle is 80%.

Table 4: Verification results of the judgment results The serial number of the form data to be determined Judgment results Feedback Baseline Truth Identify the baseline truth Correctness of judgment 1001 Manual input - Manual input correct 1002 Automatic input - Automatic input correct 1003 Manual input Automatic input Automatic input Mistake 1004 Automatic input - Automatic input correct 1005 Manual input - Manual input correct 1006 Automatic input - Automatic input correct 1007 Automatic input - Automatic input correct 1008 Automatic input - Automatic input correct 1009 Automatic input - Automatic input correct 1010 Automatic input - Automatic input correct

After multiple verification cycles of judgment, verification, and re-learning calculation to modify the judgment calculation model, the judgment accuracy of the judgment system 100 in judging whether the form data JF to be judged is automatically input or manually input can be gradually improved until the judgment accuracy is improved to a target accuracy (such as 99.99%) or above, indicating that the judgment ability of the judgment system 100 has reached a certain level of confidence. At this time, the verification cycle can be extended (for example, from once a quarter to once a year), and even the judgment result can be directly accepted, that is, the judgment result obtained by the judgment system 100 each time is directly regarded as the benchmark truth and is adopted without the need for subsequent verification.

Furthermore, inspection resources (including personnel, equipment and/or tool software, etc.) can be concentrated on inspecting form data generated by manually input data (i.e., form data with a manual input mark M) (especially the benchmark truth form data GF that has been verified to be manually input), and correcting errors in form data generated by manually input data to improve the overall accuracy of form data. In addition, the overall accuracy of form data can be efficiently improved without increasing the total workload of inspection (total inspection resources invested) by increasing the sampling rate of manually input form data and reducing the sampling rate of automatically input form data.

Since the above-mentioned feature capture module 21, supervised learning module 22, judgment module 23 and verification alarm module 24 are all generated by re-executing the judgment program JAP, the feature capture module 21, supervised learning module 22, judgment module 23 and verification alarm module 24 can essentially be (part of) the main program, sub-program of the judgment program JAP, or the program page or functional interface generated after executing the judgment program JAP. For example, anyone with general knowledge in the relevant technical field (especially the field of artificial intelligence algorithms) can use appropriate programming languages to write a judgment program JAP (including its main program or sub-program) having the functions of the above-mentioned feature capture module 21, supervised learning module 22, judgment module 23 and verification alarm module 24 according to the above-mentioned learning and judgment logic, so as to realize the above-mentioned various technologies of the present invention.

In summary, in the form data input method judgment system 100 provided by the present invention, based on the long-term observation results of the correlation and regularity between the form data automatically input or manually input, the field information amount and timestamp amount related to time and data chaos are specially selected as the important feature basis for subsequent learning training and judgment. Based on this, supervised learning training is carried out to establish a judgment calculation model with a higher confidence level and obtain a judgment result with a higher judgment accuracy in a short time.

Furthermore, by using the judgment device to perform judgment, verification, warning and generate the basic truth in a two-cycle manner, the erroneous judgment results can be corrected, and the judgment calculation model can be modified by re-learning calculation. In this way, not only can the effect of automatic judgment input method be achieved, but also the judgment accuracy can be greatly improved in a shorter time. After obtaining the judgment result with a higher judgment accuracy rate, the inspection resources (including personnel, equipment and/or tool software, etc.) can be further concentrated on the inspection of the form data generated by manually input data, so as to further achieve the effect of effectively improving the overall accuracy of the form data.

The above detailed description of the preferred specific embodiments is intended to more clearly describe the features and spirit of the present invention, but is not intended to limit the scope of the present invention by the preferred specific embodiments disclosed above. On the contrary, the purpose is to cover various changes and arrangements with equivalents within the scope of the patent application for the present invention.

100: Judgment system 1: Data storage device 2: Judgment device 21: Feature acquisition module 211: Field information acquisition unit 212: Timestamp acquisition unit 22: Supervised learning module 23: Judgment module 231: Judgment unit 232: Marking unit 24: Verification alarm module 241: Verification unit 242: Alarm unit 243: Verification cycle setting interface 244: Judgment accuracy calculation unit 3a~3c: Data input terminal device 31a~31c: Data input interface 32a~32c: Feedback operation interface GF: Benchmark truth form data JF: Form data to be judged JAP: Judgment program

The first figure is a functional block diagram showing the form data input method determination system provided by the preferred embodiment of the present invention.

100: Judgment system

1: Data storage device

2: Judgment device

21: Feature extraction module

211: Field information extraction unit

212: Timestamp quantity capture unit

22:Supervised Learning Module

23: Judgment module

231: Judgment unit

232: Marking unit

24: Verify the alarm module

241: Verification unit

242: Alarm unit

243: Verification cycle setting interface

244: Judgment accuracy calculation unit

3a~3c: Data input terminal device

31a~31c: Data input interface

32a~32c: Feedback operation interface

GF: Baseline truth table data

JF: Form data to be determined

JAP: Judgment Program

Claims (10)

A form data input mode judgment system includes: a data storage device storing a plurality of reference truth form data and a plurality of form data to be judged, wherein the reference truth form data correspondingly have a plurality of initial reference truths defined as automatic input or manual input; and a judgment device communicatingly connected to the data storage device to capture the reference truth form data and the form data to be judged, and generating after installing and executing a judgment program: A feature extraction module extracts a learning field information quantity and a learning time stamp quantity reflecting the data disorder from each of the reference truth form data, so that the reference truth form data has a corresponding plurality of the above-mentioned learning field information quantities and a plurality of the above-mentioned learning time stamp quantities, and extracts a judgment field information quantity and a judgment time stamp quantity reflecting the data disorder from each of the to-be-judged form data; A supervised learning module performs a learning calculation based on the learning field information and learning time stamp corresponding to the benchmark truth form data and one of the initial benchmark truths to generate a judgment calculation model; A judgment module judges each of the to-be-judged form data based on the judgment calculation model and the judgment field information and judgment time stamp corresponding to each of the to-be-judged form data to generate a judgment result for predicting whether each of the to-be-judged form data is automatically input or manually input, thereby generating a plurality of the above-mentioned judgment results; and A verification alarm module receives a plurality of feedback benchmark truths for defining the form data to be judged as automatically input or manually input, and issues a warning message when it is verified that one of the judgment results corresponding to one of the form data to be judged is inconsistent with one of the feedback benchmark truths, and defines a judgment abnormal form data and a confirmed benchmark truth accordingly, and stores the judgment abnormal form data and the confirmed benchmark truth in the data storage device as one of the benchmark truth form data and one of the initial benchmark truths, so as to allow the supervised learning module to re-perform the learning calculation and modify the judgment calculation model. The form data input method judgment system as described in claim 1 further includes a plurality of data input terminal devices, and the reference truth form data and the form data to be judged are transmitted from the data input terminal devices to the data storage device for storage. A form data input method judgment system as described in claim 2, wherein each of the data input terminal devices further includes a feedback operation interface for an operator of each of the data input terminal devices to input one of the feedback reference truths accordingly when verifying that one of the judgment results corresponding to one of the form data to be judged is erroneous. The form data input method judgment system as described in claim 3, wherein the verification alarm module further includes a verification cycle setting interface for setting a verification cycle, so as to periodically use the judged abnormal form data and the corresponding confirmed benchmark truth as one of the above-mentioned benchmark truth form data and one of the corresponding initial benchmark truths according to the verification cycle, so that the supervised learning module can periodically perform the learning calculation according to the verification cycle. The form data input method judgment system as described in claim 4, wherein the verification alarm module further includes a judgment accuracy calculation unit for counting the number of the form data to be judged that have completed the judgment during the verification cycle, and the number of the form data defined as abnormal, so as to calculate a judgment accuracy. A form data input method determination system as described in claim 1, wherein the data storage device is a data storage server and the determination device is a computing server. The form data input method judgment system as described in claim 1, wherein the feature extraction module further includes a field information extraction unit, and the field information extraction unit is operated according to a field entropy algorithm to obtain the learning field information and the judgment field information, and the field entropy algorithm is ,in , k represents the number of data field types, indicating that there are k types of data field types in total. represents the number of the i-th data field type among the k data field types, where i, k and All are natural numbers. The form data input method judgment system as described in claim 1, wherein the feature capture module further includes a timestamp quantity capture unit, and the timestamp quantity capture unit allows a user to specify q timestamp fields in p fields, and captures the learning timestamp quantity and the judgment timestamp quantity according to a timestamp quantity algorithm, and the timestamp quantity algorithm is ,in Indicates the number of different data content types contained in the r columns of data corresponding to the jth timestamp field among the q timestamp fields, where j, , p, q and r are all natural numbers, and p＞q. A form data input method judgment system as described in claim 1, wherein the judgment module further includes a marking unit, and the marking unit assigns an automatic input mark or a manual input mark to each of the form data to be judged based on the judgment result, and then stores it in the data storage device. A form data input method determination system as described in claim 1, wherein the learning algorithm performed by the supervised learning module includes at least one basic training algorithm and at least one fitting algorithm, the at least one basic training algorithm includes at least one of a K-nearest neighbor algorithm (KNN), a support vector machine (SVM) algorithm, a decision tree algorithm, and a regression algorithm, and the at least one fitting algorithm includes at least one of a random forest algorithm and an extreme gradient boosting (XGBoost).

Images