CN113919812A - Method and device for checking crowd queue research data - Google Patents

Abstract

The application provides a method and a device for checking crowd queue research data, wherein the method comprises the following steps: the method comprises the steps of conducting first checking on fields in a data table to be checked, if the fields all accord with preset first checking conditions, selecting any credible field in the data table to be checked as the first field, inquiring related fields of the first field and establishing a related field set, conducting correlation checking on data items in the related field set according to the correlation checking conditions of the related field set, marking the data items which do not accord with the correlation checking conditions as potential abnormal data, updating the correlation checking conditions of the related field set when the number or the proportion of the potential abnormal data which are marked as normal reaches preset threshold values, and conducting correlation checking on the data items to be checked in the related field set based on the updated correlation checking conditions. The method and the device for checking the crowd queue research data can effectively solve the problem that the checking method in the prior art is poor in accuracy.

Description

A method and device for verifying data in a population cohort study

technical field

The present application relates to the technical field of medical informatization, and in particular to a method and device for verifying data in a population cohort study.

Background technique

With the development of medical information technology, a large amount of medical data is continuously generated in the medical field, such as population cohort study data corresponding to different diseases. Therefore, it is necessary to check the population cohort study data to exclude or correct abnormal data, so as to Guarantee the accuracy and availability of data.

In the prior art, the processing method of separately performing abnormal data verification on each field in the population cohort research data table is mostly adopted, but the verification accuracy provided by this method is poor.

SUMMARY OF THE INVENTION

In view of this, the present application provides a method and device for verifying data of a population cohort study, which can effectively solve the problem of poor accuracy of the method for verifying data of a population cohort study in the prior art.

The following will give a brief overview of the application in order to provide a basic understanding of certain aspects of the application. It should be understood that this summary is not an exhaustive overview of the present application. It is not intended to identify a critical or essential part of this application, nor is it intended to limit the scope of the application. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.

According to the first aspect of the present application, a method for checking data of a population cohort study is provided, including:

Step 1: perform a first check on the fields in the data table to be checked, and if the fields all meet the preset first check conditions, select any trusted field in the data table to be checked as the first field;

Step 2: query the associated field of the first field and generate an associated field set based on the first field and the associated field of the first field;

Step 3: According to the correlation verification conditions of the associated field set, perform correlation verification on the data items in the associated field set, if the data items in the associated field set do not meet the correlation verification conditions, then The above data entry is marked as potentially abnormal data;

Step 4: Judging whether the potential abnormal data is real data, if yes, then judging whether the potential abnormal data satisfies the associated conditions corresponding to the target disease, if so, marking the potential abnormal data as normal, and at the same time, judging Whether the number or proportion of potentially abnormal data marked as normal reaches a preset threshold, and if so, update the correlation verification condition of the associated field set based on the potential abnormal data marked as normal, and based on the updated correlation The verification condition is to perform correlation verification on the data items to be verified in the associated field set.

In some embodiments, the method for verifying the data of the population cohort study further includes collecting data corresponding to the target disease type and generating an initial data table, and selecting a plurality of pieces of data in the initial data table by sampling to generate the to-be-to-be-data table. Check the data sheet.

Further, if any field in the data table to be checked does not meet the preset first check condition, then re-collect the data of the corresponding field in the initial data table, and update the initial data based on the re-collected data. table, re-sampling and selecting a plurality of pieces of data in the initial data table to generate a data table to be checked.

In some embodiments, the first check is performed on the fields in the data table to be checked, including one of garbled code checking, data format checking, information consistency checking, null value checking, value range checking, and field length checking. or more.

In some embodiments, the querying the associated field of the first field includes: in one or more preset correlation verification conditions, querying the correlation verification condition related to the first field, and based on the correlation The sex validation condition determines the associated field of the first field.

In some embodiments, after judging that the potential abnormal data is real data, if it is determined that the potential abnormal data does not meet the association condition corresponding to the target disease, the association condition corresponding to the target disease is deleted. potential anomalous data.

In some embodiments, the correlation verification condition of the associated field set includes a value range matching between the first field and its corresponding associated fields, and the data entry in the associated field set is performed The correlation verification includes judging whether the values of the first field of the data entry and its corresponding associated field are within the corresponding value range.

Further, the updating the correlation verification condition of the associated field set includes updating the value range of one or more associated fields.

In some embodiments, the method for checking data of a population cohort study further includes repeating steps 2 to 4 for other fields in the data table to be checked except the first field, until the checking of the to-be-checked data table is completed. Traversal of trusted fields in the data table.

According to a second aspect of the present application, there is provided a verification device for population cohort study data, comprising,

The first verification unit is used to perform a first verification on the fields in the data table to be verified, and if the fields all meet the preset first verification conditions, select any trusted field in the data table to be verified as the first field ;

an associated field set generating unit, configured to query the associated field of the first field and generate an associated field set based on the first field and the associated field of the first field;

A correlation verification unit, configured to perform correlation verification on the data items in the correlation field set according to the correlation verification conditions of the correlation field set, if the data items in the correlation field set do not meet the correlation verification conditions , the data entry is marked as potentially abnormal data;

A correlation verification condition update unit, configured to judge whether the potential abnormal data is real data, and if so, judge whether the potential abnormal data satisfies the correlation conditions corresponding to the target disease, and if so, mark the potential abnormal data is normal, and at the same time, determine whether the number or proportion of potential abnormal data marked as normal reaches a preset threshold, and if so, update the correlation verification condition of the associated field set based on the potential abnormal data marked as normal, and Based on the updated correlation verification conditions, correlation verification is performed on the data items to be verified in the associated field set.

According to a third aspect of the present application, an electronic device is provided, comprising:

one or more processors;

memory for storing one or more programs,

Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to the first aspect.

According to a fourth aspect of the present application, there is provided a computer-readable medium having executable instructions stored thereon, the instructions, when executed by a processor, cause the processor to perform the method according to the first aspect.

The present application proposes a method and device for verifying data in a population cohort study. By constructing an associated field set and verifying the correlation of the data in the associated field set, it effectively solves the problem that the existing population cohort study data verification method has poor verification accuracy. The problem.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present application. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort. The above and other objects, features and advantages of the present application will be more apparent from the accompanying drawings. The same reference numerals refer to the same parts throughout the drawings. The drawings are not intentionally scaled to actual size, and the emphasis is on illustrating the subject matter of the present application.

FIG. 1 is a schematic flowchart of a method for checking data of a population cohort study provided according to an embodiment of the present application.

FIG. 2 is a two-dimensional coordinate diagram of a method for checking data of a population cohort study provided according to an embodiment of the present application.

FIG. 3 is a system structure diagram of an apparatus for checking data of a population cohort study provided according to an embodiment of the present application.

FIG. 4 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures. Meanwhile, in the description of this application, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Furthermore, the term "and/or" in this application is only an association relationship to describe related objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, and A and B exist at the same time. B, there are three cases of B alone.

Hereinafter, exemplary embodiments of the present application will be described with reference to the accompanying drawings. In the interest of clarity and conciseness, not all features of an actual embodiment are described in the specification. It should be appreciated, however, that many embodiment-specific decisions may be made in the development of any such actual embodiment in order to achieve the developer's specific goals, and that these decisions may vary from embodiment to embodiment .

Here, it should also be noted that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the solution according to the present application is shown in the drawings, and the relationship with the present application is omitted. Not much other details.

It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. Herein, where feasible, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment.

The methods for checking population cohort study data provided in the prior art generally check whether the data of a certain field in the data table is a null value and whether the data value is within a certain range. However, this processing method can only check the value in a single field, and does not perform correlation verification on the fields with associated relationships, so the processing accuracy is poor.

In order to solve the above problems, the embodiments of the present application provide a method and device for checking population cohort study data from the perspective of improving the processing accuracy. The following first specifically introduces a method for checking population cohort study data provided by the embodiments of the present application. .

FIG. 1 shows a schematic flowchart 100 of a method for checking population cohort study data provided according to an embodiment of the present application, and the method specifically includes:

Step 110: Perform a first check on the fields in the data table to be checked, and select any trusted field in the data table to be checked as the first field if the fields all meet the preset first check conditions.

In the embodiment of the present application, the data table to be checked may be multiple pieces of patient data corresponding to the target disease of the study, for example, may be multiple pieces of patient data corresponding to the disease of obesity. Table 1 provides an example of a data table to be checked, and it should be noted that this example only shows part of the data in the data table to be checked, and the illustration of other data items is omitted.

Table 1 - Data table to be verified based on data of obese patients

In the embodiment of the present application, the trusted field includes a field that can identify a patient and is obtained by machine parsing or extraction from relative inherent information, for example, the relative inherent information may be obtained when a patient visits a doctor by swiping an ID card or a medical card. Information, such as medical number, ID number, gender, etc., the trusted field may be a field obtained by machine parsing or extraction from the relative inherent information of the patient, such as age, gender, etc. Correspondingly, in the embodiment of the present application, the fields other than the trusted fields in the data table to be checked are defined as untrusted fields, and the untrusted fields are mainly used to represent parameters related to the patient's condition, such as body weight, blood sugar, etc. , waist circumference, etc. The untrusted fields can be entered manually or imported in batches by a computer. In the embodiment of the present application, the age field in Table 1 is a trusted field, and the weight field is an untrusted field.

In this embodiment of the present application, the fields in the data table to be checked can be checked first from the perspectives of whether there is garbled characters, data format, information consistency, whether there is a null value, whether the numerical range of the data in the field is reasonable, and whether the data length meets the requirements, etc. Correspondingly, in the embodiment of the present application, the first verification of the fields in the data table to be verified may include garbled code verification, data format verification, information consistency verification, null value rate verification, and value range verification on the data in the field. , one or more of Field Length Check.

In the embodiment of the present application, theoretically, the field extracted by the machine has high accuracy, and it is basically considered that there is no error, so it is determined as a trusted field. However, considering that there may still be problems such as garbled data, wrong data format, incomplete data, and inconsistency between the extracted information and inherent information in the process of machine extraction, specifically, it can be the trusted field in the data table to be checked. One or more of garbled code verification, data format verification, and information consistency verification are performed on the data in the data table; while the verification of the data in the untrusted fields in the data table to be verified can be null value rate verification, value range verification One or more of Check, Field Length Check.

In the embodiments of the present application, the first verification performed on the fields in the data table to be verified is not limited to the above-mentioned dimensions, and other dimensions that can be easily thought of by those skilled in the art to verify the fields are within the protection scope of the present application.

In the embodiment of the present application, for the first verification of the fields in the data table to be verified, a first verification condition is preset, and the first verification condition may be that the data in the corresponding field passes all or part of the verification items in the first verification. Check item. For example, when the first check includes three check items: garbled code check, data format check, and information consistency check, the first check condition may be that the data in the trusted field all pass the above three check items, that is, the current The data in the field has no garbled characters, the data format is uniform, and the data is consistent with the information extracted from the inherent information such as ID number; In the case of one verification item, the first verification condition may be that the data in the field all pass the above three verification items, that is, the data in the current field does not have a null value, the value range of the data in the field is reasonable, and the data length meets the requirements . In the embodiment of the present application, the first verification condition is not limited to the above situation, and those skilled in the art can set the first verification condition by themselves according to requirements.

Further, if all the fields meet the preset first verification condition, any trusted field in the data table to be verified is selected as the first field. For example, in this embodiment of the present application, the age field in Table 1 may be selected as the first field.

Step 120: Query the associated field of the first field and generate a set of associated fields based on the first field and the associated field of the first field.

In the embodiment of the present application, one or more correlation verification conditions that can verify the correlation of the correlation fields in the data table to be checked are preset in the server, and the correlation field of the first field in the query may be in In the preset correlation verification condition, the query involves the correlation verification condition of the first field, and the associated field of the first field is determined based on the correlation verification condition.

In the embodiment of the present application, the correlation verification condition may refer to the matching of the numerical range between the related fields including the first field and its corresponding associated field. For example, when the data of obese patients is collected, Since the judgment criteria for obesity include weight value ranges corresponding to different ages, in this embodiment of the present application, the correlation verification condition corresponding to the data of obese patients may be the matching of the value ranges between the two related fields of age and weight. In the examples of the present application, it is assumed that the specific correlation verification conditions are shown in Table 2, and Table 2 gives the lower limit value and upper limit value of weight corresponding to obese patients of different ages. It should be noted that the above-mentioned correlation verification conditions for obese patient data may also include the matching of value ranges between other related fields, such as gender, age, and waist circumference.

Table 2 - Correlation verification conditions provided by the examples of this application

In the embodiment of the present application, when the trusted field is age, the correlation verification conditions including age and weight are obtained by querying the correlation verification conditions involving the age field, and then it is determined that the correlation field of the age field is weight field, and an associated field set may be generated based on the age field and the weight field.

Obviously abnormal data can be screened out through the first check of the fields in the data table to be checked. This method is usually used for checking in the prior art, but only this method is used for checking without considering the relationship between different fields, so it cannot be screened out. All abnormal data. For example, range checking, often based on empirical data, employs a universal range of values. Taking obesity as an example, when the corresponding range of obesity is ≥ 120 catties, all weight data of ≥ 120 catties can pass the verification. For actual obese patients, the corresponding weight ranges for different ages are Different, as shown in Table 2, the weight range of obesity corresponding to 42 years old is ≥125 catties. For a 42-year-old patient with a weight of 123 catties, the weight data can be checked through the range of the first verification. However, when judging by age, it does not conform to the weight range of obesity, which is abnormal data. Based on this, correlation verification can further ensure the accuracy of abnormal data screening.

In the embodiment of the present application, there may be a situation that the same trusted field involves one or more correlation verification conditions at the same time, and in this case, the corresponding generated set of associated fields is one or more. For example, the correlation verification condition for the obesity patient data table may include the matching of the numerical range between the two related fields of age and weight, and may also include the matching of the numerical range between the three related fields of age, gender, and waist circumference. Therefore, in this case, the generated associated field sets corresponding to the trusted field are two, namely {age, weight} and {age, gender, waistline} field sets.

Further, in this embodiment of the present application, after the associated field set is generated, the selected trusted field may be identified. By identifying the selected trusted field, after completing the correlation verification of the associated field set corresponding to the trusted field, it is possible to continue to select other trusted fields and perform correlation verification on the corresponding associated field set. , so as to avoid duplication when selecting trusted fields, thereby improving data processing efficiency.

In the embodiment of the present application, before step 110 of the method for verifying data of the population cohort study, the method may further include: collecting data corresponding to the target disease type of the study and generating an initial data table, where the columns in the initial data table correspond to From different fields of the data, a plurality of pieces of data in the initial data table are sampled to generate the data table to be checked.

In this embodiment of the present application, the patient data corresponding to the target disease can be searched from the server based on keywords, for example, the data of obese patients is searched based on "obesity" and similar keywords, and the searched obesity The patient's data is collected into a single table, thus forming the initial data table. Columns in the initial data table correspond to different fields of the data, for example, age, weight fields or fields of age, gender, weight, waist circumference, etc. corresponding to obese patients. Further, in order to improve the data processing efficiency, a plurality of pieces of data in the initial data table are sampled to generate the data table to be checked. In the embodiment of the present application, for example, a plurality of pieces of data of obese patients in the initial data table can be sampled to generate the data table to be checked. data sheet. It is worth noting that the patient data corresponding to each disease type can be stored in the server or in other storage media for the server to query and call.

In the embodiments of the present application, the acquisition of data corresponding to the target disease is not limited to the aforementioned method, and other methods that can easily obtain data corresponding to the target disease that can be easily conceived by those skilled in the art are within the scope of protection of the present application.

In the embodiment of the present application, in the above-mentioned process of performing the first verification on the fields in the data table to be verified, if any field in the data table to be verified does not meet the preset first verification conditions, the The data corresponding to the field in the initial data table is collected, the initial data table is updated based on the re-collected data, and multiple pieces of data in the initial data table are re-sampled to generate a data table to be checked. Here, re-collecting the data corresponding to the field in the initial data table may be collecting all data corresponding to the field, or only collecting the data corresponding to the check items in the field that do not pass the first check condition data.

Step 130: According to the correlation verification conditions of the associated field set, perform correlation verification on the data items in the associated field set, and if the data items in the associated field set do not meet the correlation verification conditions, then verify all the data items in the associated field set. The data entry described above is marked as potentially anomalous data.

In this embodiment of the present application, the data entry may correspond to a data row in the data table to be checked. After the associated field set is generated, since the associated field set is a collection of multiple field data in the data table to be checked, the data entry corresponds to the data row in the associated field set at this time. In the embodiment of this application, the data table to be checked provided in Table 1 only includes two fields: age and weight. At this time, when the trusted field is the age field and the generated associated field set is {age, weight}, the corresponding The data are shown in Table 1.

In the embodiment of the present application, the performing correlation verification on the data items in the associated field set includes judging whether the values of the first field of the data item and its corresponding associated field are within the corresponding value range. For example, when the first field is an age field and the generated set of associated fields is {age, weight}, according to the correlation verification conditions of the associated field set {age, weight}, that is, the correlation verification conditions in Table 2, The correlation verification is carried out on the data items in the associated field set. If the age field of the data items in the associated field set and the numerical range of the corresponding weight field are within the corresponding numerical range provided in Table 2, then it is judged that the data item has passed the correlation. Sexual verification, if the numerical range of the age field and the corresponding weight field of the data entry in the associated field set is not within the corresponding numerical range provided in Table 2, the data entry is marked as potential abnormal data. For example, by comparing the data in Table 1 with the correlation verification conditions provided in Table 2, it can be found that the data entries corresponding to ages 4, 18, 46, and 10 are within the value range provided by the correlation verification conditions. The correlation verification of the above data items has passed; and the weight values corresponding to the data items of age 42, 18-24 are not within the value range provided by the correlation verification conditions, these data items do not meet the correlation verification conditions, and these data items are Flagged as potentially anomalous data.

In this embodiment of the present application, the data items in the correlation field set can also be converted into coordinate points in a two-dimensional or three-dimensional coordinate system, and the numerical range in the correlation verification condition can be converted into a credible interval in the corresponding coordinate system. For example, as shown in Figure 2, a Cartesian coordinate system can be constructed with the age field in Table 1 as the X-axis and the weight field as the Y-axis, and according to the values of the data items in Table 1, the data items in Table 1 can be converted into Coordinate points in the Cartesian coordinate system; at the same time, points and lines can be drawn and connected according to the upper and lower limits of the weight values of obese patients of different ages provided in Table 2 to generate the upper and lower limits of weight corresponding to the weight. The range between the two fitting curves is taken as the credible interval, so that the correlation verification conditions in Table 2 are converted into credible intervals in the coordinate system.

At this time, the correlation verification of the data items in the associated field set may be to determine whether the coordinate point determined according to the first field of the data item and its corresponding associated field is within the credible interval, if the If the coordinate value is not within the credible interval, the data entry is marked as potential abnormal data. For example, if the coordinate point corresponding to the data item with age 42 in FIG. 2 is not within the credible interval, the data item corresponding to age 42 is marked as potential abnormal data.

Step 140: Determine whether the potential abnormal data is real data, if so, determine whether the potential abnormal data satisfies the associated conditions corresponding to the target disease, if so, mark the potential abnormal data as normal, and at the same time, determine Whether the number or proportion of potentially abnormal data marked as normal reaches a preset threshold, and if so, update the correlation verification condition of the associated field set based on the potential abnormal data marked as normal, and based on the updated correlation The verification condition is to perform correlation verification on the data items to be verified in the associated field set.

In the embodiment of the present application, for example, for the potentially abnormal data provided in the above example, it is manually verified whether the weight data in the data entry is real data, that is, whether it conforms to the actual situation of the patient. If the actual situation, it means that the weight data is not real data (maybe because of errors in the manual input process), and the data corresponding to the data entry needs to be collected again; if the weight data conforms to the actual situation of the patient, it means that the potential abnormal data is real data.

In the embodiment of the present application, the association condition corresponding to the target disease is also preset. In the embodiment of the present application, the association condition corresponding to the target disease is that the data in the data table to be checked is only caused by the target disease, not affected by other diseases. For example, when the target disease of the study is obesity, the association condition corresponding to the obesity is that the data in the data table to be checked is only caused by obesity and is not affected by other diseases. If a patient has both obesity and malignancy, it can lead to anomalous data related to obesity research. Malignant tumors, for example, may cause a patient to lose weight rapidly. However, in the patient data collection stage, due to the lack of interoperability of data among various departments in the hospital, patient data can only be collected from the corresponding department based on the target disease type. After the patient data collection is completed, the patient data of other departments can be obtained through the server to determine whether the potential abnormal data meets the associated conditions corresponding to the target disease.

Further, in the embodiment of the present application, in the case of judging that the potential abnormal data is real data, continue to judge whether the potential abnormal data satisfies the associated conditions corresponding to the target disease, and if so, the potential abnormal data is Mark as normal, which can further ensure the accuracy of data screening in the associated field set.

In the embodiment of the present application, for example, if it is determined that the data entry whose age is 42 and the weight is 56 in the above-mentioned association field set is real data, it is further determined whether the data entry satisfies the association condition corresponding to obesity, that is, the data entry in the data entry. The data is solely due to obesity and not affected by other diseases. If the data in the data entry is indeed caused only by obesity, that is, the reason why the weight value is less than the value range provided in the correlation verification condition is not caused by other diseases, then it is judged that the data entry satisfies the association condition corresponding to obesity, and mark the data entry as OK.

On the other hand, if it is further determined that the potential abnormal data does not satisfy the association condition corresponding to the target disease type, the potential abnormal data that does not satisfy the association condition corresponding to the target disease type can be deleted. In the embodiment of the present application, for example, if it is determined that the data entry with age 42 and weight 56 in the above-mentioned association field set is real data, but the patient corresponding to the data entry has other diseases besides obesity, and the other diseases As a result, the weight value in the data entry is less than the value range of the weight provided in the relevant quality control conditions. In this case, it is judged that the data entry does not meet the association conditions corresponding to obesity, and the data entry is removed from the current association field set. delete.

In the embodiment of the present application, while marking the potential abnormal data as normal, it is determined whether the number or proportion of the potential abnormal data marked as normal reaches a preset threshold, and if so, based on the potential abnormal data marked as normal The data updates the correlation validation condition of the associated field set. Because the correlation verification conditions of the associated field set are preset based on experience, but with the continuous accumulation and updating of patient data, some new data may appear that are not suitable for the preset correlation verification conditions. Therefore, It is necessary to iteratively update the correlation verification conditions to ensure the accuracy of patient data screening results.

In this embodiment of the present application, for example, when the number of data items in the associated field set {age, weight} is N, the threshold for the number of potential abnormal data marked as normal may be preset in the server to be N/3, A counter is used to count the number of potential abnormal data marked as normal in real time, and when it is determined that the number of potential abnormal data marked as normal reaches a preset threshold N/3, the correlation verification condition of the associated field set is updated. In the embodiment of the present application, the threshold for the number of potential abnormal data marked as normal is not limited to the above situation, and those skilled in the art can set the corresponding threshold according to actual needs.

In this embodiment of the present application, for example, when the number of data entries in the associated field set {age, weight} is N, the threshold for the proportion of potentially abnormal data marked as normal may be preset in the server to be 1/3 , when it is determined that the proportion of potentially abnormal data marked as normal reaches 1/3 of the preset threshold, update the correlation verification condition of the associated field set. In the embodiment of the present application, the threshold value of the proportion of the potential abnormal data marked as normal is not limited to the above situation, and those skilled in the art can set the corresponding threshold value according to actual needs.

In this embodiment of the present application, the updating the correlation verification condition of the associated field set may be updating the value range of one or more associated fields based on the value of the potential abnormal data marked as normal. For example, if the data items corresponding to age 42 and 18-24 in the correlation field set provided in the above example are all potential abnormal data marked as normal, since the weight value of the data item corresponding to age 18-24 is related to the data provided in Table 2 The sex verification conditions are relatively close, so the value range of the weight field corresponding to the age of 18-24 in the correlation verification condition can be updated based on the weight value of the data entry corresponding to the age of 18-24. Specifically, in this embodiment of the present application, it is possible to determine which values of the potential abnormal data marked as normal are closer to the provided correlation verification conditions by judging whether the coordinate points of the potential abnormal data marked as normal are located near the credible interval. For example, as shown in Figure 2, the coordinate points corresponding to the data items with age 18-24 are relatively close to the credible interval, while the coordinate points corresponding to the data items corresponding to the age of 42 are far away from the credible interval. The weight value for the data entry corresponding to age 18-24 updates the range of values for weight corresponding to age 18-24 in the correlation validation condition.

In the embodiment of the present application, the correlation verification condition for updating the associated field set may be manually updated or automatically updated.

In the embodiment of the present application, the updating of the correlation verification condition of the associated field set may also be to select a certain value corresponding to a trusted field from the potential abnormal data marked as normal, and select a certain value corresponding to a trusted field in the data table to be checked. Statistical analysis is performed on the numerical value of the associated field of the numerical value, and the correlation verification condition of the associated field set is automatically updated according to the statistical analysis result. For example, if the data items corresponding to ages 42 and 18-24 in the associated field set provided in the above example are all potential abnormal data marked as normal, the value of age 18 can be selected from these data items to be checked. Statistical analysis is carried out on the weight values and frequency of occurrence of all data entries corresponding to age 18 in the table, the numerical range of the weight with the highest frequency of occurrence is selected, and the weight corresponding to the age of 18 in the correlation verification condition is updated based on the numerical range. range of values.

It should be noted that when there are many fields in the associated field set, updating the correlation verification condition of the associated field set may also update the numerical range of multiple associated fields, which is no longer the case in the embodiment of the present application. Expand the narrative.

In this application, an association field set is formed based on the first field and the associated fields of the first field, and the correlation verification condition is used to verify the correlation of the association field set, which effectively improves the verification accuracy of the population cohort study data; in addition, in the association During the correlation verification process of the field set, the correlation verification condition of the associated field set can be updated according to the potential abnormal data marked as normal, and the updated correlation verification condition can be used to perform correlation verification on the data items to be verified in the associated field set. , which prevents the data to be verified from being identified as potentially abnormal data, thereby effectively improving the efficiency of data verification.

In the embodiment of the present application, through steps 120-140, the correlation verification of one or more associated field sets corresponding to the trusted field or the unique trusted field selected first in the data table to be checked is completed.

In the embodiment of the present application, another method for verifying data of a population cohort study is also provided. The method takes into account the fact that there are multiple credible fields in the data table to be verified. After completing steps 110 to 140, the method further includes: Steps 120 to 140 are repeated for other trusted fields in the data table to be checked until the traversal of the trusted fields in the data table to be checked is completed.

In addition, if a certain associated field set involves two or more trusted fields, the repetition of the associated field set will occur during the above data verification process. In this case, in order to improve the data processing efficiency, you can skip the correlation that has already been performed. Validated set of associated fields, while correlation validation is performed directly on other sets of associated fields.

According to the method for verifying population cohort study data provided by the embodiment of the present application, an embodiment of the present application further provides a device for verifying population cohort study data, the system structure diagram of which is shown in FIG. 3 .

The apparatus 300 for checking population cohort research data provided by the embodiment of the present application includes a first checking unit 301 , a correlation field set generating unit 302 , a correlation verifying unit 303 , and a correlation verifying condition updating unit 304 . in,

The first verification unit 301 is configured to perform a first verification on the fields in the data table to be verified, and if the fields all meet the preset first verification conditions, select any field in the data table to be verified as the first field;

an associated field set generating unit 302, configured to query the associated field of the first field and generate an associated field set based on the first field and the associated field of the first field;

The correlation verification unit 303 is configured to perform correlation verification on the data items in the correlation field set according to the correlation verification conditions of the correlation field set, if the data items in the correlation field set do not meet the correlation verification condition, the data entry is marked as potentially abnormal data;

The correlation verification condition updating unit 304 is used to judge whether the potential abnormal data is real data, and if so, then judge whether the potential abnormal data satisfies the correlation conditions corresponding to the target disease, and if so, change the potential abnormal data Mark as normal, and at the same time, determine whether the number or proportion of potential abnormal data marked as normal reaches a preset threshold, and if so, update the correlation verification condition of the associated field set based on the potential abnormal data marked as normal, And based on the updated correlation verification conditions, correlation verification is performed on the data items to be verified in the associated field set.

FIG. 4 is a schematic structural diagram of an electronic device 400 provided according to an embodiment of the present application. As shown in FIG. 4 , the electronic device 400 includes a central processing unit (CPU) 401 that can be loaded into a random access memory (RAM) 403 according to a program stored in a read only memory (ROM) 402 or a program from a storage section 408 Instead, various appropriate actions and processes are performed. In the RAM 403, various programs and data necessary for the operation of the electronic device are also stored. The CPU 401 , the ROM 402 , and the RAM 403 are connected to each other through a bus 404 . An input/output (I/O) interface 405 is also connected to bus 404 .

The following components are connected to the I/O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc. ; and a communication section 409 including a network interface card such as a LAN card, a modem, and the like. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I/O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, may be downloaded and installed from a network via communications portion 409, and/or from Removable medium 411 is installed. When the instructions are executed by the central processing unit (CPU) 401, the various method steps described in the present invention are performed.

The above-mentioned embodiments are only specific implementations of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them, and the protection scope of the present application is not limited thereto. It should be understood by those of ordinary skill in the art that: any person skilled in the art can still modify the technical solutions described in the foregoing embodiments or easily think of changes within the technical scope disclosed in this application, or Some of the technical features are equivalently replaced; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application, and should be included within the protection scope of the present application.

Claims (10)

1. A verification method for population cohort study data, characterized in that, comprising: Step 1: perform a first check on the fields in the data table to be checked, and if the fields all meet the preset first check conditions, select any trusted field in the data table to be checked as the first field; Step 2: query the associated field of the first field and generate an associated field set based on the first field and the associated field of the first field; Step 3: According to the correlation verification conditions of the associated field set, perform correlation verification on the data items in the associated field set, if the data items in the associated field set do not meet the correlation verification conditions, then The above data entry is marked as potentially abnormal data; Step 4: Judging whether the potential abnormal data is real data, if yes, then judging whether the potential abnormal data satisfies the associated conditions corresponding to the target disease, if so, marking the potential abnormal data as normal, and at the same time, judging Whether the number or proportion of potentially abnormal data marked as normal reaches a preset threshold, and if so, update the correlation verification condition of the associated field set based on the potential abnormal data marked as normal, and based on the updated correlation The verification condition is to perform correlation verification on the data items to be verified in the associated field set. 2. The verification method for population cohort study data as claimed in claim 1, further comprising: collecting data corresponding to the target disease type and generating an initial data table, and selecting a plurality of data in the initial data table by sampling. The pieces of data generate the data table to be checked. 3. the checking method of crowd cohort research data as claimed in claim 1, is characterized in that, the described field in the data table to be checked is carried out first checking, including garbled checking, data format checking, information consistency checking, null value One or more of rate check, range check, field length check. 4. The method for checking population cohort study data according to claim 1, wherein the querying the associated field of the first field comprises: in one or more preset correlation verification conditions, querying A correlation verification condition of the first field is involved, and an associated field of the first field is determined based on the correlation verification condition. 5. The checking method of crowd cohort study data as claimed in claim 2, is characterized in that, also comprises, if any field in described data table to be checked does not meet the preset first checking condition, then re-collecting all data. The data of the corresponding fields in the initial data table is updated, the initial data table is updated based on the re-collected data, and a plurality of pieces of data in the initial data table are re-sampled to generate a data table to be checked. 6. The verification method for crowd cohort study data as claimed in claim 1, further comprising, after judging that the potential abnormal data is real data, if it is judged that the potential abnormal data does not meet the corresponding conditions of the target disease type. If the association condition is satisfied, the potential abnormal data that does not meet the association condition corresponding to the target disease type is deleted. 7. The method for checking population cohort study data as claimed in claim 1, wherein the correlation verification condition of the associated field set comprises the numerical range between the associated fields including the first field and its corresponding associated field Matching, the performing correlation verification on the data items in the associated field set includes judging whether the values of the first field of the data item and its corresponding associated field are within the corresponding value range. 8 . The method for checking population cohort study data according to claim 7 , wherein the updating the correlation verification condition of the associated field set comprises updating the numerical range of one or more associated fields. 9 . 9. The verification method for crowd cohort study data as claimed in claim 1, further comprising, repeating steps 2 to 4 for other credible fields in the data table to be checked except the first field, respectively, until The traversal of the trusted fields in the data table to be checked is completed. 10. A verification device for population cohort study data, characterized in that it comprises: The first verification unit is used to perform a first verification on the fields in the data table to be verified, and if the fields all meet the preset first verification conditions, select any trusted field in the data table to be verified as the first field ; an associated field set generating unit, configured to query the associated field of the first field and generate an associated field set based on the first field and the associated field of the first field; A correlation verification unit, configured to perform correlation verification on the data items in the correlation field set according to the correlation verification conditions of the correlation field set, if the data items in the correlation field set do not meet the correlation verification conditions , the data entry is marked as potentially abnormal data; A correlation verification condition update unit, configured to judge whether the potential abnormal data is real data, and if so, judge whether the potential abnormal data satisfies the correlation conditions corresponding to the target disease, and if so, mark the potential abnormal data is normal, and at the same time, determine whether the number or proportion of potential abnormal data marked as normal reaches a preset threshold, and if so, update the correlation verification condition of the associated field set based on the potential abnormal data marked as normal, and Based on the updated correlation verification conditions, correlation verification is performed on the data items to be verified in the associated field set.

Images