WO2020052405A1 - Corpus annotation set generation method and apparatus, electronic device, and storage medium - Google Patents

Abstract

The present application discloses a corpus annotation set generation method and apparatus, an electronic device, and a computer readable storage medium. According to the technical solution provided in the present application, a corpus set to be annotated is obtained from a query log, annotation results of multiple parties on query statements in the corpus set are obtained, the query statements having similar annotation results are screened out, and then a corpus annotation set is formed according to the query statements and the corresponding annotation results thereof.

Description

Method and device for generating corpus annotation set, electronic equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on September 10, 2018, with application number 201811048957.8, and the invention name is "Method and Device for Generating Corpus Label Sets, Electronic Equipment, Storage Medium", and its entire contents Incorporated by reference in this application.

Technical field

The present application relates to the field of computer technology, and in particular, to a method and an apparatus for generating a corpus annotation set, an electronic device, and a computer-readable storage medium.

Background of the invention

In the field of voice interaction, online analysis of query sentences entered by users is performed through various data analysis models to identify user intent and provide users with accurate answers. The data analysis model is obtained by training a large number of labeled query sentences (referred to as the training set). Therefore, the accuracy of the query result annotation results in the training set directly affects the accuracy of the data analysis model and determines the intelligence level of the voice interaction function.

At present, the query statements are mainly manually labeled by the labelers. For example, the query intent of the query statement (including the intention of gossip, music on demand, weather query intent, etc.). So the cognition level of the tagger determines the tagging accuracy of the query.

Because the cognitive level of the labeler may be different from that of ordinary people, or there is a deviation in the cognition of a query sentence, it is easy to make the query sentence labels included in the training set inaccurate, which will cause analysis of the training data. The model has large errors and cannot provide users with accurate answers.

Summary of the Invention

In order to solve the problem of inaccurate labeling results of query sentences in the training set due to the bias of the cognition of the labeling staff in the related art, this application provides a method for generating a corpus labeling set.

In one aspect, this application provides a method for generating a corpus annotation set, which is executed by an electronic device and includes:

Obtaining a query log; the query log includes a query statement;

Extracting to-be-annotated query sentences from the query log to obtain a to-be-annotated corpus;

Obtaining the annotation results of multiple parties on the query sentence in the corpus to be annotated;

Filtering out query statements with similar labeling results from the corpus to be labeled according to the labeling results of the same query statement by multiple parties;

A corpus annotation set is generated from the query sentences with similar annotation results and corresponding annotation results.

On the other hand, this application provides another apparatus for generating a corpus annotation set, including:

A log acquisition module, configured to acquire a query log; the query log includes a query statement;

A corpus obtaining module for extracting query sentences to be labeled from the query log to obtain the corpus to be labeled;

A result acquisition module, configured to acquire annotated results of multiple parties on the query sentence in the corpus to be annotated;

A statement filtering module, configured to filter query statements with similar labeled results from the corpus to be labeled according to the labeled results of the same query statement by multiple parties;

The annotation set generation module is configured to generate a corpus annotation set from a query sentence with a similar annotation result and a corresponding annotation result.

Further, this application provides an electronic device. The electronic device includes:

processor;

Memory for storing processor-executable instructions;

The processor is configured to execute the method for generating the corpus annotation set.

Further, the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to complete the method for generating the corpus annotation set.

It should be understood that the above general description and the following detailed description are merely exemplary, and should not limit the present application.

Brief description of the drawings

The drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic diagram of an implementation environment involved in this application;

Fig. 2 is a block diagram of a server device according to an exemplary embodiment;

Fig. 3 is a flow chart showing a method for generating a corpus annotation set according to an exemplary embodiment;

4 is a schematic diagram of labeling results of various labeling tasks;

FIG. 5 is a schematic diagram of the division principle of multiple corpus annotation sets;

FIG. 6 is a schematic diagram of the influence curve of the corpus annotation set of each batch on the model performance;

7 is a detailed flowchart of step 330 in the embodiment corresponding to FIG. 3;

Fig. 8 is a schematic diagram illustrating a generation principle of a corpus annotation set according to an exemplary embodiment;

9 is a detailed flowchart of step 350 in the embodiment corresponding to FIG. 3;

10 is a detailed flowchart of step 370 in the embodiment corresponding to FIG. 3;

11 is a flowchart of a method for generating a corpus annotation set based on the embodiment of FIG. 3;

Fig. 12 is a block diagram of a device for generating a corpus annotation set according to an exemplary embodiment;

13 is a detailed block diagram of a corpus obtaining module in the embodiment corresponding to FIG. 12;

14 is a detailed block diagram of a result acquisition module in the embodiment corresponding to FIG. 12;

FIG. 15 is a detailed block diagram of the sentence filtering module in the embodiment corresponding to FIG. 12.

Implementation

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of devices and methods consistent with certain aspects of the application as detailed in the appended claims.

Fig. 1 is a schematic diagram of an implementation environment involved in the present application according to an exemplary embodiment. The implementation environment involved in this application includes a server device 110. The query log is stored in the server device 110, so that the server device 110 can use the corpus annotation set generation method provided in the present application to generate a corpus annotation set using the query log, thereby improving the accuracy of the query sentence annotation result in the corpus annotation set, and comparing the accuracy with The high corpus annotation set is used as a training set to train the data analysis model to improve the accuracy of the data analysis model. Subsequently, the data analysis model is used to analyze the query sentences entered by the user online to provide users with accurate answers.

As needed, the implementation environment will also include providing data, that is, the data source of the query log. Specifically, in this implementation environment, the data source may be a smart terminal 130. Although only one smart terminal 130 is shown in FIG. 1, those skilled in the art should understand that the number of smart terminals providing query logs may be large here. Each. As shown in FIG. 1, the server device 110 may be connected to the smart terminal 130 through a wired or wireless network, and obtain a query log collected and uploaded by the smart terminal 130. The query log refers to a process performed by the smart terminal 130 when the user inputs a query sentence. recording. The query log may include a time point, a query entered by a user, and a query result returned to the user. The query entered by the user may be in the form of text or voice. The query log may also include a large number of query statements entered by one or more users. The server device 110 then uses the method provided in this application to generate a corpus annotation set. The smart terminal 130 may be a smart phone, a smart speaker, or a tablet computer.

As required, the implementation environment will also include an intelligent terminal 140 that provides query statements and waits to provide a response to the user. Those skilled in the art can understand that an intelligent terminal 130 that provides query logs and an intelligent terminal that provides query statements and wait to provide a response to the user. The terminals 140 may be the same or different. As shown in FIG. 1, the server device 110 that has generated a corpus tag set according to the query log provided by the smart terminal 130 is connected to the smart terminal 140 through a wired or wireless network, receives a query sentence input by a user, and is trained based on the generated corpus tag set. The data analysis model analyzes the query entered by the user, identifies the user's intention, generates an accurate response, and then feeds it back to the smart terminal 140 through a wired or wireless network.

It should be noted that the method for generating the corpus annotation set of the present application is not limited to deploying corresponding processing logic in the server device 110, and may also be processing logic deployed in other machines. For example, processing logic for generating a corpus annotation set is deployed in a terminal device with computing capabilities.

Referring to FIG. 2, FIG. 2 is a schematic structural diagram of a server device according to an embodiment of the present application. The server device 200 may have a relatively large difference due to different configurations or performance, and may include one or more central processing units (CPU) 222 (for example, one or more processors) and a memory 232, one or More than one storage medium 230 (eg, one or one storage device) storing application programs 242 or data 244. The memory 232 and the storage medium 230 may be temporary storage or persistent storage. The program stored in the storage medium 230 may include one or more modules (not shown), and each module may include a series of instruction operations on the server device 200. Furthermore, the central processing unit 222 may be configured to communicate with the storage medium 230 and execute a series of instruction operations in the storage medium 230 on the server device 200. The server device 200 may also include one or more power sources 226, one or more wired or wireless network interfaces 250, one or more input-output interfaces 258, and / or, one or more operating systems 241, such as Windows Server ^™ , ^{mac OS XTM, UnixTM, Linux TM} , FreeBSD TM and so on. The steps performed by the server device described in the embodiments shown in FIG. 3, FIG. 7, FIG. 8, and FIG. 10 to FIG. 12 below may be based on the server device structure shown in FIG. 2.

A person of ordinary skill in the art may understand that all or part of the steps of implementing the following embodiments may be completed by hardware, or related hardware may be instructed by a program. The program may be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk.

Fig. 3 is a flow chart showing a method for generating a corpus annotation set according to an exemplary embodiment. The application scope and execution subject of the method for generating the corpus annotation set may be a server device or an electronic device for a server device described below, where the server device may be a server 110 device of the implementation environment shown in FIG. 1. As shown in FIG. 3, the method for generating the corpus annotation set may be executed by an electronic device or the server device 110, and may include the following steps.

In step 310, a query log is obtained;

The query log refers to a record collected by a device when a user inputs a query sentence, and the device may be a smart speaker, a mobile terminal, or the like. The query log can be viewed as a raw corpus containing a large number of query statements. The so-called raw corpus refers to the query sentence belonging to the original real user, which has not been manually labeled.

In step 330, extract the query sentence to be labeled from the query log to obtain a corpus to be labeled;

It should be noted that because the query log contains a large number of query statements, but not all query statements are valid, some may be input by users at random and do not represent any significance, some query statements may be too long or too short, and there are many queries Sentences may be repeated. If the query results of these query statements are used as the corpus annotation set, the accuracy of the annotation results in the corpus annotation set will be reduced, which will affect the accuracy of the data analysis model obtained by training the corpus annotation set as the training sample.

Therefore, the present application can extract the query sentence to be labeled from the query log according to a pre-configured strategy, and the query sentence to be labeled constitutes the corpus to be labeled. Among them, the extraction of the query sentence to be labeled may be an analysis of the query log, according to the configured useless / deactivated character library, remove the query sentence containing the useless / deactivated characters, remove the meaningless query sentence (such as random input A few characters without coherence), remove too long or too short query statements, remove duplicate query statements, remove already labeled query statements, and obtain the last remaining query statement as the query statement to be labeled.

In step 350, acquire the annotation results of multiple parties on the query sentence in the corpus to be annotated;

Among them, multiple parties can be multiple labeling personnel, multiple labeling devices, or multiple labeling programs in one device, which are used to indicate that there are multiple sources of labeling results for the query sentence in the corpus to be labeled. For the convenience of description, the following Labelers, equipment, or programs are collectively referred to as labelers. Each annotator can annotate the query in the annotated corpus (called "voting"). Annotation refers to adding a classification label to the query sentence in the corpus to be annotated, and multiple "voting" results can reflect the correct classification of the query sentence.

The labeling result is the classification label added by the labeler for the query. According to the different labeling tasks, the labeling results can be intent labeling results, NER (Named Entity Recognition) labeling results, slot labeling results, or segmentation labeling results. Among them, the intent labeling result refers to the result of intent classification, for example, "I'm in a bad mood today", and the labeler's intent labeling result for the query sentence is "chat intent"; for example, "please give me a soothing song", the labeler The intent labeling result of the query is "music on demand intent".

The NER labeling result is that the index names the person name, place name, institution name, proper name, etc. in the query sentence. The slot labeling result refers to adding slot labels to each phrase in the query, such as the weather business field. The slot labels include time words, place words, weather business keywords, weather phenomenon words, and interrogative words. The word segmentation result refers to dividing a query sentence into multiple phrases, and multiple phrases are used as the word segmentation result. Each phrase can be regarded as a classification label.

As shown in FIG. 4, for the corpus to be labeled, the labeler can perform intent labeling, NER labeling, slot labeling, or word segmentation labeling to obtain labeling results for each labeling task. Specifically, each party may first perform intent annotation on the query sentence in the annotated corpus (according to the intent annotation document specification) to obtain an intent annotation set including the intent annotation result of the query sentence. Furthermore, the query statement can be divided into fields according to the intent labeling, and NER labeling (according to the NER labeling document specification) and slot labeling (according to the slot labeling document specification) can be performed at the same time in the divided fields to obtain NERs containing the NER labeling results. Callout set and slot callout set containing slot callout results. Among them, at the same time as intent labeling, each labeler can also perform segmentation labeling on the corpus to be labeled to obtain a segmentation labeling set including the result of the segmentation labeling.

The intent annotation set, slot annotation set, NER annotation set, or word segmentation annotation set may be stored in the storage medium of the server device, and the server device may obtain the annotation results of the query statements in the corpus to be annotated by the multiparty from the storage medium.

In step 370, according to the annotation results of the same query sentence by multiple parties, a query sentence with a similar annotation result is filtered from the corpus to be annotated;

Among them, the query statements with similar labeling results refer to query statements with the same or similar multi-labeled results. The similarity of the multi-labeled results is greater than a preset value. It can be considered as a query with similar labeling results. The preset value can be 80%, 90%. %.

In one embodiment, assuming that the labeling result is an intent labeling result, the server device obtains the intent labeling results of the query statements in the set of corpora to be labeled by multiple parties, and sequentially compares the intent labeling results of the query statements by multiple parties for each query statement in turn. , To determine whether the annotation results of the query sentence by multiple parties are consistent (the similarity of the annotation results is greater than a preset value can be regarded as consistent), and then the query sentences with the same annotation results are selected from the corpus to be labeled.

Specifically, for the same query, if the labeling results of multiple parties are consistent, a single label labeling set is added. If they are inconsistent, the final arbiter is required to assist in reviewing the specifics of the inconsistencies:

i) If more than half of the labeled parties have the same label, take multiple labels that are consistent as the labeling result and add a single label label set;

ii) If the distribution of inconsistent results is 1: 1, it may be the case of multiple labels (use cases can be labeled with multiple labels). If the reviewer determines that it is a multi-label case, the multi-label label set is added;

iii) If the multi-party labeling results are inconsistent, it may be a multi-label sample or a difficult sample. After review, a multi-label label set or a difficult sample set is added.

In this way, a labeling task goes through the labeling process and finally gets three labeling sets: a single-label labeling set, a multi-label labeling set, and a difficult sample set. A query in a single-label callout set can be considered to be the same as the result of the multi-party callout. The single-label annotation set can be regarded as a reliable annotation set, and can be used as the training set and test set of the intent recognition model.

Similarly, assuming that the labeling result is a NER labeling result, a slot labeling result, or a word segmentation labeling result, it is also possible to filter out query statements that have the same labeling result on multiple parties.

In step 390, a corpus annotation set is generated from the query sentences with similar annotation results and corresponding annotation results.

The corpus tagging set includes a query sentence and a corresponding tagging result. The query sentence belongs to a query sentence similar to the multi-party tagging result screened in step 370. The server device generates a query corpus tag set composed of the query sentence and the tagging result by using the filtered multi-party tagging results similar to the query sentence and the tagging result of the query sentence. In other words, since the query sentence of the corpus tagging set belongs to a query sentence with similar multi-labeled results, the possibility of divergence in the tagging results of the query sentence in the corpus tagging set is small, and the accuracy of the tagging result is higher, and the accuracy is higher. The corpus annotation set is used as a training set to train data analysis models such as intent recognition models, which can improve the accuracy of the data analysis model.

As shown in FIG. 5, for the query sentence in the corpus to be annotated, multiple annotators mark the corpus to be annotated, and the server device obtains an annotation result 1 of the query sentence in the annotated corpus by the annotator 1, and the annotator 2 treats the annotated corpus. The annotation result 2 of the query sentence, the annotation party 3 is the annotation result 3 of the query sentence in the annotated corpus, and the annotation party 4 is the annotation result 4 of the query sentence in the annotated corpus. The server device merges the labeling result 1, labeling result 2, labeling result 3, and labeling result 4 to filter out four query statements with the same labeling result and add a single label labeling set. If more than half of the query statements have the same labeling result, It can be considered that the multi-party labeling results are the same. Take the consistent labeling results of multiple people as the labeling result of the query, and add the query to the single-label labeling set, and use the single-label labeling set as the corpus labeling set. Sets are merged as training and test sets.

As shown in Figure 5, if the query results 1, 2, 3, and 4 of some query statements are inconsistent, and the inconsistent results are distributed as 1: 1, these query statements may be multi-labeled. In the case of multiple labels, these query statements are added to the multi-label label set. Assuming that the query results 1, 2, 3, and 4 of some query statements are all inconsistent, and may be multi-label samples or difficult samples, these query statements can be added to the multi-label label set or difficult label set.

What needs to be explained is that the corpus label set is a single label label set, which contains query statements with the same multi-label labeling results. That is to say, there is no difference in the annotation results of query sentences in the corpus annotation set, and the accuracy of the annotation results is high. Therefore, the corpus annotation set can be used as a highly accurate training set or test set for training the data analysis model.

For example, assuming that the labeling result is an intent labeling result, and the corpus labeling set includes a query sentence with the same intent labeling result and its corresponding intent labeling result, the corpus labeling set can be used as a training set to train the intent recognition model. Assuming that the labeling result is a NER labeling result, and the corpus labeling set includes a query sentence with the same NER labeling result and its corresponding NER labeling result, the corpus labeling set can be used as a training set to train a named entity recognition model. Similarly, assuming that the labeling result is a slot labeling result, the corpus labeling set can be used as a training set to train the slot labeling model. Assuming that the labeling result is a word segmentation labeling result, the corpus labeling set can be used as a training set to train the tokenization model. .

In the technical solution provided by the foregoing exemplary embodiment of the present application, by obtaining a set of to-be-marked corpora from a query log, obtaining a plurality of user-annotated results of the query sentence in the corpus, filtering out query sentences with the same annotated result, and then using these queries The sentence and its corresponding labeling result constitute the corpus labeling set. Since the query sentence of the corpus tagging set belongs to the same or similar query sentence with multi-party tagging results, the possibility of disagreement on the tagging result of the query sentence in the corpus tagging set is small, and the accuracy of the tagging result is high, and the accuracy is further improved. A higher corpus annotation set is used as a training set to train data analysis models such as intent recognition models, which can improve the accuracy of the data analysis model.

As needed, the corpus annotation set can be added to the existing training set in an incremental and overlapping manner to retrain the data analysis model, and use the same test set to test the performance of the data analysis model, and evaluate the new corpus annotation set pair model. The performance improvement brought by performance reflects the quality and value of the new corpus annotation set.

Take the intentional corpus tag set as an example. The performance results of the conscious recognition model on the test set are used as benchmarks. After that, each batch of corpus annotation set is added to the model training set, and the performance indicators of the model trained after each batch of data is added are recorded. The curve shown in Figure 6 records the performance of the model trained after the corpus annotation set of each batch is added. The sixth batch (s6) of the data has a significant performance gain on the trained model. The corpus annotation set of this batch can be selected. Added to training data.

In addition, if the annotator is the annotator, in order to prevent the annotators from cheating each other during the annotating process, the method can be adopted to stagger the annotations in each cycle. As shown in Table 1 below.

Table 1 staggered task schedule

The above table uses four labelers as examples. In order to prevent the labelers from referencing the labeling results to each other, the labelers have different content on the same day. You can arrange labeling according to the schedule shown in Table 1, and take five days as a cycle. Results are counted for consistent and inconsistent labeled results among multiple people.

In an exemplary embodiment, as shown in FIG. 7, the above step 330 specifically includes:

In step 331, the query statements in the query log that do not satisfy the preset conditions are removed;

The query statements that do not meet the preset conditions may include one or more of the following forms: query statements containing useless / deactivated characters, meaningless query statements, excessively long or too short query statements, repeated query statements, etc. , Thereby avoiding subsequent annotation of these worthless query sentences, which not only increases the workload, but also affects the accuracy of the corpus annotation set.

In step 332, the remaining query statements in the query log are input into the multiple label prediction models that have been constructed, and the label prediction results of multiple label prediction models for the same query statement are output; Trained from different training sample sets;

Specifically, the label prediction model may be an intent recognition model for identifying the intent of a query sentence. Accordingly, the label prediction result may be an intent recognition result. The label prediction model can be trained using a large number of query sentences (that is, training sample sets) with known intent to label the results. Multiple label prediction models can be trained using different training samples. For example, all query sentences with known intent labeling results are divided into 4 batches, and each batch of query statements is trained to obtain a corresponding intent recognition model, and thus 4 intent recognition models can be obtained.

After removing the query statements that do not meet the requirements above, the remaining query statements in the query log are input into 4 intent recognition models, and the intent recognition results of the 4 intent recognition models for the same query sentence are output.

It should be noted that, depending on the labeling task, the label prediction model can also be a named entity recognition model, slot labeling model, or word segmentation model. These models can be obtained by training a large number of query statements with known NER labeling results. Known slot positions A large number of query statements with labeled results are trained, and a large number of query statements with known tokenized label results are trained. Similarly, the label prediction result can be the corresponding named entity recognition result, slot labeling result, and word segmentation result. The construction method of the label prediction model belongs to the prior art, and is not repeated here.

In step 333, according to the label prediction results of the multiple query prediction models for the same query sentence, the query sentences with inconsistent label prediction results are filtered from the remaining query sentences to obtain the corpus to be labeled.

Because the boundary sample points are of great significance to the boundary of the training model, if you find sample points with multiple intents and certain probability distributions in different classes, adding such sample points to the training set for model training is more accurate than classifying Adding the sample points to the training set will help the model's performance improvement even more.

The present application filters out query sentences with inconsistent tag prediction results from the remaining query sentences in the query log according to the tag prediction results of the same query sentence by multiple tag prediction models. That is, the model has low recognition accuracy for these query sentences, so these query sentences can be regarded as boundary sample points. Adding these boundary sample points to the corpus to be labeled for model training can improve the accuracy of the model.

In an exemplary embodiment, the foregoing step 331 may include the following steps:

Classify the query statements recorded in the query log by the constructed classifier, and remove the meaningless query statements obtained by the classification.

Among them, the meaningless query statement refers to a statement without specific intention, which may be a user error or a random input statement. A classifier is also a classification model. The role of a classifier is to identify the query statements in the query log, which are meaningful and which are meaningless. Specifically, a classifier can be trained through a large number of meaningful query sentences and non-meaningful query sentences. For example, a classifier can be obtained by training the parameters of a logistic regression model through a large number of meaningful query sentences and non-meaningful query sentences. Classifiers are a collective term for the methods used to classify samples in data mining. Classifiers are constructed in a variety of ways including algorithms such as decision trees, logistic regression, naive Bayes, and neural networks.

Specifically, the query sentence in the query log can be input to the trained classifier, and a meaningful or meaningless judgment result is output, thereby removing the meaningless query sentence in the query log. You can also remove query statements that contain useless or disabled characters in the query log based on the configured useless or disabled character libraries.

In another exemplary embodiment, the foregoing step 331 may further include the following steps:

According to the set of labeled query statements, the labeled query statements in the query log and query statements similar to the labeled query statements are removed.

The set of labeled query statements refers to a set of query statements with known labeling results. The labeled query statement set may be a generated corpus label set. According to the query statements contained in the query statement set, the query statements belonging to the set can be removed from the query statements of the query log. The labeled query statement refers to the query statements in the query statement set. Query statements similar to the labeled query statement can be calculated by calculating the similarity between the query statements to find out the query log with a higher similarity to the labeled query statement. Query statements to eliminate query statements with a high degree of similarity to labeled query statements in the query log.

That is, the remaining query statements in the above step 332 can be the removal of meaningless query statements, the removal of query statements containing useless or disabled characters, the removal of labeled query statements, and the removal of similarity to labeled query statements in the query log. After the query statement, the query statements remaining in the query log.

In another exemplary embodiment, the foregoing step 331 may further include the following steps:

A query sentence containing only a single entity word, a query sentence with a sentence length greater than a preset number of characters, or a duplicate query sentence is removed from the query log.

Among them, the entity word refers to the name of a real specific thing, for example, a song name, a singer name, and the like. A query that contains only one entity word is difficult to distinguish between intent and word segmentation, so it is not suitable to join the corpus annotation set to participate in modeling. Query statements with a statement length greater than the preset number of characters refer to longer query statements. Such query statements are difficult to label, and the length of the query statement during the modeling process will undoubtedly increase the amount of calculation. Therefore, it is not suitable. Join corpus annotation set to participate in modeling. Similarly, it is not necessary to add a corpus annotation set to participate in modeling in the repeated query statements in the query log. Therefore, to eliminate duplicate query statements, for example, three query statements are duplicated, and two query statements are retained and only one query statement is removed.

In summary, the remaining query in step 332 can also be the last remaining query in the query log after removing the query that contains only a single entity word, the query that is longer than the preset number of characters, or the duplicate query. .

As shown in Figure 8, for the new query, pre-process the new query, remove meaningless queries, remove useless / deactivated characters, remove single-entity query words, and remove extra long and repeated words. The query statements can be removed from the labeled query statements according to the labeled query statement collection, and the query statements with a high similarity to the labeled query statement collection can be removed. Further, the labels can be filtered through the above steps 332 and 333. The query sentences whose prediction results are inconsistent, and the filtered query sentences constitute a corpus to be marked. Furthermore, according to the annotation results of the corpus to be annotated by multiple parties, query sentences with similar annotation results can be filtered to generate a corpus annotation set. Furthermore, the corpus annotation set can be added to the annotated query sentence set to participate in the training of the model together.

In an exemplary embodiment, as shown in FIG. 9, the above step 350 specifically includes:

In step 351, a labeling task for the corpus to be labeled is distributed to multiple parties, and the distribution of the labeling task triggers multiple parties to execute the labeling task in parallel;

The labeling task may be an intent labeling task, a NER labeling task, a slot labeling task, or a word segmentation labeling task. For example, multiple parties may be multiple labeling devices, and the server device sends the labeling tasks carrying the corpus to be labeled to the multiple labeling devices, triggering the multiple labeling devices to execute the labeling tasks in parallel. It should be noted that the labeling device may be an intelligent labeling device obtained through training in advance with a large amount of sample data. Each labeling device is trained with different sample data sets, so the labeling accuracy of each labeling device is different.

In one embodiment, the server device may issue a labeling task carrying a set of corpora to be labeled to terminal devices to which multiple labelers belong. The terminal device to which the labeler belongs can perform the display of the corpus to be labeled and the prompt of the labeling task. Users can click intent or mark to make intent mark, NER mark, slot mark and word segmentation. The terminal device of multiple mark people can get the mark result according to the user's click option or mark, and complete the corpus to be marked. Annotate tasks.

In an exemplary embodiment, the distribution of the labeling task triggers multiple parties to execute the labeling task in parallel, which specifically includes: the distribution of the labeling task triggers multiple parties to input the set of to-be-marked corpora into a labeling model configured by themselves and outputs The labeling results of the corpus to be labeled, respectively; wherein the labeling models with multiple configurations are obtained by training with different training sample sets.

In other words, multiple parties can represent multiple labeling devices or multiple labeling programs. Each labeling party is configured with a labeling model. Since the labeling model configured with multiple parties is trained using different training sample sets, multiple labeling devices or multiple labeling programs have different labeling accuracy. It should be noted that the training sample set used by the multi-party configured labeling model in this embodiment is also different from the training sample set used by the label prediction model described above. For example, all samples can be divided into 10 training sample sets, and each training sample set can be trained to obtain the corresponding model, and then 10 models can be used as part of the label prediction model and part as the labeling model, using multiple labels The prediction model selects query sentences with inconsistent label prediction results to obtain the corpus to be labeled. Then, the multi-party labeling model is used to calculate the labeling results of the query statements in the corpus to be labeled. The multi-party labeling results of the query statements in the corpus to be labeled are obtained.

Assume that multiple parties refer to multiple labeling programs deployed in the server device. Multiple labeling programs can perform the following steps in parallel: input the corpus to be labeled into a pre-built labeling model, and output the labeling results of the corpus to be labeled. The construction of the labeling model can refer to the construction of the label prediction model.

In step 352, receiving the labeling results returned by multiple parties executing the labeling task in parallel.

Multiple labeling devices or terminal devices belonging to multiple labeling personnel perform labeling tasks in parallel to obtain labeling results and return the labeling results to the server device. The server device receives the labeling results returned by the multiple labeling devices or terminal devices belonging to the multiple labeling personnel. Corresponding to the labeling task, the labeling result can be an intent labeling result, a NER labeling result, a slot labeling result, or a segmentation labeling result.

In an exemplary embodiment, the corpus to be annotated includes a plurality of buried-point sentences with known label information; the buried-point sentence refers to a query sentence that is known to accurately label the result, and performs multi-party tagging results on the buried-point sentence. The exact labeling result of the buried point statement is called label information. As shown in FIG. 10, the above step 370 specifically includes:

In step 371, according to the labeling results of the multiple buried-point statements by multiple parties, comparing whether the labeling results of the multiple buried-point statements are consistent with corresponding label information, and calculating an accuracy rate of the multi-party labeling results;

It should be noted that when filtering the corpus of annotations based on the annotation results of multiple parties, it is necessary to first judge the accuracy of the annotation results of each annotation party, so as to remove the annotation results provided by the annotation party with a lower accuracy rate.

The accuracy of the multi-party labeling result refers to the accuracy of labeling multiple buried-point sentences by each labeling party. The calculation of the labeling accuracy of the buried-point sentence is used to evaluate the labeling accuracy of the current labeling party. During the labeling process, a “buried point” method was used to verify the accuracy of each labeling party. Among them, it can be extracted from the previous batch of labeled data sets with more than 5% consistent query statements as multiple buried point statements for the current batch of known label information. For each labeling party, according to the labeling results of multiple buried-point statements of known label information, and comparing whether the labeling results of the multiple buried-point statements are consistent with the known label information, the labeling results are calculated with The proportion of the label information is consistent, and the accuracy of the labeling result of the labeler is obtained.

In step 372, according to the accuracy rate of the multi-party labeling result, the source of the labeling result whose accuracy rate is not up to standard is removed from the multi-party sources.

Specifically, a threshold value can be set, and according to the accuracy rate of the labeling result of each user, the labeling party whose accuracy rate is less than the threshold value can be regarded as providing the labeling result whose accuracy rate does not meet the standard. This can delete the labeling results provided by such labelers whose accuracy is less than the threshold.

Alternatively, according to the accuracy rate of the labeling result of each labeler, all labelers are sorted from high to low accuracy. Several labelers with lower ranking can be regarded as labelers whose accuracy is not up to standard. As a result, the labeling results provided by the labeler with a substandard accuracy can be removed.

In step 373, according to the labeling results of the remaining sources, a query sentence with similar labeling results from multiple sources is filtered from the corpus to be labeled.

The labeling results of the remaining sources refer to the labeling results of the corpus that are to be labeled by the remaining labeling parties after deleting the labeling results with substandard accuracy from the labeling results provided by multiple parties. That is to say, in the subsequent selection of query sentences with similar annotation results from multiple corpora to be annotated, the annotation results provided by the annotated parties are no longer based on the annotation results. According to the remaining taggers with higher accuracy, the tagging results of the tagging corpus are selected, and multiple query sentences with similar tagging results are filtered out from the tagging corpus.

In an exemplary embodiment, as shown in FIG. 11, the method for generating a corpus annotation set provided in the present application further includes:

In step 1101, according to the annotation results of multiple parties on the same query sentence, a query sentence with inconsistent annotation results is filtered from the corpus to be annotated;

It should be noted that the boundary samples are very helpful for model optimization and characterization of clearer classification boundaries. Among them, the boundary samples can be selected and used from samples inconsistent from multiple parties. Specifically, the server device can filter out query statements with inconsistent labeling results from the corpus to be labeled according to the labeling results of the same query statement from multiple parties.

In step 1102, a multi-label query statement is obtained from the query statements with inconsistent labeling results, and a boundary sample point for optimizing a data analysis model is obtained.

For query statements with inconsistent labeling results for multiple users, multi-label query statements (that is, query statements with multiple labeling results) can be obtained from the review by the reviewer. Such multi-label query statements can be considered as boundary sample points. The recognition of this kind of query is difficult, so if the model can accurately identify the intent, slot, etc. of this kind of query, it will greatly improve the accuracy of the model. The data analysis model may be an intent recognition model, a named entity recognition model, a slot labeling model, a word segmentation model, and the like. The optimization of the data analysis model through such query statements can improve the recognition accuracy of the model.

For example, "I'm in a bad mood today, please give me a soothing song." The query's intent contains both chat and intent intent. This sample is a boundary sample of intent classification and can help model training. Make accurate intent boundaries.

The following is a device embodiment of the present application, which can be used to execute an embodiment of a method for generating a corpus annotation set performed by the server device 110 of the present application. For details not disclosed in the device embodiment of the present application, please refer to the embodiment of a method for generating a corpus annotation set of the present application.

Fig. 12 is a block diagram of a device for generating a corpus annotation set according to an exemplary embodiment. The device for generating a corpus annotation set can be used in the server device 110 of the implementation environment shown in Fig. 1 to execute Figs. 3 and 7 -All or part of the steps of the method for generating a corpus annotation set shown in any one of FIG. 11. As shown in FIG. 12, the device includes, but is not limited to, a log acquisition module 1210, a corpus acquisition module 1230, a result acquisition module 1250, a sentence filtering module 1270, and a label set generation module 1290.

The log acquisition module 1210 is configured to acquire a query log, where the query log includes a query statement;

A corpus obtaining module 1230, configured to extract query sentences to be labeled from the query log to obtain the corpus to be labeled;

A result acquisition module 1250, configured to acquire the annotation results of multiple parties on the query sentence in the corpus to be annotated;

A sentence filtering module 1270, configured to filter, based on the annotation results of multiple parties for the same query sentence, query sentences with similar annotation results from the corpus to be marked;

The annotation set generating module 1290 is configured to generate a corpus annotation set from a query sentence with a similar annotation result and a corresponding annotation result.

For the implementation process of the functions and functions of each module in the above device, please refer to the implementation process of the corresponding steps in the method for generating the corpus annotation set for details, and details are not described herein again.

The log acquisition module 1210 may be, for example, a wired or wireless network interface 250 with a certain physical structure in FIG. 2.

The corpus acquisition module 1230, the result acquisition module 1250, the sentence filtering module 1270, and the annotation set generation module 1290 may also be functional modules for performing corresponding steps in the method for generating the corpus annotation set. It can be understood that these modules can be implemented by hardware, software, or a combination of both. When implemented in hardware, these modules may be implemented as one or more hardware modules, such as one or more application specific integrated circuits. When implemented in software, these modules may be implemented as one or more computer programs executing on one or more processors, such as programs stored in the memory 232 executed by the central processor 222 of FIG. 2.

In an exemplary embodiment, as shown in FIG. 13, the corpus obtaining module 1230 includes:

A sentence removing unit 1231, configured to remove a query sentence in the query log that does not satisfy a preset condition;

A label prediction unit 1232 is configured to input the remaining query statements in the query log into multiple label prediction models that have been constructed, and output the label prediction results of multiple label prediction models for the same query sentence; the multiple label prediction models Obtained by training with different training sample sets;

A sentence extraction unit 1233 is configured to filter query sentences with inconsistent tag prediction results from the remaining query sentences according to the tag prediction results of the same query sentence by the multiple tag prediction models to obtain the corpus to be labeled.

In an exemplary embodiment, the sentence removing unit 1231 includes:

The classification removing subunit is configured to classify the query sentences recorded in the query log by using the constructed classifier, and remove meaningless query sentences obtained by the classification.

In an exemplary embodiment, the sentence removing unit 1231 further includes:

The first removing subunit is configured to remove the labeled query sentence and the query sentence similar to the labeled query sentence in the query log according to the labeled query sentence set.

In an exemplary embodiment, the sentence removing unit 1231 further includes:

The second removing subunit is configured to remove a query sentence containing only a single entity word, a query sentence with a sentence length greater than a preset number of characters, or a duplicate query sentence in the query log.

In an exemplary embodiment, as shown in FIG. 14, the result acquisition module 1250 includes:

A task dispatching unit 1251, configured to dispatch a labeling task to the corpus to be labeled to multiple parties, and the dispatching of the labeling task triggers multiple parties to execute the labeling task in parallel;

The result receiving unit 1252 is configured to receive a labeling result returned by multiple parties executing the labeling task in parallel.

The distribution of the labeling task triggers multiple parties to execute the labeling task in parallel, including:

The distribution of the labeling task triggers multiple parties to input the corpus to be labeled into their own labeled model in parallel, and outputs the respective labeling results for the corpus to be labeled. Among them, the labeling models configured by multiple parties use different training sample sets. Get it.

In an exemplary embodiment, the corpus to be annotated includes a plurality of buried point sentences with known tag information; as shown in FIG. 15, the sentence filtering module 1270 includes:

An accuracy calculation unit 1271, configured to compare whether the labeling results of the multiple buried-point statements are consistent with corresponding label information according to the labeling results of the multiple buried-point statements by multiple parties, and calculate the accuracy of the multi-party labeling results;

A source culling unit 1272, configured to remove, according to the accuracy rate of the multi-party labeling result, the source of the labeling result whose accuracy rate does not meet the standard;

A sentence filtering unit 1273 is configured to filter query sentences with similar multi-source annotation results from the corpus to be annotated according to the annotation results of the remaining sources.

This application also provides an electronic device that can be used in the server device 110 of the implementation environment shown in FIG. 1 to execute all or a method of generating a corpus annotation set shown in any of FIG. 3 and FIG. 7 to FIG. 11. Some steps. The electronic device includes:

processor;

Memory for storing processor-executable instructions;

Wherein, the processor is configured to execute the method for generating a corpus annotation set according to the above exemplary embodiment.

The specific manner in which the processor of the electronic device performs operations in this embodiment has been described in detail in the embodiment of the method for generating the corpus annotation set, and will not be described in detail here.

In an exemplary embodiment, a storage medium is also provided, and the storage medium is a computer-readable storage medium, and may be, for example, temporary and non-transitory computer-readable storage media including instructions. The storage medium stores a computer program that can be executed by the central processing unit 222 of the server device 200 to complete the above-mentioned method for generating a corpus annotation set.

It should be understood that the present application is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes may be performed without departing from the scope thereof. The scope of the application is limited only by the accompanying claims.

Claims (17)

A method for generating a corpus annotation set, which is executed by an electronic device and includes: Obtaining a query log; the query log includes a query statement; Extracting to-be-annotated query sentences from the query log to obtain a to-be-annotated corpus; Obtaining the annotation results of multiple parties on the query sentence in the corpus to be annotated; Filtering out query statements with similar labeling results from the corpus to be labeled according to the labeling results of the same query statement by multiple parties; A corpus annotation set is generated from the query sentences with similar annotation results and corresponding annotation results. The method according to claim 1, wherein the extracting the to-be-annotated query sentence from the query log to obtain the to-be-annotated corpus includes: Removing query statements in the query log that do not satisfy a preset condition; Inputting the remaining query sentences in the query log into multiple constructed label prediction models, and outputting the label prediction results of multiple label prediction models on the same query sentence; the multiple label prediction models adopt different training sample sets Get training According to the label prediction results of the multiple query prediction models for the same query sentence, the query sentences with inconsistent label prediction results are filtered out from the remaining query sentences to obtain the corpus to be labeled. The method according to claim 2, wherein removing the query statements in the query log that do not satisfy a preset condition comprises: Classify the query statements recorded in the query log by the constructed classifier, and remove the meaningless query statements obtained by the classification. The method according to claim 2, wherein removing the query statements in the query log that do not satisfy a preset condition comprises: According to the set of labeled query statements, the labeled query statements in the query log and query statements similar to the labeled query statements are removed. The method according to claim 2, wherein removing the query statements in the query log that do not satisfy a preset condition comprises: A query sentence containing only a single entity word, a query sentence with a sentence length greater than a preset number of characters, or a duplicate query sentence is removed from the query log. The method according to claim 1, wherein the acquiring the annotation result of the query sentence in the corpus to be annotated includes: Dispatching a labeling task to the corpus to be labeled to multiple parties, and the dispatching of the labeling task triggers multiple parties to execute the labeling task in parallel; Receive the labeling results returned by multiple parties executing the labeling task in parallel. The method according to claim 6, wherein the dispatching of the labeling task triggers multiple parties to execute the labeling task in parallel, comprising: The distribution of the labeling task triggers multiple parties to input the corpus to be labeled into their own labeled model in parallel, and outputs the respective labeling results for the corpus to be labeled. Among them, the labeling models configured by multiple parties use different training sample sets. Get it. The method according to claim 1, wherein the corpus to be annotated includes a plurality of buried point sentences with known label information; and the annotation result is filtered from the corpus to be annotated according to an annotation result of multiple parties on the same query sentence. Similar queries, including: Comparing the labeling results of the multiple buried point sentences with the corresponding label information according to the labeling results of the multiple buried point sentences by multiple parties, and calculating the accuracy rate of the multiple party labeling results; Removing the source of the labeling result that does not meet the target accuracy rate from the multiple sources according to the accuracy of the multi-party labeling result; According to the labeling results of the remaining sources, the query sentences with similar multi-source labeling results are filtered from the corpus to be labeled. A device for generating a corpus annotation set includes: A log acquisition module, configured to acquire a query log; the query log includes a query statement; A corpus obtaining module for extracting query sentences to be labeled from the query log to obtain the corpus to be labeled; A result acquisition module, configured to acquire annotated results of multiple parties on the query sentence in the corpus to be annotated; A statement filtering module, configured to filter query statements with similar labeled results from the corpus to be labeled according to the labeled results of the same query statement by multiple parties; The annotation set generation module is configured to generate a corpus annotation set from a query sentence with a similar annotation result and a corresponding annotation result. The apparatus according to claim 9, wherein the corpus obtaining module comprises: A statement removing unit, configured to remove query statements in the query log that do not satisfy a preset condition; A label prediction unit is configured to input the remaining query statements in the query log into multiple label prediction models that have been constructed, and output the label prediction results of multiple label prediction models for the same query statement. Trained with different training sample sets; A sentence extraction unit is configured to filter query sentences with inconsistent tag prediction results from the remaining query sentences according to the tag prediction results of the same query sentence by the multiple tag prediction models to obtain the corpus to be labeled. The apparatus according to claim 10, wherein the sentence removing unit comprises: The classification removing subunit is configured to classify the query sentences recorded in the query log by using the constructed classifier, and remove meaningless query sentences obtained by the classification. The apparatus according to claim 10, wherein the sentence removing unit comprises: The first removing subunit is configured to remove the labeled query sentence and the query sentence similar to the labeled query sentence in the query log according to the labeled query sentence set. The apparatus according to claim 10, wherein the sentence removing unit comprises: The second removing subunit is configured to remove a query sentence containing only a single entity word, a query sentence with a sentence length greater than a preset number of characters, or a duplicate query sentence in the query log. The apparatus according to claim 9, wherein the result acquisition module comprises: A task dispatching unit, configured to dispatch an annotation task to the corpus to be annotated to multiple parties, and the dispatching of the annotation task triggers multiple parties to execute the annotation task in parallel; The result receiving unit is configured to receive a labeling result returned by multiple parties executing the labeling task in parallel. The apparatus according to claim 9, wherein the corpus to be labeled includes a plurality of buried point sentences with known label information; and the sentence filtering module comprises: An accuracy calculation unit, configured to compare whether the labeling results of the multiple buried-point statements are consistent with corresponding label information according to the labeling results of the multiple buried-point statements by multiple parties, and calculate an accuracy rate of the multi-party labeling results; A source culling unit, configured to remove, from the multi-party source, the source of the labeled result that does not meet the standard accuracy according to the accuracy of the multi-party labeled result; A sentence screening unit is configured to filter query sentences with similar multi-source annotation results from the corpus to be annotated according to the annotation results of the remaining sources. An electronic device includes: processor; Memory for storing processor-executable instructions; The processor is configured to execute the method for generating a corpus annotation set according to any one of claims 1-8. A computer-readable storage medium stores a computer program, and the computer program is executable by a processor to complete a method for generating a corpus annotation set according to any one of claims 1-8.

Images