CN115511073A - A training method for a semantic matching model and a text matching method - Google Patents

Abstract

The invention provides a training method of a semantic matching model, which comprises the following steps: a1, obtaining a training set, wherein the training set comprises a plurality of samples, each sample comprises a preset text and two texts to be matched corresponding to each preset text, each sample corresponds to a weak label and a weight, the weak label indicates which text to be matched of the two texts to be matched contained in the corresponding sample is more relevant to the preset text, the numerical value of the initial weight is relevant to a difficulty index indicating the difficulty of the corresponding sample, and the samples with the larger difficulty are endowed with the smaller initial weight; and A2, carrying out repeated iterative training on the semantic matching model by using samples in the training set, enabling the semantic matching model to output correlation scores of the predetermined text and the text pair formed by each text to be matched respectively, determining a weighting loss value according to the correlation scores, the weak labels and the weights so as to update the semantic matching model, and dynamically adjusting the weights of the samples according to the number of times of training which is finished currently.

Description

A training method for a semantic matching model and a text matching method

technical field

The present invention relates to the field of neural networks, specifically to the field of text matching, and more specifically to a training method for a semantic matching model and a text matching method.

Background technique

In information retrieval, the user usually inputs the corresponding query text, and the system returns the relevant web page text (some documents are also called documents) as the query result. When the system searches based on the query text entered by the user, it usually goes through two matching stages. Among them, the first stage is the recall stage, which usually recalls a small number of candidate texts from a massive text database to form a candidate text set (for example, calculating the correspondence between text data and query text from a million-level or tens of millions-level web text database) The cosine distance of the semantic vector, according to the order of the cosine distance from small to large, select dozens, hundreds or thousands of candidate texts as the candidate text set). The second stage is the reordering stage (some documents are also called matching stage), which further calculates the similarity between the query text and each candidate text, and then outputs the corresponding candidate text as the query result of the query text based on similarity sorting.

The reranking stage generally models the matching scores of query text and candidate text by training a neural network model. However, if a neural network model (hereinafter referred to as a semantic matching model) is trained based on manually labeled data, the cost is very expensive. Therefore, some researchers proposed a semantic matching model method based on weakly supervised annotation training. This method uses the traditional BM25 (Best Matching, Best Matching 25) model or query likelihood model as a weak tagger, and according to the output representation text The correlation score is used to construct the weak label of the sample, and the semantic matching model is trained based on the training set with the weak label. The trained semantic matching model has the effect of surpassing the original BM25 model and the query likelihood model, which can effectively improve the retrieval efficiency. performance.

Although training the semantic matching model based on samples with a large number of weak labels does not require manual labeling of data, it reduces the time cost of the labeling process. However, the inventor found during training that when training the semantic matching model based on samples with a large number of weak labels, if the semantic matching model is trained equally from the beginning of training to the end of training, the semantic matching model will It is difficult to learn the correct knowledge, and the accuracy of the model is not high; to ensure the reliability of the trained model, this training method requires a lot of manual debugging of the model. Although the cost of manual labeling is reduced, the time cost of training and debugging is increased. Therefore, it is necessary to improve the prior art.

Contents of the invention

Therefore, the object of the present invention is to overcome the above-mentioned defects in the prior art, and provide a training method of a semantic matching model and a text matching method.

The purpose of the present invention is achieved through the following technical solutions:

According to a first aspect of the present invention, a training method for a semantic matching model is provided, including: A1. Obtain a training set, which includes a plurality of samples, each sample includes predetermined text and two texts to be matched corresponding to each predetermined text , where each sample corresponds to a weak label and a weight, the weak label indicates which of the two texts to be matched contained in the corresponding sample is more relevant to the predetermined text, and the value of the initial weight is the same as that of the corresponding sample Difficulty is related to the difficulty index, and the relatively more difficult samples are given relatively smaller initial weights; A2. Using the samples in the training set to perform multiple iterations on the semantic matching model, so that it can be matched with each to be matched according to the predetermined text The text pair formed by the text outputs the correlation score between the two, and the weighted loss value is determined according to the correlation score, weak label and weight to update the semantic matching model, where the weight of the sample is dynamically adjusted according to the number of times the training has been completed.

In some embodiments of the present invention, in the step A2, the weight of the corresponding sample is increased after each predetermined training times, wherein, each time the weight of the sample is increased, the weight of the sample whose initial weight is relatively smaller increases. Larger values are relatively larger.

In some embodiments of the present invention, in the step A2, after the number of currently completed trainings reaches a predetermined threshold, set the weights of all samples to the same value to train the semantic matching model for multiple times.

In some embodiments of the present invention, the loss value of a single sample is calculated as follows:

L=max(0,ε-y×(S ₁ -S ₂ ))×w(τ,i);

Among them, max( ) represents the function of taking the maximum value of the elements contained in it, and its output is the loss value of the sample, ε represents the interval parameter, y represents the weak label, y∈{-1,+1}, when y takes +1 means that the first text to be matched is more relevant to the predetermined text, when y is -1, it means that the second text to be matched is more relevant to the predetermined text, S ₁ means that the first text to be matched is more relevant to the predetermined text The correlation score of the text, S ₂ represents the correlation score between the second text to be matched and the predetermined text, and w(τ,i) represents the weight of the sample τ adjusted after completing i training.

In some embodiments of the present invention, the weight of the corresponding sample is increased after each predetermined number of training times, and the weight of the sample is dynamically adjusted according to the number of times of training currently completed and the difficulty index of the sample.

In some embodiments of the present invention, the weight of samples is determined in the following manner:

Among them, i represents the number of trainings that have been completed so far, c represents the predetermined number of thresholds, D(τ) represents the difficulty function, |a _τ | represents the difficulty index of the sample τ, and |a _n | represents the difficulty of any sample in the set A Index, set A represents the set of all samples in the current training batch or represents the training set, max{|a _n |∈A} represents the maximum value of all samples in the current training batch or the difficulty index of all samples in the training set .

In some embodiments of the present invention, the difficulty index of the sample is the BM25 score generated by the BM25 model for the correlation between the first text to be matched in the sample and the predetermined text minus the BM25 model for the second text to be matched in the sample. The absolute value of the difference in BM25 scores generated by the relevance of the matched text to the predetermined text.

In some embodiments of the present invention, the semantic matching model includes a BERT model and a linear prediction layer, wherein the BERT model is used to form a text pair based on the predetermined text in each sample and each text to be matched to output a text containing semantic A semantic vector of similarity information; the linear prediction layer is used to output a correlation score between the text to be matched and the predetermined text according to the semantic vector.

In some embodiments of the present invention, the method further includes: before using the training set to train the semantic matching model for the first time, performing histogram equalization processing on the difficulty index of the samples in the training set, so that the difficulty index of the samples in the training set Evenly distributed over the difficulty space.

According to a second aspect of the present invention, a text matching method is provided, including: B1, acquiring a query text and a plurality of candidate texts, forming a text pair from the query text and each candidate text; B2, combining each text pair respectively Input the semantic matching model that the training method training of semantic matching model according to claim 1-8 obtains, obtain the relevance score of query text and each candidate text; B3, query text and the correlation score pair of each candidate text The candidate texts are sorted, and the corresponding candidate texts are output according to the sorting results.

In some embodiments of the present invention, the corresponding candidate text output according to the sorting result is at least one of the following texts: the candidate text with the highest correlation score; the predetermined text selected in descending order of the correlation score number of candidate texts; candidate texts with a relevance score greater than or equal to a predetermined score threshold.

According to a third aspect of the present invention, an electronic device includes: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to The instructions are executed to implement the steps of the method described in the first aspect and/or the second aspect.

Compared with the prior art, the present invention has the advantages of:

The present invention adds a difficulty index to each weakly labeled sample, and assigns a relatively smaller initial weight to the relatively more difficult sample at the beginning of training. During the training process, as the number of iterative training increases, dynamically adjust each weight based on the difficulty index. The weight of each sample; thus, the semantic matching model pays less attention to the more difficult samples at the beginning, realizes progressive learning from easy to difficult, improves the accuracy of the model and reduces the cost of manual debugging.

Description of drawings

Embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

Fig. 1 is the schematic diagram that utilizes BM25 model to calculate the BM25 score of predetermined text and to-be-matched text according to the embodiment of the present invention;

2 is a schematic flow diagram of a training method for a semantic matching model according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of dynamically adjusting sample weights according to an embodiment of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

As mentioned in the background technology section, when training a semantic matching model based on samples with a large number of weak labels, if each sample is used to train the semantic matching model equally from the beginning of training to the end of training, it will lead to semantic matching. It is difficult for the model to learn the correct knowledge, and the accuracy of the model is not high. When the inventor used the BM25 model to construct weakly labeled data, he found that some training pairs consisted of the returned first document (top1 document, that is, the text most relevant to the predetermined text) and randomly extracted documents, and other training pairs consisted of the first A bit document and a second bit document (Top2 document). For the former, the model is relatively easy to distinguish, while the latter is more difficult to distinguish. Because the first document in the former has more words associated with the predetermined text than the randomly extracted documents. However, both the top1 document and the top2 document of the latter may have many query-related words.

In order to solve this problem, the present invention adds a difficulty index to each weakly labeled sample, and assigns relatively smaller initial weights to samples with relatively higher difficulty when starting training. During the training process, as the number of iterative training increases, based on the difficulty The index dynamically adjusts the weight of each sample; thus, the semantic matching model pays less attention to the more difficult samples at the beginning, realizes progressive learning from easy to difficult, improves the accuracy of the model and reduces manual debugging the cost of.

According to the existing algorithm or model (such as BM25 model, query likelihood model, etc.) Accuracy of Semantic Recognition Models. For ease of understanding, here is a brief introduction to the overall implementation process of the present invention by taking the BM25 model as an example, which mainly includes the following parts:

(1) Prepare a plurality of predetermined texts and a plurality of texts to be matched, score the relevance of each predetermined text and each text to be matched based on the BM25 model, and obtain the BM25 score of each predetermined text and each text to be matched;

(2) Combine the predetermined text and two texts to be matched with differences in correlation to form a sample, wherein a sample in the form of a triplet is formed in the form of a predetermined text q and two texts to be matched d ₁ and d ₂ (q, d ₁ , d ₂ ); since there are multiple predetermined texts and multiple texts to be matched, a large number of samples containing weak labels and difficulty indicators can be quickly formed to form a sample data set in this way;

(3) Construct the weak label of each sample according to the BM25 scores of the predetermined text and each text to be queried in each sample; for example, suppose the BM25 scores of the predetermined text q and d ₁ are BM25(q,d ₁ ) and q and The BM25 score of d ₂ is BM25(q,d ₂ ), if BM25(q,d ₁ )>BM25(q,d ₂ ), set the value of the weak label to +1, if BM25(q,d ₁ )< BM25(q,d ₂ ) sets the value of the weak label to -1;

(4) Construct the difficulty index of each sample according to the BM25 score of the predetermined text and each text to be queried in each sample; for example, subtract the BM25 score of predetermined text q and d ₁ contained in a sample from The absolute value of the difference between the BM25 scores of d ₂ |BM25(q,d ₁ )- BM25(q,d ₂ )| is used as the difficulty indicator for this sample;

(5) Select part samples from the sample data set formed according to the above steps as a training set, and train the semantic matching model according to the training method of the present invention;

(6) Use the trained semantic matching model to score the correlation between the query text input by the user and multiple candidate texts, and output the corresponding candidate texts.

In order to illustrate the technical solution of the present invention in more detail, the following describes four aspects of the model used, the training samples, the training process, and the application scenario.

1. The model used

According to an embodiment of the present invention, the semantic matching model includes a BERT model and a linear prediction layer, wherein the BERT model is used to form a text pair based on the predetermined text in each sample and each text to be matched to output information containing semantic similarity The semantic vector of . The semantic matching model can use Google's pre-trained BERT model, and use a fully connected layer as a linear prediction layer to convert the semantic vector output by the BERT model into a similarity score. When entering the BERT model, input the two input texts as sentence pairs. For a predetermined text q and a text to be matched d _n , the input form is, for example: [CLS] predetermined text q[SEP] to be matched text d _n , [CLS] is the start identifier of the BERT model, and [SEP] is the separator separating two sentences; after the processing of the BERT model, the semantic vector corresponding to the start identifier [CLS] is used as the semantic similarity information The semantic vector is input into the linear prediction layer to get the similarity score, which is recorded as BERT(q,d _n ). Preferably, the semantic matching model can adopt an existing model structure, for example, the structure of a BERT-Rank model can be adopted.

2. Training samples

According to an embodiment of the present invention, the sample data set of the present invention includes a plurality of samples, weak labels corresponding to each sample and difficulty indicators, and each sample is a text containing predetermined text q and two texts d ₁ and d ₂ to be matched way to form samples in the form of triplets (q, d ₁ , d ₂ ). Wherein, when the value of the weak label is +1, it indicates that the text d ₁ to be matched in the sample is more related to the predetermined text q, and when the value of the weak label is -1, it indicates that the text d ₂ to be matched in the sample is more related to the predetermined text q. The difficulty index indicates the difficulty of the corresponding sample, and the closer the correlation between the two texts to be matched and the predetermined text in the sample is, the greater the difficulty of the sample is. The difficulty index can be the absolute value of the difference between the BM25 scores of the predetermined text q and d ₁ in the sample minus the difference between the BM25 scores of the predetermined text q and d ₂ |BM25(q,d ₁ )- BM25(q,d ₂ )| . Usually a plurality of predetermined texts are set, and each predetermined text is used to simulate the query text input by the user. In other words, the predetermined text is a sample text of the constructed query text.

Referring to Figure 1, the following is an example of scoring English based on the BM25 model to illustrate the setting of weak labels and difficulty indicators:

Suppose a predetermined text is: International Organized Crime (international organized crime)

Suppose some text to be matched is:

Text to be matched 1:

A parliamentary commission accused AAA today of doing little to stopsome kind of crime and BBB international networks from pumping billions of dollars through companies of CCC. hardware for FFF. "The threat posed by organized crime has been recognized too late by the GGG office," the special investigative commission report said.

For ease of understanding, a translation of the text to be matched is given (the same below): A parliamentary committee today accused the AAA of doing nothing to prevent a certain type of crime and the BBB international network from injecting billions of dollars through CCC's companies. It said DDD was too soft on organized crime and failed to even investigate allegations that criminal EEE used cocaine profits to buy military equipment for FFF. "The GGG Office recognized the threat posed by organized crime belatedly," the special commission of inquiry report said. "

Text to be matched 2:

The precondition for taking over the task of combating crime is anincrease in personnel as well as an amendment to the Federal Constitutional Protection Law. Werthebach sees potential work for the BfV in observing organized crime structures that might exert influence over the media necessary to the issurgent methods and legal bases of intelligence services all over Europe.

例句：Taking over the task of fighting crime requires additional personnel and changes to the Federal Constitutional Protection Act. Werthebach sees the BfV's potential work in observing organized crime structures that could exert influence over the media. There is an urgent need to harmonize the methods and legal basis of intelligence services across Europe.

Text to be matched 3:

Steven Anreder, Drexel's spokesperson, said Sher's promotion had nothing to do with the SEC settlement. He said it was based simply on a decision by the firm's board to give Sher additional recognition. Securities industry analysts noted that during Joseph's addterm as chief a var executive to Joseph have been vice chairmen, and they said the Sher's promotion doesn't appear to change Joseph's status within the firm.

例句：Drexel spokesman Steven Anreder said Sher's promotion had nothing to do with the SEC settlement. He said this was based solely on the company's board of directors deciding to give Sher additional recognition. Analysts in the securities industry pointed out that during Joseph's tenure as CEO, in addition to Joseph, there were many executives who served as vice chairman. They said that Schell's promotion did not seem to change Joseph's position in the company.

Taking the predetermined text "International Organized Crime" as a query, the BM25 model can give the BM25 scores of the query and each text to be matched, which are 21.3, 18.7, and 0, respectively.

Schematically, as shown in Figure 1, some returned results can be obtained by searching for "international organized crime" through the BM25 model, and there are different BM25 scores. It is significantly more difficult for the semantic matching model to judge whether the query matches the text to be matched 1 (with a BM25 correlation score of 21) or with the text to be matched 2 (with a BM25 correlation score of 18). 21) or the difficulty of matching with the text to be matched 3 (BM25 correlation score is 0). Therefore, we can construct a weakly labeled sample with difficulty (for example: predetermined text, to-be-matched text 1, to-be-matched text 2), the weak label of this sample is +1, and the difficulty index is |21.3-18.7|=2.6.

Of course, whether it is a weak label or a difficulty index, in addition to the BM25 score, the basis for its construction can also be based on the relevance score given by the query likelihood model or the relevance score output by other semantic matching models. any restrictions.

Further, in order to avoid the unbalanced learning of the model on samples of various difficulties, according to an embodiment of the present invention, when making the sample data set, a variety of samples with different degrees of difficulty are generated, and samples of various difficulty indicators quantity balance. For example, the difficulty index of the samples is graded at a certain interval, and when making the sample data set, the number of samples of each difficulty level in the sample data set is the same. When dividing the training set, take the same number of samples at each difficulty level, so as to ensure that the number of samples of various difficulty indicators in the training set is balanced.

再通过归一化方法将难度指标映射到[0,1] 区间，得到以使各个样本的难度指标在[0,1]区间均衡分布。Because the correlation distribution between multiple documents to be matched and the predetermined text is often unbalanced, ensuring that the number of samples with various difficulty indicators is balanced according to the above-mentioned implementation method when making a sample data set will cause more samples to be unusable, thus , the sample data set can be constructed in a general way, and then the difficulty distribution of the sample can be converted through a certain processing method to ensure the accuracy of the model. According to an embodiment of the present invention, the method further includes: the sample data set or The difficulty indicators of the samples in the training set are subjected to histogram equalization processing, so that the difficulty indicators of the sample data set or the samples in the training set are uniformly distributed in the difficulty space. For example, taking the BM25 score as an example, when constructing a sample, the difficulty index corresponding to the sample is not a uniform distribution on the data. This will lead to more difficult samples or simple samples in the samples learned by the model. Histogram equalization can be used to make the distribution of samples as balanced as possible. Here the transformation T is defined so that the samples are uniformly distributed in the difficulty space, and r=|BM25(q,d ₁ )-BM25(q,d ₂ )| is a triplet τ (q,d ₁ ,d ₂ ) The absolute value of the BM25 difference. Define a transformation T(r) that transforms r into a new space s. s=T(r), use p _r (r) to represent the probability distribution function of the original BM25 difference, p _s (s) represents the transformed probability distribution function, and define the value space of r as [0,R]. p _s (s) is a uniform distribution on [0,R], then:

Then the difficulty index is mapped to the [0,1] interval through the normalization method, so that the difficulty index of each sample is evenly distributed in the [0,1] interval.

In order for the model to carry out the learning process from easy to difficult, each sample has a corresponding weight. The value of the initial weight is related to the difficulty index indicating the difficulty of the corresponding sample. The relatively more difficult samples are given relatively smaller initial weights.

3. Training process

Before training, according to a certain ratio, the data set obtained by the method of the above-mentioned embodiment is divided into a training set and a test set, and then training is performed. Referring to FIG. 2, according to an embodiment of the present invention, a training of a semantic matching model is provided The method includes: A1, obtaining a training set, which includes a plurality of samples, each sample includes predetermined text and two texts to be matched corresponding to each predetermined text, wherein each sample corresponds to a weak label and a weight, and the weak label Indicates which of the two texts to be matched contained in the corresponding sample is more relevant to the predetermined text. The value of the initial weight is related to the difficulty index indicating the difficulty of the corresponding sample. initial weight; A2, using the samples in the training set to carry out multiple iterative trainings on the semantic matching model, so that it outputs the correlation score of the two according to the text pair formed by the predetermined text and each text to be matched, and according to the correlation Score, weak label, and weight determine the weighted loss value to update the semantic matching model, where the weight of the sample is dynamically adjusted according to the number of times the training has been completed. As the number of training increases, the weight corresponding to the relatively more difficult samples is gradually increased. In this way, the present invention can at least achieve the following beneficial technical effects: the present invention adds a difficulty index to each weakly labeled sample, and assigns relatively smaller initial weights to samples with relatively greater difficulty when starting training. The number of iterative training increases, and the weight of each sample is dynamically adjusted based on the difficulty index; thus, the semantic matching model pays less attention to the more difficult samples at the beginning, and realizes progressive learning from easy to difficult, It can guide the learning process and increase the speed of learning, and finally achieve the effect of improving the accuracy of the model and reducing the cost of manual debugging.

According to an embodiment of the present invention, the weight of loss can be adjusted once per training (Epoch), but the calculation amount of such adjustment is large, and it may also be unfavorable for the model to learn knowledge stably. According to an embodiment of the present invention, the The method further includes: increasing the weight of the corresponding sample after each predetermined number of training times (Epoch), wherein, each time the weight of the sample is increased, the weight of the sample whose initial weight is relatively smaller is relatively larger. Preferably, the predetermined training times are 50-1000 times. For example, if the initial weight of sample 1 is 0.05, the initial weight of sample 2 is 0.1, and the number of scheduled training is set to 100, then in the process of 0-99 training, the initial weight is multiplied by the loss value of the sample as the weighted loss value; then increase the weight of the sample, for example: the weight of sample 1 is increased by 0.095, becoming 0.145; the weight of sample 2 is increased by 0.09, becoming 0.19, in the 100th -199th training, the next weight Multiply the loss value of the sample as the weighted loss value; and so on. For the simplest sample, its weight can be set to a fixed value and remain unchanged in the training corresponding to each training number, for example, the weight is always set to 1.

After the training reaches a certain number of times, since the model has experienced a good learning process from easy to difficult, it can be allowed to learn each sample equally to improve the accuracy of the model. According to an embodiment of the present invention, the current After the number of training times reaches the predetermined threshold, the weights corresponding to the loss values of all samples are set to the same value, and the semantic matching model is trained multiple times using the training set while keeping the value unchanged. For example: the threshold of the predetermined number of times is set to an integer value in the range of 10000-50000 times (Epoch). Take 20000 as an example. In each training, 1 is used as the weight of each sample.

When the weight of the sample needs to be adjusted each time, the weight of the sample can be determined according to the number of times the current training is completed and the difficulty index. According to an embodiment of the present invention, the weight of the corresponding sample is increased after each predetermined training times, and the weight of the sample is based on The number of trainings currently completed and the difficulty index of the samples are dynamically adjusted. Preferably, the weight of the sample is determined in the following manner:

改为

其余不变。Among them, i represents the number of trainings that have been completed so far, c represents the predetermined number of thresholds, D(τ) represents the difficulty function, |a _τ | represents the difficulty index of the sample τ, and |a _n | represents the difficulty of any sample in the set A Index, set A represents the set of all samples in the current training batch or represents the training set, max{|a _n |∈A} represents the maximum value of all samples in the current training batch or the difficulty index of all samples in the training set . It should be understood that this method is only a preferred embodiment, and it is also possible to set the value that increases each time the weight is adjusted for samples of different difficulties, so as to ensure that the value of the weight increase of the sample with a relatively small initial weight is relatively large . Or, the above formula is adjusted to easily produce other implementations, for example, the above formula

changed to

The rest remain unchanged.

According to an embodiment of the present invention, the weighted loss value of a single sample is calculated in the following manner:

L=max(0,ε-y×(S ₁ -S ₂ ))×w(τ,i);

Among them, max( ) represents the function of taking the maximum value of the elements contained in it, and its output is the loss value of the sample, ε represents the interval parameter, y represents the weak label, y∈{-1,+1}, when y takes +1 means that the first text to be matched is more relevant to the predetermined text, when y is -1, it means that the second text to be matched is more relevant to the predetermined text, S ₁ means that the first text to be matched is more relevant to the predetermined text The correlation score of the text, S ₂ represents the correlation score between the second text to be matched and the predetermined text, and w(τ,i) represents the weight of the sample τ adjusted after completing i training. ε can also be called a margin value (Margin), and its value can be determined by the implementer according to the implementation scenario. The value range is generally [0.5-1], usually 0.5 or 1. It should be understood that if the scheduled number of training times is not 1, for example, the number of scheduled training times is set to 5, the weight of the sample is still w(τ,i) in the i+2th training, because the next adjustment has not yet been reached The condition on the weight of the sample. In addition, during training, if the model is trained by stochastic gradient descent, the parameters of the model are updated each time according to the weighted loss value of a single sample; if the model is trained by batch gradient descent, each time it is based on a batch (Batch) The sum of the weighted loss values of all samples in or the average of the weighted loss values of all samples updates the parameters of the model. When each sample (q, d ₁ , d ₂ ) is input into the model, two correlation matches are performed, the first time is the correlation score between the first text to be matched and the predetermined text S1=BERT(q, d ₁ ), the second time is the correlation score between the second text to be matched and the predetermined text S2=BERT(q,d _n ), and then according to the two correlation scores, based on the above loss function, calculate the loss value and update Model. In addition, it should be understood that the correlation scores of these two times are different from the correlation scores (such as BM25 scores) forming the sample data set.

See Figure 3, schematically, assuming that the text to be matched that is more related to the predetermined text q is d ⁺ , and the other text to be matched is d ^- , and the BM25 score of each text to be matched and q is marked below it, such as the most difficult In the sample, the BM25 score of d ⁺ and q is 20, the BM25 score of d ^- and q is 19, the absolute value of BM25 score is smaller than that of its adjacent samples, and its initial weight w(τ,0) is relatively The smaller the value, the weight of the sample is dynamically adjusted every 1000 training times (Epoch), that is, the adjusted weight is used at the 1000th training time next time, and so on. If the preset number threshold is set to 10000 times, after the 0-9999 training times reach the preset number threshold, in the 10000th training and subsequent training, set the difficulty index of all samples to 1 and keep the weight unchanged Do multiple rounds of training.

4. Application scenarios

According to one embodiment of the present invention, a text matching method is provided, including: B1, acquiring query text and multiple candidate texts, forming a text pair respectively from the query text and each candidate text; B2, combining each text pair respectively Input the semantic matching model trained according to the training method of the semantic matching model of the foregoing embodiment, obtain the relevance score of query text and each candidate text; B3, query text and the relevance score of each candidate text sort the candidate text , output the corresponding candidate texts according to the sorting results. As mentioned in the background art, the plurality of candidate texts may be a small number of candidate texts related to the query text recalled from a massive text database in the recall stage. According to an embodiment of the present invention, the corresponding candidate text output according to the sorting result is at least one of the following texts: the candidate text with the highest correlation score; a predetermined number selected in descending order of the correlation score candidate texts; candidate texts with relevance scores greater than or equal to a predetermined score threshold.

In order to verify the effect of the present invention, the inventor has carried out relevant experiments, and the experiments show that if the difficulty distribution of the samples in the training set is unbalanced, after the semantic matching model is trained according to the training method of the present invention, the corresponding sorting of the semantic matching model after training The resulting MAP (mean average precision) index is about 2-2.5% higher than the existing training method, and the NDCG index (Normalized Discounted Cumulative Gain, normalized discounted cumulative gain) is about 2-3% higher. In addition, if the difficulty distribution of the samples in the training set is balanced, the training effect will be further improved. Compared with the existing training methods, the MAP will be improved by about 2-2.5%, and the NDCG index will be increased by about 3-4%.

It should be noted that although the steps are described above in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be performed concurrently, or even change the order, as long as it can be realized The required functions are sufficient.

The present invention can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present invention.

A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may include, for example, but is not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above.

Having described various embodiments of the present invention, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or technical improvement in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

Claims (13)

1. A training method of a semantic matching model is characterized by comprising the following steps:

a1, obtaining a training set, wherein the training set comprises a plurality of samples, each sample comprises a preset text and two texts to be matched corresponding to each preset text, each sample corresponds to a weak label and a weight, the weak label indicates which text to be matched of the two texts to be matched contained in the corresponding sample is more relevant to the preset text, the numerical value of the initial weight is relevant to a difficulty index indicating the difficulty of the corresponding sample, and the sample with the larger difficulty is endowed with the smaller initial weight;

and A2, carrying out repeated iterative training on the semantic matching model by using the samples in the training set, so that the semantic matching model respectively outputs a correlation score of the predetermined text and a text pair formed by each text to be matched, and determining a weighting loss value according to the correlation score, the weak label and the weight to update the semantic matching model, wherein the weight of the sample is dynamically adjusted according to the currently finished training times.

2. The training method according to claim 1, wherein in the step A2, the weight of the corresponding sample is increased after each training for a predetermined number of times, wherein each time the weight of the sample is increased, the weight of the sample whose initial weight is relatively smaller is increased by a relatively larger value.

3. The training method according to claim 2, wherein in the step A2, after the number of times that training has been completed currently reaches a predetermined number threshold, the semantic matching model is trained for a plurality of times by setting weights of all samples to the same value.

4. Training method according to claim 3, characterized in that the loss value of a single sample is calculated as follows:

L＝max(0,ε-y×(S ₁ -S ₂ ))×w(τ,i)；

wherein max (·) represents a function taking the maximum value of the elements contained in the function, the output is the loss value of the sample, epsilon represents an interval parameter, y represents a weak label, y belongs to { -1, +1}, when y takes +1, the first text to be matched is more relevant to the predetermined text, and when y takes-1, the second text to be matched is more relevant to the predetermined text, S ₁ Representing the relevance score of the first text to be matched with the predetermined text, S ₂ And w (tau, i) represents the weight of the sample tau obtained by adjusting after i times of training.

5. The training method of claim 1, wherein the weight of the corresponding sample is increased after each predetermined number of training times, and the weight of the sample is dynamically adjusted according to the number of training times currently completed and the difficulty index of the sample.

6. Training method according to claim 5, characterized in that the weights of the samples are determined in the following way:

where i represents the number of times training has been currently completed, c represents a predetermined number threshold, D (τ) represents a difficulty function, | a _τ I represents the difficulty index of the sample tau, | a _n I represents the difficulty index of any sample in a set A, the set A represents a set formed by all samples of the current training batch or represents a training set, and max { | a _n And the [ epsilon ] is equal to A ] represents that the difficulty index of all samples of the current training batch or all samples in the training set takes the maximum value.

7. The training method according to any one of claims 1 to 5, wherein the difficulty index of the sample is an absolute value of a BM25 score generated by the BM25 model on the correlation between the first text to be matched and the predetermined text in the sample minus a BM25 score generated by the BM25 model on the correlation between the second text to be matched and the predetermined text in the sample.

8. The training method according to any one of claims 1-5, wherein the semantic matching model comprises a BERT model and a linear prediction layer, wherein the BERT model is used for forming a text pair according to the predetermined text in each sample and each text to be matched and outputting a semantic vector containing semantic similarity information; the linear prediction layer is used for outputting a correlation score of the text to be matched and the preset text according to the semantic vector.

9. Training method according to any of claims 1-5, wherein the method further comprises: before the semantic matching model is trained for the first time by using the training set, histogram equalization processing is carried out on difficulty indexes of samples in the training set, so that the difficulty indexes of the samples in the training set are uniformly distributed in a difficulty space.

10. A method of text matching, comprising:

b1, acquiring a query text and a plurality of candidate texts, and respectively forming a text pair by the query text and each candidate text;

b2, inputting each text pair into the semantic matching model obtained by training according to the training method of the semantic matching model of claims 1-8 respectively to obtain a correlation score between the query text and each candidate text;

and B3, sorting the candidate texts according to the relevance scores of the query texts and each candidate text, and outputting the corresponding candidate texts according to sorting results.

11. The text matching method according to claim 10, wherein the outputting of the corresponding candidate text according to the ranking result is at least one of:

candidate text with highest relevance score;

selecting a preset number of candidate texts according to the sequence of the relevance scores from big to small;

candidate texts with a relevance score greater than or equal to a predetermined score threshold.

12. A computer-readable storage medium, having stored thereon a computer program executable by a processor for performing the steps of the method of any one of claims 1-9 and 10-11.

13. An electronic device, comprising:

one or more processors; and

a memory, wherein the memory is to store executable instructions;

the one or more processors are configured to implement the steps of the method of any of claims 1-9 and 10-11 via execution of the executable instructions.

Images