CN118193726A - Visual patent retrieval method based on pre-training language model - Google Patents

Abstract

The present invention relates to the technical field of natural language processing, and specifically to a visual patent search method based on a pre-trained language model. In a patent keyword search scenario, the present invention extracts keywords and represents word vectors for patent texts based on a pre-trained language model, reduces the dimension of patent features returned by the search, and visualizes the search results in the form of a scatter plot. Compared with the search results displayed in a list form, the method is more intuitive and provides richer search information such as the degree of similarity between patents and patent clustering.

Description

A visual patent retrieval method based on pre-trained language model

Technical Field

The present invention relates to the technical field of natural language processing, and in particular to a visual patent retrieval method based on a pre-trained language model.

Background technique

The primary task of patent search is to retrieve relevant patent texts from the massive patent database. At present, the more commonly used patent text search strategy is keyword search, which constructs a search formula based on the keywords to be searched and retrieves relevant patent texts from the patent database.

The keyword search strategy has the possibility of missing detection, and it is easy to miss a certain amount of important patent texts with high technical similarity, which makes it difficult to achieve high-precision patent search requirements. Therefore, in order to solve the above-mentioned technical problems, patent CN112000783A discloses a patent recommendation method, device, equipment and storage medium based on text similarity analysis. The patent recommendation method obtains the main keywords and descriptive keywords, and uses the main keywords and all descriptive keywords as search terms to obtain a basic similar text set, and then uses the main keywords and each descriptive keyword as search terms to obtain an extended similar text set. Then, the extended similar text set is traversed, and for each extended similar text, the similarity between the extended similar text and the basic similar text in the basic similar text set is calculated based on the text feature words of the extended similar text and the search terms corresponding to the extended similar text; and when the similarity between the extended similar text and any basic similar text in the basic similar text set is higher than a predetermined threshold, the extended similar text is moved into the basic similar text set. The patent recommendation method can improve the recall rate of similar texts and reduce the missed detection rate.

Although the above existing technologies can improve the recall rate to a certain extent, there are still the following problems in actual use:

1. The search results mainly show a rough ranking of similarities in digital form, which makes it difficult to intuitively show the differences between the different similarities. In addition, the similarity value of each patent is attached in the search results, which is not intuitive enough for a long list. You need to flip through the list to see the corresponding similarity value.

2. In the displayed search result list, the similarity between the retrieved patent texts is not calculated. Therefore, for the search personnel, it is impossible to intuitively reflect the similarity between the patent texts.

In summary, there is still much room for improvement in the current patent search results.

Summary of the invention

In order to avoid and overcome the technical problems existing in the prior art, the present invention provides a visual patent search method based on a pre-trained language model. The present invention can clearly and intuitively display the similarity relationship between the search keywords and the retrieved patent texts, as well as the similarity relationship between the retrieved patent texts.

To achieve the above object, the present invention provides the following technical solutions:

A visual patent search method based on a pre-trained language model includes the following search steps:

S1. Use patent texts annotated with existing keywords to train a RoBERTa+Bi-LSTM model to extract keywords from patents.

S2, input the patent text in the patent database into the RoBERTa+Bi-LSTM model in S1 according to the specified format, extract multiple keywords and word vectors of these keywords, and add multiple word vectors as the high-dimensional vector representation of the patent text;

S3, input multiple keywords to be searched into the RoBERTa model in S1 according to the specified format, obtain the word vector of the keyword, and add the vectors of multiple keywords as the high-dimensional vector representation of the query text;

S4, sequentially calculating the cosine similarity between the query text vector in S3 and the high-dimensional vectors of all patent texts in the patent database processed by S2; selecting patent texts with cosine similarity less than a set threshold as retrieval candidate results;

S5, the query text vector in S3 and the text vector of the candidate patents obtained in S4 are input into the Barnes-Hut t-SNE model for dimension reduction, both of which are reduced to 2 dimensions;

S6. The query text vector and the candidate patent text vector after dimension reduction are presented in the form of points in a two-dimensional plane to form a scatter plot. The similarity relationship between the search results is visualized by the distance between the points in the scatter plot.

As a further solution of the present invention: before inputting the patent text in the patent database into the RoBERTa model, it is necessary to perform data processing on the patent text and convert the input format of the patent text into T1:

T1 = ([CLS], TITLE, [SEP], ABSTRACT, [SEP], IPC_TEXT, [SEP], MAIN_TEX T);

Among them, [CLS] is a placeholder to mark the beginning of the text; [SEP] is a separator; TITLE indicates the position of the patent name after the text sequence in the patent text is expanded; ABSTRACT indicates the position of the patent specification abstract after the text sequence in the patent text is expanded; IPC_TEXT here indicates the position of the patent IPC classification number after the text sequence in the patent text is expanded; MAIN_TEXT indicates the position of the patent invention content after the text sequence in the patent text is expanded. This input format can make full use of the structured information in the patent text.

As a further solution of the present invention: there are one or more keywords to be searched, and each keyword to be searched is arranged in sequence to form a keyword sequence to be searched, and the keyword sequence to be searched is input into the RoBERTa model; before inputting the keyword sequence to be searched into the RoBERTa model, it is necessary to perform data processing on the keywords to be searched, and convert the input format of the keywords to be searched into T2, T2 = ([CLS], IPC_TEXT, [SEP], KEYWORD_1, [SEP], KEYWORD_2, ..., [SEP], KEY WORD_N), wherein the IPC_TEXT here represents the position of the patent IPC classification number after the keyword sequence to be searched is expanded; KEYWORD_1 represents the position of the first keyword to be searched after the keyword sequence to be searched is expanded; KEYWORD_2 represents the position of the second keyword to be searched after the keyword sequence to be searched is expanded; KEYWORD_N represents the position of the Nth keyword to be searched after the keyword sequence to be searched is expanded.

As a further solution of the present invention: when using the RoBERTa model to perform corresponding splicing on the keywords in the patent text or the keywords to be searched in the keyword sequence to be searched, the vectors output by the last 4 hidden layers in the RoBERTa model are selected for splicing to obtain the corresponding spliced vectors.

As a further solution of the present invention: the Barnes-Hut t-SNE algorithm is used when performing dimensionality reduction.

As a further solution of the present invention: the word vector of each keyword in the sequence state vector and the word vector of each keyword to be retrieved have a dimension of 3072. After dimensionality reduction processing by the Barnes-Hut t-SNE algorithm, the corresponding dimension is reduced from 3072 to 2.

As a further solution of the present invention: IPC_TEXT in the input format of the keyword to be searched can be retained or deleted according to the search requirements; when it is necessary to limit the IPC classification number, it is retained, otherwise it is deleted.

Compared with the prior art, the present invention has the following beneficial effects:

1. In the patent keyword search scenario, the present invention extracts keywords and represents word vectors for patent texts based on a pre-trained language model, reduces the dimensionality of patent features returned by the search, and visualizes the search results in the form of a scatter plot. Compared with the search results displayed in a list form, it is more intuitive and provides richer search information such as the degree of similarity between patents and patent clustering.

2. After the present invention uses a pre-trained model to extract the keywords of the patent, the word vector corresponding to the keyword is used as the feature representation of the patent, rather than directly storing the keyword or further vectorizing the patent into text. The same processing is done for the keywords to be retrieved. The word vector of the keyword is used as the feature representation. On the one hand, it provides a basis for dimensionality reduction visualization. On the other hand, it also provides users with the function of controlling keyword weights and thus adjusting search preferences.

3. The present invention introduces the Barnes-Hutt-SNE algorithm to reduce the dimensionality of high-dimensional word vectors. Compared with the common PCA dimensionality reduction, the Barnes-Hut t-SNE algorithm does not seek dimensionality reduction through linear transformation of features, and has a better visualization effect after the high-dimensional word vector is reduced to 2 dimensions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of the main process of the present invention.

FIG2 is a flowchart of generating keyword vectors in the present invention.

FIG3 is a similarity calculation flow chart of the present invention.

FIG. 4 is a scatter plot in the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Please refer to Figure 1. In an embodiment of the present invention, a visualized patent search method based on a pre-trained language model is provided. The present invention aims to visualize patent search results and is mainly suitable for keyword search scenarios. For keywords in patent texts, in addition to the keywords given in the patent items themselves, keywords can also be extracted from the patent texts. This process can be completed using the pre-trained language model that is popular in the field of natural language processing. The present invention uses the RoBERTa model, which is trained on Chinese corpus and is more suitable for Chinese-related scenarios. Keyword extraction can be regarded as a sequence labeling problem. A BiLSTM layer is added after the RoBERTa model to process the sequence output by the RoBERTa model to determine which positions in the sequence are keywords.

Since patent texts are usually standardized and have paragraphs such as patent name, background technology, abstract, and invention content, the original input format of the RoBERTa model is adjusted by utilizing this information, and these structured information are used as a supplement to the context. Furthermore, considering that the same keyword has different meanings in different fields, the Chinese meaning corresponding to the patent classification number is also put into the context when supplementing the context. The final text input format used is T1, T1 = ([CLS], TITLE, [SEP], ABSTRACT, [SEP], IPC_TEXT, [SEP], MAIN_TEXT), where [CLS] is a placeholder that marks the beginning of the text, [SEP] is a separator, and TITLE, ABSTRACT, IPC_TEXT, and MAIN_TEXT are the positions of the text sequence in the patent after expansion.

Use patent text with keyword position annotated, process it into the above input format, and then fine-tune the RoBERTa+Bi-LSTM model to obtain a keyword extraction model suitable for patent text scenarios.

After the keyword extraction model of the above-mentioned RoBERTa+BiLSTM structure is trained, the patent text is input to extract the keywords in the patent. It is also necessary to represent the keywords in the form of word vectors. This step can be obtained directly from the RoBERTa model. As shown in Figure 2, the present invention selects the hidden layer vectors of the last 4 layers in the RoBERTa model and concatenates them as the word vector representation of the keywords at the corresponding position in the sequence. The dimension of the hidden vector of each layer is 768, and the dimension of the word vector of each keyword obtained is 768x 4=3072, which is a very high dimension. The above model is used to extract keywords for all patent texts in the patent database and save them in the form of word vectors as keyword features of the patent.

As shown in Figure 3, during the search, for one or more keywords input by the user, the present invention constructs one or more keywords into a query text in the format of ([CLS], IPC_TEXT, [SEP], KEYWORD_1, [SEP], KEYWORD_2, ..., [SEP], KEYWORD_N) and inputs it into the RoBERTa model. At this time, there is no need to use the following BiLSTM layer, where IPC_TEXT is an optional part. If the user specifies the patent classification field to be queried, the IPC is added to the context and aligned with the training process of the above model to provide more complete context information. Afterwards, the same method as in the previous paragraph is used to extract the latent vectors of the last 4 layers of the RoBERTa model as the word vector representation of the search keyword.

The word vectors obtained from the above pre-trained model have the characteristics of context-relatedness. For example, the word apple has two representations in "apples on the fruit stand" and "Apple Company"; and to a certain extent, the addition between word vectors also shows the property of conforming to semantic expression, such as the word vector of "crown" + "male" will be close to the word vector representation of "king". Based on these properties, the present invention performs weighted summation on the vectors converted from multiple keywords provided by the user. By default, the weights are equal. Users can adjust the weights of each keyword to adjust their own search preferences.

Next, in the patent database, we first preliminarily screen out patents containing the keyword provided by the user (if there are multiple keywords, just one is sufficient), as shown in Figure 3, and sum up the multiple keyword vectors of the candidate patents to serve as the feature vector of the patent. Then, we compare the word vector of the search keyword with the feature vector of the candidate patent, calculate the cosine distance, and select the K patents with the closest distance as the search results to be returned.

After obtaining the K patents returned by the keyword search, visualization processing is finally performed. Since the word vector and patent feature vector dimensions of the search keyword are both 3072, the dimension is too high and needs to be reduced. The present invention adopts the Barnes-Hut t-SNE algorithm to reduce the above high-dimensional word vector to 2 dimensions. For samples with high-dimensional features, the Barnes-Hut t-SNE algorithm can retain the local properties of samples in the high-dimensional space after dimensionality reduction, that is, samples with similar distances will be closer after dimensionality reduction, and samples with farther distances will be farther after dimensionality reduction, which reduces the computational complexity and improves the computational speed, but is limited to obtaining 2-dimensional or 3-dimensional representations, which just meets the visualization requirements. After reducing the search keyword vector and the candidate patent feature vector to 2 dimensions, a scatter plot is used for visualization. In the final scatter plot, as shown in Figure 4, the closer the patent is to the search keyword point intuitively, the higher the degree of similarity; and the distance between the points of each patent also intuitively represents the degree of similarity between them; and in the returned patent results, a certain cluster may be presented, and some patents are closer to a certain topic than other patents, providing more abundant information for the searcher.

The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims (7)

1. A visual patent search method based on a pre-trained language model, characterized by comprising the following search steps: S1. Use patent texts annotated with existing keywords to train a RoBERTa+Bi-LSTM model to extract keywords from patent texts. S2, input the patent text in the patent database into the RoBERTa+Bi-LSTM model trained in S1 in the specified format, extract multiple keywords and word vectors of these keywords, and add multiple word vectors as the high-dimensional vector representation of the patent text; S3, input multiple keywords to be searched into the RoBERTa model in S1 according to the specified format, obtain the word vector of the keyword, and add the vectors of multiple keywords as the high-dimensional vector representation of the query text; S4, sequentially calculating the cosine similarity between the query text vector in S3 and the high-dimensional vectors of all patent texts in the patent database processed by S2; selecting patent texts with cosine similarity less than a set threshold as retrieval candidate results; S5, the query text vector in S3 and the text vector of the candidate patents obtained in S4 are input into the Barnes-Hut t-SNE model for dimension reduction, both of which are reduced to 2 dimensions; S6. The query text vector and the candidate patent text vector after dimension reduction are presented in the form of points in a two-dimensional plane to form a scatter plot. The similarity relationship between the search results is visualized by the distance between the points in the scatter plot. 2. According to claim 1, a visual patent retrieval method based on a pre-trained language model is characterized in that before the patent text in the patent database is input into the RoBERTa model, the patent text needs to be processed and the input format of the patent text is converted into T1, T1 = ([CLS], TITLE, [SEP], ABSTRACT, [SEP], IPC_TEXT, [SEP], MAIN_TEXT), wherein [CLS] is a placeholder for identifying the beginning of the text; [SEP] is a separator; TITLE indicates the position of the patent name after the text sequence in the patent text is expanded; ABSTRACT indicates the position of the patent specification abstract after the text sequence in the patent text is expanded; IPC_TEXT here indicates the position of the patent IPC classification number after the text sequence in the patent text is expanded; MAIN_TEXT indicates the position of the patent invention content after the text sequence in the patent text is expanded. 3. According to claim 3, a visual patent search method based on a pre-trained language model is characterized in that there are one or more keywords to be searched, and each keyword to be searched is arranged in sequence to form a keyword sequence to be searched, and the keyword sequence to be searched is input into the RoBERTa model; before inputting the keyword sequence to be searched into the RoBERTa model, it is necessary to perform data processing on the keywords to be searched, and convert the input format of the keywords to be searched into T2, T2 = ([CLS], IPC_TEXT, [SEP], KEYWORD_1, [SEP], KEYWORD_2, ..., [SEP], KEY WORD_N), wherein the IPC_TEXT here represents the position of the patent IPC classification number after the keyword sequence to be searched is expanded; KEYWORD_1 represents the position of the first keyword to be searched after the keyword sequence to be searched is expanded; KEYWORD_2 represents the position of the second keyword to be searched after the keyword sequence to be searched is expanded; KEYWORD_N represents the position of the Nth keyword to be searched after the keyword sequence to be searched is expanded. 4. A visual patent retrieval method based on a pre-trained language model according to any one of claims 2-3, characterized in that when using the RoBERTa model to perform corresponding splicing on the keywords in the patent text or the keywords to be searched in the keyword sequence to be searched, the vectors output by the last 4 hidden layers in the RoBERTa model are selected for splicing to obtain the corresponding spliced vectors. 5. According to the visual patent retrieval method based on a pre-trained language model described in claim 5, it is characterized in that the Barnes-Hutt-SNE algorithm is used when performing dimensionality reduction. 6. According to claim 5, a visual patent retrieval method based on a pre-trained language model is characterized in that the dimension of the word vector of each keyword in the sequence state vector and the word vector of each keyword to be searched is 3072. After dimensionality reduction processing by the Barnes-Hut t-SNE algorithm, the corresponding dimension is reduced from 3072 to 2. 7. According to claim 3, a visual patent search method based on a pre-trained language model is characterized in that the IPC_TEXT in the input format of the keyword to be searched can be retained or deleted according to the search requirements; when the IPC classification number needs to be limited, it is retained, otherwise it is deleted.