RU2834217C1 - Method and system for generating response to search query - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: invention relates to information technology and specifically to search engines in general. Method of processing search requests, performed by at least one search system, comprising steps of: receiving a search request from a user; performing full-text search of documents based on request; generating a first set of search results; performing search request tokenization and vectorization; processing the search query vector representation using a machine learning model based on a neural network trained for semantic document search; generating a second set of search results based on the documents; combining the first and second sets of search results; ranking the combined first and second sets of search results, wherein during the ranking, determining the relevance of each document with respect to the search query; generating a generative search query result based on the obtained documents; sending the result to the user.

EFFECT: high accuracy of generating a search query result.

12 cl, 3 dwg

Description

AREA OF TECHNOLOGY

[1] This technical solution generally relates to search engines in general, and more specifically to a method for processing search queries to extract information on a user's search query and generate a shortened answer to it.

LEVEL OF TECHNOLOGY

[2] Currently, search engines are an integral part of global and local communication networks and provide relevant search query results. With the development of technologies in the field of artificial intelligence, the capabilities of search engines have also expanded due to the introduction of machine learning algorithms. Thus, search engines have begun to be used in the corporate sphere, for example, in the work processes of enterprises and/or companies with a large volume of documents.

[3] However, the use of search engines, such as search engines based on machine learning models, chatbots, etc., in the corporate sphere has a number of disadvantages. As a rule, in the corporate sphere there are certain areas of technology related to the specific activities of specific corporations and/or companies, which significantly complicates the use of known search engines for work purposes. Thus, due to the presence of domain-specific terms in search queries, known search engines show low completeness of search results (provided documents and/or search query results). This is due to the fact that search results often contain text that is semantically similar to the query, but incorrect in terms of interpreting highly specialized terms. For example, the queries "data migration from a database" and "data migration from s3" are semantically similar, but essentially refer to different concepts. In addition, machine learning algorithms in well-known search engines are trained on open data that does not contain or contains too little domain-specific training data (available methods allow achieving high completeness only with a large amount of domain-specific data). Therefore, the development of search engines in domain-specific terms is a significant and non-trivial task.

[4] The prior art includes approaches based on domain adaptation of search engines to provide search results for search queries containing domain-specific terms. Thus, such approaches include GPL (found on the Internet at the link: https://arxiv.org/abs/2112.07577) (Generative Pseudo Labeling). The GPL method refers to unsupervised learning methods, uses generative models to generate pseudo-queries for a text corpus and an existing search engine to find relevant documents. Ranking model labels are used as target labels for training the DPR (Dense Passage Retrieval, semantic search for retrieving relevant documents) model, for example, RocketQA, available at the link: https://arxiv.org/abs/2010.08191. This approach allows for additional training of the DPR model on synthetic data.

[5] It is worth noting that the ranking model is often a publicly available model that was trained on publicly available data that does not contain domain-specific concepts, which can negatively affect the quality of domain adaptation. This method also requires a fairly large domain-specific text corpus to achieve high-quality results, which is a serious limitation.

[6] Accordingly, the disadvantage of this approach is the low accuracy of providing search results for search queries containing domain-specific terms, which, as a consequence, reduces the quality of generating a response to a user query.

[7] A common drawback of known solutions is the lack of an effective way of processing search queries containing domain-specific terms performed by search engines, ensuring an increase in the completeness of the output and not requiring large sizes of the domain-specific text corpus to achieve high-quality results. In addition, such an approach must have low computational costs.

DISCLOSURE OF INVENTION

[8] This technical solution is aimed at eliminating the shortcomings inherent in existing solutions known from the state of the art.

[9] The technical problem solved in this technical solution is the creation of a new and effective way of processing search queries containing domain-specific terms in a search engine.

[10] The technical result that emerges from solving the above-mentioned problem is an increase in the accuracy of generating the search query result, due to an improvement in the relevance of the retrieved documents.

[11] The specified technical result is achieved by implementing a method for processing search queries performed by at least one search engine, containing the steps of:

a) receive a search query from the user;

b) carry out a full-text search of documents based on the query received in step a);

c) generate the first set of search results;

d) carry out tokenization and vectorization of the search query;

e) process the vector representation of the search query obtained in step d) using a machine learning model based on a neural network trained for semantic search of documents, and during the training process:

i. formation of a data set containing domain-specific terms and synonyms for the specified terms;

ii. creating an unlabeled dataset from a domain-specific text corpus;

iii. augmentation of unlabeled dataset by replacing domain-specific terms with their synonyms;

iv. training a machine learning model on the augmented unlabeled dataset;

f) generate a second set of search results based on the documents obtained in step e);

g) combine the first and second sets of search results;

h) rank the combined first and second sets of search results, wherein, during the ranking, the relevance of each document with respect to the search query is determined;

i) generate a generative search query result based on the received documents;

j) send the result obtained in step i) to the user.

[12] One particular example of the implementation of the method for training a machine learning model on an augmented unlabeled dataset contains the following steps:

i. generating search queries on a dataset using large language models;

ii. generation of complex negative examples by the existing search system;

iii. obtaining pseudo-labels of search query relevance based on the original text and negative examples using their ranking;

iv. formation of a data set including: search query; source text for which the query was generated; a set of complex negative examples; pseudo-relevance labels;

v. training a machine learning model on the received data set.

[13] In another particular example of implementing the method, during the combining of the first and second sets of search results, the positional location of each search result from each set in the combined data set is selected randomly.

[14] In another particular example of implementing the method, domain-specific terms are highly specialized terms characteristic of a specific application area.

[15] In another particular example of implementing the method, during ranking of the combined first and second sets of search results, the following steps are performed:

i. obtaining pairs of text sequences, where each pair includes a search query and a potentially relevant document;

ii. converting each pair into a real number corresponding to the degree of relevance of the document to the query;

iii. determining the order of ranking of documents based on a comparison of the real numbers of each pair.

[16] In another particular example of implementing the method, ranking is performed by at least one machine learning model based on a neural network trained on a data set containing domain-specific terms and synonyms for the specified terms.

[17] In another particular example of implementing the method, a full-text search of documents is carried out by means of at least a full-text search algorithm that takes into account the length of the document and the frequency of terms.

[18] In another particular example of implementing the method, the full-text document search algorithm additionally contains a data set that includes domain-specific terms and synonyms for the specified terms.

[19] In another particular example of the implementation of the method, the generative result is formed using a neural network.

[20] In another particular example of implementing the method, the generative result is a text sequence formed on the basis of a user query and documents relevant to it from the combined first and second set of search results.

[21] In another particular example of the implementation of the method, the vectorization of the search query is carried out by means of bidirectional coding.

[22] In addition, the stated technical results are achieved through a search query processing system containing:

• at least one processor;

• at least one memory connected to the processor that contains machine-readable instructions that, when executed by the at least one processor, provide for the execution of a method for processing search queries.

BRIEF DESCRIPTION OF DRAWINGS

[23] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[24] Fig. 1 illustrates an example of the implementation of a search query processing system.

[25] Fig. 2 illustrates a block diagram of the implementation of the claimed method.

[26] Fig. 3 illustrates an example of a computing device for implementing the claimed system.

IMPLEMENTATION OF THE INVENTION

[27] The claimed technical solution proposes a new approach that provides a new and efficient way of processing search queries containing domain-specific terms in a search engine. Aspects of the claimed technical solution provide for an increase in the completeness of search results by using both full-text search and semantic search trained on domain-specific terms, as well as an additional stage of ranking the obtained search results, which improves the relevance of the retrieved documents before generating a response. In addition, the claimed technical solution provides the ability to generate a response containing a condensed or generalized presentation of the most relevant results from the combined set of search query results.

[28] The claimed technical solution may be implemented, for example, by a system, a machine-readable medium, a server, etc. In this technical solution, a system means, among other things, a computer system, a computer (electronic computer), a CNC (numerical control), a PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, clearly defined sequence of operations (actions, instructions).

[29] A command processing unit is an electronic unit or integrated circuit (microprocessor) that executes machine instructions (programs).

[30] The command processing unit reads and executes machine instructions (programs) from one or more data storage devices, such as random access memory (RAM) and/or read-only memory (ROM). ROM may include, but is not limited to, hard disk drives (HDD), flash memory, solid-state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[31] A program is a sequence of instructions intended for execution by a computer control device or a command processing device.

[32] The term "instructions" as used in this application may refer generally to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, receiving and processing data, forming a user profile, receiving and transmitting signals, analyzing received data, identifying a user, etc. The instructions may be implemented in a variety of ways, including, for example, object-oriented methods. For example, the instructions may be implemented using the C++ programming language, Java, Python, various libraries (e.g., "MFC"; Microsoft Foundation Classes), etc. The instructions that perform the processes described in this solution may be transmitted via both wired and wireless data transmission channels, such as Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.

[33] Fig. 1 shows a general view of a system 100 for processing search queries in a search engine. The system 100 includes basic functional elements such as: an interaction module 101, a full-text search module 102, a text data tokenization and vectorization module 103, a semantic search module 104, a vector database 105, a ranking module 106, and a response generation module 107. The elements of the system 100 are disclosed in more detail in Fig. 3.

[34] The system 100 may be part of and/or associated with a search engine. Thus, in one particular embodiment, the system 100 may be an element of a search engine. The search engine may be a corporate search engine designed to search for data within a company, for example, based on a combined search on the Internet and the contents of the company's databases. As a rule, the search is intended for internal users of the company for work purposes and contains domain-specific terms in search queries related to the company's technical field. In another particular embodiment, the search engine may be a web service for searching text or graphic information, a search engine based on machine learning algorithms, for example, a search engine based on LLM, etc., a local or desktop search engine for searching local files, etc. In another particular embodiment, the search engine may be a combination of several types of search engines.

[35] Domain-specific terms are narrowly specialized terms characteristic of a certain application area. Thus, domain-specific terms may relate to an application area of the corporate sphere, for example, the area of information technology, networks and communication systems, cloud solutions, etc. Thus, domain-specific terms relate to specific vocabulary inherent in a certain area of technology, for example, medicine (biological terms, names of drugs, etc.), IT technologies (names of algorithms, systems, interaction protocols, etc.).

[36] The interaction module 101 can be implemented on the basis of at least one computing device equipped with appropriate software and is intended to process incoming search queries, coordinate the work of other modules, and provide output to the user.

[37] Module 101 may contain a user interface for interaction with users of the search engine. Coordination of the work of other modules may be performed, for example, by interacting with said modules via the HTTPS protocol (API call). Coordination should be understood as the sequence of execution by modules of system 100 of the steps of method 200, which is disclosed in more detail below. Thus, module 101 may ensure processing of a search request from a user, preparation of data for submodules 101-107 and determine the order of their traversal.

[38] The full-text search module 102 may be a full-text search system, a computing device, etc.

[39] Module 102 is designed for automated document retrieval based on a search query, where the search is conducted not by document names, but by their content, all or a significant part of the document content. Thus, in one particular embodiment, full-text search can be performed by the BM25 algorithm in combination with a data set containing domain-specific terms and synonyms for the specified terms (a synonym dictionary) to improve the search efficiency. The BM25 algorithm is an improved version of TF-IDF, which takes into account the length of the document and the frequency of terms, while the synonym dictionary helps improve the completeness of the search results.

[40] Module 102 operates on a reverse index data structure, which is a set of keys and values, where the key is a word and the value(s) are the indices of documents in which it occurs. Various functions can be used to rank candidate documents from the reverse index, such as TF-IDF, BM25, etc. Using a synonym dictionary (a data set containing domain-specific terms and synonyms for the specified terms) makes it possible to add synonymous terms to the reverse index and link relevant documents to them.

[41] It is worth noting that the names of technologies are transliterated into Russian, for example, “terraform” and “terraform”, have specific abbreviations, “virtual machine” and “vm”, etc. The dictionary of synonyms allows you to take into account the occurrence of synonymous terms in documents by adding them as keys to the reverse index.

[42] Let's look at an example of a full-text search working with a dictionary:

Original request: create master node

Modified query: create (masternode OR master OR masternode)

[43] As can be seen from the presented example, the use of a dictionary of synonyms expands the range of relevant documents covered, which, accordingly, increases the completeness of search results.

[44] The text data tokenization and vectorization module 103 may be implemented on the basis of at least one computing device equipped with appropriate software and include a set of models for tokenization and detokenization of text, vectorization of tokenized text and conversion of tokens into text, for example, one or more machine learning models for converting text information into vector form, for example, BERT, ELMo, ULMFit, XLNet, RoBerta, RuGPT3 and others.

[45] In one particular embodiment, module 103 may be implemented based on system 300, which is disclosed in more detail in Fig. 3. It is worth noting that a specific tokenization and vectorization method depends on the selected language model on which module 104 is implemented. For example, when using the RuGPT3 model, tokenization is performed using the BPE (Byte Pair Encoding) method, and subsequent vectorization is performed by replacing each token with its index in the language model dictionary compiled at the stage of initial model training. In addition, in another particular embodiment, word tokenization may be used as a tokenization method.

[46] An example of tokenization by words and encoding words with indexes in a dictionary: 'migration DB' → ['migration', 'DB'] → [235, 376].

[47] The semantic search module 104 can be implemented on the basis of at least one neural network, previously trained on specific data sets consisting, for example, of triplets (search query text, relevant document text, irrelevant document text), quintuplets (query text, relevant document text (positive example), positive example relevance assessment, irrelevant document text (negative example), negative example relevance assessment), etc. Thus, triplets can be used in cases where the triple data set already contains document relevance assessments, however, in practice this is unlikely.

[48] A model from the BERT-like model family, such as xlm-roberta-base, can be used as a machine learning model implementing the function of processing the vector representation of search queries and document texts. In one particular embodiment, when implementing the claimed solution, the ML was a multilingual model, such as multilingual-e5-large. The model was trained on sources from different domains: Wikipedia, books, news, multilingual Common Crawl, etc.

[49] In another particular embodiment, the model was also further trained on highly specialized domain texts, using various text augmentation methods, in particular synonym augmentation, and GPL techniques.

[50] In total, the dataset for additional training contained about 160,000 data sets consisting of quintuples (query text, relevant document text (positive example), positive example relevance score, irrelevant document text (negative example), negative example relevance score).

[51] Example of augmented text:

Source text: create vm

Augmented text options: create a virtual machine, create an ECS, create a virtual server, etc.

[52] To obtain neural network output, module 104 compares the distances between the user query in the form of a vector and the documents of the document base 105, also presented in the form of vectors.

[53] Vector database 105 is used to store and process vector information received from module 103. Said database 105 provides fast and efficient search by vectors, supports comparison and clustering operations, and provides the ability to scale and expand taking into account the needs of the data retrieval system (search engine).

[54] The ranking module 106 can be implemented on the basis of a computing device and is intended to evaluate the relevance of text sequences.

[55] Thus, module 106 performs the process of ranking search results in accordance with a user query among the sets of search results received from modules 102 and 104. Pairs of text sequences are input to module 106, where each pair includes a search query and a potentially relevant document from the database. Module 106 then converts this pair into a real number that reflects the degree of relevance of the document to the query. By comparing these numbers, the module determines the ranking order of the documents, which allows the user to be presented with the most relevant information in response to his query.

[56] Let us consider in more detail the operating principle of module 106.

[57] After receiving two outputs (semantic and full-text) from the semantic search 104 and full-text search 102 modules, the mixed output is fed to module 106. In one particular embodiment, module 106 may be implemented based on a neural network, such as a neural network for ranking: BERT-like models, for example, bert, roberta, etc., for example, bert-multilingual-passage-reranking-msmarco. The model receives a pair of a search query and a candidate document as input, and receives a real score (from -10 to 10, more is better) as output. After receiving ranking scores for each element of the mixed output, sorting is performed by the ranking score in descending order.

[58] It is worth noting that the said ranking model, in another particular embodiment, is further trained on a data set containing domain-specific terms and synonyms for the said terms. The said process allows for the leveling of multiple meanings of the same term. Thus, the same term may have several synonyms, which, when using the standard ranking model, will lead to an incorrect evaluation of the search query results. Consequently, to increase the completeness of the search results, the ranking model is further trained on synonymous concepts.

[59] For this purpose, a dictionary of synonyms for a specific domain-specific area is also used. In the declared technical solution, the model was further trained on assessor markup consisting of approximately 5,000 triplets (search query, document, relevance assessment).

[60] Module 107 can be implemented on the basis of a generative neural network and is intended to form a generative result of a search query.

[61] Module 107 is responsible for generating the final response based on the user's query and relevant documents from the knowledge base received from other modules. Module 107 converts the received data into a final response to the user's query in the form of a text sequence.

[62] Thus, in one particular embodiment, module 107 can be implemented on the basis of a model of the LLM type, which is pre-trained (additionally trained via text prompts) to respond to a user request based on the documents most relevant to the request. Thus, in another particular embodiment, the model can be the GigaChat machine learning model, but obviously other models can also be used (YaGPT, ChatGPT, open-source LLM).

[63] An example of additional training of the model using text cues from pre-trained language machine learning models might look like this: “Your role is to act as an information retrieval system. You will be asked a question and provided with relevant excerpts from various documents. Your task is to form a short and informative answer (no more than 150 words) based solely on the presented excerpts. Be sure to use information only from these excerpts. It is important to maintain a neutral and objective tone and to avoid repetition of text. At the end, form a final answer (“FINAL ANSWER”). Do not try to invent an answer. Answer exclusively in Russian, with the exception of specific terms. If the presented documents do not contain sufficient information to form an answer, say: “I cannot answer your question using the information in the provided documentation. Try to rephrase the question.” If the document contains information relevant to the query, but the query does not imply a direct answer, then simply restate the content of the relevant document.”

[64] Accordingly, based on the search results presented to module 107 and preliminary training with text prompts, module 107 will provide a short and informative answer based on the documents contained in the sample, i.e. it will combine the most relevant parts from different documents into a single answer (answer generation).

[65] By implementing the said system 100, an increase in the completeness of search results for a search query containing domain-specific terms is ensured.

[66] Let's consider the system quality metrics:

[67] As can be seen from Table 1, the implementation of the declared technical system for processing search queries, trained to search for relevant documents taking into account domain-specific terms, has a higher accuracy of finding relevant documents in the general array of documents than existing search systems. It is worth noting that the accuracy parameter given in Table 1 characterizes the percentage of documents relevant to the search query that are included in the search results.

[68] Fig. 2 shows a block diagram of a method 200 for processing search queries performed by at least one search engine and/or a system 100 that is part of a search engine. Said method 200 is disclosed step by step in more detail below. Said method 200 consists of performing steps aimed at processing various digital data, such as text data. The processing is typically performed using a system that can represent, for example, a server, a computer, a mobile device, a computing device, etc., supplemented by hardware and/or software and hardware units, for example, modules of the system 100 that implement the present method 200. As indicated above, the elements of the system are disclosed in more detail in Fig. 3.

[69] At step 201, the system receives a search query from the user. This step may be performed by module 101.

[70] Thus, at the indicated step 202, the system receives, for example, via a user interaction interface, a user query. The user query may be received in text form, for example, via a communication network, such as a global or local communication network. In one particular embodiment, the text query contains domain-specific terms. In another particular embodiment, the search query may be received via machine-to-machine interaction, for example from an enterprise and/or company system.

[71] At step 202, the system performs a full-text search of documents, performed by the full-text search module 102, based on the request received at step 201 and transmitted by the interaction module 101 to the module 102.

[72] As mentioned above, full-text search can be performed by the BM25 algorithm in combination with a dataset containing domain-specific terms and synonyms for the specified terms (synonym dictionary) to improve the search efficiency.

[73] To perform a full-text search, work is carried out with a reverse index data structure, which is a set of keys and values, where the key is a word, and the value/-s are the indices of documents in which it occurs. Various functions can be used to rank candidate documents from the reverse index, for example, TF-IDF, BM25, etc. When using a synonym dictionary (a data set containing domain-specific terms and synonyms for the specified terms), it is possible to add synonymous terms to the reverse index and link relevant documents to them. The process of forming a data set containing domain-specific terms and synonyms for the specified terms is described in more detail below.

[74] The result of performing step 202 is a set of relevant search query results.

[75] At step 203, the system generates a first set of search results.

[76] The specified set is formed based on the search results obtained at step 202.

[77] In one particular embodiment, the first set of search results may be sent to interaction module 101 for further aggregation and processing.

[78] The first set of search results contains the results of the full-text search.

[79] Next, the method 200 proceeds to step 204, using the interaction module 101 to transmit the first set of search results to the tokenization and vectorization module 103.

[80] At the specified stage 204, tokenization and vectorization of the search query is carried out by module 103, with subsequent transfer of the vectorization results to vector DB 105 for storage and search operation.

[81] Thus, tokenization and vectorization can be performed by models for tokenization and detokenization of text, vectorization of tokenized text and transformation of tokens into text, for example, by one or more machine learning models for transformation of text information into vector form, for example, BERT, ELMo, ULMFit, XLNet, RoBerta, RuGPT3 and others.

[82] In another particular embodiment, tokenization and vectorization of text may be performed using bidirectional encoding to process and analyze text sequences, which makes it possible to extract useful features from the data and transform them into a vector space, which can then be used for more complex operations such as comparison and clustering.

[83] At step 205, the vector representation of the search query obtained at step 204 and transmitted by the interaction module 101 to the semantic search module 104 is processed using a machine learning model based on a neural network trained for semantic search of documents, wherein during the training the following is performed: forming a data set containing domain-specific terms and synonyms for the specified terms; creating an unlabeled data set from a domain-specific corpus of texts; augmenting the unlabeled data set by replacing domain-specific terms with their synonyms; training the machine learning model on the augmented unlabeled data set.

[84] Let us consider in more detail the process of training the machine learning model of module 104.

[85] To form a data set containing domain-specific terms and synonyms for these terms, the first step is to identify the most important domain-specific terms.

[86] Thus, the specified step can be performed, for example, by analyzing corporate data and identifying the most frequently encountered terms. In another particular embodiment, such terms can be cleared of terms that do not carry a semantic load. Thus, for example, in the field of cloud technologies, such terms as virtual machine, firewall, container orchestrator, etc. can be identified. In another particular embodiment, a data set can be formed by comparing words used in a specific field of technology with a dictionary of the Russian language and identifying words that are not there and then checking such words for semantic load (conducting semantic analysis). Also, the formed set can be checked using an expert assessment.

[87] Next, a list of synonyms for the terms (words, phrases) is compiled, for example: Elastic Cloud Server, ECS, virtual server, Virtual machine, virtual machine, VM, VM; email, electronic mail, E-mail, e-mail, email, email, email, email; WAF, Web application firewall, firewall, firewall.

[88] In this way, a data set is formed containing domain-specific terms and synonyms for the specified terms (a dictionary of synonyms).

[89] Example of a dictionary of synonyms:

["Elastic Cloud Server, ECS, virtual server, Virtual machine, virtual machine, VM, VM",

"balancer, balancer, balancer",

"mlspace, mlspace, ml space, ml space",

"postgresql, PSQL, postgres, postgres, PostgreSQL",

"email, email, E-mail, e-mail, email, email, email, email",

"Initialization, Activation, Initiation, Initiate",

"Directory, Folder, Directory",

"Distributed Denial of Service Attack, DDoS, DDoS attack, DDoS",

"Denial of Service Attack, DoS, DoS attack, DoS",

"Namespace, Namespace, Namespace",

"Endpoint, endpoint, endpoint",

"Worker node, Worker node, Worker, Worker node",

"Master Node, Master, Master Node"]

[90] As indicated above, the dictionary of synonyms allows for the occurrence of synonymous terms in documents to be taken into account by adding them as keys to the inverse index. The specified dictionary of synonyms can then be used in both full-text search algorithms and ranking algorithms. In one particular embodiment, the specified generated set can be added to the full-text search algorithm for queries with all sorts of substitutions (replacements) of synonyms, for example, as follows: Create vm → create, vm, virtual machine, virtual server, vm.

[91] Next, to train the machine learning model for semantic document retrieval, an unlabeled dataset is created from a domain-specific text corpus.

[92] At this step, the unlabeled dataset obtained in the previous steps is augmented.

[93] Thus, artificial data reproduction (augmentation) can be carried out, for example, by replacing domain-specific terms with synonyms (replacing a term in the text with a synonym).

[94] This results in an augmented dataset containing domain-specific terms.

[95] Example of artificial expansion of a dataset:

Source text: create vm

Augmented text options: create a virtual machine, create an ECS, create a virtual server, etc.

[96] Next, the semantic search model is trained on unlabeled data. Training can be performed, for example, using the GPL technique with artificial expansion of the dataset with synonyms (replacement with a synonym).

[97] The training process involves generating search queries to a data set using large language models (LLM); generating complex negative examples using an existing search system; obtaining pseudo-labels of the relevance of a search query to the original text and negative examples, for example, using a ranking model; compiling data sets for training, such as fives (search query; original text for which the query was generated, a set of complex negative examples, pseudo-labels of relevance, relevance scores of examples); training a DPR model on the resulting data set.

[98] Thus, at step 205, semantic search of documents is performed by module 104 using a machine learning model based on a neural network trained on highly specialized domain texts, using various text augmentation methods, in particular augmentation with synonyms, and the GPL technique. In total, about 160,000 triplets (search query, document, pseudo-relevance score) were used in the dataset for additional training.

[99] At step 206, a second set of search results is generated based on the documents obtained at step 205.

[100] Thus, the specified set is the result of the search output of the semantic search model applied to the vector representation of the search query. The specified second set of results is independent. That is, the semantic search is carried out without taking into account the full-text search.

[101] The resulting second set is sent to interaction module 101 for aggregation.

[102] At step 207, for example, by means of the interaction module 101, the first and second sets of search results (203+206) are combined, with subsequent transmission of the combined set by the interaction module 101 to the ranking module 106.

[103] Thus, in one particular embodiment, the positional placement of each search result from each set in the combined data set is chosen randomly. This feature is necessary so that when ranking the results, the system does not give preference to the most relevant results from each type of search (full-text and semantic).

[104] At step 208, the module 106 ranks the combined first and second sets of search results, and during the ranking, the relevance of each document with respect to the search query is determined.

[105] Thus, at step 208, after receiving two outputs (semantic and full-text), the mixed output is fed to the ranking module 106, which contains a neural network for ranking (BERT-like models, for example, bert, roberta, etc.), for example, bert-multilingual-passage-reranking-msmarco. The model receives a pair of a search query and a candidate document as input, and at the output we receive a real score (from -10 to 10, more is better). After receiving ranking scores for each element of the mixed output, sorting is performed by the ranking score in descending order.

[106] The resulting relevance scores are sent to interaction module 101.

[107] Next, module 101 sorts the documents from the mixed output in descending order of relevance scores. The first K documents, for example, 6 documents, 10 documents, etc., from the sorted mixed output are sent along with the user query as a text sequence to response generation module 107.

[108] At step 209, based on the documents selected by the interaction module 101, a generative result of the search query is formed.

[109] At said step 209, the received data is converted into a final response to the user's request in the form of a text sequence, for example, by generating a response to the user's request.

[110] Thus, in one particular embodiment, the response generation module 107 can be implemented on the basis of a model of the LLM type, which is programmed (trained further by means of text prompts) to respond to a user request, based on the documents most relevant to the request. Thus, in another particular embodiment, the model can be the GigaChat machine learning model, but obviously other models can also be used (YaGPT, ChatGPT, open-source LLM).

[111] The generative result is a summary of, for example, several relevant sources and the generation of a response containing the most relevant answers, for example in the form of a condensed summary of the specified sources with the addition of relevant documents.

[112] Next, at step 210, the generated search query result is sent to the user device.

[113] Thus, the above materials described a method for recognizing data containing commercial secrets in text documents.

[114] Fig. 3 shows a system 300 that implements the steps of the claimed method 200 and implements the elements of the system 100.

[115] In general, the system 300 comprises components such as: one or more processors 301, at least one memory 302, data storage means 303, input/output interfaces 304, I/O means 305, network interaction means 306, which are connected via a universal bus.

[116] The processor 301 performs the basic computing operations necessary for processing the data when performing the method 200. The processor 301 executes the necessary machine-readable instructions contained in the RAM 302.

[117] Memory 302 is typically in the form of RAM and contains the necessary software logic to provide the required functionality.

[118] Data storage means 303 can be implemented in the form of HDD, SSD disks, RAID array, flash memory, optical storage devices (CD, DVD, MD, Blue-Ray disks), etc. Means 303 allow long-term storage of various types of information, for example, the first and second sets of search results, etc.

[119] To organize the operation of the components of system 300 and to organize the operation of external connected devices, various types of I/O interfaces 304 are used. The choice of the corresponding interfaces depends on the specific design of the computing device, which may include, but are not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[120] The selection of interfaces 304 depends on the specific implementation of system 300, which can be implemented on the basis of a wide range of devices, for example, a personal computer, mainframe, laptop, server cluster, thin client, smartphone, server, etc.

[121] The following may be used as I/O data means 305: keyboard, joystick, display (touch display), monitor, touch display, touchpad, mouse, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality means, optical sensors, tablet, light indicators, projector, camera, biometric identification means (retina scanner, fingerprint scanner, voice recognition module), etc.

[122] Network interaction means 306 are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, a WLAN/Wi-Fi module, a Bluetooth module, a BLE module, an NFC module, IrDa, an RFID module, a GSM modem, etc. With the help of means 606, the organization of data exchange is ensured between, for example, a system 300, presented in the form of a server and a user's computing device, on which the received data (anonymized text document) can be displayed via a wired or wireless data transmission channel, for example, a WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.

[123] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field.

Claims (36)

1. A method for processing search queries performed by at least one search engine, comprising the steps of: a) receive a search query from the user; b) carry out a full-text search of documents based on the query received in step a); c) generate the first set of search results; d) carry out tokenization and vectorization of the search query; e) process the vector representation of the search query obtained in step d) using a machine learning model based on a neural network trained for semantic search of documents, and during the training process: i) formation of a data set containing domain-specific terms and synonyms for the specified terms; ii) creation of an unlabeled dataset from a domain-specific text corpus; iii) augmentation of unlabeled dataset by replacing domain-specific terms with their synonyms; iv) training a machine learning model on the augmented unlabeled dataset; f) generate a second set of search results based on the documents obtained in step e); g) combining the first and second sets of search results; h) rank the combined first and second sets of search results, whereby the relevance of each document to the search query is determined during the ranking; i) generate a generative search query result based on the received documents; j) send the result obtained in step i) to the user. 2. The method according to item 1, characterized in that training the machine learning model on the augmented unlabeled data set comprises the following steps: i) generating search queries on a dataset using large language models; ii) generation of complex negative examples by the existing search system; iii) obtaining pseudo-labels of search query relevance based on the original text and negative examples using their ranking; iv) formation of a data set including: a search query; the source text for which the query was generated; a set of complex negative examples; pseudo-relevance labels; v) training a machine learning model on the obtained data set. 3. The method according to claim 1, characterized in that during the combining of the first and second sets of search results, the positional location of each search result from each set in the combined data set is selected randomly. 4. The method according to paragraph 1, characterized in that the domain-specific terms are highly specialized terms characteristic of a certain application area. 5. The method according to paragraph 1, characterized in that during the ranking of the combined first and second sets of search results, the following steps are performed: i.) obtaining pairs of text sequences, where each pair includes a search query and a potentially relevant document; ii) converting each pair into a real number corresponding to the degree of relevance of the document to the query; iii) determining the ranking order of documents based on a comparison of the real numbers of each pair. 6. The method according to claim 1, characterized in that the ranking is performed by at least one machine learning model based on a neural network trained on a data set containing domain-specific terms and synonyms for the specified terms. 7. The method according to paragraph 1, characterized in that the full-text search of documents is carried out by means of at least a full-text search algorithm that takes into account the length of the document and the frequency of terms. 8. The method according to paragraph 7, characterized in that the algorithm for full-text search of documents additionally contains a data set including domain-specific terms and synonyms for the specified terms. 9. The method according to paragraph 1, characterized in that the generative result is formed using a neural network. 10. The method according to claim 1, characterized in that the generative result is a text sequence formed on the basis of a user query and documents relevant to it from the combined first and second sets of search results. 11. The method according to item 1, characterized in that the vectorization of the search query is carried out by means of bidirectional coding. 12. A search query processing system containing: • at least one processor; • at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, ensure the execution of the method according to any one of claims 1-11.