CN117272073B - Text unit semantic distance precalculation method and device, query method and device - Google Patents

Abstract

The invention discloses a text unit semantic distance pre-calculation method and device, and a query method and device, wherein the pre-calculation method comprises the following steps: acquiring all text units in a pre-calculation knowledge base, and acquiring an associated text unit set of each text unit based on an associated unit acquisition mode; acquiring knowledge representations of all object attribute text units in a pre-calculation knowledge base based on an associated text unit set by a preset object knowledge representation acquisition mode, and acquiring knowledge representations of all category attribute text units in the pre-calculation knowledge base by a preset category knowledge representation acquisition mode; based on knowledge representation of text units, calculating semantic distances of all text unit pairs in a text unit relation determining mode, and collecting all the text unit pairs with the calculated semantic distances and the corresponding semantic distances into a semantic distance library of a pre-calculation knowledge base. The semantic distance calculation process has no vector embedding, chunk process and is lossless to the original data information.

Description

Text unit semantic distance precalculation method and device, query method and device

Technical field

The invention relates to the technical field of knowledge base query, to a text unit semantic distance pre-calculation method, and in particular to a text unit semantic distance pre-calculation method and device, query method and device.

Background technique

Pre-training is a deep learning model training strategy. The purpose is to extract as many common features as possible from a large amount of data through training, and then apply them to specific task models, and then use a small amount of annotated data in relevant specific fields. "Fine-tuning" allows the model to only "learn" the "special" part of the specific task starting from the "common characteristics". However, pre-training requires the use of a large amount of data for training and learning. Due to the influence of the quality of the training data itself, the accuracy of the results is not necessarily high, and the training cost is also high. Moreover, large models use historical data for training and cannot be updated in real time.

There are currently some large model integration frameworks that can combine large models with external data to achieve real-time updates; however, the training and inference costs of large model integration frameworks are relatively high. Specifically, the external data needs to be converted into vectors and stored in a data storage that supports the vector search function. When using it, the similarity of the vectors needs to be calculated. During the vector storage process, an index needs to be built, which requires a large amount of calculation and costs time and computing power. are relatively high. At the same time, in the large model integration framework, the data embedding process and data are too long, and the data needs to be divided into chunks. In the process, the original information is damaged, and the semantic information and contextual relationship of the data are lost, resulting in the final query results and The original data is not relevant.

Therefore, for data from pre-training and large model integration frameworks, a more convenient, fast and accurate data retrieval method that does not require segmentation of data is needed to avoid damaging the original information and losing the semantic information and contextual relationships of the data.

Contents of the invention

The purpose of this application is to provide a text unit semantic distance pre-calculation method and device, query method and device, to reduce training costs and improve accuracy; to solve the problem that the existing large model integration framework requires a large amount of external data calculation and requires Data segmentation damages the original information and loses the semantic information and contextual relationship of the data, resulting in the problem that the final query result is not relevant to the original data.

In the first aspect, this application provides a text unit semantic distance precalculation method, including:

Obtain all text units in the precomputed knowledge base, and obtain the associated text unit set of each text unit based on the associated unit acquisition method;

Through the preset object knowledge representation acquisition method, the knowledge representation of all object attribute text units in the precomputed knowledge base is obtained based on the associated text unit set, and through the preset category knowledge representation acquisition method, the knowledge representation in the precomputed knowledge base is obtained. Knowledge representation of all category attribute text units;

Obtain all text unit pairs that can be formed by all the text units. Based on the knowledge representation of the text unit, calculate the semantic distance of all the text unit pairs by determining the text unit relationship, and combine all the calculated semantic distances. The text unit pairs and the corresponding semantic distance set are the semantic distance database of the precomputed knowledge base;

Wherein, the object attribute text unit is an object in the precomputed knowledge base, and the category attribute text unit is a category in the precomputed knowledge base.

In an embodiment of the present application, obtaining the associated text unit set of the text unit based on the associated unit acquisition method includes:

Obtain description pages of regular text units from the precomputed knowledge base;

Treat the text unit in the description page as the internal text unit of the regular text unit, and set all types of internal text units of the regular text unit as a set of associated text units of the regular text unit;

Wherein, the regular text unit is any text unit in the precomputed knowledge base.

In one embodiment of the present application, obtaining the knowledge representation of the object attribute text unit in the precomputed knowledge base based on the associated text unit set through a preset object knowledge representation acquisition method includes:

Use the object attribute text unit as the filtering unit to filter the associated text unit set it faces, and assemble the text units corresponding to the associated text unit set that meet the filtering conditions into the knowledge representation of the object attribute text unit;

Wherein, the object attribute text unit is any object in the precomputed knowledge base; the set of associated text units faced by the object attribute text unit is included in the precomputed knowledge base, except for the association corresponding to the object attribute text unit. All other associated text unit sets except the text unit set; the filtering condition is that the associated text unit set contains the filtering unit.

In an embodiment of the present application, obtaining the knowledge representation of a single category attribute text unit in the precomputed knowledge base through a preset category knowledge representation acquisition method includes:

Obtain the object attribute text unit belonging to the category attribute text unit as the object text unit, and collect the knowledge representation of all the object text units of the category attribute text unit into the knowledge representation of the category attribute text unit;

The category attribute text unit is any category in the precomputed knowledge base.

In one embodiment of the present application, calculating the semantic distance of the text unit pair by determining the text unit relationship includes:

Setting one of the text units in the text unit pair as the first text unit and the other text unit as the second text unit;

Determine whether there is an intersection between the knowledge representation of the first text unit and the knowledge representation of the second text unit. If so, it means that the first text unit and the second text unit are related, and based on the first text unit The knowledge representation and the knowledge representation of the second text unit calculate the semantic distance between the first text unit and the second text unit, otherwise it means that there is no relationship between the first text unit and the second text unit.

In an embodiment of the present application, based on the knowledge representation of the first text unit and the knowledge representation of the second text unit, the Ochiia coefficient calculation method or the Jaccard index calculation method is used to calculate the first text unit and the second text unit. The semantic distance between two text units.

In the second aspect, this application provides a text unit semantic distance pre-calculation device, including an associated text unit acquisition module, a knowledge representation acquisition module and a semantic distance library acquisition module:

The associated text unit acquisition module is used to acquire all text units in the precomputed knowledge base, and acquire the associated text unit set of each text unit based on the associated unit acquisition method;

The knowledge representation acquisition module is used to acquire the knowledge representation of all object attribute text units in the precomputed knowledge base based on a preset object knowledge representation acquisition method, and acquire the precomputed knowledge based on a preset category knowledge representation acquisition method. Knowledge representation of all category attribute text units in the library;

The semantic distance library acquisition module is used to obtain all text unit pairs that can be formed by all the text units. Based on the knowledge representation of the text unit, the semantic distance of all the text unit pairs is determined by a text unit relationship determination method. Perform calculations to collect all pairs of text units for which semantic distances are calculated and the corresponding semantic distances into a semantic distance library of the precomputed knowledge base;

Wherein, the object attribute text unit is an object in the precomputed knowledge base, and the category attribute text unit is a category in the precomputed knowledge base.

In the third aspect, this application provides a knowledge base text unit query method, including:

Get the text unit to be queried;

Search all text unit pairs with the text unit to be queried from the semantic distance database of the knowledge base as the query text unit pairs of the text unit to be queried, and combine all or part of the query text unit pairs based on the corresponding semantic distance. The collection of Chinese and African text units to be queried is the query result list;

Wherein, the non-query text unit in the query text unit pair is the other text unit in the query text unit pair except the text unit to be queried, and the semantic distance database of the knowledge base passes through the text unit. The unit semantic distance precomputation method is obtained.

In one embodiment of the present application, the semantic distance database of the text unit to be queried is searched from the semantic distance database as the semantic distance database to be queried, and all or part of the text units are obtained from the semantic distance database to be queried based on the semantic distance. The steps to form a query result list include:

The language type of the text unit to be queried is set to the first type of language, and the language type of the knowledge base is set to the second type of language;

If the first type of language is the same as the second type of language, then determine whether the corresponding identifier of the text unit to be queried is unique, and if so, search all texts with the text unit to be queried from the semantic distance database of the knowledge base. The unit pair is a query text unit pair of the text unit to be queried, and based on the corresponding semantic distance, all or part of the query text unit pairs that are not text units to be queried are collected into a query result list; otherwise, the text to be queried is obtained Advanced qualification conditions of the unit, and based on the advanced qualification conditions, determine the identification of the unit to be queried from all identifications corresponding to the text unit to be queried, and search all the identification of the unit to be queried from the semantic distance database of the knowledge base The text unit pairs serve as the query text unit pairs of the unit identifiers to be queried, and all or part of the query text unit pairs that are not corresponding to the unit identifiers to be queried are assembled into a query result list based on the corresponding semantic distance;

If the first type of language is different from the second type of language, convert the text unit to be queried into an equivalent text unit of the second type of language, and search for all the equivalent text units from the semantic distance database of the knowledge base. Text unit pairs of text units are used as query text unit pairs of the equivalent text units. Based on the corresponding semantic distance, non-equivalent text units in all or part of the query text unit pairs are used as equivalent query units. All the equivalent query units are The unit is converted into query results in the first language, and all the query results are assembled into a query result list;

Each text unit of meaning in the knowledge base has a unique identifier.

In the fourth aspect, this application provides a knowledge base text unit query device, including a query unit acquisition module and a query result acquisition module;

The query unit acquisition module is used to obtain the text unit to be queried;

The query result acquisition module is used to search for all text unit pairs with the text unit to be queried as the query text unit pairs of the text unit to be queried from the semantic distance database of the knowledge base, and based on the corresponding semantic distance size, The set of non-query text units in all or part of the query text units is a query result list;

Wherein, the non-query text unit in the query text unit pair is the other text unit in the query text unit pair except the text unit to be queried, and the semantic distance database of the knowledge base passes through the text unit. The unit semantic distance precomputation method is obtained.

Compared with the existing technology, one or more embodiments of the above solutions may have the following advantages or beneficial effects:

By applying the text unit semantic distance pre-calculation method provided by the embodiment of the present invention, the internal text units in the description page are queried, and then all text units with the same internal text unit are integrated to obtain the knowledge representation of each text unit. , and finally obtain the semantic distance library of text units based on knowledge representation. There is no vector embedding or chunking process in the semantic distance acquisition process, which does not damage the original information of the data and improves the accuracy. And the acquisition process of the text unit semantic distance library is the process of text unit pre-calculation. The results obtained by this process can be directly saved in various data storage or databases for query and use. Compared with the existing pre-training and large model integration frameworks It saves the time of vector embedding and calculation, that is, it saves the time cost and computing power cost of large model training; it helps to achieve semantic alignment faster during training and improve the reasoning ability of large models.

By applying the knowledge base text unit query method provided by the embodiment of the present invention, the text unit query can be realized. Furthermore, the query of text data in different languages can be quickly realized, and the query of text units with multiple semantics can also be quickly realized.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and obtained by the structure particularly pointed out in the written description, claims and appended drawings.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation of the present invention. In the attached picture.

Figure 1 shows a schematic flowchart of the text unit semantic distance pre-calculation method according to the embodiment of the present application.

Figure 2 shows a schematic structural diagram of a text unit semantic distance pre-calculation device according to an embodiment of the present application.

Figure 3 shows a schematic flow chart of the knowledge base text unit query method according to the embodiment of the present application.

Figure 4 shows a schematic structural diagram of the knowledge base text unit query device according to the embodiment of the present application.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples, so that the implementation process of how to apply technical means to solve technical problems and achieve technical effects of the present invention can be fully understood and implemented accordingly. It should be noted that as long as there is no conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other, and the resulting technical solutions are within the protection scope of the present invention.

It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the drawings only show the components related to the present application and do not follow the actual implementation of the component numbers, shapes and components. Dimension drawing, in actual implementation, the type, quantity and proportion of each component can be arbitrarily changed, and the component layout type may also be more complex.

Data curation refers to the activity of organizing and integrating data collected from different sources to build a trustworthy data resource library. It includes a series of full life cycle activities for effective management of data such as collection, screening, evaluation, preservation, maintenance, and utilization. Data curation can be done manually or processed by machines. The quality of data that has been curated is generally higher.

Knowledge base is a tool for storing and managing knowledge. It can be used to support applications such as knowledge systems, knowledge discovery, and knowledge sharing. Text units (i.e. tokens) include words, phrases and sentences. In a knowledge base, it refers to objects and categories, that is, the words, phrases, and sentences that make them up. It can be a specific token object, such as "Cogito, ergo sum", "Homohomini lupus", "Knowledge is power", "NBA Finals Most Valuable Player Award", etc., or it can be a token category, such as "NBA Finals", "Chinese actors" etc.

Knowledge representation (KR for short) refers to abstracting things, concepts, relationships, etc. in the real world into a form that can be processed by computers, so that computers can understand and process this information.

Semantic distance refers to the concept used to measure the semantic similarity or difference between two words, phrases or sentences in semantic space. It can be used to complete tasks such as word sense disambiguation, text classification, and information retrieval.

The illusion of the big model refers to the content generated by the artificial intelligence model, which is not based on any real-world data, but is the product of the extrapolation of the big model itself. The essence of this illusion is that the large model itself lacks the ability to perceive the real world, and its training data may be small in quantity or low in quality, which may lead to overfitting and quantization errors in training, as well as lack of prompt context. .

Equivalence refers to a group of tokens that express the same meaning, including a group of tokens that have the same meaning in different languages. For example, "Beijing University" and "Peking University" are equivalent, and "Peking University" in Chinese and "Peking University" in English are equivalent. "equivalence.

The large model integration framework provides standard modular components that integrate different large models and connect them to various external data sources and APIs.

The following embodiments of this application provide text unit semantic distance pre-calculation methods and devices, query methods and devices, which are used to solve the problem that the existing large model integration framework requires a large amount of external data calculation, and the need to segment the data will damage the original information. , the semantic information and contextual relationship of the data are lost, resulting in the problem that the final query result is not relevant to the original data.

The principles and implementation of a text unit semantic distance pre-calculation method and device, query method and device of this embodiment will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can understand the text of this embodiment without creative work. Unit semantic distance precalculation method and device, query method and device.

As shown in Figure 1, this embodiment provides a text unit semantic distance precalculation method, which includes the following steps.

Step S101: Acquire all text units in the precomputed knowledge base, and obtain the associated text unit set of each text unit based on the associated unit acquisition method.

The knowledge base in this embodiment can be used as an external data collection of the large model integration framework. The knowledge base in the embodiment of the present invention has undergone preliminary data curation (Data Curation), and it contains most commonly used text units (ie, tokens). The data quality of the knowledge base that has been curated in the early stage is higher. Using this knowledge base to calculate the similarity between text units can better reflect their true semantic relationships. And the results calculated by these text units are more accurate and faster than those learned by existing pre-training. Further text units (i.e. tokens) can be words, phrases and sentences. In a knowledge base, it refers to objects and categories, that is, the words, phrases, and sentences that make them up.

The knowledge base that pre-performs the semantic distance between text units is used as the pre-computed knowledge base. Get all text units in the precomputed knowledge base. Then, the associated text unit set of each text unit in the precomputed knowledge base is obtained based on the associated unit acquisition method.

Further, the process of obtaining the associated text unit set of a single text unit in the precomputed knowledge base based on the associated unit acquisition method includes: setting the text unit to any text unit in the precomputed knowledge base, and in order to distinguish it from other text units, Sets the text unit to a regular text unit. Since the text unit in this embodiment is a category or object, each text unit has a corresponding description page. First, obtain the description page of the regular text unit from the precomputed knowledge base; then obtain all text units that appear in the text of the description page as internal text units of the regular text unit; for all types of internal text units of the regular text unit Collection is performed to obtain a collection of associated text units of regular text units. Through the above method, the associated text unit set of all text units in the precomputed knowledge base can be obtained.

It should be noted that when obtaining the internal text unit of a regular text unit, there is no need to consider the number of times the internal unit appears in the description page. For example, if all types of text units appearing in the text of the description page of text unit A are text unit B and text unit C, then text unit B and text unit C are the internal text units of text unit A, and further can represent text unit B. and text unit C are respectively associated with text unit A. At this time, the set of associated text units of text unit A can be expressed as A->[B,C].

Step S102: Obtain knowledge representations of all object attribute text units in the precomputed knowledge base based on the associated text unit set through a preset object knowledge representation acquisition method, and obtain all category attributes in the precomputed knowledge base through a preset category knowledge representation acquisition method. Knowledge representation of text units.

After obtaining the associated text unit sets of all text units, the knowledge representation of each text unit can be obtained based on the associated text unit sets of all text units. The text units in this embodiment are objects and categories in the knowledge base, and the knowledge representation acquisition method when the text unit is an object is different from the knowledge representation acquisition method when the text unit is a category.

Furthermore, when the text unit is an object, through the preset object knowledge representation acquisition method, obtaining the knowledge representation of a single object attribute text unit in the precomputed knowledge base includes: using the object attribute text unit as the filtering unit, and then based on the object attribute text unit Filter the associated text unit collection faced to filter out all associated text unit collections that meet the filtering conditions; finally, collect the text units corresponding to all the filtered associated text unit collections to obtain the object attribute text unit knowledge representation.

The object attribute text unit is any object in the precomputed knowledge base. And the associated text unit set faced by the object attribute text unit is all other associated text unit sets in the precomputed knowledge base except the associated text unit set corresponding to the object attribute text unit. And the filter condition is that the associated text unit collection contains the current corresponding object attribute text unit.

For example, the set of associated text units of text unit B is B->[A,C], and the set of associated text units of text unit C is C->[A,D]. Then the knowledge representation of object attribute text unit A is KRa= [C,B], after sorting KRa=[B,C].

Each object attribute text unit is used as a filtering unit, and the knowledge representation of all object attribute text units can be obtained through the above method.

Furthermore, when the text unit is an object, obtaining the knowledge representation of a single category attribute text unit in the precomputed knowledge base through the preset category knowledge representation acquisition method includes: acquiring all the text units belonging to the category attribute text unit in the precomputed knowledge base. The object attribute text unit is used as the object text unit, and the knowledge representation of all the object text units of the category attribute text unit is collected to obtain the knowledge representation of the category attribute text unit. The category attribute text unit is any category in the precomputed knowledge base. Through the above method, the knowledge representation of all category attribute text units can be obtained.

The acquisition process of all object attribute text units belonging to category attribute text units is to precompute the setting attributes of the knowledge base itself, which will not be described in detail here.

Since each text unit of meaning in the knowledge base has a unique identifier (i.e., ID), the text units in each knowledge representation can be sorted based on the identifier size. This setting is beneficial to the subsequent determination of the relationship between text units.

Step S103: Obtain all text unit pairs that can be formed by all text units. Based on the knowledge representation of the text unit, calculate the semantic distance of all text unit pairs by determining the text unit relationship, and combine all text unit pairs for which the semantic distance is calculated. And the corresponding semantic distance set is the semantic distance library of the precomputed knowledge base.

Specifically, all text unit pairs that can be constructed by all text units are first obtained based on all text units, where the text unit pairs include any two text units among all text units. For example, all text unit pairs between text unit A, text unit B and text unit C include (A, B), (B, C) and (A, C). After obtaining the knowledge representation of all text units at the same time, the relationship between any two text units can be determined based on the knowledge representation, and then the semantic distance library of the precomputed knowledge base can be obtained. This embodiment mainly determines the relationship between two text units in all text unit pairs by determining the text unit relationship.

Further, one of the text units is set as the first text unit, and the other text unit is set as the second text unit. At this time, the process of determining the relationship between the two text units in the text unit pair through the text unit relationship determination method includes: first, determining whether there is an intersection between the knowledge representation of the first text unit and the knowledge representation of the second text unit. If so, it means that the first The text unit has a relationship with the second text unit; then the semantic distance between the first text unit and the second text unit can be calculated through the Ochiia coefficient calculation method or the Jaccard index calculation method. If there is no intersection between the knowledge representation of the first text unit and the knowledge representation of the second text unit, it means that there is no relationship between the first text unit and the second text unit. In this case, there is no need to compare the first text unit and the second text unit. The semantic distance between the first text unit and the second text unit cannot be calculated.

Among them, the process of obtaining the semantic distance between the first text unit and the second text unit through the Jaccard index calculation method is: assuming that the first text unit is A, its knowledge is expressed as KRa, and the second text unit is B, and its knowledge Expressed as KRb; the expression for obtaining the semantic distance between the first text unit and the second text unit based on the Jaccard index calculation method is:

The process of obtaining the semantic distance between the first text unit and the second text unit through the Ochiia coefficient calculation method is: assuming that the first text unit is A, its knowledge representation is KRa, the second text unit is B, and its knowledge representation is KRb; Then the expression for obtaining the semantic distance between the first text unit and the second text unit based on the Ochiia coefficient calculation method is:

The closer the semantic distance D is, the closer the relationship between tokens is. The semantic distance D is defined as follows: when D∈[0.6,1], it means that the two text units are extremely similar; when D∈[0.4,0.6], it means that the two text units are relatively similar; when D∈ [0.2,0.4], it means that the two text units are not similar; when D∈[0,0.2], it means that the two text units are not similar.

Through the above method, all the text unit pairs for which the semantic distance can be calculated can be obtained. Finally, all the text unit pairs for which the semantic distance is calculated and the semantic distance corresponding to the text unit pair are collected into the semantic distance library of the pre-computed knowledge base. When querying text units, you can quickly obtain the query results of the corresponding text units by directly querying based on semantic distance in the semantic distance database of the precomputed knowledge base.

It should be noted that the semantic distance library obtained in this embodiment of the present invention is actually a collection of semantic distances between multiple text units and is not limited to a hardware database. The semantic distance library obtained in this embodiment can be applied to storage and In the retrieval mechanism, for example, it can be stored in a high-speed storage medium based on actual application requirements, or it can be stored in a local or remote storage.

The text unit semantic distance precalculation method provided by the embodiment of the present invention queries the internal text units in the description page, and then integrates all text units with the same internal text unit to obtain the knowledge representation of each text unit. Finally, the semantic distance library of the text unit is obtained based on the knowledge representation. There is no vector embedding or chunking process in the semantic distance acquisition process, which does not damage the original information of the data and improves the accuracy. And the acquisition process of text unit semantic distance database is the process of text unit pre-calculation. The results obtained by this process can be directly saved in various data storage or databases for query and use; compared with existing pre-training and large model integration frameworks It saves the time of vector embedding and calculation, that is, it saves the time cost and computing power cost of large model training; it helps to achieve semantic alignment faster during training and improve the reasoning ability of large models.

As shown in Figure 2, this embodiment provides a text unit semantic distance pre-calculation device, which includes an associated text unit acquisition module, a knowledge representation acquisition module and a semantic distance library acquisition module.

The associated text unit acquisition module is used to acquire all text units in the precomputed knowledge base, and acquire the associated text unit set of each text unit based on the associated unit acquisition method.

The knowledge representation acquisition module is used to obtain the knowledge representation of all object attribute text units in the precomputed knowledge base based on the associated text unit set through the preset object knowledge representation acquisition method, and obtain the knowledge representation in the precomputed knowledge base through the preset category knowledge representation acquisition method. Knowledge representation of all category attribute text units.

The semantic distance library acquisition module is used to obtain all text unit pairs that can be formed by all text units. Based on the knowledge representation of text units, the semantic distance of all text unit pairs is calculated by determining the text unit relationship, and all calculated semantic distances are The text unit pairs and the corresponding semantic distance set are the semantic distance database of the precomputed knowledge base.

The object attribute text unit is an object in the precomputed knowledge base, and the category attribute text unit is a category in the precomputed knowledge base.

The text unit semantic distance precalculation device provided by the embodiment of the present invention queries the internal text units in the description page, and then integrates all text units with the same internal text unit to obtain the knowledge representation of each text unit. Finally, the semantic distance library of the text unit is obtained based on the knowledge representation. There is no vector embedding or chunking process in the semantic distance acquisition process, which does not damage the original information of the data and improves the accuracy. And the acquisition process of text unit semantic distance library is the process of text unit pre-calculation. The results obtained by this process can be directly saved in various data storage and databases for query and use; compared with existing pre-training and large model integration frameworks It saves the time of vector embedding and calculation, that is, it saves the time cost and computing power cost of large model training; it helps to achieve semantic alignment faster during training and improve the reasoning ability of large models.

As shown in Figure 3, this embodiment provides a knowledge base text unit query method, which includes the following steps.

Step S301: Obtain the text unit to be queried.

Get the text unit to be queried through text box input or selection.

Step S302: Search all text unit pairs with the text unit to be queried from the semantic distance database of the knowledge base as the query text unit pairs of the text unit to be queried, and based on the corresponding semantic distance size, all or part of the query text unit pairs are classified into non-query text units. The query text unit set is the query result list.

Query based on the specific conditions of the text unit to be queried. Query types for text units to be queried include same-language query, multi-language query and polysemy query. The specific query method is as follows. First, set the language type of the text unit to be queried as the first language, and the language type of the knowledge base as the second language; and set the semantic distance database of the knowledge base to which the text unit to be queried is based on the above text unit. Obtained by semantic distance precomputation method.

When the first language and the second language are the same, it means that the query is a query in the same language. At this time, it is also necessary to determine whether the text unit to be queried is a polysemy word, and specifically whether the identifier corresponding to the text unit to be queried is unique. If so, it means that the text unit to be queried is a single word. At this time, you can directly use the text unit to be queried as the query condition, search for all text unit pairs with the text unit to be queried from the semantic distance database of the knowledge base, and use the found text unit pairs as the query for the current text unit to be queried. text unit pairs, and then sort the non-query text units in all query text unit pairs in descending order based on the corresponding semantic distance, and finally sort all or part of the text units based on the query conditions (such as the number of displayed results, etc.) The text unit collection is the query result list. The non-query text unit in the query text unit pair is another text unit in the query text unit pair other than the text unit to be queried.

During the above query process, if the text unit to be queried is not a text unit in the knowledge base, the query of the text unit to be queried can be completed based on the equivalent of the text unit to be queried in the knowledge base. For example, query the text unit closest to the text unit "Beijing university", but the text unit does not exist in the knowledge base. Find the corresponding text unit "Peking university" in the knowledge base based on the equivalent mapping, and finally return the text unit closest to "Peking university". list of query results.

It should be noted that equivalence is a group of text units that express the same meaning, including a group of text units that have the same meaning in different languages. For example, "Beijing University" and "Peking University" are equivalent, and "Peking University" in Chinese and The English equivalent of "PekingUniversity" etc. The process of obtaining equivalences in the same language knowledge base and the process of obtaining equivalences between knowledge bases in different languages are both conventional methods and will not be described in detail here. Each text unit of meaning in the knowledge base has a unique identifier.

If it is determined that the identifier corresponding to the text unit to be queried is not unique, it means that the text unit to be queried is a polysemy. At this time, it is necessary to obtain the user's supplementary query conditions as advanced qualification conditions for the text unit to be queried. Then, the identifiers that meet the advanced qualification conditions are filtered out from all the identifiers corresponding to the text units to be queried, and the selected unit identifiers to be queried are filtered out. Then, based on the identity of the unit to be queried, the corresponding query result list is searched from the semantic distance database of the knowledge base. The specific process is: find all text unit pairs with the identity of the unit to be queried from the semantic distance database of the knowledge base, and add the queried The resulting text unit pairs are used as the query text unit pairs of the current unit identifier to be queried, and then based on the corresponding semantic distance from large to small, the text units corresponding to the non-to-be-queried unit identifiers in all query text unit pairs are sorted in order. Finally, all or part of the sorted text units are collected into a query result list based on the query conditions. The text unit corresponding to the non-query unit identifier in the query text unit pair is another text unit in the query text unit pair other than the text unit corresponding to the unit identifier to be queried. Whether all the text units in the sorted text units or selected part of the text units are combined into the query result list is implemented based on the display conditions set by the knowledge base itself. The advanced qualification conditions can be input and obtained in the form of text, or can be reflected in the form of options for the user to select.

Specifically for text units with multiple meanings, unique features can be added to distinguish them. The advanced qualification conditions set in this embodiment are the unique features for distinguishing polysemy words. For example, the unique feature can refer to "country", "field", "industry", etc., and the method of differentiation can be marked with brackets or other methods. For example, the text unit "Northwestern University" may refer to text units such as "Northwestern University (United States)" and "Northwest University (China)". The "country" feature is added here and marked with brackets. Query the text unit "Northwestern University" located in "China", then the text unit "Northwestern University" closest to the text unit "China" is the text unit "Northwestern University" that needs to be queried.

When the first language and the second language are different, it means that the query is a query between different languages. At this time, it is necessary to convert the text unit to be queried into an equivalent text unit in the second language through equivalent substitution or software translation. Then search all text unit pairs with equivalent text units from the semantic distance database of the knowledge base, and use the queried text unit pairs as the current query text unit pairs of the same text unit, and then based on the corresponding semantic distance from large to small order, sort the non-identical text units in all query text unit pairs in sequence, and finally use all or part of the sorted text units as equivalent query units of the same text unit based on the query conditions. The set of all equivalent query units obtained at this time is not the result of this query. All equivalent query units need to be converted into query results in the first language through equivalent substitution or software translation. Then all the query results are collected to obtain the query result list of the query for the text unit to be queried. Moreover, whether all the text units in the sorted text units or some text units are selected and set as a query result list is implemented based on the display conditions set by the knowledge base itself.

For example, if the knowledge base is in English, but you want to query the semantic distance between Chinese text units. For example, to query which tokens are closest to the token "I think, therefore I am", you can find the corresponding English equivalent "Cogito, ergo sum" through equivalence mapping, and query the semantic distance of this equivalence token in the English knowledge base. Recent English listings. Then convert each token in the English list into Chinese according to the equivalent body, and finally return the Chinese list.

Querying text units between different languages can also be applied to the following situations.

If the corpus quality of a certain language is not good, even if the quantity of corpus is increased, the quality of large model training results cannot be guaranteed. At this time, you can use the proven and effective training results of other languages to obtain the training results of the required language through equivalent body mapping. For example, the corpus quality of Chinese knowledge base A is not good, and the training results of large models are not ideal. The training results of English knowledge base B are very good. The tokens in B can be mapped into Chinese according to the equivalent body, thereby obtaining the semantic distance relationship between Chinese tokens. This can greatly improve the capabilities of the Chinese language model.

The knowledge base text unit query method provided by the embodiment of the present invention can realize the query of text units. Furthermore, the query of text data in different languages can be quickly realized, and the query of text units with multiple semantics can also be quickly realized.

As shown in Figure 4, this embodiment provides a knowledge base text unit query device, which includes a query unit acquisition module and a query result acquisition module.

The query unit acquisition module is used to obtain the text unit to be queried.

The query result acquisition module is used to find all text unit pairs with the text unit to be queried as the query text unit pairs of the text unit to be queried from the semantic distance database of the knowledge base, and all or part of the query text unit pairs based on the corresponding semantic distance size. The set of text units to be queried in China and Africa is the query result list.

Among them, the non-query text unit in the query text unit pair is another text unit in the query text unit pair other than the text unit to be queried, and the semantic distance database of the knowledge base is obtained through the above text unit semantic distance pre-calculation method.

The knowledge base text unit query device provided by the embodiment of the present invention can realize the query of text units. Furthermore, the query of text data in different languages can be quickly realized, and the query of text units with multiple semantics can also be quickly realized.

Although the disclosed embodiments of the present invention are as above, the described contents are only used to facilitate understanding of the present invention and are not intended to limit the present invention. Any person skilled in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope of the disclosure of the present invention. However, the protection scope of the present invention remains The scope defined by the appended claims shall prevail.

Claims (6)

1. A text unit semantic distance precalculation method, including: Obtain all text units in the precomputed knowledge base, and obtain the associated text unit set of each text unit based on the associated unit acquisition method; Through the preset object knowledge representation acquisition method, the knowledge representation of all object attribute text units in the precomputed knowledge base is obtained based on the associated text unit set, and all the knowledge representations in the precomputed knowledge base are obtained through the preset category knowledge representation acquisition method. Knowledge representation of category attribute text units; Obtain all text unit pairs that can be formed by all the text units. Based on the knowledge representation of the text unit, calculate the semantic distance of all the text unit pairs by determining the text unit relationship, and combine all the calculated semantic distances. The text unit pairs and the corresponding semantic distance set are the semantic distance database of the precomputed knowledge base; Wherein, the object attribute text unit is an object in the precomputed knowledge base, and the category attribute text unit is a category in the precomputed knowledge base; The set of associated text units to obtain text units based on the associated unit acquisition method includes: Obtain description pages of regular text units from the precomputed knowledge base; Treat the text unit in the description page as the internal text unit of the regular text unit, and set all types of internal text units of the regular text unit as a set of associated text units of the regular text unit; Wherein, the regular text unit is any text unit in the precomputed knowledge base; Using the preset object knowledge representation acquisition method, obtaining the knowledge representation of the object attribute text unit in the precomputed knowledge base based on the associated text unit set includes: Use the object attribute text unit as the filtering unit to filter the associated text unit set it faces, and assemble the text units corresponding to the associated text unit set that meet the filtering conditions into the knowledge representation of the object attribute text unit; Wherein, the object attribute text unit is any object in the precomputed knowledge base; the set of associated text units faced by the object attribute text unit is included in the precomputed knowledge base, except for the association corresponding to the object attribute text unit. All other associated text unit sets except the text unit set; the filtering condition is that the associated text unit set contains the filtering unit; Through the preset category knowledge representation acquisition method, obtaining the knowledge representation of a single category attribute text unit in the precomputed knowledge base includes: Obtain the object attribute text unit belonging to the category attribute text unit as the object text unit, and collect the knowledge representation of all the object text units of the category attribute text unit into the knowledge representation of the category attribute text unit; The category attribute text unit is any category in the precomputed knowledge base. 2. The pre-calculation method according to claim 1, characterized in that calculating the semantic distance of the text unit pair by determining the text unit relationship includes: Setting one of the text units in the text unit pair as the first text unit and the other text unit as the second text unit; Determine whether there is an intersection between the knowledge representation of the first text unit and the knowledge representation of the second text unit. If so, it means that the first text unit and the second text unit are related, and based on the first text unit The knowledge representation and the knowledge representation of the second text unit calculate the semantic distance between the first text unit and the second text unit, otherwise it means that there is no relationship between the first text unit and the second text unit. 3. The pre-calculation method according to claim 2, characterized in that, based on the knowledge representation of the first text unit and the knowledge representation of the second text unit, through the Ochiia coefficient calculation method or the Jaccard index calculation method. The semantic distance between the first text unit and the second text unit is calculated. 4. A text unit semantic distance pre-calculation device, which is characterized in that it includes an associated text unit acquisition module, a knowledge representation acquisition module and a semantic distance library acquisition module: The associated text unit acquisition module is used to acquire all text units in the precomputed knowledge base, and acquire the associated text unit set of each text unit based on the associated unit acquisition method; The knowledge representation acquisition module is configured to acquire knowledge representations of all object attribute text units in the precomputed knowledge base based on the associated text unit set through a preset object knowledge representation acquisition method, and through a preset category knowledge representation acquisition method. , obtain the knowledge representation of all category attribute text units in the precomputed knowledge base; The semantic distance library acquisition module is used to obtain all text unit pairs that can be formed by all the text units. Based on the knowledge representation of the text unit, the semantic distance of all the text unit pairs is determined by a text unit relationship determination method. Perform calculations to collect all pairs of text units for which semantic distances are calculated and the corresponding semantic distances into a semantic distance library of the precomputed knowledge base; Wherein, the object attribute text unit is an object in the precomputed knowledge base, and the category attribute text unit is a category in the precomputed knowledge base; The set of associated text units to obtain text units based on the associated unit acquisition method includes: Obtain description pages of regular text units from the precomputed knowledge base; Treat the text unit in the description page as the internal text unit of the regular text unit, and set all types of internal text units of the regular text unit as a set of associated text units of the regular text unit; Wherein, the regular text unit is any text unit in the precomputed knowledge base; Using the preset object knowledge representation acquisition method, obtaining the knowledge representation of the object attribute text unit in the precomputed knowledge base based on the associated text unit set includes: Use the object attribute text unit as the filtering unit to filter the associated text unit set it faces, and assemble the text units corresponding to the associated text unit set that meet the filtering conditions into the knowledge representation of the object attribute text unit; Wherein, the object attribute text unit is any object in the precomputed knowledge base; the set of associated text units faced by the object attribute text unit is included in the precomputed knowledge base, except for the association corresponding to the object attribute text unit. All other associated text unit sets except the text unit set; the filtering condition is that the associated text unit set contains the filtering unit; Through the preset category knowledge representation acquisition method, obtaining the knowledge representation of a single category attribute text unit in the precomputed knowledge base includes: Obtain the object attribute text unit belonging to the category attribute text unit as the object text unit, and collect the knowledge representation of all the object text units of the category attribute text unit into the knowledge representation of the category attribute text unit; The category attribute text unit is any category in the precomputed knowledge base. 5. A knowledge base text unit query method, including: Get the text unit to be queried; The language type of the text unit to be queried is set to the first type of language, and the language type of the knowledge base is set to the second type of language; If the first type of language is the same as the second type of language, then determine whether the corresponding identifier of the text unit to be queried is unique, and if so, search all texts with the text unit to be queried from the semantic distance database of the knowledge base. The unit pair is a query text unit pair of the text unit to be queried, and based on the corresponding semantic distance, all or part of the query text unit pairs that are not text units to be queried are collected into a query result list; otherwise, the text to be queried is obtained Advanced qualification conditions of the unit, and based on the advanced qualification conditions, determine the identification of the unit to be queried from all identifications corresponding to the text unit to be queried, and search all the identification of the unit to be queried from the semantic distance database of the knowledge base The text unit pairs serve as the query text unit pairs of the unit identifiers to be queried, and all or part of the query text unit pairs that are not corresponding to the unit identifiers to be queried are assembled into a query result list based on the corresponding semantic distance; If the first type of language is different from the second type of language, convert the text unit to be queried into an equivalent text unit of the second type of language, and search for all the equivalent text units from the semantic distance database of the knowledge base. Text unit pairs of text units are used as query text unit pairs of the equivalent text units. Based on the corresponding semantic distance, non-equivalent text units in all or part of the query text unit pairs are used as equivalent query units. All the equivalent query units are The unit is converted into query results in the first language, and all the query results are assembled into a query result list; Wherein, each text unit of meaning in the knowledge base has a unique identifier; Wherein, the non-query text unit in the query text unit pair is the other text unit in the query text unit pair except the text unit to be queried, and the semantic distance database of the knowledge base passes the claim 1 - Obtained by the text unit semantic distance pre-calculation method described in any one of -3. 6. A knowledge base text unit query device, which is characterized in that it includes a query unit acquisition module and a query result acquisition module; The query unit acquisition module is used to obtain the text unit to be queried; The query result acquisition module is used to set the language type of the text unit to be queried as the first language, and the language type of the knowledge base as the second language; if the first language and the second language are the same, then determine whether the corresponding identification of the text unit to be queried is unique, and if so, search all text unit pairs with the text unit to be queried from the semantic distance database of the knowledge base as the query text unit pairs of the text unit to be queried. , based on the corresponding semantic distance, all or part of the query text units and non-to-be-queried text units are collected into a query result list; otherwise, the advanced qualification conditions of the to-be-queried text units are obtained, and based on the advanced qualifications The condition determines the identity of the unit to be queried from all the identities corresponding to the text unit to be queried, and searches for all text unit pairs with the identity of the unit to be queried from the semantic distance database of the knowledge base as the query text unit for the identity of the unit to be queried. Yes, based on the corresponding semantic distance, all or part of the query text units corresponding to the non-query unit identifiers are assembled into a query result list; If the first type of language is different from the second type of language, convert the text unit to be queried into an equivalent text unit of the second type of language, and search for all the equivalent text units from the semantic distance database of the knowledge base. Text unit pairs of text units are used as query text unit pairs of the equivalent text units. Based on the corresponding semantic distance, non-equivalent text units in all or part of the query text unit pairs are used as equivalent query units. All the equivalent query units are The unit is converted into query results in the first language, and all the query results are assembled into a query result list; Wherein, each text unit of meaning in the knowledge base has a unique identifier; The non-query text unit in the query text unit pair is the other text unit in the query text unit pair except the text unit to be queried, and the semantic distance database of the knowledge base passes claims 1-3 The text unit semantic distance precalculation method described in any one of the above is obtained.