CN117829150A - Synonym recognition method, device and network equipment - Google Patents

Abstract

The present invention provides a synonym recognition method, device and network equipment, and relates to the technical field of natural language processing. The method comprises: based on the conditional random field (CRF) algorithm, entity word recognition is performed on a first test set to obtain a target test set, wherein the first test set is text data containing a point of interest (POI), and the target test set includes at least one candidate entity word, and the candidate entity word is associated with the POI; using a synonym recognition model, identifying whether each of the candidate entity words in the target test set is a synonym, and obtaining a recognition result, wherein the synonym recognition model is constructed based on at least two target similarity algorithms. The solution of the present invention solves the problem that the accuracy of the existing synonym recognition method is not high.

Description

Synonym recognition method, device and network equipment

Technical Field

The present invention relates to the technical field of natural language processing, and in particular to a synonym recognition method, device and network equipment.

Background technique

With the rapid development of social economy, mobile Internet has entered a period of prosperity and development. People can search for points of interest (POI) they are interested in through mobile phone software. In the geographic information system, a POI can be a house, a shop, a mailbox, a bus stop, etc.

Usually, users can use map navigation software to search for shopping malls and navigate to the location they want to go, or they can search for POIs around them and select the destination they want to go. At present, as people use it more and more times and in more and more scenarios, the requirements for POI search functions are getting higher and higher. People always hope to get the results they want, but sometimes the keywords entered are "informal" for the software system, resulting in inaccurate search results. For example: usually the software system stores "Traffic Management Bureau", but users usually enter "Traffic Management Bureau" for convenience when entering, and such words are often difficult for the software system to match. This requires the software system to use the word similarity calculation method to perform synonym recognition processing, identify synonyms for the keywords entered by the user, and search for these words in turn, and return the results to the user.

However, among the existing word similarity calculation methods: the dictionary-based calculation method needs to use a dictionary or vocabulary classification system to calculate the semantic similarity of words, which is greatly affected by human subjective consciousness, and sometimes it is difficult to accurately reflect objective facts, and is easily limited by the update of the dictionary and the integrity of the vocabulary, resulting in low accuracy in synonym recognition; while the method based on the context vector space model is mainly based on the word formation characteristics of Chinese synonyms. Most synonyms contain the same morphemes (i.e., characters), and synonyms are discovered by calculating the literal similarity. Therefore, this method can usually only determine synonyms that are similar in words, but it is difficult to effectively identify synonyms with different forms, so the accuracy of synonym recognition is not high. .

Summary of the invention

The purpose of the present invention is to provide a synonym recognition method, device and network equipment, which solves the problem that the existing synonym recognition method has low accuracy.

To achieve the above object, an embodiment of the present invention provides a synonym recognition method, comprising:

Based on a conditional random fields (CRF) algorithm, entity word recognition is performed on a first test set to obtain a target test set, wherein the first test set is text data containing a point of interest (POI), and the target test set includes at least one candidate entity word, and the candidate entity word is associated with the POI;

A synonym recognition model is used to identify whether each of the candidate entity words in the target test set is a synonym, and a recognition result is obtained, wherein the synonym recognition model is constructed based on at least two target similarity algorithms.

Optionally, the step of performing entity word recognition on the first test set based on the conditional random field (CRF) algorithm to obtain a target test set includes:

Based on the CRF algorithm, determining a first recognition model;

Using the first training set to train the first recognition model to obtain an entity word recognition model;

The first test set is input into the entity word recognition model to obtain the target test set.

Optionally, the using the first training set to train the first recognition model to obtain an entity word recognition model includes:

Performing a first preprocessing on the first training set to obtain at least one first character string, wherein the first preprocessing includes: word segmentation and/or part-of-speech tagging;

Using a target labeling method, labeling the at least one first character string to obtain a labeling result;

The first recognition model is trained using the annotation results to obtain the entity word recognition model.

Optionally, inputting the first test set into the entity word recognition model to obtain the target test set includes:

Performing the first preprocessing on the first test set to obtain at least one character string to be marked;

The at least one character string to be marked is input into the entity word recognition model to obtain the target test set.

Optionally, the target similarity algorithm includes: a synonym forest algorithm and a machine learning algorithm, and the synonym recognition model is used to identify whether each of the candidate entity words in the target test set is a synonym, and the recognition result is obtained, including:

Based on the synonym word forest algorithm, obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set;

Based on the machine learning algorithm, obtaining a second similarity between the first candidate entity word and the second candidate entity word;

Obtaining the synonym recognition model according to the first similarity and the second similarity;

Using the synonym recognition model, it is identified whether each of the candidate entity words in the target test set is a synonym, and a recognition result is obtained.

Optionally, obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set based on the synonym word forest algorithm includes:

Performing second preprocessing on the first candidate entity word and the second candidate entity word respectively to obtain a first word list corresponding to the first candidate entity word and a second word list corresponding to the second candidate entity word, wherein the second preprocessing includes: word segmentation and deleting non-Chinese characters;

Based on the synonym word forest algorithm, sequentially calculating the similarity between each word in the first word list and the second word list to obtain a first similarity list corresponding to the first candidate entity word, and sequentially calculating the similarity between each word in the second word list and the first word list to obtain a second similarity list corresponding to the second candidate entity word;

A first similarity between the first candidate entity word and the second candidate entity word is obtained according to the first similarity list and the second similarity list.

Optionally, obtaining a second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm includes:

The Word2Vec deep learning model is used to train the model using the first training set to obtain the target word vector set;

Based on the cosine similarity algorithm, the similarity between the first word vector and the second word vector is obtained according to the first word vector corresponding to the first candidate entity word in the target word vector set and the second word vector corresponding to the second candidate entity word in the target word vector set, and the similarity between the first word vector and the second word vector is used as the second similarity between the first candidate entity word and the second candidate entity word.

Optionally, obtaining the synonym recognition model according to the first similarity and the second similarity includes:

Using a second training set, training a logistic regression model corresponding to the synonym recognition model until the logistic regression model converges, and determining weights corresponding to the first similarity and the second similarity respectively according to a regression coefficient vector when the logistic regression model converges, wherein the second training set includes at least one group of synonyms and at least one group of non-synonymous words;

The synonym recognition model is constructed according to the weight of the first similarity and the weight of the second similarity.

To achieve the above object, an embodiment of the present invention provides a synonym recognition device, comprising:

A first processing module is used to perform entity word recognition on a first test set based on a conditional random field (CRF) algorithm to obtain a target test set, wherein the first test set is text data containing a point of interest (POI), and the target test set includes at least one candidate entity word, and the candidate entity word is associated with the POI;

The second processing module is used to use a synonym recognition model to identify whether each of the candidate entity words in the target test set is a synonym, and obtain a recognition result, wherein the synonym recognition model is constructed based on at least two target similarity algorithms.

Optionally, the first processing module includes:

A first processing submodule, used for determining a first recognition model based on the CRF algorithm;

A second processing submodule is used to train the first recognition model using the first training set to obtain an entity word recognition model;

The third processing submodule is used to input the first test set into the entity word recognition model to obtain the target test set.

Optionally, the second processing submodule includes:

A third processing unit is configured to perform a first preprocessing on the first training set to obtain at least one first character string, wherein the first preprocessing includes: word segmentation and/or part-of-speech tagging;

A fourth processing unit, configured to label the at least one first character string by using a target labeling method to obtain a labeling result;

The fifth processing unit is used to use the annotation results to train the first recognition model to obtain the entity word recognition model.

Optionally, the third processing submodule includes:

A first processing unit, configured to perform the first preprocessing on the first test set to obtain at least one character string to be marked;

The second processing unit is used to input the at least one character string to be marked into the entity word recognition model to obtain the target test set.

Optionally, the target similarity algorithm includes: a synonym forest algorithm and a machine learning algorithm, and the second processing module includes:

A fourth processing submodule is used to obtain a first similarity between the first candidate entity word and the second candidate entity word in the target test set based on the synonym word forest algorithm;

a fifth processing submodule, configured to obtain a second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm;

a sixth processing submodule, configured to obtain the synonym recognition model according to the first similarity and the second similarity;

The seventh processing submodule is used to use a synonym recognition model to identify whether each of the candidate entity words in the target test set is a synonym, and obtain a recognition result.

Optionally, the fourth processing submodule includes:

a sixth processing unit, configured to perform second preprocessing on the first candidate entity word and the second candidate entity word respectively, to obtain a first word list corresponding to the first candidate entity word and a second word list corresponding to the second candidate entity word, wherein the second preprocessing includes: word segmentation and deletion of non-Chinese characters;

A seventh processing unit is used to sequentially calculate the similarity between each word in the first word list and the second word list based on the synonym word forest algorithm to obtain a first similarity list corresponding to the first candidate entity word, and sequentially calculate the similarity between each word in the second word list and the first word list to obtain a second similarity list corresponding to the second candidate entity word;

An eighth processing unit is used to obtain a first similarity between the first candidate entity word and the second candidate entity word according to the first similarity list and the second similarity list.

Optionally, the fifth processing submodule includes:

A ninth processing unit is used to adopt a Word2Vec deep learning model, use the first training set to perform model training, and obtain a target word vector set;

The tenth processing unit is used to obtain the similarity between the first word vector and the second word vector according to the first word vector corresponding to the first candidate entity word in the target word vector set and the second word vector corresponding to the second candidate entity word in the target word vector set based on the cosine similarity algorithm, and use the similarity between the first word vector and the second word vector as the second similarity between the first candidate entity word and the second candidate entity word.

Optionally, the sixth processing submodule includes:

an eleventh processing unit, configured to train a logistic regression model corresponding to the synonym recognition model using a second training set until the logistic regression model converges, and determine weights corresponding to the first similarity and the second similarity respectively according to a regression coefficient vector when the logistic regression model converges, wherein the second training set includes at least one group of synonyms and at least one group of non-synonymous words;

A twelfth processing unit is used to construct the synonym recognition model according to the weight of the first similarity and the weight of the second similarity.

To achieve the above object, an embodiment of the present invention provides a network device, including a processor and a transceiver, wherein the processing is used to:

Based on the conditional random field (CRF) algorithm, entity word recognition is performed on the first test set to obtain a target test set, where the first test set is text data containing a point of interest (POI), and the target test set includes at least one candidate entity word, and the candidate entity word is associated with the POI;

A synonym recognition model is used to identify whether each of the candidate entity words in the target test set is a synonym, and a recognition result is obtained, wherein the synonym recognition model is constructed based on at least two target similarity algorithms.

Optionally, when the processor performs entity word recognition on the first test set based on the conditional random field (CRF) algorithm to obtain the target test set, the processor is specifically configured to:

Based on the CRF algorithm, determining a first recognition model;

Using the first training set to train the first recognition model to obtain an entity word recognition model;

The first test set is input into the entity word recognition model to obtain the target test set.

Optionally, when the processor uses the first training set to train the first recognition model to obtain the entity word recognition model, it is specifically used to:

Performing a first preprocessing on the first training set to obtain at least one first character string, wherein the first preprocessing includes: word segmentation and/or part-of-speech tagging;

Using a target labeling method, labeling the at least one first character string to obtain a labeling result;

The first recognition model is trained using the annotation results to obtain the entity word recognition model.

Optionally, when the processor inputs the first test set into the entity word recognition model to obtain the target test set, the processor is specifically configured to:

Performing the first preprocessing on the first test set to obtain at least one character string to be marked;

The at least one character string to be marked is input into the entity word recognition model to obtain the target test set.

Optionally, the target similarity algorithm includes: a synonym forest algorithm and a machine learning algorithm. When the processor uses a synonym recognition model to identify whether each of the candidate entity words in the target test set is a synonym and obtains a recognition result, it is specifically used to:

Based on the synonym word forest algorithm, obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set;

Based on the machine learning algorithm, obtaining a second similarity between the first candidate entity word and the second candidate entity word;

Obtaining the synonym recognition model according to the first similarity and the second similarity;

Using the synonym recognition model, it is identified whether each of the candidate entity words in the target test set is a synonym, and a recognition result is obtained.

Optionally, when the processor obtains the first similarity between the first candidate entity word and the second candidate entity word in the target test set based on the synonym word forest algorithm, it is specifically configured to:

Performing second preprocessing on the first candidate entity word and the second candidate entity word respectively to obtain a first word list corresponding to the first candidate entity word and a second word list corresponding to the second candidate entity word, wherein the second preprocessing includes: word segmentation and deleting non-Chinese characters;

Based on the synonym word forest algorithm, sequentially calculating the similarity between each word in the first word list and the second word list to obtain a first similarity list corresponding to the first candidate entity word, and sequentially calculating the similarity between each word in the second word list and the first word list to obtain a second similarity list corresponding to the second candidate entity word;

A first similarity between the first candidate entity word and the second candidate entity word is obtained according to the first similarity list and the second similarity list.

Optionally, when the processor obtains the second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm, it is specifically configured to:

The Word2Vec deep learning model is used to train the model using the first training set to obtain the target word vector set;

Based on the cosine similarity algorithm, the similarity between the first word vector and the second word vector is obtained according to the first word vector corresponding to the first candidate entity word in the target word vector set and the second word vector corresponding to the second candidate entity word in the target word vector set, and the similarity between the first word vector and the second word vector is used as the second similarity between the first candidate entity word and the second candidate entity word.

Optionally, when the processor obtains the synonym recognition model according to the first similarity and the second similarity, it is specifically configured to:

Using a second training set, training a logistic regression model corresponding to the synonym recognition model until the logistic regression model converges, and determining weights corresponding to the first similarity and the second similarity respectively according to a regression coefficient vector when the logistic regression model converges, wherein the second training set includes at least one group of synonyms and at least one group of non-synonymous words;

The synonym recognition model is constructed according to the weight of the first similarity and the weight of the second similarity.

To achieve the above-mentioned purpose, an embodiment of the present invention provides a network device, including a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the synonym recognition method as described above is implemented.

To achieve the above objective, an embodiment of the present invention provides a readable storage medium having a program or instruction stored thereon, wherein the program or instruction, when executed by a processor, implements the steps in the above-mentioned synonym identification method.

The beneficial effects of the above technical solution of the present invention are as follows:

The method of the embodiment of the present invention can obtain a target test set including at least one candidate entity word associated with a POI based on a CRF algorithm, and can further use a synonym recognition model to identify whether each candidate entity word in the target test set is a synonym, thereby obtaining various synonyms in the POI field, which can improve the POI search efficiency. Here, the synonym recognition model is constructed based on at least two target similarity algorithms, so the advantages of the two target similarity algorithms can be combined to make the synonym recognition result more accurate, and the recognition process is simple and the running speed is fast.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a synonym identification method according to an embodiment of the present invention;

Figure 2 is a schematic diagram of the CRF algorithm;

FIG3 is a schematic diagram of a process of training a first recognition model according to an embodiment of the present invention;

FIG4 is a schematic diagram of a process of performing entity word recognition on a first test set according to an embodiment of the present invention;

FIG5 is a structural diagram of a synonym recognition device according to an embodiment of the present invention;

FIG6 is a structural diagram of a network device according to an embodiment of the present invention;

FIG. 7 is a structural diagram of a network device according to another embodiment of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Additionally, the terms "system" and "network" are often used interchangeably herein.

In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and/or other information.

At present, there are two commonly used methods for calculating word similarity: a dictionary-based calculation method and a statistical context vector space model method, which are described as follows:

Dictionary-based calculation method: also called a method based on world knowledge or a certain classification system. In a dictionary or vocabulary classification system, all similar semantic items form a hierarchical tree structure, and the distance between corresponding nodes (called concept distance) can represent the similarity between vocabulary semantics. When using a dictionary or vocabulary classification system to calculate the similarity of vocabulary semantics, it is necessary to first convert complex vocabulary concepts into the smallest concept units included in the dictionary or vocabulary classification system, and then use the concept hierarchy system structure to calculate the semantic similarity of each concept unit. Commonly used dictionaries include "Synonym Cilin" and "Encyclopedia Encyclopedia". However, the results obtained by this method are greatly affected by human subjective consciousness, and sometimes cannot accurately reflect objective facts. In addition, synonyms are usually more inclined to written language and are easily restricted by dictionary updates and vocabulary integrity.

Method based on context vector space model: The vocabulary similarity calculation based on the document vector space model represents the context of the current vocabulary or the content of a fixed-size window as a word meaning space vector. If the word meaning space vectors of two words are similar, the two words can be considered synonyms or related words. There are many methods for calculating the similarity between vectors. The most common one is the cosine similarity calculation method, which evaluates their similarity by calculating the cosine value of the angle between two vectors. The closer the value is to 1, the higher their similarity. In this method, based on the word formation characteristics of Chinese synonyms, most synonyms contain the same morphemes (i.e., characters), and synonyms are discovered by calculating the literal similarity. However, although this method is simple to implement, it can usually only determine synonyms that are similar in literal sense, but cannot identify synonyms with different forms well, such as: Construction Bank and CBB (China Construction Bank).

As shown in FIG1 , a synonym recognition method according to an embodiment of the present invention includes:

Step 101, based on the conditional random field (CRF) algorithm, entity word recognition is performed on the first test set to obtain a target test set, wherein the first test set is text data containing points of interest (POIs), and the target test set includes at least one candidate entity word, and the candidate entity word is associated with the POI.

In this step, entities can be extracted through the CRF algorithm, that is, candidate entity words associated with the POI are extracted from the text data containing the POI. Here, entity recognition is performed, that is, entity words such as place names and institution names associated with the POI in the first test set are found. For example, in the "accommodation" category in the POI field, facilities such as hotels, guesthouses, and folk customs are entities, and hotel names such as "such and such hotel" and "such and such guesthouse" are entity words.

Step 102, using a synonym recognition model, identifying whether each of the candidate entity words in the target test set is a synonym, and obtaining a recognition result, wherein the synonym recognition model is constructed based on at least two target similarity algorithms.

In this step, synonym recognition is performed on the candidate entity words in the target test set, and the recognition results obtained are multiple groups of synonyms associated with POIs.

It should be noted that common synonym recognition algorithms often only consider factors such as font and meaning. However, in the field of POI search, specific place names such as restaurants and accommodation are usually involved, which are all nouns. Nouns do not have synonyms or synonyms in language logic, so the similarity calculation effect is not ideal. For example, when users want to query the place they want to search, they often enter an alias or abbreviation of the place name. For example, when searching for "Shenyang Public Security Bureau Traffic Police Department Vehicle Management Office", the user may not enter the full name because they do not know the full name or other reasons, but may enter words such as "Vehicle Management Office" or "Traffic Management Bureau", resulting in unsatisfactory synonym recognition. In this regard, considering that each POI has its own name in the geographic information system, the POI name can be understood as a named entity based on its characteristics. The synonym recognition algorithm for the POI field can be converted into a synonym recognition algorithm for named entities.

In this embodiment, based on the CRF algorithm, a target test set including at least one candidate entity word associated with the POI can be obtained. Further, the synonym recognition model can be used to identify whether the candidate entity words in the target test set are synonyms, thereby obtaining various synonyms in the POI field. In this way, it can be applied in geographic information fields such as navigation. Since the synonym recognition model is constructed based on at least two target similarity algorithms, it can combine the advantages of the two target similarity algorithms. Therefore, when the user searches for POI, a more accurate synonym recognition result can be obtained, and the recognition process is simple and the running speed is fast.

Here, the CRF algorithm is briefly described as follows:

CRF is an algorithm model proposed by Lafferty with the help of the maximum entropy model. It is a statistical model based on the tag sequence and a hidden Markov model. The sequence composed of words is called the observation sequence, and the tags corresponding to each word constitute the tag sequence. The CRF model is generated by calculating the probability of the tag corresponding to each word in the training set, and this model is used to calculate the joint probability of the entire tag sequence given the observation sequence.

Variable X represents the observation sequence of the text, and variable Y represents the corresponding tag sequence. A fixed-size finite character set is specified, and the representation range of all elements _Yi in set Y must not exceed this character set. P(Y|X) represents the probability of the tag sequence under the condition of a given observation sequence, and P(X) represents the probability of this observation sequence. Variables X and Y satisfy the conditions of joint probability distribution.

As shown in Figure 2, let G = (V, E) represent an undirected graph, V represents the set of points in the graph, and E represents the set of edges in the graph. Y = (Y _v ) _v∈V , T _v is an element in the tag sequence set Y, which corresponds one-to-one to the vertices in the undirected graph G. When the observation sequence X is established, the conditional probability distribution of the random variable Y _v obeys the Markov property of the graph: P(Y _v |X, Y _w w≠v) = P(Y _v |X, Y _w , w～v), where w～v means that (w, v) is an edge of the undirected graph G. At this time, we call (X, Y) a conditional random field. In theory, the structure of the graph G is arbitrary, but when building the model, CRF uses the simplest and most important first-order chain structure. As shown in Figure 2, the conditional random field (X, Y) takes the observation sequence X as the global condition and does not make any assumptions about X. This simple structure can be used to define a joint probability distribution P(Y|X) over the tag sequence. Therefore, CRF mainly focuses on two subsequences: X = ( _X1 , _X2 , ..., _Xn ) and Y = ( _Y1 , _Y2 , ..., _Yn ). Where n is an integer greater than 0, _X1 represents the first character in the observation sequence X; _X2 represents the second character in the observation sequence X; Xn represents the nth character in the observation sequence X; _Y1 represents the first character in the tag sequence Y; _Y2 represents the second character in the tag sequence Y; _Yn represents the third character in the tag sequence Y.

In the embodiment of the present invention, entity word recognition is performed on the first test set based on the conditional random field CRF algorithm, which is specifically described as follows:

As an optional embodiment, the method of performing entity word recognition on the first test set based on the conditional random field (CRF) algorithm to obtain the target test set includes:

(i) Based on the CRF algorithm, determine a first recognition model.

Here, the first recognition model is constructed based on the CRF algorithm, that is, the CRF algorithm is used as the model algorithm of the first recognition model. Therefore, the model training process of the first recognition model is also a process of probability statistical calculation, that is, the probability of each entity word appearing and the probability of the tag of the related word appearing are counted.

(ii) Using a first training set to train the first recognition model to obtain an entity word recognition model, the first training set being text data containing N types of POIs.

Here, if the POI field is divided into 16 POI categories, such as: public facilities, factories and enterprises, industrial and commercial areas, accommodation, educational and cultural places, news media, shopping and retail places, catering places, medical and health places, transportation institutions and places, financial institutions, communication institutions and places, commercial service places, resident service places, leisure and entertainment places, scenic spots and entrances and exits, then the first training set can collect training corpus (such as text materials of POI, etc.) for the above 16 POI categories to construct the first training set and the first test set, so that the data is more comprehensive and the training effect is better.

After the training is completed, a model file, namely, an entity word recognition model, can be obtained. Then, the entity word recognition model can be used to recognize the first test set to obtain a target test set.

(3) Inputting the first test set into the entity word recognition model to obtain the target test set.

The target test set includes multiple groups of synonyms associated with POI.

As shown in FIG. 4 , as an optional embodiment, the step of inputting the first test set into the entity word recognition model to obtain the target test set includes:

Performing the first preprocessing on the first test set to obtain at least one character string to be marked, wherein the first preprocessing includes: word segmentation and/or part-of-speech tagging;

The at least one character string to be marked is input into the entity word recognition model to obtain the target test set.

Here, the sentence is divided into strings to be marked, and the words are marked using the entity word recognition model.

In this embodiment, the probability of each word appearing can be obtained through Model (i.e., entity word recognition model). Combining this information, the tag sequence with the largest joint probability can be quickly found from at least one string to be tagged, and used as the output result Result (i.e., target test set). Among them, the task of CRF_Test is to calculate the tag sequence with the largest joint probability.

As shown in FIG. 3 , as an optional embodiment, the first recognition model is trained using the first training set to obtain an entity word recognition model, including:

(1) performing a first preprocessing on the first training set to obtain at least one first character string, wherein the first preprocessing includes: word segmentation and/or part-of-speech tagging.

Here, the main tasks of the first preprocessing are: word segmentation, part-of-speech tagging, and manual tagging of entity information such as place names and organizations. For example, the result of word segmentation of a text data in the first training set is: This newspaper /r News / Ng Reporter / n Meng Xianli / nr: / wThe / u "/wCody Cup/nz/wThe / wSecond/m/qChina/nsAmateur/bInternational/nStandard Dance/nContest/vn hosted by / pChina Amateur Dance Competition Association/nt, / wRecently/t/t/pBeijing Chaoyang Gymnasium/ns/ns ended/v The curtain/n. Among them, / China Amateur Dance Competition Association / Beijing Chaoyang Gymnasium are both entity words.

(2) Using a target labeling method, labeling the at least one first character string to obtain a labeling result.

Since the CRF algorithm is implemented based on sequence labeling, part-of-speech tagging is required when the entity word recognition of the first test set is performed based on the CRF algorithm. Here, illustratively, the target labeling method can use the 5-tag labeling method for text labeling. The labeling set of the 5-tag labeling method is (B1, B2, M, E, O), that is, the first character of the entity word is marked as B1, the second character of the entity word is marked as B2, the middle character is marked as M, the last character is marked as E, and non-entity words are all marked as O. For example, "Wang Houyuan Dumpling Restaurant" is a catering entity, so its entity words can be marked as "Wang/B1 Hou/B2 Yuan/M Dumpling/M Zi/M Restaurant/E".

(iii) Using the annotation results, training the first recognition model to obtain the entity word recognition model.

As shown in Figure 3, the first recognition model (i.e., CRF_Learn) is trained using the annotation results (i.e., template file) and training samples (i.e., at least one first string) to obtain Model (i.e., entity word recognition model). Among them, the main task of CRF_Learn is to comprehensively consider the characteristics of each input item, calculate and count the probability of each entity word, and provide a basis for the subsequent stages.

Optionally, the target similarity algorithm includes: a synonym forest algorithm and a machine learning algorithm, and the synonym recognition model is used to identify whether each of the candidate entity words in the target test set is a synonym, and the recognition result is obtained, including:

Based on the synonym word forest algorithm, obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set;

Based on the machine learning algorithm, obtaining a second similarity between the first candidate entity word and the second candidate entity word;

Obtaining the synonym recognition model according to the first similarity and the second similarity;

Using the synonym recognition model, it is identified whether each of the candidate entity words in the target test set is a synonym, and a recognition result is obtained.

In this embodiment, a synonym recognition model is constructed based on the synonym dictionary algorithm and the machine learning algorithm, which can combine the advantages of the two methods to make the synonym recognition model have better recognition effect and higher accuracy.

Optionally, obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set based on the synonym word forest algorithm includes:

Step 1, perform second preprocessing on the first candidate entity word and the second candidate entity word respectively to obtain a first word list corresponding to the first candidate entity word and a second word list corresponding to the second candidate entity word, wherein the second preprocessing includes: word segmentation and deletion of non-Chinese characters.

In this step, when performing the second preprocessing, a word segmentation tool can be used to perform word segmentation on two entity words to be judged (for example, the first candidate entity word and the second candidate entity word), and filter out non-Chinese characters (for example, special symbols, line breaks, spaces, blank characters, etc.) to obtain two word lists respectively. For example, the first word list corresponding to the first candidate entity word (represented by _Wa ) is {W _a1 , _Wa2 ,...,W _an }, and the second word list corresponding to the second candidate entity word (represented by W _b ) is {W _b1 , W _b2 ,...,W _bm }, where n represents the number of words in the first word list; m represents the number of words in the second word list.

Step 2, based on the synonym word forest algorithm, calculate the similarity between each word in the first word list and the second word list in turn, obtain the first similarity list corresponding to the first candidate entity word, and calculate the similarity between each word in the second word list and the first word list in turn, obtain the second similarity list corresponding to the second candidate entity word.

In this step, it is necessary to calculate the similarity between each word in W _a and each word in W _b in turn, and obtain the similarity between each word in the first word list and the second word list. For example, the similarity between W _a1 and W _b1 , W _b2 , ..., W _bm is calculated respectively, and the obtained similarity values are: p ₁ , p ₂ , ..., p _m , if the value of p ₂ is the largest among p ₁ , p ₂ , ..., p _m , then p ₂ is determined as the similarity between W _a1 and the second word list. Similarly, the similarity between W _a2 , ..., _Wan and the second word list can be obtained in turn, assuming that they are expressed as: q ₃ , ..., h ₅ , respectively, then the first similarity list corresponding to the first candidate entity word can be expressed as {p ₂ ,q ₃ ,...,h ₅ }. Similarly, the similarity between each word in the second word list and the first word list can also be calculated in turn, and the second similarity list corresponding to the second candidate entity word can be obtained.

Step 3: Obtain a first similarity between the first candidate entity word and the second candidate entity word according to the first similarity list and the second similarity list.

Here, it is assumed that the first similarity list corresponding to the first candidate entity word (W _a ) is: {SIM _a1 , ..., SIM _an }, and the second similarity list corresponding to the first candidate entity word (W _b ) is: {SIM _b1 , ..., SIM _bm }.

Then, the first similarity between the first candidate entity word and the second candidate entity word can be calculated according to the following formula:

Among them, SIM _dict represents the first similarity; n represents the number of words in the first word list; m represents the number of words in the second word list; SIM _ai represents the i-th value in the first similarity list (that is, the similarity between the i-th word in Wa and _Wa ); SIM _bj represents the j-th value in the second similarity list (the similarity between the j-th word in Wb and _Wb ).

It should be noted that when determining the similarity between a certain word in W _a ) and a certain word in W _b , it is determined in the following way:

Determine whether the codes of two words (i.e., the codes corresponding to the two words in the dictionary constructed based on the synonym forest, assuming they are _Ca and _Cb respectively) are the same; if they are the same, the similarity value of the two words is 1; if they are not the same, the similarity of the two codes needs to be calculated using the first formula, which is expressed as:

Wherein, LCS( _Ca , _Cb ) represents the nearest common parent node of code _Ca and code _Cb ; IC(C) represents the information content of code C, and IC(C) can be calculated according to the second formula, which is expressed as:

Among them, n represents the total number of ontology nodes, and k represents the number of lower-level nodes in the ontology.

As an optional embodiment of the present invention, obtaining a second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm includes:

Using the Word2Vec deep learning model, using the first training set to perform model training to obtain a target word vector set, where the first training set is text data containing N types of POIs;

Based on the cosine similarity algorithm, the similarity between the first word vector and the second word vector is obtained according to the first word vector corresponding to the first candidate entity word in the target word vector set and the second word vector corresponding to the second candidate entity word in the target word vector set, and the similarity between the first word vector and the second word vector is used as the second similarity between the first candidate entity word and the second candidate entity word.

It should be noted that Word2vec is a group of related models used to generate word vectors. These models are shallow and double-layer neural networks used for training to reconstruct the word text of linguistics. The network is represented by words, and the input words in adjacent positions need to be guessed. Under the assumption of the bag of words model in word2vec, the order of words is not important. After the training is completed, the word2vec model can be used to map each word to a vector, which can be used to represent the relationship between words. The vector is the hidden layer of the neural network. Word2Ve includes two algorithms, namely skip-gram and continuous bag of words model (Continuous Bag of Words Model, CBOW) algorithm. The biggest difference between them is that skip-gram predicts the words around the central word through the central word, while CBOW predicts the central word through the surrounding words. In the embodiment of the present invention, the skip-gram algorithm can be used to perform model training using the first training set.

In the field of natural language processing, the objects we study are usually text content, and different texts usually have different formats or include different contents. Here, the test corpus (i.e., the first training set) needs to be preprocessed. For example, non-Chinese characters in the first training set are removed, and Chinese characters, English letters, and numbers that are useful to the POI field are retained.

Here, the text in the first training set also needs to be segmented. For example, a word segmenter (such as a Chinese word segmentation system) can be used for word segmentation.

Specifically, as an optional embodiment, the parameters of the Word2Vec deep learning model can be set as follows:

(1) By default, sg=0 corresponds to the CBOW algorithm, and sg=1 corresponds to the skip-gram algorithm. Since the skip-gram algorithm is more sensitive to low-frequency words, sg can be set to 1 in the embodiment of the present invention.

(2)min_count is used to filter words. Words with a frequency less than min-count will be ignored. The default value of min_count is 5.

(3) size is the dimension of the output word vector, that is, the number of units in the hidden layer of the neural network. If the value of size is too small, the word mapping will affect the result due to conflicts. If the value of size is too large, it will consume more memory and slow down the algorithm calculation speed. In addition, when the value of size is large, more training data will be required, but the training effect will be better. Here, the size value can be set to 300 dimensions.

(4) window is the maximum distance between the current word and the target word in the sentence, that is, the window. Optionally, the size of the window movement can be set to 5.

(5) negative and sample can be fine-tuned according to the training results, where negative is the number of negative samples. The core idea of negative sampling is to transform the multi-classification problem into a binary classification problem, that is, to determine whether it is a positive sample or a negative sample. The default value is 5; sample means that higher frequency words are randomly downsampled to the set threshold. The default value is 1e-3.

(6) hs = 1 means that the softmax layer will be used. By default, hs = 0 and negative is not 0, negative sampling will be selected.

The Word2Vec deep learning model is trained using the first training set, and the training results (i.e., the target word vector set) can be saved in text form, as shown in Table 1.

Table 1:

As an optional embodiment, when calculating the second similarity between the first candidate entity word and the second candidate entity word by the cosine similarity algorithm, the first candidate entity word and the second candidate entity word can be segmented, and then a word vector (that is, a first word vector corresponding to the first candidate entity word in the target word vector set and a second word vector corresponding to the second candidate entity word in the target word vector set) can be constructed according to the Word2Vec vector model. Then, the second similarity between the first candidate entity word and the second candidate entity word is calculated by the cosine similarity algorithm. The calculation formula is expressed as follows:

Among them, SIM _word2vec represents the second similarity; _Xi represents the i-th word after the first candidate entity word (represented by X) is segmented (X is divided into n words in total); Yj represents the j-th word after the second candidate entity word (represented by Y) is segmented (Y is divided into m words in total). Among them, m and n are both integers greater than 0.

In this embodiment, a word vector model is trained using a Word2Vec deep learning model based on a test corpus in the POI field, and then a second similarity between the first candidate entity word and the second candidate entity word is calculated using a cosine similarity algorithm.

Optionally, obtaining the synonym recognition model according to the first similarity and the second similarity includes:

Using a second training set, training a logistic regression model corresponding to the synonym recognition model until the logistic regression model converges, and determining weights corresponding to the first similarity and the second similarity respectively according to a regression coefficient vector when the logistic regression model converges, wherein the second training set includes at least one group of synonyms and at least one group of non-synonymous words;

The synonym recognition model is constructed according to the weight of the first similarity and the weight of the second similarity.

Here, by performing weighted summation on the first similarity and the second similarity, the expression of the synonym recognition model can be obtained: score = θ ₁ *SIM _dict + θ ₂ *SIM _word2vec ;

Among them, score represents the similarity between the first candidate entity word and the second candidate entity word; SIM _dict represents the first similarity; SIM _word2vec represents the second similarity; θ ₁ represents the weight of the first similarity; θ ₂ represents the weight of the second similarity.

It should be noted that the parameters in the expression of the synonym recognition model, ie, the weights (θ ₁ , θ ₂ ) corresponding to the first similarity and the second similarity, respectively, can be determined by constructing a corresponding logistic regression model of the synonym recognition model.

The expression of the logistic regression model is:

Where X is the feature input, where X represents a 2D vector (X1, X2), X1 represents the first similarity (i.e., SIM _dict ), and X2 represents the second similarity (i.e., SIM _word2vec ); θ represents the regression coefficient vector θ ₁ represents the weight of the first similarity; θ ₂ represents the weight of the second similarity.

It should be noted that the value of h _θ (X) is between [0,1]. For example, it can be stipulated that: when h _θ (X)>0.5, it is considered that h _θ (X)＝1, that is, the first candidate entity word and the second candidate entity word are synonyms; when h _θ (X)<0.5, it is considered that h _θ (X)＝0, that is, the first candidate entity word and the second candidate entity word are not synonyms.

Here, the same number of synonyms and non-synonyms are manually identified as the second training set, and the values of θ ₁ and θ ₂ are continuously adjusted according to the training results and the value of h _θ (X) until the logistic regression model converges, and finally the weight of SIM _dict (ie θ ₁ ) and the weight of SIM _word2vec (θ ₂ ) are calculated.

In this embodiment, the weights of the first similarity and the second similarity can be dynamically assigned through the logistic regression model, and finally the final similarity value is obtained by weighted summing the first similarity and the second similarity. The higher the value, the higher the similarity between the first candidate entity word and the second candidate entity word. In this way, since the synonym recognition model combines two similarity algorithms (based on the synonym word forest similarity calculation method and the similarity calculation method based on the machine learning) and performs weighted summation to obtain the final similarity, the advantages of the two similarity algorithms can be combined, and the similarity finally calculated is more accurate, and the recognition effect of synonyms is better.

It should be noted that the embodiment of the present invention provides a synonym recognition method applicable to the POI field. The characteristics of the POI field are combined in the recognition process. The model is trained through the entity word recognition algorithm to obtain the aliases, abbreviations, etc. of the place nouns of the POI category as candidate entity words to form a target test set, and then the matching degree (similarity) of the candidate entity words is calculated under each POI category, so as to select the candidate entity words with high scores as synonyms. The synonym recognition method is not limited by the update of the dictionary and the integrity of the vocabulary, nor is it affected by the literal meaning; it can identify synonyms such as variant synonyms and abbreviations, and improve the recognition efficiency of variant synonyms; it can accurately reflect objective facts without being affected by human subjectivity. Make the recognition result more accurate.

The synonym recognition method of this embodiment can obtain a target test set including at least one candidate entity word associated with a POI based on the CRF algorithm, and can further use the synonym recognition model to identify whether each candidate entity word in the target test set is a synonym, thereby obtaining various synonyms in the POI field, which can improve the POI search efficiency. Here, the synonym recognition model is constructed based on at least two target similarity algorithms, so it can combine the advantages of the two target similarity algorithms, reduce the influence of human subjective factors, improve the recognition efficiency of variant synonyms, make the recognition result more accurate, and the recognition process is simple and the running speed is fast.

As shown in FIG5 , a synonym recognition device according to an embodiment of the present invention includes:

A first processing module 510 is configured to perform entity word recognition on a first test set based on a conditional random field (CRF) algorithm to obtain a target test set, wherein the first test set is text data containing a point of interest (POI), and the target test set includes at least one candidate entity word, and the candidate entity word is associated with the POI;

The second processing module 520 is used to use a synonym recognition model to identify whether each of the candidate entity words in the target test set is a synonym, and obtain a recognition result, wherein the synonym recognition model is constructed based on at least two target similarity algorithms.

In this embodiment, based on the CRF algorithm, a target test set including at least one candidate entity word associated with the POI can be obtained. Further, the synonym recognition model can be used to identify whether the candidate entity words in the target test set are synonyms, thereby obtaining various synonyms in the POI field. In this way, it can be applied in geographic information fields such as navigation. Since the synonym recognition model is constructed based on at least two target similarity algorithms, it can combine the advantages of the two target similarity algorithms. Therefore, when the user searches for POI, a more accurate synonym recognition result can be obtained, and the recognition process is simple and the running speed is fast.

Optionally, the first processing module 510 includes:

A first processing submodule, used for determining a first recognition model based on the CRF algorithm;

A second processing submodule is used to train the first recognition model using a first training set to obtain an entity word recognition model, wherein the first training set is text data containing N types of POIs;

The third processing submodule is used to input the first test set into the entity word recognition model to obtain the target test set.

Optionally, the third processing submodule includes:

A first processing unit, configured to perform the first preprocessing on the first test set to obtain at least one character string to be marked;

The second processing unit is used to input the at least one character string to be marked into the entity word recognition model to obtain the target test set.

Optionally, the second processing submodule includes:

A third processing unit is configured to perform a first preprocessing on the first training set to obtain at least one first character string, wherein the first preprocessing includes: word segmentation and/or part-of-speech tagging;

A fourth processing unit, configured to label the at least one first character string by using a target labeling method to obtain a labeling result;

The fifth processing unit is used to use the annotation results to train the first recognition model to obtain the entity word recognition model.

Optionally, the target similarity algorithm includes: a synonym forest algorithm and a machine learning algorithm, and the second processing module 520 includes:

A fourth processing submodule is used to obtain a first similarity between the first candidate entity word and the second candidate entity word in the target test set based on the synonym word forest algorithm;

a fifth processing submodule, configured to obtain a second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm;

a sixth processing submodule, configured to obtain the synonym recognition model according to the first similarity and the second similarity;

The seventh processing submodule is used to use a synonym recognition model to identify whether each of the candidate entity words in the target test set is a synonym, and obtain a recognition result.

Optionally, the fourth processing submodule includes:

a sixth processing unit, configured to perform second preprocessing on the first candidate entity word and the second candidate entity word respectively, to obtain a first word list corresponding to the first candidate entity word and a second word list corresponding to the second candidate entity word, wherein the second preprocessing includes: word segmentation and deletion of non-Chinese characters;

A seventh processing unit is used to sequentially calculate the similarity between each word in the first word list and the second word list based on the synonym word forest algorithm to obtain a first similarity list corresponding to the first candidate entity word, and sequentially calculate the similarity between each word in the second word list and the first word list to obtain a second similarity list corresponding to the second candidate entity word;

An eighth processing unit is used to obtain a first similarity between the first candidate entity word and the second candidate entity word according to the first similarity list and the second similarity list.

Optionally, the fifth processing submodule includes:

A ninth processing unit is used to adopt a Word2Vec deep learning model and use a first training set to perform model training to obtain a target word vector set, where the first training set is text data containing N types of POIs;

The tenth processing unit is used to obtain the similarity between the first word vector and the second word vector according to the first word vector corresponding to the first candidate entity word in the target word vector set and the second word vector corresponding to the second candidate entity word in the target word vector set based on the cosine similarity algorithm, and use the similarity between the first word vector and the second word vector as the second similarity between the first candidate entity word and the second candidate entity word.

Optionally, the sixth processing submodule includes:

an eleventh processing unit, configured to train a logistic regression model corresponding to the synonym recognition model using a second training set until the logistic regression model converges, and determine weights corresponding to the first similarity and the second similarity respectively according to a regression coefficient vector when the logistic regression model converges, wherein the second training set includes at least one group of synonyms and at least one group of non-synonymous words;

A twelfth processing unit is used to construct the synonym recognition model according to the weight of the first similarity and the weight of the second similarity.

It should be noted here that the above-mentioned synonym recognition device provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned synonym recognition method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be specifically repeated here.

As shown in FIG6 , a network device 600 according to an embodiment of the present invention includes a processor 610 and a transceiver 620, wherein the processor is configured to:

Based on the conditional random field (CRF) algorithm, entity word recognition is performed on the first test set to obtain a target test set, where the first test set is text data containing a point of interest (POI), and the target test set includes at least one candidate entity word, and the candidate entity word is associated with the POI;

A synonym recognition model is used to identify whether each of the candidate entity words in the target test set is a synonym, and a recognition result is obtained, wherein the synonym recognition model is constructed based on at least two target similarity algorithms.

In this embodiment, based on the CRF algorithm, a target test set including at least one candidate entity word associated with the POI can be obtained. Further, the synonym recognition model can be used to identify whether the candidate entity words in the target test set are synonyms, thereby obtaining various synonyms in the POI field. In this way, it can be applied in geographic information fields such as navigation. Since the synonym recognition model is constructed based on at least two target similarity algorithms, it can combine the advantages of the two target similarity algorithms. Therefore, when the user searches for POI, a more accurate synonym recognition result can be obtained, and the recognition process is simple and the running speed is fast.

Optionally, when the processor performs entity word recognition on the first test set based on the conditional random field (CRF) algorithm to obtain the target test set, the processor is specifically configured to:

Based on the CRF algorithm, determining a first recognition model;

Using a first training set to train the first recognition model to obtain an entity word recognition model, wherein the first training set is text data containing N types of POIs;

The first test set is input into the entity word recognition model to obtain the target test set.

Optionally, when the processor inputs the first test set into the entity word recognition model to obtain the target test set, the processor is specifically configured to:

Performing the first preprocessing on the first test set to obtain at least one character string to be marked;

The at least one character string to be marked is input into the entity word recognition model to obtain the target test set.

Optionally, when the processor uses the first training set to train the first recognition model to obtain the entity word recognition model, it is specifically used to:

Performing a first preprocessing on the first training set to obtain at least one first character string, wherein the first preprocessing includes: word segmentation and/or part-of-speech tagging;

Using a target labeling method, labeling the at least one first character string to obtain a labeling result;

The first recognition model is trained using the annotation results to obtain the entity word recognition model.

Optionally, the target similarity algorithm includes: a synonym forest algorithm and a machine learning algorithm. When the processor uses a synonym recognition model to identify whether each of the candidate entity words in the target test set is a synonym and obtains a recognition result, it is specifically used to:

Based on the synonym word forest algorithm, obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set;

Based on the machine learning algorithm, obtaining a second similarity between the first candidate entity word and the second candidate entity word;

Obtaining the synonym recognition model according to the first similarity and the second similarity;

Using the synonym recognition model, it is identified whether each of the candidate entity words in the target test set is a synonym, and a recognition result is obtained.

Optionally, when the processor obtains the first similarity between the first candidate entity word and the second candidate entity word in the target test set based on the synonym word forest algorithm, it is specifically configured to:

Performing second preprocessing on the first candidate entity word and the second candidate entity word respectively to obtain a first word list corresponding to the first candidate entity word and a second word list corresponding to the second candidate entity word, wherein the second preprocessing includes: word segmentation and deleting non-Chinese characters;

Based on the synonym word forest algorithm, sequentially calculating the similarity between each word in the first word list and the second word list to obtain a first similarity list corresponding to the first candidate entity word, and sequentially calculating the similarity between each word in the second word list and the first word list to obtain a second similarity list corresponding to the second candidate entity word;

A first similarity between the first candidate entity word and the second candidate entity word is obtained according to the first similarity list and the second similarity list.

Optionally, when the processor obtains the second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm, it is specifically configured to:

Using the Word2Vec deep learning model, using the first training set to perform model training to obtain a target word vector set, where the first training set is text data containing N types of POIs;

Based on the cosine similarity algorithm, the similarity between the first word vector and the second word vector is obtained according to the first word vector corresponding to the first candidate entity word in the target word vector set and the second word vector corresponding to the second candidate entity word in the target word vector set, and the similarity between the first word vector and the second word vector is used as the second similarity between the first candidate entity word and the second candidate entity word.

Optionally, when the processor obtains the synonym recognition model according to the first similarity and the second similarity, it is specifically configured to:

Using a second training set, training a logistic regression model corresponding to the synonym recognition model until the logistic regression model converges, and determining weights corresponding to the first similarity and the second similarity respectively according to a regression coefficient vector when the logistic regression model converges, wherein the second training set includes at least one group of synonyms and at least one group of non-synonymous words;

The synonym recognition model is constructed according to the weight of the first similarity and the weight of the second similarity.

It should be noted here that the above-mentioned network device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned synonym recognition method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

A network device according to another embodiment of the present invention, as shown in FIG7 , includes a transceiver 710, a processor 700, a memory 720, and a program or instruction stored in the memory 720 and executable on the processor 700; the processor 700 implements the above-mentioned synonym recognition method when executing the program or instruction.

The transceiver 710 is used to receive and send data under the control of the processor 700.

Among them, in Figure 7, the bus architecture can include any number of interconnected buses and bridges, specifically one or more processors represented by processor 700 and various circuits of memory represented by memory 720 are linked together. The bus architecture can also link various other circuits such as peripherals, regulators, and power management circuits together, which are all well known in the art, so they are not further described herein. The bus interface provides an interface. The transceiver 710 can be a plurality of components, that is, including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 700 is responsible for managing the bus architecture and general processing, and the memory 720 can store data used by the processor 700 when performing operations.

A computer-readable storage medium according to an embodiment of the present invention stores a program or instruction thereon, and when the program or instruction is executed by a processor, the steps in the synonym recognition method described above are implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

In the embodiment of the present invention, the module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present invention. Therefore, the present invention should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present invention will be perfect and complete, and the scope of the present invention will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms "one", "one" and "the" are intended to include these multiple forms. It will be further understood that the terms "including" and/or "comprising" when used in this specification indicate the presence of the features, integers, steps, operations, components and/or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and/or their groups. Unless otherwise indicated, when stated, a value range includes the upper and lower limits of the range and any sub-ranges therebetween.

The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (12)

1. A method of synonym identification, comprising:

performing entity word recognition on a first test set based on a Conditional Random Field (CRF) algorithm to obtain a target test set, wherein the first test set is text data containing POI (point of interest), and the target test set comprises at least one candidate entity word which is associated with the POI;

and identifying whether each candidate entity word in the target test set is a synonym or not by using a synonym identification model to obtain an identification result, wherein the synonym identification model is constructed based on at least two target similarity algorithms.

2. The method of claim 1, wherein the performing entity word recognition on the first test set based on the conditional random field CRF algorithm to obtain the target test set comprises:

determining a first recognition model based on the CRF algorithm;

Training the first recognition model by using a first training set to obtain an entity word recognition model;

and inputting the first test set into the entity word recognition model to obtain the target test set.

3. The method of claim 2, wherein training the first recognition model using the first training set to obtain the entity word recognition model comprises:

performing first preprocessing on the first training set to obtain at least one first character string, wherein the first preprocessing comprises the following steps: word segmentation and/or part-of-speech tagging;

labeling the at least one first character string by using a target labeling method to obtain a labeling result;

and training the first recognition model by using the labeling result to obtain the entity word recognition model.

4. The method of claim 3, wherein said inputting the first test set into the entity word recognition model results in the target test set comprising:

performing first preprocessing on the first test set to obtain at least one character string to be marked;

and inputting the at least one character string to be marked into the entity word recognition model to obtain the target test set.

5. The method of claim 1, wherein the target similarity algorithm comprises: the synonym forest algorithm and the machine learning algorithm, the identifying whether the candidate entity words in the target test set are synonyms or not by using a synonym identification model, and obtaining an identification result, comprises the following steps:

obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set based on the synonym Lin Suanfa;

obtaining a second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm;

obtaining the synonym recognition model according to the first similarity and the second similarity;

and identifying whether the candidate entity words in the target test set are synonyms or not by using the synonym identification model, and obtaining an identification result.

6. The method of claim 5, wherein the obtaining a first similarity between a first candidate entity word and a second candidate entity word in the target test set based on the synonym Lin Suanfa comprises:

respectively performing second preprocessing on the first candidate entity word and the second candidate entity word to obtain a first word list corresponding to the first candidate entity word and a second word list corresponding to the second candidate entity word, wherein the second preprocessing comprises the following steps: word segmentation and deletion of non-Chinese characters;

Based on the synonym word forest algorithm, sequentially calculating the similarity between each word in a first word list and a second word list to obtain a first similarity list corresponding to the first candidate entity word, and sequentially calculating the similarity between each word in the second word list and the first word list to obtain a second similarity list corresponding to the second candidate entity word;

and obtaining the first similarity between the first candidate entity word and the second candidate entity word according to the first similarity list and the second similarity list.

7. The method of claim 5, wherein the obtaining a second similarity between the first candidate entity word and the second candidate entity word based on the machine learning algorithm comprises:

adopting a Word2Vec deep learning model, and carrying out model training by using a first training set to obtain a target Word vector set;

based on a cosine similarity algorithm, according to a first word vector corresponding to the first candidate entity word in the target word vector set and a second word vector corresponding to the second candidate entity word in the target word vector set, obtaining similarity between the first word vector and the second word vector, and taking the similarity between the first word vector and the second word vector as second similarity between the first candidate entity word and the second candidate entity word.

8. The method of claim 5, wherein the obtaining the synonym recognition model from the first degree of similarity and the second degree of similarity comprises:

training a logistic regression model corresponding to the synonym recognition model by using a second training set until the logistic regression model converges, and determining weights respectively corresponding to the first similarity and the second similarity according to regression coefficient vectors when the logistic regression model converges, wherein the second training set comprises at least one group of synonyms and at least one group of non-synonyms;

and constructing the synonym recognition model according to the weight of the first similarity and the weight of the second similarity.

9. A synonym identification device, comprising:

the first processing module is used for carrying out entity word recognition on a first test set based on a Conditional Random Field (CRF) algorithm to obtain a target test set, wherein the first test set is text data containing POI (point of interest), and the target test set comprises at least one candidate entity word which is associated with the POI;

and the second processing module is used for identifying whether the candidate entity words in the target test set are synonyms or not by using a synonym identification model to obtain an identification result, wherein the synonym identification model is constructed based on at least two target similarity algorithms.

10. A network device, comprising: a transceiver and a processor; the processor is configured to:

performing entity word recognition on a first test set based on a Conditional Random Field (CRF) algorithm to obtain a target test set, wherein the first test set is text data containing POI (point of interest), and the target test set comprises at least one candidate entity word which is associated with the POI;

and identifying whether each candidate entity word in the target test set is a synonym or not by using a synonym identification model to obtain an identification result, wherein the synonym identification model is constructed based on at least two target similarity algorithms.

11. A network device, comprising: a transceiver, a processor, a memory, and a program or instructions stored on the memory and executable on the processor; the synonym identification method as claimed in any one of claims 1-8, wherein said program or instructions, when executed by said processor, implement said synonym identification method.

12. A readable storage medium having stored thereon a program or instructions, which when executed by a processor, performs the steps in the synonym identification method as claimed in any one of claims 1 to 8.