CN100476800C - A method and system for segmenting index words - Google Patents

Abstract

The invention discloses a segmentation index segmentation method. Including the following steps: read the character stream; identification described the character stream to identify Chinese characters and English characters, as well as an identification number or character; already identified Chinese and English characters or Digital and pre-built 1.1 tree comparison, the sub-set match words; English characters or figures generic fuzzy matching ASCII codes to determine English string or string of digital-term matching the above mentioned English words and string or digital string of words and non-recognition of characters referred to the character stream by order of ranking; The words and figures mentioned in the English string or strings of the sort described in the order of the character stream. The invention also openly segmentation Index segmentation system. The invention provides a cut-word indexing method and system can simultaneously address the precise words, a certain amount of redundant words and word-term problems, enhance the user experience.

Description

A method and system for segmenting index words

technical field

The invention relates to the field of information indexing, in particular to a method and system for segmenting index words.

Background technique

Existing information retrieval systems have become increasingly popular, ranging from web search engines to application-specific information retrieval systems. When it is necessary to process Chinese character information, the information retrieval system will encounter the problem of how to segment words.

There are many word segmentation algorithms at present, among which n-gram word segmentation is a mechanical word segmentation method that does not require a dictionary and is easy to implement. However, this word segmentation method has a large degree of redundancy and cannot solve the problem of single-character word segmentation.

The binary word segmentation method is to separate any two adjacent words that appear in a sentence, and build an inverted index. For example: the sentence "From the perspective of the above steps of realization" can be divided into several words such as "From above, above, stating the facts, realizing, present step, step, step, and seeing". It can be seen from the participle separated above that participles such as "reporting facts" and "present situation" have no practical meaning. Moreover, this method cannot solve the problem of word segmentation, and cannot divide English words.

The maximum matching word segmentation method is a method of matching word segmentation according to the longest word first principle. For example: the sentence "from the above realization steps" may be divided into several words such as "from, above, realization steps, view". The number of words separated by this method is relatively small, but it is not necessarily the shortest, and it is not necessarily accurate. Since this word segmentation method does not have a certain amount of redundant words, it may lead to a decrease in the recall rate, and the experience is not good in some applications.

Word segmentation methods based on statistical or semantic analysis need to solve the problem of disambiguation. The results obtained by this word segmentation method are not necessarily comprehensive, but relatively accurate. However, because this word segmentation method is cumbersome to implement, and the complex analysis process will inevitably affect the word segmentation efficiency to a certain extent, this word segmentation method is not suitable for small specific application information retrieval systems.

Contents of the invention

The purpose of the present invention is to provide a method for segmenting index word segmentation, which can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words, and single-word word segmentation, and enhance user experience.

The object of the present invention is also to provide a system for segmenting index word segmentation, which can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words, and single word segmentation, and enhance user experience.

In order to solve the above technical problems, an embodiment of the present invention provides a method for segmenting index words, including the following steps:

read character stream;

Identify the character stream, determine Chinese characters, English characters or numbers and unrecognizable characters;

Compare the determined Chinese characters, English characters or numbers with the pre-established dictionary tree to determine the matching word segmentation;

Carry out ASCII code Wild match (universal fuzzy matching) with described English character or numeral, determine the participle of English character string or numerical string;

Sorting the above-mentioned matched word segmentation and the word segmentation and unrecognizable characters of the English character string or numeric string according to the sequence of the character stream;

The character stream is divided according to the sorted word segmentation order and the length of each word segment and the above-mentioned unrecognizable characters.

Preferably, the dictionary tree is a pre-established trie character tree data structure.

Preferably, the dictionary tree is a pre-established binary stream dictionary structure.

Preferably, after the character stream is identified, the character stream is stored in an internal character buffer.

Preferably, before the character stream is stored in the internal character buffer, the character stream is subjected to uniform character processing.

Preferably, after the determination of Chinese characters, English characters or numbers and unrecognizable characters, the punctuation marks in the character stream are removed.

Preferably, the dictionary tree removes meaningless words during pre-establishment.

Preferably, after dividing the character stream according to the word segmentation order after the sorting and the length of each word segmentation and the above-mentioned unrecognizable characters, further comprising:

Regularly count the frequency of keywords received;

Keywords with a frequency higher than a predetermined value are added to the dictionary tree.

An embodiment of the present invention provides a system for segmenting index words, the system includes:

A reading unit for reading a character stream;

A character stream identification unit, configured to identify the character stream read by the reading unit, to determine Chinese characters, English characters or numbers, and unrecognizable characters;

The dictionary tree unit, the data structure unit of the dictionary tree storing phrases and phrases in advance;

The comparison unit is used to compare the Chinese characters, English characters or numbers determined by the character stream identification unit with the dictionary tree previously established by the dictionary tree unit, and determine the matching word segmentation;

The general fuzzy matching unit is used to carry out the general fuzzy matching of ASCII codes to the English characters or numbers compared by the comparison unit to determine the word segmentation of English character strings or digital strings;

The word segmentation management unit sorts the word segmentation determined by the comparison unit and the general fuzzy matching unit and the unrecognizable characters determined by the character stream identification unit according to the sequence of the character stream read by the reading unit, and records each The above word segmentation and the length of the above unrecognizable characters;

The word segmentation unit divides the character stream read by the reading unit according to the word segmentation sequence recorded by the word segmentation management unit and the length of each word segment and the above-mentioned unrecognizable characters.

The embodiment of the present invention also provides a system for segmenting index and word segmentation, the system includes:

A reading unit for reading a character stream;

A character stream identification unit, configured to identify the character stream read by the reading unit, to determine Chinese characters, English characters or numbers, and unrecognizable characters;

An internal character buffer unit for storing the character stream identified by the character stream identification unit;

The dictionary tree unit, the data structure unit of the dictionary tree storing phrases and phrases in advance;

The comparison unit is used to compare the Chinese characters, English characters or numbers determined by the character stream identification unit with the dictionary tree previously established by the dictionary tree unit, and determine the matching word segmentation;

The general fuzzy matching unit is used to carry out the general fuzzy matching of ASCII codes to the English characters or numbers compared by the comparison unit to determine the word segmentation of English character strings or digital strings;

The word segmentation management unit sorts the word segmentation determined by the comparison unit and the general fuzzy matching unit and the unrecognizable characters determined by the character stream identification unit according to the sequence of the character stream stored in the internal character buffer unit, and Record the length of each of the above-mentioned word segmentation and the above-mentioned unrecognizable characters;

The word segmentation unit divides the character stream stored in the internal character buffer unit according to the word segmentation sequence recorded by the word segmentation management unit and the length of each word segmentation and the above-mentioned unrecognizable characters;

The dictionary adaptive unit uses a pre-established statistical module to count the frequency of occurrence of keywords, and adds keywords whose frequency of occurrence is higher than a predetermined value to the dictionary tree unit.

The method for segmenting index words in the present invention can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words and single word segmentation, and enhance user experience.

The method for segmenting an index word in the embodiment of the present invention includes reading a character stream; identifying characters in the character stream, determining Chinese characters, English characters or numbers and unrecognizable characters; combining the determined Chinese characters, English characters or numbers with Compare the pre-established dictionary trees to determine the matching word segmentation; perform ASCII code Wild match on English characters or numbers to determine the word segmentation of the English string or number string; compare the above matched word segmentation with the word segmentation of the English string or number string And the unrecognizable characters are sorted according to the order in the character stream; the character stream is divided according to the order in which the word segmentation and the English character string or number string are sorted.

Because all Chinese characters, English characters or numbers are compared with a pre-established dictionary tree before dividing and segmenting, avoiding the occurrence of invalid phrases or phrases, and ensuring appropriate redundant words. When a word can be used as a word or has practical meaning, it will be processed as a normal phrase in the dictionary tree, so the division of a word as a participle can be realized. And the present invention has increased the process of ASCII code Wild match, effectively determines the participle of English character string and number string.

Therefore, the method for segmenting index words in the present invention can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words, and single-word word segmentation, and enhance user experience.

Description of drawings

Fig. 1 is a flow chart of an embodiment of the method of the present invention;

Fig. 2 is the trie character tree structural representation of the present invention;

Fig. 3 is a schematic diagram of a dictionary tree structure of a binary stream according to the present invention;

Fig. 4 is the structural diagram of the first segmentation index word segmentation system of the present invention;

FIG. 5 is a structural diagram of the second segmentation index word segmentation system of the present invention.

Detailed ways

The invention provides a method for segmenting index words, which can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words and word division, and enhance user experience.

In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Referring to Fig. 1, this figure is a flowchart of an embodiment of the method of the present invention.

S10. Read the character stream.

The read character stream may contain Chinese characters or English characters, numbers and unrecognizable characters.

S20. Identify the character stream, and determine Chinese characters, English characters or numbers, and unrecognizable characters.

Perform character recognition on the character stream read in step S10, and determine that the characters in the character stream are specifically Chinese characters or English characters or numbers or unrecognizable characters.

Since the characters in the character stream are recognized, the segmentation of various character sets can be easily realized.

S30. Compare the determined Chinese characters, English characters or numbers with the pre-established dictionary tree, and determine the matching word segmentation.

The dictionary tree can be a collection of commonly used Chinese words or phrases, or a mixture of English characters and Chinese characters.

Comparing the Chinese characters, English characters or numbers determined in step S20 with the pre-established dictionary tree, determining the matching word segmentation, and building an inverted index for the character string corresponding to the word segmentation. The inverted index is a prior art and will not be described in detail here.

When performing word segmentation for a character stream, the following strings will be built with an inverted index.

1. All substrings existing in the dictionary tree in the character stream.

2. Create an inverted index for a single Chinese character and realize a one-element word segmentation.

3. English words or English character strings composed of all English characters.

4. A numeric string composed of all numeric characters.

S40. Perform ASCII universal fuzzy matching on the English characters or numbers to determine word segmentation of the English character string or number string.

The general fuzzy matching of ASCII codes is used to ensure the integrity of the word segmentation. When an English string or number string forms a word by itself, it can be marked as a word segmentation.

If the character stream that needs to be segmented is: "useU disk", the word "U disk" exists in the dictionary tree. When the wild match scan encounters the first English character "u", it scans backward until the last English character in the string, finds the complete English string "useU" in the character stream, and determines it as an English string Participle.

S50. Sorting the above-mentioned matched word segments, word segments of the English character string or number string, and unrecognizable characters according to the sequence of the character stream.

In the entire character stream word segmentation sorting process, the Chinese character word segmentation is sorted in an orderly manner according to the position of the Chinese characters in the character stream. However, when encountering English character strings or numeric strings, the generated word segmentation may appear out of order or even repeated, and at this time, sort according to the order of the characters in the character stream. In this way, the finally obtained word segmentations are arranged in an orderly manner according to the order of each word segmentation in the original character stream, and there is no need to call a sorting module for processing.

S60. Divide the character stream according to the order of word segmentation and the length of each word segment and the aforementioned unrecognizable characters.

Step S50 has arranged the above-mentioned participles in an orderly manner. At this time, it is only necessary to divide the character stream according to the order of each participle, the length of each participle and the above-mentioned unrecognizable characters.

Since all Chinese characters, English character strings or numbers are compared with a pre-established dictionary tree before dividing and segmenting, the occurrence of invalid phrases or phrases is avoided, and appropriate redundant words are guaranteed. When a word can be used as a word or has actual meaning, it will be processed as a normal phrase in the dictionary tree, so the division of a word as a word segment can be realized. Moreover, the present invention adds a general fuzzy matching process of ASCII codes to effectively determine the word segmentation of English character strings and digital strings. Therefore, the method for segmenting index words in the present invention can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words and word division, and enhance user experience.

Although existing word segmentation methods such as binary word segmentation and trigram word segmentation do not require a dictionary tree, they generate more words, many of which are meaningless, and cannot classify mixed words in Chinese and English. The method described in the embodiment of the present invention uses a dictionary tree, and since the division of word segmentation is based on the dictionary tree, the division of meaningless words is effectively avoided.

According to the actual data, the method described in the embodiment of the present invention finally generates twice as many indexes as the binary word segmentation, but in fact, because a single Chinese character is segmented, the unary word segmentation can be realized. In the preferred solution of the present invention, meaningless words or words that do not need to be searched are removed when the dictionary tree is established.

In terms of efficiency, the method described in the embodiment of the present invention is slightly slower than the n-gram word segmentation, because the process of searching for a match in the dictionary tree is added. However, when the method described in the embodiment of the present invention is searching, the words in the dictionary tree will be segmented in the form of the shortest path, thereby greatly reducing the number of index retrieval times and greatly improving the search performance.

Compared with the maximum matching, the method described in the embodiment of the present invention adopts full segmentation, and the recall rate can reach 100%, which is more comprehensive than the maximum matching. This is sufficient for small information retrieval.

The word segmentation efficiency of the method described in the embodiment of the present invention is relatively fast. After a rough test, under linux, the corpus is a general web page, and the dictionary tree is a massive word segmentation dictionary tree, with about 200,000 words and about 80M of memory. Word segmentation is performed by the method described in the embodiment of the present invention, and 3 to 4M data can be processed in one second.

There are many processes in indexing, and word segmentation is only the first step. Compared with other processes, it is very fast to process 3 to 4M data per second. Because other processes such as merging indexes require hard disk I/O, the speed is relatively slow. Therefore, the word segmentation method of the present invention will not cause bottleneck phenomenon in the whole indexing process.

Let's take the character stream of "useU disk" as an example to illustrate the word segmentation process in detail.

Since each character will be searched, the first English character above is marked as the start of a word segment of this English string, and the marker "u" is the start of a word segment of this English string.

The character at the next position of "u" is "s", and "us" cannot be found in the dictionary, and it is not the end of the participle, so no participle is output.

By analogy, when the second "U" is encountered, it is found that there is a word "U disk" in the dictionary, indicating that this position is also the beginning of a word segmentation.

At the same time, backtracking, the first "u" at the start position of the previously marked participle and the current position form a participle "use" output. And mark the current position as the beginning of word segmentation, and then scan in turn.

The final output word segmentation is: "use" "useU" "U disk".

The general fuzzy matching process of the numeric string is similar to the process of the above-mentioned English string, and the specific general fuzzy matching process of the numeric string will not be described in detail here.

A preferred embodiment of the method of the present invention comprises the following steps:

S10. Read the character stream.

S20. Identify the character stream, and determine Chinese characters, English characters or numbers, and unrecognizable characters.

S21. Store the recognized character stream in an internal character buffer.

Before the character stream is stored in the internal character buffer, the character stream can also be processed with unified characters.

S30. Compare the determined Chinese characters, English characters or numbers with the pre-established dictionary tree, and determine the matching word segmentation.

S40. Perform general fuzzy matching of ASCII codes on the English characters or numbers to determine word segmentation of the English character string or number string.

S50. Sorting the above-mentioned matched word segments, word segments of the English character string or number string, and unrecognizable characters according to the sequence of the character stream.

S60. Divide the character stream stored in the internal character buffer according to the sequence of word segmentation and the length of each word segment and the aforementioned unrecognizable characters.

After the step S60, it may also include:

S61. Regularly count the frequencies of the received keywords.

Periodic statistics The frequency of the received keywords can be counted by a pre-established statistics module. The statistical module is used to count the frequency of keywords, and this statistical module can adopt the existing general statistical module for counting keywords by users. The working principle of the specific statistical module is common knowledge and will not be described in detail here.

S62. Add keywords with frequencies higher than a predetermined value to the dictionary tree.

According to the user's own industry or specific needs, the user can regularly count the input keywords of the user, and add keywords with a frequency higher than a predetermined value to the dictionary tree. The predetermined value can be set according to actual needs.

The user can add the counted high-frequency keywords into the dictionary tree, and then rebuild the index. In this way, the dictionary tree can be adaptive, which will greatly improve the retrieval speed.

The dictionary tree in the embodiment of the present invention may be a pre-established map mapping table or a hash_map (hash table) data structure. When the dictionary tree adopts a map mapping table or a hash_map data structure, it is necessary to search for data twice, and it is impossible to know which character the participle ends at, and it is necessary to traverse the entire sentence to find the participle.

The preferred situation of the dictionary tree of the present invention adopts a pre-established trie character tree data structure. Using the trie character tree can better meet the requirements. The trie character tree used in the dictionary tree can find multiple words in one search process, and it is easy to find the end position of the word segmentation. There are already many mature models about the trie character tree, so this article will not repeat them.

Referring to Fig. 2, this figure is a schematic diagram of the trie character tree structure of the present invention.

Each box in Figure 2 represents a node, which corresponds to a Chinese character or character. The box marked with "*" indicates that the Chinese character set or character string formed by the path from the root node to this node corresponds to a word or phrase in the dictionary tree. The number in the box is the number of the node.

For example, the recognized character flow is "Chinese people are great". Start traversal from "中" until "is" is mismatched, and two words "中华" and "中华人" are obtained. Since the words beginning with the word "中" have no specific word segmentation corresponding to the word "人", there is no need to scan backwards. Therefore, the dictionary tree search process using the trie character tree is more efficient.

The string "Chinese are great", scan from front to back, find all words starting with the first position in the dictionary tree, the process is as follows:

A. The first character is "中", which matches node 1 in the trie character tree.

B. Continue to check that the second word is "Hua", find the child node of "中" in node 1, and match to node 2.

C. Continue to check that the third word is "人", find the child node of "Hua" in node 2, and match to node 3.

D. Continue to check that the fourth word is "yes". In the child node of "person" in node 3, there is no "yes", so exit.

In this way, starting from "中", two participle words such as "Zhonghua" and "Chinese" are found. Because in the dictionary, node 2 (Hua) and node 3 (person) are marked with suffix "*". From the two paths from the root node 1 to 2 and 3, 1-2 and 1-2-3 can be used to represent the two specific participle of "China" and "Chinese".

It can be seen from the above search process that the maximum number of matches is equal to the length of the longest word in the dictionary tree in the worst case. But in fact the average value of this length is very small, it is impossible to have very long words. Therefore, the algorithm can be regarded as a constant-level algorithm, represented by a complexity function O(1).

Complexity function O(1) From the perspective of the algorithm process, it is mainly a process of sequentially scanning and recognizing characters and searching in the dictionary tree. Find a string in the dictionary tree introduced earlier, the number of searches is equal to the length of the longest word at that position. Since the length of words in the actual dictionary tree is limited, the average search times can be a constant c. The general constant c<5. When encountering English characters or numeric characters, it is necessary to look back to find the participle, but in fact the participles at the beginning of the English string are some very special words, and the probability of occurrence is very small. The impact of this part on the complexity of the entire word segmentation is negligible.

The process of character recognition is also of constant complexity. For a character stream with a length of n, the complexity of the entire algorithm is O(k×n). Where k is a constant factor, which is the linear complexity. k is generally around 10 on average.

In the preferred mode of the present invention, two character buffers are saved, which are respectively used to store the input character stream, the processed output character stream and the position and length information of the segmented words. In practical applications, long sentences without delimiters rarely exist, so the above-mentioned character buffer capacity can be set to a constant value. When the above-mentioned character buffer cannot fit, first return the word segmentation set in the character buffer, then fill the capacity of the word segmentation buffer, and finally perform word segmentation again. We use space complexity to represent the functional relationship between the memory required by the program and the size of the input data, that is, the length of the sentence. This function is a fixed value and will not increase with the length of the sentence. Therefore the space complexity is constant regardless of the length of the character stream.

The preferred mode of the character buffer capacity is 1024bit, and of course other values can be set according to specific needs.

The dictionary tree of the present invention can also adopt a binary stream dictionary tree, which can ensure that the number of child nodes of each node is small. The dictionary tree of the binary stream is directly located by the array subscript. And English strings can also be put into the dictionary tree of the binary stream, thus realizing the division of Chinese and English mixed words.

Referring to FIG. 3 , it is a schematic diagram of the dictionary tree structure of the binary stream in the present invention.

Each node of the dictionary tree structure of the binary stream in the present invention is not a Chinese character, but several bits split according to the binary data.

Assuming it is 4 bits, a Chinese character includes two bytes, so 4 nodes are needed. An ASCII character requires 2 nodes.

For example, the binary representation of "中" is 0xd0d6, then according to the bits, it is split into four data blocks of 6d 0d from low to high, then the tree of "中" can be expressed as shown in Figure 3.

"6" corresponds to the first node 31 ; the first "d" corresponds to the second node 32 ; "0" corresponds to the third node 33 ; and the second "d" corresponds to the fourth node 34 .

When the method of the present invention adopts the dictionary tree of the binary stream, the read character string is regarded as the binary stream.

For example, if you search for "Chinese", their binary format is 0xd0d60xaabb 0xcbc8, and the digital stream converted into 4bit is 6d 0d b b a a c b c 8. When doing a lookup, the matches are done sequentially. 6 matches the first node 31, the first d matches the second node 32, 0 matches the third node 33, and the second d matches the fourth node 34. The 4th node 34 is the mark of end of a word, then 6d 0d is a word together in this search process, then restores to binary form 0xd0d6 and promptly obtains word " in ". In this way, each node has a maximum of 16 child nodes, and since 4bit can only represent 16 different numbers, the space is greatly saved. The dictionary tree of the binary stream is an improved implementation of the tries tree.

When performing index word segmentation, the string is scanned from front to back, punctuation marks can be filtered out, and all words contained in the string are searched in the dictionary tree and output. A string composed of all English characters or a string composed of all numbers is considered as a word and is also output as a word segmentation. There are a certain amount of redundant words when indexing word segmentation.

The present invention also provides a system for segmenting and indexing word segmentation, which can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words and word segmentation of single characters, and enhance user experience.

Referring to FIG. 4 , this figure is a structural diagram of the first segmentation index word segmentation system of the present invention.

The first segmentation index word segmentation system of the present invention, the system includes:

The reading unit 41 is used to read the character stream.

The character stream identifying unit 42 is configured to identify the character stream read by the reading unit 41 to determine Chinese characters, English characters or numbers and unrecognizable characters.

Since the character stream read by the reading unit 41 may contain Chinese characters, English characters or numbers, and unrecognizable characters, in order to facilitate word segmentation, it needs to be recognized.

The dictionary tree unit 43 is a data structure unit of a dictionary tree for storing phrases and phrases in advance.

The dictionary tree unit 43 is a very important unit, which can effectively reduce redundant redundant words.

The comparison unit 44 is configured to compare the Chinese characters, English characters or numbers determined by the character stream identification unit 42 with the dictionary tree pre-established by the dictionary tree unit 43, and determine the matching word segmentation.

The general fuzzy matching unit 45 is used to perform ASCII general fuzzy matching on the English characters or numbers compared by the comparison unit 44 to determine the word segmentation of the English character string or the digital string.

The word segmentation management unit 46 sorts the word segmentation determined by the comparison unit 44 and the general fuzzy matching unit 45 and the unrecognizable characters determined by the character stream identification unit 42 according to the sequence of the character stream read by the reading unit 41 , and record the length of each of the above word segmentations and the above unrecognizable characters.

The word segmentation unit 47 divides the character stream read by the reading unit 41 according to the word segmentation order recorded by the word segmentation management unit 46 and the length of each word segment and the above-mentioned unrecognizable characters.

Because the comparison unit 44 of the system according to the embodiment of the present invention compares the Chinese characters, English characters or numbers determined by the character stream identification unit 42 with the dictionary tree pre-established by the dictionary tree unit 43, and determines the matching word segmentation. This can effectively avoid the occurrence of invalid phrases or phrases, and ensure a certain amount of redundant words. When a single character can be used as a word or has practical meaning, it will be processed according to a normal phrase in the dictionary tree unit 43, so the word segmentation of a single character as a word segmentation can be realized. And the system of the embodiment of the present invention adds a general fuzzy matching unit 45, which performs ASCII code general fuzzy matching on the English or numbers compared by the comparison unit 44, and effectively determines the word segmentation of English character strings and digital strings. Therefore, the first segmentation index word segmentation system of the present invention can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words, and word segmentation of single characters, thereby enhancing user experience.

Referring to FIG. 5 , it is a structural diagram of the second segmentation index word segmentation system of the present invention.

Compared with the first segmentation index word segmentation system of the present invention, the second segmentation index word segmentation system adds an internal character buffer unit 48 and a dictionary adaptive unit 49 .

The reading unit 41 is used to read the character stream.

The character stream identifying unit 42 is configured to identify the character stream read by the reading unit 41 to determine Chinese characters, English characters or numbers and unrecognizable characters.

The internal character buffer unit 48 is used to store the character stream identified by the character stream identification unit 42 and provide word segmentation text for the word segmentation unit 47 .

Of course, the character stream identification unit 42 can also unify the identified character streams before sending them to the internal character buffer unit 48 for storage.

The dictionary tree unit 43 is a data structure unit of a dictionary tree for storing phrases and phrases in advance.

The comparison unit 44 is configured to compare the Chinese characters, English characters or numbers determined by the character stream identification unit 42 with the dictionary tree pre-established by the dictionary tree unit 43, and determine the matching word segmentation.

The general fuzzy matching unit 45 is used to perform ASCII general fuzzy matching on the English characters or numbers compared by the comparison unit 44 to determine the word segmentation of the English character string or the digital string.

The word segmentation management unit 46, the word segmentation determined by the comparison unit 44 and the general fuzzy matching unit 45 and the unrecognizable characters determined by the character stream identification unit 42 are stored according to the character stream stored in the internal character buffer unit 48 Sequentially sort, and record the length of each of the above-mentioned word segmentation and the above-mentioned unrecognizable characters.

The word segmentation unit 47 divides the character stream stored in the internal character buffer unit 48 according to the word segmentation order recorded by the word segmentation management unit 46 and the length of each word segment and the above-mentioned unrecognizable characters.

The dictionary adaptation unit 49 uses a pre-established statistics module 50 to count the frequency of occurrence of keywords, and adds keywords whose frequency of occurrence is higher than a predetermined value to the dictionary tree unit 43 .

The pre-established statistical module 50 is used to count the frequency of keywords. The statistics module 50 may adopt an existing general statistics module for user statistics keywords. The working principle of the specific statistical module is common knowledge and will not be described in detail here.

The dictionary adaptive unit 49 performs an adaptive operation according to the keyword frequency counted by the statistical module 50 . When the keyword frequency counted by the statistical module 50 is greater than a predetermined value, the dictionary adaptation unit 49 adds the keyword to the dictionary tree unit 43 .

Due to the second segmentation index word segmentation system described in the embodiment of the present invention, the comparison unit 44 compares the Chinese characters, English characters or numbers determined by the character stream identification unit 42 with the dictionary tree previously established by the dictionary tree unit 43, and determines Matching participle. This effectively avoids the occurrence of invalid phrases or phrases, and ensures a certain amount of redundant words. When a single character can be used as a word or has actual meaning, the dictionary tree unit 43 can process it as a normal phrase, so the division of a single character as a word segmentation can be realized. And the second kind of segmentation index word segmentation system described in the embodiment of the present invention has general fuzzy matching unit 45, carries out ASCII code general fuzzy matching to the English character or numeral after described comparison unit 44 comparison, effectively determines English character string and numeral participle of the string. Therefore, the second segmentation index word segmentation system of the present invention can simultaneously solve the problems of accurate word segmentation, a certain amount of redundant words, and single word segmentation, and enhance user experience.

The above is only a preferred embodiment of the present invention, it should be pointed out that, for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications can also be made. It should be regarded as the protection scope of the present invention.

Claims (10)

1, a kind of method of cutting index participle is characterized in that, may further comprise the steps:

Read character stream;

Discern described character stream, determine Chinese character, English character or numeral and unrecognizable character;

Chinese character, English character or numeral of having determined and the lexicographic tree of setting up are in advance compared, determine the participle of coupling;

Described English character or numeral are carried out the general fuzzy matching of ASCII character, determine the participle of English character string or numeric string;

With the participle of above-mentioned coupling and the participle and the unrecognizable character of described English character string or numeric string, sort in proper order by described character stream;

Divide described character stream by the order of the participle after the described ordering and the length of described each participle and above-mentioned unrecognizable character.

2, the method for cutting index participle according to claim 1 is characterized in that, described lexicographic tree is the trie character tree data structure of setting up in advance.

3, the method for cutting index participle according to claim 1 is characterized in that, described lexicographic tree is the binary stream dictionary configuration of setting up in advance.

4, according to the method for the arbitrary described cutting index participle of claim 1 to 3, it is characterized in that, behind the described character stream of described identification, described character stream is stored in inner character buffer.

5, the method for cutting index participle according to claim 4 is characterized in that, described character stream is stored in before the inner character buffer, described character stream is unified the processing of character.

6, the method for cutting index participle according to claim 5 is characterized in that, behind described definite Chinese character, English character or numeral and the unrecognizable character, removes the punctuation mark in the described character stream.

According to the method for the arbitrary described cutting index participle of claim 1 to 3, it is characterized in that 7, described lexicographic tree is removed insignificant individual character when setting up in advance.

8, according to the method for the arbitrary described cutting index participle of claim 1 to 3, it is characterized in that, further comprise after dividing described character stream by the length of the order of the participle after the described ordering and described each participle and above-mentioned unrecognizable character:

Regularly add up the frequency of the keyword that receives;

The keyword that frequency is higher than predetermined value adds in the described lexicographic tree.

9, a kind of system of cutting index participle is characterized in that, this system comprises:

Reading unit is used to read character stream;

The character stream recognition unit is used for the character stream that described reading unit reads is discerned, and determines Chinese character, English character or numeral and unrecognizable character;

The lexicographic tree unit is stored the data structure cell of the lexicographic tree of phrase and phrase in advance;

Comparing unit, the Chinese character, English character or the lexicographic tree digital and that described lexicographic tree unit is set up in advance that are used for described character stream recognition unit is determined are compared, and determine the participle of coupling;

General fuzzy matching unit is used for English character or the numeral of described comparing unit after relatively carried out the general fuzzy matching of ASCII character, determines the participle of English character string or numeric string;

The participle administrative unit, with described comparing unit and definite participle and the definite unrecognizable character of described character stream recognition unit of described general fuzzy matching unit, the character stream that reads by described reading unit sorts in proper order, and writes down the length of each above-mentioned participle and above-mentioned unrecognizable character;

The participle division unit with the character stream that described reading unit reads, is divided according to the branch word order of described participle management unit records and the length of described each participle and above-mentioned unrecognizable character.

10, a kind of system of cutting index participle is characterized in that, this system comprises:

Reading unit is used to read character stream;

The character stream recognition unit is used for the character stream that described reading unit reads is discerned, and determines Chinese character, English character or numeral and unrecognizable character;

Inner character buffer location is used to store the character stream of described character stream recognition unit identification;

The lexicographic tree unit is stored the data structure cell of the lexicographic tree of phrase and phrase in advance;

Comparing unit, the Chinese character, English character or the lexicographic tree digital and that described lexicographic tree unit is set up in advance that are used for described character stream recognition unit is determined are compared, and determine the participle of coupling;

General fuzzy matching unit is used for English character or the numeral of described comparing unit after relatively carried out the general fuzzy matching of ASCII character, determines the participle of English character string or numeric string;

The participle administrative unit, with described comparing unit and definite participle and the definite unrecognizable character of described character stream recognition unit of described general fuzzy matching unit, described character stream by described inner character buffer location storage sorts in proper order, and writes down the length of each above-mentioned participle and above-mentioned unrecognizable character;

The participle division unit with the character stream of described inner character buffer location storage, is divided according to the branch word order of described participle management unit records and the length of described each participle and above-mentioned unrecognizable character;

The dictionary adaptive unit, by the frequency of occurrences of the statistical module counts keyword of setting up in advance, the keyword that the described frequency of occurrences is higher than predetermined value adds described lexicographic tree unit to.

Images