TWI636370B - Establishing chart indexing method and computer program product by text information - Google Patents

Abstract

The invention provides a chart indexing method and a computer program product thereof by using text information. The foregoing method comprises the steps of: reading the article paragraph information or sentence information associated with the target chart in the file, analyzing the article paragraph information or the sentence information collection, extracting a plurality of key vocabulary words, weighting and sorting the key words of interest, and sorting, The one or more candidate words are selected, and the virtual chart message directory and the schema index information are generated according to the candidate words.

Description

 Establishing chart indexing method and computer program product by text information  

The invention relates to an indexing method and a computer program product thereof, in particular to a chart indexing method and a computer program product thereof by using text information.

In the current file management system, in order to query a specific file context, it is often necessary to input a specific keyword to find the region in which the keyword is located, and to find the required text data in the region.

Since the prior art can only search for text content, when the user wants to query the specific chart content, the foregoing solution cannot effectively find the content of the chart, but must be manually paged in the file. Searching, causing a lot of inconvenience to users.

In summary, how to provide a technical means for solving the above problems is a technical problem that needs to be solved in the field.

In order to solve the problems disclosed above, the purpose of this case is to provide a technical solution for establishing a chart index by text information.

In order to achieve the above objectives, the present invention proposes a method for creating a chart index by using text information, and includes the following steps: reading the article paragraph information or sentence information associated with the target chart in the file, analyzing the article paragraph information or sentence information collection, to extract the plural Focus on vocabulary, weight processing and sorting of key vocabulary words to select one or more candidate vocabulary, and generate virtual chart message catalog and schema index information based on candidate words.

In order to achieve the above objectives, the present invention proposes a computer program product for creating a chart index by text information. When the computer device loads and executes the computer program product, the steps described in the method of creating a chart index by text information can be completed.

In summary, in this case, the text indexing method and the computer program product can effectively solve the shortcomings of the prior art by extracting the key words of interest to find the chart content corresponding to the keyword.

S11~S42‧‧‧Steps

FIG. 1 is a schematic flow chart of an embodiment of the present invention.

FIG. 2 is a schematic structural diagram of a sample file of the present invention.

Figure 3 is a flow chart of a model training step of the present case.

Figure 4 is a flow chart for identifying the paragraph or sentence of the article in this case.

Figure 5 is a flow chart for analyzing the passage or sentence of the article in this case.

FIG. 6 is a schematic diagram of a user browsing screen according to an embodiment of the present invention.

The specific embodiments are described below to illustrate the embodiments of the invention, but are not intended to limit the scope of the invention.

The first embodiment of the present invention provides a method for establishing a chart index by using text information. The method comprises the steps of: reading the article paragraph information or sentence information associated with the target chart in the file, analyzing the article paragraph information or the sentence information collection, extracting a plurality of key vocabulary words, weighting and sorting the key words of interest, and sorting, The one or more candidate words are selected, and the virtual chart message directory and the schema index information are generated according to the candidate words.

In another embodiment, the foregoing method extracts key vocabulary words through a collection of article information or sentence information by Name Entity Recognition. In another embodiment, the foregoing method analyzes the article paragraph information or the sentence information collection through vocabulary statistical lexical extraction to extract the focused vocabulary.

In another embodiment, the named entity identification method of the foregoing method extracts a vocabulary of a specific proper noun. In another embodiment, the proper noun of the foregoing method further includes at least one of a person name vocabulary, a place name vocabulary, and an organization name vocabulary.

In another embodiment, the lexical statistical lexical method of the foregoing method extracts a specific vocabulary using a suffix digital lexical grammar. In another embodiment, the foregoing method runs on a cloud data management system.

In another embodiment, the foregoing method performs weight processing according to word frequency and reverse file frequency. In another embodiment, the foregoing method multiplies the vocabulary of the high word frequency by the low file frequency of the vocabulary in the total number of files to generate corresponding weight values and filter specific words.

The second embodiment of the present invention further provides a computer program product for creating a chart index by using text information. When the computer device loads and executes the computer program product, the steps described in the foregoing method can be completed.

The following description of the present invention is based on the text indexing method of the first embodiment. However, the second embodiment can also achieve the same or similar technical effects by using the text information to create a chart index computer program product.

Please refer to FIG. 1 , which is a schematic flowchart of an embodiment of the present invention. In this example, the method of creating a graph index by using text information is run in the cloud data management system, but the application level is not limited to this, and the process steps are as follows:

S11: Collect article passages or sentences related to the chart in the archive.

S12: Using the named entity identification technology, analyzing the paragraphs or sentences of the articles related to the chart in the archive, and extracting the key words of interest related to the chart.

S13: Using the vocabulary statistical word-selling technique, analyzing the paragraphs or sentences of the articles related to the chart in the archive, and extracting the key words of interest related to the chart.

S14: The key vocabulary words are sorted according to the weight formula, and then the weights are sorted by the second time, and the top n key vocabulary words are selected and assigned to the chart.

S15: Generate a virtual chart message directory or a schema index to provide a user search.

The details of the above steps are as follows: S11 collects the paragraphs or sentences related to the chart in the archive. This step collects the name of the chart in the file, the name of the article appearing before and after the chart, and the file mentioned in the article. The keyword content of the chart, 4. The file refers to the front and back paragraphs of the chart, the 5. file annotation, or the 6. file tag text and other articles or sentences.

Please refer to FIG. 2 , which is a schematic structural diagram of a sample file. If the file is a Word file, and the content contains the paragraph content (paragraph 1 to paragraph 5), the schema and its description (Figure 2-2-1 image name A, and Figure 2-2-2 image name B ). In the execution, the automatic parsing sentence method is used first, and the article name A and the picture 2-2 appearing above and below the second and fifth lines of the sentence and the file are mentioned in the figure. The sentence, together with the original annotations or labels of the file, are collected as sentences for subsequent identification and abstraction.

Step S12 uses the named entity identification technology to analyze the paragraphs or sentences of the articles related to the chart in the archive and to extract the key vocabulary related to the chart. The step is mainly to analyze the collection of articles or sentences generated by S11, the details. The method includes the following steps: The named entity identification technology belongs to one of the techniques of the natural language processing project, and provides the common vocabulary words such as the name of the person, the place name and the organization name in the full-text file. A supervised model is established, and the identification model can be established according to the model training steps of FIG. described as follows:

S21: Define the label type; set the label type according to the person name, the place name, and the organization name. The label is currently set with: a person name start label (B_PER), a person name internal label (I_PER), a place name start label (B_LOC), and a place name internal label (I_LOC). ), organization name start tag (B_ORG), organization name internal tag (I_ORG), non-professional noun tag (O) and other seven categories.

S22: Collect training corpus. Collect a collection of sentences in related fields to provide model training.

S23: Define features. Define a question and judgment probability for a single word or word, such as whether the word or word is a hundred names, 1 for yes, 0 for no. There may be a lot of characteristics, and they are collected into a set of feature vectors.

S24: The feature dictionary is established with a rule dictionary.

S25: Model training is established; the combination of "tag-feature vectors" of each word in the sentence is trained according to Conditional random fields (CRFs). The conditional random field is an undirected graph model. The vertex in the graph model represents a random variable. The line between the vertices represents the dependence relationship between random variables. In the conditional random field, the distribution of the random variable Y is Conditional probability, given the observed value is the random variable X. In principle, the layout of the conditional random domain graph model can be arbitrarily given. The commonly used layout is a chained architecture. The link architecture is either training, inference, or decoding. Above, there are efficient algorithms for label discrimination and calculation.

S26: Actual corpus evaluation; given the accuracy of the test model, find the feature vector and judge the accuracy of the model. (If the corpus has a recognition error, provide pre-processing to add dictionary files or post-process manual adjustment re-training model)

When the model is established, it can be used to analyze the paragraph or sentence of the article collected through step S11). Through the identification process in Figure 4, draw the key vocabulary related to the chart. The identification process is described as follows:

S31: Feature extraction.

S32: The feature dictionary corresponds to the rule dictionary.

S33: Model identification, which identifies the paragraph or sentence of the article related to the chart in the file to be analyzed.

S34: Output format processing; the model predicts the label of each word in the sentence, and breaks the word according to the B and I of the label. For example, the phrase "the sixth chairman, Mr. Cai Lixing" can be identified as "O-O-O-O-O-O-B_PER-I_PER-I_PER-O-O" by the model. Then the vocabulary of the first B and the last I is taken as the result of a single vocabulary.

S35: Annotation result analysis; the sentence is based on the label as a word break, and the proper noun is extracted.

S13 uses the vocabulary statistical word-selling technique to analyze the paragraphs or sentences of the articles related to the chart in the archives, and draws the key vocabulary related to the chart; this step mainly analyzes the collection of articles or sentences generated by S11). The method includes the steps shown in Figure 5 below:

S41: Using the suffix array to extract vocabulary.

S42: Filter the vocabulary according to the rules.

S41: Using the suffix array to extract vocabulary. In this case, the suffix array is used to convert the string into a suffix array, and the longest prefix string in the sequence is taken as a candidate. The basic calculus concept is as follows. Suppose a string S of length n is indexed from 0 to n-1 for each of the n characters existing in the string S, and S[i] represents the suffix string of the index i, assuming S = "abracadabra", after indexing, the results are shown in Table 1 below.  

This string has a total of 11 suffixes, which are sorted in lexicographic order to produce the suffix string of Table 2. The frequency is the number of times the suffix appears in the prefix part of all suffix strings:  

If the frequency of the above suffix string is greater than one, it is a possible candidate word. However, if the candidate word is included in other candidate words and the frequency is not higher than the longer candidate word, the candidate word is filtered out. According to the above example, the two candidates "a" and "abra" can be obtained.

Take another Chinese string as an example: "Natural Science and Humanities and Social Sciences and New Generation Social Science" (Table 3), sorted by Suffix array) to get a partial suffix array as shown in the following table, and then sorted and frequency statistics will be The two candidate words appearing in "Science" three times in Table 4 and two times in "Social Science" are drawn.

S42: Filtering the vocabulary according to the rule and according to the suffix array method can capture a large number of possible candidate words from the article paragraph or sentence related to the chart in the file, but also contains a large number of useless noise words. These noise vocabularies not only consume a lot of computation time, but the wrong phrases will greatly affect the results of the chart labels. Therefore, the vocabulary is filtered according to the rules, and the useless words are filtered out. E.g:

1. Punctuation rules: Words with Chinese and English punctuation marks in the extracted words are filtered out.

2. Beginning rules: The vocabulary at the beginning of the extracted vocabulary that matches the specific character is filtered out, such as "in...", "from...".

3. The last word rule: the vocabulary at the end of the extracted vocabulary that matches the specific character is filtered out, mainly for specific words such as "...Mr.", "...Director".

4. The Maximum Matching Rule: is compared with the rule dictionary S32 prepared in advance. If the rule dictionary includes the word, all the substrings of the word in the extracted result are filtered out. For example, in the rule dictionary, the term "human resource management system" is included, and "human resource management", "resource management system", and "resource management" are all filtered out.

In step S14, the key vocabulary words are sorted according to the weight formula, and then the weights are sorted by the second time, and the top n key vocabulary words are selected and assigned to the chart; here, the weight formula is used to use the term frequency (Term-Frequency, TF) and the reverse direction. The Inverse Document Frequency (IDF) is used to calculate the weight of the focused vocabulary. In this weight formula, the calculation method of "TF-IDF algorithm" is as follows:

S14.1: First calculate the word frequency TF (Term Frequency, TF): suppose dj is "a specific file", ti is "one of the words or words used in the file", then the calculation method of tfi is ti in each The sum of the occurrences of the article dj, divided by the sum of all the words in each article, as shown in Figure 10, is mainly to emphasize that the more words appear more and more important.

S14.2: Recalculating the inverse document frequency (IDF) is a measure of the universal importance of a word. The IDF of a particular vocabulary can be obtained by dividing the total number of all files by the number of files that contain the vocabulary in the total number of files, and then obtaining the logarithm of the obtained quotient, as shown in the eleventh formula, mainly emphasizing the more files appearing. The less important the word

S14.3: Calculate tfi*idfi, multiply the high word frequency in a particular file by the low file frequency of the vocabulary in the total number of files, and then generate TF-IDF weight values, and TF-IDF tends to Filter out common words and retain important words, as shown in Figure 12.

TF - IDF _i = tf _i * idf _i

S14.4: The vocabulary weight calculation formula consists of six factors: the TF-IDF value of the chart's own name.

TF-IDF value of the article paragraph appearing before and after the chart

Archive The TF-IDF value of the keyword content of the chart mentioned in this article

The TF-IDF value of the paragraph before and after the keyword of the chart mentioned in this article

TF-IDF value of file annotation

TF-IDF value of the file label text

Vocabulary _i weight value _i = λ _var1 * +λ _var2 * +λ _var3 * +λ _var4 * +λ _var5 * +λ _var6 * +λ _ε

The parameters λ _var1 , λ _var2 , λ _var3 , λ _var4 , λ _var5 , λ _var6 , λ _ε are variables, and the extracted words constitute an equation. Let the known important words reach the maximum value of the ideal value of the formula. n vocabulary constitutes n formulas, which can be solved by parameter optimization algorithm (such as linear regression algorithm). The lower bound of each parameter λ is set to 0.1, and the upper bound is optimized by the initial training data. The maximum value of the best solution found by the algorithm is added to its upper bound value.

Then, S12 uses the named entity identification technology and the S13 vocabulary statistical word-taking technique to draw the key vocabulary related to the chart, and uses the S14) TF-IDF algorithm to select the vocabulary of the first m name. Then, the S12 named entity identification technology and the selected vocabulary are given weights α, S13, and the vocabulary selected by the lexical statistical word-taking technique is given a weight β, wherein the total of α and β is 1, and a weight score is recalculated. The first n words are assigned to the chart and become the chart index label.

A virtual chart message directory or schema index is generated in step S15 to provide a user search. According to the steps of S11 to S14, a virtual chart message directory or a graphical index can be established in the system, and the user can find the chart by focusing on the vocabulary, and can display the file name and path where the chart is located.

The user browsing screen provided by the cloud data management system is shown in FIG. 6 , which includes a chart search field, a virtual chart message directory, a chart index name, a chart thumbnail, a file name and a file path and a file hyperlink. The aforementioned chart search option provides the user with a keyword to query. The aforementioned virtual chart message directory can display the index tag directory of all current chart messages. For example, when the user enters "relativistic time" in the chart search field, he can find two files "relativistic scientist profile.pptx" and "contemporary math formula.doc" in the data, and find and key in the file. The image associated with the word (Einstein image and the relativistic formula) can quickly know the corresponding chart content in the file, and after clicking the index tab directory of the chart message, you can go to the corresponding paragraph of the file via the hyperlink. .

The detailed description of the preferred embodiments of the present invention is intended to be limited to the scope of the invention, and is not intended to limit the scope of the invention. The patent scope of this case.

Claims (8)

A method for creating a chart index by using text information, comprising: reading article paragraph information or sentence information associated with a target chart in the file; analyzing the article paragraph information or the sentence by a named entity identification analysis method and a vocabulary statistical drawing method respectively a collection of information to extract a plurality of key vocabulary words, wherein the named entity identification analysis method comprises: defining a plurality of tag types and determining a feature vector set; establishing a supervised model based on the tag types and the feature vector set; The supervised model identifies the article segment information or the collection of the sentence information to output the article segment information or the tags of the words in the sentence information; and determines the tags according to the article segment information or the tags of the words in the sentence information. Focusing on vocabulary; respectively assigning different weights to the key vocabulary extracted by the naming entity identification analysis method and the vocabulary statistical lexical lexicon, and then sorting to select one or more candidate vocabulary; and generating virtual according to the candidate word Chart message catalog and schema index information. The method for creating a chart index by using text information as described in claim 1, wherein the named entity identification method extracts a vocabulary of a specific proper noun. The method for creating a chart index by using text information as described in claim 2, wherein the proper noun further comprises at least one of a person name vocabulary, a place name vocabulary, and an organization name vocabulary. As shown in claim 1, the chart indexing method is established by using text information, wherein the vocabulary statistical lexical method extracts a specific vocabulary by using a suffix digital lexical grammar. The method of creating a chart index by using text information as described in claim 1 is running in cloud data management. system. The method for creating a chart index by using text information as described in claim 1 is based on word frequency and reverse file frequency for weight processing. The method for creating a chart index by text information as described in claim 6 is to multiply the vocabulary of the high word frequency by the low file frequency of the vocabulary in the total number of files to generate a corresponding weight value and filter the specific vocabulary. A computer program product for creating a chart index by text information. When the computer device loads and executes the computer program product, the method of any one of claims 1 to 7 can be completed.