WO2019153552A1 - Automatic tagging method and apparatus, and computer device and storage medium - Google Patents

Abstract

An automatic tagging method and apparatus, and a computer device and a storage medium. The method comprises: performing word segmentation preprocessing on a text to be tagged to obtain a preprocessed text (S101); inputting the preprocessed text to a word inverse frequency TF-IDF algorithm model to obtain a keyword set of the text to be tagged (S102); obtaining an initialized transfer matrix according to the keyword set of the text to be tagged, performing multi-time iterative multiplication operation on the initialized transfer matrix and initial keyword probability distribution until convergence to obtain final keyword probability distribution (S103); and obtaining a corresponding row of a maximum probability value in the final keyword probability distribution, obtaining a keyword corresponding to the corresponding row of the maximum probability value, and setting the keyword as a tag of the text to be tagged (S104). According to the method, an article is tagged by means of automatic learning, so that manual tagging is avoided, the tagging efficiency is improved, and the manpower cost is reduced.

Description

Automatic labeling method, device, computer device and storage medium

This application claims the priority of the Chinese patent application filed on February 12, 2018, the Chinese Patent Office, Application No. 201101145692.7, the application of the name of the "automatic labeling method, device, computer equipment and storage medium", the entire contents of which are The citations are incorporated herein by reference.

Technical field

The present application relates to the field of article classification technology, and in particular, to a method, an apparatus, a computer device and a storage medium for automatic labeling.

Background technique

The article's tags help in the search and classification of articles. The current common method is to manually tag, that is, the author edits the tags for their articles, but not all authors tag their articles. If a large number of unlabeled articles are added by manual marking, the efficiency is extremely low, and the labor cost is greatly increased.

Summary of the invention

The present application provides a method, a device, a computer device and a storage medium for automatically labeling, aiming at solving the problem that the prior art mass unlabeled articles are manually added by labeling, resulting in extremely low efficiency. And greatly increased the problem of labor costs.

In a first aspect, the present application provides a method for automatically labeling, which comprises: preprocessing a word to be tagged to obtain a preprocessed text; and inputting the preprocessed text into a reverse frequency TF-IDF algorithm model to obtain a The keyword set of the tagged text; the initial transfer matrix is obtained according to the keyword set of the tagged text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the final probability distribution of the keyword is obtained. Obtaining a corresponding row of the maximum value of the probability in the final probability distribution of the keyword, obtaining a keyword corresponding to the row corresponding to the maximum value of the probability, and setting the keyword as a tag of the text to be tagged.

In a second aspect, the present application provides an apparatus for automatically labeling, comprising:

a text preprocessing unit, configured to perform word segmentation on the tagged text to obtain a preprocessed text;

a keyword set obtaining unit, configured to input the preprocessed text into a reverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text;

The final probability distribution obtaining unit is configured to obtain an initial transition matrix according to the keyword set of the to-be-labeled text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the final probability distribution of the keyword is obtained. ;

The marking unit is configured to obtain a corresponding row of the probability maximum value in the final probability distribution of the keyword, obtain a keyword corresponding to the corresponding row of the maximum probability, and set the keyword as a label of the to-be-labeled text.

In a third aspect, the present application further provides a computer device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, the processor implementing the computer program The method of automatic labeling according to any of the preceding claims.

In a fourth aspect, the present application also provides a storage medium, wherein the storage medium stores a computer program, the computer program comprising program instructions, the program instructions, when executed by a processor, causing the processor to execute the application A method of automatic labeling as described in any of the preceding claims.

The application provides a method, device, computer device and storage medium for automatic labeling. The method labels the articles by means of automatic learning, avoids manual marking, improves marking efficiency and saves labor costs.

DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments will be briefly described below. Obviously, the drawings in the following description are some embodiments of the present application, For the ordinary technicians, other drawings can be obtained based on these drawings without any creative work.

FIG. 1 is a schematic flowchart of a method for automatically labeling according to an embodiment of the present application;

2 is a schematic flowchart of a sub-flow of a method for automatically labeling according to an embodiment of the present application;

3 is a schematic diagram of another sub-flow of a method for automatically labeling according to an embodiment of the present application;

4 is a schematic block diagram of an apparatus for automatically labeling according to an embodiment of the present application;

FIG. 5 is a schematic block diagram of a subunit of an automatic labeling apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic block diagram of another subunit of an automatic labeling apparatus according to an embodiment of the present disclosure;

FIG. 7 is a schematic block diagram of a computer device according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application are clearly and completely described in the following with reference to the drawings in the embodiments of the present application. It is obvious that the described embodiments are a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

Please refer to FIG. 1. FIG. 1 is a schematic flow chart of a method for automatically labeling according to an embodiment of the present application. The method is applied to terminals such as desktop computers, laptop computers, and tablet computers. As shown in FIG. 1, the method includes steps S101 to S104.

S101: Perform word segmentation on the to-be-labeled text to obtain pre-processed text.

As shown in FIG. 2, the step S101 includes the following steps:

S1011: Perform word segmentation on the tagged text to obtain a word segmentation text.

In this embodiment, the word segmentation method based on the probability statistical model performs word segmentation on the tagged text. The steps of the word segmentation method based on the probability and statistics model are as follows:

Step 11: For a substring S to be segmented, all candidate words w1, w2, ..., wi, ..., wn are taken out in order from left to right;

Step 12: Find the probability value P(wi) of each candidate word in the dictionary, and record all the adjacent words of each candidate word;

Step 13: Calculate the cumulative probability of each candidate word, and compare and obtain the best neighbors of each candidate word;

Step 14. If the current word wn is the end word of the string S and the cumulative probability P(wn) is the largest, then wn is the end word of S;

Step 15. Beginning with wn, in order from right to left, the best left neighbor words of each word are sequentially output, that is, the word segmentation result of S.

S1012: Set a weighting value by using a participle included in the text after the word segmentation.

In this embodiment, the weighting process is performed by the word segmentation in the tagged text that has been segmented, that is, the tagged text that has been segmented can be regarded as composed of a plurality of word segments, and the entire article has been performed. The tagged text of the word segmentation weights the word segmentation in the text from beginning to end by factors such as position, part of speech, length, etc., according to the following rules:

The first word of the text is the title, giving the weight 8*; the first word at the beginning of the paragraph is equal to the "summary", then the weight is given 5*; the first word at the beginning of the paragraph is equal to "keyword" or "conclusion", then The weight is 5*; the length of the word is equal to 2, and the weight is 3*; the part of speech is a noun, and the weight is given 2*; otherwise, the weight of each paragraph is given 1*.

S1013. Delete the stop words in the text after the word segmentation, and count the word frequency of each word segment to obtain the first triplet.

The triplet <w _i ,fre _i ,v _i > represents the processed result set of the to-be-labeled text, where w _i is the word, fre _i is the number of times the word w _{i is} weighted, and v _i is the word in the text Position weight; wherein, when the weights are included in the participles included in the text after the word segmentation, the stop words are deleted (the stop words include virtual words, modal words, adverbs, symbols, words of a word, etc.) The stop word will not be used as a candidate for the keyword), and the candidate keywords can be accurately screened for subsequent processing.

S1014. Obtain a similarity between words in the first triad that is greater than a word frequency threshold of the preset word frequency threshold;

Wherein, by word similarity calculation, all word similarities sim _ij of the word frequency fre _i >2 in the first triad <w _i , fre _i , v _i > are calculated; when sim _ij >0.9, the similarity of the two words is considered Very high, can be replaced in the text, will return the quads <w _i , w _j , sim _ij , fre _i +fre _j >, and delete the word w _j in the first triple. The quaternion <w _i , w _j , sim _ij , fre _i +fre _j > represents a set of similarities after calculating partial words in the triple, where sim _ij represents the similarity of the words w _i , w _j , fre _i +fre _j represents the sum of the word frequencies of the two words.

S1015: If the word similarity between the word segments is greater than the preset word similarity threshold, retain any one of the word segments to obtain the second triplet, and use the second triplet as the pre-processed text.

Wherein, in the first triplet <w _i ,fre _i ,v _i >, look for the words in the quads <w _i , w _j , sim _ij , fre _i +fre _j >; when the triad is fre _{i is} replaced by fre _i +fre _j in the quaternary, reconstituting the second triplet <wi,fre _i +fre _j ,vi>, the second ternary <w _i ,fre _i +fre _j ,v _i > is the pre-processed text. Through text preprocessing, the obtained preprocessed text satisfies the input criteria of the keyword screening model (that is, the text is vectorized), and the keywords of the article can be obtained more accurately. When the tagged text is segmented, the word frequency of each word can be counted as a candidate for the article tag.

S102. Enter the preprocessed text into the inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text.

In an embodiment, the word inverse frequency TF-IDF algorithm model is:

Wherein, the TF partial molecule n _i,j represents the number of occurrences of the word t _i in the text j, the denominator represents all the word frequency words in the text j, the IWF part numerator represents the sum of the frequency of all words in the corpus, and nt _j represents the word t The total frequency that _i appears in the corpus.

As shown in FIG. 3, the step S102 includes the following steps:

S1021: Generating a corpus term statistical result set;

S1022: Obtain a preprocessed text;

S1023: Input the pre-processed text and the corpus term statistical result set into the word inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text.

In this embodiment, the corpus term statistical result set is obtained based on the corpus. The corpus is that the user pre-selects a certain number of articles (such as 2000 articles), adopts the text preprocessing algorithm in steps S10111-S10115, ignores the steps of similarity calculation in step S10114, and processes the articles in the corpus separately to obtain a binary group < w _i ,fre _i >, where w _i is a word, and fre _i is the frequency at which the word w _{i is} weighted. Combine all the binary groups <w _i ,fre _i >, and get <w _i ,fre _isum >, where fre _isum is the total frequency of the word w _i appearing in the corpus, that is, <wi,freisum> is the generated corpus Word statistics result set. The pre-processed text and the corpus term statistical result set are input into the word inverse frequency TF-IDF algorithm model to obtain the keyword set of the tagged text, and the obtained keyword set has high accuracy.

S103. The initialization transition matrix is obtained according to the keyword set of the to-be-labeled text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the keyword final probability distribution is obtained.

In an embodiment, the initialization transition matrix is obtained according to the keyword set of the to-be-labeled text, and the initialization transition matrix is an n-dimensional square matrix, the dimension of the n-dimensional square matrix and the total number of keywords in the keyword set. Equal; the initial keyword probability distribution is an n-dimensional column vector whose row value is 1/n; wherein n is a positive integer equal to the total number of keywords in the keyword set;

The initialization transition matrix and the initial keyword probability distribution are recorded as V _m =MV _m-1 through multiple iterative multiplication operations, where m is a positive integer, V ₀ is an initial keyword probability distribution, and M is an initialization. Transfer matrix.

In the present embodiment, for example, the total number of keywords in the keyword set is four, which are respectively denoted as A, B, C, and D. According to the position, part of speech, length and other factors of the keyword, the initialization transfer matrix M is obtained as follows:

Suppose that the probability of each keyword being the final label of the text to be tagged is equal, ie 1/n; therefore the initial keyword probability distribution is an n-dimensional column vector V ₀ whose values are all 1/n; V _n = MV _n-1 calculates the final probability distribution Vn of the keyword (this initial transfer matrix M is multiplied by V _n-1 , and it will eventually converge after repeated iterations (generally about 30 times), and there will be no iterative conditions) .

S104. Acquire a corresponding row of the maximum probability value in the final probability distribution of the keyword, obtain a keyword corresponding to the corresponding row of the maximum value of the probability, and set the keyword as a label of the to-be-labeled text.

As another embodiment of the steps S103-S104, the initialization transition matrix may be obtained according to the keyword set of the to-be-labeled text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations, when Vn When a row of n-dimensional column vectors appears larger than the preset probability value, the iteration is stopped and the keyword corresponding to the row is used as the label of the text to be marked. If multiple lines are greater than the preset probability value at the same time, the keywords corresponding to the lines are simultaneously used as the labels of the to-be-labeled text.

In an embodiment, before the step S101, the method further includes:

Step 1. Crawl the text to be tagged and store it in the MongoDB database. That is, the original data is crawled from the Internet, and the text to be tagged is stored and stored in the MongoDB database. By crawling the data, you can set a filter condition that crawls the text of the unset label for labeling.

The method labels the articles by means of automatic learning, avoids manual marking, improves marking efficiency and saves labor costs.

The embodiment of the present application further provides an automatic labeling device for performing the method for automatically labeling any of the foregoing. Specifically, please refer to FIG. 4 , which is a schematic block diagram of an apparatus for automatically labeling provided by an embodiment of the present application. The automatic tagging device 100 can be installed in a desktop computer, a tablet computer, a laptop computer, or the like.

As shown in FIG. 4, the apparatus 100 for automatic labeling includes a text pre-processing unit 101, a keyword set acquisition unit 102, a final probability distribution acquisition unit 103, and a marking unit 104.

The text preprocessing unit 101 is configured to perform word segmentation preprocessing on the to-be-labeled text to obtain pre-processed text.

As shown in FIG. 5, the text pre-processing unit 101 includes the following sub-units:

The word segmentation unit 1011 is configured to perform word segmentation on the tagged text to obtain a word segmentation text.

In this embodiment, the word segmentation method based on the probability statistical model performs word segmentation on the tagged text. The word segmentation method based on probability and statistical model is as follows:

1) For a substring S to be word-divided, all candidate words w1, w2, ..., wi, ..., wn; 2) are taken in order from left to right to find the probability value P of each candidate word in the dictionary ( Wi), and record all the adjacent words of each candidate word; 3) calculate the cumulative probability of each candidate word, and compare the best neighbors of each candidate word; 4) if the current word wn is the string S The last word, and the cumulative probability P(wn) is the largest, then wn is the end word of S; 5) starting from wn, in order from right to left, the best left neighbor of each word is output, that is, S Word segmentation results.

The weighting unit 1012 is configured to set a weighting value for the participle included in the text after the word segmentation.

In this embodiment, the weighting process is performed by the word segmentation in the tagged text that has been segmented, that is, the tagged text that has been segmented can be regarded as composed of a plurality of word segments, and the entire article has been performed. The word-to-label text of the word segmentation weights each participle in the text from beginning to end according to factors such as position, part of speech, length, etc., according to the following rules: the first word of the text is the title, and the weight is 8*; the first one of the paragraph If the word is equal to "summary", the weight is given 5*; the first word at the beginning of the paragraph is equal to "keyword" or "conclusion", then the weight is given 5*; the length of the word is equal to 2, the weight is given 3*; the part of speech is noun , the weight is given 2*; others, each paragraph is given a weight of 1*.

The statistic unit 1013 is configured to delete the stop words in the text after the word segmentation, and count the word frequency of each word segment to obtain the first triplet.

The triplet <w _i ,fre _i ,v _i > represents the processed result set of the to-be-labeled text, where w _i is the word, fre _i is the number of times the word w _{i is} weighted, and v _i is the word in the text Position weight; wherein, when the weights are included in the participles included in the text after the word segmentation, the stop words are deleted (the stop words include virtual words, modal words, adverbs, symbols, words of a word, etc.) The stop word will not be used as a candidate for the keyword), and the candidate keywords can be accurately screened for subsequent processing.

The similarity obtaining unit 1014 is configured to obtain the similarity of words between the word segments of the first triplet that are greater than the preset word frequency threshold.

Wherein, by word similarity calculation, all word similarities sim _ij of the word frequency fre _i >2 in the first triad <w _i , fre _i , v _i > are calculated; when sim _ij >0.9, the similarity of the two words is considered Very high, can be replaced in the text, will return the quads <w _i , w _j , sim _ij , fre _i +fre _j >, and delete the word w _j in the first triple. The quaternion <w _i , w _j , sim _ij , fre _i +fre _j > represents a set of similarities after calculating partial words in the triple, where sim _ij represents the similarity of the words w _i , w _j , fre _i +fre _j represents the sum of the word frequencies of the two words.

The deleted word unit 1015 is configured to: if the word similarity between the word segments is greater than the preset word similarity threshold, retain any one of the word segments, obtain the second triplet, and use the second triplet as the preprocessed text.

Wherein, in the first triplet <w _i ,fre _i ,v _i >, look for the words in the quads <w _i , w _j , sim _ij , fre _i +fre _j >; when the triad is fre _{i is} replaced by fre _i +fre _j in the quaternary, reconstituting the second triplet <wi,fre _i +fre _j ,vi>, the second ternary <w _i ,fre _i +fre _j ,v _i > is the pre-processed text. Through text preprocessing, the obtained preprocessed text satisfies the input criteria of the keyword screening model (that is, the text is vectorized), and the keywords of the article can be obtained more accurately. When the tagged text is segmented, the word frequency of each word can be counted as a candidate for the article tag.

The keyword set obtaining unit 102 is configured to input the pre-processed text into the word inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text.

In an embodiment, the word inverse frequency TF-IDF algorithm model is:

Wherein, the TF partial molecule n _i,j represents the number of occurrences of the word t _i in the text j, the denominator represents all the word frequency words in the text j, the IWF part of the molecule represents the sum of the frequency of all words in the corpus, and nt _i represents the word t The total frequency that _i appears in the corpus.

As shown in FIG. 6, the keyword set obtaining unit 102 includes the following subunits:

a first processing unit 1021, configured to generate a corpus term statistical result set;

a second processing unit 1022, configured to acquire pre-processed text;

The keyword set calculation unit 1023 is configured to input the pre-processing text and the corpus term statistical result set into the word inverse frequency TF-IDF algorithm model to obtain a keyword set of the tagged text.

In this embodiment, the corpus term statistical result set is obtained based on the corpus. The corpus is that the user pre-selects a certain number of articles (such as 2000 articles), adopts the text pre-processing algorithm in the text pre-processing unit 101, ignores the steps of the similarity calculation, and processes the articles in the corpus separately to obtain a binary group <w _i , fre _i >, where w _i is a word, and fre _i is the frequency at which the word w _{i is} weighted. Combine all the binary groups <w _i ,fre _i >, and get <w _i ,fre _isum >, where fre _isum is the total frequency of the word w _i appearing in the corpus, that is, <wi,freisum> is the generated corpus Word statistics result set. The pre-processed text and the corpus term statistical result set are input into the word inverse frequency TF-IDF algorithm model to obtain the keyword set of the tagged text, and the obtained keyword set has high accuracy.

The final probability distribution obtaining unit 103 is configured to obtain an initial transition matrix according to the keyword set of the to-be-labeled text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the keyword final probability is obtained. distributed.

In an embodiment, the initialization transition matrix is obtained according to the keyword set of the to-be-labeled text, and the initialization transition matrix is an n-dimensional square matrix, the dimension of the n-dimensional square matrix and the total number of keywords in the keyword set. Equal; the initial keyword probability distribution is an n-dimensional column vector whose row value is 1/n; wherein n is a positive integer equal to the total number of keywords in the keyword set;

The initialization transition matrix and the initial keyword probability distribution are recorded as V _m =MV _m-1 through multiple iterative multiplication operations, where m is a positive integer, V ₀ is an initial keyword probability distribution, and M is an initialization. Transfer matrix.

In the present embodiment, for example, the total number of keywords in the keyword set is four, which are respectively denoted as A, B, C, and D. According to the position, part of speech, length and other factors of the keyword, the initialization transfer matrix M is obtained as follows:

Suppose that the probability of each keyword being the final label of the text to be tagged is equal, ie 1/n; therefore the initial keyword probability distribution is an n-dimensional column vector V ₀ whose values are all 1/n; V _n = MV _n-1 calculates the final probability distribution Vn of the keyword (this initial transfer matrix M is multiplied by V _n-1 , and it will eventually converge after repeated iterations (generally about 30 times), and there will be no iterative conditions) .

The marking unit 104 is configured to obtain a corresponding row of the probability maximum value in the final probability distribution of the keyword, obtain a keyword corresponding to the corresponding row of the maximum probability, and set the keyword as a label of the to-be-labeled text.

As another embodiment of the final probability distribution obtaining unit 103 and the marking unit 104, the initialization transition matrix may be obtained according to the keyword set of the to-be-labeled text, and the initial transition matrix and the initial keyword probability distribution are iterated multiple times. In the multiplication operation, when a row of the n-dimensional column vector of Vn appears to be greater than the preset probability value, the iteration is stopped and the keyword corresponding to the row is used as the label of the to-be-labeled text. If multiple lines are greater than the preset probability value at the same time, the keywords corresponding to the lines are simultaneously used as the labels of the to-be-labeled text.

In an embodiment, the apparatus 100 for automatic labeling further includes:

Crawl unit for crawling the text to be tagged and storing it in the MongoDB database. That is, the original data is crawled from the Internet, and the text to be tagged is stored and stored in the MongoDB database. By crawling the data, you can set a filter condition that crawls the text of the unset label for labeling.

It can be seen that the device labels the articles by means of automatic learning, avoids manual marking, improves marking efficiency and saves labor costs.

The above automatic labeling device can be implemented in the form of a computer program that can be run on a computer device as shown in FIG.

Please refer to FIG. 7. FIG. 7 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 device can be a terminal. The terminal can be an electronic device such as a tablet computer, a notebook computer, a desktop computer, or a personal digital assistant.

Referring to FIG. 7, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected by a system bus 501, wherein the memory can include a non-volatile storage medium 503 and an internal memory 504. The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method of automatic tagging. The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500. The internal memory 504 provides an environment for operation of the computer program 5032 in the non-volatile storage medium 503, which when executed by the processor 502, can cause the processor 502 to perform a method of automatic tagging. The network interface 505 is used for network communication, such as sending assigned tasks and the like. It will be understood by those skilled in the art that the structure shown in FIG. 7 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation of the computer device 500 to which the solution of the present application is applied, and a specific computer device. 500 may include more or fewer components than shown, or some components may be combined, or have different component arrangements.

The processor 502 is configured to run a computer program 5032 stored in the memory to implement the following functions: performing word segmentation on the tagged text to obtain preprocessed text; and inputting the preprocessed text into the word inverse frequency TF-IDF The algorithm model obtains a keyword set of the tagged text; the initial transfer matrix is obtained according to the keyword set of the tagged text, and the initial transfer matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, Key words final probability distribution; obtain the corresponding row of the probability maximum value in the final probability distribution of the keyword, obtain the keyword corresponding to the corresponding row of the maximum probability, and set the keyword as the label of the to-be-labeled text.

In an embodiment, the processor 502 further performs the following operations: performing word segmentation on the tagged text to obtain a word segmentation text; setting a weighting value for the segmentation word included in the segmentation word text; deleting the stop word in the text after the word segmentation, and The word frequency of each participle is counted to obtain a first triad; the word similarity between the word segments of the first triad greater than the preset word frequency threshold is obtained; if the word similarity between the word segments is greater than the preset word similarity The threshold, retaining any of the participles, obtaining the second triad, and using the second triad as the pre-processed text.

In an embodiment, the processor 502 further performs the following operations: generating a corpus term statistical result set; obtaining the preprocessed text; and inputting the preprocessed text and the corpus term statistical result set into the term inverse frequency TF-IDF algorithm model to obtain a to-be-labeled a keyword set of text; wherein the word inverse frequency TF-IDF algorithm model is:

Wherein, the TF partial molecule n _i,j represents the number of occurrences of the word t _i in the text j, the denominator represents all the word frequency words in the text j, the IWF part of the molecule represents the sum of the frequency of all words in the corpus, and nt _i represents the word t The total frequency that _i appears in the corpus.

In an embodiment, the initialization transition matrix is obtained according to the keyword set of the to-be-labeled text, and the initialization transition matrix is an n-dimensional square matrix, the dimension of the n-dimensional square matrix and the total number of keywords in the keyword set. Equal; the initial keyword probability distribution is an n-dimensional column vector whose row value is 1/n; wherein n is a positive integer equal to the total number of keywords in the keyword set;

The initialization transition matrix and the initial keyword probability distribution are recorded as V _m =MV _m-1 through multiple iterative multiplication operations, where m is a positive integer, V ₀ is an initial keyword probability distribution, and M is an initialization. Transfer matrix.

In an embodiment, the processor 502 also performs the following operations: crawling the to-be-labeled text and storing it in the MongoDB database.

It will be understood by those skilled in the art that the embodiment of the computer device shown in FIG. 7 does not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than illustrated. Or combine some parts, or different parts. For example, in some embodiments, the computer device may include only a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the embodiment shown in FIG. 7, and details are not described herein again.

It should be understood that, in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be another general-purpose processor, a digital signal processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

In another embodiment of the present application, a storage medium is provided. The storage medium can be a non-transitory computer readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The method of automatically tagging the embodiment of the present application when the program instruction is executed by the processor.

The storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or a memory of the device. The storage medium may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (SMC), a secure digital (SD) card, and a flash memory card. (Flash Card), etc. Further, the storage medium may also include both an internal storage unit of the device and an external storage device.

A person skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the device, the device and the unit described above can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

The foregoing is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any equivalents can be easily conceived by those skilled in the art within the technical scope disclosed in the present application. Modifications or substitutions are intended to be included within the scope of the present application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims (20)

A method for automatically labeling, comprising: Performing word segmentation on the tagged text to obtain preprocessed text; Entering the pre-processed text into the inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text; The initial transfer matrix is obtained according to the keyword set of the tagged text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the final probability distribution of the keyword is obtained; Obtain a corresponding row of the maximum value of the probability in the final probability distribution of the keyword, obtain a keyword corresponding to the corresponding row of the maximum value of the probability, and set the keyword as a tag of the text to be tagged. The method for automatically tagging according to claim 1, wherein the pre-processing of the word-to-be-labeled text to obtain pre-processed text comprises: The wording of the tagged text is used to obtain the word after the word segmentation; Setting a weight value for the participles included in the text after the word segmentation; Delete the stop words in the text after the word segmentation, and count the word frequency of each word segment to obtain the first triplet; Obtaining a similarity between words in the first triad that is greater than a word frequency threshold of the preset word frequency threshold; If the word similarity between the word segments is greater than the preset word similarity threshold, any one of the word segments is retained, the second triad is obtained, and the second triad is used as the pre-processed text. The method of automatically tagging according to claim 2, wherein the word inverse frequency TF-IDF algorithm model is:

Wherein, the TF partial molecule n _i,j represents the number of occurrences of the word t _i in the text j, the denominator represents all the word frequency words in the text j, the IWF part of the molecule represents the sum of the frequency of all words in the corpus, and nt _i represents the word t The total frequency of _i appearing in the corpus; And inputting the pre-processed text into the inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text, including: Generating a corpus term statistical result set; Obtain the preprocessed text; The pre-processed text and the corpus term statistical result set are input into the word inverse frequency TF-IDF algorithm model to obtain a keyword set of the tagged text. The method for automatically labeling according to claim 1, wherein the initial transfer matrix is obtained according to the keyword set of the to-be-labeled text, and the initial transfer matrix is an n-dimensional square matrix, and the n-dimensional square matrix The dimension is equal to the total number of keywords in the keyword set; the initial keyword probability distribution is an n-dimensional column vector whose row value is 1/n; wherein n is equal to the total number of keywords in the keyword set Positive integer The initialization transition matrix and the initial keyword probability distribution are recorded as V _m =MV _m-1 through multiple iterative multiplication operations, where m is a positive integer, V ₀ is an initial keyword probability distribution, and M is an initialization. Transfer matrix. The method for automatically labeling according to claim 4, wherein the step of pre-processing the word to be tagged to obtain the pre-processed text comprises: Crawl the text to be tagged and store it in the MongoDB database. An automatic labeling device, comprising: a text preprocessing unit, configured to perform word segmentation on the tagged text to obtain a preprocessed text; a keyword set obtaining unit, configured to input the preprocessed text into a reverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text; The final probability distribution obtaining unit is configured to obtain an initial transition matrix according to the keyword set of the to-be-labeled text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the final probability distribution of the keyword is obtained. ; The marking unit is configured to obtain a corresponding row of the probability maximum value in the final probability distribution of the keyword, obtain a keyword corresponding to the corresponding row of the maximum probability, and set the keyword as a label of the to-be-labeled text. The apparatus for automatically labeling according to claim 6, wherein the text preprocessing unit comprises: a word segmentation unit for segmenting the text to be tagged and obtaining the text after the word segmentation; a weighting unit for setting a weighting value for the participle included in the text after the word segmentation; a statistical unit for deleting the stop words in the text after the word segmentation, and counting the word frequency of each word segment to obtain the first triplet; a similarity obtaining unit, configured to obtain a similarity between words in the first triplet that is greater than a word frequency threshold of the preset word frequency threshold; The deleted word unit is configured to: if the word similarity between the word segments is greater than the preset word similarity threshold, retain any one of the word segments, obtain the second triplet, and use the second triplet as the preprocessed text. The automatic labeling apparatus according to claim 7, wherein the word inverse frequency TF-IDF algorithm model is:

Wherein, the TF partial molecule n _i,j represents the number of occurrences of the word t _i in the text j, the denominator represents all the word frequency words in the text j, the IWF part of the molecule represents the sum of the frequency of all words in the corpus, and nt _i represents the word t The total frequency of _i appearing in the corpus; The keyword set obtaining unit includes: a first processing unit, configured to generate a corpus term statistical result set; a second processing unit, configured to acquire preprocessed text; The keyword set calculation unit is configured to input the pre-processing text and the corpus term statistical result set into the word inverse frequency TF-IDF algorithm model to obtain a keyword set of the tagged text. The apparatus for automatically labeling according to claim 6, wherein the initialization transfer matrix is obtained according to the keyword set of the to-be-labeled text, and the initialization transfer matrix is an n-dimensional square matrix, and an n-dimensional square matrix The dimension is equal to the total number of keywords in the keyword set; the initial keyword probability distribution is an n-dimensional column vector whose row value is 1/n; wherein n is equal to the total number of keywords in the keyword set Positive integer The initialization transition matrix and the initial keyword probability distribution are recorded as V _m =MV _m-1 through multiple iterative multiplication operations, where m is a positive integer, V ₀ is an initial keyword probability distribution, and M is an initialization. Transfer matrix. The apparatus for automatically labeling according to claim 9, further comprising: Crawl unit for crawling the text to be tagged and storing it in the MongoDB database. That is, the original data is crawled from the Internet, and the text to be tagged is stored and stored in the MongoDB database. By crawling the data, you can set a filter condition that crawls the text of the unset label for labeling. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory and operative on the processor, wherein the processor, when executing the computer program, implements the following steps: Performing word segmentation on the tagged text to obtain preprocessed text; Entering the pre-processed text into the inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text; The initial transfer matrix is obtained according to the keyword set of the tagged text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the final probability distribution of the keyword is obtained; Obtain a corresponding row of the maximum value of the probability in the final probability distribution of the keyword, obtain a keyword corresponding to the corresponding row of the maximum value of the probability, and set the keyword as a tag of the text to be tagged. The computer device according to claim 11, wherein the pre-processing of the word-to-be-labeled text to obtain the pre-processed text comprises: The wording of the tagged text is used to obtain the word after the word segmentation; Setting a weight value for the participles included in the text after the word segmentation; Delete the stop words in the text after the word segmentation, and count the word frequency of each word segment to obtain the first triplet; Obtaining a similarity between words in the first triad that is greater than a word frequency threshold of the preset word frequency threshold; If the word similarity between the word segments is greater than the preset word similarity threshold, any one of the word segments is retained, the second triad is obtained, and the second triad is used as the pre-processed text. The computer apparatus according to claim 12, wherein said word inverse frequency TF-IDF algorithm model is:

Wherein, the TF partial molecule n _i,j represents the number of occurrences of the word t _i in the text j, the denominator represents all the word frequency words in the text j, the IWF part of the molecule represents the sum of the frequency of all words in the corpus, and nt _i represents the word t The total frequency of _i appearing in the corpus; And inputting the pre-processed text into the inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text, including: Generating a corpus term statistical result set; Obtain the preprocessed text; The pre-processed text and the corpus term statistical result set are input into the word inverse frequency TF-IDF algorithm model to obtain a keyword set of the tagged text. The computer device according to claim 11, wherein in the initializing transfer matrix according to the keyword set of the to-be-labeled text, the initial transfer matrix is an n-dimensional square matrix, and the dimension of the n-dimensional square matrix is The initial keyword probability distribution is an n-dimensional column vector whose row value is 1/n; wherein n is a positive integer equal to the total number of keywords in the keyword set. ; The initialization transition matrix and the initial keyword probability distribution are recorded as V _m =MV _m-1 through multiple iterative multiplication operations, where m is a positive integer, V ₀ is an initial keyword probability distribution, and M is an initialization. Transfer matrix. The computer device according to claim 11, wherein the pre-processing of the word-to-be-labeled text to obtain a pre-processed text comprises: Crawl the text to be tagged and store it in the MongoDB database. A storage medium, characterized in that the storage medium stores a computer program, the computer program comprising program instructions that, when executed by a processor, cause the processor to perform the following operations: Performing word segmentation on the tagged text to obtain preprocessed text; Entering the pre-processed text into the inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text; The initial transfer matrix is obtained according to the keyword set of the tagged text, and the initial transition matrix and the initial keyword probability distribution are subjected to multiple iterative multiplication operations until convergence, and the final probability distribution of the keyword is obtained; Obtain a corresponding row of the maximum value of the probability in the final probability distribution of the keyword, obtain a keyword corresponding to the corresponding row of the maximum value of the probability, and set the keyword as a tag of the text to be tagged. The storage medium according to claim 16, wherein the pre-processing of the word-to-be-labeled text to obtain the pre-processed text comprises: The wording of the tagged text is used to obtain the word after the word segmentation; Setting a weight value for the participles included in the text after the word segmentation; Delete the stop words in the text after the word segmentation, and count the word frequency of each word segment to obtain the first triplet; Obtaining a similarity between words in the first triad that is greater than a word frequency threshold of the preset word frequency threshold; If the word similarity between the word segments is greater than the preset word similarity threshold, any one of the word segments is retained, the second triad is obtained, and the second triad is used as the pre-processed text. The storage medium according to claim 17, wherein said word inverse frequency TF-IDF algorithm model is:

Wherein, the TF partial molecule n _i,j represents the number of occurrences of the word t _i in the text j, the denominator represents all the word frequency words in the text j, the IWF part of the molecule represents the sum of the frequency of all words in the corpus, and nt _i represents the word t The total frequency of _i appearing in the corpus; And inputting the pre-processed text into the inverse frequency TF-IDF algorithm model to obtain a keyword set of the to-be-labeled text, including: Generating a corpus term statistical result set; Obtain the preprocessed text; The pre-processed text and the corpus term statistical result set are input into the word inverse frequency TF-IDF algorithm model to obtain a keyword set of the tagged text. The storage medium according to claim 16, wherein the initialization transfer matrix is obtained according to the keyword set of the to-be-labeled text, wherein the initialization transfer matrix is an n-dimensional square matrix, and the dimension of the n-dimensional square matrix is The initial keyword probability distribution is an n-dimensional column vector whose row value is 1/n; wherein n is a positive integer equal to the total number of keywords in the keyword set. ; The initialization transition matrix and the initial keyword probability distribution are recorded as V _m =MV _m-1 through multiple iterative multiplication operations, where m is a positive integer, V ₀ is an initial keyword probability distribution, and M is an initialization. Transfer matrix. The storage medium according to claim 16, wherein the step of preprocessing the word to be tagged to obtain a preprocessed text comprises: Crawl the text to be tagged and store it in the MongoDB database.

Images