CN102567409A - Method and device for providing retrieval associated word - Google Patents

Abstract

The present invention relates to retrieval technology, and discloses a method and device for providing retrieval associated words, which are used to improve the accuracy of retrieval associated words and the retrieval efficiency of search engines; The matching method is used to obtain search related words, but based on the text data contained in the web page where the search keyword is located, the target word segmentation is extracted, and then the correlation degree between each target word segmentation and the search keyword is calculated, and the correlation degree reaches the set threshold The target participle of the value is presented as a search associated word. In this way, according to the information content contained in the web page where the search keyword is located, it can be obtained that there is a logical relationship with the search keyword, and there is or does not exist a fuzzy matching relationship. Other search related words, thereby improving the information accuracy of the search related words, avoiding the omission of some search related words, and effectively improving the search efficiency of the search engine.

Description

A method and device for providing retrieval associated words

technical field

The invention relates to retrieval technology, in particular to a method and device for providing retrieval associated words.

Background technique

With the rapid increase of the amount of information and data on the Internet, in order to serve users better, it is necessary for search engines to adopt auxiliary means to improve search efficiency, and intelligent recommendation of search related words is one of the various auxiliary means. As shown in Figure 1, the so-called intelligent recommendation of search related words means that when a user enters a search keyword, for example, ginseng, a series of words that may be associated with the search keyword entered by the user will be displayed on the search page. It is called retrieval associated words, for example, as shown in Figure 1, ginseng fruit, ginseng Jianpi pill, ginseng doll..., click on these retrieval associated words, users can conduct further searches.

At present, there are many search engines that support intelligently recommending retrieval associated words, and the methods of recommending retrieval associated words used by these search engines usually include the following:

However, different search engines have different ways of accomplishing this function. General search engines generally use simple and efficient recommendation calculation methods in terms of recommendation due to their extensive search population and wide range of fields. The common methods are:

1. System keyword fuzzy matching.

That is, after the user enters the search keyword, fuzzy matching is performed according to the content recorded in the preset system keyword library, and the search keyword with successful fuzzy matching is recommended to the user as a search related word.

2. User input keyword fuzzy matching.

That is to save the search keywords entered by the user in the user input keyword library and accumulate the number of entries. When the user enters the search keywords, fuzzy matching is performed according to the content recorded in the user input keyword library, and the fuzzy matching is successfully The search keywords are sorted according to the accumulated times, and are recommended to users as search related words.

3. Refer to keywords input by other users for recommendation.

The system records the entire process of any user inputting a search keyword and saves it in the user input keyword library. After the user enters the search keyword, it searches for a matching search keyword according to the user input keyword library, and saves the previously entered search keyword Other search keywords input by the user are recommended to the user as search keywords.

However, when the existing search engines provide search related words, they do not take into account the industry characteristics of the search related words. The recommended search related words are generally common keyword sequences, and the existing search engines usually use fuzzy matching to recommend search related words. , so that many retrieval related words that do not have a fuzzy matching relationship with the retrieval keywords but have a certain logical relationship will not be recommended. For example, when the search keyword entered by the user is "ginseng", the search related words "saponin" and "astragalus" which have a certain logical relationship with "ginseng" will not be recommended, thus affecting the search to a certain extent. The accuracy of associated words reduces the retrieval efficiency of search engines.

Contents of the invention

The embodiment of the invention discloses a method and device for providing retrieval related words, which are used to improve the accuracy of retrieval related words and improve the retrieval efficiency of a search engine.

The specific technical scheme that the embodiment of the present invention provides is as follows:

A method for providing retrieval associated words, comprising:

Acquiring webpages containing the search keywords according to the search keywords input by the user;

Extracting target word segmentation based on the text data included in the web page;

Respectively based on the density of each target participle in each web page, calculate the degree of association between each target participle and the retrieval keyword;

The target participle whose relevance degree reaches the set threshold is presented to the user as a search associated word.

A device for providing retrieval associated words, comprising:

An acquisition unit, configured to acquire a web page containing the retrieval keyword according to the retrieval keyword input by the user;

An extraction unit, configured to extract target word segmentation based on the text data included in the web page;

A computing unit, configured to calculate the degree of association between each target word and the retrieval keyword based on the density of each target word in each web page;

The presenting unit is configured to present the target word segment whose relevance degree reaches the set threshold value to the user as a retrieval related word.

In the embodiment of the present invention, for the search keyword input by the user, instead of using fuzzy matching to obtain the search associated words, the target word is extracted based on the text data contained in the webpage where the search keyword is located, and then the target word and word are calculated. Retrieve the degree of relevance between keywords, and present the target word segment whose degree of relevance reaches the set threshold value as a search related word. In this way, according to the information content contained in the web page where the search keyword is located, you can obtain the corresponding search keyword. Words are logically related, and there are other search related words with fuzzy matching relationship, so as to improve the information accuracy of the search related words, avoid the omission of some search related words, and effectively improve the search efficiency of the search engine.

Description of drawings

FIG. 1 is a schematic diagram of intelligently recommended retrieval associated words in the prior art;

Fig. 2 is a functional structural diagram of a retrieval device in an embodiment of the present invention;

Fig. 3 is a flow chart of establishing a recommended lexicon in an embodiment of the present invention;

Fig. 4 is a flow chart of providing retrieval associated words to users in an embodiment of the present invention.

Detailed ways

In the embodiment of the present invention, in order to improve the accuracy of retrieval related words, improve the retrieval efficiency of search engine, in the embodiment of the present invention, no longer adopt the mode of fuzzy matching to obtain retrieval related words, but according to the web page that retrieval keyword appears The text data included to obtain the corresponding search associated words, specifically: obtain the web pages containing the search keywords according to the search keywords input by the user, extract the target word segmentation based on the text data contained in the obtained web pages, and then based on each The density of the target participle in each web page calculates the correlation degree between each target participle and the search keyword, and finally, the target participle whose correlation degree reaches the set threshold value is presented to the user as a search related word.

Preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

In the embodiment of the present invention, the method for providing search related words is divided into two stages. The first stage is to create and update the recommended thesaurus of search related words, and the second stage is to present the search related word recommendation list according to the search keyword input by the user. ,

Referring to shown in Figure 2, in the embodiment of the present invention, the creation process of the recommended thesaurus is as follows:

Step 200: Determine whether there are unread search keywords in the search keyword list, and if so, go to step 210; otherwise, end the current process.

In this embodiment, a list of search keywords is set according to commonly used search keywords, and a corresponding recommended thesaurus is established based on the list of search keywords. The search keywords recorded in the list may be pre-input by the management personnel, or may be learned and acquired according to the search keywords input by the user during the use process.

Step 210: Read a retrieval keyword from the retrieval keyword list.

Step 220: Obtain the corresponding web page according to the retrieved keyword.

In this embodiment, web crawling tools such as WebSpider, PClawer, hpricot, etc. can be used. The webpage crawling tool obtains a series of webpage collections related to the retrieval keywords according to the retrieval keywords, and stores the crawled webpage collections in the memory for subsequent use in calculating the relevance degree.

On the other hand, when acquiring web pages, the industry attribute of the website can be restricted, and after the specified industry attribute of the website is determined, the web page containing the search keyword can be obtained from the website with the industry attribute of the website. For example, if a user needs to retrieve relevant information within the scope of the Chinese medicine industry, the search engine can be set to only obtain corresponding webpages within the Chinese medicine industry search engine or websites related to the Chinese medicine industry.

Step 230: Extract the text data contained in each webpage, and perform word segmentation to obtain target word segmentation.

Preferably, a minimum segmentation method can be used for word segmentation. The so-called minimum segmentation method is to minimize the number of target word segments obtained based on a whole sentence segmentation.

Step 240: Using a clustering algorithm to calculate the density of each target word segment in each web page.

The purpose of using a clustering algorithm is to calculate the frequency (ie density) of each target participle. Preferably, the target participle whose density does not reach the set threshold can be deleted, thereby reducing the complexity of the subsequent processing flow. For example, according to the text data contained in the web page obtained from the search keyword "ginseng", after word segmentation, the multiple target word segments obtained are: astragalus, Codonopsis pilosula, ginseng treatment, ginseng curative effect, ginseng doll... .., then, after clustering and screening, the target word segmentation whose density reaches the set threshold value is curative effect of astragalus, codonopsis, and ginseng.

Before clustering each participle, it is preferable to delete some commonly used modal particles, conjunctions, prepositions, etc., which are not helpful for the recommendation of retrieval associated words, for example, "do", "one", " Very"...in order to extract more precise target word segmentation.

Step 250: Determine the weight of each acquired web page.

In the embodiment of the present invention, when determining the weight of any web page, one or any combination of the following parameters can be referred to:

The website authority index is used to represent the authority of the website to which the web page belongs; for example, the website authority index of a well-known large website is higher than that of an ordinary website. The higher the website authority index, the higher the reference value of the webpages contained in the website.

The website professional index is used to represent the industry attribute of the website to which the web page belongs. For example, if the search is limited to the Chinese medicine industry, the website professional index of the special Chinese medicine website is higher than that of the general medical website. index. When limiting the industry attributes of the website, the higher the professional index of the website, the higher the reference value of the webpages included.

User click rate is used to represent the number of user views of a web page, and the greater the number of user views of a web page, the higher its reference value.

The number of occurrences of the target word is used to represent the cumulative number of occurrences of the target word in the web page, and the more the number of the target word is contained in the web page, the higher the reference value is.

 …

In practical applications, administrators can set various parameters that need to be referred to when determining the weight of web pages according to the specific application environment. The above-mentioned parameters are just examples and are not limited thereto.

Step 260: Based on the density of each target word segment and the weight of each webpage, respectively calculate the degree of association between each target word segment and the obtained retrieval keyword.

In the practical application of the present invention, when calculating the degree of association between any target word segmentation and retrieval key word, can adopt following formula:

Relevance = X1*w1+X2*w2+......Xm*wn Formula 1

Among them, X1, X2, . . . Xn are the densities of any target word segmentation in each web page, and w1, w2, . . . wn are the weights of each web page.

In practical applications, the density of the target word segmentation used in calculating the degree of relevance in each web page and the weight of each web page need to be updated regularly.

Step 270: Save the target participle whose relevance degree reaches the set threshold, corresponding to the search keyword input by the user, as a search related word.

In this way, by adopting steps 200-270, the recommended vocabulary can be initially established for subsequent use.

Based on the above-mentioned embodiment, referring to Fig. 3, in the embodiment of the present invention, based on the established recommended thesaurus, for the search keyword input by the user, the detailed process of providing corresponding search related words is as follows:

Step 300: Obtain the search keyword input by the user.

Step 310: Does the retrieved keyword already exist in the recommended thesaurus? If yes, go to step 320 ; otherwise, go to step 330 .

Step 320: Acquire search-related words corresponding to the search keywords from the recommended thesaurus, and present them to the user.

In this embodiment, when presenting the retrieved associated words to the user, preferably, the retrieved associated words are presented in descending page order according to the degree of relevance between each retrieved associated word and the retrieved keyword.

Step 330: Obtain the corresponding web page according to the obtained retrieval keyword

Similar to step 220, when acquiring web pages, the industry attribute of the website can be restricted, and after the specified industry attribute of the website is determined, the web page containing the search keyword can be obtained from the website with the industry attribute of the website. For example, if a user needs to retrieve relevant information within the scope of the Chinese medicine industry, the search engine can be set to only obtain corresponding webpages within the Chinese medicine industry search engine or websites related to the Chinese medicine industry.

Step 340: Extract the text data contained in each webpage, and perform word segmentation to obtain target word segmentation.

Step 350: Using a clustering algorithm to calculate the density of each target word segment in each web page.

Preferably, after calculating the density of each target word in each web page, the target word whose density does not reach the set threshold can be deleted, thereby reducing the complexity of subsequent operations.

Step 360: Determine the weight of each web page.

In the same way as step 250, in the embodiment of the present invention, when determining the weight of any web page, one or any combination of the following parameters can be referred to: website authority index, website professional index, user click rate, and occurrence times of target word segmentation. .....

Step 370: Based on the density of each target word segment and the weight of each web page, respectively calculate the degree of association between each target word segment and the retrieval keyword input by the user.

Similar to step 260, in this embodiment, Formula 1 can still be used to calculate the relevance degree of each target word segment.

Step 380: Present the target word segment whose relevance degree reaches the set threshold value to the user as a search related word.

In this embodiment, when presenting each target participle as a retrieval associated word to the user, preferably, the presentation is performed in descending order of the degree of association between each retrieval associated word and the retrieval keyword.

At the same time, it is also necessary to save the search keywords and corresponding search associated words input by the user in the recommended thesaurus.

In practical applications, with the continuous updating of various web pages, the density of the target word segmentation used to calculate the relevance degree and the weight of each web page need to be updated regularly. Correspondingly, the search associated words corresponding to each search keyword are also constantly updated. Therefore, it is also necessary to regularly update the search associated words corresponding to each search keyword.

Based on the above-mentioned embodiment, referring to shown in Fig. 4, in the embodiment of the present invention, the device that is used to provide retrieval related word, is called retrieval device, comprises:

An acquisition unit 40, configured to acquire a web page containing the retrieval keyword according to the retrieval keyword input by the user;

An extracting unit 41, configured to extract target word segmentation based on the text data contained in the web page;

Calculation unit 42, for calculating the degree of association between each target participle and the retrieval keyword based on the density of each target participle in each web page;

The presenting unit 43 is configured to present the target word segment whose relevance degree reaches the set threshold value to the user as a search related word.

As shown in FIG. 4 , the retrieval device is further provided with an update unit 44 for updating the retrieval related words corresponding to each retrieval keyword stored in the recommended thesaurus according to a set period.

To sum up, in the embodiment of the present invention, for the retrieval keyword input by the user, instead of using fuzzy matching to obtain retrieval related words, the target word segmentation is extracted based on the text data contained in the web page where the retrieval keyword is located, and then Calculate the degree of correlation between each target participle and the retrieval keyword, and present the target participle whose correlation degree reaches the set threshold value as a retrieval related word. In this way, according to the information content contained in the web page where the retrieval keyword is located, The clustering algorithm is used to obtain other search related words that have a relationship with the search keyword and have or do not have a fuzzy matching relationship, thereby improving the information accuracy of the search related words, avoiding the omission of some search related words, and effectively improving the search results. engine retrieval efficiency.

Furthermore, in the embodiment of the present invention, the industry attribute of the website as the retrieval object can also be limited, so that the retrieval range can be accurately positioned, and the retrieval efficiency of the search engine is further improved.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and equivalent technologies thereof, the present invention also intends to include these modifications and variations.

Claims (14)

1. A method for providing retrieval associated words, is characterized in that, comprising: Acquiring webpages containing the search keywords according to the search keywords input by the user; Extracting the target participle based on the text data included in the web page; calculating the degree of association between each target participle and the retrieval keyword based on the density of each target participle in each web page; The target participle whose relevance degree reaches the set threshold is presented to the user as a search associated word. 2. The method according to claim 1, wherein, when obtaining the web page containing the retrieval keyword, comprising: Determine the specified website industry attributes; In the website with the website industry attribute, the web page containing the search keyword is obtained. 3. The method according to claim 1, wherein a clustering algorithm is used to calculate the density of each target word segment in each web page. 4. The method according to claim 1, wherein, based on the density of any target participle in each web page, when calculating the degree of association between the target participle and the search keyword, the following formula is adopted: Correlation degree = X1*w1+X2*w2+......Xm*wn Wherein, X1, X2, . . . Xn is the density of any target word segment in each web page, and w1, w2, . . . wn is the weight of each web page. 5. The method according to claim 4, wherein, when determining the weight of any web page, one or any combination of website authority index, website professional index, user click-through rate and target participle occurrence times is referred to. 6. The method according to any one of claims 1-5, wherein the obtained target word segmentation is saved as the retrieval associated word of the retrieval keyword, and when the user inputs the retrieval keyword next time, directly Obtain the corresponding retrieval associated words and present them to the user. 7. The method according to claim 6, characterized in that the search related words corresponding to the search keywords are updated according to a set period. 8. A device for providing retrieval associated words, characterized in that it comprises: An acquisition unit, configured to acquire a web page containing the retrieval keyword according to the retrieval keyword input by the user; An extraction unit, configured to extract target word segmentation based on the text data included in the web page; A computing unit, configured to calculate the degree of association between each target word and the retrieval keyword based on the density of each target word in each web page; The presenting unit is configured to present the target word segment whose relevance degree reaches the set threshold value to the user as a retrieval related word. 9. The device according to claim 8, wherein, when the acquisition unit acquires the web page containing the search keyword, it determines the specified website industry attribute, and in the website with the website industry attribute, Obtain webpages containing the retrieval keywords. 10. The device according to claim 8, wherein the calculation unit uses a clustering algorithm to calculate the density of each target word segment in each web page. 11. The device according to claim 8, wherein the calculation unit calculates the degree of association between the target word and the search keyword based on the density of any target word in each webpage, using The following formula: Correlation degree = X1*w1+X2*w2+......Xm*wn Wherein, X1, X2, . . . Xm is the density of any target word segment in each web page, and w1, w2, . . . wn is the weight of each web page. 12. The device according to claim 11, wherein when the calculation unit determines the weight of any web page, it refers to one or more of the website authority index, website professional index, user click rate and target participle occurrence times. random combination. 13. The device according to any one of claims 8-12, wherein the presenting unit saves the obtained target participle as a retrieval associated word of the retrieval keyword, and waits for the user to input the retrieval keyword next time. When searching a word, directly obtain the corresponding search associated word and present it to the user. 14. The apparatus of claim 13, further comprising: An update unit, configured to update the search related words corresponding to the search keywords according to a set period.

Images