CN111538846A - Third-party library recommendation method based on hybrid collaborative filtering - Google Patents

Abstract

The invention discloses a third-party library recommendation method based on mixed collaborative filtering, which comprises the following steps: obtaining feature training data sets of the application and the third-party library according to the published application and third-party library data; training by using an unsupervised learning method to obtain a theme model; extracting entities from the application and third-party database data to construct a knowledge graph and vectorizing the knowledge graph; inputting application data to be recommended into a topic model to generate an application neighbor list; obtaining a content-based scoring list of the application to be recommended by utilizing the calling information of the application to the third-party library; inputting the application to be recommended and the third-party library list to be recommended into a knowledge graph to obtain an entity vector list; calculating the similarity of the entity vectors to obtain a knowledge graph-based rating list of the applications to be recommended; and after fusion, sorting to obtain a recommendation list based on mixed recommendation. The hybrid recommendation method provided by the invention avoids the defects of a single recommendation method, effectively solves the problems of data sparseness and cold start, and improves the recommendation accuracy.

Description

Third-party library recommendation method based on hybrid collaborative filtering

technical field

The invention relates to computer technology, in particular to a third-party library recommendation method based on hybrid collaborative filtering.

Background technique

With the rapid development of Internet technology, software and applications have become an indispensable part of people's daily life. Third-party libraries play a vital role in the development of applications. They can shorten development time, increase development efficiency, and improve development quality. However, with the rapid proliferation of third-party libraries, choosing the right third-party library is a time-consuming and laborious task even for experienced developers. How to choose a library that meets the needs from a large number of complex third-party libraries has become a difficult problem in application development. Recommendation systems can effectively improve the problem of information overload and are widely used in many scenarios in the network. The current recommendation methods of third-party libraries mainly include content-based recommendation methods and collaborative filtering-based recommendation methods. These methods have the following problems:

1. The content-based recommendation method is to discover the similarity between the third-party library and the application according to the content, and then recommend similar third-party libraries to the application based on the content of the application. This method can effectively deal with the cold start problem and data sparse problem, and has good interpretability, but this method cannot recommend third-party libraries with potential calling relationships, and the feature selection, extraction and matching of third-party libraries are difficult .

2. The recommendation method based on collaborative filtering is to first find some applications similar to the application to be recommended, and then recommend to the application the third-party libraries called by these applications but not called by the application to be recommended. Since this method of clustering applications needs to use the historical third-party library call information of the applications, there are data sparse and cold-start problems.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a third-party library recommendation method based on hybrid collaborative filtering, aiming at the defects in the prior art.

The technical solution adopted by the present invention to solve the technical problem is: a third-party library recommendation method based on hybrid collaborative filtering, comprising the following steps:

1) Obtain the published application and third-party library data from the application and third-party library server; the application and third-party library data include text description information of the application and the third-party library, the calling information of the application to the third-party library, and the application and the structured semantic information of the third-party library itself;

2) Preprocess and quantify the text description information of the application and the third-party library in step 1) by the natural language processing method to obtain the content-based feature training data set of the application and the third-party library;

3) using the content-based feature training data set of the application and the third-party library in step 2) as the corpus, and using the unsupervised learning method to train to obtain the topic model of the application and the third-party library;

4) construct the call interaction matrix of the application to the third-party library according to the call information of the application to the third-party library in step 1) according to the calling relationship of each application to the third-party library;

5) extract entities and the relationship between entities from the application obtained in step 1) and the structured semantic information of the third-party library itself, and save it into a graph database to form a knowledge map;

6) Using the knowledge graph representation learning method to map the knowledge graph obtained in step 5) to a low-dimensional space to obtain a vectorized representation of each entity and relationship;

7) Obtain the text description information of the application to be recommended and its own structured information;

8) Input the text description information of the application to be recommended obtained in step 7) into the topic model obtained in step 3), and obtain the content-based neighbor list of the application to be recommended by comparing the similarity of the text description information;

9) According to the call interaction information of the application to be recommended based on the content-based neighbor list to the third-party library to be recommended, the collaborative filtering method is used to perform a weighted sum and average value according to the similarity, and obtain the content-based score of the application to be recommended for the third-party library to be recommended. list;

10) Input the structured information of the application to be recommended and the list of third-party libraries to be recommended obtained in step 7) into the knowledge map, and find the matching entity through the entity recognition method, and then the corresponding knowledge map representation learning method can be obtained. list of entity vectors of ;

11) Calculate the similarity between the entity vectorized representation of the application to be recommended and the entity vectorized representation of the third-party library to be recommended, as a rating list based on the knowledge graph of the application to be recommended for the third-party library to be recommended;

12) Integrate the content-based and collaborative filtering scoring list of the application to be recommended obtained in step 9) with the third-party library to be recommended based on the knowledge graph scoring list obtained in step 11) to obtain the third-party library to be recommended. A list of mixed recommendation-based ratings of the application to be recommended for the third-party library to be recommended;

13) Arrange the scores of the applications to be recommended with respect to all the third-party libraries to be recommended in descending order, and obtain a Top-N third-party library recommendation list of the applications to be recommended based on mixed recommendations.

According to the above solution, the preprocessing performed by the natural language processing method in step 2) includes word segmentation, removal of stop words, punctuation marks and low-frequency words, and stemming.

According to the above scheme, the vectorization in step 2) adopts text vectorization methods including TF-IDF, Doc2Bow, and LDA.

According to the above scheme, in the described step 4), the call interaction matrix of the application to the third-party library is constructed, and the details are as follows:

Classify the application information obtained in step 1) to the third-party library by application ID, and generate a preference vector for each application. The specific preference relationship is defined as: for application a and third-party library l, if a calls l, Then y _al = 1, otherwise y _al = 1, the preference vectors of M applications for N third-party libraries constitute the interaction matrix of the calling relationship between applications and third-party libraries

According to the above scheme, in the step 8), the content-based neighbor list of the application to be recommended is obtained, as follows:

8.1) After the text description information of the application to be recommended is preprocessed by the natural language method, the text vectorization method is used to convert it into a vector;

8.2) Input the obtained vector into the topic model obtained in step 3), and through the similarity comparison, take the top k applications with the highest similarity as the content-based neighbor list of the application to be recommended N(a)=*a ₁ , a ₂ ,..., _ak +, wherein, the similarity sim(a, a _i ) is calculated by the following methods: cosine similarity method, Euclidean distance method, and Pearson correlation coefficient method.

According to the above scheme, in the step 9), the collaborative filtering method is used to perform weighted summation and take the average value according to the similarity, to obtain the content-based score list of the application to be recommended for the third-party library to be recommended, as follows:

计算得到待推荐应用a对于待推荐第三方库l的基于内容和协同过滤的评分S₁(a,l)，对待推荐第三方库列表中的所有第三方库按该上述公式计算评分，得到待推荐应用基于内容和协同过滤的第三方库评分列表。According to the call information of the content-based neighbor list N(a) of the application to be recommended for the third-party library 1 to be recommended, the collaborative filtering method is used according to the formula:

Calculate the content and collaborative filtering-based score S ₁ (a, 1) of the application a to be recommended for the third-party library 1 to be recommended, and calculate the scores for all third-party libraries in the list of recommended third-party libraries according to the above formula, and obtain the score to be recommended. Recommended application content-based and collaborative filtering third-party library scoring lists.

According to the above solution, in the step 11), the similarity between the entity vectorized representation of the application to be recommended and the entity vectorized representation of the third-party library to be recommended is calculated as the knowledge graph based on the knowledge graph of the application to be recommended for the third-party library to be recommended. Score S ₂ (a, l), wherein the similarity is calculated by the following methods: cosine similarity method, Euclidean distance method, and Pearson correlation coefficient method.

According to the above scheme, weighted fusion or feature fusion is adopted in the fusion method in step 12).

The beneficial effects that the present invention produces are:

1. The present invention designs a third-party library hybrid recommendation method to avoid the defects of a single recommendation method, while effectively solving the problems of data sparse and cold start, and improving the accuracy of the recommendation and other indicators;

2. Using the hybrid recommendation algorithm, the information of different dimensions of the application and the third-party library is fully utilized, and the third-party library with potential calling relationship can be mined, so as to avoid the recommendation results being limited to a narrow range;

3. The knowledge graph is introduced. For the recommendation results, the topological structure relationship of the knowledge graph can be used to explain.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

Fig. 1 is the method flow chart of the embodiment of the present invention;

FIG. 2 is a schematic diagram of the comparison of experimental results in an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Some current application recommendation methods have problems such as data sparseness and cold start, and cannot be recommended for third-party libraries that are not directly related, which causes a long tail effect to a certain extent. The present invention comprehensively considers the existing content-based and collaborative filtering-based recommendation methods, and designs a third-party library that is based on the knowledge map-based method, supplemented by the content-based and collaborative filtering method, and combines the three recommendation methods. recommended method.

As shown in Figure 1, the present invention provides a third-party library recommendation method based on hybrid collaborative filtering, comprising the following steps:

Step 1: Obtain published application and third-party library data from the application and third-party library servers. The application and third-party library data are divided into three parts: unstructured text description of the application and third-party library Information, which is used to describe the specific functions and application scenarios of the application and the third-party library, as well as introductions such as usage instructions; the application's call information to the third-party library, which records the list of third-party libraries called by the published application; the application and the third-party library itself structured semantic information, such as application type, developer, price, installation package volume, type, label, version and other information of third-party libraries;

Step 2: The unstructured text description information of the application and third-party library obtained in Step 1 is used for data cleaning and preprocessing with natural language methods. The processing flow includes word segmentation, removal of stop words, punctuation marks and low-frequency words, and stemming. Wait. Use TF-IDF, Doc2Bow, LDA and other text vectorization methods to vectorize the preprocessed data to obtain a "feature training data set for applications and third-party libraries";

Step 3: Use the unsupervised learning method to train the "feature training data set of the application and the third-party library" to obtain the topic model of the application and the third-party library for subsequent document similarity comparison;

Step 4: Classify the call information of the application to the third-party library obtained in step 1 according to the application ID, and generate a preference vector for each application. The specific preference relationship is defined as: for application a and third-party library l, if a calls l, then y _al = 1, otherwise y _al = 1, the preference vectors of M applications to N third-party libraries constitute the interaction matrix of the calling relationship between applications and third-party libraries

Step 5: Use the application obtained in Step 1 and the structured semantic information of the third-party library to construct knowledge graphs in related fields. The specific construction process includes: entity recognition, entity and relationship triple extraction, entity disambiguation and knowledge Completion, etc. For the triples obtained by extraction, the graph database is used for storage, and the available knowledge graph is obtained;

Step 6: Use the knowledge graph to represent the learning method, such as: TransE, TransH, TransR, etc., map the knowledge graph to a low-dimensional vector space, and each entity and relationship in the knowledge graph is represented as a vector;

Step 7: Obtain the unstructured text description information of the application to be recommended and its own structured information from the application and the third-party library server;

Step 8: After the unstructured text description information of the application to be recommended is preprocessed by natural language method, it is converted into a vector using the text vectorization method, and then input into the topic model obtained in step 3, and the first k is selected by similarity comparison. The applications with the highest similarity, as the content-based neighbor list to be recommended N(a)=*a ₁ , a ₂ ,..., a _k +, where the similarity sim(a, a _i ) can be measured in multiple ways You can choose, such as cosine similarity, Euclidean distance, Pearson correlation coefficient, etc.;

计算得到待推荐应用a对于待推荐第三方库l的基于内容和协同过滤的评分S₁(a,l)，对待推荐第三方库列表中的所有第三方库按该方式计算评分，得到待推荐应用基于内容和协同过滤的评分列表；Step 9: According to the calling information of the content-based neighbor list N(a) of the application to be recommended for the third-party library 1 to be recommended, use the collaborative filtering method according to the formula:

Calculate the content and collaborative filtering-based score S ₁ (a, l) of the application a to be recommended for the third-party library l to be recommended, and calculate the scores in this way for all third-party libraries in the list of third-party libraries to be recommended, and obtain the to-be-recommended library Apply content-based and collaborative filtering scoring lists;

Step 10: Input the structured information of the application to be recommended and the list of third-party libraries to be recommended obtained in step 7 into the knowledge graph, find the matching entity through the entity recognition method, and then obtain the corresponding generated by the knowledge graph representation learning method. list of entity vectors of ;

Step 11: Calculate the similarity between the entity vectorized representation of the application to be recommended and the entity vectorized representation of the third-party library to be recommended, as the score S ₂ (a, l) of the application to be recommended for the third-party library to be recommended based on the knowledge graph ), similarly, the similarity here has a variety of measurement methods to choose from, and those skilled in the art can also use other existing knowledge graph-related recommendation methods to calculate the score;

Step 12: Integrate the content-based and collaborative filtering scoring list of the application to be recommended obtained in step 9 for the third-party library to be recommended and the knowledge map-based scoring list of the application to be recommended obtained in step 11 to the third-party library to be recommended, to obtain the score to be recommended. Recommended application For the score list based on hybrid recommendation of the third-party library to be recommended, the fusion method can consider weighted fusion, feature fusion, etc.;

Step 13: Arrange the scores of the applications to be recommended for all the third-party libraries to be recommended in descending order, and generate a Top-N recommendation list based on mixed recommendation.

A note about the effect:

In this embodiment, in steps 2 and 8, the LDA text vectorization method is used to perform vectorization processing on the preprocessed data, and in step 12, the weighted fusion method is used to evaluate the application to be recommended obtained in step 9 to the third-party library to be recommended. The rating list based on content and collaborative filtering and the application to be recommended obtained in step 11 are fused with the rating list based on the knowledge graph of the third-party library to be recommended, and the fusion coefficient is set to 0.5.

The method of the present embodiment is compared with the existing method, and the comparison result is shown in Figure 2;

Comparative experimental settings: The experimental dataset contains 5274 applications and 471 called third-party library information; the comparison methods include collaborative filtering (CF), LDA topic model (LDA), neural collaborative filtering (NCF), knowledge graph-based methods (RippleNet), and a hybrid method based on collaborative filtering and topic models (AppLibRec); evaluation metrics include: precision (Precision@N), recall (Recall@N), F1 value (F1@N), mean precision ( MAP@N), and normalized impairment cumulative gain (NDCG@N).

The experimental results show that the method of this embodiment (TM-MKR) performs better on all evaluation indicators than all existing comparison methods.

It should be understood that, for those skilled in the art, improvements or changes can be made according to the above description, and all these improvements and changes should fall within the protection scope of the appended claims of the present invention.

Claims (8)

1. a third-party library recommendation method based on hybrid collaborative filtering, is characterized in that, comprises the following steps: 1) Obtain the published application and third-party library data from the application and third-party library server; the application and third-party library data include text description information of the application and the third-party library, the calling information of the application to the third-party library, and the application and the structured semantic information of the third-party library itself; 2) Preprocess and quantify the text description information of the application and the third-party library in step 1) by the natural language processing method to obtain the content-based feature training data set of the application and the third-party library; 3) using the content-based feature training data set of the application and the third-party library in step 2) as the corpus, and using the unsupervised learning method to train to obtain the topic model of the application and the third-party library; 4) construct the call interaction matrix of the application to the third-party library according to the call information of the application to the third-party library in step 1) according to the calling relationship of each application to the third-party library; 5) extract entities and the relationship between entities from the application obtained in step 1) and the structured semantic information of the third-party library itself, and save it into a graph database to form a knowledge map; 6) Using the knowledge graph representation learning method to map the knowledge graph obtained in step 5) to a low-dimensional space to obtain a vectorized representation of each entity and relationship; 7) Obtain the text description information of the application to be recommended and its own structured information; 8) Input the text description information of the application to be recommended obtained in step 7) into the topic model obtained in step 3), and obtain the content-based neighbor list of the application to be recommended by comparing the similarity of the text description information; 9) According to the content-based neighbor list of the application to be recommended, the call interaction information of the third-party library to be recommended is used to perform a weighted sum and average value according to the similarity using the collaborative filtering method to obtain the content-based and collaborative filtering of the application to be recommended. A filtered list of ratings; 10) Input the structured information of the application to be recommended and the list of third-party libraries to be recommended obtained in step 7) into the knowledge map, and find the matching entity through the entity recognition method, and then the corresponding knowledge map representation learning method can be obtained. list of entity vectors of ; 11) Calculate the similarity between the entity vectorized representation of the application to be recommended and the entity vectorized representation of the third-party library to be recommended, as a rating list based on the knowledge graph of the application to be recommended for the third-party library to be recommended; 12) Integrate the content-based and collaborative filtering scoring list of the application to be recommended obtained in step 9) with the third-party library to be recommended based on the knowledge graph scoring list obtained in step 11) to obtain the third-party library to be recommended. A list of mixed recommendation-based ratings of the application to be recommended for the third-party library to be recommended; 13) Arrange the scores of the applications to be recommended with respect to all the third-party libraries to be recommended in descending order, and obtain a Top-N third-party library recommendation list of the applications to be recommended based on mixed recommendations. 2. the third-party library recommendation method based on hybrid collaborative filtering according to claim 1, is characterized in that, in described step 2), carry out preprocessing by natural language processing method and comprise word segmentation, remove stop word, punctuation mark and low frequency Vocabulary and stemming. 3. The third-party library recommendation method based on hybrid collaborative filtering according to claim 1, wherein the vectorization in the step 2) adopts a text vectorization method including TF-IDF, Doc2Bow, and LDA. 4. the third-party library recommendation method based on hybrid collaborative filtering according to claim 1, is characterized in that, in described step 4), construct application to the call interaction matrix of third-party library, is specifically as follows: Classify the application information obtained in step 1) to the third-party library by application ID, and generate a preference vector for each application. The specific preference relationship is defined as: for application a and third-party library l, if a calls l, Then y _al = 1, otherwise y _al = 1, the preference vectors of M applications for N third-party libraries constitute the interaction matrix of the calling relationship between applications and third-party libraries

5. The method for recommending a third-party library based on hybrid collaborative filtering according to claim 1, wherein in the step 8), a content-based neighbor list of the application to be recommended is obtained, specifically as follows: 8.1) After the text description information of the application to be recommended is preprocessed by the natural language method, the text vectorization method is used to convert it into a vector; 8.2) Input the obtained vector into the topic model obtained in step 3), through similarity comparison, take the top k applications with the highest similarity as the content-based neighbor list of the application to be recommended N(a)={a ₁ , a ₂ ,..., _ak }, where the similarity sim(a, a _i ) is calculated by the following methods: cosine similarity method, Euclidean distance method or Pearson correlation coefficient method. 6. The third-party library recommendation method based on hybrid collaborative filtering according to claim 5, is characterized in that, in described step 9), utilize collaborative filtering method to carry out weighted sum and take average value by similarity, obtain application to be recommended for application to be A list of content-based ratings for recommended third-party libraries, as follows: According to the call information of the content-based neighbor list N(a) of the application to be recommended for the third-party library 1 to be recommended, the collaborative filtering method is used according to the formula:

Calculate the content and collaborative filtering-based score S ₁ (a, 1) of the application a to be recommended for the third-party library 1 to be recommended, and calculate the scores for all third-party libraries in the list of recommended third-party libraries according to the above formula, and obtain the score to be recommended. Recommended application content-based and collaborative filtering third-party library scoring lists. 7. the third-party library recommendation method based on hybrid collaborative filtering according to claim 1, is characterized in that, in described step 11), calculate the entity vectorization representation of application to be recommended and the entity vectorization representation of the third-party library to be recommended The similarity between the applications to be recommended is the knowledge graph-based score S ₂ (a, l) for the third-party library to be recommended, where the similarity is calculated by the following methods: cosine similarity method, Euclidean distance method or Pearson distance method Correlation coefficient method. 8 . The third-party library recommendation method based on hybrid collaborative filtering according to claim 1 , wherein the method used for fusion in the step 12) is weighted fusion or feature fusion. 9 .

Images