CN106815297A - A kind of academic resources recommendation service system and method - Google Patents

Abstract

Provide an academic resource recommendation service system and method, crawl academic resources on the Internet with an LDA-based topic crawler, use an LDA-based text classification model to classify into predetermined A categories and store them in a local academic resource database, including academic Resource model, resource quality value calculation model, user interest model, colonize the tracking software module in the user's terminal, combine the user's interest subject and historical browsing behavior data, academic resource type, subject distribution, keyword distribution and LDA potential topic distribution A total of four dimensions are used to model the academic resource model and the user interest model, calculate the similarity between the academic resource model and the user interest preference model, and then combine the resource quality value to calculate the recommendation degree, and finally provide users with academic resources according to the recommendation degree. Top-N recommendation; the present invention performs personalized and accurate recommendation of academic resources according to user identity, interest and browsing behavior, and improves the work efficiency of scientific researchers.

Description

An academic resource recommendation service system and method

technical field

The invention relates to the field of computer application technology, in particular to an academic resource recommendation service system and a method for providing academic resource recommendation services to related users by using the resource recommendation service system.

Background technique

At present, we have entered the era of big data, especially in the field of academic resources, where hundreds of millions of various academic resources are generated every year. In addition to academic papers and patents, there are also a large number of academic resources such as academic conferences, academic news, and academic community information emerging in real time. These types of academic resources are of great significance for users to accurately and efficiently grasp the current scientific research status in fields of interest. However, scientific research users usually have heavy scientific research work. This kind of academic resources have the characteristics of heterogeneity, heterogeneity and rapid growth of big data. It is difficult to search and identify academic resources based on traditional search engines, and the search process is also difficult. It is cumbersome, and users often need to spend a lot of time and energy in searching for academic resources they are interested in, which affects their work efficiency.

The current research on personalized recommendation of academic resources mainly focuses on academic papers, and the types of recommended academic resources are single; different user groups, that is, users with different identities, pay different attention to different types of academic resources. Without considering these factors, a multi-strategy recommendation scheme based on user identity cannot be formulated. Moreover, the current academic resource recommendation research is limited to the recommendation module, and the present invention provides systematic services for academic resource recommendation, from dynamic acquisition, integration and classification of academic resources to academic resource selection based on user identity, behavior and interest subjects. Personalized recommendation, forming an integrated service system centered on resource integration and recommendation.

LDA (Latent Dirichlet Allocation) is a document topic generation model, also known as a three-layer Bayesian probability model, which includes three layers of words, topics and documents. The so-called generative model means that we believe that each word in an article is obtained through the process of "selecting a certain topic with a certain probability, and selecting a certain word from this topic with a certain probability". A topic refers to a well-defined professional field or field of interest, such as aerospace, biomedicine, information technology, etc., and specifically refers to a set of related words. Documents to topics follow a multinomial distribution, and topics to words follow a multinomial distribution. LDA is an unsupervised machine learning technique that can be used to identify hidden topic information in documents. It uses the bag of words method, which treats each document as a word frequency vector, thus converting text information into digital information that is easy to model. Each document represents a probability distribution composed of some topics, and each topic represents a probability distribution composed of many words. The LDA topic model is a typical model of topic mining in natural language processing. It can extract potential topics from text corpus and provide a method for quantifying research topics. It has been widely used in topic discovery of academic resources, such as research hotspot mining, Research topic evolution, research trend prediction, etc.

In addition, with the discovery of the Internet, the Internet is filled with a large number of information texts in various forms such as news, blogs, and meeting minutes. Such information texts more or less include academic-related information, and often include the latest Academic research information is concerned by people of various related disciplines, and this kind of information text is messy and disorderly, often with overlapping disciplines, and generally does not have classification information itself. It is often difficult to make correct automatic classification of this type of information text in the existing technology. It is difficult for people in various related disciplines to search thoroughly and accurately by using traditional search engines, and the search process is also cumbersome. It often takes a lot of time and energy for users to query academic resources they are interested in, which affects their work efficiency.

The present invention is just to solve above-mentioned technical problem.

Contents of the invention

The technical problem to be solved by the present invention is to provide an academic resource recommendation service system and a method for providing academic resource recommendation services for relevant users with the resource recommendation service system in view of the above-mentioned technical status.

The technical solution adopted by the present invention to solve the problems of the technologies described above is:

An academic resource recommendation service system, the academic resources are various electronic texts published on the Internet, the academic resource recommendation service system includes a web crawler, a text classification model, and an academic resource database, crawled by the web crawler on the Internet The academic resources are characterized in that they are stored in a local academic resource database after being classified according to predetermined A categories with a text classification model, and an open API of the academic resource database is provided for display and resource recommendation module calls, and the academic resource recommendation service system also includes The academic resource model, the resource quality value calculation model, and the user interest model are embedded in the user's terminal with a tracking software module to track and record the user's online browsing behavior; The attention degree of various types of academic resources is modeled from the four dimensions of resource type, subject distribution, keyword distribution and LDA potential topic distribution. Combined with the user's interested subjects and historical browsing behavior data, the user's interest preference is modeled. model, calculate the similarity between the academic resource model and the user's interest preference model, and then calculate the recommendation degree based on the resource quality value, and finally make the Top-N recommendation of academic resources for users according to the recommendation degree.

The web crawler is a topic crawler, and also includes an LDA topic model. The LDA topic model is a three-layer Bayesian generation model of "document-topic-word". A corpus is pre-configured for the LDA topic model. In the corpus Including the training corpus, use the training corpus to train the LDA topic model according to the set topic number K, and use the word gathering function during the training of the LDA topic model to obtain K according to the set topic number K after the training corpus is trained by the LDA topic model Subject-related words set, promptly obtain K subject documents crawled by subject crawler this time; Described subject crawler further includes subject determination module, similarity calculation module, URL prioritization module on the basis of common web crawler; Described subject crawler is Multiple distributed crawlers distributed according to the number of academic topics, each distributed crawler corresponds to an academic topic, and each distributed crawler obtains academic resources of multiple academic topics at the same time; during each crawling process of a topic crawler, the topic determination module of the topic crawler Determine the target theme and its theme document, use the theme document to guide the calculation of theme similarity, the similarity calculation module calculates and judges the theme similarity of each anchor text on the crawled page and combines the content of the page, and removes the anchor The text is combined with the hyperlink whose theme similarity of the page is less than the set threshold, and the selected anchor text is combined with the URL whose theme similarity of the page is greater than the set threshold, and the theme crawler maintains an unvisited page pointed by the hyperlink of the visited page The URL queue of the web page, the URL queue is arranged in descending order according to the similarity. The theme crawler continuously visits the web pages of each URL according to the order of the URL queue, crawls the corresponding academic resources, and continuously classifies and labels the crawled academic resources Afterwards, it is stored in the database, and for the theme documents crawled this time, until the queue URL is empty; the academic resources crawled by the theme crawlers are used as new corpus for LDA theme model training; and the theme crawlers are constantly repeated During the crawling process, the collected subject related words of each subject document can be continuously supplemented and updated, and the crawled academic resources can be continuously supplemented and updated to a degree recognized by humans.

The corpus also includes a verification corpus with clear categories, which is used to use the verification corpus in advance to allow the text classification model to perform classification verification according to the predetermined number A of categories, so that the text classification model can accurately classify each category in the A categories. rate, as the classification credibility index of each category in the A category by the text classification model; the accuracy rate is the ratio of the correctly classified corpus among all the verification corpora classified into a certain category by the text classification model, And preset classification accuracy threshold.

Divide all subjects into 75 subject categories, that is, the number of categories A is 75 categories, the number of topics K is set to 100 when using the LDA topic model training, and the classification accuracy is preset when the text classification model performs classification verification The threshold is 80%.

A method for providing academic resource recommendation services for relevant users with a resource recommendation service system, the academic resources are various electronic texts published on the Internet, including using a web crawler to crawl academic resources on the Internet, characterized in that, using The text classification model classifies and stores the crawled academic resources according to predetermined A categories to form an academic resource database, provides an open API of the academic resource database for display and resource recommendation module calls, and uses resource quality value calculation models and user interest models , the tracking software module is implanted in the user's terminal to track and record the user's online browsing behavior; the process of recommending the corresponding academic resources to the user includes the cold start recommendation stage and the second recommendation stage. The cold start recommendation stage is based on the subject of interest as Users recommend high-quality resources that meet their interests. The high-quality resources are academic resources with high resource quality values calculated by the resource quality value calculation model. The resource quality values are resource authority, resource community popularity, and resource updates. In the secondary recommendation stage, the user interest model and the resource model are modeled separately, the similarity between the user interest model and the resource model is calculated, and then the recommendation degree is calculated by combining the resource quality value, and finally according to The recommendation degree is the Top-N recommendation of academic resources for users.

The calculation of the resource quality value Quality includes that the calculation formula of the resource authority degree Authority is as follows:

Among them, Level is the quantified score of resource publishing publication level, and the publication level is divided into 5 levels, and the scores are 1, 0.8, 0.6, 0.4 and 0.2 points. Top journals or conferences such as Nature and Science get 1 point, second-level ones like ACM Transaction get 0.8 points, and the lowest level ones get 0.2 points; the calculation formula of Cite is as follows:

Cite=Cites/maxCite (2)

Cite is the quantitative result of resource citations, Cites is the citation of resources, and maxCite is the largest citation in the resource source database;

The formula for calculating the Popularity of the resource community is as follows:

Popularity=readTimes/maxReadTimes (3)

readTimes is the number of readings of the paper, and maxReadTimes is the maximum number of readings in the resource source database;

The calculation method of the recentness of resources is the same, and the formula is as follows:

year and month are the publication year and month of the resource respectively; minYear, minMonth, maxYear and maxMonth are the earliest and latest publication year and month of all resources in the source database of this type of resource;

The resource quality value Quality is calculated as follows:

The academic resource model is represented as follows:

M _r ＝{T _r ,K _r ,C _t ,L _r } (6)

Among them, T _r is the subject distribution vector of academic resources, which is the probability value of the academic resources distributed in A subject category, which is obtained by the Bayesian polynomial model;

K _r ＝{(k _r1 ,ω _r1 ),(k _r2 ,ω _r2 ),…,(k _rm ,ω _rm )}, m is the number of keywords, k _ri (1≤i≤m) means a single academic The i-th keyword of the resource, ω _ri is the weight of the keyword k _ri , which is obtained through the improved tf-idf algorithm, and the calculation formula is as follows:

w(i,r) represents the weight of the i-th keyword in document r, tf(i,r) represents the frequency of the i-th keyword appearing in document r, Z represents the total number of documents in the document set, and L represents The number of documents containing keyword i; Lr is the potential topic distribution vector, L _r = {l _r1 , l _r2 , l _r3 ..., l _rN1 }, N1 is the number of potential topics; Ct is the resource type, and the value of t can be 1, 2, 3, 4, 5 are five major categories of academic resources: papers, patents, news, conferences and books;

According to the behavior characteristics of users using mobile software, the user's operation behavior on an academic resource is divided into opening, reading, star rating, sharing and collection. The user interest model is based on the user's background and the academic resources that have been browsed. According to the user's different browsing Behavior, combined with the academic resource model, constructs a user interest model. The user interest model is expressed as follows:

M _u ＝{T _u ，K _u ，C _t ，L _u } (8)

Among them, T _u is the subject distribution vector T _r of a certain type of academic resources browsed by the user within a period of time, and the user's subject preference distribution vector is formed after the user's behavior, that is

Among them, sum is the total number of academic resources that the user has acted on, and s _j is the "behavior coefficient" after the user has acted on the academic resource j. The larger the value, the more the user likes the resource. T _jr represents the subject distribution vector of the jth resource. The calculation of s _j comprehensively considers behaviors such as opening, reading, evaluating, collecting and sharing, and can accurately reflect the user's preference for resources.

K _u = {(k _u1 , ω _u1 ), (k _u2 , ω _u2 ),..., (k _uN2 , ω _uN2 )} is the user's keyword preference distribution vector, N ₂ is the number of keywords, k _ui (1≤i≤N ₂ ) represents the i-th user preference keyword, ω _ui is the weight of keyword k _ui , and the "keyword distribution vector" K _r is calculated.

K _jr ′=s _j *K _jr (10)

The new keyword distribution vector of each resource can be calculated according to formula 10, and then TOP-N2 of the new keyword distribution vector of all resources is selected as the user keyword preference distribution vector K _u ;

L _u is the user's LDA latent topic preference distribution vector, which is calculated from the LDA latent topic distribution vector of academic resources L _r = {l _r1 , l _r2 , l _r3 ..., l _rN1 }, the method is the same as T _u :

The similarity between user interests and resource models is calculated as follows:

The Academic Resource Model says:

M _r = {T _r , K _r , C _t , L _r } (12)

The user interest model represents:

M _u = {T _u , K _u , C _t , L _u } (13)

The similarity between user subject preference distribution vector T _u and academic resource subject distribution vector T _r is calculated by cosine similarity, namely:

The similarity between the user LDA latent topic preference distribution vector L _u and the academic resource LDA latent topic distribution vector L _r is calculated by cosine similarity, namely:

The similarity calculation between the user keyword preference distribution vector K _u and the academic resource keyword distribution vector K _r is calculated through Jaccard Similarity:

Then the similarity between the user interest model and the academic resource model is:

Among them, σ+ρ+τ=1, and the specific weight distribution is obtained from experimental training.

Introduce the concept of Recommendation_degree. The higher the recommendation degree of an academic resource, the more suitable the resource is for the user's interests and preferences, and the higher the quality of the resource. The calculation formula of the recommendation degree is as follows:

Recommendation_degree＝λ ₁ Sim(M _u ,M _n )+λ ₂ Quality(λ ₁ +λ ₂ =1) (18)

The second recommendation stage is to make Top-N recommendations based on the recommendation degree of academic resources.

The web crawler includes addressable crawler and topic crawler, and also includes an LDA topic model. The LDA topic model is a three-layer Bayesian generation model of "document-topic-word", and a corpus is pre-configured for the LDA topic model , the corpus includes the training corpus, use the training corpus to train the LDA topic model according to the set topic number K, use the word gathering function during the training of the LDA topic model to obtain the training corpus according to the set topic number K after the training corpus is trained by the LDA topic model Form K subject-associated word collections, promptly obtain the K theme documents crawled by the theme crawler this time; The theme crawler further includes a theme determination module, a similarity calculation module, and a URL prioritization module on the basis of a common web crawler; Theme crawlers are multiple distributed crawlers distributed according to the number of academic topics. Each distributed crawler corresponds to an academic theme, and each distributed crawler obtains academic resources of multiple academic topics at the same time; The theme determination module determines the target theme and its theme document, uses the theme document to guide the calculation of theme similarity, and the similarity calculation module performs theme similarity calculation and judgment on each anchor text on the crawled page combined with the content of the page , remove the hyperlinks whose anchor text combined with the theme similarity of the page is less than the set threshold, select the URL whose anchor text combined with the theme similarity of the page is greater than the set threshold, and the theme crawler maintains a hyperlink pointed to by the visited web page The URL queue of the unvisited webpages, the URL queue is arranged in descending order according to the similarity, the topic crawler continuously visits the webpages of each URL according to the order of the URL queue, crawls the corresponding academic resources, and continuously crawls the crawled academic resources After the resource classification label is stored in the database, for the theme document crawled this time, until the URL of the unvisited queue is empty; the academic resources crawled by the theme crawler each time are used as new corpus for LDA theme model training; and continuously Repeating the crawling process of the topic crawler enables the collection of topic related words in each topic document to be continuously supplemented and updated, and the crawled academic resources are continuously supplemented and updated to an artificially recognized level.

The corpus also includes a verification corpus with clear categories, which is used to use the verification corpus in advance to allow the text classification model to perform classification verification according to the predetermined number A of categories, so that the text classification model can accurately classify each category in the A categories. rate, as the classification credibility index of each category in the A category by the text classification model; the accuracy rate is the ratio of the correctly classified corpus among all the verification corpora classified into a certain category by the text classification model, And preset classification accuracy rate threshold; When using the text classification model to carry out text classification for each text to be classified, specifically include the following steps:

Step 1. Preprocess each text to be classified. Preprocessing includes word segmentation, removing stop words, and retaining proper nouns. Calculate the feature weights and feature weight values of all words in the text after preprocessing Proportional to the number of times it appears in the text, and inversely proportional to the number of times it appears in the training corpus, the calculated word set is arranged in descending order according to its characteristic weight value, and the original word set of each text to be classified is extracted. The front part is used as its feature word set;

Step 2. Using the text classification model, select the original feature word set of each text to be classified to calculate the probability value that the text may belong to each of the predetermined A categories, and select the category with the highest probability value as the text classification category;

Step 3. Judging the text classification results of step 2. If the classification accuracy value of the text classification model for this category reaches the set threshold, the result will be output directly; if the classification accuracy value of the text classification model for this category does not reach the set threshold threshold, go to step 4;

Step 4, input each preprocessed text into the LDA topic model, use the LDA topic model to calculate the weight value of each topic in the K topics corresponding to the text, and select the one with the largest weight value topic, and add the first Y words in the topic related words under the topic obtained after the LDA topic model training in advance to the original feature word set of the text as the expanded feature word set, and use the text again The classification model calculates the probability values that the text may belong to each of the predetermined A categories, and selects the category with the highest probability value as the final classification category of the text.

The main calculation formula of the text classification model is:

Where P(c _j |x ₁ , x ₂ ,..., x _n ) represents the probability that the text belongs to category cj when the feature word (c _j |x ₁ , x ₂ ,..., x _n ) appears at the same time; Among them, P(c _j ) represents the ratio of the text belonging to category c _j in the training text set, and P(x ₁ , x ₂ ,..., x _n |c _j ) indicates that if the text to be classified belongs to category c _j , Then the feature word set of this text is the probability of (x ₁ , x ₂ ,..., x _n ), P(c ₁ , c ₂ ,..., c _n ) represents the joint probability of all given categories .

The resource recommendation service system for multi-type academic resources described in the present invention has the following characteristics:

(1) The present invention realizes the dynamic acquisition of various types of academic resources such as academic papers, patents, academic conferences and academic news, and efficiently acquires target academic resources based on the theme crawler module.

(2) The present invention realizes the subject classification of various types of academic resources based on subject attributes.

(3) Different user groups pay different attention to different types of academic resources. The present invention implements a multi-strategy academic resource recommendation scheme based on different user groups, and recommends various types of academic resources in different proportions for users with different identities.

(4) Based on user browsing habits, the present invention realizes personalized recommendation of various types of academic resources based on different behaviors of users.

The present invention performs personalized recommendation of academic resources according to user identity, interest and browsing behavior, can recommend academic resources to users more accurately, greatly improves the work efficiency of scientific researchers, and creates convenience and speed for scientific research workers to better conduct scientific research. The information acquisition environment effectively resolves the contradiction between information overload of academic resources and user resource acquisition.

In addition, the present invention adopts the LDA-based academic resource acquisition method and classification method, and uses the LDA topic model to deeply mine topic semantic information, construct a good guidance foundation for topic crawlers of academic resources, and integrate machine learning into the academic resource acquisition method , improve the quality and efficiency of academic resource acquisition; the academic resources obtained by the theme crawler are used for LDA theme update, the theme model can be updated at any time, follow up the trend of academic development, and provide scientific researchers with cutting-edge resources in related fields; the invention proposes The text classification method based on selective feature expansion is suitable for complex application scenarios. It selectively adds topic information to data with less information, and avoids adding noise to data with sufficient information, which provides a way for the optimization of text classification models. , and has the characteristics of strong scene adaptability, high availability of results, and easy update and maintenance of classification models.

Description of drawings

Fig. 1 is the frame schematic diagram of the whole academic resource recommendation service system of the present invention;

Fig. 2 is the schematic diagram of LDA model;

FIG. 3 is a schematic diagram of a text before preprocessing of a certain text;

FIG. 4 is a schematic diagram of a text after preprocessing of a certain text;

Fig. 5 is the schematic diagram of topic and topic document after training corpus through LDA topic model training;

Fig. 6 is a schematic flow diagram of the present invention using an LDA-based academic resource acquisition method;

Fig. 7 is the schematic flow chart that the present invention adopts the text classification method based on LDA;

Figure 8 is a schematic diagram of the recall rate of three experiments in some subjects;

Figure 9 is a schematic diagram of the precision rate of three experiments in some subjects

Fig. 10 is a schematic diagram of the recommendation process of the present invention.

detailed description

Specific embodiments of the present invention will be described in detail below.

The academic resource recommendation service system of the present invention, as shown in Figure 1, includes a web crawler, a text classification model, and an academic resource database, crawls academic resources on the Internet by the web crawler, uses the text classification model to classify and store in predetermined A categories The local academic resource database provides an open API of the academic resource database for display and resource recommendation module calls; the academic resource recommendation service system of the present invention also includes an academic resource model, a resource quality value calculation model, and a user interest model, which are colonized in the user's terminal The tracking software module is used to track and record users' online browsing behavior; based on the historical browsing behavior data of different groups of users, calculate the degree of attention of users with different identities to various types of academic resources, from resource types, subject distribution, keyword distribution and LDA There are four dimensions of potential topic distribution to model academic resources, combine user interest subjects and historical browsing behavior data, model user interest preferences, calculate the similarity between academic resource models and user interest models, and combine resource quality values Calculate the recommendation degree, and finally recommend Top-N academic resources for users according to the recommendation degree. According to the subject categories in the "Catalogue of Postgraduate Disciplines" issued by the Ministry of Education, all first-level subjects are organized into 75 subject categories, that is, the number of categories A is 75 categories.

1. Acquisition of academic resources

The web crawler of the present invention is mainly a topic crawler, and also includes a corresponding LDA topic model, and the LDA topic model is a three-layer Bayesian generation model of "document-topic-word", as shown in Figure 2; Set the topic number K to let the LDA topic model train. Of course, each training corpus needs to be preprocessed before training. The preprocessing includes word segmentation and de-staying words; use the word gathering function during the training of the LDA topic model to pass through the LDA topic model in the training corpus. After training, K sets of subject-related words are gathered according to the set number of topics K, which are also called topic documents; when using the LDA topic model for training, the number of topics K can be set to 50 to 200, and the number of topics K is preferably 100; various forms of documents in various disciplines can be randomly crawled from the Internet, and documents such as papers that are very long but have standardized abstracts can only be extracted from their abstracts, or ready-made databases can be used as training corpus, the number of documents It should reach a considerable scale, at least tens of thousands of articles, and as many as several million articles. If the number of topics K is selected as 100, all words in the training corpus will be aggregated into 100 topic-related word sets during the training process of the LDA topic model, that is, 100 topic documents; we can artificially name each set according to the meaning of each set of words The topic name, or not naming each topic name, but only using numbers or codes to indicate the difference. Three of the topic documents are shown in Figure 5.

The topic crawler further includes a topic determination module, a similarity calculation module, and a URL prioritization module on the basis of an ordinary web crawler; the topic crawler is a plurality of distributed crawlers distributed according to the number of academic topics, and each distributed crawler corresponds to a Academic topics, each distributed crawler obtains academic resources of multiple academic topics at the same time; during each crawling process of the topic crawler, the topic determination module of the topic crawler determines the target topic and its topic document, and uses the topic document to guide the calculation of topic similarity. The degree calculation module calculates and judges the topic similarity of each anchor text on the crawled page combined with the content of the page, eliminates the hyperlinks whose anchor text combined with the topic similarity of the page is less than the set threshold, selects the anchor text combined with the hyperlink For URLs whose theme similarity of the page is greater than the set threshold, the theme crawler maintains a URL queue of unvisited webpages pointed to by the hyperlinks of the visited webpages. The order of arrangement is to continuously visit the webpages of each URL, crawl the corresponding academic resources, and continuously store the crawled academic resources into the database after classifying and labeling them. For the theme documents crawled this time, until the unvisited queue URL is empty; The academic resources crawled by the topic crawler each time are used as new corpus for LDA topic model training; and the crawling process of the topic crawler is repeated continuously, so that the topic associated words collected in each topic document can be continuously supplemented and updated, and the crawled academic resources Resources are constantly being replenished to an artificially acceptable level.

For the convenience of operation, the summary of academic resources can be used as the training corpus, and the topics and topic documents can be calculated through the LDA topic model. As a new corpus for the LDA training model, the open API of the academic resource database is provided for display and calling; the specific steps are as follows:

Step 1. Download and preprocess the summaries of existing academic resources in multiple fields, artificially divide them into different categories according to the academic fields, and use them as training corpora for multiple topics of LDA;

Step 2. Input the LDA topic model parameters. The LDA topic model parameters include K, α, and β, where the value of K represents the number of topics, the value of α represents the weight distribution of each topic before sampling, and the value of β represents the weight of each topic pair word. Prior distribution, training to get more subdivided topics and topic documents of multiple topics, each topic document is used to guide a crawler;

Step 3. Each crawler starts from the selected high-quality seed URL, maintains a crawling URL queue, and continuously calculates the similarity between the text in the web page and the text and topic pointed to by the anchor text link in the web page, and sorts and updates according to the similarity Crawl the URL queue and grab the most relevant web page content;

Step 4. The academic resources obtained by the topic crawler are tagged with corresponding topic tags, stored in the database, and used as new corpus for training LDA for updating topic documents;

Step 5: Provide the open API of the academic resource database for display and call.

Wherein step one includes following specific sub-steps:

(a) Corpus collection: download summaries of existing academic resources in multiple fields as training corpus;

(b) Text preprocessing: extract summary, Chinese word segmentation, remove stop words;

(c) Classification into the corpus: artificially divided into different categories according to the academic field, and used as the training corpus for multiple topics of LDA.

Step three includes the following specific sub-steps:

(a) the initial seed URL selects a better seed site facing a specific theme;

(b) extracting webpage content: download the page pointed to by the URL with high priority, and extract the required content and URL information according to the HTML tag;

(c) analysis and determination of topic relevance, and decision of page selection; the present invention mainly adopts combining existing VSM technology and SSRM technology to calculate topic relevance;

(d) sorting the importance of URLs of unvisited webpages;

(e) Repeat steps (b) to (d) until the unvisited queue URL is empty.

In sub-step (c), when the topic crawler crawls through each electronic document to analyze and judge the topic relevance, it uses the generalized vector space model GVSM, which combines the VSM and SSRM two topic similarity calculation algorithms, to calculate the crawled The topic relevance of the page determines the choice of the page.

A topic is represented by a group of semantically related words and a weight indicating that the word is related to the topic, that is, topic Z={(w ₁ ,p ₁ ),(w ₂ ,p ₂ ),…,(w _n ,p _n )}, where the i-th word w _i is a word related to topic Z, p ₁ is the measure of the correlation between the word and Z, expressed as Z={(w ₁ , p(w ₁ |z _j )), (w ₂ , p(w ₂ |z _j )), …, (w _n , p(w _n |z _j ))}, where w _i ∈ W, p(w _i |z _j ) is The probability of choosing word w _i when the topic is Z _j , z _j is the jth topic.

The subject document generation process is a probability sampling process of the model, including the following specific sub-steps:

(a) For any document d in the corpus, generate document length N, N～Poisson(ε), obey Poisson distribution;

(b) For any document d in the corpus, generate a θ~Dirichlet(α), which obeys the Dirichlet distribution;

(c) Generation of the i-th word wi in document d: First, generate a topic z _j ～Multinomial(θ), which obeys multinomial distribution; then, for topic z _j , generate a discrete variable Obey the Dirichlet distribution; finally generate such that The most probable word. The LDA model is shown in Figure 3.

Among them, the value of α represents the weight distribution of each topic before sampling, and the value of β represents the prior distribution of each topic to words.

All variables in the LDA model and their distributions are as follows:

The entire model can actually be transformed into a joint distribution of P(w|Z) by integrating possible variables. where w refers to words and is observable. Z is the variable of the topic, which is the target product of the model. It can be seen that α and β are the initial parameters of the model. Then by integrating the variables present in it, we get:

Among them, N is the vocabulary length, w is a word, for θ～Dirichlet(α), Integrate θ to get:

in, Indicates the number of times feature word w is assigned to topic j, Indicates the number of feature words assigned to topic j, Indicates the number of feature words assigned to topic j in text d, Indicates the number of all feature words assigned topics in text d.

It can be seen from the above that the three variables that affect LDA modeling are mainly α, β and the number of topics K. In order to select a better number of topics, the values of α and β are fixed first, and then the change of the value of the formula after integrating other variables is calculated.

When the LDA model is used to model the text set, the number of topics K has a great influence on the performance of the LDA model fitting the text set, so the number of topics needs to be set in advance. This paper determines the optimal number of topics by measuring the classification effect under different number of topics, and compares it with the classification effect when using the Perplexity value to determine the best fit of the model. On the one hand, the method in this paper can obtain a more intuitive and accurate optimal number of topics. On the other hand, the optimal number of topics determined by the Perplexity value can find out the gap between the corresponding classification effect and the actual result. The formula for the Perplexity value is:

Among them, M is the number of texts in the text set, N _m is the length of the mth text, P(d _m ) is the probability of the LDA model generating the mth text, the formula is:

The theme crawler of the present invention adds three modules on the basis of the general crawler: a theme determination module, a similarity calculation module, and a URL priority sorting module, thereby completing the filtering and theme matching of crawled pages, and finally obtaining a highly relevant theme Content.

1. Theme determination module: Before the theme crawler works, it needs to determine the relevant subject word set of the theme crawler, that is, to establish the theme document. There are usually two ways to determine the subject term set, one is manually determined, and the other is extracted from the initial page set. Manually determine the subject word set, the training selection of keywords is subjective, and the keywords extracted from the initial page have high noise and low coverage. The number of topic words is used as the dimension of the topic vector, and the corresponding weight is the value of each component of the topic vector. Note that the keyword set vector is: K={k1, k2, ..., kn}, where n is the number of keyword.

2. Similarity calculation module: In order to ensure that the web pages obtained by the crawler can be as close as possible to the topic, the web pages must be filtered, and the web pages with low topic relevance (less than the set threshold) are eliminated, so that they will not be crawled in the next step. Process links in this page. Because if the topic relevance of a page is very low, it means that some keywords are likely to appear on the page occasionally, and the topic of the page may have little to do with the specified topic, and the meaning of processing the links in it is very small. This is the theme crawler and The fundamental difference between ordinary reptiles. Ordinary crawlers process all links according to the set search depth, and as a result, a large number of useless web pages are returned, which further increases the workload. It is obviously not feasible to use the entire text for similarity comparison. Usually, it is necessary to refine and extract the text and transform it into a data structure suitable for comparison and calculation, while ensuring that the theme of the text is reflected as much as possible. The feature selection used by common theme crawlers is VSM, which also involves the TF-IDF algorithm. This article uses the semantic similarity calculation based on "HowNet". By calculating the similarity between the words in the document and the keyword document, the similarity value between the entire article and the topic is obtained.

3. URL priority sorting module: The URL priority sorting module is mainly to screen out potential pages with high similarity to the topic from unvisited URLs, and sort them according to the level of similarity. The higher the similarity, the higher the priority. Prioritize visits with high similarity as much as possible to ensure that the visited pages are highly topic-related. When sorting unvisited URLs, the similarity between the page where the URL is located and the URL anchor text (text describing the URL) can be combined as an influencing factor for prioritization.

The present invention utilizes the definition of the semantic information of each word by "HowNet" to calculate the similarity between words. In HowNet, for two words W ₁ and W ₂ , let W ₁ have a concept: W2 has m concepts: The similarity of W ₁ and W ₂ is each concept of W ₁ Every concept with W ₂ The maximum value of the similarity, the formula is as

In this way, the similarity between two words can be transformed into the calculation of the similarity between concepts. All concepts in HowNet are finally attributed to the representation of the original meaning, so the calculation of the similarity between concepts can also be attributed to the corresponding meaning. Calculation of the similarity between the original. Suppose concept c ₁ and concept c ₂ have p and q sememes respectively, denoted as The similarity between concept c ₁ and concept c ₂ is that each sememe of c ₁ and each sememe of c ₂ The maximum value of the similarity, the formula is:

All concepts in "HowNet" are finally attributed to the representation of sememes, so the calculation of similarity between concepts can also be attributed to the calculation of similarity between corresponding sememes. Since all sememes constitute a tree-like sememe hierarchy system according to the upper and lower relations, the semantic distance of sememes in the sememe hierarchy system can be used to calculate the sememe similarity, and then obtain the concept similarity[27] . Assuming that two sememes and the path distance in the sememe hierarchy are Dis(s ₁ , s ₂ ), the formula for calculating the similarity of the sememes is:

Among them, Dis(s ₁ , s ₂ ) is the path length of s ₁ and s ₂ in the sememe hierarchy system, and the sememe-hypernym relationship is used here, which is a positive integer.

The design of the subject reptile of the present invention is based on common reptiles, and further expands its functions. Steps in the whole process of webpage processing: determination of initial seed URL, extraction of webpage content, topic correlation analysis, and URL sorting.

(a) The initial seed URL selects a better seed site for a specific topic, so that the topic crawler can crawl smoothly.

(b) Extract web page content: download the page pointed to by the URL with high priority, and extract the required content and URL information according to the HTML tags.

(c) Topic correlation analysis is the core module of the topic crawler, which determines the selection of pages. The present invention mainly adopts the generalized vector space model GVSM which combines the existing VSM technology and SSRM technology to calculate the subject correlation.

For topic correlation analysis, TF-IDF is used to extract text keywords, and the weight of words is calculated to analyze the relevance of web pages.

TF-IDF related calculations:

Where w _di is the weight of word i in document d, tf _i is the word frequency of word i, idf _i is the inverse document frequency of word i, f _i is the number of times word i appears in document d, and f _max is the word i in document d The number of occurrences with the highest frequency among all words, N is the number of all documents, and N _i is the number of documents containing word i. TF-IDF is still the most effective method to extract keywords and calculate the weight of words.

VSM topic correlation calculation:

in is the word vector of document d, is the word vector of topic t, w _di and w _ti are the TF-IDF values of word i in document d and topic t, and n is the number of common words that appear in document d and topic t. This algorithm only considers the frequency vector of the same word in the document as a judgment of document similarity, and does not take into account the semantic relationship between words, such as synonyms, synonyms, etc., which affects the accuracy of similarity .

SSRM topic correlation calculation:

Where w _di and w _ti are the TF-IDF values of word i in document d and topic t, n and m are the number of words in document d and topic t respectively, and Sem _ij is the semantic similarity between word i and word j.

Among them, C ₁ and C ₂ are two concepts, equivalent to word w ₁ and word w ₁ , Sem(C ₁ , C ₂ ) is the semantic similarity between concept C ₁ and concept C ₂ , and C ₃ is C ₁ and C ₂ The lowest common concept shared, Path(C ₁ , C ₃ ) is the number of nodes on the path from C ₁ to C ₃ , Path(C ₂ , C ₃ ) is the number of nodes on the path from C ₂ to C ₃ , Depth(C ₃ ) is the number of nodes on the path from C ₃ to the root node in some different ontologies. Using the SSRM algorithm, only the semantic relationship is considered. If the words in two articles are all synonyms or synonyms, then the similarity of this document will be calculated as 1, that is, they are exactly the same, which is obviously not accurate.

The present invention adopts the method for calculating similarity in combination with VSM and SSRM, also known as generalized vector space model, referred to as GVSM, and its calculation formula is:

Where Sim(d _k , t) is the subject similarity of document d _k , the present invention takes into account the word frequency factor of the document and the semantic connection between words, and adopts the method of combining VSM and SSRM to effectively improve the calculation efficiency of subject similarity precision.

(d) Ranking the importance of URLs of unvisited webpages. The following formula is used to sort the URLs:

Among them, priority(h) is the priority value of unvisited hyperlink h, N is the number of retrieved webpages containing h, Sim(f _p , t) is the subject similarity of the full text of webpage p (including hyperlink h), Sim(a _h , t) is the topic similarity of the anchor text of the hyperlink h, and λ is the weight value for adjusting the full text and the anchor text. The similarity calculation in the formula also adopts the combination of VSM and SSRM, which optimizes the priority ranking of the uncrawled URL link queue, and also effectively improves the accuracy of subject academic resource acquisition.

The theme crawler of the present invention is a network information grabbing tool specially designed for grabbing certain theme resources. The degree of relevance to the topic determines whether to crawl the webpage, and maintains a queue of URLs to be crawled, and visits the pages according to the priority of the URLs to ensure that pages with high relevance are accessed first.

There are some deficiencies in the current topic crawler: (1) the topic crawler should determine the relevant subject word set of the topic crawler before working. There are usually two ways to determine the keyword set, one is manually determined, and the other is obtained through initial page analysis. There is a certain degree of subjectivity in the manual determination method; and the method of extracting keywords through the initial page is generally insufficient in topic coverage. The two traditional methods will cause a lot of deviation when the topic crawler calculates the similarity of webpage topics. (2) The core of the current text-based heuristic theme crawler is page similarity calculation. To judge whether the currently crawled webpage is similar to the theme, in addition to the accuracy of the theme determination module, the most important thing is the similarity calculation algorithm, usually using VSM (Vector Space Model), based on the assumption that different words are not related, uses word vectors to represent text, and calculates the similarity between documents through the common word frequency. This algorithm often ignores the semantic relationship between words. , which reduces the similarity value of highly semantically related articles.

The theme crawler of the present invention is designed on the basis of a general crawler, adding three core modules: a theme determination module, a theme similarity calculation module and a URL sorting module to be crawled. Aiming at the above deficiencies, the present invention proposes a theme crawler based on the theme model LDA, and improves the theme similarity algorithm and the URL prioritization algorithm, and improves the content quality and accuracy of the theme crawler from the initial crawling and crawling process. Main contributions: (1) Through the LDA topic model, deeply mine the semantic information of the corpus topic, construct a good guidance foundation for the topic crawler, integrate machine learning into the resource acquisition method, and improve the accuracy and quality of resource acquisition. (2) In the topic similarity calculation module of the topic crawler, the method of semantic similarity calculation based on "HowNet" is adopted to balance the cosine similarity and semantic similarity to achieve better topic matching effect.

2. Classification of academic resources

The present invention adopts the text classification method based on LDA, as shown in Figure 7, uses the Bayesian probability calculation model as the text classification model, extracts a group of feature words that can best reflect the characteristics of the text to be classified as the input text classification model The feature word set, the original feature word set is the front part after the original word set is sorted according to the characteristic weight, and the text classification model is used to calculate the probability that the feature word combination belongs to each of the predetermined A categories, and the probability value is the largest According to the subject categories in the "Catalogue of Postgraduate Disciplines" of the Ministry of Education, all subjects are divided into 75 subject categories, that is, the number of categories A is 75 categories. Use the LDA topic model mentioned above and the 100 topic documents trained by it to assist the text classification model for text classification. Also use the verification corpus with clear categories in advance to allow the text classification model to perform classification verification according to the predetermined number of categories A, so as to obtain the classification accuracy rate of the text classification model for each category in the A categories, as the text classification model for A categories. The classification credibility index of each category; the accuracy rate is the ratio of the correctly classified corpus among all the verification corpora classified into a certain category by the text classification model, and the classification accuracy threshold is preset; the text classification model performs It is more appropriate to preset the classification accuracy threshold to be 80% during classification verification. When using the text classification model to classify each text to be classified, it specifically includes the following steps:

Step 1. For each text to be classified, calculate the feature weights of all the words of the text after preprocessing. The feature weight value of the word is proportional to the number of times it appears in the text, and is proportional to the number of times it appears in the training corpus. Inversely proportional to the number of times, the calculated word sets are arranged in descending order according to their characteristic weight values, and the front part of the original word set of each text to be classified is extracted as its feature word set.

Step 2. Using the text classification model, select the original feature word set of each text to be classified to calculate the probability value that the text may belong to each of the predetermined A categories, and select the category with the highest probability value as the text classification category;

Step 3. Judging the text classification results of step 2. If the classification accuracy value of the text classification model for this category reaches the set threshold, the result will be output directly; if the classification accuracy value of the text classification model for this category does not reach the set threshold threshold, go to step 4;

Step 4, input each preprocessed text into the LDA topic model, use the LDA topic model to calculate the weight value of each topic in the K topics corresponding to the text, and select the one with the largest weight value topic, and add the first Y words in the topic related words under the topic obtained after the LDA topic model training in advance to the original feature word set of the text as the expanded feature word set, and use the text again The classification model calculates the probability values that the text may belong to each of the predetermined A categories, and selects the category with the highest probability value as the final classification category of the text. Specifically, 10 to 20 words can be selected. For example, the first 15 words in the subject-related words are added to the original feature word set of the text as the expanded feature word set; even if the newly added words are repeated with the original feature words does not matter.

The main calculation formula of the text classification model is:

where P(c _j |x ₁ , x ₂ ,...,x _n ) represents the probability that the text belongs to category cj when the feature words (x1, x2,..., xn) appear at the same time; where P(c _j ) represents the training In the text set, the ratio of the text belonging to category c _j to the total number, P(x ₁ , x ₂ ,..., x _n |c _j ) means that if the text to be classified belongs to category c _j , then the feature word set of this text is the probability of (x ₁ , x ₂ , ..., x _n ), and P(c ₁ , c ₂ , ..., c _n ) represents the joint probability of all classes given.

Obviously, for all given categories, the denominator P(c ₁ , c ₂ ,..., c _n ) is a constant, and the classification result of the model is the category with the highest probability in formula (1). Solving the maximum probability of formula (6) The value can be transformed into the maximum value for solving the following formula

According to the Bayesian hypothesis, the text feature vector attributes x ₁ , x ₂ , ..., x _n are independent and identically distributed, and their joint probability distribution is equal to the product of the probability distribution of each attribute feature, namely:

P(x ₁ , x ₂ ,..., x _n |c _j )=Π _i P(x _i |c _j ) (36)

So formula (7) becomes:

That is, the classification function used for classification is sought.

The probability values P(c _j ) and P( _xi |c _j ) in the classification function are still unknown, therefore, in order to calculate the maximum value of the classification function, the prior probability values in (9) are estimated as follows:

Among them, N(C=c _j ) represents the number of samples belonging to category c _j in the training text; N represents the total number of training samples.

Among them, N(X _i = _xi , C=c _j ) represents the number of training samples containing attribute x _i in category c _j ; N(C=c _j ) represents the number of training samples in category c _j ; M represents training samples The number of keywords in the collection after removing useless words.

LDA is a statistical topic model for modeling discrete data sets proposed by Blei et al. in 2003. It is a three-layer Bayesian generative model of "document-topic-word". The original model only introduced a hyperparameter to the "document-topic" probability distribution to make it obey the Dirichlet distribution, and then Griffiths et al. introduced a hyperparameter to the "topic-word" probability distribution to make it obey the Dirichlet distribution. The LDA model is shown in Figure 2. Among them: N is the number of words in this document, M is the number of documents in the document set, K is the number of topics, is the probability distribution of topic-word, θ is the probability distribution of document-topic, Z is the hidden variable representing the topic, W is the word, α is the hyperparameter of θ, β is super parameter.

The LDA topic model regards a document as a set of words, there is no order between words, and a document can contain multiple topics, each word in the document is generated by a topic, the same Words can also belong to different topics, so the LDA topic model is a typical bag-of-words model.

The key to training the LDA model is the inference of the hidden variable distribution, that is, to obtain the hidden text-topic distribution θ and topic-word distribution of the target text If the model parameters α, β are given, the joint distribution of the random variables θ, z and w of the text d is:

Since there are multiple hidden variables in the above formula at the same time, directly calculate θ, It is impossible, so it is necessary to estimate and infer the parameters. The current common parameter estimation algorithms include Expectation Maximization (EM), variational Bayesian inference and Gibbs sampling. In this paper, Gibbs sampling is used to infer model parameters. Griffiths pointed out that Gibbs sampling is superior to variational Bayesian inference and EM algorithm in terms of Perplexity value and training speed. Due to the local maximization of the likelihood function of the EM algorithm, the model often finds a local optimal solution, while the model obtained by variational Bayesian inference deviates from the real situation. Gibbs sampling can quickly and effectively extract data from large-scale data sets. Extracting topic information has become the most popular LDA model extraction algorithm.

MCMC is a set of approximate iterative methods for extracting sample values from complex probability distributions. Gibbs sampling is a simple implementation form of MCMC. sample. During the training process, the algorithm only samples the subject variable _zi , and its conditional probability calculation formula is as follows:

Among them, the meaning on the left side of the equation is: the probability that the current word w _i belongs to the topic k under the condition that the current word w i knows the topics to which other words belong; the n _i -1 on the right side of the equation is the i-th word under the k-th topic number minus 1; n _k -1 is the number of topics k in the document minus 1; the first multiplier is the probability of the word w _i under topic k; the second multiplier is the kth topic The probability that the topic is in the document.

The specific steps of Gibbs sampling are as follows:

1) Initialization, randomly assign topics to each word w _i , z _i is the topic of the word, initialize z _i to a random integer between 1 and K, i is from 1 to N, and N is the characteristic word token of the text set , this is the initial state of the Markov chain;

2) i loops from 1 to N, calculates the probability that the current word w _i belongs to each topic according to formula (2), and resamples the topic for word w _i according to this probability, and obtains the next state of the Markov chain;

After iterating step 2) enough times, it is considered that the Markov chain has reached a steady state. So far, each word in this document has a specific topic; for each document, the text-topic distribution θ and topic-word distribution The value of can be estimated by the following formula:

in, Indicates the number of times feature word w is assigned to topic k, Indicates the number of feature words assigned to topic k, Indicates the number of feature words assigned to topic k in text d, Indicates the number of all feature words assigned topics in text d.

The classification accuracy rate as the credibility index of the text classification model is calculated by probability. The specific formula is as follows:

Among them, i represents the category, N _i represents the number of times the classifier correctly predicts the i category, and M _i represents the total number of times the classifier predicts the i category.

The precision rate P, the recall rate R and the comprehensive evaluation index F1 _of the two can be used as the final evaluation index. The precision rate P measures the proportion of the test samples that are correctly judged as the category to the test samples that are judged as the category. , the recall rate R measures the correct determination of the proportion of test samples of this category to all test samples of this category. Taking a certain category C _i as an example, n ⁺⁺ indicates the number of samples that are correctly judged to belong to category C _i , n ^+- indicates the number of samples that do not belong to but are judged to be in category C _i , n ^-+ indicates that they belong to but are judged not to be The number of samples belonging to class C _i . For category C _i , the values _of recall rate R, precision rate P and comprehensive index F1 are:

The inventor has conducted three sets of experiments: Experiment 1, classifier performance test based on the original feature set; Experiment 2, classifier performance test based on the expanded feature set; Experiment 3, based on the feature set after selective feature expansion. Classifier performance testing, where the confidence threshold is set to 0.8. Table 2 shows the recall and precision of the three experiments in some subjects:

Table 2 Recall rate and precision rate of some subjects

It can be seen from Table 2 that when the experiment is performed based on the original feature set, the history recall rate is higher, but the precision rate is lower, indicating that there are more data that do not belong to the subject of history and are classified as history by the classifier. At the same time, it is found that The recall rate of the history of science and technology subject is low, indicating that a lot of data belonging to this subject has been classified into other subjects. Since the themes of these two subjects are very similar, it is likely that the classifier has more data belonging to the history of science and technology. Classified as History. A similar situation also occurs in the disciplines of geological resources and geological engineering and geology. Based on the expanded feature set, the above problems have been improved, but it has had an impact on the previously highly recognized subjects. On the one hand, the selective feature expansion avoids the impact on the subjects with high recognition degree, and on the other hand, it improves the subjects with low recognition degree due to insufficient information to a certain extent.

According to the above experimental results, the average recall rate, average precision rate and average F1 value _of the three experiments can be calculated. The result is as follows:

Table 3 Experimental comparison

It can be seen from Table 3 that in the face of complex classification scenarios, the method based on selective feature expansion in the present invention has better adaptability than the method based on the original feature set or the extended feature set, and the average recall rate , the average precision rate and the average F ₁ value are obviously higher than other schemes, and can achieve better practical results.

Figure 6 is a schematic diagram of the recall rate of three experiments on some subjects; Figure 7 is a schematic diagram of the precision rate of three experiments on some subjects.

With the advent of the big data era, resource classification is facing more and more challenges. Different application scenarios require different classification technologies, and no one technology is suitable for all classification tasks. The method based on selective feature expansion proposed by the present invention is suitable for complex application scenarios, selectively adding topic information to data with less information, while avoiding adding noise to data with sufficient information, and the method of the present invention has universal adaptability .

3. Recommendation of academic resources

The process of the present invention recommending corresponding academic resources to users includes a cold-start recommendation stage and a secondary recommendation stage. In the cold-start recommendation stage, users are recommended high-quality resources that meet their interest subjects based on the subjects of interest. The value calculation model calculates and compares academic resources with high resource quality values. The resource quality value is the arithmetic mean or weighted mean of resource authority, resource community popularity and resource freshness; in the second recommendation stage, the user interest The model and resource model are modeled, and the similarity between the user's interest and the resource model is calculated, and then the recommendation degree is calculated based on the resource quality value, and finally the Top-N recommendation of academic resources is made for the user according to the recommendation degree.

1. Recommended algorithm in the cold start phase:

Table 4 Attributes and measurement standards of the five categories of resources

Quality academic resources attract and retain new users. In the cold start stage, this paper intends to recommend high-quality resources that match the disciplines of interest to users. High-quality resources refer to academic resources with high quality value. The measurement criteria of quality value mainly include attributes such as authority, community popularity, and novelty. The attributes and measurement standards of the five categories of resources are shown in Table 4.

The formula for calculating the authoritative degree of a paper is as follows:

Level is the quantified score of the publication level of the paper. In this paper, the publication level is divided into 5 levels, and the scores are 1, 0.8, 0.6, 0.4 and 0.2 points. Top journals or conferences such as Nature and Science get 1 point, second-level ones like ACM Transaction get 0.8 points, and the lowest level ones get 0.2 points. The calculation formula of Cite is as follows:

Cite＝Cites/maxCite. (2)

Cite is the quantified result of the paper's cited quantity, Cites is the paper's cited quantity, and maxCite is the maximum cited quantity in the paper's source database.

The calculation of the authority of the other four types of resources is similar to that of papers, but the quantification method is different.

The formula for calculating the Popularity of the paper community is as follows:

Popularity=readTimes/maxReadTimes. (3)

readTimes is the number of times the paper has been read, and maxReadTimes is the maximum number of times the paper has been read in the source database.

The calculation method of the recentness of all resources is the same, and the formula is as follows:

year and month are the publication year and month of the resource, respectively. minYear, minMonth, maxYear, and maxMonth are the earliest and latest published year and month of all resources in the source database of this type of resource.

The calculation method of the quality value of the paper is as follows:

2. The algorithm of the second recommendation stage:

In this stage, the recommendation method that integrates user behavior and resource content is adopted to model the user interest model and resource model respectively, calculate the similarity between the two, and then calculate the recommendation degree based on the resource quality value, and finally make recommendations based on the recommendation degree.

The academic resource model is represented as follows:

M _r = {T _r , K _r , C _t , L _r } (6)

Among them, T _r is the subject distribution vector of academic resources, which is the probability value of the distribution of academic resources in 75 subjects, obtained from the Bayesian polynomial model.

K _r = {(k _r1 , ω _r1 ), (k _r2 , ω _r2 ),..., (k _rm , ω _rm )}, m is the number of keywords, k _ri (1≤i≤m) means For the i-th keyword of a single academic resource, ω _ri is the weight of the keyword k _ri , obtained through the improved tf-idf algorithm, and the calculation formula is as follows:

w(i, r) represents the weight of the i-th keyword in document r, tf(i, r) represents the frequency of the i-th keyword appearing in document r, Z represents the total number of documents in the document set, and L represents The number of documents containing keyword i.

L _r is the LDALDA potential topic distribution vector, L _r = {l _r1 , l _r2 , l _r3 ..., l _rN1 }, N1 is the number of potential topics.

C _t is the resource type, and the value of t can be 1, 2, 3, 4, 5, that is, five types of academic resources: academic papers, academic patents, academic news, academic conferences and academic books.

According to the behavior characteristics of users using mobile software, the user's operation behavior on an academic resource is divided into opening, reading, star rating, sharing, and collection. Star rating is an explicit behavior, and the others are implicit behaviors. Explicit behaviors can clearly reflect the degree of user interest and preference, such as star ratings. The higher the score, the more the user likes the resource; although the implicit behavior cannot clearly reflect the user's interest preference, the amount of information and information value it contains is often more than the explicit behavior. Feedback is more and higher.

The user interest model is mainly based on user background and browsed academic resources. According to different browsing behaviors of users, combined with the academic resource model, a user interest model can be constructed, and this model will be dynamically adjusted as the user interest changes. The user interest model is expressed as follows:

M _u ＝{T _u ，K _u ，C _t ，L _u } (8)

Among them, T _u is the subject distribution vector T _r of a certain type of academic resources browsed by the user within a period of time, and the user's subject preference distribution vector is formed after the user's behavior, that is

Among them, sum is the total number of academic resources that the user has acted on, and s _j is the "behavior coefficient" after the user has acted on the academic resource j. The larger the value, the more the user likes the resource. T _jr represents the subject distribution vector of the jth resource. The calculation of s _j comprehensively considers behaviors such as opening, reading, evaluating, collecting and sharing, and can accurately reflect the user's preference for resources.

K _u = {(k _u1 , ω _u1 ), (k _u2 , ω _u2 ),..., (k _uN2 , ω _uN2 )} is the user keyword preference distribution vector, N ₂ is the number of keywords, k _ui (1≤i≤N ₂ ) represents the i-th user's preferred keyword, ω _ui is the weight of keyword k _ui , and is calculated through the "keyword distribution vector" K _r of all academic resources that user u has performed in a period of time get.

K _jr ′=s _j *K _jr (10)

The new keyword distribution vector of each academic resource can be calculated according to formula 10, and TOP-N2 of the new keyword distribution vector of all resources is selected as the user's keyword preference distribution vector K _u .

L _u is the user's LDA latent topic preference distribution vector, which is calculated from the LDA latent topic distribution vector of academic resources L _r = {l _r1 , l _r2 , l _r3 ..., l _rN1 }, the method is the same as T _u .

Calculation of the behavior coefficient: s represents the behavior coefficient, T is the reading time threshold, and δ is an adjustment parameter. The reading time threshold is added to prevent accidental clicks, so this value is very small. If the time spent by the user reading resource j is less than the threshold T, it is considered that the user clicked by mistake, s=0. Under the condition that the user is willing to spend a long time reading, that is, the reading time is greater than or equal to T, if the user makes an evaluation and the evaluation value is greater than the mean mean of all his previous evaluations, it is considered that he likes j, and s is increased by δ. If the user collects or shares j, it means that the user likes j very much, and increases s by δ. The present invention considers that reading, evaluating, collecting, and sharing reflect user's interests and preferences from shallow to deep. The value of s mainly depends on the initial value and the adjustment parameter δ. We want to map all user behaviors to a value from 0 to 2, so the initial value is 1, and the adjustment parameter δ=0.333333.

Calculation of similarity between academic resource model and user interest model:

The Academic Resource Model says:

M _r = {T _r , K _r , C _t , L _r } (12)

The user interest model represents:

M _u = {T _u , K _u , C _t , L _u } (13)

The similarity between user subject preference distribution vector T _u and academic resource subject distribution vector T _r is calculated by cosine similarity, namely:

The similarity between the user’s LDA latent topic distribution vector L _u and the academic resource’s LDA latent topic distribution vector L _r is calculated by cosine similarity, namely:

The similarity calculation between user keyword preference distribution vector K _u and academic resource keyword distribution vector K _r is calculated by Jaccard Similarity:

Then the similarity between the user interest model and the academic resource model is:

Among them, σ+ρ+τ=1, and the specific weight distribution is obtained from experimental training.

In order to recommend high-quality resources that users are interested in, the concept of Recommendation_degree is introduced. The higher the recommendation degree of a certain academic resource, the more suitable the resource is for the user's interest preference, and the higher the quality of the resource. The recommendation calculation formula is as follows:

Recommendation_degree=λ ₁ Sim(M _u , M _n )+λ ₂ Quality(λ ₁ +λ ₂ =1) (18)

The second recommendation stage is to make Top-N recommendations based on the recommendation degree of academic resources.

The entire recommendation process is shown in Figure 10. From Figure 2, we can see that the overall recommendation process of the system includes three parts: the construction of the resource model, the recommendation in the cold start phase, and the secondary recommendation process. Their specific steps are as follows:

The construction process of the resource model:

1) Obtain five types of academic resource data through web crawler and data interface technology;

2) Analyze and extract the relevant information of each academic resource, and insert it into the resource library;

3) Preprocessing each piece of data in the resource library, including word segmentation and removing stop words;

4) Calculate the subject distribution, keyword distribution and LDA potential topic distribution of each resource through the three types of models that have been trained. The three types of models are Bayesian polynomial model, VSM and LDA model;

5) According to the subject distribution vector, the subject category of the resource is obtained, and the subject category of the resource is the first three subjects with higher probability in the subject distribution vector;

6) Calculate the quality value of each resource;

7) Insert the subject distribution vector, keyword distribution vector, LDA latent topic distribution vector, subject category and quality value into the resource library.

Recommended procedure for the cold start phase:

1) Select academic resources that meet the user's interest disciplines

2) Recommend high-quality resources according to the quality value of academic resources.

The recommendation process in the second recommendation stage:

1) Obtain the user's browsing records and calculate the "behavior coefficient";

2) Construct user interest model;

3) Calculate the similarity between the resource model and the user interest model;

4) Calculate the recommendation degree according to the similarity and quality value;

5) Perform Top-N recommendation according to the recommendation degree of resources.

In order to facilitate subsequent calculations, we built a resource model in advance. When users use the system for the first time, we adopt the recommendation strategy in the cold start phase to recommend academic resources for them; when the user's behavior data reaches a certain amount, we adopt the second recommendation strategy Recommend academic resources for it.

The present invention mainly proposes corresponding recommendation strategies based on the continuous accumulation and changes of academic resources and user data. In the cold start stage, users are recommended high-quality resources that meet their interests; in the secondary recommendation stage, various academic resources are modeled from four dimensions: resource type, subject distribution, keyword distribution, and LDA potential topic distribution. Interest preference modeling, and finally Top-N recommendation based on resource recommendation.

The experimental results show that the academic resource recommendation strategy adopted by the present invention can fully cater to the subjects of interest of users, and achieve obvious effects in improving the CTR of resources; in the secondary recommendation stage, it can be seen from the experimental results that the modeling used in the present invention The recommendation strategy under the method is significantly higher than the recommendation strategy under the current two common resource modeling methods in terms of Precision.

Claims (10)

1. An academic resource recommendation service system, the academic resource is various electronic texts published on the Internet, the academic resource recommendation service system includes a web crawler, a text classification model, a local academic resource database to be recommended, by The web crawler crawls academic resources on the Internet, which is characterized in that the text classification model is used to classify according to predetermined A categories and stored in a local academic resource database to be recommended, and an open API of the academic resource database is provided for display and resource recommendation module calls. The academic resource recommendation service system also includes an academic resource model, a resource quality value calculation model, and a user interest model, and a tracking software module is embedded in the user's terminal to track and record the user's online browsing behavior; Behavioral data, calculate the degree of attention of users with different identities to various types of academic resources, model academic resources from four dimensions of resource type, subject distribution, keyword distribution and LDA potential topic distribution, combined with users' interest subjects and historical browsing Behavioral data, model the user interest model, calculate the similarity between the academic resource model and the user interest model, and then calculate the recommendation degree based on the resource quality value, and finally make Top-N recommendations for academic resources for users based on the recommendation degree. 2. the academic resource recommendation service system as claimed in claim 1, is characterized in that, described web crawler is theme crawler, and configures LDA subject model, and described LDA subject model is a three-layer of "document-subject-word" The Bayesian generation model configures a corpus for the LDA topic model in advance. The corpus includes training corpus, and the training corpus is used to set the topic number K to allow the LDA topic model to train. The word gathering function during the training of the LDA topic model is used during training. After the corpus is trained by the LDA topic model, it is obtained according to the set topic number K to be aggregated into K topic related word sets, that is, K topic documents crawled by the topic crawler this time; the topic crawler further includes topic A determination module, a similarity calculation module, and a URL prioritization module; the subject crawlers are a plurality of distributed crawlers distributed according to the number of academic topics, each distributed crawler corresponds to an academic topic, and each distributed crawler obtains multiple academic topics at the same time. Subject academic resources; during each crawling process of the subject crawler, the subject determination module of the subject crawler determines the target subject and its subject document, uses the subject document to guide the calculation of subject similarity, and the similarity calculation module compares the crawled pages Each anchor text is combined with the content of the page to calculate and judge the topic similarity. Eliminate the hyperlinks whose topic similarity combined with the anchor text and the page is less than the set threshold, and select the hyperlinks whose topic similarity combined with the anchor text and the page is greater than the set threshold. URL, the theme crawler maintains a URL queue of unvisited webpages pointed to by the hyperlinks of the visited webpages. The URL queue is arranged in descending order according to the similarity, and the theme crawler continuously visits the webpages of each URL according to the order of the URL queue , crawl the corresponding academic resources, and continuously store the crawled academic resources into the database after classifying and labeling them. For the theme documents crawled this time, until the unvisited queue URL is empty; The academic resources are used as new corpus for LDA topic model training; and the crawling process of the topic crawler is repeated continuously, so that the topic related words collected in each topic document can be continuously supplemented and updated, and the crawled academic resources can be continuously supplemented and updated to a human-made degree of recognition. 3. The academic resource recommendation service system according to claim 2, wherein the corpus also includes a verification corpus with clear categories, which is used to use the verification corpus in advance to classify the text classification model according to the predetermined number A of categories Verification, to obtain the classification accuracy rate of the text classification model for each category in the A categories, as the classification credibility index of the text classification model for each category in the A categories; the accuracy rate is the text classification model The ratio of correctly classified corpora among all verification corpora assigned to a certain category, and a preset classification accuracy threshold. 4. The academic resource recommendation service system as claimed in claim 3, wherein all subjects are divided into 75 subject categories, that is, the category number A is 75 categories, and the number of topics is set when utilizing the LDA topic model training. K is 100, and the preset classification accuracy threshold is 80% when the text classification model performs classification verification. 5. A method for providing relevant users with a resource recommendation service system for academic resource recommendation services, the academic resources are various electronic texts published on the Internet, including using web crawlers to crawl academic resources on the Internet, characterized in that , use the text classification model to classify the crawled academic resources according to predetermined A categories and store them to form an academic resource database, provide an open API of the academic resource database for display and resource recommendation module calls, use the academic resource model, resource quality value Calculation model, user interest model, and tracking software modules are implanted in the user's terminal to track and record the user's online browsing behavior; the process of recommending the corresponding academic resources to the user includes the cold start recommendation stage and the second recommendation stage, cold start In the recommendation stage, high-quality resources are recommended for users based on the subjects of interest. The high-quality resources are academic resources with high resource quality values calculated by the resource quality value calculation model. The resource quality values are resource authority, resource Arithmetic or weighted average of community popularity and resource freshness; in the second recommendation stage, model the user interest model and resource model respectively, calculate the similarity between the user interest model and the resource model, and then combine the resource quality value Calculate the recommendation degree, and finally recommend Top-N academic resources for users according to the recommendation degree. 6. The method according to claim 5, wherein the calculation of the resource quality value Quality comprises that the calculation formula of the Authority of the resource is as follows: $\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; L</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; v</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; C</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$ Among them, Level is the quantified score of resource publishing publication level, and the publication level is divided into 5 levels, and the scores are 1, 0.8, 0.6, 0.4 and 0.2 points. Top magazines or conferences such as Nature and Science get 1 point, second-level ones like ACM Transaction get 0.8 points, and the lowest level ones get 0.2 points; the calculation formula of Cite is as follows: Cite=Cites/maxCite (2) Cite is the quantitative result of resource citations, Cites is the citation of resources, and maxCite is the largest citation in the resource source database; The formula for calculating the Popularity of the resource community is as follows: Popularity=readTimes/maxReadTimes (3) readTimes is the number of readings of the paper, and maxReadTimes is the maximum number of readings in the resource source database; The calculation method of the recentness of resources is the same, and the formula is as follows: $\mathrm{<span\; class="notranslate">\; Re</span>}\mathrm{<span\; class="notranslate">\; c</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; min</span>}\mathrm{<span\; class="notranslate">\; Y</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; min</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; max</span>}\mathrm{<span\; class="notranslate">\; Y</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; min</span>}\mathrm{<span\; class="notranslate">\; Y</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; max</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; min</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$ year and month are the publication year and month of the resource respectively; minYear, minMonth, maxYear and maxMonth are the earliest and latest publication year and month of all resources in the source database of this type of resource; The resource quality value Quality is calculated as follows: $\mathrm{<span\; class="notranslate">\; Q</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; P</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 3</span>}}\mathrm{<span\; class="notranslate">\; Re</span>}\mathrm{<span\; class="notranslate">\; c</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$ 7. The method according to claim 5, wherein the academic resource model is represented as follows: M _r ＝{T _r ,K _r ,C _t ,L _r } (6) Among them, T _r is the subject distribution vector of academic resources, which is the probability value of the academic resources distributed in A subject category, which is obtained by the Bayesian polynomial model; K _r ＝{(k _r1 ,ω _r1 ),(k _r2 ,ω _r2 ),…,(k _rm ,ω _rm )}, m is the number of keywords, k _ri (1≤i≤m) means a single academic The i-th keyword of the resource, ω _ri is the weight of the keyword k _ri , which is obtained through the improved tf-idf algorithm, and the calculation formula is as follows: w(i,r) represents the weight of the i-th keyword in document r, tf(i,r) represents the frequency of the i-th keyword appearing in document r, Z represents the total number of documents in the document set, and L represents The number of documents containing keyword i; Lr is the potential topic distribution vector, L _r = {l _r1 , l _r2 , l _r3 ..., l _rN1 }, N1 is the number of potential topics; Ct is the resource type, and the value of t can be 1,2,3,4,5 are five major categories of academic resources: papers, patents, news, conferences and books; According to the behavior characteristics of users using mobile software, the user's operation behavior on an academic resource is divided into opening, reading, star rating, sharing and collection. The user interest model is based on the user's background and the academic resources that have been browsed. According to the user's different browsing Behavior, combined with the academic resource model, constructs a user interest model. The user interest model is expressed as follows: M _u ＝{T _u ,K _u ,C _t ,L _u } (8) Among them, T _u is the subject distribution vector of a certain type of academic resources browsed by the user within a certain period of time, and T _r is the user's subject preference distribution vector formed after the user's behavior, that is ${\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 9</span>}<span\; class="notranslate">\; )</span>$ Among them, sum is the total number of academic resources that the user has acted on, and s _j is the "behavior coefficient" after the user has acted on the academic resource j. The larger the value, the more the user likes the resource. T _jr represents the subject distribution vector of the jth resource; the calculation of s _j comprehensively considers the behaviors of opening, reading, evaluating, collecting and sharing, and can accurately reflect the user's preference for resources. K _u ＝{(k _u1 ,ω _u1 ),(k _u2 ,ω _u2 ),…,(k _uN2 ,ω _uN2 )} is the user preference keyword distribution, N ₂ is the number of keywords, k _ui (1≤ i≤N ₂ ) represents the i-th user's preference keyword, ω _ui is the weight of keyword k _ui , which is obtained by calculating the "keyword distribution vector" K _r of all academic resources that user u has acted in a period of time; K′ _jr =s _j *K _jr (10) The new keyword distribution vector of each academic resource can be calculated according to formula 10, and then TOP-N2 of the new keyword distribution vector of all resources is selected as the user keyword preference distribution vector K _u ; L _u is the user's LDA latent topic preference distribution vector, which is calculated from the LDA latent topic distribution vector of academic resources L _r ={l _r1 ,l _r2 ,l _r3 ...,l _rN1 }, the method is the same as T _u : ${\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 11</span>}<span\; class="notranslate">\; )</span>$ The similarity between user interests and resource models is calculated as follows: The Academic Resource Model says: M _r ＝{T _r ,K _r ,C _t ,L _r } (12) The user interest model represents: M _u ＝{T _u ,K _u ,C _t ,L _u } (13) The similarity between user subject preference distribution vector T _u and academic resource subject distribution vector T _r is calculated by cosine similarity, namely: $\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 14</span>}<span\; class="notranslate">\; )</span>$ The similarity between the user LDA latent topic preference distribution vector L _u and the academic resource LDA latent topic distribution vector L _r is calculated by cosine similarity, namely: $\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 15</span>}<span\; class="notranslate">\; )</span>$ The similarity calculation between the user keyword preference distribution vector K _u and the academic resource keyword distribution vector K _r enters the calculation through JaccardSimilarity: $\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span>\frac{{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; \&cap;</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}{{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; \&cup;</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; )</span>$ Then the similarity between the user interest model and the academic resource model is: $\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&tau;</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}{\sqrt{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&rho;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&tau;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 17</span>}<span\; class="notranslate">\; )</span>$ Among them, σ+ρ+τ=1, and the specific weight distribution is obtained from experimental training. Introduce the concept of Recommendation_degree. The higher the recommendation degree of an academic resource, the more the resource is in line with the user's interests and preferences, and the resource is of higher quality. The calculation formula of the recommendation degree is as follows: Recommendation_degree＝λ ₁ Sim(M _u ,M _n )+λ ₂ Quality(λ ₁ +λ ₂ =1) (18) The second recommendation stage is to make Top-N recommendations based on the recommendation degree of academic resources. 8. The method according to claim 5, wherein the web crawler is a topic crawler, and configures an LDA topic model, and the LDA topic model is a three-layer Bayesian generation of "document-topic-word" Model, configure a corpus for described LDA topic model in advance, include training corpus in the corpus, let LDA topic model train by setting topic number K with training corpus, utilize the polyword function during LDA topic model training in training corpus through LDA topic After the model is trained, it is obtained by setting the number of topics K to gather K topic-associated word sets respectively, that is, to obtain K topic documents crawled by the topic crawler this time; the topic crawler further includes a topic determination module, a similar Degree calculation module, URL prioritization module; the theme crawler is a plurality of distributed crawlers distributed according to the number of academic topics, each distributed crawler corresponds to an academic theme, and each distributed crawler obtains academic resources of multiple academic topics at the same time ; During each crawling process of the theme crawler, the theme determination module of the theme crawler determines the target theme and its theme document, uses the theme document to guide the calculation of theme similarity, and the similarity calculation module calculates each anchor text on the crawled page And combine the content of the page to calculate and judge the topic similarity, remove the hyperlinks whose anchor text combined with the topic similarity of the page is less than the set threshold, select the URL whose anchor text combined with the topic similarity of the page is greater than the set threshold, and use the theme The crawler maintains a URL queue of unvisited web pages pointed to by the hyperlinks of the visited web pages. The URL queue is arranged in descending order according to the similarity. Academic resources, and continuously classify and store the crawled academic resources into the database, for the subject documents crawled this time, until the unvisited queue URL is empty; use the academic resources crawled by the subject crawler each time as The new corpus used for LDA topic model training; and the crawling process of the topic crawler is repeated continuously, so that the topic associated words collected in each topic document can be continuously supplemented and updated, and the crawled academic resources can be continuously supplemented and updated to a human-recognized level. 9. The method according to claim 5, characterized in that, the corpus also includes clear verification corpus, which is used to use the verification corpus in advance to allow the text classification model to perform classification verification according to the predetermined category number A, so as to obtain The classification accuracy rate of the text classification model for each of the A categories is used as the classification credibility index of the text classification model for each of the A categories; the accuracy rate is assigned to a certain category by the text classification model The ratio of the correctly classified corpus in all verification corpora of the category, and the preset classification accuracy threshold; when using the text classification model to classify each text to be classified, it specifically includes the following steps: Step 1. Preprocess each text to be classified. Preprocessing includes word segmentation, removing stop words, and retaining proper nouns. Calculate the feature weights and feature weight values of all words in the text after preprocessing Proportional to the number of times it appears in the text, and inversely proportional to the number of times it appears in the training corpus, the calculated word set is arranged in descending order according to its characteristic weight value, and the original word set of each text to be classified is extracted. The front part is used as its feature word set; Step 2. Using the text classification model, select the original feature word set of each text to be classified to calculate the probability value that the text may belong to each of the predetermined A categories, and select the category with the highest probability value as the text classification category; Step 3. Judging the text classification results of step 2. If the classification accuracy value of the text classification model for this category reaches the set threshold, the result will be output directly; if the classification accuracy value of the text classification model for this category does not reach the set threshold threshold, go to step 4; Step 4, input each preprocessed text into the LDA topic model, use the LDA topic model to calculate the weight value of each topic in the K topics corresponding to the text, and select the one with the largest weight value topic, and add the first Y words in the topic related words under the topic obtained after the LDA topic model training in advance to the original feature word set of the text as the expanded feature word set, and use the text again The classification model calculates the probability values that the text may belong to each of the predetermined A categories, and selects the category with the highest probability value as the final classification category of the text. 10. The method according to claim 9, wherein the main calculation formula of the text classification model is: $\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; ...</span><span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; ...</span><span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; ...</span><span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 19</span>}<span\; class="notranslate">\; )</span>$ Wherein P(c _j |x ₁ ,x ₂ ,…,x _n ) represents the probability that the text belongs to category C _j when feature words (x1,x2,…,xn) appear at the same time; where P(c _j ) represents the training text Concentration, the ratio of texts belonging to category c _j to the total, P(x ₁ ,x ₂ ,…,x _n |c _j ) means that if the text to be classified belongs to category c _j , then the feature word set of this text is (x ₁ ,x ₂ ,…,x _n ), P(c ₁ ,c ₂ ,…,c _n ) represents the joint probability of all given categories.