CN101968819B - Audio and video intelligent cataloging information acquisition method facing wide area network - Google Patents

Abstract

The invention relates to a method for obtaining audio and video intelligent cataloging information for a wide area network, and belongs to the field of computer application. The invention is characterized in that a weighting algorithm based on the position factor of keyword feature items is proposed, and different weighting factors are assigned to feature items at different positions in the document, so as to more accurately calculate the subject similarity of web page content; the three factors of web page content similarity, URL directory hierarchy information of hyperlinks, and anchor text information of hyperlinks are comprehensively utilized to optimize the selection of links with higher subject similarity. For the searched subject pages, cataloging information is automatically extracted using an information extraction method based on ontology and HTML. The extracted cataloging information is standardized using an improved semantic similarity calculation method. The invention can intelligently and automatically provide cataloging item information to catalogers, reduce manual labor, improve cataloging efficiency, meet the different needs of professional and non-professional catalogers, and adapt to wide area network environments.

Description

Method for Acquiring Audio and Video Intelligent Cataloging Information for Wide Area Network

technical field

The invention belongs to the field of computer application technology, relates to the cataloging of digital audio and video materials in the wide area network environment, provides an efficient, automatic and intelligent cataloging method for professional and non-professional content creators and catalogers in the wide area network environment, and can solve the problems faced by the current cataloging system. Problems such as local area network, strong professionalism, heavy workload, repetitive labor, and low degree of automation can significantly improve cataloging efficiency and reduce manual labor.

Background technique

The cataloging quality and speed of digital audiovisual materials directly affect resource management, cost-effectiveness, and resource reuse.

The large-scale video cataloging work in China basically started in 2002. Since the establishment of a media asset management platform requires considerable technical and material conditions, there are only a few units in China that have established a large-scale media asset management system and cataloged large-scale production. Among them, the largest and highest annual output is CCTV audio-visual materials Museum, the cataloging software it uses is mainly produced by two companies, Sobei and Zhongke Dayang.

In the existing literature, some studies have been done on the cataloging of libraries, news and other media forms and cataloging automation technology. For example, the "News Video Cataloging Method and System" developed by the Institute of Automation, Chinese Academy of The host and audio mute point information automatically cataloged the news video, and this method only analyzes, separates and matches some local content of the news video itself to determine the catalog information. There are also other researches on automatic frame extraction and shot segmentation. For example, the "Internet Media Intelligent Cataloging System" developed by Beijing Nufront Network Technology Co., Ltd., the "Automatic Cataloging System" can perform video structure on media files. Humanization analysis, face analysis, subtitle analysis, station logo analysis, sound analysis, so as to form corresponding catalog information. These methods aim at the video content itself, the algorithm is relatively complex, the adaptability is very weak, the realization accuracy is low, and the quality of the video content itself is greatly affected.

At present, there are still the following main problems in the cataloging of digital audio and video materials: First, there are many cataloging items, which require catalogers to enter manually one by one according to their professional knowledge, which is a heavy workload and prone to errors; second, the degree of intelligence and automation is low, and they cannot be automatically obtained. Cataloging information is inefficient; third, the cataloging environment is generally a local area network, which limits the sources and channels for automatic acquisition of cataloging information; fourth, the professional level of cataloging personnel is high, and with the gradual popularization of audio and video collection equipment, content production It began to be personalized and publicized, and the cataloging work gradually tended to be completed by non-professionals.

Contents of the invention

In order to overcome the above-mentioned problems existing in the existing cataloging system, the present invention proposes a wide area network-oriented audio and video intelligent cataloging information extraction method, which can intelligently and automatically provide cataloging item information for catalogers, reduce manual labor, and improve cataloging efficiency. Moreover, it can adapt to the different needs of professional and non-professional catalogers, and adapt to the wide area network environment.

The present invention is characterized in that: a weighting algorithm based on the location factors of keyword feature items is proposed, and different weighting factors are given to feature items in different positions in the document, so as to more accurately calculate the subject similarity of webpage content; comprehensively utilize webpage content similarity degree, hyperlink URL directory level information, and hyperlink anchor text information to optimize and select links with higher topic similarity. For the searched subject pages, the catalog information is automatically extracted using ontology and HTML-based information extraction methods. The extracted catalog information is normalized by using an improved semantic similarity calculation method.

The overall process of the present invention is shown in Figure 1. The invention provides the user with a friendly cataloging interface. After the interface is opened, the audio and video files to be cataloged are selected to be played, and then recorded in the cataloging input box. In the process of description, first enter the title proper and keywords as input values, and then follow the steps below on the computer to automatically extract the catalog information:

1. Theme crawler searches the web

Aiming at the fact that the current traditional search engines are difficult to meet the requirements for specific catalog information retrieval, the invention uses a vertical search engine to search for network resources related to specific topics. The title proper and keywords entered by the cataloger are used as the subject collection of the crawler.

The process of the theme crawler search webpage designed by the present invention is as follows:

(1) Page document preprocessing

Obtain and parse the webpage corresponding to the initial seed URL, segment the title text and body text to form a set of keyword feature items, and match it with the topic set to obtain a feature item vector with the same dimension as the topic vector.

(2) Weight calculation of keyword feature items

The invention improves the TF weight algorithm in the traditional vector space model. The traditional TF weight algorithm only pays attention to the frequency of keyword feature items in a web page, but when browsing a web page, the importance of title text and text is obviously different, and the TF weight algorithm ignores the frequency of keyword feature items in a web page. The location factor leads to errors in the similarity between keyword vectors and topic vectors. The present invention proposes " based on the weight algorithm of keyword feature item position factor ", calculation method is as follows:

a) Define different positions where feature items appear, and assign different position weight factors to feature items at different positions.

The position where the keyword feature item appears is defined as three categories: the title (Title) tag, the title (H1-H6) tag, and other positions in the text. The importance of these three types of positions for feature items is in descending order.

Then the position weight factor PG is introduced to indicate the importance of feature items at different positions. The larger PG is, the more important the feature items at this position are. Define PG _i (i=1, 2, 3) as the weight factors corresponding to different positions of feature items, and i represents the above three types of positions. Since the importance of the three types of positions to the feature items is in descending order, it is stipulated that: PG _i >=PG _i+1 (1<=i<=2).

Define TF' _i (i=1, 2, 3) as the frequency of feature item t appearing in different positions.

b) Calculate the weight of feature items based on location factors.

According to the position of the feature item, calculate the weight DWeight(t) of a certain keyword feature item t in the page document D, the formula is:

$\mathrm{DWeight}\left(t\right)={\mathrm{\&Sigma;}}_{i=1}^3({{\mathrm{TF}}^{\&prime;}}_i\&times;{\mathrm{PG}}_i)$ (Formula 1)

(3) Calculation of page content topic similarity

According to the position-based weight of each keyword feature item, the similarity Sim(D) between the keyword feature item in the searched page document D and the topic Topic is calculated as follows:

$\mathrm{Sim}\left(\mathrm{D.}\right)=\frac{{\mathrm{\&Sigma;}}_{j=1}^z\mathrm{DWeight}\left(j\right)\&times;\mathrm{TopicWeight}\left(j\right)}{\sqrt{{\mathrm{\&Sigma;}}_{j=1}^z{\mathrm{DWeight}\left(j\right)}^2}\&times;\sqrt{{\mathrm{\&Sigma;}}_{j=1}^z{\mathrm{TopicWeight}\left(j\right)}^2}}$ (Formula 2)

In formula 2, the keyword feature item in the document D of the searched page is equal to the dimension of the topic Topic, denoted by z; DWeight(j) indicates the weight of the jth keyword feature item in document D, and TopicWeight(j) indicates The weight of the jth topic item in the topic Topic, 1≤j≤z.

According to the experimental analysis, the value range of the similarity threshold TH is 0.4-0.6, which can obtain the topic-related web pages more accurately. If the similarity value Sim(D)≥TH between the webpage document and the topic, it is determined that the webpage document is similar to the topic; otherwise, the document is not similar to the topic.

(4) Link similarity calculation

This step determines the search direction of the topic crawler. Through the topic search strategy designed by the invention, the URL links are judged on similarity, and URL priorities are sorted, so as to provide crawlers with optimal URL links and improve crawling efficiency.

The present invention comprehensively utilizes the following three aspects of information to calculate the theme similarity of candidate URLs: the content similarity of this webpage, the URL directory level information of hyperlinks in this webpage, and the anchor text information of hyperlinks in this webpage. The specific calculation steps are:

a) Calculate the subject similarity of the current page content, that is, use the calculation result of step (3);

b) For each link of the current page, judge whether the link and the current page are adjacent to each other based on URL directory information. If yes, predict the similarity of the target page according to the topic similarity of the current page content, if not, evaluate the similarity of the link by analyzing the anchor text. Calculated as follows:

(Formula 3)

in:

D - the current web page;

L—the hyperlink anchor text in the D web page;

C - the target page pointed to by L;

Sim(D)—the topic similarity of web page D;

Sim(L) - topic similarity of the anchor text of the link L;

Sim(C)——the estimated value of the topic similarity of L pointing to the target webpage C;

x——Influence coefficient, x=0~1, x is used to adjust the weight distribution of the topic similarity of the two factors of D and L, the larger the x, the more inclined the formula is to the anchor text, and the smaller the x is, the more inclined the parent Thematic similarity of web pages. According to experiments, x can be set to 0.7-0.8.

If the similarity value Sim(C)≥TH between the link and the topic, it is determined that the link is similar to the topic, otherwise it is not similar to the topic.

(5) Download the webpages similar to the subject to the database and build an index for the next step of cataloging information extraction.

2. Cataloging Information Extraction Based on Ontology and HTML

For the topic pages with high similarity searched in step 1, the catalog information extraction method based on ontology and HTML is used to extract the catalog information.

In the usual web information extraction, the structure of the webpage existing in the form of HTML is easy to change, and the content of the webpage lacks semantic description. The present invention aims at this problem, combines the characteristics of the information extraction technology based on the HTML structure and the information extraction technology based on the ontology, adopts the extraction principle based on the HTML structure when locating the information block where the extracted information is located, and uses the Based on ontology extraction principle, it solves the semantic problem of extraction item description.

The specific execution process is shown in Figure 2, and the steps are as follows:

(1) Construct ontology: construct multimedia content extraction ontology, the concept in the ontology is defined as the catalog entry of multimedia file content description information, the attribute is defined as the relationship between concepts, and the label attribute of concept defines the extraction and description in each extraction data source Polysemous words corresponding to the item.

(2) Parsing the web page: cleaning the HTML page, correcting the error information on the page, removing redundant information, converting it into an XHTML document, and then parsing the document into a DOM tree structure.

(3) Generate extraction rules: Usually, the extracted information from each Web extraction source is concentrated in a continuous information block. The system locates information based on the combination of tree path and text content, generates XPath paths, and generates extraction rules.

(4) Read extraction rules: read the extraction rules generated in step (3).

(5) Read the ontology: read the multimedia content to extract the ontology, and operate on the classes, attributes and instances in the ontology.

(6) Execute the extraction algorithm: take the extraction rules and ontology in steps (4) and (5) as input, and execute the extraction algorithm. The specific steps are: divide the specific information in the information block to be extracted in the DOM tree formed by HTML parsing into key-value; read the concept in the multimedia file and extract the label attribute value of the concept; if the key in the DOM tree is consistent with the ontology The tag attribute value of the concept in the ontology is corresponding, then the concept of the ontology and the corresponding value value are saved in the XML file, that is, all the data contained in the web page information block are extracted. The extracted specific information can be added to the extracted ontology as a concrete instance of the concept in the ontology to extend the ontology model.

(7) Information fusion storage: extract corresponding extraction results according to each web page data source, multiple data sources correspond to each extraction result file, and the information contained in each result file may be the same or different. The system compares and analyzes the information in each extraction result file, performs information fusion, and finally generates an extraction result file.

3. Normalization of cataloging information based on natural language

Firstly, the present invention initializes a standardized thesaurus of bibliographic items; then, according to the catalog information extracted in step 2, executes the semantic similarity calculation algorithm based on HowNet to generate standardized catalog bibliographic items.

The system reads in the catalog item information extracted in step 2, as well as HowNet thesaurus and sememe tree files, finds two matching words, and then calculates the semantic similarity of the two words. A Chinese word is composed of one or more meanings (concepts), and the similarity between two words is the maximum value of the semantic similarity of each concept. The similarity between two Chinese words is attributed to the relationship between two concepts. However, all concepts are finally represented by sememe, so the following steps will be followed, starting from the sememe similarity calculation, and gradually calculating the semantic similarity of two Chinese words.

1) Calculate the semantic similarity of the sememe

In the tree-like hierarchical system composed of sememes, assuming that the path distance between two sememes X and Y is dis (a positive integer), the semantic similarity Sim(X, Y) between these two sememes is expressed by Equation 4 calculates:

$\mathrm{Sim}(x,Y)=\frac{\mathrm{\&alpha;}}{\mathrm{dis}+\mathrm{\&alpha;}}$ (Formula 4)

Among them, α is the path length parameter when the similarity is 0.5, and according to the experimental analysis, α is taken as 1.6.

According to formula 4, the semantic similarity of the first independent sememe, other independent sememe, relational sememe and symbolic sememe are calculated.

2) Calculate the semantic similarity of two concepts

The similarity of two concepts is obtained by the weighted average of the similarity of the above four types of sememes. Therefore, the semantic similarity Sim(S ₁ , S ₂ ) between concept S ₁ and concept S ₂ is calculated according to formula 5:

$\mathrm{Sim}(S_1,S_2)=\underset{k=1}{\overset{4}{\mathrm{\&Sigma;}}}{\mathrm{\&beta;}}_k\underset{q=1}{\overset{k}{\mathrm{\&Pi;}}}{\mathrm{Sim}}_q(x,Y)$ (Formula 5)

Among them, X and Y represent two sememes, Sim _q (X, Y) represents the semantic similarity of the qth type of sememe, 1≤q≤4; β _k (1≤k≤4) is the four types of sememes The weight represents the degree of influence of the four types of sememes on the conceptual semantic similarity, and β ₁ + β ₂ + β ₃ + β ₄ = 1, β ₁ ≥ _{β 2} ≥ _{β 3} ≥ _{β 4} .

In general, the first type of sememe reflects the most important feature of the concept, therefore, the weight is larger, and the similarity of this type of sememe is usually larger, but in practice, the similarity ratio of the first type of sememe also exists. If the similarity of the third or fourth type of sememes is small, the overall similarity of concepts will be larger than the actual one. In Equation 5, for sememes with small weight but possibly high similarity, the overall similarity is reduced by multiplying the similarity of multiple sememes, that is, the similarity value of the main sememe is relatively large compared with that of the secondary sememe. The similarity value plays a restrictive role. If the similarity of the main sememe is relatively low, the effect of the similarity of the secondary sememe on the overall similarity will also be reduced.

3) Calculate the semantic similarity of two Chinese words

For two Chinese words W ₁ and W ₂ , if W ₁ has n concepts: S ₁₁ , S ₁₂ , ..., S _1n , W ₂ has m concepts: S ₂₁ , S ₂₂ , ..., S _2m , Use steps 1) and 2) to calculate the semantic similarity of each pair of concepts of two words, and then take the maximum value of the result, that is, the semantic similarity Sim(W ₁ , W ₂ ) of word W ₁ and word W ₂ is The maximum value of the similarity of each concept is calculated by formula 6:

$\mathrm{Sim}(W_1,W_2)=\underset{v=1...\mathrm{no};w=1...m}{\mathrm{Max}}\mathrm{Sim}(S_{1v},S_{2w})$ (Formula 6)

Among them, S _1v represents the vth concept in word _W1 , 1≤v≤n, n is the number of concepts in word _W1 ; _S2w represents the wth concept in word _W2 , 1≤w≤m, m is the number of concepts in word W ₂ ; Sim(S _1v , S _2w ) represents the semantic similarity between concepts S _1v and S _2w . Take the maximum value of the similarity of each pair of concepts to get the semantic similarity of two words.

The invention can intelligently and automatically provide bibliographic item information for catalogers, reduce manual labor, improve cataloging efficiency, and can adapt to different needs of professional and non-professional catalogers and wide area network environment.

Description of drawings

Figure 1: The overall solution process for WAN-oriented audio and video intelligent cataloging information acquisition

Figure 2: Execution flow of cataloging information extraction based on ontology and HTML

Figure 3: Implementation of a topic crawler searching web pages

Detailed ways

An embodiment of the invention is configured according to FIG. 1 . In this embodiment, the computer is "DELL E02S, CPU: Intel Xeon 5504 2Ghz, 4G memory, and the operating system is Server 2003 Standard Edition sp2".

The cataloger first opens the cataloging system interface, and then chooses to play the video file to be cataloged; then performs description in the description area on the right side of the interface. After entering the description items of "proper title" and "keyword", the system will automatically connect to the subject crawler to search Relevant subject webpages, these webpages will extract relevant catalog information through ontology and HTML-based catalog information extraction functions, and these possibly non-standard catalog information will be normalized by the natural language-based catalog information normalization function, and finally the system will automatically prompt to the corresponding description text box for users to refer to and select. Specific steps are as follows:

1. Theme crawler searches the web

By improving the key algorithm of the subject crawler, the invention designs and realizes an efficient subject crawler information collection function.

Use the title proper and keywords entered by the cataloger as the subject collection of the crawler, and perform the following steps (as shown in Figure 3):

(1) Put the initial seed URL into the queue to be crawled, and wait for the topic crawler to crawl.

(2) Obtain and analyze the webpage corresponding to the initial seed URL, obtain the text content in the <p> tag and <title> tag on it, and obtain the content by analyzing the <a> tag, <area> tag, and <base> tag All URL links on the web page.

(3) Segment the title text and body text on the obtained webpage with the Lucene Chinese tokenizer, form a set of keyword feature items after deduplication analysis, and match it with the subject set, and obtain the subject vector through word frequency calculation Equal-dimensional feature term vectors.

(4) According to the "weight algorithm based on the position factor of the keyword feature item" proposed by the present invention, calculate the weight of each feature item based on the position factor, and then calculate the similarity between the page document and the topic. If the similarity is greater than or equal to the threshold (taken as 0.4), it will be downloaded to the database and indexed, otherwise, it will be discarded. And put the URL link into the completion queue.

(5) analyze all links that above-mentioned webpage comprises, and it is carried out repeatability judgment with the URL in the completion queue, repeats then discards, does not repeat then according to the link theme similarity algorithm (formula 3) that the present invention designs, calculates its theme Similarity, according to the priority of the subject similarity, put it in the queue to be climbed.

(6) When the queue to be crawled is empty and all threads are idle, the crawling work ends. The crawled webpage links will be used for the next step of cataloging information extraction.

In order to evaluate the theme crawling effect, a crawler example with the theme of "2008 Beijing Olympics" is designed as follows:

First configuration parameters are: thread number=3, crawling depth=1, initial seed number=3, and then subject crawling is carried out according to the above-mentioned method designed in the present invention.

The general topic crawler uses the traditional vector space model to determine the topic similarity and a single search strategy based on content evaluation, but the present invention uses "weight algorithm based on keyword feature item position factors" to calculate the topic similarity of web page content, And comprehensively consider the three factors of webpage content similarity, URL directory level information of hyperlinks in webpages, and anchor text information of hyperlinks in webpages to calculate the theme similarity of each link, so that pages with similar themes can be found more accurately.

In this experiment, the general theme crawler is compared with the theme crawler designed by the present invention, and the data in Table 1 are obtained.

Table 1 general theme crawler and theme crawler comparative analysis of the present invention

General theme crawler The theme crawler designed by the present invention The total number of links analyzed (pieces) 1280 1419 Number of topic-related links (pieces) 413 494 The number of timed out connections (pieces) 35 45 The number of discarded links (pieces) 832 880 Total time spent (seconds) 3855.843 2191.344 Crawling rate (pieces/second) 0.332 0.648

Through the analysis of experimental data, it can be seen that the theme crawler designed by the present invention has the following advantages:

(1) Aspect the total number of links, the theme crawler designed by the present invention increases the number of links analyzed by the general theme crawler, which shows that the search strategy adopted by the present invention is under the premise of ensuring that the links relevant to the theme are analyzed as much as possible. Alleviated the "myopia" problem of theme crawlers and improved the recall rate of crawlers;

(2) In terms of the acquisition of topic-related links, the topic crawler designed by the present invention also has an increased amount of acquisition than general topic crawlers, which shows that the topic similarity judgment algorithm of the present invention is different from the topic similarity judgment based on the traditional vector space model Compared with the algorithm, web documents related to the topic can be obtained more accurately, which improves the accuracy rate of crawlers and avoids misjudgment of documents related to the topic;

(3) In terms of crawling time and speed, the theme crawler designed by the present invention has significantly improved efficiency than general theme crawlers.

2. Cataloging Information Extraction Based on Ontology and HTML

For the topic pages with high similarity searched in step 1, the catalog information extraction method based on ontology and HTML is used to extract the catalog information. The HTML page structure is analyzed and located by the information extraction method based on the HTML structure, the extraction rules are generated, and the semantic problem is solved by the information extraction method based on the ontology. The specific processing steps are as follows:

(1) Using Protégé, an ontology construction tool developed by Stanford University, to construct a multimedia content extraction ontology, the concept in the ontology is defined as the catalog entry of multimedia file content description information, the attribute is defined as the relationship between concepts, and the label attribute of the concept defines each Extract the polysemous words corresponding to the extracted bibliographic items from the data source. The created ontology is stored as a file in OWL format.

(2) Use the HTML Tidy toolkit to clean the HTML page, correct the error information, remove the redundant information, convert it into an XHTML document, and then use the XML parser to parse the document into a DOM tree structure.

(3) Locate information based on the combination of tree path and text content, generate XPath path, and generate extraction rules.

(4) Read the extraction rules generated in step (3).

(5) Using Jena, an open source semantic web toolkit based on Java developed by HP, to read the multimedia content extraction ontology and operate on the classes, attributes and instances in the ontology.

(6) Take the extraction rules and ontology of steps (4) and (5) as input, and execute the extraction algorithm. The specific steps are: divide the specific information in the information block to be extracted in the DOM tree formed by HTML parsing into key-value; read the concept in the multimedia file and extract the label attribute value of the concept; if the key in the DOM tree is consistent with the ontology If the label attribute value of the concept in the ontology corresponds, the concept of the ontology and the corresponding value value are saved in the XML file.

(7) For the result files extracted from multiple webpage data sources, by comparing and analyzing the information of each catalog item in them, performing information fusion, and finally generating an extraction result file (XML document).

In order to evaluate the effect of cataloging information extraction, for ten films and television drama videos, nine pages in the subject pages searched in step 1 were respectively used, and the cataloging information extraction method of the present invention was used to analyze and extract them. The website sources corresponding to these nine pages are Tencent.com, Sohu.com, Sina.com, Douban.com, Mtime.com, Tudou.com, Youku.com, Ku6.com, and Xunlei Kankan. Each website source provides a film and television drama video.

The statistical analysis of the extraction results of nearly a hundred extraction instances from nine different website sources is as follows:

Table 2 Analysis of cataloging information extraction results based on ontology and HTML

in,

Recall rate R = correctly extracted information / correct information that should be extracted

Accuracy rate P = correct extracted information / extracted information

其中，δ是召回率和准确率的相对权重，δ＝1时，召回率与准确率同样重要，δ＞1时，准确率更重要，δ＜1时，召回率更重要。实验中，取δ＝1，使召回率和准确率同样重要。The F index is the weighted geometric mean of the recall rate R and the accuracy rate P. In order to comprehensively evaluate the extraction performance,

Among them, δ is the relative weight of the recall rate and the precision rate. When δ=1, the recall rate is as important as the precision rate. When δ>1, the precision rate is more important. When δ<1, the recall rate is more important. In the experiment, take δ=1, so that the recall rate and precision rate are equally important.

As can be seen from the data in the above table, the recall rate is almost all above 80%, mostly close to 90%, and the F index is also basically above 80%, which shows that the method proposed by the present invention is effective in processing simple extraction page information and complex semantic description extraction pages When extracting information, it can achieve a better extraction effect, has a strong adaptability to page structure changes, and has high stability.

3. Normalization of cataloging information based on natural language

First, store the relationship tree between HowNet thesaurus and sememes as two text files, which has the advantage of reducing database storage time and helping users to independently expand and modify thesaurus; then the system automatically reads in the two text files , matching the catalog information extracted in step 2 with thesaurus content one by one, automatically analyzing the character string, separating the part of speech and sememe string of the word; then by the algorithm designed in the present invention, at first calculate the similarity of the sememe (take α=1.6), and then calculate the similarity between all the concepts contained in the two words (take β ₁ =0.5, β ₂ =0.2, β ₃ =0.17, β ₄ =0.13), and take the maximum The value is taken as the semantic similarity of two words, and the normalized bibliographic items whose similarity exceeds the threshold are displayed on the system interface. According to the statistical analysis of a large number of experiments, setting the threshold at 0.7 can get more accurate results.

Based on the extraction results of nearly a hundred extraction examples from nine different website sources in step 2, the word similarity algorithm of the present invention is used to carry out a cataloging information standardization experiment. The experimental results show that the words selected by the present invention according to the similarity It is consistent with the actual situation.

Taking the item "audience" as an example, the content of the item extracted in step 2 is "children", while in the standardized thesaurus established in step 3, the items of "audience" include "minor", "adult" and "public ". By applying the semantic similarity algorithm designed by the present invention, the semantic similarity of "children" and "minor", "adult" and "popular" three normalized entries are calculated respectively, and "minor" and "children" are The semantics are the closest, so the system automatically displays the calculation result "underage" in the text box of the "audience" bibliographic item.

Taking the "language" item as an example again, the content of the item extracted in step 2 is "Chinese", and by applying the semantic similarity algorithm designed by the present invention, each "language" entry in "Chinese" and the standard lexicon is calculated respectively The semantic similarity of "Chinese" and "Chinese" in the standard thesaurus is the closest in semantics, so the system automatically displays the calculation result "Chinese" in the "Language" description item text box.

The present invention adopts a weighting algorithm based on the location factors of keyword feature items, and assigns different weighting factors to feature items in different positions in the document, thereby more accurately calculating the subject similarity of webpage content; comprehensively utilizing webpage content similarity, hyperlink Based on the three factors of URL directory hierarchy information and hyperlink anchor text information, links with higher topic similarity are optimized and selected. For the searched subject pages, the catalog information is automatically extracted using ontology and HTML-based information extraction methods. The extracted catalog information is normalized by using an improved semantic similarity calculation method. The invention can intelligently and automatically provide bibliographic item information for catalogers, reduce manual labor, improve cataloging efficiency, adapt to different needs of professional and non-professional catalogers, and also adapt to wide area network environment.

Claims (1)

1. A wide area network-oriented audio and video intelligent cataloguing information acquisition method is characterized by comprising the following steps:

(1) topic crawler search web page

Calculating the topic similarity of the webpage content by adopting a weight algorithm based on the position factors of the keyword characteristic items; comprehensively utilizing three factors of webpage content similarity, URL directory hierarchical information of hyperlinks and anchor text information of the hyperlinks to calculate link topic similarity;

the weighting algorithm based on the position factors of the keyword feature items comprises the following specific steps:

1) defining different positions where the feature items appear, and giving different position weight factors to the feature items at the different positions;

defining the position of the occurrence of the keyword feature item as 3 types: subject label, title label, other positions of the text; the importance of the 3 types of positions to the feature items is sequentially reduced;

then, a position weight factor PG is introduced to express the importance of the characteristic items at different positions, and the PG is defined_i(i is 1, 2, 3) is a weighting factor corresponding to the feature term at different positions, i represents the above 3 types of positions, and defines: PG (Picture experts group)_i＞＝PG_i+1(1＜＝i＜＝2)；

TF'_i(i ═ 1, 2, 3) is the frequency with which the characteristic term t appears at different positions;

2) calculating the weight of the feature item based on the position factor;

according to the position of the feature item, calculating the weight DWeight (t) of the feature item t of a certain keyword in the page document D, wherein the formula is as follows:

(formula 1)

The steps of calculating the similarity of the linking topics are as follows:

1) calculating the topic similarity of the current page content:

(formula 2)

In formula 2, D is the searched page document, z represents the dimension of the Topic Topic, DWeight (j) represents the weight of the jth keyword feature item in D, Topicweight (j) represents the weight of the jth Topic item in the Topic Topic, and j is more than or equal to 1 and less than or equal to z;

2) for each link of the current page, judging whether the link is adjacent to the current page in directory position or not through URL directory information, if so, estimating the similarity of the target page according to the topic similarity of the content of the current page, and if not, evaluating the similarity of the link through the analysis of the anchor text; the calculation formula is as follows:

(formula 3)

Wherein:

d-current webpage;

hyperlink anchor text in L-D web page;

c-the target web page pointed to by L;

sim (D) -topic similarity of web page D;

sim (L) -topic similarity of anchor text linking L;

sim (C) -a predicted value for similarity of subject matter C of L-directed target web pages;

x is an influence coefficient, wherein x is 0-1, x is used for adjusting weight distribution of theme similarity of the D factor and the anchor L factor, the larger x is, the more the anchor text is inclined to a formula, and the smaller x is, the more the theme similarity of a parent webpage is inclined to;

(2) ontology and HTML based catalog information extraction

Analyzing and positioning an HTML page structure by adopting an information extraction method based on an HTML structure, generating an extraction rule, and solving a semantic problem by adopting an information extraction method based on a body; the specific treatment steps are as follows:

1) constructing an ontology: constructing a multimedia content extraction body, wherein concepts in the body are defined as cataloguing and bibliographic items of multimedia file content description information, attributes are defined as relations among the concepts, and label attributes of the concepts define polysemous words corresponding to the bibliographic items extracted from each extraction data source;

2) analyzing the webpage: cleaning HTML page, correcting error information of page, removing redundant information, converting into XHTML document, and then analyzing the document into DOM tree structure;

3) and (3) generating an extraction rule: generating an XPath path and an extraction rule according to positioning information based on the combination of the tree path and the text content;

4) reading an extraction rule: reading the extraction rule generated in the step 3);

5) reading the body: reading a multimedia content extraction body, and operating the class, the attribute and the example in the body;

6) executing an extraction algorithm: taking the extraction rules and the body of the steps 4) and 5) as input, and executing an extraction algorithm; the method comprises the following specific steps: dividing specific information in an information block to be extracted in a DOM tree formed by HTML (hypertext markup language) analysis into key-values; reading concepts in the multimedia file extraction ontology and label attribute values of the concepts; if the key in the DOM tree corresponds to the label attribute value of the concept in the ontology, storing the concept of the ontology and the corresponding value into an XML file, namely, extracting all data contained in the webpage information block; the extracted specific information can be added into the extracted ontology as a specific example of a concept in the ontology to expand the ontology model;

7) information fusion and storage: extracting corresponding extraction results according to each webpage data source, wherein the plurality of data sources correspond to each extraction result file, and the information contained in each result file is the same and different; the system performs information fusion by comparing and analyzing information in each extraction result file, and finally generates an extraction result file;

(3) cataloging information normalization based on natural language

Adopting a semantic similarity calculation method and providing an improved calculation method of the similarity of the concept semantic expression, and performing similarity calculation on the cataloguing information extracted in the step (2) and the information of the normative lexicon to further determine the contents of the normative cataloguing items; the specific process is as follows:

the system reads in the cataloguing information extracted in the step (2), a known network word stock and a semantic tree file, finds two matched words and calculates the semantic similarity of the two words; the method comprises the following steps that one Chinese word is composed of one or more concepts, the similarity of the two words is the maximum value of the semantic similarity of each concept, the similarity problem between the two Chinese words is solved to the similarity problem between the two concepts, all the concepts are finally expressed by a semantic primitive, and the semantic similarity of the two Chinese words is calculated step by step from the calculation of the similarity of the semantic primitives according to the following steps;

1) calculating semantic similarity of sememes

In the tree hierarchy formed by the sememes, assuming that the path distance between two sememes X and Y is dis, which is a positive integer, the semantic similarity Sim (X, Y) between the two sememes is calculated according to formula 4:

(formula 4)

Wherein α is a path length parameter indicating a similarity of 0.5;

calculating the semantic similarity of four types of sememes, namely a first independent sememe, other independent sememes, a relation sememe and a symbol sememe according to a formula 4;

2) calculating semantic similarity of two concepts

The similarity of the two concepts is obtained by weighted average of the similarity of the four types of the sememes; concept S₁And concept S₂Semantic similarity Sim (S) between₁，S₂) Calculated according to equation 5:

(formula 5)

Wherein X and Y represent two sememes, Sim_q(X, Y) represents the semantic similarity of the q-th class of sememes, and q is more than or equal to 1 and less than or equal to 4; beta is a_k(1 ≦ k ≦ 4) is the weight of the four classes of sememes, representing the degree of influence of the four classes of sememes on the semantic similarity of the concept, and has β₁+β₂+β₃+β₄＝1，β₁≥β₂≥β₃≥β₄；

In formula 5, for the semantic atoms with smaller weight but possibly larger similarity, the overall similarity is reduced by the product of the similarities of a plurality of the semantic atoms, that is, the similarity of the primary semantic atoms has a restriction effect on the similarity of the secondary semantic atoms, and if the similarity of the primary semantic atoms is lower, the similarity of the secondary semantic atoms also has a reduction effect on the overall similarity;

3) calculating semantic similarity of two Chinese words

For two Chinese words W₁And W₂If W is₁There are n concepts: s₁₁，S₁₂，……，S_1n，W₂There are m concepts: s₂₁，S₂₂，……，S_2mCalculating the semantic similarity of each pair of concepts of the two terms by adopting the steps 1) and 2), and then taking the maximum value of the result, namely the term W₁And the word W₂Semantic similarity Sim (W) of₁，W₂) Is the maximum value of the similarity of each concept, and is calculated by equation 6:

(formula 6)

Wherein S is_1vMeans word W₁The v-th concept in the specification, v is more than or equal to 1 and less than or equal to n, and n is the word W₁The number of concepts in; s_2wMeans word W₂W is more than or equal to 1 and less than or equal to m which is a word W₂The number of concepts in; sim (S)_1v，S_2w) Represents the concept S_1vAnd S_2wThe semantic similarity of the two words is obtained by taking the maximum value of the similarity of each pair of concepts.

Images