CN103778256A - Method for realizing extraction of Internet audiovisual programs based on context - Google Patents
Method for realizing extraction of Internet audiovisual programs based on context Download PDFInfo
- Publication number
- CN103778256A CN103778256A CN201410065728.2A CN201410065728A CN103778256A CN 103778256 A CN103778256 A CN 103778256A CN 201410065728 A CN201410065728 A CN 201410065728A CN 103778256 A CN103778256 A CN 103778256A
- Authority
- CN
- China
- Prior art keywords
- audiovisual material
- audio
- website
- audiovisual
- webpage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Information Transfer Between Computers (AREA)
Abstract
本发明涉及一种基于上下文环境实现互联网视听节目提取的方法，其中包括加载预定义的视听节目元数据库；加载需要提取视听节目网站的种子地址；下载需要提取视听节目网站的网页内容；判断下载的网页是否为一视听节目的播放页面；对于播放页面，查找该视听节目的上文并生成视听节目单，对于不是播放页面的，根据加载的视听节目元数据库量化该下载的网页内容作为视听节目上文并存入上文集合中。采用该种基于上下文环境实现互联网视听节目提取的方法，实现了在不针对特定网站创建提取模板的情况下，提取所有互联网上网站的视听节目，可以规避视听节目信息抓取过程中不必要的干扰，保证视听节目信息抓取的准确性，具有更广泛的应用范围。
The invention relates to a method for extracting Internet audio-visual programs based on the context environment, which includes loading a predefined audio-visual program metadata database; loading the seed address of the audio-visual program website that needs to be extracted; downloading the webpage content of the audio-visual program website that needs to be extracted; judging the downloaded Whether the webpage is a play page of an audio-visual program; for the play page, search for the above text of the audio-visual program and generate an audio-visual program list; for those that are not play pages, quantify the downloaded webpage content as the audio-visual program on the basis of the loaded audio-visual program metadata database The text is stored in the above collection. Using this context-based method for extracting Internet audio-visual programs realizes the extraction of audio-visual programs from all websites on the Internet without creating an extraction template for a specific website, which can avoid unnecessary interference in the process of capturing audio-visual program information , to ensure the accuracy of capturing audio-visual program information, and has a wider range of applications.
Description
Technical field
The present invention relates to Internet technical field, relate in particular to internet audiovisual material information and play link extraction field, specifically refer to a kind of method that realizes internet audiovisual material extraction based on context environmental.
Background technology
The general extracting method of current existing internet audiovisual material is: each type program of website is created to a kind of template of extracting, the detailed elements path of extracting programme information is set, then collect page elements by reptile according to template, finally gather generating video programme information.This scheme is for the huge audiovisual Websites quantity in internet, and each website generates one and extracts template, if website revision or renewal page structure just need to be revised corresponding reptile configuration template.
Under prior art, if crawl the audiovisual material on all internets, and form consistent audiovisual material table, the configuration amount of template is as astronomical figure, adds the website renewal of can ceaselessly upgrading, and safeguards so most according to being impossible mission.
Summary of the invention
The object of the invention is to overcome the shortcoming of above-mentioned prior art, provide a kind of can realize do not create for specific website extract template in the situation that, extract website on all internets audiovisual material, the accuracy that guarantees audiovisual material information scratching, there is broader applications scope realize based on context environmental the method that internet audiovisual material is extracted.
To achieve these goals, of the present inventionly realize based on context environmental the method that internet audiovisual material extracts and there is following formation:
Should realize the method that internet audiovisual material is extracted based on context environmental, its principal feature is that described method comprises the following steps:
(1) load predefined audiovisual material metadatabase;
(2) load the seed address that need to extract audiovisual material website;
(3) download the web page contents that need to extract audiovisual material website;
(4) judge whether the webpage of downloading is the broadcasting page of an audiovisual material, if so, continues step (5), otherwise continue step (6);
(5) search this audiovisual material above and generate audiovisual material list;
(6) web page contents that quantizes this download according to the audiovisual material metadatabase loading as audiovisual material above and deposit in set above.
Preferably, described audiovisual material metadata comprises director, protagonist, performer, issuing time, update time and the program outline of audiovisual material.
Preferably, described loading need to be extracted the seed address of audiovisual material website, is specially:
Load the seed address that need to capture audiovisual material website from Xml file or database.
Preferably, described download need to be extracted the web page contents of audiovisual material website, is specially:
Use Http client or reptile that the web page contents of the named web page of intended target website is downloaded to this locality from server.
Preferably, described search this audiovisual material above and generate audiovisual material list, comprise the following steps:
(51) broadcasting type corresponding to this audiovisual material identified;
(52) from searching this audiovisual material above set above;
(53) merge the above complete documentation of this audiovisual material of web content data Information generation of metadata information and this download.
More preferably, described identifies broadcasting type corresponding to this audiovisual material, is specially:
Identify broadcasting type corresponding to this audiovisual material and utilize corresponding player to verify broadcasting to this audiovisual material.
Preferably, the described web page contents that quantizes this download according to the audiovisual material metadatabase loading as audiovisual material above and deposit in set above, comprises the following steps:
(61) judge whether this webpage is the details page of an audiovisual material, if so, continues step (62), otherwise continue step (3);
(62) according to the rule of audiovisual material metadatabase definition, this webpage is quantized and judge this webpage be whether an audiovisual material above, if so, continue step (63), otherwise continue step (64);
(63) using this webpage as an audiovisual material above and deposit above set in, then continue step (64);
(64) judge whether this webpage is last webpage of website, if so, finishes to exit, otherwise, step (65) continued;
(65) analyze the hyperlink of this webpage and add webpage queue to be downloaded, then continuing step (3).
Adopt and realized based on context environmental the method that internet audiovisual material is extracted in this invention, there is following beneficial effect:
(1) adopt audiovisual material information characteristics quantization method, can evade unnecessary interference in audiovisual material information scratching process, thereby can guarantee that the audiovisual material grabbing is accurately.
(2) utilize the unchangeability of audiovisual material metadata information, for the renewal of website layout or content, capture as long as implement the increment of the method, can grab the latest update audiovisual material information of website.
(3) utilize the checking of player rule, can guarantee that the audiovisual material grabbing is the audiovisual material that can play.
(4) adopt a small amount of configuration, not for specific website, but can identify the audiovisual material on internet by the relation between webpage, obtain essential information and the broadcast address of audiovisual material, can be in the situation that not creating extraction template for specific website, extract the audiovisual material of website on all internets, there is range of application widely.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of realizing the method for internet audiovisual material extraction based on context environmental of the present invention.
Embodiment
In order more clearly to describe technology contents of the present invention, conduct further description below in conjunction with specific embodiment.
The audiovisual material grasping means of existing internet, is all for page layout and content configuration template, thus identification audiovisual material.
The present invention is from audiovisual material, carry out abstract to the metadata of audiovisual material, such as: audiovisual material generally all can issuing time/update time, director, performer, the present invention carries out template configuration for these metadata exactly, main contents display area at webpage is identified these metadata, and then the information recording of formation audiovisual material above.
According to the invention process, as long as the once template of (or on a small quantity several times) audiovisual material metadata of configuration, just can avoid the template of a large amount of different web sites that configure under prior art, and the website space of a whole page upgrade after later maintenance, because for existing audiovisual material, its basic metadata information can not become, as: director and the performer of film " decisive battle greatly " can not become all the time.
Audiovisual material on internet, have the details page of audiovisual material, details page has been collected most of metadata of this audiovisual material, these data can form a part for audiovisual material information, have link in details page and be associated with the broadcasting page, play together with the information of the page and the information combination of details page, form the context of an audiovisual material, in conjunction with context, system generates an audiovisual material record.
Realization flow:
1, system starts, and loads meta data category, definition in predefined audiovisual material metadatabase, Web page loading player recognition feature;
2, load the seed address of the spiders of configuration, in these addresses, may have the audiovisual material information of expection;
3, the network being defined by reptile is downloaded logic, downloads the web page contents being present in queue to be crawled;
4, analyzing web page content:
Whether first identify this page by player identification module is the broadcasting page of an audiovisual material;
Whether be the details page of an audiovisual material by this webpage of audiovisual material metadata collecting Module recognition;
Collected the hyperlink of this page by URL analysis module, these hyperlink be likely an audiovisual material above below, also may be a new audiovisual material above, these hyperlink are added in the queue to be climbed of reptile, for continuing the crawl of the next page, complete the traversal to whole website with this;
If 5 current pages are broadcasting pages of an audiovisual material,, merge above metadata information and this page metadata information from this page of set search above above, generate the complete documentation of an audiovisual material;
If 6 current pages are not the broadcasting pages of an audiovisual material, according to the rule of metadata definition, quantize this page, with judge this page be whether an audiovisual material above, if quantized result meets the rule above of an audiovisual material, deposit current page in above set;
If 7 systems need further to capture, jump to 3;
8, system has completed the page to be analyzed, completes audiovisual material and extracts.
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing, the specific embodiment of the present invention is further described.
Fig. 1 is that a kind of that the embodiment of the present invention provides realizes based on context environmental the method that internet audiovisual material is extracted, and comprising:
Step (1): loading audiovisual material feature database is metadatabase.
Particularly, audiovisual material is all accompanied with director, protagonist, performer, issuing time, update time, program outline etc., according to different audiovisual material types, can configure different audiovisual material metadata combinations.
Step (2): loading need to be extracted the seed address of audiovisual material website.
Particularly, can load the website that need to capture audiovisual material from Xml file or database.
Step (3): downloading web pages content.
Particularly, use Http client or reptile, the named web page that refers to targeted website is downloaded to this locality from server.
Step (4): analyzing web page content, determine whether this page is the broadcasting page of audiovisual material.
Whether particularly, first identifying this page by player identification module is the broadcasting page of an audiovisual material, and which kind of player identifies be, as Flash player; Whether be the details page of an audiovisual material by this webpage of audiovisual material metadata collecting Module recognition, arrange out audiovisual material metamessage and define needed content; Collected the link of this page by URL analysis module, for continuing the crawl of the next page.
Step (5): search audiovisual material above, generate audiovisual material list.
Particularly, the audiovisual material information getting according to step (4), if current page is the broadcasting page of an audiovisual material, from this page of set search above above, merge above metadata information and this page metadata information, generate the complete documentation of an audiovisual material.
Step (6): take audiovisual material feature database as criterion, quantize web page contents, as audiovisual material above.
Particularly, the audiovisual material information getting according to step (4), quantize this page, with judge this page be whether an audiovisual material above, if quantized result meets the rule above of an audiovisual material, deposit above current page in set, this set can be a HASH table, or a tables of data in database.
Adopt and realized based on context environmental the method that internet audiovisual material is extracted in this invention, there is following beneficial effect:
(1) adopt audiovisual material information characteristics quantization method, can evade unnecessary interference in audiovisual material information scratching process, thereby can guarantee that the audiovisual material grabbing is accurately.
(2) utilize the unchangeability of audiovisual material metadata information, for the renewal of website layout or content, capture as long as implement the increment of the method, can grab the latest update audiovisual material information of website.
(3) utilize the checking of player rule, can guarantee that the audiovisual material grabbing is the audiovisual material that can play.
(4) adopt a small amount of configuration, not for specific website, but can identify the audiovisual material on internet by the relation between webpage, obtain essential information and the broadcast address of audiovisual material, can be in the situation that not creating extraction template for specific website, extract the audiovisual material of website on all internets, there is range of application widely.
In this instructions, the present invention is described with reference to its specific embodiment.But, still can make various modifications and conversion obviously and not deviate from the spirit and scope of the present invention.Therefore, instructions and accompanying drawing are regarded in an illustrative, rather than a restrictive.
Claims (7)
1. realize based on context environmental the method that internet audiovisual material is extracted, it is characterized in that, described method comprises the following steps:
(1) load predefined audiovisual material metadatabase;
(2) load the seed address that need to extract audiovisual material website;
(3) download the web page contents that need to extract audiovisual material website;
(4) judge whether the webpage of downloading is the broadcasting page of an audiovisual material, if so, continues step (5), otherwise continue step (6);
(5) search this audiovisual material above and generate audiovisual material list;
(6) web page contents that quantizes this download according to the audiovisual material metadatabase loading as audiovisual material above and deposit in set above.
2. the method that realizes internet audiovisual material extraction based on context environmental according to claim 1, is characterized in that, described audiovisual material metadata comprises director, protagonist, performer, issuing time, update time and the program outline of audiovisual material.
3. the method that realizes internet audiovisual material extraction based on context environmental according to claim 1, is characterized in that, described loading need to be extracted the seed address of audiovisual material website, is specially:
Load the seed address that need to capture audiovisual material website from Xml file or database.
4. the method that realizes internet audiovisual material extraction based on context environmental according to claim 1, is characterized in that, described download need to be extracted the web page contents of audiovisual material website, is specially:
Use Http client or reptile that the web page contents of the named web page of intended target website is downloaded to this locality from server.
5. according to claim 1ly realize based on context environmental the method that internet audiovisual material is extracted, it is characterized in that, described search this audiovisual material above and generate audiovisual material list, comprise the following steps:
(51) broadcasting type corresponding to this audiovisual material identified;
(52) from searching this audiovisual material above set above;
(53) merge the above complete documentation of this audiovisual material of web content data Information generation of metadata information and this download.
6. the method that realizes internet audiovisual material extraction based on context environmental according to claim 5, is characterized in that, described identifies broadcasting type corresponding to this audiovisual material, is specially:
Identify broadcasting type corresponding to this audiovisual material and utilize corresponding player to verify broadcasting to this audiovisual material.
7. the method that realizes internet audiovisual material extraction based on context environmental according to claim 1, it is characterized in that, the described web page contents that quantizes this download according to the audiovisual material metadatabase loading as audiovisual material above and deposit in set above, comprises the following steps:
(61) judge whether this webpage is the details page of an audiovisual material, if so, continues step (62), otherwise continue step (3);
(62) according to the rule of audiovisual material metadatabase definition, this webpage is quantized and judge this webpage be whether an audiovisual material above, if so, continue step (63), otherwise continue step (64);
(63) using this webpage as an audiovisual material above and deposit above set in, then continue step (64);
(64) judge whether this webpage is last webpage of website, if so, finishes to exit, otherwise, step (65) continued;
(65) analyze the hyperlink of this webpage and add webpage queue to be downloaded, then continuing step (3).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410065728.2A CN103778256B (en) | 2014-02-26 | 2014-02-26 | Method for realizing extraction of Internet audiovisual programs based on context |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410065728.2A CN103778256B (en) | 2014-02-26 | 2014-02-26 | Method for realizing extraction of Internet audiovisual programs based on context |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103778256A true CN103778256A (en) | 2014-05-07 |
| CN103778256B CN103778256B (en) | 2017-02-01 |
Family
ID=50570491
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410065728.2A Active CN103778256B (en) | 2014-02-26 | 2014-02-26 | Method for realizing extraction of Internet audiovisual programs based on context |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103778256B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241307A (en) * | 2018-08-29 | 2019-01-18 | 山东浪潮商用系统有限公司 | A kind of performers and clerks' contents management method and system |
| CN115002068A (en) * | 2022-05-09 | 2022-09-02 | 北京市博汇科技股份有限公司 | Internet audio-visual program address automatic analysis method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090049122A1 (en) * | 2006-08-14 | 2009-02-19 | Benjamin Wayne | System and method for providing a video media toolbar |
| CN102630041A (en) * | 2012-04-01 | 2012-08-08 | 央视国际网络有限公司 | Processing method, device and system for television program data |
| CN102902785A (en) * | 2012-09-29 | 2013-01-30 | 合一网络技术(北京)有限公司 | Webpage information acquisition system and method |
| CN103428525A (en) * | 2013-07-22 | 2013-12-04 | 华中科技大学 | Online inquiry and play control method and system for network videos and television programs |
-
2014
- 2014-02-26 CN CN201410065728.2A patent/CN103778256B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090049122A1 (en) * | 2006-08-14 | 2009-02-19 | Benjamin Wayne | System and method for providing a video media toolbar |
| CN102630041A (en) * | 2012-04-01 | 2012-08-08 | 央视国际网络有限公司 | Processing method, device and system for television program data |
| CN102902785A (en) * | 2012-09-29 | 2013-01-30 | 合一网络技术(北京)有限公司 | Webpage information acquisition system and method |
| CN103428525A (en) * | 2013-07-22 | 2013-12-04 | 华中科技大学 | Online inquiry and play control method and system for network videos and television programs |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241307A (en) * | 2018-08-29 | 2019-01-18 | 山东浪潮商用系统有限公司 | A kind of performers and clerks' contents management method and system |
| CN115002068A (en) * | 2022-05-09 | 2022-09-02 | 北京市博汇科技股份有限公司 | Internet audio-visual program address automatic analysis method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103778256B (en) | 2017-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101853300B (en) | Method and system for identifying and evaluating video download service websites | |
| KR20210053825A (en) | Method and apparatus for processing video | |
| US10216856B2 (en) | Mobilizing an existing web application | |
| US9608870B1 (en) | Deep link verification for native applications | |
| CN110008378B (en) | Corpus collection method, device, equipment and storage medium based on artificial intelligence | |
| CN102819591B (en) | A kind of content-based Web page classification method and system | |
| CN101364979B (en) | Downloaded material parsing and processing system and method | |
| CN103744853A (en) | Method and device for providing web cache information in search engine | |
| US8972374B2 (en) | Content acquisition system and method of implementation | |
| CN107925696A (en) | Systems and methods for identifying, indexing and navigating to the deep state of a mobile application | |
| CN103631875A (en) | Method and browser for network search on browser side | |
| WO2017107620A1 (en) | Method and system for loading page data | |
| US10614500B2 (en) | Identifying search friendly web pages | |
| KR102024998B1 (en) | Extracting similar group elements | |
| CN112115266B (en) | Malicious website classification method, device, computer equipment and readable storage medium | |
| CN102609412A (en) | RSS (Really Simple Syndication)-based multi-thread graphic information synchronization crawling control method and system | |
| US11250080B2 (en) | Method, apparatus, storage medium and electronic device for establishing question and answer system | |
| CN104615700A (en) | Method for collecting webpage objects in browser, browser client side and system | |
| CN103631806A (en) | Network information fetching method and device | |
| CN103778256A (en) | Method for realizing extraction of Internet audiovisual programs based on context | |
| CN104572945B (en) | A kind of file search method and device based on cloud storage space | |
| KR101837341B1 (en) | Method and system for dividing viedo content | |
| CN101517574A (en) | Illegal contents auto-searching system and method using access/search application on internet | |
| US9367590B1 (en) | Method and system to cloud-enabled large-scaled internet data mining and data analytics | |
| CN104363237A (en) | Method and system for processing internet media resource metadata |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20171214 Address after: KIC Business Center No. 433 Shanghai 200433 Yangpu District Shanghai Road No. 6 building 11 layer Patentee after: SHANGHAI CERTUSNET Inc. Address before: 210042 Xuanwu District, Xuanwu District, Jiangsu, Nanjing, No. 699-22, building 18 Patentee before: CERTUSNET CORP. |
|
| TR01 | Transfer of patent right | ||
| PP01 | Preservation of patent right |
Effective date of registration: 20250109 Granted publication date: 20170201 |
|
| PP01 | Preservation of patent right |