CN101404026A - Crawler system construction method for video-previewing search engine - Google Patents

Abstract

The invention discloses a method for constructing a crawler system capable of previewing a video search engine. The method comprises the following steps: (1) hyperlinks are mapped into a list; (2) list status is detected; (3) summary image processing; (4) Video processing; (5) Video title processing. By applying the method described in the present invention, a general design method can be provided for the crawler system of a video search engine that can be previewed; preview data sets can be provided for a video search engine that can be previewed, simplifying the design and implementation of other parts of the video search engine that can be previewed Development, greatly reducing the development cost of the previewable video search engine crawler system and the previewable video search engine.

Description

Method for constructing a crawler system capable of previewing video search engines

Technical field

The invention relates to a method for constructing a network data acquisition system, and in particular to a method for constructing a crawler system capable of previewing a video search engine.

Background technology

With the advent of the information age and the development of video technology, video has attracted more and more people to appreciate it because of its incomparable advantages and strong visual impact. However, due to the huge amount of video data and the limitation of general network bandwidth, it is difficult for people to watch videos conveniently on their local computers. It is precisely because of this main reason that many video websites have been established on the wide area network to implement online playback of video data so that people can enjoy videos conveniently and quickly in real time. However, with the surge in the amount of video data on video websites, it is difficult for people to find the desired video on the wide area network simply and quickly, so video search engines are born. Although video search engines can bring great convenience, videos are not as easy to identify as text information, and online videos also need to download and buffer video data for smooth playback. In addition, the amount of video data is large, which occupies more bandwidth, and the user's bandwidth and traffic are limited. Therefore, users hope to make a pre-judgment before opening the video webpage whether the video is what they are looking for and whether it is worth watching. If it is not what you need, there is no need to waste time and bandwidth to watch the video. Therefore, the previewability of video search engines has received keen attention.

Since video websites all contain summary pictures and video names of videos, the main visual content of the video can be reflected in a concentrated manner through the summary pictures and video names, and users can preview and judge the video through the summary pictures and names. Therefore, the collection of preview data of the video is a top priority in the process of building a previewable video search engine. At present, there is no systematic and effective method for constructing a collection system for video preview data. The present invention effectively collects the preview data of the video by introducing a hyperlink mapping list technology and a search technology based on the mapping list.

Contents of the invention

In view of the problems existing in the prior art, the purpose of the present invention is to provide a method for constructing a crawler system that can preview a video search engine.

To achieve the above object, the method of the present invention comprises the following steps:

(1) Hyperlinks are mapped into lists;

(2) Check list status;

(3) Abstract image processing;

(4) Video processing;

(5) Video title processing.

In the above method, step (3) further comprises:

(31) In the hyperlink mapping list, search for the summary image;

(32) Download and store summary images.

In the above method, step (4) further comprises:

(41) Search for the video in the hyperlink mapping list;

(42) Download and store videos;

In the above method, step (5) further comprises:

(51) Download video playback page;

(52) Extract the stored video title.

The beneficial effect of the present invention is that, by applying the method described in the present invention, a general design method can be provided for the crawler system of the previewable video search engine; a preview data set can be provided for the previewable video search engine, thereby simplifying the design and development of other parts of the previewable video search engine, and greatly reducing the development cost of the previewable video search engine crawler system and the previewable video search engine.

Other features and advantages of the present invention will become more apparent from the following description of preferred embodiments, which illustrate the principles of the present invention by way of example, in conjunction with the accompanying drawings.

Description of the attached figure

FIG. 1 is a flow chart of a method according to one embodiment of the present invention.

Specific implementation method

The specific implementation modes of the present invention will be described in detail below with reference to the accompanying drawings.

FIG1 is a flow chart of a method according to an embodiment of the present invention. The process starts at step 101. It should be noted that the video websites mentioned below are only examples, and the specific video websites do not constitute a limitation to the present invention. Then in step 102, all hyperlinks of the video web page are analyzed, and all hyperlinks are extracted one by one in the order from top to bottom and from left to right in the web page source code, and finally mapped into a list. It should be noted that the starting web page should be a web page rich in video hyperlinks, such as a video playback page, etc. This is only an optimal example, and the difference in the starting video web page does not constitute a limitation to the present invention.

The hyperlinks are mapped into a list. One implementation method is to analyze and extract the list from the structure of the video web page. This is further explained below by taking an example.

<a    href＝″http://www.tudou.com/programs/view/c74iyYGuDIc/″title＝″New Record of Dominoes″target＝″new″class＝″inner″><imgsrc＝″http://i01.img.tudou.com/data/imgs/i/023/746/281/m10.jpg″alt＝″New Record of Dominoes″width＝″120″height＝″90″class＝″pack_clipImg″</a>

The above is a source code of a video webpage that contains video hyperlinks. It contains two hyperlinks, namely:

http://www.tudou.com/programs/view/c74iyYGuDIc/

http://i01.img.tudou.com/data/imgs/i/023/746/281/m10.jpg

The first one is the hyperlink address pointing to the video playback page, and the second one is the hyperlink address of the summary image corresponding to the video. The characteristic of the video website construction structure is that the hyperlink pointing to the video playback page and the hyperlink to the summary image corresponding to the video are adjacent, and both are marked with html markup language. As can be seen from the above code snippet, there is no other hyperlink between the two hyperlinks, and the hyperlink pointing to the video playback page is marked with href=, and the hyperlink to the summary image corresponding to the video is marked with img src=. Therefore, for a web page containing a video, you can use regular expressions to match the href= and img src= tags to find all hyperlinks in the web page. For example, in the above example,

href="http://www.tudou.com/programs/view/c74iyYGuDIc/" and imgsrc="http://i01.img.tudou.com/data/imgs/i/023/746/281/m10.jpg" are used to search for them, and then all the hyperlinks are listed in the order of search, that is, a hyperlink mapping list is generated, and finally the hyperlink mapping list of the current web page is put at the end of the original hyperlink mapping list. One storage implementation method of a mapping table is to write the current mapping list directly to the end of the original hyperlink mapping list in text form. It should be pointed out that the text form is only an example, and there are other storage forms such as relational databases. The specific storage form does not constitute a limitation to the present invention. The above is an embodiment of mapping hyperlinks into a list, and other different implementation examples do not constitute a limitation to the present invention.

After step 102 , the process proceeds to step 103 .

In step 103, the state of the hyperlink mapping list is analyzed and detected. A specific implementation method for detecting the state of the hyperlink mapping list is to add one bit from the mark of the hyperlink in the processed table to see if it is empty. If it is empty, it means that the mapping list has been processed completely; if it is not empty, it means that the mapping list has not been processed completely. The above is an implementation method for analyzing and detecting the state of the hyperlink mapping list, and other different implementation methods do not constitute a limitation to the present invention.

If the processing is not completed, the process proceeds to step 104; if the processing is completed, the process proceeds to step 110.

In step 104, the hyperlink map list generated in step 102 is searched for a summary image hyperlink. A specific implementation method for searching for summary images is through string matching, such as the example of the code snippet in step 102, matching the string img src = in the hyperlink map list, and the following content:

http://i01.img.tudou.com/data/imgs/i/023/746/281/m10.jpg is the hyperlink of the abstract image. The above is an implementation method for searching for abstract images, and other different implementation methods do not constitute a limitation to the present invention.

After step 104 , the process proceeds to step 105 .

In step 105, the summary picture found in step 104 is downloaded and stored. One embodiment is to use a relational database system to store the downloaded summary pictures, which is convenient for data management. The above is an implementation method for downloading and storing summary pictures, and other different implementation methods do not constitute a limitation of the present invention.

After step 105 , the process proceeds to step 106 .

In step 106, the hyperlink of the video corresponding to the summary picture downloaded and stored in step 105 is searched in the hyperlink mapping list generated in step 102. A specific implementation method of searching for the corresponding video is to first locate the hyperlink of the summary picture by string matching, and then match a hyperlink forward at the position of the hyperlink of the summary picture. The above specific implementation method is based on such a principle: in the construction of the video website, the hyperlink of the video and the hyperlink of the corresponding summary picture are connected together front and back, without any other hyperlink in the middle, and the video hyperlink is in front of the picture hyperlink. As the example of the code fragment in step 102, in the hyperlink mapping list, the two hyperlinks are closely adjacent. From step 104, the hyperlink of the summary image can be known, and the hyperlink mapping list is matched and located: img src = "http://i01.img.tudou.com/data/imgs/i/023/746/281/m10.jpg", and then the tag href = is matched forward to obtain the hyperlink of the corresponding video: http://www.tudou.com/programs/view/c74iyYGuDIc/. The above is an implementation method of searching for the corresponding video, and other different implementation methods do not constitute a limitation to the present invention.

After step 106 , the process proceeds to step 107 .

In step 107, the video found in step 106 is downloaded. In one embodiment, the real video address is first obtained by redirection technology, and then the downloaded video is stored using a relational database system, which can be inserted after the summary image data stored in step 105, so that the associated data set of the two can be obtained. The above is an implementation method for downloading and storing videos, and other different implementation methods do not constitute a limitation of the present invention.

After step 107 , the process proceeds to step 108 .

In step 108, the video playing page is downloaded, that is, the video hyperlink found in step 106 is downloaded. A specific implementation method of downloading the video playing page is to send a data request to the host corresponding to the hyperlink. For example, in step 102, a data request for programs/view/c74iyYGuDIc is sent to the www.tudou.com host to download data. The above is an implementation method of downloading the video playing page, and other different implementation methods do not constitute a limitation of the present invention.

After step 108 , the process proceeds to step 109 .

In step 109, the title of the video is extracted and stored, that is, the title tag <title> is searched for the video playback page downloaded in step 108. A specific implementation method for extracting and storing the title of the video is to match <title> in the playback page through a string search. As in the example in step 102, in the following video playback page: http://www.tudou.com/programs/view/c74iyYGuDIc/, <title>New Record of Dutch Dominoes</title> can be obtained, and the middle part is the title of the video. The middle part is extracted, and then the video title is stored using a relational database system, which can be inserted before the summary image data stored in step 105, so that the three related data sets can be obtained. The above is an implementation method for extracting and storing the title of the video, and other different implementation methods do not constitute a limitation of the present invention.

After step 109, the video playback page downloaded in step 108 is processed in step 102.

At step 110, the system ends.

The above describes the specific implementation of the present invention in conjunction with the accompanying drawings. Various examples do not limit the essential content of the invention. The present invention is not limited to the implementation details provided above, and can be implemented in other embodiments without departing from the characteristics of the present invention. After reading the specification, ordinary technicians in the relevant technical field can make modifications or deformations to the specific implementations described above without departing from the essence and scope of the invention.

Claims (4)

1. a kind of construction method that can preview the crawler system of video search engine, it is characterized in that comprising the following steps: (1) Hyperlinks are mapped into lists; (2) Check the status of the list; (3) Abstract image processing; (4) Video processing; (5) Video title processing. 2. the construction method of the crawler system that can preview video search engine according to claim 1, is characterized in that: step (3) further comprises: (31) In the hyperlink mapping list, search for the summary picture; (32) Download and store summary pictures. 3. the construction method of the crawler system that can preview video search engine according to claim 1, is characterized in that: step (4) further comprises: (41) in the hyperlink mapping list, find the video; (42) Download and store video. 4. the construction method of the crawler system that can preview video search engine according to claim 1, is characterized in that: step (5) further comprises: (51) Download the video playback page; (52) extract and store the video title.

Images