KR20180080535A - Apparatus and method for scrapping web data - Google Patents

Abstract

Provided are a web data scraping system and an operating method thereof. The system simulates a target site specified by a user and renders the target site in a separate frame, so that a site which does not conform to a web standard can also be accurately simulated through a proxy setting. The user can intuitively and easily select a document object model (DOM) from the corresponding frame, and can generate an event controlling the DOM as a data collection rule. Therefore, the event is reflected on a scenario, thereby providing a web scraping service such as automatic repetition performing.

Description

[0001] APPARATUS AND METHOD FOR SCRAPPING WEB DATA [0002]

The present invention relates to a web data scraping device and its operation method, and more particularly to a web data scrapping service that can be easily utilized by a user without expert knowledge.

Web scraping (also called "web harvesting", "web data extraction", "web crawling", etc.) extracts and secures data from a web site. People can do it, but web scraping software is also available. Existing web scraping software or service sites have been difficult for general users who are not developers and not experts.

On the other hand, according to the same-origin policy of the World Wide Web Consortium (W3C), documents and scripts loaded from one source are not allowed to interact with resources from other sources, Providing a dedicated frame for pinging has difficulties.

Korean Patent No. 10-1125083 (Publication date March 21, 2012) U.S. Pat. No. 7,647,351 (Published on January 12, 2010)

According to one aspect, a method of operating a web data scraping device is provided. The method includes: rendering a target site designated by a user and rendering the target site on a target site display unit; Performing graphical processing so that the first DOM selected by the user is distinguished from other objects among at least one DOM (Document Object Model) included in the rendered result; And displaying at least one user input event that can be applied to the first DOM in a form of a menu in an input event display section.

According to one embodiment, the method further includes adding a first scenario line corresponding to a rule for applying a first event selected from the user to the first DOM among the displayed at least one user input event to the data collection rule display unit, And reflecting it. Also, at least one editing interface may be provided when the first scenario line is displayed on the data collection rule display unit. Illustratively, but not exclusively, the editing interface may include additional input interfaces to the first scenario line, deletion of the first scenario line, and change of enforcement order to the first scenario line, .

According to one embodiment, a method includes downloading HTML resources necessary for controlling the target site, converting at least one resource path on the HTML resource into a proxy path of a proxy server and setting the proxy path to a proxy corresponding to the target site, And a proxy setting step of generating a site.

According to an embodiment, the proxy setting step may include generating a first key, which is a unique identifier corresponding to the URL of the target site, and managing the proxy site in association with the first key. In this case, if there is another proxy setting request for the URL in the target site, the first key may be confirmed and the proxy site may be returned through the first key.

According to another aspect, a web data scraping device is provided. The apparatus includes a target site management unit for downloading the target site information and rendering the target site information on the target site display unit when receiving the URL of the target site selected by the user; And displaying at least one event selectable for a first DOM selected by a user among at least one DOM (Document Object Model) included in the rendered result on an input event display unit, And a data collection rule management unit that generates a first data collection rule to be applied to the first DOM.

According to an embodiment, the data collection rule management unit may provide the first data collection rule to the data collection rule display unit in the form of a first scenario line. In this case, the data collection rule management unit displays the first scenario line on the data collection rule display unit, and displays the additional input interface for the first scenario line, the deletion for the first scenario line, ≪ / RTI > and / or change the enforcement order for < / RTI >

On the other hand, the device downloads HTML resources necessary for controlling the target site, converts at least one resource path on the HTML resource into a proxy path of the proxy server, and creates a proxy site corresponding to the target site And may further include a control server. In this case, the control server may further include: a proxy dynamic setting unit configured to perform proxy dynamic setting to download the HTML resource from the target site; An HTML downloader for downloading HTML resources necessary for controlling the target site in a web data collection system; An HTML parser for parsing the path of the HTML resource, converting the resource path on the downloaded HTML resource to a proxy path, recognizing the target site through the converted path, and outputting it to the screen; And an HTML converter for converting the resource path on the HTML resource into the proxy path.

According to an embodiment, the HTML converter may generate a first key, which is a unique identifier corresponding to the URL of the target site, and manage the proxy site in association with the first key. If there is another proxy setting request for the URL in the target site, the first key may be confirmed and the proxy site may be returned through the first key.

According to another aspect, there is provided a program (e.g., a software application) for performing a method of providing a web data scrapping service stored on a computer readable recording medium. According to one embodiment, the method comprises the steps of: rendering a target site designated by a user and rendering the target site on a target site display; Performing graphical processing so that the first DOM selected by the user is distinguished from other objects among at least one DOM (Document Object Model) included in the rendered result; And displaying at least one user input event that can be applied to the first DOM in a form of a menu in an input event display section.

According to an embodiment, the method may further comprise the step of adding a first scenario line corresponding to a rule for applying a first event selected from a user to the first DOM among the at least one user input event displayed on the data collection rule display unit And may further include a step of reflecting on the scenario. The method also includes downloading HTML resources required to control the target site, converting at least one resource path on the HTML resource into a proxy path of the proxy server, and creating a proxy site corresponding to the target site And may further include a proxy setting step. Here, a first key, which is a unique identifier corresponding to the URL of the target site, is generated, the proxy site can be managed in association with the first key, and another proxy setting request for the URL of the same target site The proxy site can be returned through the first key by checking the first key. While each of the above-described steps may be executed with the instruction sets of the program stored on the computer-readable recording medium.

1 is a block diagram illustrating a configuration of a web data collection system according to an embodiment.
FIG. 2 illustrates a screen configuration that a web data collection system according to an embodiment provides to a user.
3 is a diagram illustrating a process of selecting a DOM in a web data collection system according to an exemplary embodiment of the present invention.
FIG. 4 is a diagram illustrating a process of providing an event menu corresponding to a DOM selected according to an exemplary embodiment of the present invention.
FIG. 5 is a diagram for explaining setting and editing of a scenario according to an embodiment.
FIG. 6 is a diagram illustrating a Web data proxy setting and path modification process according to an embodiment.
7 is a flowchart illustrating an operation of a web data collection system according to an exemplary embodiment of the present invention.
8 is a diagram for explaining a configuration of a data collecting robot according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the rights is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

The terms used in the following description are chosen to be generic and universal in the art to which they are related, but other terms may exist depending on the development and / or change in technology, customs, preferences of the technician, and the like. Accordingly, the terminology used in the following description should not be construed as limiting the technical thought, but should be understood in the exemplary language used to describe the embodiments.

Also, in certain cases, there may be a term chosen arbitrarily by the applicant, in which case the meaning of the detailed description in the corresponding description section. Therefore, the term used in the following description should be understood based on the meaning of the term, not the name of a simple term, and the contents throughout the specification.

Web scraping interface

Software and / or Web services that perform web scraping have remained in the domain of developers. General users without expertise could not design code for Web scraping. According to embodiments, a user intuitively selects such a web scraping object on a graphical user interface basis, and sets an event to be performed on the selected object. Setting up a series of events for each of one or more objects is referred to as a scrapping scenario configuration, which allows a user to construct scenarios easily and intuitively. Then, if a cycle for executing the scenario is set, the robot automatically scrapes the Web according to the same scenario every cycle. Accordingly, the user can create a web scraping scenario as if he or she recorded a macro service in an editing application. Web scraping can be performed automatically by setting a predetermined period and / or a repetition frequency.

DOM control of Web scraping software

In the web scraping interface according to the embodiment, an object to be an object to which an event is to be applied is called a Document Object Model (DOM). The DOM is an api (application programming interface) that processes html and xml documents. Html is a document consisting of many tags such as head, body, table, tr, td, etc. These tags are all made to create a visual representation that users can see. The DOM means a part of a web document made up of this tag, and the user can select a user interface device (illustratively, keyboard, mouse, touch screen, etc.) Drag, touch, etc., but not limited to this). According to embodiments, a web screen is implemented in an i-frame, which is a separate browsing frame, so that a user can select and control an individual DOM. When a specific DOM is selected, a plurality of events applicable to the corresponding DOM are provided on the right menu screen. When the user selects one or more events to complete an event to be applied to a specific DOM, It is added as a scenario line. Hereinafter, embodiments will be described in more detail with reference to the drawings.

Example system configuration

The system according to one embodiment may be configured as shown in FIG. The web data collection system includes a web data search area 100 and a web data collection area 140. The web data search area 100 includes a target site management unit 110, a data collection rule management unit 120, a control server 130 and a data collection robot 150. The web data collection area 140 includes a data collection rule creation unit 180 and a target site setting unit 190. [

The target site management unit 110 receives information on the target site as a request signal 170 from the target site setting unit 190 of the web data collection area 140 and stores the information, Can be provided. The data collection rule management unit 120 receives information on a data collection rule as a request signal 170 from the data collection rule creation unit 180 of the web data collection area 140 and stores the information, Information about collection rules can be provided. The control server 130 may receive information on the target site from the target site management unit 110 and may control the data collection procedure so as to download data from the target site. The control server 130 will be described in detail with reference to FIG.

The data collection robot 150 receives information on the target site from the target site management unit 110 and receives information on the data collection rules from the data collection rule management unit 120 and periodically and / It is possible to collect target data from the target site repeatedly according to data collection rules.

The data collection rule generation unit 180 may generate data collection rules through a mouse click, a keyboard input, and the like using the user interface of the web data collection system provided by the user. The data collection rule creation unit 180 receives an event that the user specifies for the DOM selected by the user through the graphical user interface. Such an event may be one, but preferably a plurality of events may be generated as a scenario with a sequence of scenes. The user may also set up a dedicated web and / or an app-to-site site for the app, in which case the target site setting unit 190 handles this setting. The unit 190 can set a target site for collecting web data, using a user interface of the web data collection system provided by the user.

The block diagram shown in FIG. 1 may refer to the division of a physical element, but may be a logical division, not a physical division. Thus, it is also possible for one or more units to be implemented together by substantially the same hardware element. Therefore, it should be understood that the constitution performing substantially the same function is not limited to FIG. 1 which is an exemplary block diagram but is also included in the scope of the invention.

Implementation of 'i-frame'

According to one embodiment, a web page to be scrapped (hereinafter referred to as a 'target page') may be divided into a separate frame (an 'i-frame' Quot;). A web data collection system frame 200 is provided to render a target page in an i-frame, where DOM selection and control event selection are enabled. 2 is a diagram illustrating an implementation frame 200 of a web data collection system according to one embodiment. Such a frame 200 may include a graphical user interface (GUI) for allowing a user to easily and intuitively select / control a target site and a DOM included therein, and to select an event to create / manage a web data collection scenario Lt; RTI ID = 0.0 > web-based, < / RTI > For example, the frame 200 may be provided by a dedicated program based on an APP (APP). Although there is no mention in the following, the form and the manner of providing the frame 200 should not be construed to be limited to some of the exemplified applications.

The frame 200 that allows a user to provide web data collection on a graphical basis includes a target site display unit 210, an input event display unit 220, and a data collection rule display unit 230. By way of example but not limitation, the target site display unit 210 may be the i-frame mentioned above. The web data collection system is based on a graphical user interface. The target site to which the web data is to be collected may be recognized as a converted path through the proxy server and displayed on the screen. The proxy server and the proxy site using the proxy server will be described later in detail. The target site display unit 210, which is an i-frame, simulates and renders a target site, and provides a graphic that displays a selectable DOM in the target site, and can select and select the target DOM. Such graphics can provide graphics such as highlighting the DOM, or displaying an outline when mouse-over, for example. For a selected DOM, it may provide a graphical process that is persistent and distinct from others until a temporary or other DOM is selected.

The input event display unit 220 may be a menu window including a plurality of submenus. The input event display unit 220 may display a data collection rule event applicable to the DOM selected by the user. For example, it can represent input events such as corresponding mouse click, drag, keyboard input, touch. It is possible to provide all events regardless of the selected DOM, but preferably only those that are applicable to the selected DOM can be displayed. In addition, it is also possible to display the entire event menu, to activate only those that can be applied to the selected DOM, to enable selection, and to deactivate other event menus not selected. When a user selects a particular event to appear on the target site display 210, it is created with a rule that applies that event to the selected DOM, which may be reflected as part of the scenario. Illustratively, one event line can be added to the scenario if the DOM selection and event selection for it is performed. The user can select the DOM in this way and intuitively select the events to apply to it, and the actual code line for it is automatically generated, so that even if the user does not have expertise in computing code for html or web scraping, Lt; / RTI >

As described above, the data collection rule display unit 230 may be a display unit of a scenario in which the flow of individual processes for applying a specific event to a specific DOM is displayed in a time-series manner. When a user views a target site that is rendered in an i-frame, selects the DOM included in the target site, and selects an event from the menu, it is added to the scenario one by one as an individual rule. The scenario displayed in the data collection rule display unit 230 can be subsequently edited such as delete, add, edit, change order, and so on for each line and / or related line group. The type of such editing is not limited to the illustrated ones and may include other things as well as to improve the data collection efficiency in the process of scraping the web data and / or to improve some errors or necessary as the user needs change . The editability of the scenario itself, which is provided in accordance with the embodiments, has great advantages because it provides the flexibility to change the web data collection rules intuitively, even if the non-developer does not understand the source code.

Rendering of the target site, menu processing for the selected DOM, creation and editing of the scenario, etc. will be described in more detail with reference to the following drawings.

Display of target site and selection of DOM

FIGS. 3 through 5 illustrate a case where a user selects a DOM while viewing a target site and a DOM included therein, in a frame layout presented by the web data collection system to a user, creates an event rule to be applied thereto, and creates a scenario Shows a series of processes going on.

3 includes a target site display unit 300 called i-frame, an input event display unit 320 also called a menu window, and a data collection rule display unit 330 called a scenario window. When the user selects a target site, for example, "http://www.goole.com", the target site is rendered in the i-frame. According to one embodiment, the general-purpose browser itself does not browse the actual target site, but rather renders the proxy site that simulates the target site. It bypasses the same source policy of W3C, which will be described later, and enables site rendering, DOM selection and data collection for sites that do not comply with web standards.

A target site may contain multiple DOMs. For example, they can be individual DOMs, such as text, images, flash, etc., objects with hyperlinks, and text input windows for entering queries.

When a user takes an action to select a DOM, such as mouse-over, target hovering, touching, or clicking, on the target site display unit 300, which is an i-frame, The selected DOM can be graphically processed to distinguish it from other parts. Such graphical processing may be a variety of things, such as highlight processing, outline display, color or shade change, blinking, etc., as described above, and should be understood as any manner of graphical processing that can distinguish the currently selected DOM from the others . In the illustrated example, DOM, which is a text input window for inputting a text query, was selected and the outline 310 of the selected DOM was displayed. Therefore, the user can intuitively recognize which DOM he or she has selected.

Event menu applicable to selected DOM

4, when a text input window DOM to be graphically processed 410 in the i-frame 400 is selected, the input event display 420, which is a menu window, displays a plurality of event menus applicable to the selected DOM / RTI > If a user selects a text input event 440 as shown, a rule for inputting a text for the corresponding DOM is generated and a scenario line 450 is created in the data collection rule display unit 430, which is a scenario window. The scenario line is provided with an edit button 460 that allows the individual event to be deleted, allowing the user to undo the scenario line creation. In addition, although not shown, edit buttons such as changing the order of the scenario events, copying and reproducing specific events, and the like can also be provided.

Additional settings within the scenario line

As shown in FIG. 5, the user may additionally input specific text for the text input scenario in the scenario window 530. FIG. If the event selected by the user is a text input, it is a process of inputting / setting subsequent to which text to insert. Illustratively, when the user inputs text into the DOM as "KBS 9 o'clock news ", the user can input it into the input window 540. The user then selects the search icon DOM 510 in the i-frame 500. Then, events applicable to the currently selected DOM are displayed in the menu 520 window 520, and the user can select a specific event 550 'click element'. Then, a scenario line 560 for the corresponding DOM is added to the scenario window 530 again. This process allows users to configure / edit scenarios for scrapping Web data intuitively and easily.

Overcoming the W3C Same Origin Policy and Targeting Site Rendering

According to one embodiment, the system does not browse the target site itself in the i-frame, downloads the target site, sets it as a proxy site, and renders the proxy site in the i-frame. In this process, the user may not be able to distinguish whether the rendered page is a real target page or a simulated page. In this sense, this process can be called target site simulation in i-frame.

There are several reasons why such an embodiment is provided. First, it is to overcome the same source policy of W3C. The same-origin policy of the World Wide Web Consortium (W3C) specifies that documents and scripts loaded from one source are not allowed to interact with resources from other sources . Therefore, it is difficult to provide a dedicated frame for web scraping, and hyperlink, flash, or Java based scripting may not be normally rendered in a separate simulated frame. Also, if the target site does not conform to the web standard, it may be the reason why the target site is not normally copied in the frame for web scraping. In particular, Web sites (for example, portal sites or media content providing sites) that are serviced in the Republic of Korea often do not comply with Web standards. Therefore, existing web crawler bots are often mistaken for collecting data and collecting such websites. Another possible cause is a script that takes a loop loop, such as self-referencing within a DOM object.

For some reason, the existing Web crawling services did not properly map the target site, and the DOMs could not be displayed at all. However, the system according to the embodiments simulates the target site through the proxy setting to prevent such error. FIG. 6 illustrates a Web data proxy routing and link path modification process according to an embodiment.

PROXY site / page setup

6, the control server 620 includes a proxy dynamic setting unit 624, an HTML downloader 622, an HTML parser 626, and an HTML converter 628, as shown in FIG. The proxy dynamic setting unit 624 can perform a proxy dynamic setting 635 with a proxy server 630 to download HTML resources from a target site. The HTML downloader 622 may download the HTML resources required to control the target site in the web data collection system. In this download process, some objects may not be downloaded due to various reasons such as the above-mentioned same source policy. According to one embodiment, which will be described later, various alternatives overcome the incompleteness of such downloads and accurately simulate the target pages.

The HTML parser 626 parses the path of the HTML resource to render the resource path on the HTML resource to the proxy server 630 via the proxy server 630, Path, and recognizes the target site through the converted path and outputs it to the screen. Through this, the target site recognized through the converted path is output to the output screen of the web data collection system. Thus, a state is provided in which the user can select and control the DOM while viewing the target site equally within the web data collection system.

HTML converter 628 may convert the resource path on the HTML resource to a proxy path. The HTML resource path conversion of the target page through the proxy server will be described in detail. Illustratively, but not exclusively, the system can receive the URL of the target site page, set it on the proxy server for that host, and proceed with key generation. This key is an identifier for distinguishing individual pages newly set in the proxy server corresponding to the corresponding URL. If a key is added to the proxy site path, a proxy URL dedicated to the target site can be generated. For example, if 'http://www.A.com' is the target site, a key for 'http://www.A.com' can be generated. For example, if the key is "abc", then 'http://proxy.B.com/abc' can be the dedicated proxy URL of the target site 'http://www.A.com'. In this case, B may be a host of a web data collection system that provides services according to one embodiment.

On the other hand, the modification of the proxy path of the web data will be described. The proxy dynamic setting unit 624 of the control server 620 may perform the proxy dynamic setting 635 and the proxy server 630 to download the HTML resource from the target site. The HTML downloader 620 may access 650 the target site 660 and download 640 the HTML resource from the target site 660. The downloaded HTML resource can be used to control the target site in the web data collection system. The HTML parser 630 parses the path of the downloaded HTML resource and converts the resource path on the HTML resource to the proxy path through the proxy server 630 without recognizing the downloaded HTML resource in the web data collection system as it is And the target site is recognized through the converted path so as to be displayed on the screen of the web data collection system. The resource path of the HTML resource may be converted to a proxy path through the HTML converter 628. [ The HTML resource path converted to the proxy path can be transferred 610 to the web data collection system. The web data collection system recognizes the target site through the HTML resource path converted to the proxy path, displays the target site screen on the target site display unit 605 of the web data collection system frame 600, Access.

PROXY Reduces resources by preventing redundant path setup

Meanwhile, according to an exemplary embodiment, the once generated proxy URL can be reused. If different users are allowed to generate proxy URLs for the same target URLs, this can be counter to the efficient use of resources. Therefore, when a key is assigned to a once generated proxy URL, it is separately managed, thereby preventing duplication of proxy settings for the same target URL. For example, if the first user has set 'http://www.C.com' as the target site and the key is set to 'def', then this key is managed, then the second user is' When http://www.C.com 'is set as the target site, the target user site of the second user can check the managed key to recycle the preset proxy URL. Thus, server resources can be significantly reduced through this consideration.

System flow for scenario creation

7 illustrates an operational flow of a web data collection system according to one embodiment. The web data collection system includes a target site path modification step 700, a user's web data collection system control step 720, and a data collection step 740. In step 703 of the target site path modification step 700, the user can access the target site for collecting data through the web data collection system and designate the URL of the target site. In step 706, the control server of the search area of the web data collection system can download the HTML resource of the target site designated by the user through the HTML downloader. Then, in step 709, the control server may convert the path of the downloaded HTML resource after proxy dynamic setting to a proxy path using the HMTL parser and HTML converter.

Then, in step 726 of the web data collection system control step 720, the target site recognized as the converted path may be output to the target site display unit of the web data collection system frame to be controlled by the user. In step 724, the web data collection system frame is provided with a DOM event control environment in which a user can generate a web data collection rule through a target site display unit, an input event display unit, and a data collection rule display unit. In step 722, the user can control the DOM using the user interface in the provided DOM control environment and select an input event to generate a data collection rule.

Next, in step 742 of the data collection step 740, the collection area of the web data collection system creates a data collection rule database for the data collection rules generated by the user. In step 746, data can be periodically and repeatedly searched according to the data collection rules generated by the user. In step 748, the HTML resource of the result page for the found data may be downloaded. Then, in step 750, the path of the HTML resource can be converted according to each file type of the resource such as image, css, etc. with respect to the downloaded HTML resource.

Then, in step 728 of the web data collection system control step 720, the result page is reproduced with the converted path and output to the web data collection system frame to provide the data collection result to the user.

The behavior of the data collection bot

FIG. 8 is a diagram showing a configuration of a data collecting robot according to an embodiment. The data collection bot 800 may be comprised of a search performing robot 810 and a data repository 820. The search performing robot 810 can periodically and repeatedly collect web data at predetermined time intervals according to the data collection rules to be generated. The collected web data may be stored in the data store 820.

Upon receipt of the data output request 840 from the user terminal 850, the web data collected in the meantime and stored in the data store 820 may be immediately retrieved and transmitted 830 to the user terminal 850.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the drawings, various modifications and variations may be made by those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (18)

A method of operating a web data scraping device, the method comprising:
Rendering a target site designated by a user and rendering the target site on a target site display unit;
Performing graphic processing to distinguish a first DOM selected by a user among at least one DOM (Document Object Model) included in the rendered result, from other objects; And
Displaying at least one user input event that can be applied to the first DOM in a form of a menu on an input event display
≪ / RTI > The method according to claim 1,
Adding a first scenario line corresponding to a rule for applying a first event selected by the user to the first DOM among the at least one user input event to be displayed to the data collection rule display unit and reflecting the added first scenario line to the scenario
≪ / RTI > 3. The method of claim 2,
Displaying at least one editing interface on a data collection rule display unit, the additional input interface for the first scenario line, the deletion for the first scenario line, and the change of enforcement order for the first scenario line
≪ / RTI > The method according to claim 1,
A proxy setting step of downloading an HTML resource necessary for controlling the target site, converting at least one resource path on the HTML resource into a proxy path of the proxy server and creating a proxy site corresponding to the target site,
≪ / RTI > 5. The method of claim 4,
The proxy setting step includes generating a first key that is a unique identifier corresponding to the URL of the target site and managing the proxy site in association with the first key. 6. The method of claim 5,
If there is another proxy setting request for the URL in the target site, checking the first key and returning the proxy site through the first key. A target site managing unit for downloading the target site information and rendering the target site information on the target site display unit when the URL of the target site selected by the user is received; And
Displaying at least one event selectable for a first DOM selected by a user among at least one DOM (Document Object Model) included in the rendered result on an input event display unit, 1 < / RTI > data collection rule management unit < RTI ID = 0.0 >
The web data scraping device comprising: 8. The method of claim 7,
Wherein the data collection rule management unit provides the first data collection rule to a data collection rule display unit in the form of a first scenario line. 9. The method of claim 8,
Wherein the data collection rule management unit displays the first scenario line on the data collection rule display unit and displays the additional input interface for the first scenario line, the deletion for the first scenario line, and the execution for the first scenario line And displaying at least one editing interface of the change order. 8. The method of claim 7,
Downloading an HTML resource required for controlling the target site, converting at least one resource path on the HTML resource to a proxy path of the proxy server, and setting a proxy site corresponding to the target site,
Lt; / RTI > 11. The system of claim 10, wherein the control server comprises:
A proxy dynamic setting unit configured to perform a proxy dynamic setting to download HTML resources from the target site;
An HTML downloader for downloading HTML resources necessary for controlling the target site in a web data collection system;
Parsing the path of the HTML resource), converting the resource path on the downloaded HTML resource into a proxy path, recognizing the target site through the converted path, and outputting it to the screen; And
An HTML converter for converting the resource path on the HTML resource into the proxy path;
/ RTI > 12. The method of claim 11,
Wherein the HTML converter generates a first key that is a unique identifier corresponding to the URL of the target site and manages the proxy site in association with the first key. 13. The method of claim 12,
And if there is another proxy setting request for the URL in the target site, checking the first key and returning the proxy site through the first key. A program for performing a method of providing a web data scraping service stored on a computer readable recording medium, the method comprising:
Rendering a target site designated by a user and rendering the target site on a target site display unit;
Performing graphical processing so that the first DOM selected by the user is distinguished from other objects among at least one DOM (Document Object Model) included in the rendered result; And
Displaying at least one user input event that can be applied to the first DOM in a form of a menu on an input event display
Readable recording medium. 15. The method of claim 14,
Adding a first scenario line corresponding to a rule for applying a first event selected by the user to the first DOM among the at least one user input event to be displayed to the data collection rule display unit and reflecting the added first scenario line to the scenario
Further comprising a computer readable medium. 15. The method of claim 14,
A proxy setting step of downloading an HTML resource necessary for controlling the target site, converting at least one resource path on the HTML resource into a proxy path of the proxy server, and setting a proxy site corresponding to the target site Further comprising: 17. The method of claim 16,
Wherein the proxy setting step includes: generating a first key that is a unique identifier corresponding to a URL of the target site, and managing the proxy site in association with the first key. 18. The method of claim 17,
And if there is another proxy setting request for the URL of the target site, confirms the first key and returns the proxy site via the first key.

Images