GR1010585B - Web crawling and content summarization - Google Patents

Abstract

The web crawler prioritizes web crawling and generates news summaries for news topics of different structure and content type and incorporates comments, posts and all kinds of interaction from present social networking and blogs. It uses automatically generated web wrappers to detect, analyze, and extract web page content based on their structure and content using XPath expressions, neural networks, and natural language processing and understanding. Web wrappers are evaluated for crawling effectiveness and automatically updated when the structure of web page matches changes. Web crawling policies are also constantly updated to reflect reputation, influence, impact, and mentions of a web page/website. Detection uses multiple parallel queues which incorporate different micro-tasks each andconverge to a priority which is used for detection according to the detection policies. The crawled content is analyzed, aggregated and summarized.

Description

Web Crawling and Content Summarization

DESCRIPTION

Background: Web crawling is a widely used mechanism for searching for content on websites and web pages on the Internet. It is widely used by search engines to paginate the Internet to facilitate subsequent searches, or is used specifically to collect content related to some parameter, such as a news topic.

Web crawling typically uses crawling policies that define what, where, and how to search for and retrieve web content, and often involves prioritizing access to web content sources to make better use of scarce and finite crawling resources.

Building and maintaining a large-scale web crawler and content aggregator is a painstaking task that requires constant updating and adaptation. Typically, many web content sources do not use automatic feeds, and the structure of their web pages varies significantly between web sites and changes from time to time. All of these aspects make web crawling the need to constantly create and modify web page wrappers used to crawl web pages. Such a process is either done manually or semi-automatically, requiring significant manual input and oversight on a regular basis.

Furthermore, the creation of crawling policies is typically done based on the frequency of website visits, thus ignoring other important qualitative characteristics related to the actual content of the website.

Similarly, web content should be summarized for better presentation and readability. Current approaches are manual or semi-automated, with inherent human intervention costs and delays when managing massive amounts of data.

Problem Definition: A crawling engine is needed that can automatically generate, and continuously adapt, crawling policies and web wrappers with the goal of automatically crawling and thematically summarizing large volumes of web news content from various sources.

Proposed Solution - Abstract: A system and method for web crawling and content summarization is presented.

As used herein and in the appended claims, references to the singular include references to the plural unless the context clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains (detection webs, Internet, computers, data processing). Although any techniques similar or equivalent to those described herein may be used in the practice or testing of the invention, the preferred techniques are described herein.

This innovative web crawling solution focuses on crawling for news and social media content that is not available for retrieval through Rich Site Summary or Really Simple Syndication (RSS) feeds or other structured and automated mechanisms.

This innovative solution implements a source ranking mechanism for all content sources, which takes into account the influence, reputation and impact of the source and prioritizes the most influential updated news sites by topic content. The innovative solution takes into account the number of citations from other sources (including news websites and social media) to articles published by a first site (e.g. news site) and the number of articles (on the first or other sites) that discuss the same event/topic/entity after it has been first discussed by the first site's articles. The present detection solution uses multiple parallel queues for the elements to be retrieved and multiple threads consuming the queues. These threads and queues are implemented in either software, hardware, or a combination thereof, and are either local to a computing device (eg, server where the present detection solution is running) or distributed across more than one computing device. In alternative exemplary embodiments, the computing devices may be selected from servers, desktop computers, laptops, tablets, smartphones, and the like.

Since the resources of a crawler are not unlimited, configuring a crawling solution for collecting content across websites usually involves prioritizing what to crawl, when, and how many resources to allocate to each content source (whether it's a site with a collection of similar of web pages or a single web page within a website).

Typically, prioritizing web crawling tasks involves using the popularity of web content and web sources to crawl. However, it is also important to allocate resources not only to web sources that update their content frequently (as web crawlers usually do) but also to influential web sources (e.g., web sources that are widely accepted as reputable sources, viewed preferably etc.). Considering the influence of content sources can help increase the efficiency of the crawling process and the impact of crawled content, which can then be processed to create aggregated content served to end users.

In addition, the present innovative solution automates the process of collecting content from web sources that do not use RSS or other structured and/or automated techniques for content collection. For each specific web source, this innovative solution uses a combination of machine learning and rule-based techniques to identify text segments of special interest (e.g. article title, body, links to articles, links to article categories) and automatically determine tracking policies for each web content source. Crawling policies (page wrappers, crawl priorities, etc.), which differ between web sources, are created automatically the first time a source is crawled and are updated when the structure of a source changes (e.g. when the structure, elements and their order is modified in a web page or website) .

In the absence of a common method (such as RSS) to access this content, existing news crawlers are either limited to sources that provide RSS feeds or manually create and maintain different crawl policies (wrappers, priorities, etc.) for each source Web. Different crawl policies are required as each news source uses its own standard for serving content. The work involved in both manually creating and maintaining crawling policies is large, and this limits companies that offer web crawling and web content processing and aggregation services from aggregating hundreds or thousands of sources. Despite the advances in automatic wrapper maintenance found in the literature, initial wrapper generation still remains a manual task.

Therefore, there is a need for an automatic solution to create and continuously update web crawling policies. Such a method should automatically adapt to changes in the structure of Web content sources, their impact, and their priorities.

The present novel solution provides an elegant solution to the above problem by teaching a new page wrapper creation and maintenance mechanism, combined with a new detection priority mechanism based on the impact and reputation of the web content source and implemented with a number or parallel queues that they are consumed to form a terminal queue used for continuous tissue detection. The content collected by the crawling process is continuously aggregated with an innovative content clustering/summarization method, which aggregates similar articles (discussing the same event, topic or entity), identifies the main sub-topics discussed by the articles in the cluster and generates suggestions that describe each subtopic. A grouping/summary section can generate new content from multiple reports of an event in traditional web media. The clustering/summarization module combines an extraction step and an abstract summarization, where the first step selects the topics of a cluster and the most representative groups of sentences for each topic, and the second step composes new lexical chains covering each topic in the cluster. Although the summary extraction step has been extensively studied by researchers and commercial systems, the abstract step exploits the repeated content in the news cluster to generate more comprehensive summaries than existing seq2seq models.

In the case of hundreds or thousands of web sources, the content overlap is huge and the grouping mechanism is important for presenting the news content. Users interested in an event get a quick idea from the automatically generated summary and can then browse at their own time the articles grouped in the same cluster. This automatic summarization process improves the reading experience and enables the development of new content delivery services, e.g. news services that provide a summary of the news and the summary is continuously updated taking into account not only the hit rate or web sources but also the impact, influence and other parameters of the web sources and includes social media content, data and metadata related to original web content.

Architecture of web detection system

Figure 1 shows a schematic diagram of an exemplary implementation of the proposed innovative detection system. The detection system (100) includes at least one server detector in communication with at least one web server and optionally with at least one computing device. In the illustrated exemplary embodiment of Figure 1, a first detection server (160) is connected to a database (165). The database (165) is implemented either internally in the detection server (160) or externally, e.g. as a networked database server or other configuration obvious to a reader of ordinary skill in the relevant art. The database (165) includes various types of data such as crawl policies, crawl results, crawl queues, etc. This data can be stored encrypted, unencrypted, compressed, descriptive (eg, an XML type or other descriptive language can be used), or a combination thereof.

The detection server (160) has access to the first database (165) for reading and writing policies and detection data during the detection processes and displaying the results to the stakeholders. The detection server (160) is also connected to a second database (170), which database may contain additional policies and/or detection results and is configured similarly to the first database (165) and to an optional second detection server (150). The detection server (150) is also connected to the first database (165), the second database (170), and (in an alternative embodiment) a third database (155), which third database (155) is configured similarly to the first database (165) and contains additional policies and/or detection results.

The first (160) and second detection servers (150) are configured to operate in parallel and speed up the detection process. In one aspect, the first detection server (160) acts as a master, running software that manages the parallelization of the detection process and the workload and results of the two detection servers. In another aspect, the second detection server (150) is the master, while in another aspect, the switching of the master state during the detection mode occurs as a result of changing operating conditions (e.g., a decrease in server processor performance as result of overheating, reduction of network performance, etc.).

In an alternative embodiment, additional databases and trace servers can be added, or in cases where massive parallelization is required, trace servers can be used either locally or in a cloud infrastructure. In yet another exemplary embodiment, only the first crawl server (160) and the first database (165) are used, and all processing and storage needed for web crawling is implemented in these two system components (100).

The first and second detection servers (160), (150) communicate via any available type of wired or wireless network (140) (cloud infrastructures are an example of such a network) with a first web server (110), which web server hosts at least one website or web portal consisting of at least one web page. In practice, the first web server (110) hosts dozens of web pages or portals, each consisting of tens or hundreds of web pages stored in databases or file systems local or remote to the first web server (110).

In another aspect, the first (160) and second (150) detected servers are also connected to a second web server (120) and a third web server (130) configured similarly to the first web server (110). In alternative exemplary embodiments, there are additional web servers that may be configured as a web server cluster or otherwise.

The scan results are stored by the first (160) and second (150) scan servers in local storage (the scan servers) or databases (165), (170), (155). These results are accessible directly from the detection servers (160), (150) or more commonly from a computing device (eg, a desktop computer (195), a server (not shown), a laptop (not shown) , a tablet (175), a smartphone (185) or other electronic device (not shown) with sufficient computing power such as an Internet of Things (IoT) network device, etc.). Such computing devices access the detection results via a fourth web server (180), which server is connected to the database (170), which database is connected via servers (160), (170) to databases (165), (155). In an alternative embodiment, the foregoing computing devices access the detection results through any of the first (110), second (120), or third (130) web servers.

Alternative embodiments of the detection system (100) may be used where some elements are omitted or merged and others are introduced. As a basic configuration, one can consider a single crawl server, a network infrastructure, and a web server. Various configurations of the system (100) may also include cloud-based components while the components depicted in Figure 1 may be physical or virtual.

Hardware architecture

Figure 2A shows a hardware architecture block diagram for the servers and computing device units of the sensing system (100). The hardware element (200) comprises a series of modules (210)-(290). These modules can be replaced by other similar modules, and some or all of these modules can be duplicated to ensure uninterrupted operation in the event of a module failure, or to increase the capacity and capabilities of a module, as in the case of a memory module ( 240). Modules (210)-(290) are interconnected via a data bus and a control bus where a suitable logic unit acts as a master unit.

In alternative exemplary embodiments, some or all of the modules (210)-(290) are equipped with microprocessors or microcontrollers and any one of them can become a master that communicates with other modules via a multi-core bus, e.g. a controller area network (CAN) bus.

The hardware unit (200) includes a central controller or processor (250), which contains digital logic and/or software for integrating data from other units and making operational decisions, one or more memory units (240) (these can implemented as volatile or non-volatile memory, hard disk or other type of data storage device) to store the data to be handled by the processor (250) or exchanged between the unit (200) and other units of the system (100), the communications unit (280) for communication with other system units (100) and battery units (260) for power supply unit (200) (can be implemented in the form of a simple rechargeable battery, a smart battery, a Universal Power Supply (UPS ) or other similar technology). Depending on the type of hardware module (200), additional modules may be included, such as the graphics module (210) for generating graphical content for display to the user (200) of the module, the display (220) for displaying the graphics from the module (210), the interaction/data input module (230) that allows the user to interact, the microphone (290) for recording the user's voice commands or ambient sound, and the camera (270) for capturing visual content such as screenshots, page scans, user gestures for interaction, etc.

The units of the system (200) may be reduced, merged or new units added depending on the type of implementation of the system unit (100), e.g. a camera and microphone are included in a smartphone but not in a web server or crawl server.

In an alternative exemplary embodiment of unit (200), units (210)-(270) and (290) are connected to communications unit (280), where the communications unit is responsible for managing communication in unit (200) between units (210)-(270) and (290), handles the control and data bus and the execution of communication tasks outside the unit (200) with other units of the system (100).

Each unit (200) may be in the form of a commercially available computing device (server, desktop, or computing device) or an embedded system, i.e., a microcontroller-based system (i.e., Central Processing Units (CPUs) with built-in memory or peripherals interfaces). In embedded system unit implementations, the processors used are of types ranging from general-purpose processors to processors specialized in certain classes of computations, or even custom processors designed for the present application. In one aspect, specialized processor chips and systems are embedded Application Specific Integrated Circuit (ASIC) chips or systems. Both design solutions i.e. general purpose microcontroller systems and ASICs run embedded software and each module can have its own system clock, bus communication capabilities, bus controller etc. Such a design allows simple plug-n-play connection of any of the system (200) modules (210)-(290) and replacement of any module with a replacement module in case of module failure or with a module suitable for a specific use case or customer, in accordance with local regulations, marketing and service agreement-related concerns and choices, or with newer or improved versions and types of modules as they become available. Additionally, such a system design also allows for hot (ie, on-the-fly) replacement or addition of system units without interrupting system operation (200). This is especially important to not interrupt system logging, diagnostics, and monitoring, and therefore not to compromise integrity and security.

The memory unit (240) may be implemented in any known storage technology including, but not limited to, magnetic disks, optical disks, flash memory, random access memory (RAM), read only memory (ROM), erasable programmable only memory read-only (EPROM) or other, and may copy for performance, redundancy or other reasons. In an exemplary embodiment, all storage units are of the same type, while in an alternative exemplary embodiment, a combination of storage units of different technologies may be used. New technologies may be used in storage units as these technologies become available.

The graphics unit (210) is selected to match the capabilities of the display (220), which display may be of any type, including an alphanumeric display, a monochrome or color display, a touch screen, a 3D display, or another type of display. Likewise, the interaction/data input unit (230) may include key input, touch screen input, gesture input, remote device (eg, portable computing or interaction device) wirelessly or via wired input communication, or other input technology.

The battery module (260) is responsible for supplying power to the module (200). In an alternative embodiment, the battery module (260) also provides power to the system buses, which buses then provide power to all other system modules (200) that do not have a battery module. In one aspect, the battery module (260) includes a power management module which can regulate power and perform functions such as power off if instructed by the processor (200), (250) of the module. For example, a power outage may occur if the system detects hazardous operating conditions that may damage the system hardware, operation is prohibited for any reason (eg, unauthorized access), etc.

In alternative implementations the battery module (260) may also include the logic to make decisions (eg, to cut power) on its own in situations where the processor module (250) has failed, based on data from sensor modules ( not depicted in Figure 2A but optionally included in some exemplary embodiments) or commands received from other computing devices in the system (100).

The communications module (280) allows the module (200) to communicate with other modules and/or systems. The communications module (280) is in the form of an input/output (I/O) module (for wired or wireless data communication between the module (200) and other modules (200) in the system) or a data/control bus driver/transducer (for wired data/control signal communication between modules (210)-(290) of module (200)).

The communication modules (280) include hardware modules for I/O with at least one fast wireless, wired and optical network data transfer. Wired connections are preferred over wireless connections due to the significantly higher performance of wired connections over wireless connections, although wireless data transfer could be used in alternative applications. By way of example, the wired connection may include any fast wired data bus or preferably an optical link. The exact protocol of the selected link is beyond the scope of this novel solution and any such protocol can be used.

The processor unit (250) is the main processing element of the system (200) and includes either an off-the-shelf processing chip or processor card, or an embedded processor unit, while in alternative embodiments the unit (250) includes a redundant set of embedded processors units to ensure uninterrupted operation in the event of processor failure. The firmware runs on the central controller unit (250), while in alternative multiprocessor implementations a hypervisor is used to control virtual machines running on the other processors (eg, in parallel processing configurations with multiple processor chips/cards).

Figure 2B shows an exemplary high-level hardware architecture of a processing unit according to the present innovative solution. The processor unit (250) consists of a series of processing sub-units. In particular, the processor module (250) has a policy detection module (251), a web page wrapper module (252), a detection module (253), a clustering module (254), a page parser (255) and a summary module ( 256). In alternative exemplary embodiments, subunits (251)-(256) may be merged, duplicated (eg, for speed or redundancy), or distributed across more than one processor unit (250) on the same computing device or networked devices (eg network servers, cloud infrastructure, etc.) .

In other exemplary embodiments, the processor unit (250) is designed to include more than one processing core. Each or some of the processing cores of the processor unit (250) contain all or some of the sub-modules (251)-(256). As a result of such multi-core designs, the steps of the present innovative solution are executed serially in the processing core when each processing core contains one or more of the subsections (251)-(256).

Alternatively, when each processing core contains all subunits (251)-(256), the steps of the present novel solution are performed in parallel, where each subunit (251)-(256) produces results for e.g. a type of entry sites etc.

Software Architecture

Figure 3 shows a software architecture block diagram for a non-server computing device of the system (100). At the lowest level of the software components (300) are the device specific capabilities (360) which are the commands to control the different hardware device components of the unit (200). Moving to higher levels are the Operating System (OS) (350), Virtual Machines (340) (such as a Java virtual machine), Device/User Manager (330), Application Manager (320), and at the top level the Applications (310). These applications may access, manipulate and display data. The software components (300) may be implemented in any preferred programming language (including descriptive languages such as eXternsible Markup Language (XML)) or in a combination of two or more programming languages of the same or different level.

Figure 4 shows the main software components of a server. At the lowest level of software components (400) is the OS Kernel (460) followed by the Hardware Abstraction Layer (450), Services/Applications Framework (440), Services Manager (430), Applications Manager (420) and Services (410) and Applications (470). The software components (400) may be implemented in any preferred programming language (including descriptive languages such as extensible Markup Language (XML)) or in a combination of two or more programming languages of the same or different level.

It is noted that the software and hardware components shown in Figures 3 and 4 are by way of example and other components may be present but not shown in these Figures, or some of the components shown may be omitted or replaced with others without removing from the framework of this innovative solution.

The present innovative solution may also be implemented by software running on the probe server (160) or on additional probe servers or on one or more distributed computers that are not displayed (e.g., cloud infrastructure, remote servers, forms servers, or other computing devices , etc.), or any combination thereof. The present novel solution can be implemented in any computer language or an abstract language (eg, a description based on metadata which is then interpreted by a software and/or hardware component). The software running on the aforementioned hardware effectively transforms a general-purpose or special-purpose hardware or computer into one configured to specifically implement the present innovative solution.

In the exemplary embodiment where the electronic units (200) run embedded software to perform their tasks, a layered architectural design is used without limiting the embedded software design to a particular instance. Other designs are used in other alternative embodiments.

Detection and summarization

Figure 5 is a high-level flow diagram of the detection and summarization process. For ease of description, Figure 5 describes an example implementation where a simple detection server (160) is used. It is assumed that the process can be extended to include additional detection servers, although this is not specifically detailed below.

The process (500) begins with step (510) where the detection server (160) receives the detection policies (or in alternative embodiments creates the detection policies) that describe what, how and where to detect. The crawling policies can be stored in local storage (memory, hard disk, etc.) of the crawling server (160) or in the databases (165), (170) or in the cloud. In alternative implementation examples, detection policies may be broken into subpolicies and distributed across more than one database, detection server, or cloud module. In such a case, sub-policies may be needed to compose the policy to be used by the detected server (160), or where multiple detection servers are used, a sub-policy is assigned (or more sub-policies are assigned) to each detection server.

After the detection policies have been analyzed, the detection server (160) proceeds to step (520), where the detection server obtains web page wrappers (or in alternative implementation examples creates and/or updates the web page wrappers), which have already been created and which stored in the same or alternate locations as the detection policies.

Web page wrappers (can be considered as elements of a crawling policy) are descriptions of the structure and content of a web page, where elements such as title, advertisements, article body, contents, links to other pages, images, etc. have been identified. Page wrappers are created by analyzing multiple pages of a website in terms of the aesthetic aspects and layout of page elements, text length (eg titles are shorter than article bodies), font types and sizes, placeholders etc. Other types of analytics are also used, such as Natural Language Processing (NLP) and Natural Language Analysis (NLA), machine learning, etc. Page wrappers can then be used to automatically process (detect) web pages of the same type, e.g. web pages on the same site or web sites and pages using the same or similar layout and design, etc. As a result, clusters of similar web pages can be crawled using the same web wrapper.

The crawl server (160) uses the retrieved web wrappers to crawl news sites (530) and news and article pages (540). All crawled news pages and articles are then aggregated (550) and analyzed (560) to generate news summaries by news topic (570).

Crawling news sites are part of crawling policies or are otherwise part of other data structures.

The high level flow diagram of Figure 5 is broken down in Figures 6-13 where more details are presented as to the implementation of steps (510)-(570).

Figure 6 is a flowchart showing how a detection policy is created and updated. The process (600) begins by retrieving an existing detection policy (610) from its storage location, if such a policy exists.

Table 1 shows an example of a detection policy. The sample detection policy is implemented as a configuration file. The configuration file contains, in the form of key-value pairs, all the information the wrapper needs to extract the appropriate content from a news source. The first part of the file configures the wrapper to access the news source homepage and extract links to articles. It also contains some general settings for the source, such as content language and encoding, and metadata such as source name, etc. The second part of the file contains instructions that configure the wrapper to extract specific content of interest from the article pages, e.g. as title, body, author, etc., which are in the form of Xpath expressions and additional settings for how the exported content is processed (eg date formats, tags to ignore, etc.).

#Mon Feb 02 14:53:10 EEST 2016

javascript=true

rss=http://www. protothema.gr/rss/news/general/

charset=utf-8

name=ProtoTheme

tagsToRemove=script,div

xpath body=//div[@class="article-content"]

xpathsurtitle=//div[@class="article-title"]/p

xpathsubtitle=//div[@class="article-content"]/h3 xpathauthor=//div[@class="article-info"]/span[@class="name"] jquerymidtitle=b

jquerycaption=

xpathlead=//div[@class="article-content"]/h3

xpathedition=

dateFormat=EEE, d MMM yyyy HH:mm:ss Z

bodyremoveregex=_ Enter your email to be the first to receive all the breaking news! Send\\s*Follow us at_ _(\V\\*< !) *?(>\\*\V)

parQuery=h3_(<br>)+

toReplace={ "<h3>" : "</h3>" : "","<br><br>" : "</p> <p>","<br/>":"</p> <p>","<br />":"</p><p>" }

presstitle= FIRST TOPIC

Table 1 : Example of crawling policy

Based on the crawl policy, the crawl server (160) knows what (eg, type/topic of news article), how (ie, what to look for in web page layout/content), and where to look (ie, what is the website/page URLs to crawl) and proceeds to calculate the impact of the web pages to be crawled by counting the number of references to an article on a site from other sites (620) and the number of mentions (or plays) of the article on social media and other news sites (630).

The crawl server then aggregates the frequency of article citations per site (640) and uses this frequency, along with additional data (posting rate, subject categorization of sources, history of source reputation, etc.), to estimate its reputation website in different thematic categories (650). This additional data includes previously calculated news site reputation data, negative comments on the site (eg negative user comments), sanctions by regulators against a site for poor journalistic practices, etc. For example, a website that publishes a lot of financial news/items, which are quickly replicated by other news sites or quickly discussed in the blogosphere or social media is a very reliable source in the financial sector, compared to others that only once published an important news item, among the dozens of articles they publish daily on different topics/fields. Unlike existing crawling techniques that prioritize news sources based on their update rate and regardless of domain, the current crawler focuses on source reputation per domain and results in collecting content more often from authoritative sources and indirectly increasing his authority's rating. The reputation of a web content source is also affected by source ratings, as these may be created by expert actors or by readers.

A list of previously crawled sites, if any, is retrieved (660) and the site reputation is used to create or update the crawl (prioritization) policy (670) to be used in subsequent crawl processes. For example, the detection policy will rank the trusted source of financial news higher than other sources of lower reputation and therefore crawl this source of financial news before and more often than crawling other sites with lower reputation and assigned priority.

It is noted that there may be one detection policy or multiple detection sub-policies that are created, updated and used according to one or more criteria, such as one policy for each detection server (if more than one detection server is used), or site, or cluster or sites, or news topic, or cluster of news topics, etc. Additional criteria for detection policies may apply, as will be apparent from this description to any reader of ordinary skill in the relevant art.

An innovative aspect of the present detection solution is that the solution uses Web Site Reputation to create detection policies and subsequently update detection policy and prioritize detection tasks as opposed to frequency of occurrence used in detection techniques mentioned in the prior art. Reputation is a safer and more robust metric as it allows to get rid of the influence of sites etc. that produce fake, low-quality, questionable news articles, which are either not replicated or have a rapidly diminishing interest. Unfortunately, with the widespread use of the Internet, thousands of websites report news that they know is fake or that has not been vetted according to standard journalistic practices (eg, never report a story unless it has been verified by at least three different sources or unless tangible evidence is available). In such cases it regularly proves after some time that such (fake) news has no value, however it can seriously damage personal, corporate or state reputation, breed advertising expenditure on sources with seemingly high visibility and deprive the sites that deserve that money, get it. Thus, the reputation measure proves valuable for selecting and prioritizing news sources while removing bias from “bad” sources. Another advantage is the ability to evaluate the performance of detection policies in real time and react immediately when a policy fails. For example, when a news site changes its layout or template, it is very likely that the crawling mechanism will fail. This failure is automatically detected and a self-healing mechanism is activated. The engine repeats the policy extraction process and adapts to the new site template.

Figure 7 is a flow chart showing how web wrappers used in crawling web and web content are created and updated. The process (700) begins by processing all web pages in all web sites (710) listed or allowed by the updated crawl policy (or sub-policies). Processing starts with the first page (720) of the first website and continues with the category pages (730) of the first website and all article pages (740) of the first website and repeats until all sites have been processed web.

To process the first web page (720) of the first website, all hyperlinks extracted (722) from the first page by the crawl server are classified into one of three categories (724) (ie external links located outside the website , links to category pages, links to non-category pages within the site, i.e. navigation pages, articles, etc.), links to category pages are grouped based on their position on the webpage (using the public ancestor <div > or other similar HTML nodes) and an XML Path Language (Xpath) expression (726) is created that identifies page areas with links to category pages.

XPath is a query language for selecting nodes from an XML document, defined by the World Wide Web Consortium (W3C), which in this discussion is related to the structure and content of a website/page. In addition, XPath can be used to calculate values (eg strings, numbers, or Boolean values) from the content of an XML document. XPath was defined by the World Wide Web Consortium (W3C). The XPath language is based on a tree representation of the XML document and provides the ability to navigate around the tree, selecting nodes by various criteria.

Using the Xpath expression creates a first page wrapper (728) (or updates it if it already exists so that it adapts to changes in the style and content of the first page). The new or customized first page web wrapper is then used in the next iteration of the process for the same web site (729), specifically in the step (722) where the links to the first web page are extracted. More than one web wrapper may be created and/or updated.

Link extraction (722) can be done by any known algorithm. For example, a text analyzer can be used to identify links and words and select useful links by checking for file extensions (eg .html, .jpg, etc.) or in another exemplary embodiment, syntax analysis ( e.g. <a> tags in HTML), NLP, NLA, Natural Language Understanding (NLU) or other known techniques (e.g. regular expressions, e.g. a sequence of characters defining a search pattern as in the regular expression "a." where "a" (one-one-one) is a literal character that matches only 'a', while is a metacharacter that matches every character except a newline).

Figure 14 shows an example of processing a first web page in steps (720). The first web page (1400) is edited using links in the category bar (1410) and links in the footer (1420).

One level below the site structure, to process the category page (730) of the first site, the links are extracted (732) from the category page by the crawl server, sorted (734) and grouped by the Xpath expression (736) . Links are now classified into one of three categories (ie external links to other sites, links to article pages, and links to non-article pages within the site) and similarly areas of pages that contain links to article pages are identified, after processing several category pages from the same website.

Link extraction (732) can be done by any known algorithm. For example, a text parser can be used to identify links and words and select useful links by checking for file extensions (eg .html, .jpg, etc.) or in another exemplary embodiment, syntax analysis (<a> tags in HTML), NLP, NLA, NLU or other known techniques (eg regular expressions) may be used.

Using the Xpath Expression a category page wrapper (738) is created (or updated if it already exists to adapt to changes in the style and content of the category webpage). The new or customized category webpage wrapper is then used in the next iteration of the process for the same website (739), specifically in step (732) where the links to the category webpage are extracted.

Figure 15 shows an example of processing a category page in steps (730). The category page (1500) is processed using top news content (1510), newsroom content (1520) and latest news content (1530).

Another level below the site structure, to process an article page (740) of the first site, the article content is extracted (742) from the article page by the crawl server, sorted (744) and grouped by the Xpath expression (746). This final step involves a combination of machine learning techniques (i.e. classification) to identify the elements of interest on an article page (e.g. <div> elements containing author information, article title, article text, etc.) and extract information and NLP techniques to extract the exact content of interest, avoiding noise (eg in text ads, etc.).

Extracting article content (742) can be done with any known algorithm. For example, a text parser can be used to identify large chunks of text (ie, body text as opposed to titles, etc.) and select useful text. of the article while ignoring other content on the web page (eg, image captions, headlines, advertisements that typically contain very short text message or no text at all).In another exemplary embodiment, syntax analysis, natural language processing, parsing and understanding, or other known techniques they can be used.

Using the Xpath expression creates an article page wrapper (748) (or updates it if it already exists to adapt to changes in the style and content of the article page). The new or customized article web page wrapper is then used in the next iteration of the process for the same web site (749), specifically in step (742) where the links to the category web page are extracted.

Step (740) is repeated for all article pages in the website and each extracted article content (742) is checked for validity (750). The check can be performed in several ways, manually or automatically. Manual testing can be done initially to train the server crawler and then switch to supervised or fully automatic mode. Automatic operation can be done using any suitable algorithm, with neural networks, machine learning and statistical techniques known in the prior art being prime candidates. Other techniques known in the art may also be used. The purpose of step (750) is to verify that no junk text has been received (eg, image captions, titles, advertisements that typically contain a very short text message or no text and are identified as article content.

If the content of the extracted article is valid (760), the content is preserved (stored in a database or other storage structure) and a decision is made (770) to store (780) the web wrappers for the first, category, and article pages from steps (728), (738), (748), respectively.

If the content of the exported article is not valid (760), repeat procedure (700) to re-export the content of the article and to update the 3-page wrappers (produced in steps (728), (738), ( 748)) until a positive decision is made in step (770).

Figure 16 shows an example of processing an article page in steps (740). The article page (1600) is edited using the title (1610), body (1620) and image (1630).

Figure 8 is a flowchart showing how web pages are crawled. The process (800) begins by selecting a crawling implementation (810) between sequential crawling (820) (where one crawling server is used to crawl all web sites), parallel crawling 1 (830) (where a single crawling server is used crawling per site or groups of sites), and parallel crawling 2 (840) (where more than one crawler is used per web site).

Having selected the crawl implementation, the crawl server (or primary crawl server when multiple crawl servers are used) creates Universal Resource Locator (URL) queues for site crawling (850). Preferably more than one queue is used per web crawl server to facilitate implementation of the various steps of the process (500). These queues are shown in Figure 13 and are processed (860) to create a single prioritized URL queue (870) used in site crawling. If parallel 1 or parallel 2 detection implementations are selected, one priority queue is created in step (870) per detection server.

Queues are implemented in software, hardware, or a combination of the two and contain ordered lists of URLs.

Figure 9 is a flowchart showing the steps for grouping news pages and articles. The process (900) begins by representing the extracted text of each new article received (raw text) using an intermediate representation model such as Bag-of-Words (BoW) (ie, a simplified representation used in NLP) and Vector Space Model or any other word embedding method for representing text (910). In one sense, this intermediate representation is a high-level representation.

Once the raw article text is mapped to the intermediate representation, it is forwarded to the article classifier (920), which decides whether the article is classified (ie, grouped) with the articles of an existing cluster or not (930).

Any text classification method can be applied in this step (eg Naive Bayes, Support Vector Machine (SVM), single or multi-layer convolutional neural networks (CNN) or Recurrent Neural Networks (RNN)).

Articles not classified into an existing cluster are forwarded to the article clustering process (940), which uses a text similarity method and a clustering algorithm to form new clusters (950). Articles that are similar to an existing cluster are attached to the cluster and the cluster is updated accordingly (960). The last update date is preserved with the cluster.

A cluster that has not been updated for a long time has been archived (970).

Figure 17 shows an example of a cluster of similar species. The example cluster (1700) contains a collection of similar articles (1710), containing 104 news articles, 9 blog articles, 1 opinion article, 88 media articles, and comments.

Figure 10 is a flowchart showing the steps for parsing news pages and articles. The process (1000) begins with a representation of the article's text and metadata (author, URL, media, etc.) and the metadata of the article's source site (domain name URL, site type, subject orientation, etc.). ). Deep and wide learning models are used to represent both textual information and metadata (1010). Any classification technique (eg, Naive Bayes, SVM, single or multi-layer CNN or RNN) (1020) can be applied to assign an article to one or more predefined categories (labels) (1030) and appends this information to article (1040).

Figure 11 is a flowchart showing how a news topic summary is created by aggregating subtopic text. Process 1100 begins by selecting all detected articles that have been grouped together because they discuss the same news topic (1110) and preprocessing those articles using NLP techniques (1120). The preprocessed articles are then analyzed using NLU techniques (1130) and the main subtopics discussed by the selected web articles of each cluster are identified (1140).

Having analyzed the web articles in each cluster, sentences describing the identified sub-topics are detected in the analyzed web articles or generated (1150) by mixing and matching sentence fragments using the NLU results.

The process (1100) ends by aggregating the sentences identified or generated by step (1150) into a summary of the news articles for each group of articles (1160).

Figure 18 shows an example news topic summary page. The news topic summary page (1800) contains news summaries (1810) related to a news topic (1820).

Figure 12 is a flowchart showing how a topic summary is generated with natural language synthesis (NLS) techniques. The process (1200) begins by selecting all detected articles into groups discussing the same news (1210) and preprocessing them using NLP techniques (1220). The preprocessed articles are then analyzed using NLU techniques (1230) and the main subtopics discussed by the selected web articles of each cluster are identified (1240).

The process (1200) ends by creating a summary of the news articles for each group of articles using the NLS techniques (1250) on the results of step (1240).

Figure 13 is a block diagram of the main logic modules and queues of the system (100). In the system (1300), the processor module (250) of the detection server (160) (and optionally all the detection server plug-in processing modules used in alternative exemplary embodiments) should be considered as a custom-made module (or modules in alternative examples implementation) specifically tailored to implement this innovative detection solution. From a hardware perspective, the processing unit (250) can be divided into several logical sub-units that each implement a specific task (e.g.,

analysis, prioritization, classification, etc.) as shown in Figure 2B. These modules can be real (ie in hardware), virtual (ie in software) or hybrid (ie a mixture of hardware and software) and implemented on a single computing device or on more than one computing device. Real sub-modules are built as ASICs or other technologies, while virtual sub-modules simply use existing processors to run specialized or embedded software that ultimately transforms the virtual sub-modules by simulating the behavior of purpose built hardware sub-modules. From a hardware perspective, hardware subunits are associated with (connected to) at least one hardware (or software, or combination thereof) queue. Hardware queues are implemented either in dedicated queue sub-units in the processor unit (250) or in the memory unit (240), either on the same computing device or on more than one computing device.

From a software perspective, the software or embedded software running on the processor unit (250) creates virtual modules and queues and performs the same function as described above for the hardware. This software approach may lack slightly in speed, but is more flexible as it allows for easy reconfiguration according to new performance needs and other system considerations (100). In alternative exemplary embodiments, a combination of hardware and software modules for processing and for detection queues is used to achieve higher processing speed and easier reconfiguration.

A seeding module (1310) generates a list of URL sources (ie URLs of web sites and web pages) to be retrieved during crawling. The list of URL sources is entered into the Seed Queue (1312) and the queue is read by the retrieval module (1320), which retrieves the content of each URL source. The fetched content is inserted into the fetched_queue (1322) and the queue is read by the parsing module (1330), which processes the fetched content and extracts content elements (or elements) (e.g., title, content, author, media , etc.) using the resolution specifications read from the database (1345) (or other data structure). The extracted content items are stored in the content_item_queue (1334) and the queue is read by the item sorting module (1350), which processes the retrieved content items and stores them either in the article_item-queue (1356) if the item is a news article or to hyperlink_queue (1352) if the element is a hyperlink, or to RSS_queue (1354) if the element is an RSS feed.

The content of the article stored in the article_item_queue (1356) is read by the item processor module (1390) (additional processing modules (1390) are used in alternative implementation examples), which performs at least one of Named Entity Recognition (NER), analysis sentiment, language detection, feature mining, clustering, summarization, etc. and produces the result of the present innovative detection solution.

The hyperlinks stored in the hyperlink_queue (1352) are read by the URL classifier module (1360), which sorts the URLs into one of three categories: a URL that points to a new source domain with related content, a URL that points to an unrelated source or a URL that points to a new resource (eg article) to retrieve.

URLs pointing to irrelevant resources are ignored, or in alternative implementations are put into an irrelevant_queue for elimination from other queues.

URLs pointing to new source domains are entered into the domain_queue (1362) and the domain_queue (1362) is read along with the RSS_queue (1354) by the deduplicator module (1370). The deduplicator module (1370) checks if the source is already in the list of known sources, and if it is not, then it inserts the source into the sources_queue (1372), which is read by the prioritizer module (1380), otherwise the source is ignored as it already exists in seed_queue (1312). The Prioritizer module (1380) reads the sources_queue (1372) and gives high priority to selected domains or URLs or RSS feeds so that they are fetched more often and its output is stored in the seed_queue (1312), essentially creating a closed loop where all subqueues converge to a single queue, e.g. seed_queue (1312). Ranking takes into account the number of posts from a source that have been cited by other sources more than a threshold in the recent past. Alternatively, multiple priority queues are used per data source, each priority queue for a different subject matter, in order to assign different subject matter priorities to the same source. A single queue is fed from the multiple queues and merges their data using information about the popularity of a topic's content in the recent past (eg, the number of articles published per topic).

The URL retrieval frequency imposed by the priority program module (1380) on the retrieval module (1320) via the seed_Q (1312) is a very important innovation of the present novel detection solution and takes advantage of the fact that an article may originate from a important (eg reputable or fast update or other) source. Additionally, retrieval frequency is important because it can be used to account for additional sources such as user comments on news articles, amateur journalism where news coverage is based on individual comments/tweets/articles/etc., and social media posts networking and comments, blog posts and comments, etc.

During the process described in Figure 13, the retrieval module (1320) may fail to retrieve the contents of a source URL for various reasons, such as network problem, routing/domain name server (DNS) problems, etc. web server problems, source site problems, etc. To deal with such problems, the recovery module (1320) stores the failed URLs in the failed queue (1324) and the queue failed_queue (1324) is read (along with the new_item_queue (1364) which contains the storage of new items detected by the URL_classifier by 1 postponer module (1365), which module (1365) stores the URLs it reads in the seed_queue (1312).

The parsing module (1330) may fail to process the received content and extract content elements. In such a case, the parser module (1330) stores the failed content in the error_fetched_queue (1332) and the queue is read by the parser specification generator (1340), which processes groups of content elements fetched from the same source/domain and generates specifications (ie, Xpath expressions) for analysis module (1330). The new specifications are stored in the parser specification database (1345) (or other data structure) and read (either periodically or when an interrupt is generated by the parser specification generator (1340)) by the parser module (1330).

As a result of the above queue group, the seed_queue (1312) contains a sorted list or URLs to crawl, where priority is given to those URLs that are more important, from sources with higher reputation, change more frequently, and are new as they come from related metadata and thresholds.

In addition, hot topics that temporarily receive many article publications will be temporarily promoted and their respective priority lists will be merged into the final, higher priority list.

The logic behind Figure 13 is sufficient to handle errors in the detection process and support changing states in the process, while feeding prioritization and aspects of web article sources such as reputation, update frequency, web page layout and websites etc. Various modifications of the order, name and number of subunits and queues are possible without deviating from the scope of the present innovative detection solution. One can, for example, add new queues or modules (parallel and/or serial) to handle different content from different types of sources (e.g. social media) or to model integrations using parallel processing in the form of multi-server detection , server crawl farm, cloud-based crawl servers, multiple processor cores per processor or multiple processors per crawl server, etc. Alternatively, one can merge some of the system queues (1300).

In addition to these modifications, further modifications may be introduced for synergistic computing where the processing resources used and/or detection results produced for a particular application, user or client are shared or partially shared for other applications, users or clients. In such cases, at least the parser specifications and article_item_queues are further processed according to specifications related to data privacy, privacy of tracking preferences, etc., so that personal data is protected in accordance with applicable law (e.g. h.

Civil Data Protection Legislation in the US, Data Protection Law in the UK, General Data Protection Regulation (GDPR) in the EU, etc.).

The procedures presented above are referenced in news websites and news articles and are given as an example. It is noted that the scope of the present innovative detection solution is not limited to websites and articles related to news, but it applies to the detection of all sources of online content and includes other form of this text, i.e. images, videos, multi-channel audio, vlogs, etc. .etc Where non-textual content is used, the proposed detection method may use metadata accompanying the non-textual content or in other exemplary embodiments apply additional processing steps where the textual data is derived from the non-textual content. For example, voice speech and audio recognition can be used to create text versions of the speech or textual description of the sound (eg song title, metadata describing an explosion, etc.) or image/video/ 3D to create metadata describing visual content. Techniques for processing audiovisual content are well known in the prior art and are not included in the present disclosure as they are obvious to any reader of ordinary skill in the relevant art.

Raw, processed and parsed data are stored in queues and databases (such as SQL, noSQL or others) while in alternative implementation examples they are stored in log files, data structures, etc. Data is stored in ASCII, XML, compressed, encrypted or other file representations and formats.

The above example embodiments are simplified and do not include hardware and software elements used in the embodiments but which are not part of the current innovative solution, are not needed to understand the embodiments, and are obvious to any user of ordinary skill in the relevant art. In addition, variations of the described technique, system architecture, and software architecture are possible, where, for example, technique steps and hardware and software elements may be rearranged, omitted, or new ones added.

Various embodiments of the invention are described above in the Detailed Description. While these descriptions directly describe the above embodiments, it is understood that those skilled in the art may conceive of modifications and/or variations (such as adding, deleting, or rearranging process steps and software and hardware modules) to the specific embodiments shown and described herein. Any such modifications or variations falling within the scope of this specification are also intended to be incorporated herein. Unless otherwise specifically noted, it is the inventor's intention that the words and phrases in the specification and claims be given the ordinary meaning given by persons of ordinary skill in the applicable art.

The foregoing description of a preferred embodiment and best mode of the invention known to the applicant at the time of filing has been presented and is for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed and many modifications and variations are possible in light of the above teachings. The embodiment has been selected and described to better explain the principles of the invention and its practical application and to enable others skilled in the art to make better use of the invention in various embodiments and with various modifications suitable for the particular use contemplated. Therefore, it is intended that the invention not be limited to the specific embodiments disclosed for carrying out this invention, but that the invention shall include all embodiments falling within the scope of the appended claims.

Those skilled in the art will understand that marks can be represented using any of a variety of different techniques. For example, data, software, instructions, signals that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, light, or any combination thereof.

Those skilled in the art would further appreciate that the various illustrative RF or analog circuit blocks described in connection with the disclosure herein may be implemented in a variety of different circuit topologies, in one or more integrated circuits, separately or in combination with logic circuits and systems. while performing the same functions described in the present disclosure.

Those skilled in the art would also appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this hardware and software interchangeability, various illustrative elements, blocks, modules, circuits and steps have been described above generally in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific implementation and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed to cause a departure from the scope of this disclosure.

The various illustrative logic blocks, modules, and circuits described in connection with the disclosure herein may be implemented or executed with a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic array field gate array (FPGA) or other programmable logic device, discrete logic gate or transistor, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, but alternatively, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g. a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any media that facilitates the transfer of a computer program from one place to another. A storage medium can be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carrying or storing the desired program code in the form of instructions or data structures and which can be accessed by a computer or any other device or device acting as a computer. Also, any connection is properly called computer-readable media. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital flexible disc (DVD), floppy disk, and blu-ray disc where the discs typically reproduce data magnetically or laser optically. Combinations of the above should also be included in the field of computer readable media.

An exemplary storage medium is connected to the processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and storage medium may reside as separate components in a user terminal.

The foregoing description of the disclosed exemplary embodiments is provided to enable anyone skilled in the art to make or use the present invention. Various modifications to these exemplary embodiments will be readily apparent to those skilled in the art, and the general principles set forth herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments set forth herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (19)

1. System for automatically optimizing the use of computing resources and increasing the efficiency of web crawling and content summarization, which includes: at least one detection server (160), which has at least one processor; and at least one database (155), (165), (170) connected to the at least one detection server (160), wherein the at least one database (155), (165), (170) is configured to store at least one of (a) at least one detection policy and (b) detection results? wherein the at least one processor is configured to receive web page wrappers, create and update the web page wrappers by analyzing multiple pages of a website based on aesthetic aspects and layout of page elements, text length, font type and size, and placeholders, and_ includes : a seeding module (1310) configured to produce using a web wrapper a list of web site and web page URL sources to download during web crawling; a queue seed_queue (1312) formatted to store the list of source URLs in a sorted list that is formatted to prioritize those URLs that are more important, from sources with higher reputation, change more frequently, and are they new as they come from relevant metadata and thresholds? fetch module (1320) configured to retrieve the content of each URL source in the URL source list in the seed_queue (1312) and to store the content of each URL in a fetched_queue (1322); a parsing module (1330) configured to process the content of each URL source from the fetched_queue (1322) and to extract content elements using the parsing specifications read from the database (155), (165), (170) and from web wrappers? a content_item_queue (1334) formatted to store the extracted content items? an item sorting module (1350) configured to read the content_item_queue (1334), process and store the content items in one of (c) an article_item-queue (1356) if the item is a news article, (d) in a hyperlink_queue (1352) if the element is a hyperlink, and in (e) an RSS_queue (1354) if the element is an RSS feed? and an item processor module (1390) configured to read content from the article_item-queue (1356), perform at least one of Named Entity Recognition (NER), sentiment analysis, language detection, aspect mining, clustering, and summarization, and produce a web crawler and content digest system result? a URL classifier module (1360) configured to read hyperlinks stored in the hyperlink_queue (1352) and to sort the URLs into one of (f) a first class that includes URLs that point to a new URL source domain with an associated content, (g) in a second category that includes URLs that point to an unrelated URL source, and (h) in a third category that includes URLs that point to a new resource to retrieve, and insert (i) URLs from the first category in a domain_queue (1362), URLs from the second category in an irrelevant_queue, and (j) URLs from the third category in a new_items_queue; a Deduplicator module (1370) configured to read URLs from domain_queue (1362) and RSS_queue (1354) and to insert the read URLs into a sources_queue (1372) if the read URLs are not already in a list of known sources ; and a prioritizer module (1380) configured to read the sources_queue (1372) and give a high priority to URLs from selected domains and RSS feeds so that they are fetched more often and to store prioritized URLs in the seed_queue (1312), and enforce URL retrieval frequency in the retrieval module (1320). where all queues form a closed loop and converge to the seed_queue (1312), and the prioritizer module (1380) is configured to prioritize URLs taking into account the number of posts from a source that have been referenced by other sources more than a threshold in recent past in order to optimize the use of computing resources and increase the efficiency of web crawling and content summarization. 2. A system according to claim 1, further comprising: a failed queue (1324) configured to store failed URLs when the recovery module (1320) fails to retrieve contents of a URL source? a new_item_queue (1364) configured to store new articles that are crawled by the URL classifier module (1360); and a postponer (1365) module configured to read URLs from the failed queue (1324) and new_item_queue (1364), and store the URLs in the seed_queue (1312). 3. A system according to claim 2, further comprising: an error_fetched_queue (1332) formatted to store failed URLs when the parsing module (1330) fails to fetch content and extract content elements? a parser specification generator (1340) configured to read the error_fetched_queue (1332), process groups of content elements fetched from the same source, and generate specifications for the parser module (1330) and store new specifications in a database; wherein the database is formatted to be read by the analysis module (1330). A system according to claim 1, wherein: the high priority given by the prioritizer module (1380) to URLs from one of the selected categories and RSS feeds, is given to those URLs that are most important, from sources with a high reputation, change frequently, are new, and temporarily accept a plethora of published articles as extracted from metadata and thresholds? and the new resource is an article. A system according to claim 1, wherein the content elements are elements selected from title, content, author and media. A system according to claim 3, wherein the specifications are Xpath expressions. 7. A method of automatically optimizing the use of computing resources and increasing the efficiency of crawling web pages and summarizing content implemented by at least one processor of at least one crawling server (160) of the system of any one of claims 1-6, which includes: retrieve web crawl policies using a crawl policy module (251); retrieving web page wrappers using a web page wrapper module (252) and creating and updating web page wrappers by analyzing multiple pages of a website based on aesthetic aspects and layout of page elements, text length, font type and size, and placeholders; in a crawling module (253), (a) using the web crawling policies to create a seed_queue (1312) for crawling web pages, wherein the seed_queue (1312) stores the list of source URLs in an ordered list which is formatted to give prioritize those URLs that are more important, from higher reputation sources, change more frequently and are new as they come from relevant metadata and thresholds and (b) crawl web pages using seed_queue (1312) and web wrappers? grouping crawled web pages and articles into a number of clusters using a news topic criterion and a clustering module (254); parsing web pages on each of the plurality of clusters using a page parser (255) which uses the parsing specifications read from a database (155), (165), (170) and from the web page wrappers; and digest web pages in each of the multiple clusters to generate a digest for each cluster using a digest module (256); where the above steps are implemented through multiple queues, which create a closed loop and converge to the seed_queue (1312). 8. A method according to claim 7, wherein the websites include at least one of news websites, social networking websites, blog articles, vlog articles, opinion websites, media articles, commentaries and opinion articles. 9. The method of claim 7, wherein the seed_queue (1312) includes a prioritized list of Universal Resource Locator (URL) items for crawling web pages, the seed_queue (1312) is populated with the processed items of a new_item_queue (1364), a sources_queue (1372) and a failed queue (1324) and wherein new_item_queue (1364), sources_queue (1372) and failed queue (1324) are filled with the processed content items of at least one of: a fetched_queue (1322) that includes fetched content from a list of URL sources from the seed_queue (1312); a content_item_queue (1334) containing processed content items extracted from the fetched content of the fetched_queue (1322); an article_item-queue (1356) containing article content items which are selected from content items in content_item_queue (1334); a hyperlink_queue (1352) containing hyperlinks selected from content items in content_item_queue (1334) ; a Rich Site Summary (RSS) queue (1354) which includes RSS feeds selected from content items in content_item_queue (1334); an irrelevant queue that contains URL content elements that are removed from other queues? a domain_queue (1362) that contains URL content elements that point to new source domains? a sources_queue (1372) that includes content elements of URLs that point to new sources not contained in the RSS_queue (1354); a failed queue that contains content elements of URLs for which crawling failed? and an error_fetched_queue (1332) which consists of fetched content for which parsing of the content elements failed. 10. A method according to claim 7, wherein crawling web pages comprises first crawling a first web page on a first page, then crawling a category page of the same site, and finally crawling all web pages of the site category page before detection of a second site. 1 1. A method according to claim 7, wherein the websites are news websites. 12. A method according to claim 7, wherein the web crawling policies and the seed_queue (1312) are based on influence, reputation and impact of the website as extracted from metadata and thresholds. 13. A method according to claim 12, wherein the website reputation is calculated from at least one of the following: frequency of article news reporting article publications per site? article publication rate? thematic categorization of source domains and source pages? source reputation history? negative comments about the site that originate from negative user comments on at least one website, website pages, off-site pages, news pages, social media pages, blog articles, vlog articles, opinion pages, media articles, and website comments of the website and in opinion articles? and regulatory sanctions against a website for poor journalistic practices. A method according to claim 7, further comprising creating or updating web crawl policies by the crawl policy module (251) and web page wrappers by the web page wrapper module (252). A method according to claim 14, wherein at least one of the following is true: are different web page wrappers created or updated by the web page wrapper module (252) for (a) the first page, (b) the category pages, and (c) the web pages of a first website, and are different web wrappers created for pages of a second web page? and a plurality of seed_queues (1312) are used per data source, where each seed_queue (1312) is used to assign a different priority to each topic of the data source. 16. A method according to claim 15, wherein creating or updating a web page wrapper by the web page wrapper module (252) comprises: extract links from website page? link sorting? link grouping by Xpath expression? create or customize the page wrapper? using the created or updated page wrapper to extract content from a web page? validation of exported content? and update the page wrapper if the extracted content is invalid; 17. A method according to claim 7, wherein web page detection is implemented in one of the following ways: sequential crawling of web pages? parallel web crawling? and parallel crawling of web pages on the same site. 18. A method according to claim 7, wherein web pages in each of the plurality of clusters are analyzed by the page parser sub-module (255) using Natural Language Processing (NLP) and Natural Language Understanding (NLU) to determine subtopics discussed in the web pages. A method according to claim 7, wherein the summary for each cluster is generated by the summarization module (256) by first detecting or generating sentences describing the subtopics of web pages in the cluster and then aggregating the sentences found or generated in the summary.