WO2020185110A1 - Method and system for searching for relevant news - Google Patents

Abstract

The present solution relates to the field of information technology, and more particularly to search engines designed to identify relevant information from different types of data sources. The technical result is that of enabling the formation of a related set of information from news sources with grouping according to companies that are the subject of news or given types of events. In a first preferred embodiment of the claimed solution, a computer-implemented method for searching for relevant news is proposed in which a set of news is received on a control server from at least one news aggregator server; an analysis of the received set of news is performed on the control server, which includes lemmatizing the texts of each piece of news from said set of news, processing the resulting lemmas of the news texts with the aid of a machine learning model containing a predetermined set of company data and a list of events, wherein for each event in the machine learning model a given set of lemmas is predetermined, identifying news containing lemmas that characterize given events, and forming links between the identified events and at least one company; whereupon, a list of relevant news is generated on the basis of the analysis performed.

Description

METHOD AND SYSTEM FOR SEARCHING FOR RELEVANT NEWS

FIELD OF TECHNOLOGY

[0001] The present technical solution generally relates to the field of information technology, and in particular to search engines designed to identify relevant information from heterogeneous data sources.

LEVEL OF TECHNOLOGY

[0002] Currently, data mining (English "Data Mining") is an important component for various areas of business, especially in the areas of analytics and forecasting. Often, the source of data on information on topics of interest are public resources on the Internet, for example, news resources (websites, channels in instant messengers, etc.).

[0003] When analyzing data, the main problem is the aggregation of an array of news sources, in particular the linking of actual events to companies for subsequent search. As a rule, today there are no effective means for filtering the collected news content to create aggregated arrays of information with reference to news objects, for example, companies.

[0004] Various algorithms for collecting data are known in the prior art, for example the solution described in WO 1999005614 (by Louis Gay et al., Published: 02/04/1999), which allows data to be aggregated from multiple sources and tracked back to back. collected data.

[0005] From the patent RU 2382401 (patentee: MICROSOFT CORPORATION, published: 02/20/2010) an approach is known for analyzing and comparing collections of documents, according to which documents can presumably be organized into groups according to their content or source and analyzed for intergroup and intragroup differences and commonalities. For example, comparing two groups of documents on the same topic, but obtained from two different sources, for example, an informational overview of an incident in different parts of the world, can show interesting differences of opinion and general interpretations of situations. By moving content from static collections to collections of articles generated over time, its evolution can be considered. For example, a feed of news articles by general description can be viewed over time to highlight really informative news stories and filter out the many articles that pretty much convey "much the same thing."

[0006] A common drawback of existing approaches is the lack of a method for identifying relevant news in relation to binding to a news object, for example, a company and a corresponding event associated with it, which does not allow efficient collection of relevant information from multiple data sources.

DISCLOSURE INVENTION

[0007] The technical problem or technical problem to be solved using the claimed approach is to provide a process for searching and generating a set of news with reference to a given set of company names as news objects and events about which information appears in open data sources.

[0008] The technical result achieved when solving the above technical problem is to provide the formation of a related set of information from news sources with a grouping by companies that are the object of news and given types of events.

[0009] An additional technical result is to improve the accuracy of identifying information about companies for a given type of event in publicly available information sources.

[0010] The specified technical result is achieved through the implementation of a computer-implemented method for searching relevant news, in which:

receiving at the management server a set of news from at least one news aggregator server;

analyze the received set of news on the control server, which includes lemmatization of the texts of each news from the mentioned set of news; processing the obtained lemmas of news texts using a machine learning model, which contains an established set of company data and a list of events, and for each event in the machine learning model a given set of lemmas is set;

defining news containing lemmas identifying the specified events and forming a connection of the identified events with at least one company;

form a list of relevant news based on the analysis performed. [OOP] In one of the particular examples of implementation of the method, when receiving a set of news, duplicate news is filtered.

[0012] In another particular embodiment of the method, the filtering is performed by calculating the Jacquard measure between news signatures.

[0013] In another particular embodiment of the event method, the assigned company news is stored in a database.

[0014] In another particular embodiment of the method, the news aggregator updates the news list using information channels.

[0015] In another particular embodiment of the method, the information channels are web sites on the Internet and / or messenger channels.

[0016] In another particular embodiment of the method, during the analysis of news, the determination of whether the event belongs to a parent or subsidiary company is performed.

[0017] In another particular embodiment of the method, the ownership of the company mentioned in the news is determined using a decision tree algorithm.

[0018] In another particular embodiment of the method, during the lemmatization of news texts, they are cleared of punctuation marks, stop words and named entities.

[0019] In another particular embodiment of the method, a statistical measure is calculated for each lemma of the news text during lemmatization.

[0020] In another particular embodiment of the method, the machine learning algorithm is a logical regression, classifying the belonging of the news to the event based on the analysis of the statistical measure of the lemmas.

[0021] In another particular embodiment of the method, for each news text, a determination of frequency phrases with a length of 2 to 10 lemmas is performed.

[0022] In another particular embodiment of the method, the machine learning algorithm is a gradient boosting trained to classify an event based on the number of sentences containing lemmas identifying an event from a search query.

[0023] In another particular embodiment of the method, after assigning an event from a company news, extraction of lemmas and / or sentences containing lemmas identifying said event is performed.

[0024] In another preferred embodiment of the claimed solution, a relevant news search system is provided, comprising at least one processor and at least one memory that contains machine-readable instructions that, when executed by at least one processor, perform the above method.

BRIEF DESCRIPTION OF DRAWINGS

[0025] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[0026] FIG. 1 illustrates the interaction of the elements included in the claimed solution.

[0027] FIG. 2 illustrates the general flow of the method.

[0028] FIG. 3 illustrates the processing of text data.

[0029] FIG. 4 shows an example of a graphical user interface when interacting with a service for the selection of relevant news.

[0030] FIG. 5 illustrates a general view of a computing device.

CARRYING OUT THE INVENTION

[0031] FIG. 1 shows the general computing architecture (100) of the presented solution. Main functionality for collection and processing information is executed on a control server (ON), which, through a data transmission channel, receives information from a news aggregator server (120), which is connected via the Internet (150) to a plurality of news resources (130). The server (software) provides interaction with users (10) to display data on the collected news information, as well as additional functionality, which will be disclosed later in the application materials.

[0032] The Internet or Intranet can be used as a data transmission channel between the management server (110) and the server of the news aggregator (120). In this case, the server of the news aggregator (120) can represent several devices that are part of a different network environment, for example, a set of servers, routers, clusters, etc. A data transmission channel can be organized using various types of known data transmission protocols, both wired and wireless, for example, TCP / IP, 802.11, Ethernet, FTP, etc., providing the formation of various network interactions, in particular LAN, WAN, PAN , WLAN, etc.

[0033] the Management server (software) performs the main processing of information received from the server of the news aggregator (120), stores and generates data for display to users (10). Information display can be formed using a specialized graphical user interface. Users (10) can interact with the control server (110) using a web portal or other type of software application that provides access to aggregated news information. Access can be provided, for example, through an API. The interaction of users (10) can be carried out using various electronic devices, which can be, for example, a computer, laptop, smartphone, tablet, game console, smart wearable electronic device, thin client, as well as devices of augmented, mixed or virtual reality and dr.

[0034] The server of the news aggregator (120) is connected via the Internet (150) with various information resources (130) or information channels providing news information. Such resources (130) can be, for example, websites, messenger channels (Telegram ™, WatsApp ™, Viber ™, etc.), social networks (Facebook ™, Vkontakte ™, etc.). Saving the received information on the server (110) can be carried out in JSON format in a data store, for example, a database. In this case, the source of obtaining news information and the date of its placement on the corresponding resource can be taken into account (130).

[0035] FIG. 2 shows a general process for performing the claimed method of searching for relevant news information (200). Information from news sources, collected and stored on the server of the news aggregator (120), is transmitted (201) to the control server (software). Information from the server of the news aggregator (120) can be transmitted online or offline. In online mode, data from the Internet (150) is transmitted as soon as it appears on a web resource to which the news aggregator server has a connection (120). In offline mode, news is stored on the news aggregator server (120), for example, in a database, and at a set time (for example, every hour, once a day, etc.) or upon request from the control server (software) are transmitted to it ...

[0036] Data from the news aggregator server (120) can be transmitted in various formats, for example, xml, html, txt, and the like. The data format for transmission can also change depending on the mode of information transmission to the control server (1-10). In addition to the text itself, the news data contains information about the companies mentioned in the text.

[0037] On the control server (ON) there is a generated list containing the names of companies (2021) and events (2022), for which the analysis of incoming news information from the server of the news aggregator (120) is carried out. The specified data is stored in the database of the management server (software). Events can be, for example, arrest / freezing of company accounts, company bankruptcy, claims against the company, stock collapse / growth, etc. The list of events (2022) and companies (2021) can be updated or changed over time.

[0038] The search for relevant information according to the data received from the server of the news aggregator (120) is carried out by processing (202) the obtained data array using a machine learning model, which is trained to search by company names (2021) and corresponding events (2022) in the array textual information and make judgments about the relevance of the relevant information. Data processing at server (110), is performed upon receipt of a new data array from the news aggregator server (120), or according to a predetermined scenario. An automatic script can be configured as a script, which at a set time activates a machine learning model for data processing (202).

[0039] When executing the processing step (202), an access is made to the information store of the management server (UE), which contains data received from the news aggregator server (120) from news sources (130). When accessing the information stored on the control server (HO), it is processed (202) to identify relevant data and bind data (203) from news to the corresponding types of events during information processing using a machine learning model.

[0040] FIG. 3 shows a process (300) for processing news data received from a news aggregator server (120), which is performed during steps (202) - (203). At the first step (301), the news text data received from the server of the news aggregator (120) is lemmatized, during which the text corpus of each news is divided into lemmas. From the received data, news text and metadata from files are extracted.

[0041] During the execution of the text lemmatization process (301), the body of the news is divided into words by all punctuation separators, and then reduced to normal form, for example, using the pymorphy2 library. Then the text is converted, in particular, the text is cleared of punctuation marks, stop words (prepositions, conjunctions, pronouns) and nominal entities. In this case, a nominal entity is considered to be any word that begins with a capital letter and is not the first word in a sentence. Also, the process of N-gramming (https://ru.wikipedia.org/wiki/N-rpaMMa) can be performed, in which the most frequent word combinations of length from 2 to 10 lemmas are highlighted in the text. The list of the most frequent phrases was obtained by automatic analysis of a large body of text and contains more than 9 million objects.

[0042] Also, incoming news goes through a deduplication process, during which duplicate news is filtered out. During the deduplication procedure, the MinHash signature is calculated for each news item (see. https://en.wikipedia.org/wiki/MinHash), after which, for each pair of news, the similarity of signatures is calculated according to the Jaccard measure (sometimes the Jaccard coefficient). If the similarity of a pair of news exceeds the specified threshold, for example, 0.7, then the shorter news from the pair of text corpuses is considered duplicate and is not subjected to further processing.

[0043] In the next step (302), after lemmatization of the news texts, processing of the normalized text is performed. The text of the news is searched for the names of companies that do not have homonyms (for example, Sberbank ™). All phrases with a capital letter and in quotes are found, after which the lemmas of the found phrases are searched in the list of companies stored in the server database (110).

[0044] Found company names are classified according to the "main" or "additional" company (ie, which is indirectly referred to in the text of the news). A company is considered "main" if it is the subject of news, and "additional" if the name of the company is simply mentioned in the body of the news. The classification is carried out using a machine learning model, in particular a decision-making algorithm, for example, using decision trees. The decision tree feature list is as follows:

1) Number of the sentence of the first mention of the company (0, if it is a title);

2) Number of the proposal of the first mention of the company, normalized to the number of proposals;

3) The length of the text in characters;

The classification threshold is 0.5.

[0045] To determine the relevance of an event for the companies indicated in the news body, the received lemmas from the news body are processed at step (303) using machine learning models.

[0046] As one example of a machine learning model, logical regression can be applied by calculating the TF-IDF statistical measure for text lemmas (see https://ru.wikipedia.org/wiki/TF-IDF). For each lemma in the text, a statistical measure is calculated, after which a pre-trained logistic regression judgment is made on the obtained features. In addition to processing using a machine learning model, a list of given lemmas is compiled, for example, a list may contain 30-40 lemmas that have most weight in logistic regression. The list is built for each event after the logistic regression learning process. At the output of the model, the weight of each lemma is determined, according to which the selection of lemmas for the list is carried out based on the values of their weights.

[0047] If the obtained value of the probability of judgment of the model (303) is higher than a predetermined threshold and at least one lemma from the mentioned list occurs in the news text, then at least one event is assigned to news (304) for the company name identified in the text.

[0048] Additionally, for each event, a set of lemmas can be specified, for example, 10-15 lemmas, which are most appropriate for the event, which are selected from a previously defined list of lemmas, and if the event was assigned to news at step (304), then all lemmas found in the text from mentioned set are highlighted in the text.

[0049] A second example of an application of the machine learning model is a gradient boosting classifying algorithm such as LightGBM (https://lightgbm.readthedocs.io). For each news text, the number of sentences containing pairs of lemmas characteristic of the event is counted. Pairs of characteristic lemmas are selected for each event during the training of the classifier. The characteristic lemmas (and their number) are selected automatically during training.

[0050] On the features (pairs of lemmas) obtained in this way, a judgment is made using the aforementioned machine learning model (303). If the obtained probability value is higher than a predetermined threshold, then the event is assigned to news (304) for one or more companies indicated in the news. Additionally, sentences containing pairs of lemmas characteristic of the event can be highlighted in each text.

[0051] If during the processing of news data, the identification of relevant events for the specified names of companies is not carried out, then such information is not taken into account (305).

[0052] FIG. 4 shows an example of a graphical user interface (400) for interacting with a service for the selection of relevant news information. The interface (400) provides functionality for displaying and managing the content of the provided data. The search query is formed using the name information input panel companies (401). In the main field (404) for displaying current or found information, a list of companies is presented for which processing of identifying relevant information from the server database (110) is carried out.

[0053] Companies in field (404) can be displayed in a different hierarchical order, for example, alphabetically, by the number of news, and the like. The information can be filtered by the time range, which is set in the date input field (402).

[0054] Also, the interface (400) contains a control panel for setting parameters of search queries (403). Using the control panel (403), you can configure the detection of certain types of events, link companies, configure service parameters, etc. The field (405) displays a list of identified news sources in accordance with the specified events for the companies.

[0055] Users (10) can also set an alert function for selected company names. Notifications about the arrival of new news can be sent via e-mail messages, PUSH notifications, SMS notifications, etc. When configuring the notification function, the user (10) can configure the required parameters, for example, the name of the company, the type of events related to companies.

[0056] The generated information on processed news can also be displayed using a filter customized with respect to the role of the user (10) interacting with the interface (400). Taking into account the parameter of the user account (10), only those news that contain the type of events associated with his role can be displayed to him.

[0057] FIG. 5 shows an example of a general view of the device (500), which provides the implementation of the presented solution. A different range of computing devices can be implemented on the basis of the device (500), for example, a control server (110), a news aggregator server (120), user devices (10), etc.

[0058] In general, the device (500) contains one or more processors (501) united by a common data exchange bus, memory means such as RAM (502) and ROM (503), input / output interfaces (504), input / output (505), and a device for networking (506). [0059] The processor (501) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example, manufacturers such as: Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™, etc. Under the processor or one of the processors used in the device (500), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU or Graphcore, the type of which is also suitable for full or partial execution of the method (200), and can also be used for training and applying machine models. training in various information systems.

[0060] RAM (502) is a random access memory and is intended for storing machine-readable instructions executed by the processor (501) for performing the necessary operations for logical data processing. RAM (502) typically contains executable instructions of an operating system and associated software components (applications, software modules, etc.). In this case, the available memory of the graphics card or graphics processor can act as RAM (502).

[0061] ROM (503) is one or more means for permanent storage of data, for example, hard disk drive (HDD), solid state data storage device (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0062] Various types of I / O interfaces (504) are used to organize the operation of the components of the device (500) and to organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific version of the computing device, which can be, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0063] To ensure the interaction of the user with the computing system (500), various means (505) I / O information are used, for example, a keyboard, display (monitor), touch display, touch-pad, joystick, mouse manipulator, light pen, stylus, touchpad, trackball, speakers, microphone, augmented reality, optical sensors, tablet, light indicators, projector, camera, biometric identification (retina scanner, fingerprint scanner, voice recognition module), etc.

[0064] The networking tool (506) provides data transmission via an internal or external computer network, for example, Intranet, Internet, LAN and the like. One or more means (506) may be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and dr.

[0065] Additionally, satellite navigation aids can be used as part of the device (500), for example, GPS, GLONASS, BeiDou, Galileo.

[0066] The specific choice of device elements (500) for the implementation of various software and hardware architectural solutions can vary while maintaining the required functionality provided from one or another type of device.

[0067] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation, not going beyond the scope of the claimed legal protection, which are obvious to specialists in the relevant field of technology.

Claims

FORMULA 1. A computer-implemented method of searching for relevant news, containing the stages at which: • receive on the control server a set of news from at least one server of the news aggregator; • analyze the received set of news on the control server, which includes lemmatization of the texts of each news from the mentioned set of news; processing of the received lemmas of news texts using a machine learning model, which contains a set set of company data and a list of events, and for each event in the machine learning model a given set of lemmas is set; about the definition of news containing lemmas identifying the specified events and the formation of a connection of the identified events with at least one company; • form a list of relevant news based on the analysis of the set of news. 2. The method according to claim 1, characterized in that when receiving a set of news, duplicate news is filtered. 3. The method according to claim 2, characterized in that the filtering is carried out by calculating the Jaccard measure between news signatures. 4. A method according to claim 1, characterized in that the events assigned news about the company are stored in a database. 5. The method according to claim 1, characterized in that the news aggregator updates the news list using information channels. 6. The method according to claim 5, characterized in that the information channels are Internet websites and / or messenger channels. 7. The method according to claim 1, characterized in that during the analysis of news, the event belongs to the main or subsidiary company. 8. The method according to claim 7, characterized in that the affiliation of the company mentioned in the news is determined using a decision tree algorithm. 9. The method according to claim 1, characterized in that during the lemmatization of news texts, they are cleared of punctuation marks, stop words and nominal entities. 10. The method according to claim 1, characterized in that during the lemmatization for each lemma of the text of the news, a statistical measure is calculated. 11. The method according to claim 10, characterized in that the machine learning algorithm is a logical regression classifying the belonging of the news to the event based on the analysis of the statistical measure of the lemmas. 12. The method according to claim 1, characterized in that for each news text, the definition of frequency phrases with a length of 2 to 10 lemmas is performed. 13. The method of claim 1, wherein the machine learning algorithm is a gradient boosting trained to classify an event based on the number of sentences containing lemmas identifying the event from a search query. 14. The method according to claim 1, characterized in that after assigning an event from the company news, extraction of lemmas and / or sentences containing lemmas identifying said event is performed. 15. A device for searching relevant news, containing at least one processor and at least one memory containing machine-readable instructions, which, when executed by at least one processor, execute the method according to any one of claims. 1-14. 16. A search engine for relevant news, containing at least one management server; at least one news aggregator server, configured to receive news data from at least one news source, and the control server is configured to receive news data from at least one news aggregator server; analysis of the data obtained, during which is performed: o lemmatization of the texts of each news from the mentioned set of news; o processing the received lemmas of news texts using a machine learning model, which contains a set set of company data and a list of events, and for each event in the machine learning model a given set of lemmas is set; about the definition of news containing lemmas identifying the specified events and the formation of a connection of the identified events with at least one company; generating a list of relevant news based on the analysis of the news set.