RU2691855C1 - Training classifiers used to extract information from natural language texts - Google Patents

Abstract

FIELD: data processing.SUBSTANCE: invention relates to a system and methods of extracting information from natural language texts. Method of extracting information from natural language texts includes: training information extraction classifier to extract first plurality of information objects from text in natural language, wherein determination of information extraction classifier includes one or more hyperparameters; obtaining a list of extracted information objects by performing a conflict arbitration function with respect to a plurality of conflicting information objects; changing values of hyperparameters of information extraction classifier; and optimizing the information extraction quality factor for the list of extracted information objects by iterative repetition of the information extraction classifier training operations, performing the conflict arbitration function and changing the hyperparameter values.EFFECT: high efficiency and quality of extracting information from natural language texts.25 cl, 16 dwg

Description

TECHNICAL FIELD

[0001] The present invention relates generally to computing systems, and more specifically to systems and methods for processing natural language.

BACKGROUND

[0002] Extraction of information may include the analysis of a natural language text to identify and classify information objects in accordance with a given set of categories (for example, the names of individuals, organizations, places, expressions of time, number, sums of money, interest, etc.). Information retrieval may additionally include defining relationships between recognized named entities and (or) information objects.

DISCLOSURE OF INVENTION

[0003] In accordance with one or more embodiments of the present invention, an example of a classifier training method used to extract information from texts in natural language may include: training information extraction classifier to extract the first set of information objects from natural language text, and the definition of extraction classifier information includes one or more hyperparameters; obtaining a final list of extracted information objects by performing the function of “arbitration” of conflicts with respect to a set of conflicting information objects, so that the first information object from the set of conflicting information objects to the first set of information objects and the second information object from the set of conflicting information objects belongs to the second set of information objects extracted from natural language text; changing the values of the hyperparameters of the information extraction classifier; and optimization of the information extraction quality indicator for the final list of extracted information objects by iteratively repeating the training operations of the information extraction classifier, performing the “arbitration” of conflicts and changing the values of hyperparameters.

[0004] In accordance with one or more embodiments of the present invention, another example of a classifier training method used to extract information from natural language texts may include: extracting, by the first method of extracting information, a first set of information objects from natural language text; extracting, by the second method of extracting information, the second set of information objects from the text in natural language; changing the values of one or several hyperparameters of the conflict arbitration classifier; learning the conflict arbitration classifier that creates a list of extracted information objects by performing the function of arbitrating conflicts regarding a set of conflicting information objects, so that the first information object of a set of conflicting information objects belongs to the first set of information objects the second information object of a set of conflicting information objects belongs to the second set of information objects; optimization of the information extraction quality metric, reflecting the quality of the list of extracted information objects, iteratively repeating operations of extracting the first set of information objects, extracting the second set of information objects, changing the values of hyperparameters and training the conflict arbitration classifier.

[0005] In accordance with one or more embodiments of the present invention, an example of a system for extracting information from text in natural language may include: a storage device and a processor connected to the storage device. This processor can be configured to: - learn information extraction classifiers to extract the first set of information objects from natural language text, with the definition of the information extraction classifier including one or more hyperparameters; retrieving the final list of extracted information objects by performing the function of “arbitration” of conflicts with respect to the set of conflicting information objects, for example, the first information object from the set of conflicting information objects belonging to the first set of information objects and the second information object from the set of conflicting information objects belonging to the second set of information objects extracted from the text in natural language; changes in the values of the hyperparameters of the information extraction classifier; and optimization of the information extraction quality indicator for the final list of extracted information objects by iteratively repeating the training operations of the information extraction classifier, performing the “arbitration” of conflicts and changing the values of hyper parameters.

[0006] In accordance with one or more embodiments of the present invention, an example of a computer-readable readable data carrier may contain executable instructions that, when executed by a computing system, cause the following actions of a computing system: training information extraction classifiers to extract the first set of information objects from natural language text, moreover, the definition of the classifier information extraction includes one or more hyperparameters; obtaining a final list of extracted information objects by performing the function of “arbitration” of conflicts with respect to a set of conflicting information objects, for example, the first information object from the set of conflicting information objects belonging to the first set of information objects and the second information object from the set of conflicting information objects belonging to the second set of information objects extracted from the text in natural language; changing the values of the hyperparameters of the information extraction classifier; and optimization of the information extraction quality indicator for the final list of extracted information objects by iteratively repeating the training operations of the information extraction classifier, performing the “arbitration” of conflicts and changing the values of hyperparameters.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present invention is illustrated using examples rather than a limiting method, and may be better understood when considering the following description of preferred embodiments in conjunction with the drawings, in which:

[0008] FIG. 1 is a block diagram of one illustrative example of a method for extracting information using several extraction methods in accordance with one or more embodiments of the present invention;

[0009] FIG. 2 schematically shows an example of “arbitration” of conflicting objects, which may be performed in accordance with one or more embodiments of the present invention, in accordance with one or more embodiments of the present invention;

[00010] FIG. 3 shows a block diagram of one example of a classifier training method used to extract information from natural language texts in accordance with one or more embodiments of the present invention;

[00011] FIG. 4 shows a block diagram of another example of a classifier learning method used to extract information from natural language texts in accordance with one or more embodiments of the present invention;

[00012] FIG. 5 shows a block diagram of one illustrative example of a method 400 for semantic-syntactic analysis of a natural language sentence in accordance with one or more embodiments of the present invention;

[00013] FIG. 6 schematically shows an example of a lexico-morphological structure of a sentence in accordance with one or more embodiments of the present invention;

[00014] FIG. 6 schematically shows language descriptions representing a natural language model in accordance with one or more embodiments of the present invention;

[00015] FIG. 8 schematically shows examples of morphological descriptions in accordance with one or more embodiments of the present invention;

[00016] FIG. 9 schematically shows examples of syntactic descriptions in accordance with one or more embodiments of the present invention;

[00017] FIG. 10 schematically shows examples of semantic descriptions in accordance with one or more embodiments of the present invention;

[00018] FIG. 11 schematically shows examples of lexical descriptions in accordance with one or more embodiments of the present invention;

[00019] FIG. 12 schematically shows examples of data structures that can be used within one or more of the methods implemented in accordance with one or more embodiments of the present invention;

[00020] FIG. 13 schematically shows an example of a graph of generalized components in accordance with one or more embodiments of the present invention;

[00021] FIG. 14 shows an example of the syntactic structure corresponding to the sentence shown in FIG. 13;

[00022] FIG. 15 shows the semantic structure corresponding to the syntactic structure shown in FIG. 14;

[00023] FIG. 16 is a diagram of an example computing system implementing the methods of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[00024] This document describes methods and systems for extracting information from natural language texts. The systems and methods presented in this document can find application in a wide variety of situations where natural language processing is required, in particular, it can be machine translation, semantic indexing, semantic search (including multilingual semantic search), document classification, search and submission of electronic documents (e-discovery), etc.

[00025] In this document, the term "computing system" means a data processing device equipped with a universal or specialized processor, memory, and at least one communication interface. Examples of computing systems that can use the methods described in this document are, in particular, desktop computers, laptops, tablet computers and smartphones.

[00026] "Ontology" in this document means a model that represents objects belonging to a specific area of knowledge (domain), and the relationship between these objects. An ontology can include definitions of a certain set of classes, where each class corresponds to a domain concept. Each class definition may include definitions of one or more objects assigned to the class. According to generally accepted terminology, an ontology class can also be called a “concept”, and an object belonging to a class can mean an instance of a given concept. In some implementations of the invention, the class may be an ancestor or descendant of another class. In certain embodiments, an object may be associated with two or more classes.

[00027] The definition of each class may further include one or more definitions of the relationships of one or more objects assigned to the class. Relationships define different types of interactions between related objects. In some implementations of the invention, different relationships may be organized in a general taxonomy, for example, the “paternity” and “motherhood” relationships may be included in the more general “parent” relationship, which in turn may be included in the more general “ancestor” relationship .

[00028] The attribute of an information object may be represented by another information object (for example, the place of birth of a person may be represented by an information object of the class "City"; and the employer of a person may be represented by an object of the class "Company"). A class definition can include one or more constraints associated with certain attributes of an object of this class, for example, a cardinality restriction attribute (for example, a person can have only one birthplace) or a type restriction (for example, a given attribute can be represented by objects of a specified set of classes ).

[00029] Information extraction may include analysis of a natural language text to identify and classify information objects referenced by natural language text fragments. Fragments of text that refer to an information object are called “object annotations”. Text fragments that refer to an attribute of an information object are called “attribute annotations.” An annotation may be defined by its position in the text in natural language, including the position of the beginning (word) and the position of the end (word).

[00030] When extracting information, the identified information objects may be classified according to a predefined set of categories (for example, names of people, names of organizations, places, expressions of time, quantities, sums of money, interest, etc.) Such categories can be represented by concepts of a predefined or dynamically formed ontology. Information retrieval may additionally include defining relationships between recognized named entities and (or) information objects. Examples of such relationships can be the work of a person X in the organizational unit Y, the location of the object X in geo-location Y, the acquisition by the organizational unit X of an organizational unit Y, etc. Similar relationships can be expressed in natural language fragments, which can contain a variety of words from one or more sentences. Relationships of information objects can be expressed by attributes of an information object that refer to other information objects (for example, the information object “person” can have attributes such as “place of birth”, “place of residence” or “employer”, each of which is represented by reference to information object of the corresponding class).

[00031] The extracted information, which can be represented by an RDF graph, can be used to perform various operations and tasks of natural language processing, including machine translation, semantic indexing, semantic search, document classification, search and presentation of electronic documents (e-discovery) and etc.

[00032] Information extraction can be performed by various extraction methods using variable sets of rules, automatic classification methods (also called “machine learning-based classifiers”), heuristic approaches, and / or a combination of these methods. Information extraction methods can be based on ontologies and / or other document models for analyzing morphological, lexical, syntactic, semantic, and / or other attributes of natural language texts.

[00033] In an illustrative example, retrieving information can include applying a set of production rules, for example, matching the pattern specified on the left side of a production rule with a semantic structure representing a piece of text in natural language can trigger the action of the right side of a production rule that can assign certain values to one or more attributes of the information object. In another illustrative example, a classifier based on machine learning can determine the degree of relatedness of a given piece of text in natural language (for example, represented by a set of morphological, lexical, syntactic, semantic and / or other attributes) with a certain class of information objects. Classifier parameters can be defined or changed as a result of further training of the classifier using previously existing or dynamically created training data samples, on the basis of which the classes of information objects are correlated with morphological, lexical, syntactic, semantic and (or) other attributes of text in natural language. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00034] Different methods for extracting information may have different performance characteristics, such as computational accuracy, completeness, and (or) computational complexity. Accordingly, different methods of extracting information can create different results when applied to the same natural language text. Certain information objects extracted based on two or more methods of extracting information may conflict with each other, for example, the same word in natural language can be recognized as a reference to a geographical location or the name of a person.

[00035] The present invention improves the efficiency and quality of information extraction by creating systems and methods that use various combinations of information extraction methods, followed by defining and resolving conflicts among the extracted information objects, thus achieving a result that exceeds the quality and efficiency of the traditionally used methods. In an illustrative example, a computing system implementing this method may apply two or more methods of extracting information for the same natural language text. The computing system can then identify sets (for example, pairs, triplets, quadruplets, etc.) of potentially conflicting objects extracted by various methods, for example, by identifying overlapping text annotations and / or violations of various constraints associated with the attributes of information objects (for example , attribute of cardinality constraint). After determining the set of conflicting objects, the computing system may attempt to resolve the conflict by applying the function of “arbitration” of the conflicting objects, which can perform an analysis of morphological, lexical, syntactic, semantic and / or other attributes of the conflicting information objects in order to obtain a result confirming more potentially conflicting objects, changing the attributes of one or more potentially conflicting objects, deleting one or more potentially conflicting objects and / or merging one or more potentially conflicting objects.

[00036] In certain embodiments, the function of “arbitration” of conflicting information objects may apply one or more custom rules for evaluating logical conditions defined for morphological, lexical, syntactic, semantic, and / or other attributes of the extracted information objects. Alternatively, the “arbitration” function of conflicting objects may include one or more classifiers based on machine learning. In an illustrative example, a classifier used by the “arbitration” function of conflicting objects can provide a degree of justification for one or more possible results of “arbitration” (for example, merging possible conflicting objects into a single object, confirming one or more extracted objects, changing one or more extracted objects and (or) deleting one or more retrieved objects). In another illustrative example, a classifier used by the “arbitration” function of conflicting objects can provide the similarity of two or more information objects representing the same real-life object. In another illustrative example, the classifier used by the function of “arbitration” of conflicting objects can create degrees of confidence associated with each information object from a set of conflicting information objects.

[00037] Classifiers used by the “arbitration” function of conflicting objects can be trained on the basis of pre-existing or dynamically created training data samples, based on which the results of “arbitration” can be correlated with morphological, lexical, syntactic, semantic and (or) other attributes of the extracted information objects. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00038] In certain embodiments, training data samples for classifiers based on machine learning can be created by applying methods based on rules for selecting texts in natural language and / or by verifying by the user the results of classification obtained through the use of trained classifiers. In an illustrative example, classifiers can iteratively undergo re-training based on user-verified classification results, due to which, with each iteration of training, the quality of classification is gradually increased. Outlines of such training based on feedback can be applied to classifiers that perform the initial extraction of information objects, as well as to classifiers that implement the functions of “arbitration” of conflicting objects, thereby gradually increasing the overall quality of information extraction.

[00039] In some embodiments, the definition of a classifier used to extract information may contain one or more parameters, the values of which may be adjusted using machine learning methods, and may also include one or more hyperparameters, the values of which may be determined by some external to methods of machine learning by operations or processes. In an illustrative example, the values of one or more hyperparameters of the information extraction classifier can be adjusted iteratively by iteratively performing learning operations of the information extraction classifier, training the conflict arbitration classifier, performing the conflict arbitration function and changing the hyperparameter values, thus iteratively optimizing the quality indicator using which evaluates the final list of extracted information objects, as described in more detail in Also in this document.

[00040] Systems and methods operating in accordance with one or more embodiments of the present invention can be used to perform various natural language processing operations, such as machine translation, semantic search, classification and clustering of objects, etc. In some implementations, the extracted information can be visually represented using a graphical user interface, for example, by displaying the identifiers of classes of information objects in a visual comparison with the corresponding fragments of the natural language text.

[00041] Various aspects of the methods and systems mentioned above are described in detail later in this document using examples, and not a way of limitation.

[00042] FIG. 1 shows a block diagram of one illustrative example of a method for extracting information using several extraction methods in accordance with one or more embodiments of the present invention. Method 100 and / or each of its individual functions, standard programs, subroutines, or operations may be performed by one or more processors of a computing system (eg, computing system 1000 in FIG. 16) implementing this method. In some embodiments, method 100 may be performed in a single processing thread. In an alternative approach, method 100 may be implemented using two or more processing threads, with each one implementing one or more separate functions, procedures, subroutines, or actions of this method. In one illustrative example, processing threads that implement method 100 can be synchronized (for example, using semaphores, critical sections, and / or other thread synchronization mechanisms). In an alternative approach, processing flows implementing method 100 can be performed asynchronously with respect to each other.

[00043] At step 110, the computing system implementing the method may receive the text in natural language. In an illustrative example, a computing device may receive text in natural language in the form of an electronic document, which can be obtained by scanning or by applying a different image method from a paper document and then performing optical character recognition (OCR) to obtain the text of the document. In another illustrative example, a computing system may receive text in natural language in the form of one or more formatted files, such as text editor files, e-mail messages, digital data files, etc.

[00044] In step 120, a computing system may apply a first method of extracting information to natural language text. In certain embodiments, the first method of extracting information can use variable sets of rules, automatic classification methods (also called “machine learning-based classifiers”), heuristic approaches, and / or a combination of these methods. When applying the first method of extracting information to a natural language text, a variety of information objects of one or more types of information objects can be created.

[00045] In one of the illustrative examples, the first method of extracting information can use a variable set of production rules to interpret the set of semantic-syntactic structures resulting from the semantic-syntactic analysis of a natural language text. Each semantic structure can represent a sentence in the appropriate natural language. Each semantic structure can be represented by an acyclic graph, which includes a set of nodes corresponding to semantic classes, and a set of arcs corresponding to semantic relations, which is described in more detail below with reference to FIG. 15.

[00046] Production rules that apply to a variety of semantic structures may contain interpretation rules and identification rules. An interpretation rule may contain a left part, represented by a set of logical expressions defined on one or more patterns of the semantic structure, and a right part, represented by one or more statements regarding information objects representing entities referenced in natural language text. A semantic structure pattern may include certain elements of a semantic structure {for example, a connection with a specific lexical / semantic class, a connection with a certain surface or deep position in a specified place of the semantic structure, the presence of ancestor-descendant relations between certain nodes of the semantic structure and a number of levels of the semantic hierarchy between the parent and daughter classes in a specific ontology, the presence of a common ancestor for certain nodes of the semantic structure, the presence of a certain ennyh grammemes or semantics, etc.). Relations between elements of semantic structures can be defined using one or more logical expressions (conjunction, disjunction and denial) and / or operations that characterize the mutual arrangement of nodes in the syntactic-semantic tree. As an illustrative example, such an operation can check whether a node belongs to a subtree of another node.

[00047] The right part of a production rule can assign and (or) change the values of one or more attributes of information objects that display morphological, lexical, syntactic, semantic, and (or) other attributes of a piece of text in natural language. In one of the illustrative examples, the right side of the rule of interpretation may contain a statement relating a piece of text in natural language to a class of information objects.

[00048] The identification rule can be used to establish associative relationships for a pair of information objects that represent the same essence of the real world. An identification rule is a production rule, the left part of which contains one or more logical expressions pointing to the nodes of the semantic tree corresponding to information objects. If the specified pair of information objects satisfies the conditions specified by logical expressions, then the information objects are merged into one information object.

[00049] In an illustrative example, an information extraction method may use one or more machine learning classifiers. Each machine learning classifier can determine the degree of connectedness of a given piece of text in natural language (for example, represented by a set of morphological, lexical, syntactic, semantic, and (or) other attributes) with a certain class of information objects. A computing system can iteratively process many pieces of text in natural language and determine for each text fragment the degree of its connection with one or more classes of information objects. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00050] Classifiers can be trained using pre-existing and (or) dynamically created training data samples, on the basis of which the classes of information objects are correlated with lexical, syntactic, semantic and (or) other attributes of a natural-language text. A training sample of data can include one or more natural language texts accompanied by metadata that define some information objects, their classification, attributes, and corresponding text annotations. In an illustrative example, metadata can be represented by markup associated with text in natural language. In certain embodiments, the training data sample can be iteratively improved by adding new texts in natural language, accompanied by user-verified metadata. Validation of metadata may include receiving, through a graphical user interface, user input confirming or correcting the extracted information objects and their attributes.

[00051] The extracted information objects can be represented as an RDF-graph. RDF (Resource Definition Framework - a resource definition environment) assigns each information object a unique identifier and stores information about such an object as sets of three elements (triplets) of an SPO, where S means "subject" and contains the object identifier, P means "predicate" and defines a property of this object, and O means “object” and stores the value of the property of the object in question. This value can be either a primitive data type (examples: a string, a number, a boolean (logical) value), or an identifier for another object. In one of the illustrative examples, the SPO triplet may associate a fragment from a natural language text with a category of information objects.

[00052] In certain implementations, information extraction methods can be based on ontologies and / or other document models for analyzing lexical, syntactic, semantic, and / or other attributes of a natural language text. Thus, the information retrieval operations indicated at step 120 may be preceded by one or more document preprocessing operations performed to determine the structure of the document and determine the appropriate document model. In one illustrative example, the structure of the document may include a hierarchical multi-level structure in which sections of the document are separated by headings and subheadings. In another illustrative example, the document structure may include one or more tables containing several rows and columns, at least some of which may be associated with headings, which in turn may be organized into a multi-level hierarchy. In another illustrative example, the document structure may include a page structure containing a page header, a page body, and / or a page footer. In another illustrative example, the document structure may include certain text fields associated with predetermined types of information, such as a signature field, date field, address field, name field, etc. The computing system 100 in which this method is implemented can interpret the structure of the document to obtain specific information about the structure of the document, which can be used to expand the textual information contained in this document. In some implementations of the invention, when analyzing structured documents, the computing system may use various auxiliary ontologies containing classes and concepts reflecting the specific structure of the document. Classes of auxiliary ontology can be associated with certain production rules and (or) classifier functions that can be applied to several semantic structures obtained by semantic-syntactic analysis of the corresponding document for inclusion in the resulting set of semantic structures of certain information transmitted by the structure of this document.

[00053] At step 130, a computing system may apply a second information extraction method to natural language text that is different from the first information extraction method. In certain embodiments, the second method of extracting information can use variable rule sets, automatic classification methods (also called “machine learning classifiers”), heuristic approaches, and / or a combination of these methods, which are described in more detail below with reference to step 120. When applying the second method of extracting information can be created the second set of information objects, which may differ from the first set of information objects created by applying the first Methods for information retrieval.

[00054] FIG. 1 shows only two sets of information objects retrieved using two different methods of extracting information, however, various embodiments of method 100 may involve using two or more different methods of extracting information to extract two or more sets of information objects.

[00055] At step 140, the computing system may create an intermediate 250 list of information objects, which may include at least a subset for each set of information objects retrieved by applying various methods for extracting information in steps 120-130. In an illustrative example, a computer system may include in the intermediate list information objects associated with one or more predetermined classes of information objects, with the exception of information objects associated with other classes. In another illustrative example, a computer system may exclude from the intermediate list information objects of one or more predetermined classes of information objects. As described above, the intermediate list may include one or more sets of potentially conflicting information objects, the coexistence of which violates one or more specified rules (for example, overlapping text annotations or cardinality constraints). An intermediate list can be represented by an RDF-graph created by merging RDF-graphs representing the corresponding sets of extracted information objects.

[00056] At step 150, the computing system may determine one or more sets of conflicting information objects, each including two or more information objects, at least two of which have been extracted by various methods of extracting information. In an illustrative example, the set of conflicting information objects may include two information objects, with the first information object related to the first set of information objects extracted by the first information retrieval method, and the second information object related to the second set of information objects extracted by the second information extraction method.

[00057] In an illustrative example, the definition of potentially conflicting information objects may include the definition of the extracted information objects that have at least partially overlapping text annotations. The computing system that implements this method can iteratively process an intermediate list of extracted information objects in order to identify sets of information objects (for example, pairs, triplets, quadruplets, etc.) with intersecting text annotations. In certain embodiments, the computing system may determine that two or more retrieved information objects conflict if their corresponding annotations intersect at least in a threshold word count.

[00058] In another illustrative example, the definition of potentially conflicting information objects may include defining various constraints associated with the attributes of the information objects. In certain embodiments of the implementation of the computing system that implements this method, can perform an assessment of the logical conditions defined for the attributes of information objects (for example, comparing the attribute of one information object with the attribute of another information object). In an illustrative example, a computer system can detect a violation of a cardinality constraint by defining an information object that has a number of attributes of a certain type (for example, a person can have only one date of birth and one place of birth). Cardinality and other restrictions may be specified in the model of the document or ontology associated with the document being processed.

[00059] At step 160, the computing system can use the “arbitration” function of conflicting objects for each identified set of conflicting information objects to resolve conflicts in the intermediate list of information objects, thereby creating a final list of information objects extracted from natural language texts. For each set of conflicting information objects, the function of “arbitration” of conflicting objects can resolve the conflict by analyzing morphological, lexical, syntactic, semantic and / or other attributes of conflicting information objects in order to create a result confirming one or more potentially conflicting objects that change one or more potentially conflicting objects, removing one or more potentially conflicting objects and / or merging one or more sweats potentially conflicting objects. Based on the results of the “arbitration” function of the conflicting objects, the computer system can change the RDF-graph, which represents an intermediate list of extracted information objects, on the basis of which a final RDF-graph is created.

[00060] FIG. 2 schematically shows an example of “arbitration” of conflicting objects, which can be performed in accordance with one or more embodiments of the present invention. As shown in FIG. 2, when applied to the text in natural language 210 of the first information extraction method 220A, the first set of information objects 230A is created, and when applied to the text in natural language 210 of the second information extraction method 220B, a second set of information objects 230B is created. The first set of information objects 230A may include an information object 240A, which may be potentially conflicting with the information object 240V included in the second set of information objects 230V. When applying the “arbitrage” function of conflicting objects 250 to potentially conflicting information objects 260, one of the results of “arbitration” of conflicting objects 260 can be created. (for example, by changing the class of the information object and appropriately resolving an obvious conflict). The result of the “arbitration” of conflicting objects 260B includes the deletion of the information object 240B. At least some of the attributes of the deleted information object 240B may be copied to the remaining information object 240A. The result of the “arbitration” of the conflicting objects 260C includes the merging of information objects 240A and 240B into a new information object 240C.

[00061] In certain implementations, the arbitration function of conflicting information objects may apply one or more custom production rules for evaluating logical conditions defined for morphological, lexical, syntactic, semantic, and / or other attributes of the extracted information objects. In an illustrative implementation example, the attributes of the retrieved information objects, evaluated by the production rules, may include the degrees of confidence associated with the respective information objects. Degrees of confidence can be generated by the methods of extracting information used in steps 120-130.

[00062] Alternatively, the “arbitration” function of conflicting objects may include one or more machine learning classifiers. In an illustrative example, a classifier used by the “arbitration” function of conflicting objects can provide a degree of justification for one or more possible results of “arbitration” (for example, merging possible conflicting objects into a single object, confirming one or more extracted objects, changing one or more extracted objects and (or) deleting one or more retrieved objects). In another illustrative example, the classifier used by the function of “arbitration” of conflicting objects can fix the similarity of two or more information objects representing the same real-life object. If the probability exceeds the threshold value, the function of “arbitration” of the conflicting objects can merge the identified information objects. In another illustrative example, the classifier used by the function of “arbitration” of conflicting objects can create degrees of confidence associated with each information object from a set of conflicting information objects. The “arbitration” function of conflicting objects can delete one or more information objects with a minimum degree of confidence and (or) a degree of confidence below a predetermined threshold.

[00063] Classifiers used by the “arbitration” function of conflicting objects can be trained on the basis of pre-existing or dynamically created training data samples, based on which the results of “arbitration” can be correlated with morphological, lexical, syntactic, semantic and (or) other attributes of the extracted information objects. In certain embodiments, attributes of the retrieved information objects assessed by machine learning based classifiers may include the degrees of confidence associated with the respective information objects. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00064] The classifier can be trained using pre-existing and (or) dynamically created training data samples, on the basis of which the possible results of “arbitration” are correlated with lexical, syntactic, semantic and (or) other attributes of natural-text text. A training sample of data can include one or more natural language texts accompanied by metadata that define the sets of conflicting information objects, their classification, attributes, and the corresponding results of “arbitration” of the conflicting objects. In an illustrative example, metadata can be represented by markup associated with text in natural language. In certain embodiments, the training data sample can be iteratively improved by adding new texts in natural language, accompanied by user-verified metadata. Validation of metadata may include receiving, through a graphical user interface, user input confirming or correcting the extracted information objects and their attributes. In some implementations, user-verified texts and accompanying metadata specifying the extracted information objects, their attributes and text annotations can be used to generate and update training samples used to train classifiers as described in detail below.

[00065] With reference to FIG. 1, at step 170, the computing system can use the final list of information objects to perform tasks or natural language processing operations, including machine translation, semantic indexing, semantic search (including semantic search in multiple languages), document classification, search and presentation of electronic documents (e-discovery), etc. In some implementations, the computer system can display the extracted information objects with visual references to fragments of text in natural language and take into account user actions that confirm or modify text annotations and / or the result of the classification of an object.

[00066] In some embodiments, the definition of a classifier used to extract information may contain one or more parameters, the values of which may be adjusted using machine learning methods, and may also include one or more hyperparameters, the values of which may be determined by some external to methods of machine learning by operations or processes. In other words, the values of the hyper-parameters of the classifier are determined before applying the methods of machine learning to fine-tune the parameters of the classifier.

[00067] In one of the illustrative examples, the hyperparameter of the information extraction classifier can be represented by the regularization parameter of the gradient boosting classifier. In another illustrative example, the hyperparameter of the information extraction classifier can be represented by the parameter β F-measure, obtained by evaluating an information object issued by the classifier, which is defined as follows:

[00068] F _β = (1 + β ² ) * (Accuracy * Completeness) / ((β ² * Accuracy) + Completeness),

where Accuracy = t _p / (t _p + f _p ) and Completeness = t _p / (t _p + f _n ),

t _p is the number of true positive results (correctly classified information objects retrieved), f _p is the number of false positive results (an information object that does not belong to a certain class was classified as belonging to this class), af _{n is the} number of false negative results (an information object that belongs to a particular class has not been classified as belonging to this class).

[00069] In some embodiments, the values of one or more hyperparameters of the information extraction classifier can be adjusted iteratively by iteratively performing learning operations of the information extraction classifier, training the conflict arbitration classifier, performing the conflict arbitration function, and modifying the hyperparameter values based on the result of applying the “ arbitration ”of conflicts, thus, an iterative optimization of the quality indicator is made to evaluate the final list of information objects, as will be described in more detail later in this document with reference to FIG. 3

[00070] FIG. 3 is a block diagram of one example of a classifier training method used to extract information from natural language texts in accordance with one or more embodiments of the present invention. The method 300 and / or each of its individual functions, programs, subroutines, or operations may be executed by one or more processors of the computing system {for example, the computing system 1000 in FIG. 16) implementing this method. In some embodiments, the method 300 may be performed in a single processing thread. In an alternative approach, the method 300 may be implemented by two or more processing threads, with one or more separate functions, procedures, subroutines, or operations of the method implemented in each stream. In an illustrative example, processing threads that implement method 300 can be synchronized {for example, using semaphores, critical sections, and / or other thread synchronization mechanisms). In an alternative approach, processing flows implementing the method 300 may be performed asynchronously with respect to each other.

[00071] At step 310, the computing system implements a method that allows the classifier to retrieve information to be trained in extracting a set of information objects from natural language text. In some embodiments, the classifier definition may include one or more parameters, the values of which are adjusted during the classifier training operation and may additionally include one or more hyperparameters, the values of which may be determined by some operation or process external to the machine learning. In one of the illustrative examples, the hyperparameter of the information extraction classifier can be represented by the regularization parameter of the gradient boosting classifier. In another illustrative example, the hyperparameter of the information extraction classifier can be represented by the parameter β F-measure, obtained by evaluating an information object issued by the classifier.

[00072] In one of the illustrative examples, the annotated corpus of texts used to train the information extraction classifier can be divided into a training data sample and a set of verification data. The training data sample can then be used to determine the values of the parameters of the classifier, and the set of verification data can be used to calculate an indicator for the quality of information extraction (that is, to be represented by a parameterized F-measure). In one of the illustrative examples, the learning process may include setting one or more parameters of the information extraction classifier until the selected quality indicator applied to the retrieved information objects corresponds to a predetermined threshold value.

[00073] The separation of the corpus of texts may include cross-checking of the training sample of data and the set of verification data. To reduce the variability of the result, several iterations of cross-validation can be performed using different divisions, and the model verification results can be aggregated (for example, averaged) by iterations. In an illustrative example to the body of texts in a natural language, the method of k-fold cross-checking can be applied. The method may include randomly splitting the original body of text into k data sets of the same size, one of which is then used as a set of verification data, and the remaining k-1 additional sets are then used as training data samples. The cross-validation process can then be repeated k times so that each of the k data sets is used once as the verification data. After that, k results can be aggregated to create a single calculation.

[00074] At step 320, the computing system may use a classifier for extracting information to a set of verification data, thereby obtaining a first set of information objects.

[00075] At step 330, the computing system may apply to the same set of verification data a second method of extracting information that is different from the previously mentioned information extraction classifier, thereby obtaining a second set of information objects. In one of the illustrative examples, the second method of extracting information can use a set of custom production rules to interpret a set of semantic-syntactic structures obtained by semantic-syntactic analysis of natural language text, as described in more detail later in this document. In another illustrative example, the second information extraction method may use one or more machine learning classifiers that differ from the above information extraction classifier that was used to extract the first set of information objects.

[00076] At step 340, the computing system may determine one or more sets of conflicting information objects, each including two or more information objects, at least two of which have been extracted by various methods of extracting information. In an illustrative example, the set of conflicting information objects may include two information objects, with the first information object related to the first set of information objects extracted using the information extraction classifier, and the second information object related to the second set of information objects extracted using a different information retrieval method as discussed in more detail above.

[00077] At step 350, a computing system may apply a conflict arbitration function to each detected set of conflicting information objects to resolve conflicts, thus creating a final list of information objects extracted from natural language texts, as described in more detail above.

[00078] In an illustrative example, the function of “arbitration” of conflicting objects can be implemented by a classifier based on machine learning, trained to produce a degree of “reasonableness” of one or more possible results of “arbitration” (for example, merging possible conflicting objects into a single object, confirming one or more extracted objects, changing one or more extracted objects and (or) deleting one or more extracted objects). In another illustrative example, a classifier used by the “arbitration” function of conflicting objects can provide the similarity of two or more information objects representing the same real-life object. If the probability exceeds the threshold value, the function of “arbitration” of the conflicting objects can merge the identified information objects. In another illustrative example, a classifier used by the “arbitration” function of conflicting objects can create degrees of confidence associated with each information object from a set of conflicting information objects. The “arbitration” function of conflicting objects can delete one or more information objects with a minimum degree of confidence and (or) a degree of confidence below a predetermined threshold.

[00079] Alternatively, a conflict arbitration function can be implemented by one or more custom rules that evaluate logical conditions defined on morphological, lexical, syntactic, semantic, and / or other attributes of the extracted information objects, as described in more detail here.

[00080] At step 360, the computing system may evaluate a predetermined quality indicator applied to the final set of retrieved information objects. In one illustrative example, a quality score is provided by an F-measure.

[00081] In response to the determination at step 370 that the predefined termination conditions are not met, processing continues at step 380 by changing one or more hyperparameters of the information extraction classifier. In one of the illustrative examples, the hyperparameter can be represented by the parameter β F-measure, which is used to assess the quality of information extraction. Thus, the computing system can modify the value of the parameter β to increase the accuracy or completeness of the results obtained from the information extraction classifier. In some embodiments, the computing system may implement a grid search of parameters, iteratively changing the values of one or more hyperparameters to predetermined increments so that the parameters remain within certain ranges, thus optimizing the quality indicator applied to the final set of extracted information objects. After performing the operations in step 380, the method may return to step 310.

[00082] In response to the determination at step 370 of the fulfillment of the termination conditions, the method can exit the cycle and continue working at step 390. In one of the illustrative examples, the termination conditions can be determined by the maximum number of iterations performed. In response to determining whether a specified number of operations have been completed, the computer system may, at step 390, identify among the values indicated above, predetermined quality indicators, evaluated at step 370, which were generated on all performed iterations, the optimal (i.e. minimum or maximum) quality indicator value and properly identify the values of the hyperparameters of the information extraction classifier that was used in the iteration, which showed the optimal quality indicator.

[00083] In another illustrative example, the termination conditions, tested at step 370, can determine a threshold value corresponding to the quality indicator, which was evaluated at step 370. In response to determining whether the quality indicator matches the specified threshold value, the computing system can detect hyper parameters at step 390 information extraction classifier, which was used in the last iteration, which gave the value of the quality indicator that meets the conditions for termination tions. In response to determining the values of the hyperparameters of the information extraction classifier, the method may stop working. The trained information extraction classifier may then be used in the information extraction method 100.

[00084] As noted above, in some implementations, the function of arbitrating conflicts can be performed by a classifier based on machine learning. The definition of such a classifier may include one or more parameters, the values of which can be adjusted using machine learning methods and may additionally include one or more hyperparameters, the values of which may be determined by some external to the machine learning, operations or process. In other words, the classifier hyperparameter values are determined before applying machine learning methods to fine tune the classifier parameters.

[00085] In an illustrative example, the conflict arbitration classifier hyperparameter can be represented by the regularization parameter of the gradient boosting classifier. In another illustrative example, the hyper-parameter of the conflict arbitration classifier can be represented by the parameter β of the F-measure indicator by evaluating information objects received by the classifier.

[00086] In some implementations, the values of one or more hyperparameters of the conflict arbitration classifier can be iteratively corrected by iteratively performing operations of extracting the first set of information objects, extracting the second set of information objects, changing the values of hyper-parameters based on evaluating the quality of information extraction, and training the classifier of conflict arbitration thus iteratively optimizing the quality metric, evaluating the final list of extracted information objects. In some implementations, the computer system can implement the search for the best parameters on the grid (grid search) by iteratively changing the values of one or several hyperparameters using predetermined increments, while the parameters remain in certain ranges, thus optimizing the quality metric applied to the final set retrieved information objects, as described in more detail below with reference to FIG. four.

[00087] FIG. 4 shows a flowchart of another example of a classifier training method used to extract information from natural language texts, in accordance with one or more aspects of the present disclosure. The method 400 and / or each of its individual functions, subroutines, subroutines, or operations may be performed by one or more computer system processors (for example, computer system 1000 in FIG. 16) implementing this method. In some implementations, method 400 may be performed by a single processing thread. Alternatively, method 400 may be performed by two or more processing threads, with each thread implementing one or more separate functions, subroutines, subroutines, or operations of this method. In an illustrative example, a method for implementing 400 processing threads may be synchronized (for example, using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the method 400 of implementing processing flows may be performed asynchronously with respect to each other.

[00088] At step 410, a computer system implementing the method may apply a first method of extracting information to the natural language text. In some implementations, the first method of extracting information can use custom rulesets, automatic classification methods (also known as “machine learning-based classifiers”), heuristic approaches, and / or combinations thereof. Applying the first method of extracting information to a natural language text can create a first set of information objects of one or more types of information objects.

[00089] At block 420, a computer system may apply a second method of extracting information to the natural language text that is different from the first method of extracting information. In some implementations, the second method of extracting information can use custom rulesets, automatic classification methods (also known as “machine learning-based classifiers”), heuristic approaches, and / or combinations thereof. The application of the second method of extracting information to the text in natural language may create a second set of information objects, which may differ from the first set of information objects created using the first method of information extraction.

[00090] At step 430, the computer system may change the values of one or more hyperparameters of the conflict arbitration classifier. In an illustrative example, the hyperparameter can be represented by the regularization parameter of the gradient boosting method. In another illustrative example, the hyperparameter can be represented by the parameter β of the F-measure metric by evaluating the information objects received by the classifier. In some implementations, the computer system can implement the search for the best parameters on the grid (grid search) by iteratively changing the values of one or several hyperparameters using predetermined increments, while the parameters remain in certain ranges, thus optimizing the quality metric applied to the final set extracted information objects.

[00091] At step 440, the computer system may train a conflict arbitration classifier to perform a conflict arbitration function on a set of conflicting information objects in order to compile a final list of the extracted information objects. In an illustrative example, a set of conflicting information objects may contain two information objects, so that the first information object belongs to the first set of information objects that was extracted by the information extraction classifier, while the second information object belongs to the second set of information objects that were extracted using the second extraction method information as described in more detail above.

[00092] In an illustrative example, a classifier used by the arbitration function of conflicts can give a degree of validity to one or several possible results of arbitration (for example, combining two or more potentially conflicting objects into one object, confirming one or more extracted objects, changing one or more extracted objects and / or deleting one or more extracted objects). In another illustrative example, the classifier used by the conflict arbitration function can estimate the likelihood that two or more information objects represent the same real-life object. If the probability exceeds a threshold value, the conflict arbitration function may combine the identified information objects. In another illustrative example, the classifier used by the conflict arbitration function can produce confidence levels associated with each information object from a set of conflicting information objects. The conflict arbitration function can remove one or more information objects with the least confidence and / or confidence that are below a predetermined threshold.

[00093] In some implementations, the definition of a conflict arbitration classifier may include one or more parameters, the values of which are configured using the classifier training procedure and may additionally include one or more hyperparameters, the values of which may be determined by some external process in relation to machine learning or operation.

[00094] Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc. In an illustrative example, the learning process may include setting one or more parameters of the information extraction classifier until a predetermined threshold of the selected quality metric is reached.

[00095] Classifiers used in the function of arbitration of conflicts can be trained using pre-existing and / or dynamically created training data sets that establish the relationship of the results of arbitration with morphological, lexical, syntactic, semantic and / or other attributes of the extracted information objects.

[00096] A training dataset can contain one or more natural language texts, accompanied by metadata that define a set of conflicting information objects, their classification, attributes, and corresponding arbitration results. In an illustrative example, metadata can be represented by markup associated with text in natural language. In some implementations, the training data set can be iteratively expanded by adding new texts in natural language, followed by user-verified metadata. Validation of metadata may include, through a graphical user interface (GUI), user input confirming or correcting the retrieved information objects and their attributes.

[00097] In an illustrative example, an annotated text corpus used to train a conflict arbitration classifier can be divided into a training data set and a test data set. The training data set can then be used to determine the values of the parameters of the classifier, while the test data set can be used to calculate the information extraction quality metric (for example, by the presented parameterized F-measure). In an illustrative example, the learning process may include adjusting one or more parameters of the conflict arbitration classifier until the selected quality metric applied to the final list of retrieved information objects corresponds to a predetermined threshold.

[00098] At step 450, the computer system may use the conflict arbitration function for each detected set of conflicting information objects to resolve conflicts, thus creating a final list of information objects extracted from natural language texts, as described in more detail above.

[00099] At step 460, the computer system may calculate a predetermined quality metric as applied to the final set of retrieved information objects. In an illustrative example, a quality metric can be represented by an F-measure.

[000100] In response to the determination at step 470 that the predetermined termination condition is not fulfilled, the method may return to block 410; otherwise, the method may exit the loop and continue processing in block 480. In an illustrative example, the termination condition may indicate the maximum number of iterations to be performed. In response to performing the specified number of iterations, the computer system may, at step 470, identify among the values of the above predetermined quality metric, calculated at step 460, which were obtained with all the iterations performed, the optimal (for example, maximum or minimum) value of the quality metric and identify those values of the hyperparameters of the conflict arbitration classifier used in the iteration process, which gave the optimal quality metric.

[000101] In another illustrative example, the termination condition, which is evaluated at block 470, may indicate a threshold value that must be met by the quality metric that was calculated at step 460. In response to determining that the quality metric corresponds to the specified threshold value, the computer system may, at step 490, identify the values of the hyperparameters of the information extraction classifier that were used at the last iteration performed, which gave a quality indicator that satisfies the condition complete. In response to identifying the values of the hyperparameters of the conflict arbitration classifier, the method may end. A trained conflict arbitration classifier can then be used by method 100 to extract information.

[000102] As stated above, production rules, classifiers and (or) other methods of extracting information and “arbitration” of conflicting objects can perform an analysis of various morphological, lexical, syntactic, semantic and (or) other attributes of natural text text. Such attributes can be created by performing semantic-syntactic analysis of text in natural language, which is shown schematically in FIG. five.

[000103] FIG. 5 is a block diagram of one illustrative example implementation of method 200 for performing semantic-syntactic analysis of a natural language sentence 212 in accordance with one or more aspects of the present invention. Method 400 may be applied to one or more syntactic units (eg, sentences) included in a particular text corpus to form a set of semantic-syntactic trees corresponding to syntactic units. In various illustrative examples, natural language sentences to be processed by method 400 can be extracted from one or more electronic documents, which can be created by scanning (or using other means of obtaining images of paper documents) and optical character recognition (OCR) to obtain texts corresponding to these documents. Natural language sentences can also be extracted from various other sources, including emails, texts from social networks, files with digital content, processed using speech recognition methods, etc.

[000104] In block 214, the computing device that implements this method can conduct a lexico-morphological analysis of the sentence 212 to establish the morphological meanings of the words that make up the sentence. In this document, the word “morphological meaning” means one or more lemmas (that is, canonical or dictionary forms) corresponding to a word, and a corresponding set of values of grammatical features that define the grammatical meaning of a word. Such grammatical features may include the lexical category (part of speech) of a word and one or more morphological and grammatical features (for example, case, gender, number, conjugation, etc.). Due to homonymy and (or) matching grammatical forms corresponding to different lexical-morphological meanings of a certain word, two or more morphological meanings can be established for a given word. A more detailed description of an illustrative example of a lexico-morphological analysis of a sentence is given in more detail later in this document with reference to FIG. 6

[000105] In block 215, the computing device may perform a rough syntax analysis of sentence 212. A rough syntax analysis may include the use of one or more syntax models that can be correlated with elements of sentence 212, followed by the establishment of surface (i.e. syntax) links within sentence 212 for obtaining a graph of generalized components. In this document, "component" means a group of adjacent words of the original sentence, functioning as one grammatical essence. The component includes the core in the form of one or more words and may also include one or more child components at lower levels. A child component is a dependent component that can be correlated with one or more parent components.

[000106] In block 216, the computing device may conduct an exact syntax analysis of the sentence 212 to form one or more syntax trees of the sentence. Among the various syntactic trees based on a specific evaluation function, taking into account the compatibility of the lexical meanings of the words of the original sentence, surface relationships, depth relationships, etc. one or more of the best syntax trees corresponding to sentence 212 may be selected.

[000107] In block 217, a computing device can process syntax trees to form a semantic structure 218 corresponding to sentence 212. Semantic structure 218 can include many nodes corresponding to semantic classes and can also include many arcs corresponding to semantic relations (for more details, see below in this document).

[000108] FIG. 6 schematically illustrates an example of a lexico-morphological structure of a sentence in accordance with one or more aspects of the present invention. An example of a lexico-morphological structure 300 may include a plurality of “lexical meaning - grammatical meaning” pairs for example sentences. As an illustrative example, "ll" can be correlated with the lexical meaning of "shall" and "will" 514. The grammatical meaning associated with the lexical meaning of 512 is <Verb, GTVerbModal, ZeroType, Present, Nonnegative, Composite II>. The grammatical meaning associated with the lexical meaning 314 is <Verb, GTVerbModal, ZeroType, Present, Nonnegative, Irregular, Composite II>.

[000109] FIG. 7 schematically illustrates the used language descriptions 210, including morphological descriptions 201, lexical descriptions 203, syntactic descriptions 202 and semantic descriptions 204, as well as relations between them. Among them, morphological descriptions 201, lexical descriptions 203 and syntactic descriptions 202 depend on the language. The set of language descriptions 610 is a model of a specific natural language.

[000110] As an illustrative example, a certain lexical meaning in lexical descriptions 203 can be correlated with one or more surface models of syntactic descriptions 202 corresponding to the lexical meaning. A certain surface model of syntactic descriptions 202 can be correlated with a deep model of semantic descriptions 204.

[000111] FIG. 8 schematically illustrates several examples of morphological descriptions. The components of morphological descriptions 201 may include: descriptions of inflection 710, grammatical system 720, descriptions of word formation 730, and others. The grammatical system 720 includes a set of grammatical categories, such as part of speech, case, gender, number, face, reflexivity, time, type and their meanings (so-called "grammymes"), including, for example, adjective, noun or verb; nominative, accusative or genitive; feminine, masculine or neuter, etc. The corresponding grammemes can be used to compile a description of inflection 710 and a description of word formation 730.

[000112] The description of inflection 710 defines the forms of a given word depending on its grammatical categories (for example, case, gender, number, time, etc.) and broadly includes or describes various possible forms of the word. Description of word formation 730 determines which new words can be formed from a given word (for example, compound words).

[000113] In accordance with one aspect of the present invention, constituent models can be used in establishing syntactic relationships between elements of the original sentence. The constituent is a group of adjacent words in a sentence that behave as a whole. The core of the component is the word, it can also contain the child components of lower levels. A child component is a dependent component and can be attached to other components (parent) to build syntactic descriptions 202 of the original sentence.

[000114] FIG. 9 shows examples of syntactic descriptions. The components of the syntactic descriptions 202 may include, among other things, surface models 910, descriptions of surface positions 920, descriptions of referential and structural control 956, descriptions of management and agreement 940, descriptions of non-tree syntax 950 and analysis rules 960. possible syntactic structures of the original sentence in a given natural language, taking into account the free linear order of words, non-wood syntactic phenomena (for example, coordination , ellipsis, etc.), referential relations and other factors.

[000115] Surface models 910 can be represented as a set of one or several syntactic forms (“synth forms” 912) for describing possible syntactic structures of sentences included in syntactic description 202. In general, the lexical meaning of a word in natural language can be associated with surface (syntactic) models 910. A surface model can be components that are possible if the lexical meaning acts as a "core". The surface model may include a set of surface positions of child elements, a description of the linear order and (or) diathesis. In this document, "diathesis" means a certain relationship between surface and deep positions and their semantic roles, expressed through deep positions. For example, the diathesis can be expressed by the verb's voice: if the subject is an agent of the action, the verb is in the active voice, and when the subject is the direction of the action, this is expressed by the passive voice of the verb.

[000116] In the constituent model, a plurality of surface positions of 915 child components and descriptions of their linear order 916 can be used to describe the grammatical meanings 914 of possible placeholders for these surface positions. Diathesis 917 represents the correspondence between surface positions 915 and depth positions 514 (as shown in Fig. 11). Communicative descriptions 980 describe the communicative order in a sentence.

[000117] The linear order description (916) can be represented as linear order expressions, reflecting the sequence in which different surface positions (915) can appear in a sentence. Linear order expressions may include variable names, surface position names, parentheses, grammemes, the or operator (or), etc. As an illustrative example, the linear order description of a simple sentence "Boys play football" can be represented as "Subject Core Object_Direct" (Subject - Core - Direct Supplement), where Subject (Subject), Core (Core) and Object_Direct (Direct Supplement) are the names of the surface positions 915, corresponding to the word order.

[000118] Communicative descriptions 980 can describe the word order in the syntforme 912 from the point of view of communicative acts, presented in the form of communicative order expressions that are similar to linear order expressions. The descriptions of control and negotiation 940 may include rules and restrictions on the grammatical meanings of the components being joined, which are used during the syntax analysis.

[000119] The descriptions of non-tree syntax 950 can be created to reflect various linguistic phenomena, such as ellipsis and agreement, they are used in the transformations of syntactic structures that are created at various stages of analysis in various embodiments of the invention. Descriptions of non-wood syntax 950 may, among other things, include descriptions of ellipsis 952, descriptions of concordance 954, and descriptions of referential and structural controls 930.

[000120] The analysis rules 960 may describe the properties of a particular language and be used as part of semantic analysis. Analysis rules 960 can include rules for calculating semantics 962 and normalization rules 964. Normalization rules 964 can be used to describe transformations of semantic structures that may vary in different languages.

[000121] FIG. 10 shows an example of semantic descriptions. The components of semantic descriptions 204 are language independent and may, among other things, include the semantic hierarchy 510, the descriptions of the underlying positions 520, the semantics system 530 and the pragmatic descriptions 540.

[000122] The core of semantic descriptions can be represented by the semantic hierarchy 510, which can include semantic concepts (semantic entities), also called semantic classes. The latter can be arranged in a hierarchical structure, reflecting the relationship "parent-child".

In general, a child semantic class may inherit one or more properties of its direct parent and other semantic ancestral classes. As an illustrative example, the semantic class SUBSTANCE (Substance) is a subsidiary semantic class of the ENTITY class (Entity) and the parent semantic class for the classes GAS, (Gas), LIQUID (Liquid), METAL (Metal), WOOD MATERIAL (Wood), etc. .

[000123] Each semantic class in the semantic hierarchy 510 may be accompanied by a deep model 512. The deep model 512 of the semantic class may include multiple depth positions 514, which may reflect the semantic roles of the child components in different sentences with objects of the given semantic class as the core of the parent component. Depth model 512 may also include possible semantic classes that act as placeholders for deep positions. Depth positions (514) can express semantic relationships, including, for example, "agent" (agent), "addressee" (addressee), "instrument" (instrument), "quantity" (quantity), etc. A child semantic class can inherit and refine the deep model of its immediate parent semantic class.

[000124] The descriptions of the deep positions 520 reflect the semantic roles of the child components in the deep models 512 and can be used to describe the general properties of the deep positions 514. The descriptions of the deep positions 520 may also contain grammatical and semantic restrictions on the place holders of the deep positions 514. Properties and restrictions related with depth positions 514 and their possible placeholders in different languages, can be largely similar and often identical. Thus, the deep positions 514 are language independent.

[000125] The semantics system 530 may be a set of semantic categories and semantics that represent the meanings of semantic categories. As an illustrative example, the semantic category "DegreeOflComparison" (Degree of Comparison) can be used to describe the degree of comparison of adjectives and include the following semantemes: "Positive", "ComparativeHigherDegree" (Comparative degree of comparison), "SuperlativeHighestDegree" (Excellent degree of comparison) and others. As another illustrative example, the semantic category "RelationToReferencePoint" can be used to describe an order (spatial or temporal in the broad sense of the words analyzed), such as before or after a point or event, and include the semanthemes "Previous" ( Previous) and "Subsequent" (Subsequent). As another illustrative example, the semantic category "EvaluationObjective" (Evaluation) can be used to describe an objective assessment, such as "Bad", "Good", etc.

[000126] The semantem system 530 may include language independent semantic attributes that can express not only semantic characteristics, but also stylistic, pragmatic and communicative characteristics. Some semanthemes can be used to express an atomic meaning that finds regular grammatical and / or lexical expression in natural language. According to their intended purpose and use of the system, semanthemes can be divided into categories, for example, grammatical semanthemes 532, lexical semanthemes 534 and classifying grammatical (differentiating) semantemes 536.

[000127] Grammar semanthemes 532 can be used to describe the grammatical properties of constituents when converting a syntactic tree to a semantic structure. The lexical semantemes 534 can describe specific properties of objects (for example, being flat and be liquid) and used in the descriptions of the deep positions 520 as a restriction to the fillers of the deep positions (for example, for the verbs "face (with ) "(lined) and" flood "(fill), respectively). Classifying grammatical (differentiating) semanthemes 536 can express the differential properties of objects within a single semantic class. As an illustrative example in the semantic class HAIRDRESSER (HAIRDRESSER), the semantheme “RelatedToMen” (Refers to men) is assigned to the lexical meaning “barber” unlike other lexical meanings that also belong to this class, for example, “hairdresser”, “hairstylist” and etc. Using data language-independent semantic properties that can be expressed as elements of a semantic description, including semantic classes, in-depth positions and semantics, you can extract semantic information in accordance with one or more aspects of the present invention.

[000128] Pragmatic descriptions 540 allow you to assign a specific theme, style or genre to the texts and objects of the semantic hierarchy 510 (for example, Economic Policy, Foreign Policy, Jurisprudence, Legislation, Trade, Finance, etc. .d.) Pragmatic properties can also be expressed by semanthemes. As an illustrative example, a pragmatic context can be taken into account in semantic analysis.

[000129] FIG. 11 shows an example of lexical descriptions. Lexical descriptions (203) are a set of lexical meanings 612 of a specific natural language. For each lexical meaning 612, there is a link 602 with its language-independent semantic parent in order to indicate the position of any given lexical meaning in the semantic hierarchy 510.

[000130] The lexical meaning 612 in the lexico-semantic hierarchy 510 can be correlated with the surface model 910, which in turn through one or more diathesis 917 can be correlated with the corresponding depth model 512. The lexical meaning 612 can inherit its parent's semantic class and specify its deep model 512.

[000131] The surface model 910 of a lexical meaning may include one or more syntax forms 912. The surface format 912 of the surface model 910 may include one or more surface positions 915, including the corresponding descriptions of their linear order 916, one or more grammatical meanings 914, expressed in the form of a set of grammatical categories (gramme), one or more semantic restrictions related to placeholder placeholders, and one or more diathesis 917. Semantic restrictions related to Those with a certain surface position filler can be represented as one or more semantic classes whose objects can fill this surface position.

[000132] FIG. 12 schematically illustrates examples of data structures that may be used within one or more of the methods of the present invention. Referring again to FIG. 5, in block 214, a computing device implementing this method can perform a lexico-morphological analysis of a sentence 212 for constructing a lexico-morphological structure 722 according to FIG. 12. The lexico-morphological structure 722 may include multiple correspondences of lexical and grammatical meanings for each lexical unit (eg, words) of the original sentence. FIG. 6 schematically illustrates an example of a lexico-morphological structure.

[000133] Returning again to FIG. 6, at block 215, the computing device may perform a crude syntactic analysis of the original sentence 212 to build a graph of generalized components 732 according to FIG. 12. Rough syntax analysis involves applying one or more possible syntactic models of possible lexical meanings to each element of the set of elements of the lexico-morphological structure 722, in order to establish a set of potential syntactic relations within the original sentence 212, represented by the graph of generalized components 732.

[000134] The graph of generalized components 732 can be represented by an acyclic graph including a set of nodes corresponding to the generalized components of the original sentence 212 and including a set of arcs corresponding to surface (syntactic) positions that can express different types of relationships between the generalized lexical values. Within this method, a set of potentially applicable syntactic models can be used for each element of the set of elements of the lexico-morphological structures of the original sentence 212 to form a set of components of the original sentence 212. Then the method can consider the many possible components of the original sentence 212 to build a graph of generalized components 732 basis of a set of components. The graph of generalized components 732 at the surface model level may reflect the set of potential connections between the words of the original sentence 212. Since the number of possible syntactic structures may be relatively large, the graph of generalized components 732 may, in general, include redundant information, including a relatively large number of lexical values on certain nodes and (or) surface positions on certain arcs of the graph.

[000135] The graph of generalized components 732 can initially be constructed in the form of a tree, starting from the leaf nodes (leaves) and moving further to the root, by adding child components that fill the surface positions 915 of the set of parent components, so that all lexical units of the original sentence are covered. 212.

[000136] In some embodiments, the implementation of the root of the graph of generalized components 732 is a predicate. In the process described above, a tree can become a graph, since certain lower level components can be included in one or more top level components. The set of components that represent certain elements of the lexico-morphological structure can then be generalized to obtain generalized components. Constituents may be generalized based on their lexical meanings or grammatical meanings 914, for example, based on parts of speech and the relationship between them. FIG. 13 schematically illustrates an example of a graph of generalized components.

[000137] In block 216, the computing device may conduct an exact syntax analysis of the sentence 212 to form one or more syntax trees 742 according to FIG. 12 on the basis of the graph of generalized components 732. For each syntactic tree, the computing device can determine the integral estimate on the basis of a priori and calculated estimates. The tree with the best score can be selected to build the best syntactic structure 746 of the original sentence 212.

[000138] In the course of building the syntactic structure 746 based on the selected syntax tree, the computing device can establish one or several non-wood links (for example, by creating an additional link among at least two graph nodes). If this process fails, the computing device may select a syntactic tree with a conditionally optimal estimate that is closest to optimal, and an attempt is made to establish one or more non-wood links in the tree. Finally, as a result of accurate parsing, a syntactic structure 746 is created, which is the best syntactic structure corresponding to the original sentence 212. In fact, as a result of selecting the best syntactic structure 746, the best lexical values 240 for the elements of the original sentence 212 are determined.

[000139] In block 217, the computing device can process the syntax trees to form the semantic structure 218 corresponding to sentence 212. The semantic structure 218 may reflect the semantics transmitted by the original sentence in terms of language independent terms. The semantic structure 218 can be represented as an acyclic graph (for example, a tree, possibly supplemented by one or more non-wood links (arc of a graph). The words of the original sentence are represented by nodes with corresponding language-independent semantic classes of the semantic hierarchy 510. The arcs of the graph are deep (semantic a) relations between elements of the sentence. The transition to the semantic structure 218 can be carried out using analysis rules 960 and involves the correlation of one or more attributes ( reflecting the lexical, syntactic and (or) semantic properties of words of the original sentence 212) with each semantic class.

[000140] FIG. 14 shows an example of the syntactic structure of the sentence generated from the graph of generalized components shown in FIG. 13. Node 901 corresponds to the lexical element "life" (life) 906. Using the method of syntactic-semantic analysis described in this document, a computing device can establish that the lexical element "life" (life) 906 represents one of the forms of the lexical meaning correlated with the semantic class "LIVE" (904) and fills the surface position $ Adjunct_Locative 905) in the parent component represented by the Verb: succeed: succeed: TO_SUCCEED (907) control node.

[000141] FIG. 15 shows the semantic structure corresponding to the syntactic structure in FIG. 15. With regard to the aforementioned lexical element "life" (906) in FIG. 13, the semantic structure includes a lexical class 1010 and a semantic class 1030, corresponding to those shown in FIG. 15, however, instead of the surface position (905), the semantic structure includes the deep position "Sphere" (sphere of activity) 1020.

[000142] As noted earlier in this document, a “ontology” can be a model that represents objects related to a specific area of knowledge (subject area) and the relationship between these objects. Thus, ontology differs from the semantic hierarchy, despite the fact that it can be correlated with the elements of the semantic hierarchy through certain relationships (also called “anchors”). An ontology can include definitions of a certain set of classes, where each class corresponds to a domain concept. Each class definition may include definitions of one or more objects assigned to the class. According to generally accepted terminology, the ontology class can also be called a “concept”, and an object belonging to a class can mean an instance of a given concept.

[000143] In accordance with one or more aspects of the present invention, a computing device in which the methods described in the present description are implemented may index one or more parameters resulting from semantic-syntactic analysis. Thus, the methods of the present invention allow to consider not only the set of words in the source text corpus, but also the set of lexical meanings of these words, preserving and indexing all the syntactic and semantic information obtained during the syntactic and semantic analysis of each sentence of the original text corpus. Such information may additionally include data obtained during the intermediate stages of analysis, as well as the results of lexical choice, including the results obtained during the resolution of ambiguities caused by homonymy and (or) concurrent grammatical forms corresponding to different lexical and morphological meanings of some words of the original language.

[000144] For each semantic structure, you can create one or more indexes. The index can be represented as a data structure in memory, for example, in the form of a table consisting of several records. Each entry can be a mapping between a certain element of a semantic structure (for example, one word or several words, a syntactic relationship, a morphological, syntactic or semantic property, or a syntactic or semantic structure) and one or more identifiers (or addresses) of cases when a given element is used semantic structures in the source text.

[000145] In some embodiments, the implementation of the index may include one or more values of morphological, syntactic, lexical and (or) semantic parameters. These values can be created in a two-stage semantic analysis process (for a more detailed description, see this document). The index can be used to perform various natural language processing tasks, including for performing semantic search.

[000146] A computing device that implements this method can extract a wide range of lexical, grammatical, syntactic, pragmatic and (or) semantic characteristics in the course of conducting syntactic-semantic analysis and creating semantic structures. In an illustrative example, the system can extract and save certain lexical information, data on the belonging of certain lexical units to semantic classes, information on grammatical forms and linear order, information on the use of certain forms, aspects, tonality (for example, positive or negative), deep positions, non-timber connections, semanthemes, etc.

[000147] A computing device that implements the methods described here can analyze using one or more text analysis methods described in this document and index any one or more parameters of language descriptions, including lexical meanings, semantic classes, grammes, semantemes, and etc. Indexing a semantic class can be used in various tasks of natural language processing, including semantic search, classification, clustering, text filtering, etc. Indexing lexical values (instead of indexing words) allows you to search not only words and forms of words, but also lexical values, those. words that have certain lexical meanings. A computing device that implements the methods of the present invention can also store and index syntactic and semantic structures created by one or more text analysis methods described herein for using these structures and / or indices when conducting semantic search, classification, clustering and filtering documents .

[000148] FIG. 16 illustrates an illustrative example of a computing system 1000 that can execute a set of instructions that cause the computing system to execute any one or more of the methods of the present invention. A computer system can be connected to another computer system via a local network, a corporate network, an extranet network, or the Internet. The computing system may operate as a server or client in a client-server network environment or as a peer-to-peer computing device in a peer-to-peer (or distributed) network environment. A computer system can be represented by a personal computer (PC), tablet PC, set-top box (STB), pocket PC (PDA), cell phone, or any computer system capable of executing a set of commands (sequentially or otherwise) defining the operations that should be performed by this computing system. In addition, although only one computing system is shown, the term “computing system” may also include any combination of computing systems that separately or jointly execute a set (or more sets) of instructions for executing one or more of the methods discussed herein. .

[000149] An example of a computing system 1000 includes a processor 502, main storage device 504 {for example, read-only memory (ROM) or dynamic random access memory (DRAM) and storage device 518, which communicate with each other via bus 530.

[000150] The processor 502 may be represented by one or more general-purpose computing systems, for example, a microprocessor, a central processor, etc. In particular, processor 502 may be a full instruction set microprocessor (CISC), a reduced instruction set microprocessor (RISC), a microprocessor with superlong length command words (VLIW), a processor implementing another instruction set or processors implementing a combination of instruction sets. The processor 502 may also be one or more special purpose computing systems, such as a custom integrated circuit chip (ASIC), a user-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, and the like. The processor 502 is implemented with the ability to execute commands 526 to implement the operations and functions discussed herein.

[000151] The computing system 1000 may further include a network interface device 522, a display device 510, a character input device 512 (eg, a keyboard), and an input device in the form of a touch screen 514.

[000152] The storage device 518 may comprise a computer readable storage medium 524, in which one or more sets of instructions 526 are stored and in which one or more of the methods or functions discussed herein are implemented. Commands 526 may also be located entirely or at least partially in the main storage device 504 and / or in the processor 502 during execution thereof in the computer system 1000, wherein the random access memory 504 and the processor 502 are also computer readable storage media. Commands 526 may also be transmitted or received over network 516 through network interface device 522.

[000153] In some embodiments of the invention, instruction set 526 may contain instructions of method 100, 300 and / or 400 for extracting information from natural language texts and training classifiers in accordance with one or more embodiments of the present invention. Although computer readable storage medium 524 is shown in the example of FIG. 16 as one medium, the term “computer readable medium” should be understood in a broad sense, implying one or more media (for example, a centralized or distributed database and / or corresponding caches and servers) in which one or more instruction sets are stored. The term “computer readable storage medium” is also to be understood as including any medium that can store, encode, or transfer a set of instructions to be executed by a machine and that ensures that the machine executes any one or more methods of the present invention. Therefore, the term “computer readable storage medium” refers, inter alia, to solid-state storage devices, as well as optical and magnetic media.

[000154] The methods, components, and functions described in this document can be implemented using discrete hardware components, or they can be built into the functions of other hardware components, such as ASICS (customized custom integrated circuit), FPGA (programmable logic integrated circuit) , DSP (Digital Signal Processor) or similar devices. In addition, methods, components, and functions may be implemented using firmware modules or hardware functional diagrams. Methods, components and functions can also be implemented using any combination of hardware and software components, or exclusively using software.

[000155] In the above description, numerous details are set forth. However, it should be obvious to anyone skilled in the art who has read this description that the present invention can be practiced without these specific details. In some cases, well-known structures and devices are shown in block diagram form without detail in order not to complicate the description of the present invention.

[000156] Some parts of the description of preferred embodiments of the invention are presented in the form of algorithms and a symbolic representation of operations with data bits in a computer storage device. Such descriptions and presentation of algorithms are tools used by specialists in the field of data processing, which provides the most efficient transfer of the essence of the work to other specialists in this field. In the context of the present description, as is customary, the algorithm is called a logically consistent sequence of operations leading to the desired result. Operations involve actions that require physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and carried out other manipulations. It is sometimes convenient, first of all for ordinary use, to describe these signals in the form of bits, values, elements, symbols, terms, numbers, etc.

[000157] However, it should be borne in mind that all of these and similar terms must be associated with the corresponding physical quantities and that they are only convenient notation applied to these quantities. Unless explicitly stated otherwise, it is assumed that in the following description, the terms "definition", "calculation", "calculation", "receipt", "establishment", "definition", "change", etc. relate to actions and processes of a computing system or a similar electronic computing system that uses and converts data presented as physical (for example, electronic) values in registers and storage devices of a computing system into other data also represented as physical values in memory devices or computer system registers or other devices for storing, transmitting, or displaying such information.

[000158] The present invention also relates to a device for performing the operations described herein. Such a device can be specially designed for the required purposes, or it can be a universal computer that is selectively driven or additionally configured using a program stored in a computer storage device. Such a computer program can be stored on a machine-readable data carrier, for example, among other things, on any type of disk, including floppy disks, optical disks, CD-ROMs and magneto-optical disks, permanent storage devices (RAM), random access memory (RAM), EPROM , EPROM, magnetic or optical cards and media of any type suitable for storing electronic information.

[000159] It should be understood that the above description is intended to illustrate and not to limit the essence of the invention. Various other embodiments of the invention will become apparent to those skilled in the art after reading and understanding the above description. On this basis, the scope of the invention should be determined by taking into account the appended claims, as well as all the uses of equivalent methods to which the claims are equally applicable.

Claims (65)

1. A method of extracting information from natural language texts, including: training information extraction classifier to extract the first set of information objects from the text in natural language, and the definition of information extraction classifier includes one or more hyperparameters; retrieving a list of retrieved information objects by performing the arbitration function of conflicts against a set of conflicting information objects, for example, the first information object from a set of conflicting information objects belonging to the first set of information objects and the second information object from the set of conflicting information objects belonging to the second set of information objects objects extracted from natural language text; changing the values of the hyperparameters of the information extraction classifier; and optimization of the information extraction quality index for the list of extracted information objects by iteratively repeating the training operations of the information extraction classifier, performing the function of arbitration of conflicts and changing the values of hyperparameters. 2. The method according to p. 1, characterized in that the function of arbitration of conflicts additionally includes: training classifier arbitration conflicts used to implement the function of arbitration conflicts. 3. The method according to p. 1, characterized in that the training classifier information extraction further includes: determining a set of values of the set of parameters of the information extraction classifier for optimizing the quality indicator, with the quality parameter being provided as a hyperparameter. 4. A method according to claim 1, characterized in that the training of the information extraction classifier is performed using a training sample of data containing annotated text in a natural language, including a set of text annotations, where each text annotation is associated with an information object of a known category. 5. The method of claim 1, wherein the conflict arbitration function performs at least one of the following actions: changing the first information object, deleting the first information object, or merging two or more information objects from a set of conflicting information objects. 6. The method according to claim 1, further comprising: the use of the information extraction classifier for performing a natural language processing operation. 7. A method according to claim 1, further comprising: identifying a set of conflicting information objects by determining that the first text annotation associated with the first information object intersects with the second text annotation associated with the second information object. 8. The method according to claim 1, further comprising: identifying a set of conflicting information objects by evaluating a logical condition that includes the first attribute of the first information object and the second attribute of the second information object. 9. The method according to claim 1, further comprising: identifying a set of conflicting information objects by defining an information object that has a number of attributes of a certain type in an amount above the threshold number of attributes of a certain type. 10. The method according to p. 1, characterized in that the function of arbitration of the conflicting objects further includes: applying a set of production rules to a set of attributes of a natural language text. 11. The method according to p. 1, characterized in that the function of arbitration of the conflicting objects further includes: applying to the set of attributes of a natural language text an “arbitration” classifier of conflicts, on the basis of which at least one of the following is determined: similarity of the first information object and the second information object, representing the same object, degree of confidence of the first information object or degree of confidence of the second information object. 12. The method according to p. 11, further comprising: acceptance of user input confirming the list of information objects; adding to the training sample data in natural language accompanied by metadata, including definitions and text annotations of one or more information objects of the list of information objects; training, using a training sample of data, classifier "arbitration" of conflicts. 13. The method according to claim 1, further comprising: acceptance of user input confirming the list of information objects; adding to the training sample data in natural language, accompanied by metadata, including definitions and textual annotations of one or more information objects of the final list of information objects; training, using the training data sample, information extraction classifier. 14. The method according to claim 1, further comprising: for each information object from the list of extracted information objects, displaying the class identifier of the information object in a visual association with the corresponding fragment of the natural language text. 15. A method of extracting information from texts in natural language, including: extracting the first set of information objects from the text in natural language using the first method of extracting information; extracting the second set of information objects from the text in natural language using the second method of extracting information; changing the values of one or several hyperparameters of the conflict arbitration classifier; Conflict arbitration classifier training, which creates a list of extracted information objects by performing the conflict arbitration function on a set of conflicting information objects, so that the first information object of a set of conflicting information objects belongs to the first set of information objects the second information object of a set of conflicting information objects belongs to the second set of information objects ; optimization of the information extraction quality index, reflecting the quality of the list of extracted information objects, iteratively repeating operations of extracting the first set of information objects, extracting the second set of information objects, changing the values of hyperparameters and training the conflict arbitration classifier. 16. The method of claim 15, wherein the conflict arbitration classifier returns at least one of: the probability that the first information object and the second information object represent the same object, the degree of confidence of the first information object, or the degree of confidence of the second information object. 17. The method according to claim 15, in which the function of arbitration of conflicts performs at least one of: changing the first information object, deleting the first information object, or combining two or more information objects of multiple conflicting information objects. 18. The method according to p. 15, in which the extraction of the first set of information objects further comprises: training of a classifier that provides the degree of association of a fragment of a natural language text with a predetermined class of information objects. 19. A method according to claim 15, in which the extraction of the first set of information objects further comprises: applying a set of production rules to a set of attributes of a natural language text. 20. The method according to p. 15, further comprising: using the conflict arbitration classifier to perform a natural language processing operation. 21. The method according to p. 15, further comprising: getting user input checking the list of extracted information objects; adding a natural language text to the set of training data, accompanied by metadata containing the definitions and textual annotations of one or more information objects from the list of information objects; and using a training data set to train a conflict arbitration classifier. 22. A system for extracting information from natural language texts, including: Memory device; the processor associated with the specified storage device, and this processor is configured to: teach the information extraction classifier to extract the first set of information objects from the text in natural language, the definition of the information extraction classifier includes one or more hyperparameters; get a summary list of extracted information objects by performing the function of arbitration of conflicts in relation to a set of conflicting information objects, for example, the first information object from the set of conflicting information objects belonging to the first set of information objects and the second information object from the set of conflicting information objects belonging to the second a variety of information objects extracted from natural language text; change the values of the hyperparameters of the information extraction classifier; and optimize the information extraction quality indicator for the final list of extracted information objects by iteratively repeating the training operations of the information extraction classifier, performing the function of arbitration of conflicts and changing the values of hyperparameters. 23. The system of claim. 22, characterized in that the processor is configured to: use the information extraction classifier to perform natural language processing operations. 24. A permanent machine-readable data carrier containing executable instructions for extracting information from natural language texts that, when executed, induces a computing system: teach the information extraction classifier to extract the first set of information objects from the text in natural language, the definition of the information extraction classifier includes one or more hyperparameters; retrieve a list of retrieved information objects by performing the function of arbitrating conflicts against a set of conflicting information objects, for example, the first information object from the set of conflicting information objects belonging to the first set of information objects and the second information object from the set of conflicting information objects belonging to the second set of information objects objects extracted from natural language text; change the values of the hyperparameters of the information extraction classifier; and optimize the information extraction quality index for the list of extracted information objects by iteratively repeating the training operations of the information extraction classifier, performing the function of arbitration of conflicts and changing the values of hyperparameters. 25. A permanent machine-readable storage medium according to claim 24, further comprising executable instructions that, when executed, force the computing system: use the information extraction classifier to perform natural language processing operations.

Images