RU2795202C2 - Methods and servers for determining metric-dependent thresholds used with multiple nested metrics for binary classification of a digital object - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: invention is aimed at reducing errors in processing a digital object in case of binary classification. It is achieved due to the fact that the method includes obtaining object-specific test data sets and a set of nested metrics and generating a set of predictions using them. During the first iteration, the server compares the predictions with the first combination of metric-dependent thresholds and generates the first accuracy parameters and the first recall parameters for the first iteration. During the second iteration, the server corrects the prediction from the first combination and thus generates the second combination, compares the predictions with the second combination, and generates the second accuracy parameters and the second recall parameters for the second iteration. The method includes selecting the first combination or the second combination as the target combination of metric dependent thresholds.

EFFECT: reducing errors in processing a digital object in case of binary classification.

18 cl, 7 dwg

Description

The field of technology to which the invention belongs

[01] The present technology relates to systems and methods for binary classification of digital objects. In particular, the present technology relates to methods and servers for determining a combination of metric-dependent thresholds used with a plurality of nested metrics when performing a binary classification of a digital object.

State of the art

[02] Machine learning algorithms (MLA, Machine Learning Algorithm) are used to meet a variety of needs in computer technology. Typically, MLA algorithms are used to generate a prediction based on the data provided to them. Some MLA algorithms are called classifiers and are generally capable of assigning objects to one or more classes. In other words, some MLA algorithms trained on a training dataset containing observations (or instances) of a known category are able to solve the problem of determining which category of a set of categories (or subsets) a new observation belongs to.

[03] The resulting predictive class can be used as additional information about the object to provide users with improved interactive electronic services. For example, the information resulting from the classification of an object can be used by search engines (for example, document classification), content recommendation services (for example, content classification), email services (for example, email message classification), e-commerce services (for example, user classification), etc.

[04] In particular, the classifier can be trained on a training dataset associated with an object and containing information associated with the object and a control class of the object. The classifier learns which information about the object is more likely to point to the control class of training objects. This classifier is then used to determine the predicted class of the use-phase object based on the information available for that use-phase object.

[05] Incorrect classification of objects can lead to errors in further processing of the object and usually negatively affects the quality of interactive services that require such classification.

Disclosure of invention

[06] The developed embodiments of the present technology are based on the developers' understanding of at least one technical problem associated with known approaches to object classification.

[07] According to one aspect of the present technology, a server capable of executing a plurality of computer algorithms, referred to as a classification engine, is implemented and is generally capable of performing binary classification of digital objects. In the context of the present technology, digital objects may correspond to elements associated with one or more interactive services such as search engines, content recommendation services, e-commerce services, e-mail services, and so on. The nature of the digital object, among other things, depends on the specific implementations of the present technology.

[08] The classification engine comprises a plurality of MLA algorithms capable of predicting the class of a digital object based on the object's associated past events. It can be said that a set of MLA algorithms are predictive models that model a set of metrics and are applied to data representing the past events of an object for a digital object in order to determine the probability that this object belongs to a certain class.

[09] According to at least one aspect of the present technology, the set of metrics used by the classification engine is able to use specific subsets of past events of an object. It should be noted that some kinds of object events can be linked in a special way.

[010] In particular, some kinds of object events may have a nested relationship. We can say that the events of an object of the second kind are nested in the events of an object of the first kind if the events of an object of the second kind can only occur if the events of the first kind of object have occurred. For example, object events (second type events) indicating user "clicks" on search engine results can only occur after the user submits queries (first type events). It can also be said that the events of the second kind object are nested in the events of the first kind object when the events of the second kind object are a subset of the events of the first kind object. For example, object (second kind) events indicating "long clicks" are a subset of object (first kind) events indicating "clicks".

[011] In at least some embodiments of the present technology, the set of metrics used by the classification engine can be said to be "nested" in the sense that the object events used by the first metric of the set of metrics to generate predictions are nested in the object events, used by the second metric from the set of metrics to form its own predictions.

[012] The classification engine comprises a computer algorithm capable of applying a "target combination of metric dependent thresholds" on predictions generated by a plurality of nested metrics to perform a binary classification of a digital object. In a non-limiting example of the present technology, where three nested metrics are used to form three predictions based on respective subsets of an entity's past events indicative of the likelihood that a digital entity belongs to the first class, the classification engine may apply a target combination of three metric-dependent thresholds, each of which corresponds to one nested metric out of three nested metrics. In some embodiments of the invention, if at least one prediction from a plurality of predictions exceeds a corresponding threshold from a target combination of metric-dependent thresholds, then the classification engine may determine that the digital object belongs to the first class. In some embodiments of the invention, if there are no predictions from the set of predictions that exceed the corresponding threshold from the target combination of metric-dependent thresholds, then the classification engine may determine that the digital object belongs to a different class.

[013] According to at least one aspect of the present technology, a server is implemented that is capable of selecting a target combination of metric-dependent thresholds for a plurality of nested metrics from among a plurality of candidate combinations of metric-dependent thresholds. The server may perform an iterative checking process to select a target combination of metric-dependent thresholds from a plurality of candidate combinations of metric-dependent thresholds. In some embodiments of the invention, the server may use an object-specific test data to obtain test data on the classification performance of a plurality of nested metrics with various combinations of metric-dependent thresholds. For example, the classification performance of multiple nested metrics with various combinations of metric-dependent thresholds can be measured in terms of accuracy, recall, error, and the like. In some embodiments, the server may generate a current metric dependent threshold candidate combination (to be used during the current iteration of the iterative validation process) based on a previous metric dependent threshold candidate combination (which was used during a previous iteration of the iterative validation process).

[014] According to a first aspect of the present technology, a method is implemented for determining a target combination of metric-dependent thresholds for use with a plurality of nested metrics when performing a binary classification of a digital object into a first class or a second class. The object is associated with past events of the object, an indication of which is stored in the data store. The method is performed by a server that has access to the data store. The method includes receiving by the server a plurality of object-specific validation datasets. The set of the plurality of object-specific validation datasets contains an indication of a plurality of past object events associated with the corresponding validation object, and a validation object control class representing the first class or the second class. The method includes applying a plurality of nested metrics to a plurality of object-specific test sets by a server and thereby generating a plurality of predictions. The prediction indicates the probability that the test object belongs to the first class or to the second class. The method during the first iteration includes comparing by the server a plurality of predictions with corresponding thresholds from the first combination of metric dependent thresholds to determine the predicted test object classes for the first iteration. The method during the first iteration includes generating by the server first precision parameters and first recall parameters for the plurality of nested metrics for the first iteration by comparing the control classes with the corresponding predicted test object classes for the first iteration. The method during the second iteration includes adjusting the threshold from the first combination of metric dependent thresholds by the server and thereby generating a second combination of metric dependent thresholds. The method during the second iteration includes the server comparing the plurality of predictions with corresponding thresholds from the second combination of metric dependent thresholds to determine the predicted test object classes for the second iteration. The method during the second iteration includes generating by the server second precision parameters and second recall parameters for the plurality of nested metrics for the second iteration by comparing the control classes with the corresponding predicted test object classes for the second iteration. The method includes selecting by the server the first combination of metric-dependent thresholds or the second combination of metric-dependent thresholds as the target combination of metric-dependent thresholds by comparing (a) the first accuracy parameters and the second accuracy parameters with the accuracy threshold and/or (b) the first parameters completeness and second parameters of completeness with the threshold of completeness. A target combination of metric-dependent thresholds is to be used with a set of nested metrics in a use mode to perform a binary classification of a digital object such that if the prediction of the use step of at least one metric from the set of nested metrics for the digital object exceeds the corresponding threshold from the target combination of metric-dependent thresholds , then the belonging of the digital object to the first class is determined.

[015] In some embodiments, the method further includes performing a plurality of iterations by the server until (a) the precision parameters for the iteration are greater than a precision threshold and/or (b) the recall parameters for the iteration are greater than the recall threshold, and selecting by the server a combination of metric-dependent thresholds from this iteration as the target combination of metric-dependent thresholds.

[016] In some embodiments, the method further includes receiving by the server in use mode an object-specific data set containing an indication of a plurality of past object events associated with the object. The method includes applying, in use mode, a plurality of nested metrics to an object-specific set of data, and thereby generating one or more predictions indicating whether the object is likely to belong to the first class or the second class. The method includes a server in use mode comparing one or more predictions with appropriate thresholds from a target combination of metric-dependent thresholds. The method includes determining by the server in use mode whether the object belongs to the first class if at least one prediction exceeds a corresponding threshold from a target combination of metric-dependent thresholds. The method includes determining by the server in use mode whether the object belongs to the second class if there is no prediction that exceeds the corresponding threshold from the target combination of metric-dependent thresholds.

[017] In some embodiments of the method, the first metric of the plurality of nested metrics is based on the events of the first kind of object, and the second metric of the plurality of nested metrics is based on the events of the second kind of object. Events of an object of the second kind occur only if events of an object of the first kind have occurred.

[018] In some embodiments of the method, the second kind object events are a subset of the first kind object events.

[019] In some embodiments of the method, the target combination of metric-dependent thresholds is selected simultaneously for multiple nested metrics during one iteration.

[020] In some embodiments of the method, the object is an email message, with the first class corresponding to spam and the second class corresponding to no spam.

[021] In some embodiments of the method, the object is an e-commerce platform user, with the first class corresponding to fraud and the second class corresponding to no fraud.

[022] In some embodiments of the method, the object is a document, with the first class corresponding to relevance and the second class corresponding to irrelevance.

[023] According to a second aspect of the present technology, a server is implemented for determining a target combination of metric-dependent thresholds to use with a plurality of nested metrics when performing a binary classification of a digital object into a first class or a second class. The object is associated with past events of the object, an indication of which is stored in the data store. The data store is available to the server. The server is capable of receiving a plurality of entity-specific test sets. The set of the plurality of object-specific validation datasets contains an indication of a plurality of past object events associated with the corresponding validation object, and a validation object control class representing the first class or the second class. The server is able to apply multiple nested metrics to multiple object-specific test sets and thus generate multiple predictions. The prediction indicates the probability that the test object belongs to the first class or to the second class. The server during the first iteration is able to compare the plurality of predictions with the corresponding thresholds from the first combination of metric dependent thresholds to determine the predicted test object classes for the first iteration. The server during the first iteration is able to generate the first accuracy parameters and the first recall parameters for the set of nested metrics for the first iteration by comparing the control classes with the corresponding predicted test object classes for the first iteration. The server during the second iteration is able to adjust the threshold from the first combination of metric-dependent thresholds and thus generate a second combination of metric-dependent thresholds. The server during the second iteration is able to compare the plurality of predictions with the corresponding thresholds from the second combination of metric dependent thresholds to determine the predicted test object classes for the second iteration. The server during the second iteration is able to generate second precision parameters and second recall parameters for the plurality of nested metrics for the second iteration by comparing the control classes with the corresponding predicted test object classes for the second iteration. The server is able to select the first combination of metric dependent thresholds or the second combination of metric dependent thresholds as the target combination of metric dependent thresholds by comparing (a) the first precision parameters and the second precision parameters with the precision threshold and/or (b) the first recall parameters and the second fullness parameters with a fullness threshold. A target combination of metric-dependent thresholds is to be used with a set of nested metrics in a use mode to perform a binary classification of a digital object such that if the prediction of the use step of at least one metric from the set of nested metrics for the digital object exceeds the corresponding threshold from the target combination of metric-dependent thresholds , then the belonging of the digital object to the first class is determined.

[024] In some embodiments, the server is further capable of performing multiple iterations until (a) the precision parameters for the iteration are greater than the precision threshold and/or (b) the recall parameters for the iteration are greater than the recall threshold, and select a combination of dependent from the threshold metric from this iteration as the target combination of metric-dependent thresholds.

[025] In some embodiments, the server is further capable of receiving in use mode an object-specific data set containing an indication of a plurality of past object events associated with the object. The server is further capable of applying, in usage mode, a plurality of nested metrics to the object-specific dataset and thereby generating one or more predictions indicating the likelihood of the object being in the first class or the second class. The server is further operable to compare in use one or more predictions with corresponding thresholds from the target combination of metric dependent thresholds. The server is further capable of determining in use whether an object belongs to the first class if at least one prediction exceeds an appropriate threshold from the target combination of metric dependent thresholds. The server is further capable of determining in use whether an object belongs to the second class if there is no prediction that exceeds an appropriate threshold from the target combination of metric-dependent thresholds.

[026] In some embodiments of the server, the first metric of the plurality of nested metrics is based on the events of the first kind of object, and the second metric of the plurality of nested metrics is based on the events of the second kind of object. Events of an object of the second kind occur only if events of an object of the first kind have occurred.

[027] In some embodiments of the server, the second kind object events are a subset of the first kind object events.

[028] In some server embodiments, a target combination of metric-dependent thresholds is selected simultaneously for multiple nested metrics during a single iteration.

[029] In some server embodiments, the object is an email message, with the first class corresponding to spam and the second class corresponding to no spam.

[030] In some embodiments of the server, the object is an e-commerce platform user, with the first class corresponding to fraud and the second class corresponding to no fraud.

[031] In some embodiments of the server, the object is a document, with the first class corresponding to relevance and the second class corresponding to irrelevance.

[032] In the context of the present description, unless expressly indicated otherwise, under the electronic device, server, remote server and computer system refers to any hardware and/or software suitable for solving the task. Thus, some non-limiting examples of hardware and/or software include computers (servers, desktops, laptops, netbooks, etc.), smartphones, tablets, network equipment (routers, switches, gateways, etc.). .) and/or combinations thereof.

[033] As used herein, unless expressly indicated otherwise, "computer-readable media" and "memory" include any type of media, non-limiting examples of which include random access memory (RAM), read only memory (ROM), disks (CD-ROMs, DVDs, floppy disks, hard drives, etc.), USB drives, flash memory cards, solid state drives and tape drives.

[034] In the context of the present description, unless explicitly stated otherwise, the information element itself can be an indication of an information element, as well as a pointer, link, hyperlink, or other indirect means by which the recipient of data can find a place in the network, memory , database, or other machine-readable storage medium from which the information element can be retrieved. For example, a reference to a document may include the document itself (i.e., its contents), or the reference may be a unique document descriptor pointing to a file on a specific file system, or some other means to indicate a location on a network to the recipient of the data. , a memory address, a table in a database, or another location where the file can be accessed. One skilled in the art would appreciate that the degree of precision required for such an indication depends on the amount of prior knowledge regarding the interpretation of the information exchanged between the sender and recipient of the data. For example, if it is known before the communication between the sender and the recipient that the reference to the information element is the database key for the element in a certain table of the predefined database containing this information element, then in order to efficiently transmit this information element to the recipient, it is enough to send the database key data, even if the information element itself is not transmitted between the sender and receiver of the data.

[035] In the context of the present description, unless explicitly stated otherwise, the numerals "first", "second", "third", etc. are used only to indicate the difference between the nouns they refer to, but not to describe any specific relationship between these nouns. For example, it should be clear that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy, or classification of, in this case, servers, and that their use (by itself) does not imply a "second server" in every situation. In addition, as occurs herein in another context, reference to a "first" element and a "second" element does not exclude that the two elements may in fact be the same element. Thus, for example, in some cases the "first" server and the "second" server may be the same software and/or hardware, and in other cases they may be different software and/or hardware.

[036] Each embodiment of the present technology relates to at least one of the above goals and/or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology, associated with an attempt to achieve the above goal, may not meet this goal and/or may meet other goals not explicitly mentioned here. Additional and/or alternative features, aspects and advantages of embodiments of the present technology are contained in the following description, in the accompanying drawings and in the claims.

Brief description of the drawings

[037] The following description is provided for a better understanding of the present technology, as well as other aspects and their features, and should be used in conjunction with the attached drawings.

[038] In FIG. 1 is a diagram of a system in accordance with at least some non-limiting embodiments of the present technology.

[039] In FIG. 2 shows the structure of the data stored in the database of the system shown in FIG. 1 according to non-limiting embodiments of the present technology.

[040] In FIG. 3 is a representation of the object-specific data set received by the server of the system shown in FIG. 1 according to non-limiting embodiments of the present technology.

[041] In FIG. 4 is a representation of the stage of using the classification mechanism hosted on the server of the system shown in FIG. 1 according to non-limiting embodiments of the present technology.

[042] In FIG. 5 shows the scheme of learning the first nested metric and the second nested metric by the server of the system shown in FIG. 1 according to non-limiting embodiments of the present technology.

[043] In FIG. 6 is a representation of an iterative verification process performed by the server of the system shown in FIG. 1 to select a target combination of metric dependent thresholds according to non-limiting embodiments of the present technology.

[044] In FIG. 7 is a flow diagram of a method for determining a target combination of metric-dependent thresholds performed by a server of the system shown in FIG. 1 according to embodiments of the present technology.

Implementation of the invention

[045] The examples and conventions presented herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically given examples and terms. It is obvious that specialists in the art are able to develop various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its essence and scope.

[046] In addition, to facilitate a better understanding, the following description may contain simplified implementations of the present technology. A person skilled in the art should be clear that other embodiments of this technology can be significantly more complex.

[047] In some cases, useful examples of modifications to the present technology are provided. They contribute to understanding, but also do not define the scope or boundaries of the present technology. The presented list of modifications is not exhaustive and the person skilled in the art can develop other modifications within the scope of this technology. In addition, if modifications are not described in some cases, this does not mean that they are impossible and / or that the description contains the only possible implementation of one or another element of the present technology.

[048] The description of principles, aspects, and embodiments of the present technology, as well as their specific examples, are intended to cover their structural and functional equivalents, whether they are currently known or will be developed in the future. For example, those skilled in the art will appreciate that any block diagrams described herein correspond to conceptual representations of illustrative circuit diagrams embodying the fundamentals of the present technology. It should also be obvious that any flowcharts, process diagrams, state transition diagrams, pseudocodes, etc. correspond to various processes that may be represented on a computer-readable physical storage medium and may be executed by a computer or processor, whether such computer or processor is explicitly shown or not.

[049] The functions of the various elements shown in the drawings, including any functional unit labeled "processor" or "graphics processing unit", may be implemented using specialized hardware, as well as using hardware capable of executing the corresponding software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general purpose processor such as a central processing unit (CPU) or a specialized processor such as a graphics processing unit (GPU). In addition, explicit use of the term "processor" or "controller" should not be construed as referring solely to the hardware capable of executing the software and may refer to, but is not limited to, digital signal processor (DSP) hardware, network processor, ASIC (ASIC), Field Programmable Gate Array (FPGA), Software ROM, RAM, and Non-Volatile Memory. Other general purpose and/or custom hardware may also be contemplated.

[050] Software modules, or simply modules that are intended to be implemented in software, may be represented here as any combination of flowchart elements or other elements indicating the steps of a process and/or containing a textual description. Such modules may be implemented in hardware, as shown or implied.

[051] In the following, in light of the foregoing principles, some non-limiting examples are provided illustrating various embodiments of aspects of the present technology.

[052] In FIG. 1 is a diagram of a system 100 (not shown in the drawings) suitable for non-limiting embodiments of the present technology. Obviously, the system 100 is shown only to demonstrate an implementation of the present technology. Thus, the following description of the system is a description of examples illustrating the present technology.

[053] In the illustrated example, system 100 may be used to provide interactive services to a user. To this end, the system 100 includes, among other things, an electronic device 102 associated with the user 101, a server 106, a plurality of resource servers 108, and a database system 150.

[054] In a non-limiting example, system 100 may be used to provide search engine services. In this example, the user 101 may use the electronic device 102 to send a request to the server 106, which in response is able to provide search results to the user 101. The server 106 generates these search results based on information received, for example, from a plurality of resource servers 108 and stored in the system 150 Database. The search results provided by the system 100 may be relevant to the request submitted. We can say that the server 106 may contain a search engine 120.

[055] As described below, in addition to (or instead of) providing search engine services, other interactive services, such as content recommendation services, email services, e-commerce services, and the like, may be provided to the user 101. For example, server 106 may contain one or more of a plurality of interactive services 160, including a search engine 120, an e-commerce platform 130, and an email platform 140.

[056] In the context of the present technology, a system 100 providing one or more interactive services is capable of performing binary classification of digital objects associated with one or more interactive services. The nature of digital objects and the purpose of their classification for various interactive services are described in more detail below.

Electronic device

[057] As mentioned above, system 100 includes an electronic device 102 associated with a user 101. Electronic device 102 or simply device 102 may sometimes be referred to as a client device, terminal device, or client electronic device. It should be noted that the association of the electronic device 102 with the user 101 is not meant to suggest or imply any mode of operation, such as logging in, registering, or the like.

[058] In the context of the present description, unless expressly indicated otherwise, the term "electronic device" or "device" means any computer hardware capable of executing programs suitable for solving the task. Thus, some non-limiting examples of device 102 include personal computers (desktops, laptops, netbooks, etc.), smartphones, tablets, and the like. Device 102 includes hardware and/or application software and/or firmware (or a combination thereof) known in the art for running a browser-based application (not shown).

[059] In general, a browser application provides the user 101 with access to one or more web resources. There are no special restrictions on the implementation of a browser application. For example, a browser application executed by device 102 may be implemented as a Yandex™ browser. For example, the user 101 may use a browser application to (a) navigate to a search engine website and (b) submit a request to be provided with relevant search results. In another example, the user 101 may use a browser application to (a) navigate to an e-commerce website and (b) buy and/or sell an item or service. In yet another example, the user 101 may use a browser application to (a) navigate to an email website and (b) access their email subscriber account to view email messages associated with their account.

[060] The device 102 is capable of generating a request 180 to communicate with the server 106. The request 180 may be one or more data packets containing information, for example, indicative of a request sent by the user 101. The device 102 is also capable of receiving a response 190 from the server 106. The response 190 may be one or more data packets containing information, for example, indicative of search results relevant to the request submitted, and machine-readable instructions for the browser application to display those search results to user 101.

Communication network

[061] System 100 includes communication network 110. In a non-limiting example, the communication network 110 may be the Internet. In other non-limiting examples, communications network 110 may be implemented in other ways, such as any wide area network, local area network, private network, and the like. In practice, the implementation of the communication network 110, which is not subject to any restrictions, may, among other things, depend on the implementation of other elements of the system 100.

[062] The communication network 110 is designed to provide communication between at least some elements of the system 100, such as the device 102, a plurality of resource servers 108, and a server 106. For example, this means that a plurality of resource servers 108 is available through the communication network 110 to the device 102. In another example, this means that a plurality of resource servers 108 are available via communication network 110 to server 106. In yet another example, this means that server 106 is accessible via communication network 110 to device 102.

[063] Communication network 110 may be used to transfer data packets between device 102, a plurality of resource servers 108, and server 106. For example, communication network 110 may be used to send request 180 from device 102 to server 106. In another example, communication network 110 may be used to send a response 190 from the server 106 to the device 102.

Many resource servers

[064] As described above, a plurality of resource servers 108 can be accessed via communication network 110. A plurality of resource servers 108 may be implemented as conventional computer servers. In a non-limiting embodiment of the present technology, a server of multiple resource servers 108 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. The server of the plurality of resource servers 108 may also be implemented using any other suitable hardware and/or application software and/or firmware, or a combination thereof.

[065] The plurality of resource servers 108 contains resources (or web resources) available to the device 102 and/or server 106. There are no restrictions on the kind of resources contained in the plurality of resource servers 108. In some embodiments of the present technology, the resources may contain digital documents or simply documents that are web pages.

[066] For example, a plurality of resource servers 108 may contain web pages, ie. a plurality of resource servers 108 may store documents that are web pages and are accessible by device 102 and/or server 106. A document may be written in a markup language and may, among other things, contain (a) the content of the corresponding web page, and (b) machine-readable instructions for displaying the corresponding web page (the content it contains).

[067] The device 102 may contact a server of the plurality of resource servers 108 to obtain a document stored on that server. For example, user 101 may enter a web address associated with a web page into a browser application of device 102. In response, device 102 may contact a resource server containing the web page to obtain a document representing the web page to displaying the content of this web page using a browser application.

[068] Server 106 may contact a server of a plurality of resource servers 108 to obtain a document stored on that resource server. The purpose of a server 106 accessing and receiving documents from a plurality of resource servers 108 is described in more detail below.

Database system

[069] The server 106 is connected to the database system 150. In general, database system 150 is capable of receiving data from server 106, storing that data, and/or providing it to server 106 for later use.

[070] In some embodiments, database system 150 may store information associated with one or more interactive services hosted on server 106. For example, if server 106 hosts a search engine 120, then database system 150 may store information about previously searches performed by the search engine 120, as well as information about queries previously sent to the server 106 and about documents provided by the search engine of the server 106 as search results.

[071] In this example, it is assumed that the database system 150 can store query data associated with the corresponding queries sent to the search engine 120. The query data associated with a query can be of various kinds and is not subject to any restrictions. For example, database system 150 may store, for appropriate queries, query data such as (but not limited to):

- query popularity;

- frequency of sending a request;

- the number of "clicks" associated with the request;

- indications of other sent requests related to the request;

- indications of documents related to the request;

- other statistics related to the request;

- search terms associated with the query;

- the number of characters in the request;

- other characteristics inherent in the request.

[072] In this example, the database system 150 may also store document data associated with respective documents. The document data associated with a document can be of various types and is not subject to any restrictions. For example, database system 150 may store, for relevant documents, document data such as (but not limited to):

- the popularity of the document;

- coefficient of "clicks" for the document;

- time per "click" associated with the document;

- indications of requests related to the document;

- other statistical data related to the document;

- text associated with the document;

- document file size;

- other characteristics inherent in the document.

[073] In this example, the database system 150 may also store user data associated with respective users. The user data associated with the user can be of various types and is not subject to any restrictions. For example, the database system 150 may store, for respective users, the following user data (but not limited to):

- web session data;

- data of the sent request;

- history of "clicks";

- interaction data;

- user preferences.

[074] In at least some embodiments of the present technology, it is contemplated that the database system 150 may store data associated with an interactive service item or object. It can be said that the database system 150 can store object-specific data. It is contemplated that the server 106 may store data about the various interactive service objects for each object separately without departing from the present technology.

[075] For example, if the server 106 hosts the search engine 120, then the database system 150 may store data associated with its users (the first kind of digital objects or elements associated with search engine services). Thus, in this example, the database system 150 may store user-specific data for each user separately. In another example, if server 106 hosts search engine 120, then database system 150 may store data associated with digital documents that were used as search results (the second kind of digital objects or elements associated with search engine services). Thus, in this example, the database system 150 may store document-specific data for each document separately.

[076] In yet another example, if the server 106 hosts the email platform 140, then the database system 150 can store data associated with its users (a first kind of digital objects or items associated with an email service). Thus, in this example, the database system 150 may store user-specific data for each user separately. In another example, if server 106 hosts email platform 140, then database system 150 may store data associated with email messages (the second kind of digital objects or items associated with an email service). Thus, in this example, the database system 150 may store email-specific data for each email message separately.

[077] Therefore, it can be said that the database system 150 can store various object-specific data depending, among other things, on the type of interactive service (or services) hosted on the server 106, and on the types of objects associated with this interactive service. (or services).

[078] As described below, server 106 is capable of executing a variety of computer algorithms, referred to as classification engine 170, that is generally capable of performing binary classification of digital objects from one or more interactive services provided by server 106.

[079] In at least some embodiments of the present technology, the database system 150 may store labeled object-specific data to test the performance of one or more of the classification algorithms described herein. For example, the markup object-specific data for a validation object may contain markup data pointing to the validation object's control class. There are no particular restrictions on the collection and/or generation of markup data and their subsequent storage in the database system 150 . In some cases, markup data may be collected from human evaluators who have been given the task of "marking" test items.

[080] Referring to FIG. 2 describes in more detail the information that may be included in the object-specific data and the marked-up object-specific data stored in the database system 150.

Server

[081] System 100 includes a server 106, which may be implemented as a conventional computer server. In an exemplary embodiment of the present technology, server 106 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. Obviously, the server 106 may be implemented using any other suitable hardware and/or application software and/or firmware, or a combination thereof. In the present non-limiting embodiment of the present technology, the server 106 is a single server. In other non-limiting embodiments of the present technology, the functions of the server 106 may be distributed among multiple servers.

[082] As shown in FIG. 1, server 106 may contain a plurality of 160 interactive services. For example, server 106 may include a search engine 120 for providing search engine services, an e-commerce platform 130 for providing e-commerce services, and an email platform 140 for providing e-mail services. Described below is a possible implementation of a search engine 120, an e-commerce platform 130, and an email platform 140 in at least some embodiments of the present technology.

[083] In some embodiments, the server 106 may be managed and/or administered by a search engine service provider (not shown), such as the Yandex™ search engine operator. The server 106 may include a search engine 120 to perform one or more searches in response to queries submitted by users of the search engine 120.

[084] For example, server 106 may receive request 180 from device 102 indicating a request submitted by user 101. Server 106 may perform a search in response to the submitted request to generate search results relevant to the submitted request. As a result, the server 106 may generate a response 190 indicating the search results and may send a response 190 to the device 102 to display the search results to the user 101, such as using a browser application.

[085] The search results generated for the submitted request can be presented in any form. However, in a non-limiting example of the present technology, the search results generated by the server 106 may point to documents relevant to the submitted request. Determination and receipt by the server 106 of documents relevant to the submitted request is described below.

[086] Server 106 may also run a crawler application (not shown). In general, the crawler application may be used by the server 106 to "visit" resources available through the communications network 110 and obtain or download them for later use. For example, a crawler application may be used by server 106 to access multiple resource servers 108 and to retrieve or download documents that are web pages contained on multiple resource servers 108 .

[087] It is contemplated that the crawler application may be periodically executed by the server 106 to retrieve or download documents that have been updated and/or made available via the communication network 110 since the previous execution of the crawler application.

[088] In other embodiments, the server 106 may be managed and/or administered by an e-commerce service provider (not shown), such as, for example, the Yandex.Market™ e-commerce platform operator. Thus, the server 106 may include an e-commerce platform 130 for offering one or more goods and/or services for purchase or sale by users of the e-commerce platform 130.

[089] In general, the e-commerce platform corresponds to one or more computer algorithms by which the server 106 is able to provide e-commerce services to the user 101 of the electronic device 102. For example, the user 101 may be a client of the e-commerce platform 130. The user 101 can enter a Universal Resource Locator (URL) associated with the e-commerce platform 130 in the browser application command interface and access their account on the e-commerce platform 130.

[090] It should be noted that the server 106 may collect information about customers and products available on the e-commerce platform 130. For example, server 106 may collect client-specific information regarding client interactions with various products. In this example, server 106 may collect information about products viewed, products selected, products purchased, products recommended, and the like, from a client. In another example, server 106 may collect product-specific information regarding various products. In this example, the server 106 may collect views, clicks, purchases, and the like for a product.

[091] In other embodiments of the invention, the server 106 may be managed and/or administered by an email service provider (not shown), such as the operator of the Yandex.Mail™ email service. Accordingly, the server 106 may include an email platform 140 for providing email services to users of the email platform 140 .

[092] In general, the email platform 140 corresponds to one or more computer algorithms by which the server 106 is able to provide email services to the user 101 of the electronic device 102. For example, the user 101 may have an email subscriber account associated with the email platform 140 . The user 101 can enter a URL associated with the email platform 140 in a browser application command interface and access their email subscriber account on the email platform 140 .

[093] In some embodiments of the present technology, in addition to or instead of the foregoing, electronic device 102 may run an email client application (not shown) associated with (server) platform 140. In general, the email client application is designed to provide a user with 101 ability to view a list of emails (read and unread), read emails, open attachments, compose new emails, reply to emails, forward emails, delete emails, manage unwanted emails, assign categories of e-mail messages, arrange e-mail messages in folders, create and access an address book, and so on.

[094] Regardless of whether the user 101 uses a browser application and/or an email client application to access their email subscriber account, it is contemplated that the user 101 may be provided with an email interface (not shown) to perform one or more actions regarding email messages in your email subscriber account. The functions of the email platform 140 are described in more detail below.

[095] In general, the email interface is designed to provide user interactions between a user of email platform 140 (such as user 101, for example) and email messages in their email subscriber account. In a non-limiting example, an email interface may include one or more panels, one or more menus, one or more buttons, and may provide other functionality for user interaction with email messages. It should be noted that various e-mail interfaces are possible in the context of the present technology.

[096] For example, the email interface may include a sidebar pointing to one or more email folders (predefined and/or personalized) associated with an email account, such as an inbox, an outbox, a folder drafts, spam folder, deleted items folder, etc. In another example, the email interface may include one or more buttons for performing various actions on email messages, such as a "compose" button for composing a new email message, a "send" button for sending an email message, a button "save" to save the current version of the email message, "read" button to indicate that the email message has been read or viewed by the user, "unread" button to indicate that the email message has not been read or viewed by the user, " spam" button to indicate that the email message should be categorized as "spam" and/or to indicate that the email message should be moved to the spam folder, a "deleted" button to indicate that the email message should be deleted and/or that the e-mail should be moved to the "deleted" folder, etc. In yet another example, the email interface may support user interaction with other types of email messages, such as a drag and drop functionality that allows the user to efficiently select an email message in a first folder and move that email message to a second folder.

[097] In the context of the present technology, the server 106 may contain a classification engine 170. In general, the classification engine 170 is capable of using data stored in conjunction with an interactive service digital object and performing a binary classification of that digital object. For example, server 106 may obtain object-specific data for a digital object from database system 150 and use classification engine 170 to determine whether the digital object belongs to a first class or a second class.

[098] As described in more detail below, the classification engine 170 comprises (a) a plurality of nested metrics for generating predictions of the probability that a digital object will belong to the first class, and (b) a combination of metric-dependent thresholds against which the respective predictions are compared to determine membership. digital object to the first class. The server 106 is also capable of determining a target combination of metric-dependent thresholds from a plurality of candidate metric-dependent threshold combinations to be used in conjunction with a plurality of nested metrics.

[099] FIG. 2 is a view 200 for at least some of the data stored in the database system 150. As shown, the database system 150 is capable of storing object-specific data 210 and object-specific verification data 220.

[0100] For illustration purposes, the objects in this example may be assumed to represent users of the search engine 120. However, the types of objects may depend on various implementations of the present technology, among other things.

[0101] Object-specific data 210 is stored in the database system 150 on a per-object basis or, as shown in a non-limiting example, on a per-user basis. For example, the object-specific data 210 contains a plurality of object-specific data sets (not numbered). The set of object-specific datasets comprises an object-specific dataset 215, which in the non-limiting example shown where the objects correspond to users of the search engine 120, is also referred to as a user-specific dataset.

[0102] It should be noted that the object-specific data set 215 contains information for uniquely identifying the corresponding object. For example, the object-specific dataset 215 may be associated with an object identifier (in this example, a user ID) that uniquely identifies a particular object of the object-specific dataset 215. The object-specific data set 215 also contains information about the object collected by the corresponding interactive service. In the illustrated example, object-specific data set 215 may contain user interaction data about a particular user collected by search engine 120. Referring to FIG. 3 describes in more detail the information that may be stored as part of the object-specific dataset 215.

[0103] As described above, in addition to the object-specific data 210, the database system 150 is also capable of storing object-specific verification data 220. For illustration, it can be assumed that the verification objects in this example are users of the search engine 120. Obviously, like the appearance of the objects in the object-specific data 210, the appearance of the check objects may depend on various implementations of the present technology, among other things. However, it is contemplated that the appearance of the objects in the object dependent data 210 may be the same as the appearance of the objects in the object dependent data 220.

[0104] Object-specific data 210 is stored in the database system 150 on a per-object basis or, as shown in a non-limiting example, on a per-user basis. For example, object-specific verification data 220 contains a plurality of object-specific verification datasets (not numbered). The set of object-specific validation datasets comprises an object-specific validation dataset 225, which in the non-limiting example shown where the validation objects correspond to users of the search engine 120, is also referred to as a user-specific validation dataset.

[0105] It should be noted that the object-specific validation data set 225 contains information for uniquely identifying the corresponding object. For example, the object-specific validation dataset 225 may be associated with an object identifier (in this example, a user ID) that uniquely identifies a particular validation object of the object-specific validation dataset 225. The object-specific validation data set 225 also contains information about the validation object collected by the corresponding interactive service. In the example shown, the object-specific validation data set 225 may contain user interaction data about a particular user collected by the search engine 120.

[0106] However, unlike the object-specific dataset of the object-specific data 210, the object-specific validation dataset 225 additionally contains an indication of the validation object's control class. As described above, the database system 150 may store markup data indicative of check object check classes. In the non-limiting example shown, the entity-specific validation dataset 225 (user-specific validation dataset) may contain information about the validation object (verifier user) control class, such as whether the corresponding search engine validation user is or is not a fraudster. 120.

[0107] Obviously, not only the kind of objects and check objects depends, among other things, on various implementations of the present technology, but the kinds of binary classes also depend, among other things, on various implementations of the present technology. For example, if the check object is a digital document of the search engine 120, then the check class may indicate whether the digital document is relevant or not. In another example, if the check object is a product of the e-commerce platform 130, then the check class may indicate whether the product is counterfeit or not. In yet another example, if the validation object is an e-mail message of the e-mail platform 140, then the control class may indicate whether the e-mail message is spam or not.

[0108] As described below, the server 106 is able to use the object-specific data 210 in the use step of the classification engine 170, and the object-specific test data 220 is used in the learning check step of the classification engine 170 to determine a candidate combination of metric-dependent thresholds to be selected. as the target combination of metric-dependent thresholds. It can be said that in some embodiments of the present technology, the server 106 can (a) perform a training step of a set of MLA algorithms (multiple nested metrics) by using object-specific test data 220, (b) perform a test step to select a target combination of metric-dependent thresholds by applying the object dependent test data 220 and (c) performing the step of using the classification engine 170 comprising the thus trained set of MLA algorithms and the thus selected target combination of metric dependent thresholds to classify one or more objects from the object dependent data 210.

[0109] The following describes in more detail the performance by the server 106 of one iteration of the binary classification (one iteration of the step of using the classification engine 170) of an object from object-specific data 210.

[0110] In FIG. 3 (left) is a representation 300 of the information retrieval operation performed by the server 106 on an object to be classified as either a first class or a second class. The server 106 is capable of accessing the database system 150 and obtaining information associated with the digital object to be classified. For example, the server 106 may send a request 302 to the database system 150 containing an indication of an object identifier. In response, the server 106 may receive an object-specific data set 310 .

[0111] Dependent on the object data set 310 contains information about the corresponding object to be classified. In particular, the object-specific data set 310 contains an indication of a set of past events 312 of the object associated with the corresponding object to be classified. As described above, object-specific data, such as object events, may be collected by the server 106 and stored in the database system 150. For example, in a non-limiting example where the object is a search engine user 120 (e.g., user 101), the object's past events 312 may include submit request events, view events, "click" events, "long click" events, etc. .d. Thus, it can be said that an object's past events associated with an object may include object-dependent events of various kinds.

[0112] Obviously, the types of past events of an object that can be stored in combination with the object, among other things, depend on the type of interactive service to which the object belongs, on the type of object, and on specific implementations of the present technology. However, it should be noted that certain kinds of object events from past object events 312 may be related in a special way. In particular, in at least some embodiments of the present technology, it is contemplated that certain kinds of object events may have nested relationships with each other.

[0113] To better illustrate this, it can be assumed that past object events 312 for a user of search engine 120 comprise first kind and second kind object events and that first kind object events represent query submissions and second kind object events correspond to search engine result selection events. ("clicks"). In other words, some object events from object past events 312 indicate that the user (which is the object in this example) has submitted one or more queries, and other object events from object past events 312 indicate that the user has selected the results. search engine provided in response to these submission requests. In this example, we can say that the events of an object of the second kind are, in a certain sense, nested in the events of an object of the first kind, since the events of an object of the second kind can only occur if the events of the first kind of object have occurred, i.e. object events indicating the selection of search engine results can only occur after the user submits appropriate queries.

[0114] You can also say that the events of the second kind of object are nested in the events of the first kind of object, when the events of the second kind of object are a subset of the events of the first kind of object. To better illustrate this, the following description is given with reference to a representation 350 of a set of object events 360 from past object events 312 . It can be assumed that the object event set 360 contains "clicks" on search engine results. It can also be assumed that some of the user's "clicks" on search engine results are "long clicks" on search engine results. For example, some user-selected search engine results may have been viewed for a significant period of time (“long clicks”).

[0115] For example, object event set 360 contains object events 351, 352, 353, 354, 355, 356, 357, 358 indicative of a user performing "clicks" on search engine results. In the same example, it can be assumed that object events 356, 357, 358 indicate a user performing "long clicks" on search engine results. In this example, the object event set 360 (containing "clicks" on search engine results) can be said to contain a subset 370 of object events containing "long clicks" on search engine results. In this example, we can say that the events of the second kind of object ("long clicks") are, in a sense, nested in the events of the first kind of object ("clicks"), since the events of the second kind of object are a subset of the events of the first kind of object.

[0116] Thus, the server 106 may perform a lookup operation to obtain an object-specific data set 310 for an object containing past events 312 of the object. Past object events 312 contain object events of various kinds, with some object events having a "nesting" relationship with each other that depends on the respective object event types.

[0117] The server 106 is able to use the object-specific data set 310 for the object in order to perform its binary classification. In FIG. 4 is a representation 400 of a binary classification procedure performed by the classification engine 170 of the server 106. As shown, the server 106 is capable of inputting an object-specific dataset 310 into the classification engine 170.

[0118] In general, the classification engine 170 comprises a plurality of nested metrics 410 capable of (a) receiving as input at least some of the data from the object-specific data set 310 and (b) producing a plurality of 440 predictions for the object-dependent digital object set 310 data.

[0119] As described in more detail below with reference to FIG. 5, a corresponding metric of nested metrics set 410 may be implemented by server 106 as an MLA algorithm capable of deriving a subset of data from object-specific dataset 310, based on which this nested metrics set 410 is able to generate a prediction indicative of the probability of belonging to to the first class (or to the second class) of the digital object of the object-specific data set 310 .

[0120] For example, the first nested metric 412 may use a subset 422 of the object-specific data set 310 to form a first prediction 432 for a digital object indicating the probability that the digital object belongs to the first class. In the same example, the second nested metric 414 may use a subset 424 of the object-specific data set 310 to generate a second prediction 434 for the digital object indicating the probability that the digital object belongs to the first class. Below with reference to FIG. 5 describes in more detail the training of the nested metrics set 410 to generate predictions and the data from the object-specific dataset 310 included in subset 422 and subset 424.

[0121] As described above, the classification engine 170 is able to compare a plurality of predictions 440 with a target metric-dependent threshold combination 450 to determine a predictive class 460 (whether first class or second class) for the corresponding digital object. For example, the target metric dependent threshold combination 450 comprises a first target metric dependent threshold 452 (depending on the first nested metric 412) and a second target metric dependent threshold 454 (depending on the second nested metric 414).

[0122] It should be noted that if at least one prediction from the prediction set 440 exceeds the corresponding threshold from the target metric-dependent threshold combination 450, then the server 106 may determine that the corresponding object belongs to the first class. In other words, in this case, the server 106 determines that the predicted class 460 for the digital object corresponds to the first class. If there is no prediction from the prediction set 440 that exceeds the corresponding threshold from the target metric-dependent threshold combination 450, then the server 106 may determine that the corresponding object belongs to the second class. In other words, in this case, the server 106 determines that the predicted class 460 for the digital object corresponds to the second class.

[0123] In at least some embodiments of the present technology, it can be said that the classification engine 170 may comprise one or more computer algorithms, hereinafter referred to as threshold logic, and is capable of performing one or more logic operations based on a target combination 450 of metric-dependent thresholds and set 440 predictions to determine the predicted class 460 of the digital object.

[0124] The developers of the present technology have found that the use of one combination of metric-dependent thresholds instead of another combination of metric-dependent thresholds as part of the threshold logic of the classification engine 170 can affect the classification efficiency of the classification engine 170. For example, the classification performance of the classification engine 170 can be measured using a variety of parameters, including error, accuracy, recall, and so on. In at least some embodiments of the present technology, the developers of the present technology have developed methods and servers for selecting from a plurality of candidate combinations of metric-dependent thresholds such a target combination of 450 metric-dependent thresholds such that each metric of the set of 410 nested metrics has (a) an accuracy value , exceeding the minimum predetermined precision value, and/or (b) the recall value exceeding the minimum predetermined recall value.

[0125] Referring to FIG. 6 describes in more detail the determination by the server 106 of the target combination 450 of the metric dependent thresholds. In doing so, learning a set of nested metrics 410 to predict based on an object-specific data set 310 is first described.

[0126] As described above, the server 106 is capable of implementing a plurality of nested metrics 410 as a plurality of MLA algorithms. In general, an MLA algorithm is first "built" (or trained) using training data and training targets. During the training iteration, the MLA algorithm receives training input data and generates an appropriate prediction. The server 106 is then, in a sense, able to update the MLA algorithm based on a comparison of the prediction with the corresponding training target for the training inputs. For example, the adjustment may be performed by server 106 using one or more machine learning techniques such as backpropagation and the like. After a large number of training iterations, the MLA algorithm is modified to generate predictions based on the input data that are close to the corresponding training goals.

[0127] Briefly, the implementation by the server 106 of a metric from the set of nested metrics 410 can be divided into two main stages: a training stage and a use stage. First, the MLA algorithm is trained in the training phase. Then, the MLA algorithm built on the training data is applied by the classification engine 170 using the use phase data in the use phase (in this example, the use phase of the MLA algorithm is performed during the use phase of the classification engine 170).

[0128] In FIG. 5 shows a representation 500 of one training iteration of the first MLA algorithm (first nested metric 412) and a representation 550 of one training iteration of the second MLA algorithm (second nested metric 414). One iteration of training the first MLA algorithm and the second MLA algorithm is described below.

[0129] The server 106 is configured to generate a training set 502 for one training iteration of the first nested metric 412. The server 106 is configured to generate a training set 502 based on the object-specific validation data set stored in the database system 150. In the presented non-limiting example, it can be assumed that the server 106 generates the training set 502 based on the object-specific test set 225 data.

[0130] The training set 502 contains a target 504 pointing to the test object's control class. The training set 502 also contains object-specific inputs 506 representing at least some of the object events associated with the object-specific test set 225.

[0131] It should be noted that the type of object events associated with the object-specific validation dataset 225 to be used as the object-specific input 506 may be predetermined by the operator of the server 106. For example, the operator may specify that the first nested metric 412 should predict the probability of an object belonging to the first class based on the events of an object of some kind. As a result, the server 106 can determine a subset of events of that kind of object from the object-specific validation data set 225 .

[0132] It can be assumed that object events of some kind to be used as object-specific input 506 are "clicks". Thus, the server 106 can determine from the object-specific validation data set 225 a subset of object events that are "clicks" and use this subset as the object-specific input 506. After receiving the object-specific input 506, the first MLA algorithm ( the trainable first nested metric 412) is capable of generating a prediction 508. The prediction 508 indicates the probability that the test object belongs to the first class and is based on object events of some kind associated with the first MLA algorithm (e.g., object events representing "clicks"). The server 106 is then able to generate a comparison result 510 indicating the difference between the target 504 and the prediction 508.

[0133] For example, if the check object's control class is the first class, then target 504 may correspond to "1". In the same example, if the check object's control class is the second class, then target 504 may correspond to "0". In this example, prediction 508 may be equal to a value between 0 and 1. Thus, it can be said that comparison result 510 may indicate similarity or dissimilarity between prediction 508 and target 504. Server 106 uses comparison result 510 to train or adjust the first MLA algorithm so so that the first MLA algorithm generates predictions that are as close as possible to the corresponding goals.

[0134] After a large number of training iterations performed like one training iteration of the first MLA algorithm shown in FIG. 5, the first nested metric 412 is capable of using the appropriate kind of object events associated with the first nested metric 412 (eg, "clicks") to generate predictions indicative of the likelihood that the corresponding objects belong to the first class.

[0135] In at least some embodiments of the present technology, a nested metric can use input data containing object events of several appropriate kinds without departing from the boundaries of the present technology.

[0136] The server 106 is configured to generate a training set 552 for one training iteration of the second nested metric 414. The server 106 is configured to generate a training set 552 based on the object-specific validation data set stored in the database system 150. In the presented non-limiting example, it can be assumed that the server 106 generates the training set 552 based on the object-specific test set 225 data. However, as described below, the training sets for training the first nested metric 412 and the second nested metric 414 may or may not be determined based on the same object-specific test datasets.

[0137] Training set 552 contains a target 554 pointing to the test object's control class. Training set 552 also contains object-specific input 556 representing at least some of the object events associated with the object-specific test set 225.

[0138] It should be noted that the type of object events associated with the object-specific validation data set 225 to be used as the object-specific input 556 may be predetermined by the operator of the server 106. For example, the operator may specify that the second nested metric 414 should predict the probability of an object belonging to the first class based on the events of an object of a different kind. As a result, the server 106 can determine a subset of other kind of object events from the object-specific validation data set 225 .

[0139] However, it is important to note that some kind of object events to be used to train the first nested metric 412 are specifically related to other kind of object events to be used to train the second nested metric 414, i.e. nested metrics set 410 uses object nested events. For example, if some kind of object events used by the first nested metric 412 are "clicks", then other kind of object events to be used by the second nested metric 414 are nested in "clicks" kind object events. Thus, metrics from a set of nested metrics can be called nested, since they are based on the types of object events, between which there are relationships of the “nesting” type.

[0140] Thus, it can be assumed that other kind of object events used as object dependent input 556 are "long clicks". Therefore, the server 106 can determine from the object-specific test set 225 a different subset of object events that are "long clicks" and use this other subset as the object-specific input 556. After receiving the object-specific input 556, the second algorithm The MLA (learnable second nested metric 414) is capable of generating a prediction 558. The prediction 558 indicates the probability that the test object belongs to the first class and is based on other kind of object events associated with the second MLA algorithm (for example, object events representing "long clicks" ). The server 106 is then able to generate a comparison result 560 indicating the difference between the target 554 and the prediction 558.

[0141] We can say that the comparison result 560 may indicate the similarity or dissimilarity of the forecast 558 and the goal 554. The server 106 uses the comparison result 560 to train or adjust the second MLA algorithm so that the second MLA algorithm generates predictions that are as close as possible to the respective goals.

[0142] After a large number of training iterations performed like one training iteration of the second MLA algorithm shown in FIG. 5, the second nested metric 414 is capable of using the appropriate kind of object events associated with the second nested metric 414, such as "long clicks", to generate predictions indicating the likelihood that the corresponding objects belong to the first class.

[0143] As described above, the training sets for training the first nested metric 412 and the second nested metric 414 may or may not be determined based on the same object-specific test datasets. In the shown in FIG. 5, as a non-limiting example, it can be assumed that the object-specific validation data set 224 for the validation object contains "click" events and that among these "click" events there are "long click" events. However, not every check object may have events corresponding to nested events of nested metrics set 410 .

[0144] In some embodiments of the present technology, the first metric and the second metric of a plurality of nested metrics can be trained independently of each other. In some embodiments, the first metric and the second metric of a plurality of nested metrics may be trained separately by server 106.

[0145] In one example, the object-specific validation data set 225 may contain "click" events, but not contain "long click" events. In this example, the object-specific test set 225 can be used by the server 106 to train the first nested metric 412 because it contains the appropriate kind of object events associated with the first nested metric 412 (eg, "click" kind). However, the server 106 cannot use the object-specific test set 225 to train the second nested metric 414 because it does not contain object events of the appropriate kind associated with the second nested metric 414 (eg, long click). In such an example, the server 106 may use a different object-specific test set to train the second nested metric 414.

[0146] However, it should be noted that if the server 106 has determined that the object-specific test set contains object events associated with the second nested metric 414, then the server 106 may also determine that the object-specific test set contains object events associated with the first nested metric 412, since there is a "nesting" relationship between the object events of the species used by the corresponding metrics from the nested metrics set 410 .

[0147] As described above, the server 106 is able to select a target metric dependent threshold combination 450 from a plurality of candidate metric dependent threshold combinations. In FIG. 6 is a representation 600 of an iterative check process performed by the server 106 to select a target combination 450 of metric dependent thresholds.

[0148] It is contemplated that the server 106 may receive object-specific verification data 220 containing a plurality of object-specific verification datasets to perform an iterative verification process.

[0149] The server 106 may generate the first prediction data 610 and the second prediction data 620 based on the object-specific test data 220 and the set 410 of nested metrics. For example, as described above, the server 106 may input at least a subset of the object-specific validation dataset 225 into the first nested metric 412 and at least another subset of the object-specific validation dataset 225 into the second nested metric 414 to generate the first prediction 612 and second prediction 614, respectively, for the test object (and its associated events) associated with the object-specific test dataset 225.

[0150] The server 106 may similarly use (a) the first nested metric 412 to generate a set of first predictions 622 based on a set of object-specific test sets, and (b) a second nested metric 414 to generate a set of 624 second predictions based on a set of object-dependent validation dataset object. It is assumed that only some of the metrics from the set of 410 nested metrics can be used for the test object. For example, server 106 may use those metrics from nested metrics set 410 for which the corresponding object-specific test dataset contains a non-zero subset of objects of the respective kinds.

[0151] We can say that the server 106 can apply a set of nested metrics 410 in relation to a set of object-specific test datasets and thus generate a set 625 predictions (comprising first predictions 622 and second predictions 624) indicating the probability of belonging to the test object (and related events) to the first class or to the second class.

[0152] In FIG. 6 also shows a view 602 of the first iteration of the iterative verification process and a view 604 of the second iteration of the iterative verification process.

[0153] During the first iteration, the server 106 may use the first metric-dependent threshold candidate combination 650 to compare against the set 625 of predictions. In particular, during the first iteration, the server 106 may use the first metric-dependent threshold 630 of the first metric-dependent threshold candidate 650 to compare against the first predictions 622 and the second metric-dependent threshold 640 of the first metric-dependent threshold candidate 650 for comparison with second forecasts 624.

[0154] In some embodiments, it is contemplated that the first candidate metric dependent threshold combination 650 may be initialized by server 106. For example, server 106 may use a predefined initial candidate metric dependent threshold combination to perform the first iteration. As described below, a predetermined initial candidate metric dependent threshold combination may be used by the server to determine a plurality of candidate metric dependent threshold combinations by adjusting one or more metric dependent thresholds in that candidate combination to generate a new candidate combination.

[0155] For example, the server 106 may compare the first prediction 612 for the test object associated with the object-specific test set 225 with the first metric-dependent threshold 630 and determine the predicted class 672 as a result. For example, if the first prediction 612 is greater than the first dependent from the threshold metric 630, then the server 106 may determine that the predicted class 672 indicates the first class. Otherwise, if the first prediction 612 is less than the first metric-dependent threshold 630, then the server 106 may determine that the predicted class 672 points to the second class. The server 106 may similarly compare each prediction from the first prediction set 622 to form a first prediction class set 674 . It can be said that the first prediction class data 670 is generated based on the comparison of the first prediction data 610 with the first metric dependent threshold 630.

[0156] In the same example, server 106 may compare a second prediction 614 for the test object associated with the object-specific test set 225 against a second metric-dependent threshold 640 and determine a predicted class 682 as a result. Server 106 may similarly compare each a prediction from the second prediction set 624 to form a second prediction class set 684 . It can be said that the second class prediction data 680 is generated based on the comparison of the second prediction data 620 with the second metric dependent threshold 640.

[0157] The server 106 may compare the first predictive class data 670 and the second predictive class data 680 with the markup data 690 from the object-specific check data 220.

[0158] On the one hand, markup data 690 contains indications of the control classes of the check objects. On the other hand, predictive classes from the first predictive class set 674 (predicted using the first nested metric 412) and from the second predictive class set 684 are associated with the corresponding test object.

[0159] Thus, the server 106 may compare the first predictive class data 670 and the second predictive class data 680 with the markup data 690 to evaluate the classification performance of the first nested metric 412 and the second nested metric 414 used in conjunction with the first metric dependent threshold 630 and a second metric dependent threshold 640 (first candidate combination 650 metric dependent thresholds). For example, for the first iteration, the server 106 may determine a first precision parameter 692 and a first recall parameter 694 for the first nested metric 412, as well as a different first precision parameter 696 and a different first recall parameter 698 for the second nested metric 414.

[0160] In general, the accuracy parameter (also called positive predictive value) is the proportion of relevant instances among the received instances, and the recall parameter (also called sensitivity) is the proportion of actually received instances out of the total number of relevant instances. In other words, the completeness parameter indicates how many relevant instances were received by the metric, and the accuracy parameter indicates how many received instances were actually correctly classified.

[0161] Thus, it can be said that the server 106 during the first iteration can generate the first precision parameters (the first precision parameter 692 and the other first precision parameter 696) and the first completeness parameters (the first completeness parameter 694 and the other first completeness parameter 698) for nested metric sets 410 used in conjunction with the first metric-dependent threshold combination 650 by comparing control classes from the object-specific test data 220 with the corresponding predicted test object classes.

[0162] It should be noted that the server 106 may store an indication of the first metric-dependent threshold candidate combination 650 in combination with the first precision parameter 692 and the first recall parameter 694 for the first nested metric 412, and also in combination with another first precision parameter 696 and another first completeness parameter 698 for the second nested metric 414.

[0163] During the second iteration, the server 106 may use a second candidate metric dependent threshold combination 660 to compare against the set 625 of predictions.

[0164] As described above, it is contemplated that the server 106 can determine the second metric dependent threshold candidate combination 660 based on the first metric dependent threshold candidate 650 combination. In one embodiment, the server 106 may adjust one threshold from the first candidate combination 650 while keeping the remainder of the first candidate combination 650 unchanged.

[0165] Thus, the server 106 can determine the second combination candidate 660 by adjusting the second metric dependent threshold 640 from the first combination candidate 650. In this example, the second combination candidate 660 comprises a first metric dependent threshold 630 and an adjusted second dependent metric threshold 642 (based on second metric dependent threshold 640).

[0166] In at least some embodiments of the present technology, server 106 may apply an adjustment algorithm capable of controlling direction (eg, increase or decrease) and absolute value to adjust the metric-dependent threshold from one iteration to the next. As described below, in one embodiment of the invention, the current adjustment value by which the metric-dependent threshold may be adjusted for the next iteration may depend on the previous adjustment value by which the metric-dependent threshold was adjusted for the previous iteration.

[0167] During the second iteration, the server 106 may use a first metric dependent threshold 630 from the second candidate combination 660 to compare against the first predictions 622 and an adjusted second metric dependent threshold 642 from the second candidate combination 660 to compare against the second predictions 624.

[0168] For example, the server 106 may compare the first prediction 612 for the test object associated with the object-specific test set 225 against the first metric-dependent threshold 630 and determine the predicted class 872 as a result. For example, if the first prediction 612 is greater than the first dependent from the threshold metric 630, then the server 106 may determine that the predicted class 872 indicates the first class. Otherwise, if the first prediction 612 is less than the first metric-dependent threshold 630, then the server 106 may determine that the predicted class 872 points to the second class. The server 106 may similarly compare each prediction from the first prediction set 622 to form a third prediction class set 874 . It can be said that the third prediction class data 870 is generated based on the comparison of the first prediction data 610 with the first metric dependent threshold 630.

[0169] In the same example, the server 106 may compare the second prediction 614 for the test object associated with the object-specific test set 225 with the adjusted second metric-dependent threshold 642 and determine the predicted class 882 as a result. The server 106 may similarly compare each prediction from a plurality of 624 second predictions to form a fourth plurality of 884 predictive classes. It can be said that the fourth class prediction data 880 is generated based on the comparison of the second prediction data 620 with the adjusted second metric dependent threshold 642.

[0170] The server 106 may compare the third predicted class data 870 and the fourth predicted class data 880 with the markup data 690 from the object-specific check data 220. Thus, the server 106 may compare the third predictive class data 870 and the fourth predictive class data 880 with the data 690 markups for evaluating the classification performance of the first nested metric 412 and the second nested metric 414 used in conjunction with the first metric dependent threshold 630 and the adjusted second metric dependent threshold 642, respectively. For example, the server 106 may define a second precision parameter 892 and a second recall parameter 894 for the first nested metric 412, as well as a different second precision parameter 896 and a different second recall parameter 898 for the second nested metric 414 when used with the second candidate combination 660.

[0171] Thus, it can be said that the server 106 during the second iteration can generate the second precision parameters (the second precision parameter 892 and the other second precision parameter 896) and the second completeness parameters (the second completeness parameter 894 and the other second completeness parameter 898) for nested metric sets 410 by comparing the control classes from the object-specific test data 220 with the predicted test object classes.

[0172] It should be noted that the server 106 may store an indication of the second candidate combination 660 metric dependent thresholds in combination with a second precision parameter 892 and a second recall parameter 894 for the first nested metric 412, and also in combination with another second precision parameter 896 and another second fullness parameter 898 for the second nested metric 414.

[0173] The server 106 may perform a large number of iterations, similar to how the server 106 performs the first and second iterations. In some embodiments, the server 106 may iterate the iterative verification process until a stop criterion is reached. For example, server 106 may continue the iterative verification process until one or more precision parameters and/or one or more recall parameters for the training iteration exceed one or more predetermined thresholds, as described below.

[0174] As described above, it is contemplated that the server 106 can select a metric-dependent threshold adjustment direction from a candidate combination to generate a new (adjusted) candidate combination depending on the previous adjustment direction of that metric-dependent threshold.

[0175] For example, it can be assumed that during the third iteration, the server 106 uses the second candidate combination 660 to form a new (third) candidate combination. The server 106 may retain the adjusted second metric dependent threshold 642 in the third candidate combination unchanged and use the adjusted first metric dependent threshold instead of the first metric dependent threshold 630. In other words, the third candidate combination may contain an adjusted (new) first dependent on metric threshold and an adjusted second metric dependent threshold 642. Server 106 may perform a third iteration of the iterative verification process using this third candidate combination, similar to the first and/or second iteration described above.

[0176] In the same example, when server 106 generates a new (fourth) candidate combination for the fourth iteration, server 106 may determine to keep the adjusted first metric dependent threshold from the third candidate combination unchanged and adjust the adjusted second dependent on metric threshold 642. To this end, in some embodiments, server 106 may access information indicative of a previous metric-dependent threshold adjustment for second nested metric 414. For example, server 106 may obtain the previous value that server 106 previously adjusted the second metric-dependent threshold 640 to form an adjusted second metric-dependent threshold 642. The previous value may have a sign (adjustment direction) and modulus (absolute adjustment value).

[0177] It is contemplated that server 106 can determine a new value by which server 106 should adjust the previously adjusted second metric dependent threshold 642 to generate a re-adjusted second metric dependent threshold in the fourth candidate combination by determining a value that has (a ) sign different from the sign of the previous value (different direction of adjustment), and (b) smaller modulus (smaller absolute value of adjustment). The server 106 may then perform a fourth iteration of the iterative verification process using this fourth candidate combination comprising the adjusted first metric-dependent threshold and the re-adjusted second metric-dependent threshold.

[0178] It is also contemplated that server 106 can select a candidate combination from a plurality of candidate combinations of metric dependent thresholds (associated with respective iterations of the iterative validation process) by comparing precision and recall parameters with respective parameter thresholds. It is contemplated that the parameter thresholds may comprise a predetermined accuracy threshold and/or a predetermined completeness threshold (eg, as defined by an interactive service operator).

[0179] In one embodiment, server 106 may use a predetermined accuracy threshold indicative of a minimum accuracy parameter that an operator deems appropriate to perform binary classification of objects. In this embodiment, for the first iteration, the server 106 may compare the first accuracy parameter 692 and another first accuracy parameter 696 with a predetermined accuracy threshold. If the first accuracy parameter 692 and the other first accuracy parameter 696 are greater than a predetermined accuracy threshold, then the first candidate metric-dependent threshold combination 650 is stored for later processing. If the first accuracy parameter 692 and/or the other first accuracy parameter 696 is less than a predetermined accuracy threshold, then the first candidate metric dependent threshold combination 650 is excluded from further processing. In this embodiment, for the second iteration, the server 106 may compare the second accuracy parameter 892 and another second accuracy parameter 896 with a predetermined accuracy threshold. If the second accuracy parameter 892 and the other second accuracy parameter 896 are greater than a predetermined accuracy threshold, then a second candidate metric-dependent threshold combination 660 is stored for later processing. If the second accuracy parameter 892 and/or other second accuracy parameter 896 is less than a predetermined accuracy threshold, then the second candidate metric dependent threshold combination 660 is excluded from further processing. In this embodiment, the server 106 may select a candidate combination from among the stored candidate combinations associated with the highest recall parameters from among the recall parameters of other stored candidate combinations as the target combination of metric dependent thresholds.

[0180] Additionally or alternatively, server 106 may determine a candidate combination from a plurality of candidate combinations to be stored based on a predetermined completeness threshold (the minimum completeness parameter that the operator considers appropriate). The server 106 may then select the candidate combination among the stored candidate combinations associated with the highest accuracy parameters among the accuracy parameters of the other stored candidate combinations as the target combination.

[0181] In FIG. 7 is a flow diagram of a method 700 in accordance with at least some non-limiting embodiments of the present technology. The various steps of a method 700 for determining a target combination of metric-dependent thresholds to be used with a plurality of nested metrics to perform a binary classification of a digital object are described in more detail below.

Step 702: Obtaining a plurality of object-specific test sets.

[0182] The method 700 begins at step 702, where the server 106 may receive a plurality of object-specific validation datasets. For example, server 106 may receive object-specific validation data 220 containing a plurality of object-specific validation datasets. The set of the plurality of object-specific verification datasets contains an indication of a plurality of past object events associated with the verification object, and a verification object's control class, which is the first class or the second class.

[0183] In some embodiments, the object may be an email message, with the first class corresponding to spam and the second class corresponding to no spam. Alternatively, the entity may represent an e-commerce platform user, with the first class corresponding to fraud and the second class corresponding to no fraud. In some cases, the object may be a document, with the first class corresponding to relevance and the second class corresponding to irrelevance.

Step 704: Applying a plurality of nested metrics to a plurality of object-specific test sets.

[0184] The method 700 continues at step 704, where the server 106 may apply a set of nested metrics 410 to a set of object-specific test sets and thus generate a set 625 of predictions. The prediction indicates the probability that the test object belongs to the first class or to the second class.

[0185] In some embodiments of the invention, the first metric of the set of nested metrics may be based on the events of the first type of object, and the second metric of the set of nested metrics may be based on the events of the second type of object, while the events of the second type object occur only if the object events occurred first kind. In some embodiments of the invention, the events of the second kind of object may be a subset of the events of the first kind of object.

Step 706: comparing, during the first iteration, the set of predictions with the corresponding thresholds from the first combination of metric dependent thresholds to determine the predicted test object classes for the first iteration.

[0186] The method 700 continues at step 706, where the server 106 during the first iteration may compare the prediction set 625 with the corresponding thresholds from the first metric dependent threshold combination 650 to determine the predicted test object classes for the first iteration. For example, server 106 may determine first predictive class data 670 and second predictive class data 680.

Step 708: generating during the first iteration the first precision parameters and the first recall parameters for the set of nested metrics for the first iteration by comparing the control classes with the predicted test object classes for the first iteration.

[0187] The method 700 continues at step 708, where the server 106 may generate first precision parameters and first recall parameters for the nested metric set 410 for the first iteration by comparing the control classes with the predicted test object classes for the first iteration.

[0188] For example, server 106 may generate a first precision parameter 692 and a first recall parameter 694 for the first nested metric 412, as well as another first precision parameter 696 and a different first recall parameter 698 for the second nested metric 414 (see FIG. 6).

Step 710: Adjusting during the second iteration of the threshold from the first combination of metric dependent thresholds and thus generating a second combination of metric dependent thresholds.

[0189] The method 700 continues at step 710, where the server 106 during the second iteration may adjust the threshold from the first metric-dependent threshold combination 650 and thus generate the second metric-dependent threshold combination 660.

[0190] For example, server 106 may adjust the metric-dependent threshold 640 and thus determine an adjusted metric-dependent threshold 642. As a result, the second metric-dependent threshold combination 660 comprises an adjusted metric-dependent threshold 642 instead of a metric-dependent threshold 640.

Step 712: comparing, during the second iteration, the plurality of predictions with the corresponding thresholds from the second combination of metric dependent thresholds to determine the predicted test object classes for the second iteration.

[0191] The method 700 continues at step 712, where the server 106 during the second iteration may compare the prediction set 625 with the corresponding thresholds from the second metric dependent threshold combination 660 to determine the predicted test object classes for the second iteration. For example, server 106 may determine third predictive class data 870 and fourth predictive class data 880.

Step 714: generating, during the second iteration, second precision parameters and second recall parameters for the set of nested metrics for the second iteration by comparing the control classes with the corresponding predicted test object classes for the second iteration.

[0192] The method 700 continues at step 714, where the server 106 during the second iteration may generate second precision parameters and second recall parameters for the nested metric set 410 for the second iteration by comparing the control classes with the corresponding predicted test object classes for the second iteration.

[0193] As shown in FIG. 6, the server 106 may generate a second precision parameter 892 and a second recall parameter 894 for the first nested metric 412. The server 106 may generate a different second precision parameter 896 and a different second recall parameter 898 for the second nested metric 414.

Step 716: Server selection of the first metric dependent threshold combination or the second metric dependent threshold combination as the target metric dependent threshold combination.

[0194] The method 700 continues at step 716, where the server 106 may select the first combination 650 or the second combination 660 as the target combination of metric-dependent thresholds by comparing (a) the first accuracy parameters and the second accuracy parameters with a precision threshold and/or ( b) the first parameters of completeness and the second parameters of completeness with a threshold of completeness. The target combination 450 dependent on the metric thresholds is to be used with a set of nested metrics 410 in the use mode to perform a binary classification of a digital object such that if the prediction of the stage of using at least one metric from the set 410 nested metrics for the digital object exceeds the corresponding threshold from the target combination 450 thresholds depending on the metric, then the belonging of the digital object to the first class is determined.

[0195] It should be noted that in some embodiments of the invention, the target combination of metric-dependent thresholds is selected simultaneously for multiple nested metrics during one iteration.

[0196] It is obvious that not all of the technical effects mentioned in this description should be present in every embodiment of the present technology. For example, there may be embodiments of the present technology where the user does not receive some of these technical effects, or other implementations where the user receives other technical effects, or when there is no technical effect.

[0197] Some of these steps and transmitted or received signals are well known in the art and are therefore omitted in some parts of the description for simplicity. Signals may be transmitted or received using optical means (such as a fiber optic connection), electronic means (such as a wired or wireless connection), and mechanical means (such as based on pressure, temperature, or any other suitable physical parameter).

[0198] For a person skilled in the art, possible changes and improvements to the above-described embodiments of the present technology may be obvious. The foregoing description is for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of this technology is determined solely by the scope of the appended claims.

Claims (52)

1. A method for determining a target combination of metric-dependent thresholds for use with a plurality of nested metrics when performing a binary classification into a first class or a second class of a digital object associated with past events of the object, an indication of which is stored in a data store, performed by a server capable of accessing the store data and includes: - receiving by the server a plurality of object-specific verification datasets, each of which contains an indication of the set of past events of the object associated with the verification object, and the control class of the verification object, which is the first class or the second class; - the application by the server of a plurality of nested metrics in relation to a plurality of object-specific test data sets and thus the formation of a plurality of predictions, each of which indicates the probability that the test object belongs to the first class or to the second class; - during the first iteration: - comparing the set of predictions by the server with the corresponding thresholds from the first combination of metric-dependent thresholds to determine the predicted test object classes for the first iteration; - the formation by the server of the first accuracy parameters and the first recall parameters for the set of nested metrics for the first iteration by comparing the control classes with the corresponding predicted classes of test objects for the first iteration; - during the second iteration: - adjusting the threshold by the server from the first combination of metric-dependent thresholds and thus generating a second combination of metric-dependent thresholds; - comparing the set of predictions by the server with the corresponding thresholds from the second combination of metric-dependent thresholds to determine the predicted test object classes for the second iteration; - the formation by the server of the second parameters of accuracy and the second parameters of completeness for the set of nested metrics for the second iteration by comparing the control classes with the corresponding predicted classes of test objects for the second iteration; And - selecting by the server the first combination of metric-dependent thresholds or the second combination of metric-dependent thresholds as the target combination of metric-dependent thresholds by comparing (a) the first accuracy parameters and the second accuracy parameters with the accuracy threshold and/or (b) the first recall parameters and the second recall parameters with a recall threshold, wherein the target combination of metric-dependent thresholds is to be used with a plurality of nested metrics in a usage mode to perform a binary classification of a digital object such that if the prediction of the usage stage of at least one metric from the plurality of nested metrics for the digital object exceeds the corresponding threshold from the target combination of thresholds dependent on the metric, then the belonging of the digital object to the first class is determined. 2. The method according to p. 1, characterized in that it additionally includes: - the server performs multiple iterations until (a) the precision parameters for the iteration are greater than the precision threshold and/or (b) the recall parameters for the iteration are greater than the recall threshold; And - selection by the server of a combination of metric-dependent thresholds from the iteration as the target combination of metric-dependent thresholds. 3. The method according to p. 1, characterized in that in the mode of use it additionally includes: - receipt by the server of an object-specific data set containing an indication of a set of past events of the object associated with the object; - the application by the server of a set of nested metrics in relation to the data set dependent on the object and thus the formation of one or more predictions indicating the probability of belonging to the object in the first class or in the second class; - comparison by the server of one or more forecasts with the corresponding thresholds from the target combination of metric-dependent thresholds; - determining by the server that the object belongs to the first class if at least one prediction exceeds the corresponding threshold from the target combination of metric-dependent thresholds; And - determination by the server that the object belongs to the second class if there is no forecast that exceeds the corresponding threshold from the target combination of metric-dependent thresholds. 4. The method according to claim 1, characterized in that the first metric from the set of nested metrics is based on the events of the first type object, and the second metric from the set of nested metrics is based on the events of the second type object, while the events of the second type object occur only if events of the first type object. 5. The method according to claim 4, characterized in that the events of the second type object are a subset of the events of the first type object. 6. The method of claim. 1, characterized in that the target combination of metric-dependent thresholds is selected simultaneously for multiple nested metrics during one iteration. 7. The method of claim. 1, characterized in that the object is an e-mail message, while the first class corresponds to spam, and the second class corresponds to the absence of spam. 8. The method of claim. 1, characterized in that the object is a user of the e-commerce platform, while the first class corresponds to fraud, and the second class corresponds to the absence of fraud. 9. The method of claim. 1, characterized in that the object is a document, while the first class corresponds to relevance, and the second class corresponds to irrelevance. 10. Server for determining the target combination of metric-dependent thresholds for use with a plurality of nested metrics when performing binary classification into the first class or second class of a digital object associated with past events of the object, an indication of which is stored in the data store, configured to access the store data and configured to: obtaining a plurality of object-specific verification datasets, each of which contains an indication of a plurality of past object events associated with the verification object, and the verification object's control class, which is the first class or the second class; applying a plurality of nested metrics to a plurality of object-specific test sets and thus generating a plurality of predictions, each of which indicates the probability that the test object belongs to the first class or to the second class; - during the first iteration: comparing the plurality of predictions with corresponding thresholds from the first combination of metric-dependent thresholds to determine the predicted test object classes for the first iteration; - generating first accuracy parameters and first recall parameters for the set of nested metrics for the first iteration by comparing the control classes with the corresponding predicted test object classes for the first iteration; - during the second iteration: - adjusting the threshold from the first combination of metric-dependent thresholds and thus generating a second combination of metric-dependent thresholds; comparing the plurality of predictions with the corresponding thresholds from the second combination of metric-dependent thresholds to determine the predicted test object classes for the second iteration; - generating second accuracy parameters and second recall parameters for the set of nested metrics for the second iteration by comparing the control classes with the predicted test object classes for the second iteration; And - selecting the first combination of metric-dependent thresholds or the second combination of metric-dependent thresholds as the target combination of metric-dependent thresholds by comparing (a) the first accuracy parameters and the second accuracy parameters with the accuracy threshold and/or (b) the first recall parameters and the second parameters of completeness with a threshold of completeness, wherein the target combination of metric-dependent thresholds is to be used with a plurality of nested metrics in the usage mode to perform such a binary classification of a digital object that if the prediction of the stage of using at least one metric from a plurality of nested metrics for a digital object exceeds the corresponding threshold from the target combination of thresholds depending on the metric, then the belonging of the digital object to the first class is determined. 11. The server according to claim 10, characterized in that it is additionally configured to: - performing multiple iterations until (a) the precision parameters for the iteration are greater than the precision threshold and/or (b) the recall parameters for the iteration are greater than the recall threshold; And - selecting a combination of metric-dependent thresholds from this iteration as the target combination of metric-dependent thresholds. 12. The server according to claim 10, characterized in that in the mode of use it is additionally configured to: - obtaining an object-specific data set containing an indication of a set of past events of the object associated with the object; - applying a plurality of nested metrics to an object-specific data set and thereby generating one or more predictions indicating the probability of an object belonging to the first class or to the second class; - comparing one or more forecasts with the corresponding thresholds from the target combination of metric-dependent thresholds; - determining whether the object belongs to the first class if at least one prediction exceeds the corresponding threshold from the target combination of metric-dependent thresholds; And - determining whether an object belongs to the second class if there is no forecast that exceeds the corresponding threshold from the target combination of metric-dependent thresholds. 13. The server according to claim 10, characterized in that the first metric from the set of nested metrics is based on the events of the first type object, and the second metric from the set of nested metrics is based on the events of the second type object, while the events of the second type object occur only if events of the first type object. 14. The server according to claim 13, characterized in that the events of the object of the second type are a subset of the events of the object of the first type. 15. The server according to claim 10, characterized in that the target combination of metric-dependent thresholds is selected simultaneously for multiple nested metrics during one iteration. 16. The server according to claim. 10, characterized in that the object is an e-mail message, while the first class corresponds to spam, and the second class corresponds to the absence of spam. 17. The server according to claim 10, characterized in that the object is a user of the e-commerce platform, the first class corresponds to fraud, and the second class corresponds to the absence of fraud. 18. The server according to claim 10, characterized in that the object is a document, while the first class corresponds to relevance, and the second class corresponds to irrelevance.

Images