JP2023539430A - Electronic trading system and method based on point-to-point mesh architecture - Google Patents

Abstract

The electronic trading system and corresponding method are based on a point-to-point mesh architecture and include a gateway, a core compute node and a sequencer, the core compute node performs an electronic transaction matching function, and the gateway sends a message to the core compute node in a first Send via direct connection. The gateway sends the message to the sequencer over a second direct connection, the sequencer sends the order-marked message to the core compute node over a third direct connection, and the core compute node sends the order-marked message to the others based on the order-marked message. determine the relative ranking among the messages to complete the electronic transaction matching function. The gateway, core compute node, sequencer and each direct connection form part of a point-to-point mesh architecture that exhibits low latency, fairness and fault tolerance to perform fast, deterministic electronic transactions of financial instruments.

Description

(Cross reference to related applications)
This application is a continuation-in-part of U.S. Patent Application No. 16/988510, filed August 7, 2020, and a continuation-in-part of United States Patent Application No. 16/988491, filed August 7, 2020. The entire teachings of the above applications are incorporated herein by reference.

Current financial instruments trading systems allow traders to submit orders and receive confirmations, market data, and other information electronically via communication networks. The ``electronic'' market enabled by this so-called ``electronic trading system'' is an ``electronic'' market in which all traditional traders or their agents physically stand in a designated position, i.e., the trading pit, and communicate verbally and visually. It has largely replaced the pit-based "open outcry" trading system of trading with each other through hand gesture communication and has become the dominant means of trading most financial instruments.

When trading stocks and/or other financial instruments in an electronic trading system, the electronic trading system typically generates trades by matching trade orders, ie, sell orders and corresponding buy orders. Depending on the conditions related to the transaction, the process of matching sell price orders and buy price orders may become complicated.

One example of an electronic trading system is a computerized exchange that typically includes a central matching engine residing within a central server and multiple distributed servers or gateways. In such computerized exchanges, typical processing may be as follows. Order entry messages, e.g., buy and sell orders, are sent from a client or participant device, e.g., a trader terminal, to a computerized exchange, and the computerized exchange processes the order entry messages. This order entry message is received at the central server via the gateway. Processing at a computerized exchange, eg, at a central server, may include, among other things, performing order matching based on received order entry messages.

An order processing approval message generated by the central server is then sent back to the participant device, typically via a gateway that forwards the transaction. The gateway may perform additional processing before the order processing approval message is sent back to the participant device. The central server disseminates information about the order processing authorization message, either in the same format as it was received or in other formats, to one or more other gateways that perform processing of the order processing authorization message to the market data stream. Market data output may be generated via. Market data output is typically transferred to participant devices or other subscribers to the market data stream through various communication mechanisms and requires additional processing at the gateway.

According to an exemplary embodiment, an electronic trading system includes a gateway, a core compute node (also interchangeably referred to herein as a core compute engine or a compute engine) configured to perform electronic transaction matching functions, and a sequencer. The gateway and core compute node are coupled via a first direct connection. The gateway and sequencer are coupled via a second direct connection. The sequencer and core compute node are coupled via a third direct connection. The first, second and third direct connections have respective unshared bandwidths. The gateway is configured to send a message representing an electronic trading request with a limit price to buy or sell a financial instrument to the core compute node via the first direct connection. In contrast, messages are received by core compute nodes. The gateway is further configured to send the message to the sequencer via the second direct connection, and the sequencer in turn is configured to send the order-marked message to the core compute node via the third direct connection. It is composed of A sequencer is inserted between the gateway and the core compute node via second and third direct connections. The order marked messages sent by the sequencer are ordered versions of the messages sent by the gateway. In contrast, order-marked messages are received by core compute nodes. The core compute node is configured to determine a relative ranking (i.e., order) of the order-marked message among order-marked versions of other messages received by the core compute node in the electronic trading system. The core compute node is further configured to complete an electronic trade matching function for the electronic trade request in accordance with the determined relative ranking, match the limit price with a counterparty limit price for the financial instrument, and enable electronic trading of the financial instrument. be done.

The message and the order marked message may contain the same user data. User data is associated with electronic transaction requests.

The message may be a gateway message sent by the gateway in response to receipt of an incoming message received by the gateway from a participant device. The sequencer is further configured to send the order marked message to the gateway via the second direct connection. In contrast, order-marked messages are received by the gateway. The order marked message is the first order marked message. The core compute node is further configured to transmit the core compute node message to the gateway via the first direct connection in response to receiving the gateway message. The core compute node is further configured to send the core compute node message to the sequencer via the third direct connection, and the sequencer is further configured to send the second ordered marked message to the gateway via the second direct connection. Configured to send over a direct connection. The second order marked message is an ordered version of the core compute node message. The gateway is further configured to determine a relative ranking of the second order-marked message and order-marked versions of other messages sent from the core compute node to the gateway. The gateway is further configured to send outgoing messages to participant devices. Outgoing messages are sent according to the determined relative ranking. The sequencer is further configured in response to send the second order-marked message to the core compute node via the third direct connection.

The gateway may be a given gateway among a plurality of gateways. A core compute node may be a given core compute node among a plurality of core compute nodes. Each gateway of the plurality of gateways may be coupled to a respective core compute node of the plurality of core compute nodes via a respective first direct connection. The sequencer may be coupled to each gateway of the plurality of gateways via a respective second direct connection and to each core compute node of the plurality of core compute nodes via a respective third direct connection. The plurality of gateways, the plurality of core compute nodes, the sequencer and their respective direct connections may constitute at least a portion of a point-to-point mesh system.

Within a point-to-point mesh system, each gateway of the plurality of gateways may be configured to send messages destined for the respective compute node sent therefrom to all compute nodes and sequencers of the plurality of core compute nodes. It should be understood that messages addressed to a compute node may be interchangeably referred to herein as "destination to compute node" messages. Additionally, a message addressed to a gateway may be interchangeably referred to herein as a "gateway addressed" message. Each core compute node of the plurality of core compute nodes may be configured to transmit messages destined for a respective gateway sent therefrom to all gateways and sequencers of the plurality of gateways. The sequencer may be further configured to send respective order-marked messages to the plurality of gateways and the plurality of core compute nodes in response to receiving the respective compute node-directed message or the respective gateway-directed message.

The sequencer may be a given sequencer among multiple sequencers in a point-to-point mesh system. Each gateway of the plurality of gateways may be coupled to each sequencer of the plurality of sequencers via a respective second direct connection. Each core compute node of the plurality of core compute nodes may be coupled to each sequencer of the plurality of sequencers via a respective third direct connection. A given sequencer may be a currently active sequencer serving a point-to-point mesh system. Each other sequencer of the plurality of sequencers may be a standby sequencer waiting to take over for the currently active sequencer. Each sequencer of the plurality of sequencers may be coupled to each other sequencer of the plurality of sequencers via a respective fourth direct connection. Each gateway of the plurality of gateways may be further configured to send messages destined for a respective compute node to a given sequencer of the plurality of sequencers. Each core compute node of the plurality of core compute nodes may be further configured to send a respective gateway-directed message to a given sequencer of the plurality of sequencers. The given sequencer is further configured to send order marked messages to each other sequencer of the plurality of sequencers via each respective fourth direct connection so that a standby sequencer can It may be configured to allow the currently active sequencer to take over.

Each compute node-directed message sent by a given gateway may be the same message received by multiple core compute nodes. Multiple core compute nodes may be configured to generate response messages in response to receiving the same message. A response message may be received at a given gateway from among multiple core compute nodes. A given gateway may be further configured to take action based on a given response message of the response messages generated in response to receipt of the same message. A given response message may arrive at a given gateway first relative to other response messages generated in response to receipt of the same message. The gateway may further be configured to ignore other response messages that arrive after a given response message.

Messages destined for multiple compute nodes representing the same message may be received from among multiple gateways at a given compute node. A given compute node may be further configured to take an action based on a given compute node-directed message of the plurality of compute node-directed messages. A message destined for a given compute node may arrive at the given compute node first relative to messages destined for other compute nodes among multiple compute node messages representing the same message. A given core compute node may be further configured to ignore messages destined for other compute nodes that arrive after messages destined for the given compute node.

The electronic trading system may further include an order book accessible by the core compute node. The core compute node may be further configured to match trade orders for financial instruments using electronic trade matching functionality. The core compute node may further be configured to maintain a balance of financial instruments in the order book. Amount discrepancies in financial instruments may result from performing electronic transaction matching functions. The remaining balance may include amount discrepancies in financial instruments. It should be understood that balances can convey more information than quantities. For example, positions may be bullish and bearish (sides). According to an exemplary embodiment, the balance conveys both price and quantity.

The electronic trading system may further include a clock. The gateway, core compute node and sequencer may be synchronized based on a clock.

The gateway may further be configured to serve at least one participant device and send messages to the sequencer and core compute nodes in response to receiving incoming messages at the gateway. Incoming messages may be dispatched by at least one participant device. The sequencer further comprises marking the message with the unique ordinal identifier, or by creating an indication of the received message, marking the indication with the unique ordinal identifier, and transmitting the marked indication. , may be configured to generate order-marked messages. The marked indication may be an ordinal marked message.

The electronic trading system may further comprise at least one respective redundant direct connection for the first direct connection, the second direct connection and the third direct connection or some set thereof.

The gateway may be a given gateway of a plurality of gateways communicatively coupled to each other via a shared gateway network. A core compute node may be a given core compute node of a plurality of core compute nodes communicatively coupled to each other via a shared core compute node network. The sequencer may be a given sequencer of a plurality of sequencers communicatively coupled to each other via a shared sequencer network or via respective fourth direct connections.

The electronic trading system may further include a system status log. A given sequencer may be configured to transmit a system state log to at least one other sequencer of the plurality of sequencers via a shared sequencer network or to store the system state log in a data store. A data store may be accessible to multiple sequencers via a shared sequencer network.

The electronic trading system may be an active electronic trading system, and at least one sequencer of the plurality of sequencers may be communicatively coupled to a disaster recovery site. A disaster recovery site may include a standby electronic trading system. The standby electronic trading system may be a replica of the active electronic trading system and may be configured to allow electronic trading to continue in the event of a failure of the active electronic trading system.

The gateway, core compute node, sequencer, and first, second, and third direct connections may configure a first point-to-point mesh system. The electronic trading system may be a first electronic trading system communicatively coupled to the proxy node. The proxy node may further be communicatively coupled to the at least one participant device and the second electronic trading system. The second electronic trading system may include a second point-to-point mesh system. The proxy node may be configured to send a message to the first and second electronic trading systems in response to receiving an incoming message from the at least one participant device. The first and second electronic trading systems may be configured to generate respective responses to messages sent by the proxy node. The proxy node is further configured to send a response to the at least one participant device in response to receiving the first arriving response of the respective responses generated and received from the first or second electronic trading system. can be done.

According to another exemplary embodiment, a method for performing an electronic transaction includes transmitting a message representing an electronic transaction request with a limit price to buy or sell a financial instrument from a gateway to a core compute node via a first direct connection. Equipped with steps. The method, in turn, further comprises receiving a message at a core compute node to perform an electronic trading function in the electronic trading system. The method includes the steps of transmitting a message from the gateway to a sequencer in an electronic trading system via a second direct connection, and transmitting an order-marked message from the sequencer to a core compute node via a third direct connection. Furthermore, it is equipped with. The first, second and third direct connections have respective unshared bandwidths. A sequencer is inserted between the gateway and the core compute node via second and third direct connections. The order marked messages sent by the sequencer are ordered versions of the messages sent from the gateway over the second direct connection. The method, in turn, further comprises receiving the order-marked message at the core compute node. The method further comprises receiving, at the core compute node, another message from the gateway, and receiving, at the core compute node, an order-marked version of the other message from the sequencer. The method further comprises determining at the core compute node a relative ranking of the order marked message among order marked versions of other messages received by the core compute node in the electronic trading system. The method further comprises completing, at the core compute node, an electronic trade matching function for the electronic trade request in accordance with the determined relative ranking, the completing step matching a limit price with a counterparty limit price for the financial instrument; Enabling electronic trading of financial products.

Alternative method embodiments are similar to those described above in connection with the exemplary electronic trading system embodiments.

According to other exemplary embodiments, a sequencer for an electronic trading system includes a non-transitory computer-readable medium encoding a series of instructions that, when loaded into and executed by the sequencer, causes the sequencer to electronically The point-to-point mesh system of the transaction system is in direct communication with the gateway via a first direct connection. The series of instructions further causes the sequencer to communicate directly with the core compute node via the second direct connection in the point-to-point mesh system of the electronic trading system. A sequencer is inserted between the gateway and the core compute node via first and second direct connections. The first and second direct connections have respective unshared bandwidths. The set of instructions further causes the sequencer to generate an order-marked message by marking the message or representation thereof with a unique order identifier, the message being received by the sequencer via the first or second direct connection, respectively. be done. The series of instructions further causes the sequencer to send order marked messages to the gateway and core compute nodes.

According to another exemplary embodiment, an electronic trading system includes a gateway coupled to a core compute node via an activation link, a ranking path, and a sequencer electronically located within the ranking path. The gateway is configured to send messages to the core compute nodes via the activation link and the ranking path. The core compute nodes are configured to receive messages and order-marked versions of messages from the gateway and the sequencer, respectively. The order marked version includes an order identifier indicating the definitive position of the message among the plurality of messages communicated via the activation link and received by the sequencer via the ranking path. Messages and order marked versions contain common metadata.

The core compute node (i) initiates a matching function activity for the electronic transaction in response to receiving the message via the activation link, and (ii) in response to receiving the order-marked version via the ranking path. , may be configured to prioritize completion of matching function activities toward servicing electronic transactions using the ordering identifier.

Messages and order marked versions may include common metadata. The core compute node may be further configured to correlate the message to the order marked version based on the common metadata in response to receiving the order marked version via the ranking path.

According to yet other exemplary embodiments, an electronic trading system may include a gateway, a sequencer, and a core compute node arranged in a point-to-point mesh topology. The core compute nodes may be configured to perform matching functions toward servicing transaction requests received from participant devices and introduced into the point-to-point mesh topology via the gateway. The point-to-point mesh topology may include a first direct connection, a second direct connection, and a third direct connection. The sequencer is configured to: (i) respond to messages communicated via a first direct connection between the gateway and the core compute node and received by the sequencer via a second or third direct connection from the gateway or core compute node, respectively; It may be configured to determine a definitive ranking (ie, order). The sequencer further comprises: (ii) transmitting order-marked versions of messages representing transaction requests or responses thereto to the gateway and core compute nodes, respectively, within the deterministic ranking; The message may be configured to communicate the location of the message.

It should be understood that the example embodiments disclosed herein can be implemented in the form of a method, apparatus, system, or computer-readable medium having program code embodied therein.

The foregoing will become apparent from the following more detailed description of exemplary embodiments, as illustrated in the accompanying drawings, in which like numerals refer to like parts throughout the various drawings. The drawings are not necessarily to scale, emphasis instead being placed upon the illustrated embodiments.

FIG. 1A is a block diagram of an exemplary embodiment of a market for trading financial instruments. FIG. 1B-1 is a block diagram of an exemplary embodiment of an electronic trading system. FIG. 1B-2 is a block diagram of an exemplary embodiment of another electronic trading system. FIG. 1C is a block diagram of an exemplary embodiment of a point-to-point mesh system. FIG. ID is a block diagram of another example embodiment of an electronic trading system. FIG. 1E is a table of an example embodiment of fields of a message format for a transaction message. FIG. 1F is a flow diagram illustrating an exemplary embodiment of the operation of an electronic trading system. FIG. 1G is a flow diagram illustrating another exemplary embodiment of the operation of an electronic trading system. FIG. 2 is a block diagram of an exemplary embodiment of a mesh node in a point-to-point mesh architecture of an electronic trading system. FIG. 3 is a block diagram of another exemplary embodiment of a point-to-point mesh system. FIG. 4 is a block diagram of another exemplary embodiment of a point-to-point mesh system. FIG. 5 is a flow diagram of an exemplary embodiment of a method for performing electronic transactions. FIG. 6 is a block diagram of an exemplary embodiment of a sequencer of an electronic trading system. FIG. 7 is a block diagram of another exemplary embodiment of an electronic trading system.

A description of example embodiments follows.

It should be understood that the dedicated/direct connections disclosed herein are point-to-point connections that do not go through a shared network switch.

Although current electronic trading systems attempt to provide performance advantages, many suffer from performance degradation due to an increase in transaction volume due to an increase in the number of market participants. Practices by some market participants are often based on high-frequency trading methods in which high-speed computers automatically monitor the market and react to market events, usually in an overwhelming manner. There is also a continued demand for further reductions in processing latencies and response times, in order to maintain the performance experienced by each market participant and avoid deleterious consequences such as capacity exhaustion and unfair access. This has led to a need for further improvements in capacity and performance.

An increase in the speed with which market participants can evaluate and respond to changes in market data, such as in response to market events, increases the speed at which transactions are received into electronic trading systems and increases the time between receptions. The need for a higher degree of identification narrows and determines the order in which those transactions are received. Deterministic behavior of the electronic trading system is based on these, for example with respect to trading order assignments and the like. Furthermore, the addition of channels of communication to an electronic trading system allows more transactions to be submitted to the electronic trading system via multiple parallel paths to increase capacity and opportunity as the bandwidth of each channel increases. Become.

Therefore, it would be useful for electronic trading systems to identify incoming transactions received within a short period of time. It would be further useful for such systems to arbitrate between transactions that were received simultaneously or so close in time that they were considered to be received simultaneously. In addition to increased capacity and lower latency, the global nature of business has further increased the need for fault tolerance to increase the availability and reliability of electronic trading systems.

Business transactions are conducted in accordance with one or more relevant business rules (industry, legal or regulatory requirements or customs) to achieve a business or commercial purpose, which may include compliance with industry, regulatory or legal requirements. may be defined as one or more actions or effects contemplated according to A business transaction may be implemented by one or more computer processing and/or database operations/program actions, and may itself be referred to as a transaction. A business transaction, as defined by the associated business rules, may be characterized as deterministic in the following respects: It is defined by business rules and/or dependence on the order in which they are processed, such as temporal order, to enable business/commercial objectives and/or meet participants' expectations. Business transactions can be characterized by interdependencies or relationships that influence their outcome, such as dependence on real-time processing. This is referred to here as "transaction determinism." In general, a set of deterministic transactions will give a particular result when executed in one order (i.e., order) and will give a different result when executed in a different order (i.e., order). Become. In some applications, deterministic processing may be preferred/preferred over real-time processing.

High performance electronic trading systems improve trading opportunities, fault tolerance, low latency processing, high capacity (e.g. processing many messages per second), risk mitigation and market protection with minimal impact and information. It is useful to guarantee transaction determinism under increasing load while providing fair access to transactions and opportunities.

The exemplary embodiments disclosed herein provide a high-speed electronic trading system that provides a market where orders to buy and sell financial instruments such as stocks, bonds, commodities, futures, options, etc. are traded between market participants such as traders and brokers. Regarding. The exemplary embodiments of electronic trading systems disclosed herein exhibit low latency, fairness, fault tolerance, and other features described more fully below.

FIG. 1A is a block diagram of an exemplary embodiment of a market 90 that includes an exemplary embodiment of an electronic trading system 100 used to trade financial instruments (not shown). Electronic trading system 100 is primarily responsible for "matching" trading orders with each other and employs a point-to-point mesh system 102 to do so. In one example, a limit price to "buy" a financial instrument is matched by the matching engine of the point-to-point mesh system 102 to a corresponding counterparty limit price to "sell." The matched limit price and counterparty limit price must at least partially satisfy the desired price, and any remaining unfilled quantity is passed to other appropriate counter orders. The matched trading orders are then paired and the trade executed.

Orders that are not fully filled or partially filled are maintained in a data structure called an "order book" (not shown). Pending information regarding unmatched trading orders can be used by the matching engine to satisfy subsequent trading orders. An order book is typically maintained for each financial instrument and generally defines or represents the state of the market 90 for that particular instrument, ie, for that particular instrument. The order book may include, for example, recent prices and quantities at which market participants have expressed intent to buy or sell.

The results of the matching may be made visible to market participants via a streaming data service (not shown) called a market data feed (not shown). Market data feeds typically include individual messages conveying pricing and related information such as volume and other statistics for each financial instrument traded.

In market 90, market participants include two traders: a first trader 104a and a second trader 104b. It should be understood that market 90 is not limited to market participants who are traders, and market 90 is not limited to two traders. In the market 90, market participants, such as a first trader 104a and a second trader 104b, submit trading orders and receive confirmations, market data, and other information electronically via a communications network (not shown). can be received.

In the exemplary embodiment of FIG. 1A, a first trader 104a purchases a financial instrument (not shown) via a first incoming message 3a sent to electronic trading system 100 via first participant device 130a. A first trading order (not shown) has been submitted. Electronic trading system 100 employs a point-to-point mesh system 102 that matches a first trading order to a second trading order (not shown) to sell financial instruments. A second trading order to sell a financial instrument is submitted by a second trader 104b via a second incoming message 3b sent to the electronic trading system 100 via a second participant device 130b. .

In the exemplary embodiment, the electronic trading system 100 transmits the first outgoing message 5a and the second outgoing message 5b to the first participant device 130a and the second participant device 130b, respectively, to send the first outgoing message 5a and the second outgoing message 5b to the first participant device 130a and the second participant device 130b, respectively. 104a and second trader 104b, respectively, of successful execution of their respective trading orders. Point-to-point mesh system 102 enables electronic trading system 100 to perform fast, deterministic electronic trading of financial instruments. An exemplary embodiment of a point-to-point mesh system 102 is disclosed below with reference to FIGS. 1B-1 and 1B-2.

FIG. 1B-1 is a block diagram of an exemplary embodiment of the electronic trading system 100 of FIG. 1A disclosed above. In particular embodiments, electronic trading system 100 includes gateway 120-1 coupled to core compute node 140-1 via activation link 180-1-1 and ranking (ie, ordering) path 117. Electronic trading system 100 further includes a sequencer 150-1 electronically located within ranking path 117. Gateway 120-1 is configured to send messages (not shown) to core compute node 140-1 via activation link 180-1-1 and ranking path 117. The message may be the message 106 disclosed below with respect to FIG. 1B-2. Core compute node 140-1 is configured to receive messages and order-marked versions (not shown) of the messages from gateway 120-1 and sequencer 150-1, respectively. The order-marked version of the message may be order-marked message 106' of FIG. 1B-2, further disclosed below. The order marked version (i.e., the order marked message 106') may be included in the order ID field 110-14 of the order marked message 106' as further disclosed below with respect to FIG. 1E for a non-limiting example. Contains a sequence identifier (ID). The order ID determines the order of the message among order-marked versions of other messages communicated via activation link 180-1-1 and received by sequencer 150-1 via ranking path 117. Indicates the definitive position of the marked version. The plurality of messages whose order IDs indicate definitive positions also include order-marked versions of other messages received by core compute node 140-1 via ranking path 117. Messages (eg, unordered or unordered messages) and order-marked versions include common metadata (not shown). As further disclosed below, a message's order ID is identified by correlating the message to its order marked version via common metadata. The order ID further indicates the definitive position of the message among all messages communicated throughout electronic trading system 100 that have passed through sequencer 150-1 and have been order-marked as a result by sequencer 150-1.

Elements of electronic trading system 100 timestamp messages communicated therein, while the order ID determined by sequencer 150-1 determines the position (rank/priority) of messages communicated in electronic trading system 100. It should be understood that Multiple systems may timestamp a message with the same timestamp, and as a result, a ranking/priority for the message may have to be determined at its recipient. Such does not occur in electronic trading system 100 because sequencer 150-1 may be the sole determiner of the order/priority of messages communicated throughout electronic trading system 100.

Core compute node 140-1 (i) initiates matching function activity for electronic transactions in response to receiving the message via activation link 180-1-1; and (ii) initiates matching function activity via ranking path 117. In response to receiving the order marked version, the order identifier may be configured to prioritize completion of matching function activities toward servicing electronic transactions.

Messages and order marked versions may include common metadata. Core compute node 140-1 may be further configured to correlate the message to the order-marked versions based on common metadata in response to receiving the order-marked versions via ranking path 117.

In the example embodiment of FIG. 1B-1, the message is sent to core compute node 140-1 via activation link 180-1-1 in the activation link forward direction, i.e., at act-link-fwd-dir 113a. and is sent to core compute node 140-1 via ranking path 117 in the ranking path forward direction, ie, at order-path-fwd-dir 115a. Following completion of the matching function activity, core compute node 140-1 sends a response (not shown) to the activation link reverse direction (i.e., act-link-rev-dir 113b) and the ranking path reverse direction (i.e., order- path-rev-dir 115b) via activation link 180-1-1 and ranking path 117 to gateway 120-1. The response sent to gateway 120-1 via activation link 180-1-1 and ranking path 117 may be response 107, further disclosed below with respect to FIG. 1B-2.

Activation link 180-1-1 is a single direct connection, while ranking path 117 includes multiple direct connections. For example, activation link 180-1-1, also referred to as first direct connection 180-1-1, is a single direct connection, while ranking path 117 is a It may include two direct connections 180-gw1-s1 and a third direct connection 180-c1-s1.

Gateway 120-1, sequencer 150-1, and core compute node 140-1 are arranged in a point-to-point mesh topology. Core compute node 140-1 provides matching functionality (i.e., electronic transaction matching functionality) for servicing transaction requests received from participant devices and introduced into the point-to-point mesh topology via gateway 120-1. ). This participant device is further disclosed below with respect to FIG. 1B-2. The point-to-point mesh topology includes a first direct connection, a second direct connection, and a third direct connection, as disclosed above and further disclosed below with respect to FIG. 1B-2. Sequencer 150-1 is configured to: (i) communicate via a first direct connection between gateway 120-1 and core compute node 140-1 to transmit data from gateway 120-1 or core compute node 140-1, respectively; It may be configured to determine a definitive ranking (ie, order) for messages received by sequencer 150-1 via the third direct connection. Sequencer 150-1 further comprises: (ii) transmitting an order-marked version of the message to gateway 120-1 and core compute node 140-1 via second and third direct connections, respectively; may be configured to convey the location of the message. The message represents a transaction request or a response thereto, as disclosed below with respect to FIG. 1B-2.

FIG. 1B-2 is another block diagram of the exemplary embodiment of the electronic trading system 100 of FIG. 1A disclosed above. Electronic trading system 100 includes a gateway 120-1, a core compute node 140-1 configured to perform electronic transaction matching functions, and a sequencer 150-1. Gateway 120-1 and core compute node 140-1 are coupled via a first direct connection 180-1-1. Gateway 120-1 and sequencer 150-1 are coupled via a second direct connection 180-gw1-s1. Sequencer 150-1 and core compute node 140-1 are coupled via a third direct connection 180-c1-s1. The first direct connection 180-1-1, the second direct connection 180-gw1-s1 and the third direct connection 180-c1-s1 have respective non-shared bandwidths.

In some drawings of the disclosure, arrows such as the first direct connection 180-1-1, second direct connection 180-gw1-s1 and third direct connection 180-c1-s1 shown in FIG. 1B-2 are shown. Arrows are included on direct connections (ie, direct links). An arrow on such a direct connection/link indicates that data can flow in both directions via the direct connection/link. Although other figures of the disclosure may not have the above arrows applied to direct connections/links, such as FIG. 1B-1 disclosed above and FIG. 1D further disclosed below, the data may be may flow bidirectionally along a connection/link, and such connection/link may be a single communication link or two links in parallel, each of the two links being unidirectional; It should be understood that data flows in opposite directions through the two links.

Continuing with reference to FIG. 1B-2, gateway 120-1 sends message 106 (i.e., B) representing an electronic transaction request with a limit price to buy or sell a financial instrument to core compute node 140-1 on a first 180-1-1, to which message 106 is configured to be received by core compute node 140-1. Gateway 120-1 is further configured to send message 106 (i.e., B) to sequencer 150-1 via second direct connection 180-gw1-s1, to which sequencer 150-1 , configured to send the order-marked message 106' (i.e., C) to the core compute node 140-1 via the third direct connection 180-c1-s1, as further disclosed below with respect to FIG. 1D. Ru. Sequencer 150-1 is inserted between gateway 120-1 and core compute node 140 via second and third direct connections. The order marked message 106' sent by sequencer 150-1 is an ordered version of the message sent by gateway 120-1. In contrast, order-marked messages are received by core compute node 140-1. Core compute node 140-1 determines a relative ranking of order-marked message 106' among order-marked versions (not shown) of other messages received by core compute node 140-1 in electronic trading system 100. configured to do so. The core compute node 140-1 further completes an electronic trade matching function for the electronic trade request according to the determined relative ranking, and matches the limit price with the counterparty limit price for the financial instrument to match the electronic trade request. Configured to enable transactions.

It should be understood that the electronic trade matching function may involve more than matching trade orders per se. For example, the electronic transaction matching function may include sending an authorization message as further disclosed below with respect to FIG. 1D. Core compute node 140-1 may also perform some of the electronic transaction matching functions prior to receiving the order marked message 106' that enables core compute node 140-1 to facilitate it. For example, following receipt of message 106 (e.g., an out-of-order marked message), core compute node 140-1 determines the stock symbol field 110-2, as further disclosed below with respect to FIG. 1E, for non-limiting examples. The electronic trade matching function may be initiated by loading data regarding the stock symbols identified in message 106, such as symbols, into high speed memory for later access. A stock symbol may represent a traded security, such as a stock symbol or stock ticker. However, it should be understood that core compute node 140-1 may initiate (trigger) the electronic transaction matching function by performing any type of activity related to the content of message 106.

Sequencer 150-1 may be further configured to generate order-marked messages 106' by marking messages 106 or representations thereof with unique order identifiers (not shown). The unique order identifier may have a value corresponding to the time of arrival of message 106 at sequencer 150-1, and the relative order of message 106 among a plurality of messages (not shown) received at sequencer 150-1. It may also indicate the location.

Core compute node 140-1 may initiate electronic transaction functionality in response to receiving unordered message 106 and thereby begin processing unordered message 106, as described above, while core compute node 140-1 may not complete processing of message 106 and/or commit to the results of processing message 106 until core compute node 140-1 receives order-marked message 106'. Without a definitive ranking for processing messages, such as specified via an order identifier in order marked message 106', processing of messages by compute node 140-1, for example, may be unpredictable. A non-limiting example of a possible unpredictable outcome may be multiple outstanding unordered messages, each representing a potential match for a counterparty in an exchange of securities. It would be useful to have a deterministic way to arbitrate between multiple potential matches. This is likely because only a subset of potential matches will be available for execution for a given trading order of a counterparty.

According to some embodiments, after receiving both the unordered message 106 and the ordered message 106', the compute node 140-1 identifies the identification in both versions of the message, as described below in connection with FIG. 1E. The unordered message 106 may be correlated to the ordered message 106' via information. When the compute node 140-1 receives the order-marked message 106', the compute node 140-1 determines whether the message 106/106' is to be processed relative to other messages throughout the electronic trading system 100. Messages 106/106 may determine the appropriate order and send an appropriate response message, possibly referencing the order identifier assigned by the sequencer to the order marked message 106'. You may complete the process of ''. Returning to the non-limiting example of multiple messages representing potential matches for a counterparty in an exchange of securities, when an order-marked version of a message representing a potential match is received by compute node 140-1: Compute node 140-1 may determine the exact order in which potential matches occur to complete the electronic transaction matching function.

According to the example embodiment, in addition to transmitting the order marked message 106' to the compute node 140-1 via the third direct connection 180-c1-s1, the sequencer 150-1 further sends the order marked message 106' to the compute node 140-1 via the third direct connection 180-c1-s1. 106' (ie, C) may be sent to gateway 120-1 via a second direct connection 180-gw1-s1. By providing the order marked message 106' (i.e., C) to the sender of the message 106, the sender, i.e., the gateway 120-1, determines the sequence number assigned to the message, i.e., the message 106 (FIG. 1E). may be correlated to other identifying information in the message (as described below in connection with the Message). Thereby, the sender can easily address subsequent messages that reference that sequence number, as further disclosed below with respect to FIG. 1D.

The gateway 120-1, the core compute node 140-1, the sequencer 150-1, the first direct connection 180-1-1, the second direct connection 180-gw1-s1, and the third direct connection 180-c1-s1 are , configures a point-to-point mesh system 102. According to an exemplary embodiment, the point-to-point mesh system 102 includes a first direct connection 180-1-1, a second direct connection 180-gw1-s1, and a third direct connection 180-c1-s1, or one of them. The sets of units may be protected by at least one respective redundant direct connection (not shown). If this direct connection fails, a respective redundant direct connection can be adopted in its place. As such, electronic trading system 100 may further include at least one respective redundant direct connection to the first, second and third direct connections or some set thereof.

According to example embodiments, electronic trading system 100 may further include a clock, and gateway 120-1, core compute node 140-1, and sequencer 150-1 clock the clock of FIG. 1D, as further disclosed below. 195.

Message 106 may be referred to as a "gateway" message because it comes from a gateway, gateway 120-1. Message 106 may also be referred to as a "directed to compute node" message because it is addressed to a compute node, ie, core compute node 140-1, in the exemplary embodiment. Message 106 is a gateway message sent by gateway 120-1 in response to receipt of incoming message 103 (ie, A) received by gateway 120-1 from a participant device (not shown). Order marked message 106' may be a first order marked message. Core compute node 140-1 further, in response to receiving message 106, a gateway message, sends a core compute node message, response 107 (i.e., D) to gateway 120-1 through a first direct connection 180. -1-1 and core compute node messages (ie, responses 107) to sequencer 150-1 via a third direct connection 180-c1-s1. Sequencer 150-1 further connects gateway 120-1 with a second direct connection 180-gw1-s1 in response to a second order-marked message, namely, order-marked response 107' (i.e., E). configured to send via. The second order marked message is an ordered version of the core compute node message. Gateway 120-1 further determines the relative order-marked versions of the second order-marked message (i.e., order-marked response 107') and other messages sent from core compute node 140-1 to gateway 120-1. The system may be configured to determine a ranking. Gateway 120-1 may be further configured to send outgoing message 105 (ie, F) to the participant device in response to the determined relative ranking. It should be understood that messages 106 and responses 107 relate to trading activity.

Sequencer 150-1 further transmits a second order-marked message, namely, order-marked response 107' (i.e., E) to core compute node 140-1 via third direct connection 180-c1-s1. It is possible. By providing an order-marked response 107' (i.e., E) to the sender of the response 107, the sender, i.e., core compute node 140-1, assigns the ordinal number assigned to the message, i.e., the response 107 ( It can be correlated to other identifying information in the message (as described below in connection with FIG. 1E). Thereby, the sender can easily address subsequent messages that reference that sequence number, as further disclosed below with respect to FIG. 1D.

Similar to the above in connection with messages 106 and 106' received by compute node 140-1, upon receipt of unordered response message 107, gateway 120-1 determines that gateway 120-1 receives ordered response 107. The processing of the response message may be activated even before the response message is received. By way of non-limiting example, activating (initiating) the process may include updating the state of the gateway 120-1's open trade order database and/or sending an outgoing message to the participant device. 105. However, in some embodiments, the gateway 120-1 sends an order-marked response message that includes an order identifier that specifies a definitive position of the response message 107 in a series of messages that include other messages in the electronic trading system 100. 107' is received by gateway 120-1 before processing of order marked response message 107, which includes sending outgoing message 105 to the participant device, may not be complete. In some embodiments, after receiving both the unordered message 107 and the ordered response message 107', the gateway 120-1 may include the identification information in both versions of the message, as described below in connection with FIG. 1E. The unordered response message 107 may be correlated to the ordered response message 107' via. Thereby, the definitive position of the response message 107/107' is determined in response to the receipt of the order-marked response message 107'. In some embodiments, processing of the response message may then be completed, including ensuring that the outgoing message 105 is sent to the participant device.

Electronic trading system 100 may further include an order book (not shown) accessible by core compute node 140-1. Core compute node 140-1 may be further configured to match trade orders for financial instruments (not shown) based on electronic trade matching functions performed. For example, core compute node 140-1 may be further configured to match trade orders for financial instruments using electronic trade matching functionality. Core compute node 140-1 may further be configured to maintain a balance of financial instruments (not shown) in the order book. Amount discrepancies in financial instruments may result from performing electronic transaction matching functions. The remaining balance may include amount discrepancies in financial instruments. It should be understood that balances can convey more information than quantities. For example, positions may be bullish and bearish (sides). According to an exemplary embodiment, the balance conveys both price and quantity.

Gateway 120-1 further serves at least one participant device (not shown) and, in response to receipt of incoming message 103 at gateway 120-1, forwards message 106 to sequencer 150-1 and core compute node 140-. 1. Incoming messages 103 are dispatched by at least one participant device. Sequencer 150-1 further includes marking the message with the unique order identifier or creating an indication of the received message, marking the indication with the unique order identifier, and transmitting the marked indication. may be configured to generate order-marked messages by. The marked indication may be an ordinal marked message.

According to an exemplary embodiment, gateway 120-1 may be a given gateway of a plurality of gateways, such as the plurality of gateways of FIGS. 1C, ID, 3, and 4, as further disclosed below. Also, core compute node 140-1 may be a given core compute node of a plurality of core compute nodes, such as the plurality of core compute nodes of FIGS. 1C, ID, 3, and 4, as further disclosed below.

FIG. 1C is a block diagram of an exemplary embodiment of point-to-point mesh system 122. Point-to-point mesh system 122 includes multiple gateways 120, multiple core compute nodes 140, and sequencer 150-1. Each gateway 120-1, 120-2...120-g of the plurality of gateways 120 provides a respective 1 direct connection, ie, a first direct connection 180-a. The sequencer 150-1 is coupled to each gateway of the plurality of gateways via a respective second direct connection, ie, the second direct connection 180-b, and is coupled to each of the plurality of core compute nodes via a respective second direct connection 180-b. It is coupled via a third direct connection, ie third direct connection 180-c. The plurality of gateways, the plurality of core compute nodes, the sequencer 150-1, and their respective direct connections form at least a portion of the point-to-point mesh system 122.

Within the point-to-point mesh system 122, each gateway of the plurality of gateways 120 sends messages destined for the respective compute nodes sent therefrom to all of the compute nodes of the plurality of core compute nodes 140 and the sequencer 150-1. configured to do so. Within the point-to-point mesh system 122, each core compute node of the plurality of core compute nodes 140 sends messages destined for the respective gateways sent therefrom to all gateways of the plurality of gateways 120 and to the sequencer 150-1. configured to do so. Within the point-to-point mesh system 122, the sequencer 150-1 further transmits each order-marked message to the plurality of gateways 120 and the plurality of core compute nodes in response to receiving a message destined for the respective compute node or a message destined for the respective gateway. and configured to transmit to node 140.

Each compute node-directed message sent by a given gateway of the plurality of gateways 120 is the same message received by the plurality of core compute nodes 140. At least two compute nodes 140 of the plurality of core compute nodes 140 may be configured to generate a response message in response to receiving the same message. The response message may be received at a given core compute node from among the multiple core compute nodes. A given gateway may be further configured to take action based on a given response message of the response messages generated in response to receipt of the same message. A given response message may arrive at a given gateway first relative to other response messages generated in response to receipt of the same message. A given gateway may further be configured to ignore other response messages that arrive after a given response message. Such response messages may also be referred to as functionally equivalent messages. Functionally equivalent messages represent the same message that causes a recipient of the functionally equivalent message to perform the same function (eg, activity or activities). Functionally equivalent messages represent the same message, and in some embodiments may actually be identical; in other embodiments, functionally equivalent messages may, as a non-limiting example, They do not necessarily have to be the same, as they may contain different metadata such as different timestamps, different source identifiers, etc. In the interest of clarity, the term "functionally equivalent message" does not refer to an out-of-order marked message that is related to its ordinal-marked message counterpart. Rather, functionally equivalent messages generally refer to multiple ordered marked messages representing the same out-of-order marked message, although in some embodiments there may also be multiple ordered marked messages representing the same ordered marked message. by which we may refer to functionally equivalent out-of-order marked messages or functionally equivalent in-order marked messages. The discussion below with respect to FIG. 1E describes at least one example of how a recipient node may determine that two or more such response messages are functionally equivalent and represent the same message. do.

Multiple functionally equivalent messages (not shown) may be received at a given gateway from among multiple core compute nodes 140, multiple sequencers 150, or a combination thereof. The given gateway may be further configured to take action based on the given functionally equivalent message of the plurality of functionally equivalent messages, and the given functionally equivalent message is the first to arrive at a given gateway. A given gateway may be further configured to ignore other functionally equivalent messages of the plurality of functionally equivalent messages that arrive after the given functionally equivalent message. Such messages may be understood as "functionally" equivalent. This is because, for the same message received at multiple core compute nodes, each of the multiple core compute nodes independently generates a response message, so each response may have the same result, even if not exactly the same. This is because it provides functionality. For example, such a response message may at least have a different originating core identifier included therein to uniquely identify the particular core compute node sending the response.

According to an example embodiment, the at least two "functionally equivalent messages" may arrive from the gateway/sequencer to a given core compute node, such as core compute node 140-1. A given core compute node may be configured to process only the first of such functionally equivalent messages to arrive. In this way, multiple functionally equivalent messages may be received from multiple gateways, multiple sequencers, or a combination thereof at a given compute node. A given compute node may be further configured to take an action based on a given functionally equivalent message of a plurality of functionally equivalent messages, The message is the first one to arrive at a given compute node. A given compute node may be further configured to ignore other functionally equivalent messages of the plurality of functionally equivalent messages that arrive after the given functionally equivalent message.

In this manner, messages destined for multiple compute nodes representing the same message may be received from among multiple gateways 120 at a given compute node, such as compute node 140-1. This may be the case if electronic trading system 100 is configured for high availability (HA). For example, redundant flows from participant devices may be received between multiple gateways, and thus messages destined for multiple compute nodes representing the same message may be sent by that gateway to compute node 140. According to example embodiments, a given core compute node may be configured to take action based on a given compute node-directed message of a plurality of compute node-directed messages; may be the first of multiple compute node messages representing the same message to arrive at a given compute node relative to messages addressed to other compute nodes. A given core compute node may be further configured to ignore messages destined for other compute nodes that arrive after messages destined for the given compute node. Messages addressed to multiple compute nodes that represent the same message may be called functionally equivalent messages. The discussion below with respect to FIG. 1E provides at least one example of how a recipient node may determine that messages addressed to two or more such compute nodes are functionally equivalent and represent the same message. Explain.

Electronic trading system 100 includes an active electronic trading system in which at least one sequencer of the plurality of sequencers may be communicatively coupled to a disaster recovery site including a standby electronic trading system, such as disaster recovery site 155 of FIG. 1D, disclosed below. It can be a trading system.

FIG. ID is a block diagram of another example embodiment of an electronic trading system. FIG. 1D shows a number of gateways 120-1, 120-2, ..., 120-g (collectively referred to as gateways 120), a set of core compute nodes 140-1, 140-2, ..., 140- 150-s (collectively core compute node 140 or compute node 140) and one or more sequencers 150-1, 150-2, ..., 150-s (collectively sequencer 150). Accordingly, in some embodiments, gateway 120, core compute node 140, and sequencer 150 are considered nodes in electronic trading system 100. As described in more detail below, in one embodiment, gateway 120, compute node 140, and sequencer 150 are directly connected to each other, preferably via a dedicated low-latency connection 180.

The term "peer" in the context of the description of electronic trading system 100 refers to other devices that generally serve the same functionality in electronic trading system 100 (e.g., "gateway" vs. "core compute node" vs. "sequencer"). ”). For example, gateways 120-2, ..., 120-g are peers of gateway 120-1, core compute nodes 140-2, ..., 140-c are peers of core compute node 140-1, Sequencers 150-2, . . . , 150-s are peers of sequencer 150-1.

The terms "active" and "standby" in connection with the description of system 100 may refer to the high availability (HA) role/state/mode of the system/component. Generally, a standby system/component is a redundant (backup) system/component that is capable of taking over the functions performed by the active system/component when powered on. Such switchover/failover, i.e. switching from a standby role/state/mode to an active role/state/mode, may occur automatically in response to, by way of non-limiting example, failure of the currently active system/component. can be executed.

Electronic trading system 100 processes trading orders from one or more participant computing devices 130-1, 130-2, ..., 130-p (collectively participant devices 130) and provides related information thereto. do. Participant device 130 interacts with electronic trading system 100 and may be one or more personal computers, tablets, smartphones, servers, or other data processing devices configured to display and receive trading order information. obtain. Participant device 130 may be operated by a human via a graphical user interface (GUI) or via a high-speed automated trading method running on a physical or virtual data processing platform. Each participant device 130 may exchange messages with (ie, send and receive messages to) electronic trading system 100 via the connection established by gateway 120. Although FIG. 1D depicts each participant device 130 as connected to electronic trading system 100 via a single connection to gateway 120, participant device 130 may have multiple connections to one or more gateway devices 120. It should be understood that the electronic trading system 100 may be connected to the electronic trading system 100 via the electronic trading system 100.

Note that each gateway 120-1 may serve a single participant device 130, but typically serves multiple participant devices 130.

Compute nodes 140-1, 140-2, ..., 140-c (also referred to herein as matching engine 140 or compute engine 140) provide the matching functionality described above and further distribute to one or more participant devices 130. may generate an outgoing message to be sent. Each compute node 140 is a high-performance data processor and typically maintains one or more data structures that retrieve and maintain one or more order books 145-1, 145-2, ..., 145-b. do. Order book 145-1 may be maintained, for example, for each product for which core compute node 140-1 is responsible. One or more of the compute nodes 140 and/or one or more of the gateways 120 may provide the market data feed 147. Market data feed 147 may be broadcast (eg, multicast) to participants, which may be participant devices 130 or any other suitable computing devices.

Some outgoing messages generated by core compute nodes 140 are synchronous, i.e., one or more participating messages, such as an outgoing "approval message" or "execution message" in response to a corresponding incoming "new order" message. may be generated directly by one core compute node 140 in response to one or more incoming messages received from user device 130. However, in some embodiments, at least some outgoing messages may be asynchronous and may be initiated by trading system 100, such as certain "unsolicited" cancellation messages and "abort" or "transaction" messages. It could be a "bankrupt" message.

A distributed computing environment, such as electronic trading system 100, can be configured with multiple matching engines running in parallel on multiple compute nodes 140.

Sequencer 150 ensures that the proper order of any rank-dependent operations is maintained. To ensure that actions on incoming messages are not performed out of order, incoming messages received at one or more gateways 120, e.g., a new trade order message from one of participant devices 130, are typically , then pass through at least one sequencer 150 (e.g., a single currently active sequencer, and possibly one or more standby sequencers), and the incoming message will pass through at least one sequencer (e.g., a single currently active sequencer, and possibly one or more standby sequencers) (by one currently active sequencer) with a sequence identifier. The identifier is used throughout the distributed system 100 (e.g., electronic trading system 100) during subsequent processing to determine relative rankings among the messages and to uniquely identify the messages throughout the electronic trading system 100. can be a unique monotonically increasing value used in In some embodiments, the order identifier may be an indication of the order (ie, order) in which the messages arrived at the sequencer. For example, the order identifier may be a value that is monotonically incremented or decremented by the sequencer according to a fixed interval for each arriving message; for example, the order identifier may be incremented by one for each arriving message. However, it should be understood that ordinal identifiers, while unique, are not limited to monotonically increasing or decreasing values. In some embodiments, the original unmarked message and the ordinal marked message may be essentially identical except for the value of the ordinal identifier included in the marked version of the message. Once an incoming marked message, i.e., an order-marked message, is ordered, it is typically then forwarded by the sequencer 150 to other downstream compute nodes 140 to perform potentially order-dependent processing on the message. be done. Therefore, in addition to uniquely identifying messages throughout electronic trading system 100, the ordinal identifier assigned by sequencer 150 also determines the relative ranking of each marked message among other marked messages in electronic trading system 100. can determine the attachment.

Thus, in contrast to other purposes for which ordinal identifiers may be employed, the unique ordinal identifiers disclosed herein are intended to ensure a definitive ranking (i.e., order) for electronic transaction message processing. can be used for. The unique ordering identifier represents a unique, deterministic ranking (i.e., order) that dictates the processing of a given electronic trading message relative to other trading messages within the electronic trading system. According to an example embodiment, the order identifier may be added to the order ID field 110-14 of the message, as further disclosed below with respect to FIG. 1E as a non-limiting example.

In some embodiments, messages may flow in the other direction, ie, from the core compute node 140 to one or more of the participant devices 130, through one or more of the gateways 120. Outgoing messages generated by such one core compute node 140 may also be rank-dependent (i.e., order-rank-dependent) and are therefore also typically marked with a sequencer 150 with an order identifier. can be passed first. Sequencer 150 may then forward the marked response messages to gateway 120 for communication to participant devices 130 in an appropriately deterministic order.

The use of sequencer 150 to generate unique sequence numbers and thereby mark messages or representations thereof, i.e., to generate sequence marked messages, ensures that the correct ranking of operations is possible in distributed systems, i.e., electronic transactions. The compute node or set of compute nodes 140 processes messages regardless of whether they are maintained throughout the system 100. This approach provides "state determinism", e.g. the entire state of the system is deterministic and reproducible (possibly somewhere else, such as a disaster recovery site), provides fault tolerance, high availability and provide disaster recovery potential.

A generating node (i.e., a node that introduces new messages into electronic trading system 100, e.g., by generating new messages and/or forwarding messages received from participant device 130) and its peer nodes: It may also be important to receive the sequence number assigned to the message. Receiving sequence numbers for the messages it generates is used not only to process the messages in order according to their sequence numbers, but also to process the messages generated by the node throughout the rest of the electronic trading system 100. It may also be useful for producing nodes and their peer nodes to correlate message order identifiers. Such a correlation between an unmarked version of a message as introduced into an electronic trading system by a generating node and an ordinal marked version of the same message output by a sequencer is discussed further below in connection with FIG. 1E. This can be done through identification information in both versions of the message. Subsequent messages generated within electronic trading system 100 may also be assigned their own sequence numbers, but may still be referenced to one or more sequence numbers of associated preceding messages. Therefore, a node may need to quickly reference (by sequence number) messages that it has previously generated. This is because, for example, the sequence number of a message generated by a node is used as a reference in subsequent messages.

In some embodiments, the producing node first sends the message to the sequencer 150 and receives a sequence number for the message from the sequencer before the producing node forwards the message to other nodes in the electronic trading system 100. You may wait.

In an alternative exemplary embodiment, after receiving an unordered message from a production node, sequencer 150 generates an ordered version of the message to avoid at least one hop that may add unnecessary latency increases within electronic trading system 100. (i.e., an order-marked message) to a destination node, an ordered version of the message may be sent back to the sending node and its peers substantially simultaneously. For example, after assigning a sequence number to an incoming message sent from gateway 120 to core compute node 140, sequencer 150 not only forwards the ordered version of the message to core compute node 140, but also forwards the ordered version of the message to core compute node 140. 120-1 and other gateways 120. Therefore, if any subsequent message generated at core compute node 140 is based on that sequence number, then any gateway 120 can facilitate the associated message originally generated by gateway 120-1 by that sequence number. can be identified.

Similarly, in some further embodiments, the ordered version of the outgoing message generated by the core compute node 140 and sent therefrom to the gateway 120 and ordered by the sequencer 150 is forwarded by the sequencer 150 to the gateway 120. It may also be sent back to the core compute node 140 along with the data.

Some embodiments provide for high availability, e.g., to ensure that other sequencers are available if a first sequencer fails, as further disclosed below with respect to FIG. A plurality of sequencers 150 may be included. For embodiments having multiple sequencers 150 (eg, a currently active sequencer 150-1 and one or more standby sequencers 150-2, . . . , 150-s), the currently active sequencer 150-1 is the sequencer 150-s. A system state log (not shown) of all messages that pass through one and the messages' associated sequence numbers may be maintained. This system state log may be sent continuously or periodically to the standby sequencer to provide them with the essential system state and enable them to take over for the active sequencer if necessary. Alternatively, the system state log may be stored in a data store that is accessible to multiple sequencers 150.

The system state log may be continuously or periodically replicated to one or more sequencers in a standby replicated electronic trading system (not shown in detail) at disaster recovery site 155, thereby providing a primary In the event of a catastrophic failure of the site, electronic transactions can continue in exactly the same condition at the disaster recovery site 155.

According to an example embodiment, a currently active sequencer of the plurality of sequencers may store a system state log in a data store (not shown). A data store may be accessible to multiple sequencers via a shared sequencer network, such as sequencer-wide shared network 182-s, further disclosed below with respect to FIG. 1D. When a given sequencer among the sequencers switches its role (state) from standby to active, that sequencer retrieves the system state log from the data store to synchronize its state with that of the previous active sequencer. obtain.

In some embodiments, the system state log may be provided to a drop copy service 152 that may be implemented by one or more of the sequencers and/or by one or more other nodes in the electronic trading system 100. Drop copy service 152 may, for example, provide a record of daily trading activity through electronic trading system 100 that may be distributed to regulators and/or clients that may be connected via participant device 130. In alternative embodiments, drop copy service 152 may be implemented at one or more of gateways 120. Still further, in addition to or in lieu of referencing system status logs, drop copy service 152 provides a record of trading activity based on the content of incoming and outgoing messages sent throughout electronic trading system 100. You can. For example, in some embodiments, the gateway 120 that implements the drop copy service 152 transfers all messages exchanged throughout the electronic trading system 100 from the sequencer 150 (and/or to the core compute nodes 140 and other gateways 120). ) can be received. Participant devices 130 configured to receive records of daily trading activity from drop copy service 152 do not necessarily send trading orders to or utilize the matching functionality of electronic trading system 100. good.

Messages exchanged between participant devices 130 and gateway 120 may follow any suitable protocol that may be used for financial transactions (referred to for convenience as a "financial transaction protocol"). For example, messages may be exchanged according to custom protocols or established standard protocols, including both binary protocols (such as Nasdaq OUCH and NYSE UTP) and text-based protocols (such as NYSE FIX CCG). In some embodiments, electronic trading system 100 may support exchanging messages simultaneously according to multiple financial transaction protocols, including multiple protocols simultaneously at the same gateway 120. For example, participant devices 130-1, 130-2, and 130-3 may simultaneously establish trading connections or send messages to gateway 120-1 in accordance with Nasdaq Ouch, NYSE UTP, and NYSE FIX CCG, respectively. Sometimes they are exchanged.

Still further, in some embodiments, gateway 120 converts messages received from participant devices 130 that conform to financial transaction protocols to standardization used to exchange messages between nodes within electronic trading system 100 (e.g., (standardized) message format. The standardized transaction format may be an existing protocol and is typically of a different size and data format than any financial transaction protocol used to exchange messages with participant devices 130. Good too. For example, the standardized transaction format may optionally include one or more additional fields or parameters when compared to the financial transaction protocol of the original incoming message received at gateway 120 from participant device 130; Any of the above fields or parameters may be omitted and/or each field or parameter of a message in a standardized format may be of a different data type or size than the corresponding message received at gateway 120 from participant device 130. It's okay. Similarly, in the reverse direction, gateway 120 sends outgoing messages generated by electronic trading system 100 in a standardized format to the format of one or more financial transaction protocols used by participant devices 130 to communicate with gateway 120. It can be translated into a message.

FIG. 1E is a table of example embodiments of fields of a message format 110 for a trading message, such as a trading message exchanged between nodes in the electronic trading system 100 disclosed above. In the exemplary embodiment of FIG. 1E, message format 110 is configured for internal (i.e., within electronic trading system 100) display of trading messages as they are exchanged between nodes within electronic trading system 100. is a standardized message format intended for use in In this exemplary embodiment, gateway 120 exchanges messages between participant 130 and electronic trading system 100 and formats the messages as specified by one or more financial transaction protocols used by participant 130. , translate messages to and from standardized trading formats used between nodes in electronic trading system 100. It is noted that fields 110-1 through 110-17 are for non-limiting examples, that message format 110 may include more, fewer, or different fields, and that the order of such fields is as shown in the figure. It should be understood that the present invention is not limited to that shown in IE.

Although the fields in message format 110 are shown in a single message format in this example, they may be distributed across multiple message formats or encapsulated in layered protocols. For example, in other embodiments, some sets of fields in message format 110 are included as part of header, trailer, or extension fields in a layered protocol that encapsulates other fields of message format 110 into the message payload. Good too. According to some example embodiments, message format 110 may include other message formats, including, without limitation, the respective payload portions of IP datagrams, UDP datagrams, TCP packets, or by way of non-limiting example, A payload of a message data frame format, such as a data frame format, or other data frame formats, including InfiniBand, Universal Serial Bus (USB), PCI Express (PCI-e), and High-Definition Multimedia Interface (HDMI). (data) part may define one or more fields of data encapsulated in a field.

Message format 110 includes fields 110-1...110-6 that correspond to information that may be included in a message sent or received in accordance with a financial transaction protocol for communication with one or more participant devices 130. As a non-limiting example, message type field 110-1 indicates a transaction message type. Certain transaction message types (such as message types "New Order", "Replacement Order", or "Cancellation Order") correspond to messages received from participant device 130, while other message types (such as "New Order Approval", "Replacement Order") correspond to messages received from participant devices 130; ``Order Approval'', ``Cancellation Order Approval'', ``Execution'', ``Execution Report'', ``Unsolicited Cancellation'', ``Trade Failure'', or various rejection messages) are generated by the electronic trading system 100 and sent to the participant device 130. corresponds to the message contained in the transaction message sent.

Message format 110 also includes a stock symbol field 110-2 that contains an identifier for the traded security, such as a stock symbol or stock ticker. For example, "IBM" is the stock symbol for "International Business Machines Corporation." Side field 110-3 in message format 110 may be used to indicate the "side" of the trading message, such as whether the trading message is a "buy," "sell," or "short sell." Similarly, price field 110-4 may be used to indicate a desired price to buy or sell a security. Quantity field 110-5 may be used to indicate the desired quantity (eg, number of shares) of the security. Message format 110 may include an order token field 110-6, which is appended to an “order token” or “client order ID” originally provided by participant device 130 and sent via gateway 120. A new order may be uniquely identified in the context of a particular trading session (ie, "connection" or "flow") established between participant device 130 and the electronic trading system.

Although fields 110-1...110-6 are typical fields typically included for most message types following most financial transaction protocols, message format 110 may be particularly specific to a particular message type or a particular financial transaction protocol. It should be understood that additional or alternative fields may also be included to support. For example, according to many financial trading protocols, "replacement order" and "cancellation order" message types are used by participant 130 to represent a replaced or canceled order by supplying additional order tokens to It is necessary to distinguish it from the order of Similarly, "replacement order" and "cancellation order" typically also include a replacement/cancellation quantity field, and a "replacement order" may include a replacement price field. These additional replacement/cancellation order token fields, replacement price fields, and replacement/cancellation quantity fields may be included in the corresponding approval message sent by electronic trading system 100.

Additionally, message format 110 includes fields 110-11 . . . 110-17 that may be used internally within electronic trading system 100 but do not necessarily correspond to fields in messages exchanged with participant devices 130. For example, node identifier field 110-11 may uniquely identify each node in electronic trading system 100. In some embodiments, the producing node may include the identifier in messages that it introduces into the electronic trading system 100. For example, each gateway 120 may include its node identifier in messages it forwards from participant devices 130 to compute nodes 140 and/or sequencers 150. Similarly, each compute node 140 generates messages that are sent to other nodes in electronic trading system 100 (e.g., asynchronous messages that are intended to be ultimately transmitted to one or more participant devices 130) The node identifier may be included in the message (acceptance, execution, or type). Accordingly, each message introduced into the electronic trading system 100 may be associated with the message's generating node via the node identifier field 110-11 in the message.

Message format 110 may also include a flow identifier field 110-12. In some embodiments, each trading session (i.e., “connection” or “flow”) established between participant device 130 and gateway 120 may be unique throughout electronic trading system 100. May be identified by an intended flow identifier. As discussed above in connection with FIG. 1D, participant devices 130 may be connected to electronic trading system 100 via one or more flows and via one or more of gateways 120. In such embodiments, all messages exchanged between participant devices 130 and electronic trading system 100 via a particular flow are configured according to standardized message format 110 (used between nodes in electronic trading system 100). The version of the message will include a unique identifier for the flow in flow identifier field 110-12. In some embodiments, the flow identifier field 110-12 is added by the generating node of the message. For example, gateway 120 may add to flow identifier field 110-12 the identifier of a flow associated with a message it receives from participant 130 that gateway 120 introduces into electronic trading system 100. Similarly, core compute node 140 adds a flow identifier associated with a message it generates (i.e., an acknowledgment message or other outgoing message, including response messages such as fills or asynchronous messages) to flow identifier field 110-12. Good too.

In some embodiments, the flow identifier field 110-12 includes a value that uniquely identifies a logical flow that is highly available in practice, as multiple redundant trading session connections, and possibly through multiple gateways. may be implemented for the purpose of That is, in some embodiments, the same flow ID may be assigned to two or more redundant flows between participant device 130 and gateway 120. In such embodiments, redundant flows may be in either an active/standby configuration or an active/active configuration. In an active/active configuration, functionally equivalent messages may be exchanged in parallel between participant devices 130 and gateway 120 via multiple redundant flows at the same time. That is, a trading client simultaneously sends functionally equivalent messages to the electronic trading system 100 via multiple redundant flows in parallel, and sends multiple functionally equivalent responses from the electronic trading system 100 via multiple redundant flows. can be received in parallel via. However, electronic trading system 100 may only take action on a single such functionally equivalent message. In an active/standby configuration, a single flow among the multiple redundant flows at a time may be designated as the active flow, while another flow among the multiple redundant flows may be designated as the standby flow. Transaction messages will actually only be exchanged via currently active flows. Messages exchanged over any of the redundant flows, regardless of whether the redundant flows are configured in an active/active or active/standby configuration, are configured in the flow identifier field 110- of the standardized message format 110. 12 by the same flow identifier stored by the message's generating node.

As mentioned above, in some embodiments, messages exchanged between nodes in electronic trading system 100 are sent to sequencer 150 and marked with an ordering identifier. Accordingly, message format 110 includes an order identifier field 110-14. In some embodiments, an "unmarked message" may be sent with an empty blank (eg, zero) value sequence identifier field 110-14. In other embodiments, the order identifier field 110-14 of unmarked messages may be set to some predetermined value that the sequencer does not assign to messages, or an invalid value. Still other embodiments may specify that a message is not marked via an indicator in another field (not shown) of the message, such as a Boolean or flag value that indicates whether the message is ordered or not. . Upon receiving an unmarked message, sequencer 150 may then generate an "order marked message" by adding a valid order identifier value to the order identifier field 110-14 of the unmarked message. A valid order identifier value in the order identifier field 110-14 of an order marked message uniquely identifies the message and also determines the relative rank of the marked message among other marked messages throughout electronic trading system 100. Specifies the definitive position of a marked message in the marking. In this example, an "order marked message" sent by sequencer 150 is subsequently received by the sequencer, except that the order marked message's order identifier field 110-14 contains a valid order identifier value. can be identical to the corresponding unmarked message.

Message format 110 may also include a reference order identifier field 110-15 in some embodiments. The producing node may add the value of the order number of the previous message associated with the message being produced to the base order identifier field 110-15 of the new message it produces. The values of the reference order identifier fields 110-15 allow nodes in the electronic trading system 100 to correlate messages with previous related messages.

The previous related message referenced in the reference order identifier field 110-15 may be a previous message in the same "order chain" (ie, "trade order chain"). According to most financial trading protocols, messages may be logically grouped into "order chains," which are sets of messages through a single flow that refer to or "derive from" a common message. The order chain typically begins with a "new order message" sent by participant device 130. The next message in the order chain is typically a response by the electronic trading system (e.g., a "new order accepted" message if the message is accepted by the trading system, or perhaps an invalid price, etc., as non-limiting examples) (either a "New Order Rejected" message if the message is rejected by the trading system because it has an invalid format or invalid parameters). The order chain cancels at least a portion of the quantity of the new order that was previously approved (but includes at least some quantity that is still outstanding, i.e., not canceled and/or filled). It may also include "cancel order" messages sent by. The "Cancel Order" message will again be approved or rejected by the electronic trading system with a "Cancel Order Approved" or "Cancel Order Rejected" message, which may also be part of the order chain. The order chain may also include a "replacement order" message sent from participant device 130 that replaces the quantity and/or price of a previously approved (but still outstanding) new order. The "Replacement Order" message will again be approved or rejected by the electronic trading system by a "Replacement Order Approved" or "Replacement Order Rejected" message, which may also be part of the order chain. A previously confirmed order that is still outstanding may be matched with one or more counter orders on the opposite side (ie, a "buy" on one side and a "sell" or "short" on the other side). The electronic trading system 100 then sends either a complete "fill" message (if all of the outstanding order quantity is filled in a single match) or one or more partial "fill" messages (if all of the outstanding order quantity is filled in a single match). If only a portion of the quantity is filled in a single match), these "fill" messages may also be part of the order chain. As mentioned above, the reference order identifier may generally identify other previous messages in the same order chain.

For example, returning to the base sequence identifier field 110-15, the value for the base sequence number is assigned by the sequencer for an "incoming" message originating from the participant device 130 and introduced into the electronic trading system 100 by the gateway 120. A corresponding "outgoing" message, such as a response message generated by compute node 140, may refer to the sequence number value of the incoming message to which it is responding. In this example, a "new order approval" message or "fill" message generated by compute node 140 is a response in which compute node 140 responds with a "new order approval" message or fulfills an order with a "fill" message. The reference order identifier field 110-15 will contain the value for the order identifier assigned to the "New Order" message. However, in general, the value for the base order identifier field 110-15 need not necessarily be that of a message being responded to directly by electronic trading system 100, but that of a previous message that is part of the same order chain, e.g. , can be a sequence number for "New Order" or "New Order Approval".

In some embodiments, for at least some message types, the gateway 120 adds the value of the sequence identifier for the associated previous message to the base sequence identifier field 110-15 in messages that they introduce into the electronic trading system 100. It's okay. For example, the gateway 120 populates the base order identifier field 110-15 in a "cancellation order" or "replacement order" message with the value of the order identifier assigned to the previous corresponding "new order" or "new order approval" message. May be added. Similarly, the core compute node 140 indicates that the order identifier field 110-15 for a corresponding "cancellation order approval" or "replacement order approval" message is not of a message to which the compute node 140 is directly responding (e.g., Also add the value of the order identifier for "new order" or "new order approval" (rather than the order identifier for "cancel order" or "replace order" message). Again, the reference order identifier fields 110-15 allow nodes in the electronic trading system 100 to generally correlate a message to one or more previous messages in the same order chain.

The producing node may include a node-specific timestamp field 110-13 in messages that it introduces into the electronic trading system 100. Although the order identifier contained in the order identifier field 110-14 of the order marked message output by the sequencer 150 is intended to be unique throughout the electronic trading system 100, the order identifier in the node-specific timestamp field 110-13 The value may be unique among some set of messages introduced into electronic trading system 100 by a particular producing node. Although referred to herein as a "timestamp," the value contained in the node-specific timestamp field 110-13 may be any suitable value that is unique among messages generated by that node. For example, a node-specific timestamp may actually be a timestamp or any suitable monotonically increasing or decreasing value.

Some embodiments may include other timestamp fields in the message format. For example, some message formats may include a reference timestamp field, which may be a timestamp value assigned by the generating node of a previous associated message. In such embodiments, the compute node 140 may include a new timestamp value in the node-specific timestamp field 110-13 for messages it generates, and the reference timestamp field for messages it generates. may include timestamp values from associated messages at. For example, a "new order approval" message generated by a compute node may include a "new order" timestamp value to which it responds in the "new order approval message" reference timestamp field. Still further, in some embodiments, compute nodes 140 may not include new timestamp values in node-specific timestamp fields 110-13 in messages they generate, but simply The timestamp value from the previous related message may be added to 110-13.

Message format 110 may also include an entity type field 110-16 and an entity count field 110-17. The entity type of a message determines whether it is introduced into electronic trading system 100 by gateway 120 or by compute node 140; in other words, an incoming message where the message is received at gateway 120 from participant device 130. or whether it is an outgoing message being generated by compute node 140 that is sent to participant device 130. For example, in some embodiments, the incoming message is considered to be of entity type "Flow" (and the entity type field 110-16 has a value added to the incoming message by the gateway 120 representing the type "Flow"); , the outgoing message is considered to be of entity type "Stock Symbol" (and the entity type field 110-16 has a value added to the outgoing message by the compute node 140 representing the type "Stock Symbol"). In such an embodiment, an entity count of type "flow" is maintained by gateway 120 and an entity count of type "stock symbol" is maintained by compute node 140.

Considering the entity type "Flow", the gateway 120 maintains an incoming message count per flow, counting the incoming messages received by the gateway 120 via each flow active on the gateway 120. For example, if four non-redundant flows are active on gateway 120, each flow would be assigned a unique flow identifier as described above, and gateway 120 would maintain an incoming message count per flow and The number of incoming messages received through each of the book flows will be counted. In such embodiments, the gateway 120 enters the Entity Count field 110-17 of the incoming message (identified throughout the electronic trading system 100 by the flow identifier value added to the message's Flow Identifier field 110-12). Add the incoming message count per flow associated with the flow of incoming messages (such as).

Active/active configuration (i.e., a configuration in which multiple flows receive the same message, or at least functionally equivalent messages, in parallel from participant devices 130 connected via one or more gateways 120, as described above) In the case of redundant flows in Can be kept separate from the flow. Participant devices 130 are expected to send the same set of messages in the same rank (i.e., order) to electronic trading system 100 via each of the redundant flows and thus are received via separate redundant flows. It is also expected that the entity counts assigned to functionally equivalent messages should be the same. These functionally equivalent incoming messages may be forwarded by gateway 120 to sequencer 150 and compute node 140. Thus, in such embodiments, sequencer 150 and compute node 140 may receive multiple functionally equivalent incoming messages associated with the same flow identifier, but the entity If the counts are the same, sequencer 150 and compute node 140 may identify such messages as being functionally equivalent. In some embodiments, sequencer 150 and compute node 140 may track, for each flow, the highest entity count that was included in entity count field 110-17 of an incoming message associated with each flow. This allows sequencer 150 and compute nodes 140 to take action only on the first of multiple functionally equivalent incoming messages received by each node, and to act on other functionally equivalent incoming messages that arrive later. You can ignore the message. For example, sequencer 150 may, in some embodiments, only order such functionally equivalent incoming messages that arrive first, and compute node 140 may order only such functionally equivalent incoming messages that arrive first. Processing may only begin if the If an incoming message received by a node (i.e., sequencer 150 or compute node 140) has an entity count that is less than or equal to the highest entity count that the node knows about for that flow, then that node A message can be presumed to be functionally equivalent to other previously received incoming messages, and later received functionally equivalent incoming messages may simply be ignored.

Considering now the case of the entity type "Stock Symbol", the Compute node 140 maintains an outgoing message count per stock symbol that is generated and sent by the Compute node 140 for each stock symbol handled by the Compute node 140. may count outgoing messages sent from. For example, if four stock symbols (e.g., MSFT, GOOG, IBM, ORCL) are handled by the compute node 140, each stock symbol is added to the stock symbol field 110-2 of the message, as described above. Assigned to a symbol identifier, the compute node 140 will maintain an outgoing message count per stock symbol, counting the number of outgoing messages it has generated and sent while handling each of these four stock symbols. In such embodiments, the compute node 140 determines the stock symbol associated with the stock symbol of the outgoing message (as identified throughout the electronic trading system 100 by the value added to the message's stock symbol field 110-2). Adds the outgoing message count to the incoming message's entity count field 110-17.

In some embodiments, as described further below, the compute nodes may be configured such that multiple compute nodes handle a particular stock symbol in parallel for high availability reasons. Because of the deterministic ranking of messages throughout electronic trading system 100 provided by sequencer 150, even if multiple compute nodes are handling a given stock symbol, they will rank the same stock symbol in the same order (i.e. , order), thereby ensuring that functionally equivalent response messages are generated in parallel. Considering outgoing messages sent for a particular stock symbol across multiple compute nodes 140, each outgoing message that references that stock symbol is sent by each other compute node 140 that actively handles that stock symbol. should have functionally equivalent messages. All of these outgoing messages may be sent by compute node 140 to sequencer 150 and gateway 120. Thus, in such embodiments, sequencer 150 and gateway 120 may receive multiple functionally equivalent incoming messages related to the same stock symbol, but sequencer 150 and gateway 120 may receive multiple functionally equivalent incoming messages that have the same stock symbol identifier. If the entity count is the same for multiple messages, the messages may be identified as functionally equivalent. In some embodiments, sequencer 150 and gateway 120 may track, for each stock symbol, the highest entity count that was included in the entity count field 110-17 of the outgoing message associated with the stock symbol. This allows sequencer 150 and gateway 120 to take action only on the first of multiple functionally equivalent outgoing messages received by each node, and to act on other functionally equivalent outgoing messages that arrive later. can be ignored. For example, sequencer 150 may order only such functionally equivalent outgoing messages that arrive first in some embodiments. Similarly, gateway 120 may begin processing only the first such functionally equivalent message that arrives. If an outgoing message received by a node (i.e., sequencer 150 or gateway 120) has an entity count less than or equal to the highest entity count previously known to that stock symbol, that node , an outgoing message can be presumed to be functionally equivalent to other previously received outgoing messages, and later received functionally equivalent outgoing messages may simply be ignored.

In embodiments where sequencer 150 orders only the first of multiple functionally equivalent messages arriving at the sequencer, the sequencer can do so in a variety of ways. In one example, other subsequent arriving messages that are functionally equivalent to the first arriving functionally equivalent message may simply be ignored by the sequencer (in this case, for a set of functionally equivalent messages) only a single ordered marked message may be output by the sequencer). Another possibility is for the sequencer to make an association between, for example, a message's entity count, its flow identifier or stock symbol identifier (for messages with entity types of "flow" and "stock symbol" respectively), and its sequence number. to keep track of the sequence number assigned to the first functionally equivalent message. Thereby, the sequencer can output an ordered version of each functionally equivalent message, and in that ordered version of each functionally equivalent message, an ordered version of all functionally equivalent messages can be output. The value of the order identifier field 110-14 is the same as that assigned by the sequencer 150 to the first arriving message among the functionally equivalent messages received by the sequencer 150.

In other embodiments, sequencer 150 may not track whether a message is functionally equivalent, and whether the message is one of a plurality of functionally equivalent messages. Regardless, each unordered message arriving at sequencer 150 may be assigned a unique sequence number. In such embodiments, each of the order marked versions of the plurality of functionally equivalent messages is assigned a different order identifier by the sequencer as a value in the order identifier field 110-14. To determine a valid ordering identifier for a set of functionally equivalent messages, a recipient node of an ordered functionally equivalent message in such embodiments determines the ordering identifier of the ordered functionally equivalent message that arrives at the node first. An order identifier in an ordered version of a functionally equivalent message may be used. In embodiments where there are direct point-to-point connections between nodes in electronic trading system 100, the order-marked versions of the messages are sent out in an ordered order by sequencer 150 so that all nodes directly connected to the sequencer should be received in the same ordering order. Therefore, for every node that receives an order-marked message via its respective direct point-to-point connection with a sequencer, the order-marked message that arrives first of multiple functionally equivalent order-marked messages is , should have the same value in the order identifier field 110-14.

As will be apparent from the above, in addition to message order identifiers, in embodiments with message formats such as message format 110, there are numerous other aspects that uniquely identify messages throughout electronic trading system 100. It is possible. For example, in embodiments where a message includes both a node identifier and a node-specific timestamp, the presence of these two identifiers in the message may be sufficient to uniquely identify the message throughout electronic trading system 100. Such fields can be understood as containing metadata. Multiple such messages containing the same metadata may be understood as containing common metadata. Similarly, in embodiments where flow identifiers are unique throughout electronic trading system 100, the combination of a message's flow identifier and node-specific timestamp is sufficient to uniquely identify the message throughout electronic trading system 100. It can be. Still further, the combination of flow identifier and entity count is sufficient to uniquely identify a message of entity type "Flow", and the combination of stock symbol identifier and entity count is sufficient to uniquely identify a message of entity type "stock symbol". This may be sufficient for identification.

However, in addition to the ordinal identifier assigned to a message, there may be other aspects that uniquely identify a message throughout electronic trading system 100 that are generated by other nodes throughout electronic trading system 100. It should be noted that order identifiers are still needed to specify the relative ranking of messages among other messages in a fair and deterministic manner. For example, if a node-specific timestamp is actually implemented as a timestamp value, even if the system clocks between the nodes are perfectly synchronized, each of two different messages each generated by a different node will They may be assigned the same timestamp value by the producing node, and the relative ranking between these two messages becomes ambiguous. Even if the messages were uniquely identifiable, the recipient nodes of both messages would still need a way to determine the relative ranking of the two messages before taking possible actions on the messages. .

One possible approach for the recipient node to resolve the ambiguity is by randomly selecting one message to precede the other in the relative ranking of messages throughout the electronic trading system 100, for example. This may be due to the use of sex. However, using randomness to resolve ambiguity does not support "state certainty" throughout the electronic trading system 100. Different recipient nodes will randomly determine different relative rankings among the same set of messages, leading to unpredictable and non-deterministic behavior within the electronic trading system 100, which may lead to fault tolerance, high availability and This hinders the proper implementation of important features such as disaster resiliency.

Other approaches for recipient nodes to resolve ranking ambiguities include, for example, through predetermined prioritization methods based on node identifiers associated with messages. However, such approaches work against the important goal of fairness by giving higher priority to some messages solely based on the node identifier of the node that introduced the message to electronic trading system 100. For example, some participant devices 130 may end up being prioritized simply because they happen to be connected to electronic trading system 100 via a gateway 120 that is deemed high in a predetermined prioritization method. Become.

Given that a message is uniquely identified via an entity count and either a stock symbol identifier or a flow identifier depending on whether the message has an entity type of "stock symbol" or "flow" respectively, the There is a definitive ranking between the stock symbol (for messages with an entity type of "stock symbol") or other messages associated with the flow (for messages with an entity type of "flow"). obtain. However, the ranking among other messages associated with different stock symbols and flows will still be non-deterministic.

Therefore, even though other fields in message format 110 may be sufficient to uniquely identify a message throughout electronic trading system 100, it is not fair to rank a message relative to other messages in electronic trading system 100. And for deterministic designation, an ordering identifier assigned to the message by sequencer 150 may still be required. In such embodiments, sequencer 150 (or the single currently active sequencer if multiple sequencers 150 are present) provides a truly deterministic system of sequence-marked messages throughout electronic trading system 100. Act as an authoritative source for rankings.

In some embodiments, the nodes in the electronic trading system 100 have two versions of the message: an unordered (unmarked) version of the message, as introduced into the electronic trading system 100 by the generating node, and an unordered (unmarked) version of the message, as assigned by the sequencer 150. A (marked) version of the message containing the ordered order identifier may be received. This may occur in embodiments where the producing node sends unmarked messages to one or more recipient nodes and sequencer 150. Sequencer 150 may then send order-marked versions of the same message to a set of nodes that include the same recipient node.

As discussed above, the order-marked version of a message is useful for determining the message's relative processing order (i.e., position in the order) among other marked messages in electronic trading system 100; may also be useful for receiving unmarked versions of messages. For example, it is certainly possible that in unexpected cases (eg, in embodiments where there is a direct connection between nodes), an unmarked version of a message is received before a marked version of the message. This is because the marked version of the message is sent through sequencer 150 via intervening hops. Thus, in some embodiments, even before a recipient node receives a marked version of the message that reliably indicates the relative ranking of the marked message among other marked messages, A recipient node has an opportunity to activate unmarked message processing in response to receiving an unmarked message.

A node that receives both marked and unmarked versions of the same message can correlate the two versions of the message via the same identification information or "common metadata" in both versions of the message. For example, as discussed above, a producing node may stamp the messages it produces (i.e., unmarked messages) with a node identifier and a node-specific timestamp that together may uniquely identify each message throughout electronic trading system 100. May be included. In embodiments where the marked and unmarked versions of a message are essentially the same except for the ordinal identifier assigned by sequencer 150, the marked message has the same node identifier and node-specific information that is also included in the corresponding unmarked message. A timestamp will also be included, allowing recipient nodes of both versions of the message to correlate the marked and unmarked versions. Thus, although a marked message indicates a relative ranking of the marked message to other marked messages throughout the electronic trading system 100, due to the correlation that may be made between unmarked and marked versions of the same message. In addition, a marked message indicates (at least indirectly via the correlation described above) the relative ranking of that message relative to other messages (marked or unmarked) throughout electronic trading system 100. It should be appreciated that nodes in electronic trading system 100 may correlate order-marked versions of messages to unmarked versions by other aspects of uniquely identifying messages as described above. For example, correlation between order-marked and unmarked messages may be performed by a combination of flow identifiers and node-specific timestamps. Such correlation may additionally or alternatively be performed by the entity count of the message along with the stock symbol identifier or flow identifier in the message, for messages with entity types "stock symbol" and "flow", respectively.

In the era of high-speed trading where microseconds or even nanoseconds matter, participant devices 130 that exchange messages with electronic trading system 100 are often very latency sensitive, and low, predictable latency is preferred. . The configuration shown in FIG. 1D addresses this requirement by providing a point-to-point mesh 172 architecture between at least each of the gateways 120 and each of the compute nodes 140. In some embodiments, each gateway 120 in mesh 172 may have a dedicated high-speed direct connection 180 to compute nodes 140 and sequencers 150.

For example, dedicated connection 180-1-1 is provided between gateway 120-1 (i.e., GW1) and core compute node 140-1 (i.e., Core1), and dedicated connection 180-1-2 is provided between gateway 120-1 (i.e., GW1) and core compute node 140-1 (i.e., Core1); (ie, GW1) and core compute node 140-2 (ie, Core2), and so on. An example connection 180-gc is then provided between gateway 120-g and compute node 140-c, and an example connection 180-sc is provided between sequencer 150 and core compute node 140-c (i.e., Core c ), an example connection 180-gw1-s1 is provided between the gateway 120-1 (i.e., GW g) and the sequencer 150-1, and an example connection 180-c1-s1 is provided between the core compute node 140 -1 (ie, Core1) and the sequencer 150-1.

It should be understood that each dedicated connection 180 in point-to-point mesh 172 is, in some embodiments, a direct point-to-point connection that does not utilize a shared switch. A dedicated or direct connection may be interchangeably referred to herein as a direct or dedicated "link," which is a direct connection between two endpoints that is dedicated (i.e., non-shared) to communication between them. Such dedicated/direct link may be any suitable interconnect or interface, as further disclosed below, and is limited to network links such as wired Ethernet network connections or other types of wired or wireless network links. I can't do it. A dedicated/direct connection/link may also be referred to herein as an end-to-end path between two endpoints. Such an end-to-end path may be a single connection/link or may include a series of connections/links. However, the bandwidth of the dedicated/direct connection/link in its entirety, i.e. from one endpoint to the other endpoint, is non-shared, and neither the bandwidth nor the latency of the dedicated/direct connection/link Even if an element crosses it, it cannot be affected by its resource usage. For example, a dedicated/direct connection/link may traverse one or more buffers or other elements whose bandwidth or latency is not affected based on its utilization. However, dedicated/direct connections/links do not traverse shared network switches. This is because such switches can impact bandwidth and/or latency due to their shared usage.

For example, in some embodiments, dedicated connections 180 in point-to-point mesh 172 include 10 Gigabit Ethernet (GigE), 25GigE, 40GigE, 100GigE, InfiniBand, Peripheral Component Interconnect-Express (PCIe), Rap idIO, Small Computer System Interface (SCSI), FireWire, Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or custom serial or parallel buses. Accordingly, although the compute engine 140, gateway 120, sequencer 150, and other components may be referred to herein as "nodes," other types of interconnections or interfaces are possible; ”, “sequencer node” or “mesh node” should not be construed to mean that the particular components are necessarily connected using network links. Additionally, a "node" as disclosed herein may be any suitable hardware component, software component, firmware component, or combination thereof configured to perform the respective functions described above for a node. As explained in more detail below, a node may be a programmed general-purpose processor, but may also be a specialized hardware device such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other It may be a hardware device or devices, logic within a hardware device, a printed circuit board (PCB), or other hardware component.

The nodes disclosed herein may be independent elements or within a single FPGA, ASIC, or other element configured to execute logic and perform the functions of the nodes as hereinbefore described. It should be understood that they may be integrated together within a single element, such as. A node may also be an entity of software executing logic executed by a general purpose computer and/or any of the above devices.

Traditional approaches of connecting components such as compute engine 140, gateway 120, and sequencer 150 through one or more shared switches do not provide the lowest possible latency. These traditional approaches also result in unpredictable spikes in latency during periods of increased message traffic.

In the exemplary embodiment, dedicated connections 180 are also provided directly between each gateway 120 and each sequencer 150 and between each sequencer 150 and each core compute node 140. Still further, in some embodiments, the dedicated connections 180 are for all sequencers, such that the example sequencer 150-1 has a dedicated connection 180 for each other sequencer 150-2,..., 150-s. provided in between. Although not shown in FIG. 1D, in some embodiments, the dedicated connection 180 also connects each gateway 120-1 to each other gateway 120-2, as further disclosed below with respect to FIG. , 120-g may be provided between all gateways 120 to have dedicated connections 180 to them. Similarly, in some embodiments, the dedicated connection 180 also connects the example core compute node 140-1 to each other core compute node 140-2, . . . , as further disclosed below with respect to FIG. 140-c is provided between all compute nodes 140 to have a dedicated connection 180 to 140-c.

A dedicated connection 180 between two nodes (e.g., between any two nodes 120, 150, or 140) is, in some embodiments, a dedicated connection 180 between those same two nodes for redundancy and increased reliability. It should also be understood that it may also be implemented as multiple redundant dedicated connections between. For example, dedicated connection 180-1-1 between gateway 120-1 and core compute node 140-1 may actually be implemented as a pair of dedicated connections.

Further, according to some embodiments, any message sent by a node is sent in parallel to all nodes directly connected to it in point-to-point mesh 172. Each node in the point-to-point mesh 172 may decide for itself whether to take some action in response to receiving a message or simply ignore the message, based, for example, on the node's configuration. In some embodiments, a node may not ignore the message completely. Even if a node, due to its configuration, takes no substantial action in response to receiving a message, it may take at least a minimal action, such as consuming any sequence number assigned to the message by sequencer 150. You can also take it. That is, in such embodiments, a node may use the last received order number to ensure that if the node takes more substantive action on the message, it does so in the appropriate ordering order. May be tracked.

For example, a message containing a trade order to "Sell 10 shares of Microsoft for $190.00" originates from participant device 130-1, such as a trader's personal computer, and arrives at gateway 120-1 (i.e., GW1). shall be taken as a thing. The message is sent to all core compute nodes 140-1, 140-2, ..., 140-c, even though only core compute node 140-2 is currently performing matching for Microsoft stock orders. Ru. All other core compute nodes 140-1, 140-3,..., 140-c may ignore the message or take only minimal action on the message upon receipt. For example, the only action taken by 140-1, 140-3,..., 140-c may be to consume the sequence number assigned to the message by sequencer 150-1. The message is sent to sequencers 150-1, 150-2, ..., 150-s even if a single sequencer (in this example, sequencer 150-1) is the currently active sequencer serving the mesh. It will also be sent to all. The other sequencers 150-2, ..., 150-s also receive the message and if sequencer 150-1 (currently active sequencer) fails or by moving a different active sequencer, electronic trading system 100. Give them an opportunity to take over the currently active sequencer if the overall reliability of the sequencer is to increase. One or more of the other sequencers (eg, sequencer 150-2) are also responsible for relaying system state to the disaster recovery site 155. Disaster recovery site 155 may include a replica of electronic trading system 100 at another physical location, which replica may be a physical or virtual entity of some or all of the individual components of electronic trading system 100. Equipped with

By sending each message to all directly connected nodes in parallel, electronic trading system 100 reduces complexity and further promotes redundancy and high availability. If all directly connected nodes receive all messages by default, multiple nodes may be configured to act on the same message in a redundant manner. Returning to the above example of the order to "sell 10 shares of Microsoft for $190.00," in some embodiments, multiple core compute nodes 140 may concurrently perform matching for orders of Microsoft stock. For example, both core compute node 140-1 and core compute node 140-2 will perform matching on Microsoft stock messages at the same time, and after receiving the incoming message for the "sell" order, core compute node 140- 1 and core compute node 140-2 may each independently generate a response message, such as an approval or execution message, that is transmitted to gateway 120 through sequencer 150 and passed to one or more participant devices 130.

Because of the strict ranking and state determinism guaranteed by sequencer 150, each of the associated response messages independently generated by and transmitted from core compute nodes 140-1 and 140-2 is substantially It is possible to guarantee that the two products are functionally equivalent. Thus, the architecture of electronic trading system 100 readily supports redundant processing of messages, thereby increasing system availability and resiliency. In such embodiments, gateway 120 may receive multiple related outgoing messages from core compute node 140 for the same corresponding incoming message. Due to the fact that these multiple related response messages can be guaranteed to be equivalent, the gateway 120 need only process the first received outgoing message and not the subsequent one that corresponds to the same incoming message. Ignore related outgoing messages. In some embodiments, the "first" and "subsequent" messages may be identified by their associated order numbers, as the messages may be order marked messages. However, in other embodiments, such as those in which sequencer 150 assigns a single ordering identifier among multiple functionally equivalent messages, the messages may have an entity type field, as described further above in connection with FIG. 1E. 110-16 and as functionally equivalent based on other identifying information in the message, such as the values in the entity count field 110-17.

Therefore, allowing gateway 120 to take action on the first of multiple functionally equivalent related response messages to arrive improves the overall latency of electronic trading system 100. It could also happen. Still further, electronic trading system 100 is easily configurable such that any incoming message is processed by multiple compute nodes 140, each of which has an equivalent Generate a response message. Such an architecture allows compute nodes 140 to handle incoming messages for a period of time (whether due to system failure, node reconfiguration, or maintenance activities) without appreciable impact on latency. Provide high availability.

The point-to-point mesh 172 architecture of such an electronic trading system 100, in addition to maintaining low predictable latency and redundant processing of messages, also provides for multiple built-in redundant paths. As can be seen, multiple paths exist between any gateway 120 and any compute node 140. If the direct connection 180-1-1 between the gateway 120-1 and the compute node 140-1 becomes unavailable, communications can still be made via an alternative connection, such as by traversing one of the sequencers 150 instead. It is still possible between these two elements via a path. Thus, more generally, in point-to-point mesh 172, multiple paths exist between any node and any other node.

Still further, this point-to-point mesh architecture inherently maintains another important goal of financial trading systems: fairness. A point-to-point architecture with direct connections between nodes ensures that the paths between any gateway 120 and any core compute node 140 or sequencer 150 and any other node have identical or at least very similar latencies. Ensure that you have the following. Therefore, two incoming messages sent to sequencer 150 from two different gateways 120 at the same time should arrive at sequencer 150 at substantially the same time. Similarly, outgoing messages sent from core compute node 140 should be sent to all gateways 120 simultaneously and received by each gateway at substantially the same time. Because the point-to-point mesh topology does not favor any one gateway 120, the chance that being connected to a particular gateway 120 could give a participant device 130 an unfair advantage is minimized.

Additionally, the point-to-point mesh architecture of electronic trading system 100 allows for easy reconfiguration of a node's functionality, ie, whether the node is currently acting as a gateway 120, core compute node 140, or sequencer 150. It is particularly easy to perform such reconfiguration in embodiments where each node has a direct connection between itself and each other node in the point-to-point mesh. If each node is connected to each other node in the mesh via a direct connection, changing the functionality of the node within the mesh (e.g., changing the functionality of the node from the core compute node 140 to the gateway 120 or from the gateway 120 to the sequencer) 150) requires rewiring or recabling of the connections 180 (whether physical or virtual) within the point-to-point mesh 172. In such embodiments, any necessary internal reconfiguration of point-to-point mesh 172 may be easily accomplished through remotely performed configuration changes. When a node is reconfigured to act as a new gateway 120 or from acting as a gateway 120 to other functions, some incidental networking changes external to point-to-point mesh 172 occur. may be required, but the internal wiring of the mesh may be left as is.

Thus, in some embodiments, reconfiguration of a node's functionality may be accomplished live, even dynamically, during trading hours. For example, due to changing load characteristics or new demands on electronic trading system 100, it may be useful to reconfigure core compute node 140-1 to act as an additional gateway 120 instead. . After some possible redistribution of state or configuration to other compute nodes 140, a new gateway 120 becomes available and will begin accepting new connections from participant devices 130.

In some embodiments, slower and potentially higher latency shared connections 182 may be provided between system components, such as between gateway 120 and/or core compute node 140. These shared connections 182 are in contrast to messages related to trading activities performed over dedicated connections 180 in point-to-point mesh 172, such as messages 106 and responses 107 disclosed above with respect to FIG. 1B-2. , for maintenance, control operations, administrative operations, and/or similar operations that do not require very low-latency communications. In contrast to first direct connection 180-a, second direct connection 180-b, and third direct connection 180-c, which carry traffic related to trading activity, shared connections 182g and 182c carry traffic related to non-trading activity. carries type of traffic. Shared connections 182 carrying non-transaction traffic may be through one or more shared networks and through one or more network switches, and the nodes in the mesh may be distributed among these shared networks in different ways. Good too. For example, in some embodiments, the gateways 120 are all in the gateway-wide shared network 182-g, the compute nodes 140 are in their own respective compute node-wide shared network 182-c, and the sequencers 150 are in their own may be in separate sequencer-wide shared networks 182-s, but in other embodiments, all nodes in the mesh may communicate via the same shared network for their latency-insensitive work. .

Distributed computing environments, such as electronic trading system 100, may follow high-resolution clocks to maintain tight synchronization among the various components. To this end, one or more of the nodes 120, 140, 150 may be provided with access to a clock, such as a high-resolution Global Positioning System (GPS) clock 195, in some embodiments. . For purposes of the following description, gateways 120, compute nodes 140, and sequencers 150 connected to point-to-point mesh 172 are referred to as "mesh nodes" and have an architecture as further disclosed below with respect to FIG. obtain.

FIG. 1F is a flow diagram of an exemplary embodiment of a process 125 that may be performed by electronic trading system 100 to process messages. Referring to FIG. 1B-2, gateway 120-1 may receive a message from a participant device (131) and in response may send message 106 to sequencer 150-1 and compute node 140-1 (132). . Gateway 120-1 may optionally include an identifier generated by gateway 120-1 in message 106; in an electronic trading system with multiple compute nodes and/or sequencers, gateway 120-1 may include an identifier generated by gateway 120-1 in message may be sent to all such nodes. In response to receiving the message 106, the compute node 140-1 first determines whether the message 106 has been assigned for processing by the compute node 140-1, and identifies one or more values or identifiers of the message 106. This determination may be made based on the value of a common parameter (e.g., the stock symbol representing a given financial instrument or the type of transaction). If the message 106 is assigned for processing, the compute node 140-1 may perform preprocessing on the message 106 (134) to generate preliminary results. For example, compute node 140-1 may read into memory information related to the value referenced in message 106 (eg, a stock symbol for a financial instrument). Alternatively or additionally, in some embodiments, pre-processing includes, by way of non-limiting example, generating a response message, including an authorization message or an execution message, the values referenced in message 106 (e.g., financial This may include performing matching functions, such as performing a preliminary update of an order book for a product (stock symbol). Such preprocessing may be performed so that it has no harmful side effects, or if necessary, if the message on which the preprocessing was performed was received out of order. It may be "rolled back" atomically (ie, on a transaction-by-transaction basis) when determined.

At the same time, sequencer 150-1 assigns messages to unique order identifiers that allow, for example, the position of message 106 to be identified within a series of messages that may need to be processed in a given order (i.e., order). Messages 106 may be ordered by associating (133) 106 to produce an order-marked version of message 106, ie, order-marked message 106'. Once complete, sequencer 150-1 may send an order-marked message 106' containing message 106 (or an indication thereof) and an order identifier to compute node 140-1 (135). When compute node 140-1 receives an order-marked message 106', it determines, in some embodiments, whether the (unordered) message 106 on which it performed preprocessing was received in order. ie, the order may be verified (136). As part of verifying order, compute node 140-1 may correlate message 106 to order-marked message 106' with common metadata or common identification information in both versions of the message. In process 125 of the exemplary embodiment, message 106 has been received in order and compute node 140-1 can proceed to continue processing message 106 (137). As a result of the pre-processing, the compute node 140-1 has already completed some operations (e.g., cache memory lookup of the value) that it would have performed after receiving the order-marked message 106'; Thereby, compute node 140-1 may complete the steps of processing message 106 sooner than it would have completed in the absence of the preprocessing operations.

FIG. 1G is a flow diagram of an example embodiment of process 165 illustrating another example embodiment of the operation of an electronic trading system. Process 165 may be performed by electronic trading system 100 to process the message. In the embodiment of FIG. 1G, electronic trading system 100 includes at least a second gateway, gateway 120-2. Referring to FIG. 1B-2, gateway 120-1 receives (166) a first message (M1) from a participant device and in response sends message 106 corresponding to first message M1 to sequencer 150. -1 and compute node 140-1 (167). Message 106 may also be referred to as first message M1, since the gateway may have modified the message to include different metadata, such as, by way of non-limiting example, an identifier for gateway 120-1. , it should be understood that message 106 may not be the same as M1. Gateway 120-2 may receive (168) a second message (M2) from other participant devices and send a message corresponding to second message M2 to sequencer 150-1 and compute node 140-1. (169). The message corresponding to the second message M2 may also be referred to as second message M2 here. In this example, the first and second messages belong to a common set of messages such that the relative ranking (i.e., order) in which the first and second messages are processed affects the state of electronic trading system 100. can be given. In the exemplary embodiment of FIG. 1G, compute node 140-1 receives a second message before receiving the first message, and the second message is one that compute node 140-1 has assigned for processing. Once confirmed, compute node 140-1 may then perform pre-processing on the message (171). Upon receiving the first message (M1), compute node 140-1 may also perform preprocessing on the first message (M1) (173).

Sequencer 150-1 may receive both the first and second messages. The sequencer 150-1 may then assign each of the first and second messages a respective unique ordering identifier that may be used to determine the relative ranking, or order, of the first and second messages. (170, 174). In some embodiments, a sequence identifier assigned to a message by a sequencer may be a monotonically increasing sequence number, as a non-limiting example. Still further, in some embodiments, the value of the ordering identifier assigned to a message by the sequencer may be based on the time of arrival of the message at sequencer 150-1. That is, in some embodiments, as a non-limiting example, messages are placed in a first-in-first-out (FIFO) queue as they are received by sequencer 150-1, and they are removed from the FIFO queue. An ordering identifier is assigned to each item. In the exemplary embodiment of FIG. 1G, message M1 was received earlier than message M2 at sequencer 150-1, so in embodiments where the sequence identifier is a monotonically increasing sequence number, sequencer 150-1 will be assigned a lower sequence number relative to the other sequence number assigned to message M2. Once the process of assigning an ordering identifier to each message is complete, the sequencer 150-1 sends the marked messages (i.e., messages containing the ordering identifier) corresponding to the first and second messages or representations thereof to the compute node 140-1. (175, 177). In some embodiments, when compute node 140-1 receives a marked message, it determines whether the preprocessed (unordered) message 106 was received in order, i.e., in order. may be verified (176). As part of verifying order, compute node 140-1 may correlate unordered message 106 to order-marked message 106' with common metadata or common identification information in both versions of the message. In operation 165, compute node 140-1 receives the second message out of order. In this result, some or all of the artifacts of preprocessing (e.g., cache lookups for values of common parameters of messages, as a non-limiting example) may optionally be discarded, or other Type preprocessing may be atomically rolled back. Compute node 140-1 may then proceed to process (178) both the first and second messages in order (ie, order) as indicated by sequencer 150-1.

Even if it is determined that the unordered message was received out of order at compute node 140-1, node 140-1 will still benefit from performing pre-processing on the message. By pre-processing the message, compute node 140-1 has determined that this message will likely need to be processed in the future, thereby allowing it to take proactive action to save time. Information needed to fully process a message, such as information about current open trade orders for the same stock symbol, is transferred to faster side memory (e.g., Figure 2) before the message can be fully processed. FPGA memory, such as fixed logic memory 250 shown in FIG. (such as DRAM 280 shown in FIG. Although this information may not currently exist in high-speed memory for the value of a common parameter, such as a stock ticker, for messages that have not been recently referenced, compute node 140-1 can The memory copy operation can be performed in advance, thereby saving time if the ordered messages arrive quickly afterwards in the proper order (ie, order). By allowing a compute node to prefetch information for a large number of values in advance, provided that the fast-side memory is large enough to hold the information for multiple values, It may even be beneficial to receive a series of consecutive out-of-order messages. Therefore, rather than ignoring the results of pre-processing operations, compute node 140-1 may temporarily store the results for reference during later processing operations.

In some example embodiments, the artifacts of preprocessing by a compute node may still be valid in instances that do not follow order ranking. For example, an implementation in which each computing node handles some set of operations, such as some set of values for a common parameter of a message (e.g., a limited number of stock tickers or other indicators of a financial instrument). In this case, a computing node may still receive messages that refer to other stock symbols that it does not handle, but the ranking of the ordering changes only insofar as it affects the values that the computing node handles. Can be related to nodes. Also, the ranking of messages referencing different values may be considered independent of each other. For example, if messages are received out of order relative to other messages that refer to stock symbols of different values, the order in which these two messages are processed will not affect the result of the matching function operation; No further action regarding preprocessing (eg, rollback) is required, and the preprocessed results may ultimately be used without modification and/or bound to the state of the distributed system.

It is understood that in some embodiments, verifying the order (136 in FIG. 1F and 176 in FIG. 1G) may not be necessary depending on the nature of the pre-processing performed by compute node 140-1. Should. For example, preprocessing performed on an unordered message has no harmful side effects and processing of other messages (e.g., loading data related to stock symbols referenced in the message into memory on the fast side) It may not be necessary to verify that the preprocessing of unordered messages was performed according to the order determined by sequencer 150-1, and certainly the preprocessing There may be no need to discard or atomically roll back the results. In such embodiments, the compute node 140-1 simply subsequently marks the order mark at the order rank specified by the order identifier in the order marked message 106' whenever utilizing the preprocessed results. Processing of the message may be completed in response to receiving the attached message 106'.

The amount of preprocessing that may be done on an unordered message, and whether the results of that preprocessing need to be discarded or rolled back, is determined by the message type field 110-1, stock symbol field, according to the embodiment of FIG. 1E. 110-2, side field 110-3, or price field 110-4. It determines whether other unordered messages that refer to the same value for a common parameter in the message, such as the same stock ticker, are currently outstanding (i.e., for which a corresponding order-marked message has not yet been received). You may choose to do so.

For example, if an unordered message with a message type of "new order" is received by core compute node 140-1, core compute node 140-1 stores stock symbol information associated with the relevant portion of the order book in high-speed memory. If the new order is a match for an open order in the order book, compute node 140-1 thereby begins generating a "Fill" message, but does not receive an ordered version of that message. Until then, the customer will refrain from committing to updating the order book and from sending "Confirmed" messages. However, Compute node 140-1 does not accept any other outstanding unordered "new orders" that refer to the same stock symbol, side, price, etc. that are also potential matches for the same open orders in the order book. If a message is also being received, core compute node 140-1 may perform its preprocessing differently. In some embodiments, core compute node 140-1 determines that each of the two outstanding unordered "new order" messages may act as a match against a potential conflicting "fill" for each outstanding order. Messages may be generated. Based on the ordered version of the messages, one of the potential "fill" messages may be discarded and the other will be committed to the order book and sent to gateway 120. In other embodiments, if two or more possible outstanding unordered messages can potentially match the same outstanding order, Compute node 140-1 determines which messages need to be discarded or rolled back. No pre-processing may be performed (e.g., without creating any potential "fill" messages) or any such pre-processing may be aborted or interrupted for these unprocessed unordered messages.

As another example, an outstanding unordered "new order" message that would be a potential match for an open order in the order book would act as a potential match for a "new order" message in the order book. may conflict with outstanding unordered "replace order" or "cancel order" messages that attempt to replace or cancel the same open order, respectively. In this case, depending on the relative order assigned by the sequencer of the "New Order" message versus the "Replacement/Cancellation Order" message, the final result is the difference between the open order and the "New Order" message in the order book. There may be a match, or the outstanding order may be canceled or replaced by a new order at a different price or quantity. Compute node 140-1 cannot determine which of these two outcomes should occur until the order-marked versions of the conflicting outstanding unordered messages are received by sequencer 150-1.

In this case, compute node 140-1 may perform preprocessing differently. In some embodiments, if there are multiple conflicting unordered messages outstanding, the compute node 140-1 sends the relevant portions of the order book associated with the stock symbols referenced in both conflicting messages to the fast side. One may simply perform preprocessing, such as loading into memory, that does not need to be rolled back or discarded. In other embodiments, compute node 140-1 may perform additional pre-processing, such as configuring one or more tentative potential responses, each corresponding to one of a plurality of conflict scenarios. For example, compute node 140-1 may create a potential "Fill" message and/or a potential "Replacement Approval" or "Cancellation Approval" message, and in some cases select one of multiple possible outcomes. You may provisionally update the order book corresponding to one or more items. In some embodiments, Compute node 140-1 may perform this additional pre-processing for all such conflict scenarios, while in other embodiments, Compute node 140-1 may perform this additional pre-processing for all conflict scenarios. It may be performed for only one or some of the sets. For example, compute node 140-1 may perform additional preprocessing on outstanding unordered messages only if there are no other outstanding conflicting unordered messages. Alternatively or additionally, Compute node 140-1 may perform additional pre-processing on unprocessed conflicting unordered messages according to the amount of time and/or complexity involved in rolling back or discarding the results of the pre-processing. Priority can be given to the execution of Upon receiving the order-marked versions of the outstanding unordered messages, the compute node 140-1 then determines the order (as assigned by the sequencer 150-1) in which the outstanding unordered messages are to be processed. , may complete processing of messages in that order. This may include rolling back or discarding one or more results of the preprocessing in some embodiments.

In addition to the types of pre-processing already described above, in some embodiments, compute node 140-1 additionally or alternatively performs pre-processing regarding the validity of the message to accept or reject the message. can be determined. For example, preprocessing may include ensuring that the price or quantity specified by the message does not exceed a maximum value (i.e., "Maximum Price Check" or "Maximum Quantity Check"), that the stock symbol in the message is a known stock symbol ( (i.e., "Unknown Stock Symbol Confirmation"), trading is currently permitted for the stock symbol (i.e., "Stock Symbol Confirmation"), or the price is properly specified according to the correct number of decimal places. This may include subjecting messages to real-time risk confirmations, such as confirming that they are true (i.e., “subpenny confirmations”). In some embodiments, the type of pre-processing is that if "self-trading prevention" is enabled for a particular client or trading order, certain potential matches become self-trading, i.e. May include a "self-trading prevention" validation to prevent a trading client from matching against itself. If a trading order fails one or more of these validation checks, electronic trading system 100 may respond with an appropriate rejection message. Although these validation checks are described in the above embodiments as being performed by compute node 140-1, at least some of these types of validation checks may be performed by gateway 120 or electronic transaction in some embodiments. It should be understood that it may alternatively or additionally be performed by other nodes in system 100.

In further embodiments, it may be beneficial or necessary for gateway 120-1 to be informed of a unique system-wide ordering identifier associated with a message originating from a client. This information may enable gateway 120-1 to match the original incoming message to a unique sequence number. This is used to ensure proper ranking of messages throughout electronic trading system 100. Such configuration at the gateway may require electronic trading system 100 to implement state determinism to provide fault tolerance, high availability, and disaster recovery with respect to activity at the gateway. One solution for configuring the gateway 120-1 to maintain sequence identifier information associated with incoming messages is to have the gateway 120-1 send the message to the sequencer 150 along with the sequence identifier before forwarding the message to the compute node 140-1. This is to wait for a reply from -1. Such an approach may add latency to message processing. In a further example, in addition to forwarding to compute node 140-1 the order-marked messages that it originally received from gateway 120-1, sequencer 150-1 forwards order-marked messages (e.g., FIG. 1B-2 Marked messages 106') as shown in FIG. 1 may be sent in parallel to the gateway 120-1. As a result, gateway 120-1 can maintain order identifier information while minimizing latency in electronic trading system 100.

FIG. 2 is a block diagram of an exemplary embodiment of a mesh node in a point-to-point mesh architecture of an electronic trading system, such as electronic trading system 100 disclosed above. FIG. 2 shows an exemplary embodiment of a mesh node 200 in a point-to-point mesh 172 architecture of electronic trading system 100. Mesh node 200 may represent, for example, gateway 120, sequencer 150, or core compute node 140. Although in this example the functionality in mesh node 200 is distributed across both hardware and software, mesh node 200 may be implemented in any suitable combination of hardware and software, including pure hardware and pure software implementations. In some embodiments, any or all of gateway 120, compute node 140, and/or sequencer 150 may be implemented with commercially available components.

In the embodiment shown in FIG. 2, some functions are implemented in hardware in fixed logic device 230 while other functions are implemented in software in device driver 220 and mesh software application 210 to achieve low latency. Implemented. Fixed logic device 230 may be implemented in any suitable manner, including an application specific integrated circuit (ASIC), an embedded processor, or a field programmable gate array (FPGA). Mesh software application 210 and device driver 220 may be implemented as instructions executing one or more programmable data processors, such as a central processing unit (CPU). Different versions or configurations of mesh software applications 210 may be installed on mesh nodes 200 depending on their roles. For example, different versions or configurations of mesh software application 210 may be installed based on whether mesh node 200 is operating as gateway 120, sequencer 150, or core compute node 140.

Any suitable physical communication link layer may be employed (Universal Serial Bus (USB), Peripheral Component Interconnect (PCI) Express (PCI-Express, or PCI-E), High Definition Multimedia Interface (HDMI), 10 Gigabit Ethernet (GigE), 40GigE, 100GigE, or InfiniBand (IB) (via fiber or copper cable), in this example, the mesh node 200 supports multiple low-latency 10 Gigabit Ethernet Small Form Factor Pluggable Plus (SFP+) It has connectors (interfaces) 270-1, 270-2, 270-3, . . . , 270-n (collectively shown as connector 270). Connector 270 may be connected directly to other nodes in a point-to-point mesh via dedicated connection 180, connected via shared connection 182, and/or connected to participant device 130 via gateway 120, for example. . These connectors 270, in this example, are electrically connected to respective 10 GigE media access (MAC) cores 260-1, 260-2, 260-3, ..., 260-n (collectively shown as GigE core 260). combined, which in this embodiment is implemented by a fixed logic device 230 to ensure minimal latency. In other embodiments, 10GigE MAC core 260 may be implemented by functionality external to fixed logic device 230, for example, in a PCI-E network interface card adapter.

In some embodiments, fixed logic device 230 may also include other components. In the example of FIG. 2, fixed logic device 230 also includes fixed logic 240 components. In some embodiments, fixed logic component 240 performs different functions depending on the role of mesh node 200, e.g., whether it is gateway 120, sequencer 150, or core compute node 140. obtain. Fixed logic device 230 also includes fixed logic memory 250, which can be memory accessed with minimal latency by fixed logic 240. Fixed logic device 230 also includes a PCI-E core 235 that may implement PCI Express functionality. In this example, PCI Express transfers data between hardware and software, and more specifically, between fixed logic device 240 and mesh software application 210 via PCI Express bus 233 by device driver 220. Used as a conduit mechanism. However, any suitable data transfer mechanism may be employed between hardware and software, including direct memory access (DMA), shared memory buffers, or memory mapping.

In some embodiments, mesh node 200 may also include other hardware components. For example, depending on its role in electronic trading system 100, mesh node 200, in some embodiments, is equipped with a high-resolution clock 195 (Fig. 1D). Dynamic random access memory (DRAM) 280 may also be included in mesh node 200 as additional memory in conjunction with fixed logic memory 250. DRAM 280 may be any suitable volatile or nonvolatile memory, including one or more random access memory banks, hard disks, and solid state disks, and may be accessed through any suitable memory or storage interface. . As disclosed above, mesh node 200 may represent gateway 120, sequencer 150, or core compute node 140, and as disclosed above with respect to FIGS. 1A-D and further below with respect to FIG. Can be configured in a two-point mesh architecture.

In view of the exemplary embodiments described in FIGS. 1B-1, 1B-2, 1C, 1D, 1E, and 2, point-to-point mesh architecture 172 of electronic trading system 100 provides for improved latency in a number of ways. It should be obvious. In embodiments where messages are exchanged between nodes within electronic trading system 100 via direct dedicated connections 180 without traversing a switch, bypassing the switch itself allows for a number of latency-related beneficial effects. Become.
a) The time required for a switch to process a single message (the time interval between when a message enters the switch and when it leaves the switch) is completely eliminated. The latency of this switch process is typically about 1.0 microseconds and includes buffering time and time to route the message to the appropriate destination, as specified in the message's header.
b) two transmission times (transmission time between the sending node and the switch and transmission time between the switch and the receiving node) are replaced by a single transmission time for transmitting the message directly between the sending node and the receiving node It will be done.
c) Communicating via a direct point-to-point connection without a switch may be required by a switch, but may add processing time overhead to the sending and receiving nodes as well. It also eliminates the requirement to exchange messages within a mesh that follows a protocol. the time required for the sending node to perform protocol-specific processing such as constructing the TCP/IP header, calculating the checksum, and for the receiving node to further interpret the TCP/IP header; The time required to perform similar protocol-specific processing, such as verifying a checksum, is about 0.5 microseconds for each of the sending and receiving nodes, for a total of about 1.0 microseconds. This protocol-specific overhead can optionally be applied to point-to-point mesh architectures that exchange messages according to one or more custom protocols rather than a specific established protocol required by the switch, such as TCP/IP. 172 may be excluded in some embodiments.
d) Messages may be broadcast to directly connected nodes at exactly the same time and received by directly connected recipient nodes at exactly the same time, as described further below. Different messages may also be received completely simultaneously from multiple directly connected sender nodes, as discussed further below.

Embodiments in which direct dedicated connections 180 between nodes are implemented via dedicated communication logic and dedicated interfaces, such as GigE MAC core 260 and connector 270, respectively, as described above in connection with FIG. 2, improve latency. obtain. In some such embodiments, each GigE MAC core 260 handles message communication between its node and a single other node, and each connector 270 handles message communication between its node and a single other node, and each GigE MAC core 260 handles message communication between its node and a single other node, and each to other nodes via a dedicated connection 180. For example, considering three nodes (gateway 120-1, core compute node 140-1, and core compute node 140-2) in the point-to-point mesh of the embodiment of FIG. 1D, gateway 120-1 is the core compute node 140-1 via a dedicated connection 180-1-1, gateway 120-1 is also connected to core compute node 140-2 via a separate dedicated connection 180-1-2. If each of these three nodes is implemented according to the embodiment of FIG. GigE MAC core 260-1 and connector 270-1 for core compute node 140-1 may be used solely for communication with gateway 120-1. Similarly, via dedicated connection 180-1-2, GigE MAC core 260-2 and connector 270-2 to gateway 120-1 are used solely for communication with core compute node 140-2, and GigE MAC core 260-1 and connector 270-1 for node 140-2 may be used solely for communication with gateway 120-1. Thus, according to such embodiments, a dedicated compute resource, such as one of the GigE MAC cores 260 and one of the connectors 270, is used for each node to communicate with each other node having a dedicated connection 180. Ru.

These dedicated compute resources per dedicated connection 180 allow messages to be sent by the sending node to the sending node (e.g., if the GigE core 260 is implemented in hardware such as fixed logic device 230). completely simultaneously between other mesh nodes that are directly connected (via). Conversely, a broadcast message from a sending node may be received by all receiving nodes completely simultaneously. Each receiving node has a respective dedicated connection 180 to the sending node, and for each node directly connected to the receiving node, reception is performed by dedicated connection communication logic and a dedicated interface. Since each receiving node receives the broadcast message exactly at the same time, and the internal message latency between different nodes in the mesh should be the same, no receiving node is preferred. Furthermore, a receiving node may receive different messages completely simultaneously from each different node directly connected to it.

The ability to send and receive multiple messages completely simultaneously is in contrast to other environments where communication between servers occurs through switches, and such environments typically require that message sending and receiving be serialized. It takes. When communication is through a switch, each server typically has a single connection (or possibly multiple redundant connections, but is still considered a single logical connection) between itself and the switch. have). When a server needs to broadcast a message to multiple other servers in the system, those messages are neither actually sent to nor received by the switch at the same time, but instead , must be sent/received serially and sequentially over a single logical connection between the server and the switch. Similarly, it is possible for a switch to simultaneously receive multiple different messages destined for the same server but originating from different sending servers (because the switch may have a single connection to each of the originating servers). Even if possible, these multiple different messages would still be serialized by the switch in order to send them one by one over a single logical connection between the switch and the destination server. There is a need. The message serialization required in such an environment not only increases the overall latency of the system, but also results in unpredictable latency between servers within such a system. For example, if a message needs to be broadcast from a server to 15 other servers in the system, these 15 messages need to be serialized as they are sent to the switch, and the latency difference can be large depending on whether the message addressed to a particular destination server is the first or the 15th message in a series of broadcast messages.

In addition to the latency benefits that can be obtained from the use of dedicated connections 180 that do not require traversal of switches, point-to-point mesh architecture 172 allows for yet other latency improvements. As discussed above in connection with FIG. 1B-1, messages being sent between two nodes within a point-to-point mesh, such as from gateway 120-1 to core compute node 140-1, are To ensure that messages can be processed throughout electronic trading system 100 in a deterministic order, they may be sent via ranking path 117. As the message traverses through ranking path 117, it passes through sequencer 150-1, which marks the message with a unique ordering identifier, after which sequencer 150-1 passes to the destination node (in this example, a core compute generate an order-marked message to send to node 140-1); Receipt of the order-marked message causes the destination node (in this example, core compute node 140-1) to mark the order with respect to other messages within electronic trading system 100 (which are also order-marked by the sequencer). It may be ensured that messages are processed in the correct deterministic order, as specified by the order identifier of the attached message.

As discussed above in connection with FIG. 1B-1, this ranking path 117 includes multiple direct connections: a direct connection between the transmitting node and the sequencer (a direct connection 180 between the gateway 120-1 and the sequencer 150-1); -gw1-s1) and further direct connections between the sequencer and the destination node (such as the direct connection 180-c1-s1 between sequencer 150-1 and compute node 140-1). These multiple direct connections with ranking path 117, in some embodiments, do not pass through a switch and can benefit from the beneficial effects of latency already mentioned above. Nevertheless, in such embodiments, the latency of ranking pass 117 is determined by the processing time of the message at sequencer 150-1 (i.e., when sequencer 150-1 receives an out-of-order marked (i.e., unordered) message; the time required to mark it with an order identifier and send the order-marked message to its destination) and the transmission time of the message over the two direct connections. (i.e., the unordered marked version is first sent over a direct connection from the sender node to the sequencer, and then the ordered marked version is sent from the sequencer to the destination node over a separate direct connection). Testing will determine the impact of this latency on the ranking path 117 when compared to a single direct connection, depending on factors such as whether a custom protocol is used rather than a specific established protocol such as TCP/IP. Accordingly, it indicates an additional 0.5-1.0 microseconds for the ranking pass 117.

Therefore, as an optimization to minimize the potential latency impact of message traversal through ranking path 117, out-of-order marked messages are also used directly between gateway 120-1 and core compute node 140-1. It may be transmitted in parallel over activation link 180-1-1, which may be a single direct connection between the sender node and the destination node, such as connection 180-1-1. Accordingly, in some embodiments, out-of-order marked messages are sent to a destination node (e.g., core compute node 140-1) via activation link 180-1-1 and as part of ranking path 117. , may be sent completely simultaneously by the sending node (e.g., gateway 120-1) to sequencer 150-1 via a direct connection between the sending node and sequencer 150-1 (e.g., direct connection 180-gw1-s1). . According to such embodiments, sequencer 150-1 and a destination node (e.g., core compute node 140-1) may simultaneously receive an out-of-order marked version of a message, thereby allowing the destination node to It becomes possible to immediately activate the processing of out-of-order marked messages in response to the request. At the same time that ordered marked messages are being generated and sent to the destination node by sequencer 150-1, the destination node may begin processing out-of-order marked messages in parallel; , loading data related to stock symbols referenced in the message into high-speed memory, constructing a preliminary response message, or initiating matching function activities for electronic trading. As discussed above, upon receiving an order-marked message, a destination node may complete processing of the message according to the proper deterministic order (ie, order) as specified by the order identifier. However, by the time the destination node receives the ordered marked message via ranking path 117, the destination node has a chance (e.g., 0.5 to 1. 0 microseconds of processing time) than if the destination node had not yet received the out-of-order marked version of the message over the activated link 180-1-1. It becomes possible to complete processing of ordered marked messages and generate corresponding lower latency response messages.

Thus, as discussed above, point-to-point mesh architectures (eg, 102 and 172) provide for improved latency in a number of ways. Through testing, at least some of these latency improvements have been quantified empirically. For example, the latency improvement resulting from the use of a direct connection between nodes that avoids switches would total at least 1.0 microseconds, and in some cases up to 3.0 microseconds in each direction (i.e., incoming vs. outgoing). becomes. Additionally, in parallel with the processing and transmission of messages via ranking path 117, latency due to the opportunity for the destination node to process out-of-order marked messages received via activation link 180-1-1. The improvement is an additional 0.5-1.0 microseconds in each direction. Therefore, the overall possible latency improvement from the time an incoming message enters electronic trading system 100 from participant device 130 via gateway 120 to the time a corresponding response message is sent from gateway 120 to participant device 130 is , at least 2.5 to 7.0 microseconds. In latency-sensitive applications such as electronic trading, even microsecond latency differences can be very important. These latency improvements enable some embodiments of electronic trading system 100 to reliably respond according to desired response time latencies of 5.0 to 7.0 microseconds. This is a significant improvement over prior art trading systems, the best of which currently have response time latencies in the range of 50-75 microseconds.

FIG. 3 is a block diagram of another exemplary embodiment of a point-to-point mesh system 302. Point-to-point mesh system 302 includes multiple gateways 320, multiple core compute nodes 340, and sequencer 350. Each gateway 320-1, 320-2, . 340 core compute nodes 340-1, 340-2, . . . , 340-c. The sequencer 350 is coupled to each gateway of the plurality of gateways 320 via a respective second direct connection, i.e., a second direct connection 380-b, and a respective third direct connection, i.e., a third direct connection. Coupled to each core compute node of the plurality of core compute nodes via connection 380-c. 1B-1, 1B-2, 1D and 3, gateway 120-1 can be connected via a shared gateway network as disclosed above with respect to FIG. 1D or directly connected 384a, 384b, . . . of FIG. , may be a given gateway of a plurality of gateways 320 communicatively coupled to each other via 384g. The plurality of gateways 320 include gateways 320-1, 320-2, . . . , 320-g in the exemplary embodiment. Core compute nodes 140-1 are communicatively coupled to each other via a shared core compute node network as disclosed above with respect to FIG. 1D or via direct connections 386a, 386b, ..., 386c of FIG. A given core compute node may be one of a plurality of core compute nodes 340, including core compute nodes 340-1, 340-2, . . . , 340-c. It should be understood that the number g of the gateway may be the same or different from the other number c of the core compute nodes in the point-to-point mesh system 302. According to an exemplary embodiment, the sequencer 150-1 is connected via a shared sequencer network, such as the sequencer-wide shared network 182-s disclosed above with respect to FIG. 1D, or through a fourth direct connection 482, such as the fourth direct connection 482 disclosed below with respect to FIG. A given sequencer may be one of a plurality of sequencers communicatively coupled to each other via a direct connection.

FIG. 4 is a block diagram of another exemplary embodiment of a point-to-point mesh system 402. Referring to FIGS. 1B-2 and 4, sequencer 150-1 is configured to perform one of the following sequencers in point-to-point mesh system 402, including sequencer 450-1 and at least one other sequencer 450-2. It can be a sequencer.

In the point-to-point mesh system 402, each gateway 420-1, 420-2, ..., 420-g of the plurality of gateways 420 is connected via a respective first direct connection, i.e., first direct connection 480a. and is coupled to each core compute node 440-1, 440-2, . . . , 440-c of the plurality of core compute nodes. Each gateway of the plurality of gateways 420 connects each sequencer 450-1 of the plurality of sequencers 450-1, . . . , 450-2. Each core compute node 440-1, 440-2, . -c-2 to each of the sequencers 450-1, . . . , 450-2 of the plurality of sequencers.

A given sequencer, i.e., a particular sequencer of sequencers 450-1...450-2, may be the currently active sequencer serving a point-to-point mesh system, and other sequencers of multiple sequencers may be Each sequencer may be a standby sequencer waiting to take over for the currently active sequencer. Each sequencer of the plurality of sequencers may be coupled to each other sequencer of the plurality of sequencers via a respective fourth direct connection, such as fourth direct connection 482. Each of the gateways 420-1, 420-2, . . . , 450-2. Each of the core compute nodes 440-1, 440-2, . 1, . . . , 450-2. At least one other sequencer is on standby.

A standby state (standby role) may represent a passive state in which at least one other sequencer is powered on and configured to be in a "listening only" mode. In "listening-only" mode, at least one sequencer is configured to receive messages, but not to send messages, and is capable of taking over as the currently active sequencer if the currently active sequencer fails. There is. The currently active sequencer is in an active state (active role) in which it can send and receive messages. Typically, a standby sequencer is a redundant (backup) sequencer that is powered on, capable of receiving messages, and capable of taking over for the active sequencer. Such a switchover, i.e. from a standby state (roll) to an active state (roll), may occur due to a failure of the currently active sequencer or, by way of non-limiting example, to software applied to the currently active sequencer. It may be based on receiving a switching command issued by a controller (not shown) for other reasons, such as an update.

A given sequencer (which is in the active state) may also receive an order mark if the currently active sequencer fails or is no longer designated as the active sequencer for any other reason, such as, by way of non-limiting example, a software update. order-marked messages 106' and order-marked responses 107' to each other sequencer of the plurality of sequencers 450-1, . . . , 450-2 via each respective fourth direct connection 482. may be configured to allow the standby sequencer to take over for the currently active sequencer. However, the active sequencer, i.e., a given sequencer in the active state, need not forward ordered and marked messages to the standby sequencer, i.e., the sequencer in the standby state; alternatively, continuously, It should be understood that the journal, also known as the state log (which will contain the ordering information therein), may be broadcast/replicated to a standby sequencer such as disclosed above with respect to 1D.

Electronic trading system 100 may further include a system status log (not shown) as described above with reference to FIG. 1D. The active sequencer may be configured to transmit a system state log from the active sequencer to at least one other sequencer of the plurality of sequencers via a shared sequencer network. For example, a shared sequencer network may include a fourth direct connection 482 such that the system state log is logged from sequencer 450-1 to sequencer 450-2 when sequencer 450-1 is a given sequencer in the active state. can be sent to.

FIG. 5 is a flow diagram 500 of an example embodiment of a method for performing electronic transactions. The method begins (502) and sends a message representing an electronic trading request with a limit price to buy or sell a financial instrument from a gateway to a core compute node via a first direct connection (504). Messages, in turn, are received by core compute nodes to perform electronic trading functions in the electronic trading system. The method includes transmitting (506) a message from the gateway to a sequencer of an electronic trading system via a second direct connection, and in response transmitting an order-marked message from the sequencer to a core compute node via a third direct connection. and send it (508). The first, second and third direct connections have respective unshared bandwidths. A sequencer is inserted between the gateway and the core compute node via second and third direct connections. The order marked messages sent by the sequencer are ordered versions of the messages sent from the gateway over the second direct connection. Order marked messages, in turn, are received by core compute nodes. The method includes, at a core compute node, receiving another message from a gateway, receiving an order-marked version of the other message from a sequencer, and, at the core compute node, receiving another message received by the core compute node in an electronic trading system. A relative ranking of the order-marked messages is determined (510) among the order-marked versions of the order-marked messages. The core compute node completes the electronic transaction matching function for the electronic transaction request according to the determined relative ranking. Upon completion of this, the limit price is matched with the counter limit price for the financial product, and electronic trading of the financial product becomes possible. The method then ends (512) in the exemplary embodiment.

The message may be a gateway message. The method may further comprise receiving an incoming message from a participant device at the gateway. Sending the gateway message may include sending the gateway message by the gateway in response to receiving an incoming message by the gateway from a participant device. The order marked message may be a first order marked message. The method may further comprise transmitting the first order marked message from the sequencer to the gateway via the second direct connection. In contrast, the first order marked message may be received by the gateway. The method includes, in response to receiving the message, transmitting a core compute node message from the core compute node to the gateway via the first direct connection; and transmitting the core compute node message to the sequencer via the third direct connection. and, in response, transmitting a second order-marked message from the sequencer to the gateway via the second direct connection. The second order marked message is an ordered version of the core compute node message. The method includes determining, at the gateway, a relative ranking of a second order-marked message and order-marked versions of other messages sent from a core compute node to the gateway; and transmitting the outgoing message to a participant device. and transmitting the second order-marked message from the sequencer to the core compute node via the third direct connection. I can prepare.

The gateway may be a given gateway of a plurality of gateways, and the core compute node may be a given core compute node of a plurality of core compute nodes. The method may further comprise sending a respective compute node-directed message from each gateway of the plurality of gateways to all core compute nodes and the sequencer of the plurality of core compute nodes. The method includes the steps of: transmitting a respective gateway-bound message from each core compute node of the plurality of core compute nodes to all gateways of the plurality of gateways and a sequencer; and transmitting each compute node-bound message received at the sequencer and the respective and transmitting respective order-marked messages from the sequencer to the plurality of gateways and the plurality of core compute nodes in response to the gateway-directed message.

The sequencer may be a given sequencer among a plurality of sequencers. The method includes the steps of: sending a message addressed to each compute node from each gateway of the plurality of gateways to a given sequencer; and sending a message addressed to each gateway from each core compute node of the plurality of core compute nodes to the given sequencer. and sending an order-marked message from a given sequencer to each other of the plurality of sequencers.

The method includes receiving the same message at a plurality of core compute nodes, the same message being a message addressed to each compute node sent by a given gateway; The method may further include generating a response message at the plurality of core compute nodes, and receiving the response message from among the plurality of core compute nodes at a given gateway. The method may further comprise performing an action at a given gateway based on a given response message of the response messages generated in response to receiving the same message. A given response message may be the first to arrive at a given gateway relative to other response messages generated in response to receipt of the same message. The method may further comprise ignoring other response messages that arrive at the given gateway after the given response message. Thus, the "same message" refers to a single message that is broadcast from a single gateway to all Compute nodes such that each Compute node obtains one copy of the single message. say. Each of the multiple compute nodes then responds to the same single message, and multiple functionally equivalent responses are broadcast to all gateways, such that they are received by a given gateway. It can be. Gateways, including a given gateway, can then sort these multiple functionally equivalent messages on a "first come, first served" basis.

The method may comprise receiving, at a given compute node, messages destined for the plurality of compute nodes from at least two gateways of the plurality of gateways. Each Compute node-destined message of a plurality of Compute node-destined messages may represent a respective version of the same message, each version entering a respective one of the at least two gateways via a respective high availability flow. has been done. The method may further include performing an action at the given compute node based on the given compute node-directed message of the plurality of compute node-directed messages. A message destined for a given compute node may be the first one to arrive at the given compute node relative to other messages destined for the compute node among the plurality of messages destined for the compute node. The method may further comprise ignoring messages destined for other compute nodes that arrive at the given compute node after the message destined for the given compute node.

The method may further comprise matching trade orders for the financial instrument at the core compute node based on the electronic trade matching function performed. The method may further comprise maintaining a balance balance of the financial instrument in the order book, where the balance balance is an amount mismatch of the financial instrument resulting from the executed electronic trade matching function.

The method may further include synchronizing the gateway, the core compute node, and the sequencer based on a clock.

The message may be a gateway message. The method includes the steps of: providing service to at least one participant device at a gateway; receiving an incoming message at the gateway; and transmitting a gateway message from the gateway to a sequencer and a core compute node in response to receiving the incoming message at the gateway. The method may further include the step of: The incoming message may have been dispatched by at least one participant device. The method may further comprise generating an ordinal marked message by marking the message or representation thereof with a unique ordinal identifier.

The method may further comprise protecting the first direct connection, the second direct connection, and the third direct connection, or a subset thereof via at least one respective redundant direct connection.

The gateway is a given gateway of the plurality of gateways, the core compute node is a given core compute node of the plurality of core compute nodes, and the sequencer is a given sequencer of the plurality of sequencers. obtain. The method includes the steps of: enabling a plurality of gateways to communicate via a shared gateway network; enabling a plurality of core compute nodes to communicate via a shared core compute node network; communicable via a shared sequencer network or via a respective fourth direct connection.

The method includes the steps of transmitting a system state log from a given sequencer to at least one other sequencer of a plurality of sequencers via a shared sequencer network, or storing the system state log by a given sequencer in a data store. It may further include: A data store may be accessible to multiple sequencers via a shared sequencer network.

The electronic trading system may be an active electronic trading system. The method may include enabling at least one sequencer of the plurality of sequencers to communicate with a disaster recovery site. A disaster recovery site may include a standby electronic trading system. The standby electronic trading system is a replica of the active electronic trading system and may be configured to allow electronic trading to continue if the active electronic trading system fails.

FIG. 6 is a block diagram of an exemplary embodiment of a sequencer 650 of an electronic trading system, such as electronic trading system 100 disclosed above. In the exemplary embodiment, sequencer 650 is configured to communicate directly with gateway 620 via first direct connection 680a in a point-to-point mesh system, such as point-to-point mesh system 102 of electronic trading system 100 disclosed above. A first communication module 652 is provided. Sequencer 650 further comprises a second communication module 654 configured to communicate directly with core compute node 640 via a second direct connection 680b in a point-to-point mesh system in an electronic trading system. Sequencer 650 further includes sequencing logic 656 coupled to first communication module 652 and second communication module 654 .

The ordering logic 656 is configured to generate an order-marked message 606' by marking the message 606 or an indication thereof with a unique order identifier (not shown), such that the message 606 is associated with the first direct connection 680a or Received by first communication module 652 or second communication module 654 via second direct connection 680b. Ordering logic 656 is further configured to send order marked messages 606' to gateway 620 and core compute node 640 via first communication module 652 and second communication module 654, respectively.

According to an exemplary embodiment, the ordering logic is, for example, part of the logic of an FPGA or an ASIC in the fixed logic device 230 of FIG. 2 disclosed above. Additionally, both communication modules 652 and 654 may be implemented in an FPGA, such as the 10 GigE MAC core 260 disclosed above with reference to FIG. 2.

FIG. 7 is a block diagram of another exemplary embodiment of an electronic trading system, such as electronic trading system 100 of FIGS. 1B-1 and 1B-2, disclosed above. 1B-2 and FIG. 7, gateway 120-1, core compute node 140-1, sequencer 150-1, first direct connection 180-1-1, second direct connection 180-gw1-s1 and A third direct connection 180-c1-s1 constitutes a first point-to-point mesh system 702a. Electronic trading system 100 may be a first electronic trading system 700a communicatively coupled to proxy node 772. Proxy node 772 may further be communicatively coupled to at least one participant device 730 and second electronic trading system 700b. The second electronic trading system 700b may include a second point-to-point mesh system 702b, such as the point-to-point mesh system 102 disclosed above with respect to FIG. 1B-2.

Proxy node 772 is configured to send a message 703' to first electronic trading system 700a and second electronic trading system 700b in response to receiving an incoming message 703 from at least one participant device 730. . The first and second electronic trading systems may be configured to generate respective responses to messages sent by proxy node 772. The proxy node 772 further sends a response, i.e., an outgoing message 705, to at least one incoming response 705' received from the first electronic trading system 700a or the second electronic trading system 700b. The information may be configured to transmit to participant device 730. The first electronic trading system 700a and the second electronic trading system 700b may be synchronized (777), for example, based on a common clock that provides date and time, for example. However, the first electronic trading system 700a and the second electronic trading system 700b are not limited to being synchronized based on a common clock (777).

According to an exemplary embodiment, a single active/primary sequencer (not shown) for the entire system that includes both electronic trading system 700a and electronic trading system 700b is a It can be employed to adjust the number. For example, a single active/primary sequencer may adjust the sequence number such that an incoming response 705' is assigned the same sequence number by both electronic trading system 700a and electronic trading system 700b, thereby allowing the proxy Node 772 can appropriately determine the relative ranking of messages. For example, if there is a single active/primary sequencer in electronic trading system 700a, the active sequencer may be configured to communicate with electronic trading system 700b by communicating with a standby sequencer in electronic trading system 700b, for example; A sequence number is assigned to each message.

The above architectures, such as point-to-point mesh architectures, may be used in applications other than electronic trading systems. For example, to monitor data streams flowing through a network, capture packets, decode the raw data of the packets, analyze the contents of the packets in real time, and provide responses for purposes other than processing securities trading orders. It is possible that it can be used for

Additionally, the example embodiments disclosed herein may be implemented using a computer program product, for example, controls may be programmed into software for implementing the example embodiments. Further example embodiments may include a non-transitory computer-readable medium containing instructions executable by a processor that, when loaded and executed, cause the processor to complete the methods described herein. can. Elements of the block diagrams and flow diagrams may include one or more arrangements of the circuits of FIG. 2 disclosed above, or equivalents thereof, firmware, custom designed semiconductor logic, application specific integrated circuits (ASICs), field programmable gates, etc. It should be understood that it may be implemented in software or hardware, such as by an array (FPGA), a combination thereof, or other similar implementations to be determined in the future.

Furthermore, the elements of the block diagrams and flow diagrams described herein may be combined or divided in any manner in software, hardware, or firmware. If implemented in software, the software may be written in any language that may be compatible with the example embodiments disclosed herein. The software may be stored on any form of computer readable media, such as one or more random access memory (RAM), read only memory (ROM), compact disk read only memory (CD-ROM), and the like. In operation, a general or special purpose processor or processing core loads and executes software in a manner that is well understood in the art. Additionally, it is to be understood that the block diagrams and flow diagrams may include more or fewer elements, be arranged or oriented in different manners, or be represented differently. It should be understood that the examples may determine the number of block diagrams, flow diagrams and/or network diagrams and block diagrams and flow diagrams illustrating the implementation of the embodiments disclosed herein.

Accordingly, further embodiments may be implemented with a variety of computer architectures, physical, virtual, cloud computers, and/or any combination thereof; thus, the data processing systems described herein are exemplary. and is not intended as a limitation of the embodiments.

While the exemplary embodiments have been specifically shown and described, those skilled in the art will appreciate that various changes in form and detail may be made therein without departing from the scope of the embodiments encompassed by the appended claims. should be understood.

Claims (30)

An electronic trading system,
gateway and
a core compute node configured to perform electronic transaction matching functionality;
sequencer and
Equipped with
The gateway and the core compute node are coupled via a first direct connection, the gateway and the sequencer are coupled via a second direct connection, and the sequencer and the core compute node are coupled via a third direct connection. coupled via a direct connection, the first, second and third direct connections having respective non-shared bandwidths;
The gateway is configured to send a message representing an electronic trading request with a limit price to buy or sell a financial instrument to the core compute node via the first direct connection; is received by the core compute node, and the gateway is further configured to send the message to the sequencer via the second direct connection, and the sequencer receives an order-marked message therefrom. configured to transmit to the core compute node via the third direct connection, the sequencer inserted between the gateway and the core compute node via the second and third direct connections; The order-marked messages sent by the sequencer are ordered versions of the messages sent by the gateway, whereas the order-marked messages are received by the core compute node and The core compute node is configured to determine a relative ranking of the order-marked message among order-marked versions of other messages received by the core compute node in the electronic trading system, and the core compute node further completes the electronic transaction matching function for the electronic transaction request according to the determined relative ranking, matches the limit price with a counterparty limit price for the financial instrument, and enables electronic trading of the financial instrument. An electronic trading system configured as follows. 2. The electronic trading system of claim 1, wherein the message and the order marked message include the same user data, and the user data is associated with the electronic trading request. the message is a gateway message sent by the gateway in response to receipt of an incoming message received by the gateway from a participant device; via the second direct connection, wherein the order-marked message is received by the gateway, and the order-marked message is a first order-marked message; The core compute node further includes:
configured to send a core compute node message to the gateway via the first direct connection in response to receiving the gateway message;
The core compute node message is configured to send to the sequencer via the third direct connection, and the sequencer is further configured to send a second order marked message to the gateway via the second direct connection. configured to send over a connection, the second order-marked message being an ordered version of the core compute node message, the gateway further configured to:
configured to determine the relative ranking of order-marked versions of the second order-marked message and other messages sent from the core compute node to the gateway;
The sequencer is configured to send an outgoing message to the participant device, the outgoing message being sent in accordance with the determined relative ranking, and the sequencer further configured to send the second order marked message thereto. The electronic trading system of claim 1, configured to transmit to the core compute node via the third direct connection. the gateway is a given gateway of a plurality of gateways; the core compute node is a given core compute node of the plurality of core compute nodes;
each gateway of the plurality of gateways is coupled to each core compute node of the plurality of core compute nodes via a respective first direct connection;
The sequencer is coupled to each of the plurality of gateways via a respective second direct connection and to each of the plurality of core compute nodes via a respective third direct connection. and the plurality of gateways, the plurality of core compute nodes, the sequencer and their respective direct connections constitute at least a portion of a point-to-point mesh system, within the point-to-point mesh system:
each gateway of the plurality of gateways is configured to send messages destined for a respective compute node sent therefrom to all compute nodes of the plurality of core compute nodes and the sequencer; each said core compute node is configured to send messages sent therefrom addressed to a respective gateway to all said gateways and said sequencer of said plurality of gateways;
The sequencer is further configured to send each of the order-marked messages to the plurality of gateways and the plurality of core compute nodes in response to receiving a message directed to the respective compute node or a message directed to the respective gateway. The electronic trading system according to claim 1, wherein the electronic trading system is the sequencer is a given sequencer among a plurality of sequencers in the point-to-point mesh system;
each said gateway of said plurality of gateways is coupled to each said sequencer of said plurality of sequencers via a respective said second direct connection;
each of the core compute nodes of the plurality of core compute nodes is coupled to each of the sequencers of the plurality of sequencers via a respective third direct connection;
The given sequencer is the currently active sequencer serving the point-to-point mesh system, and each other sequencer of the plurality of sequencers is a standby sequencer waiting to take over the currently active sequencer. can be,
each said sequencer of said plurality of sequencers is coupled to each other of said plurality of sequencers via a respective fourth direct connection;
Each gateway of the plurality of gateways is further configured to send a message addressed to a respective compute node to the given sequencer of the plurality of sequencers;
Each core compute node of the plurality of core compute nodes is further configured to send a respective gateway-directed message to the given sequencer of the plurality of sequencers;
The given sequencer is further configured to transmit the order marked message to each other sequencer of the plurality of sequencers via each respective fourth direct connection if the currently active sequencer fails. 5. The electronic trading system of claim 4, wherein the standby sequencer is configured to allow the currently active sequencer to take over. the respective compute node-directed messages sent by the given gateway are the same messages received by the plurality of core compute nodes, and the plurality of core compute nodes, in response to receiving the same message, configured to generate a response message, the response message being received at the given gateway from among the plurality of core compute nodes, the given gateway further comprising:
configured to take action based on a given one of the response messages generated in response to receipt of the same message, the given response message being generated in response to reception of the same message; arriving at said given gateway first relative to other response messages, said given gateway further comprising:
5. The electronic trading system of claim 4, configured to ignore the other response messages that arrive after the given response message. Messages addressed to multiple compute nodes representing the same message are received from among the multiple gateways at the given compute node, and the given compute node further:
The device is configured to take action based on a given Compute node-directed message of the plurality of Compute node-directed messages, where the given Compute node-directed message is one of the plurality of Compute node-directed messages representing the same message. arrives at the given Compute node first relative to messages addressed to other Compute nodes, and the given Compute node further:
5. The electronic trading system of claim 4, configured to ignore messages destined for the other compute nodes that arrive after the message destined for the given compute node. further comprising an order book accessible by the core compute node, the core compute node further comprising:
configured to match trade orders regarding the financial instrument using the electronic trade matching functionality;
configured to maintain a balance balance of the financial instrument in the order book, wherein an amount mismatch of the financial instrument results from performing the electronic transaction matching function; The electronic trading system according to claim 1, wherein the electronic trading system includes the amount mismatch of. The electronic trading system of claim 1, further comprising a clock, and wherein the gateway, the core compute node, and the sequencer are synchronized based on the clock. The gateway further includes:
configured to provide service to at least one participant device;
the sequencer is configured to send the message to the sequencer and the core compute node in response to receiving an incoming message at the gateway, the incoming message is routed by the at least one participant device, and the sequencer further comprises: said message by marking said message with an identifier, or by creating an indication of said message received, marking said indication with said unique ordinal identifier, and sending said marked indication; 2. The electronic trading system of claim 1, configured to generate an order-marked message, and wherein the marked representation is the order-marked message. 2. The electronic trading system of claim 1, further comprising at least one respective redundant direct connection for the first direct connection, the second direct connection, and the third direct connection, or a subset thereof. the gateway is a given gateway of a plurality of gateways communicatively coupled to each other via a shared gateway network;
The core compute node is a given core compute node among a plurality of core compute nodes communicatively coupled to each other via a shared core compute node network;
2. The electronic transaction of claim 1, wherein the sequencer is a given sequencer of a plurality of sequencers communicatively coupled to each other via a shared sequencer network or via a respective fourth direct connection. system. further comprising a system state log, the given sequencer transmitting the system state log to at least one other sequencer of the plurality of sequencers via the shared sequencer network, or transmitting the system state log to data. 13. The electronic trading system of claim 12, configured to store data in a store, the data store being accessible to the plurality of sequencers via the shared sequencer network. The electronic trading system is an active electronic trading system, and at least one of the sequencers of the plurality of sequencers is communicatively coupled to a disaster recovery site, the disaster recovery site including a standby electronic trading system, and the standby electronic trading system. 13. The electronic trading system of claim 12, wherein the system is a replica of the active electronic trading system and is configured to allow electronic trading to continue in the event of a failure of the active electronic trading system. the gateway, the core compute node, the sequencer and the first, second and third direct connections constitute a first point-to-point mesh system;
The electronic trading system is a first electronic trading system communicatively coupled to a proxy node, the proxy node further communicatively coupled to at least one participant device and a second electronic trading system; the second electronic trading system includes a second point-to-point mesh system;
The proxy node is configured to transmit the message to the first and second electronic trading systems in response to receiving an incoming message from the at least one participant device, The trading system is configured to generate respective responses to the messages sent by the proxy node, and the proxy node further generates respective responses to the messages generated and received from the first or second electronic trading system. The electronic trading system of claim 1, configured to transmit a response to the at least one participant device in response to receiving a first of the responses to arrive. A method of performing an electronic transaction, the method comprising:
sending a message representing an electronic trading request with a limit price to buy or sell a financial instrument from the gateway to the core compute node via the first direct connection;
In contrast, receiving the message at the core compute node to perform an electronic trading function in an electronic trading system;
transmitting the message from the gateway to a sequencer in the electronic trading system via a second direct connection, and transmitting an order-marked message from the sequencer to the core compute node via a third direct connection; the first, second and third direct connections have respective non-shared bandwidths, and the sequencer connects the second and third direct connections between the gateway and the core compute node. the order-marked message inserted over a connection and sent by the sequencer is an ordered version of the message sent from the gateway over the second direct connection;
In contrast, receiving the order-marked message at the core compute node;
receiving, at the core compute node, another message from the gateway, and receiving, at the core compute node, an order-marked version of the other message from the sequencer;
determining at the core compute node a relative ranking of the order marked message among the order marked versions of the other messages received by the core compute node in the electronic trading system;
completing, in the core compute node, an electronic transaction matching function for the electronic transaction request according to the determined relative ranking, the completing step including determining the limit price and the counterparty limit price for the financial instrument; matching and enabling electronic trading of the financial instrument;
How to prepare. 17. The method of claim 16, wherein the message and the order marked message include the same user data, and the user data is associated with an electronic transaction request. The message is a gateway message, and the method further comprises receiving an incoming message from a participant device at the gateway, and transmitting the gateway message includes receiving an incoming message from the participant device by the gateway. transmitting the gateway message by the gateway in response to receiving a message, the order marked message being a first order marked message, the method comprising:
transmitting the first order-marked message from the sequencer to the gateway via the second direct connection, wherein the first order-marked message is received by the gateway; step,
In response to receiving the message, transmitting a core compute node message from the core compute node to the gateway via the first direct connection;
sending the core compute node message to the sequencer via the third direct connection, and in response sending a second order marked message from the sequencer to the gateway via the second direct connection; the second order marked message is an ordered version of the core compute node message;
determining, at the gateway, the relative ranking of the second order-marked message and the order-marked versions of other messages sent from the core compute node to the gateway;
sending an outgoing message to the participant device, the outgoing message being sent in response to the determined relative ranking;
sending the second order-marked message from the sequencer to the core compute node via the third direct connection;
17. The method of claim 16, further comprising: the gateway is a given gateway of a plurality of gateways, the core compute node is a given core compute node of a plurality of core compute nodes, and the method includes:
transmitting a message addressed to each compute node sent from each of the plurality of gateways to all of the core compute nodes and the sequencer of the plurality of core compute nodes;
transmitting a message addressed to each gateway sent from each core compute node of the plurality of core compute nodes to all of the gateways and the sequencer of the plurality of gateways;
transmitting respective order-marked messages from the sequencer to the plurality of gateways and the plurality of core compute nodes in response to the respective messages addressed to the respective compute nodes and the respective gateways received at the sequencer; and,
17. The method of claim 16, further comprising: The sequencer is a given sequencer among a plurality of sequencers, and the method includes:
transmitting a respective message addressed to a compute node sent from each gateway of the plurality of gateways to the given sequencer;
transmitting a message addressed to the respective gateway sent from each core compute node of the plurality of core compute nodes to the given sequencer;
transmitting the order marked message from the given sequencer to each other of the plurality of sequencers;
20. The method of claim 19, further comprising: receiving the same message at the plurality of core compute nodes, the same message being a message sent by the given gateway and addressed to the respective compute node;
generating a response message at the plurality of core compute nodes in response to receiving the same message;
receiving, at the given gateway, the response message from among the plurality of core compute nodes;
performing an action at the given gateway based on a given response message of the response messages generated in response to receipt of the same message, the given response message being the first of the response messages generated in response to receiving a message to arrive at the given gateway relative to other response messages;
ignoring the other response messages arriving at the given gateway after the given response message;
20. The method of claim 19, further comprising: receiving, at the given compute node, a plurality of compute node-directed messages from among the plurality of gateways, the plurality of compute node-directed messages representing the same message;
performing an action at the given compute node based on a given compute node addressed message of the plurality of compute node addressed messages, the given compute node addressed message being the first to arrive at the given compute node relative to other compute node-bound messages of the plurality of compute node-bound messages representing;
ignoring messages destined for the other compute nodes that arrive at the given compute node after the messages destined for the given compute node;
20. The method of claim 19, further comprising: matching trade orders regarding the financial instrument at the core compute node based on the electronic trade matching function performed;
maintaining a balance of the financial instrument in an order book, the balance being an amount mismatch of the financial instrument resulting from the executed electronic trade matching function;
17. The method of claim 16, further comprising: 17. The method of claim 16, further comprising synchronizing the gateway, the core compute node, and the sequencer based on a clock. The message is a gateway message, and the method includes:
servicing at least one participant device at the gateway;
receiving an incoming message at the gateway;
transmitting the gateway message from the gateway to the sequencer and the core compute node in response to receiving the incoming message at the gateway, the incoming message being dispatched by the at least one participant device; There are steps and
generating the ordinal marked message by marking the message or representation thereof with a unique ordinal identifier;
17. The method of claim 16, further comprising: 17. The method of claim 16, further comprising protecting the first direct connection, the second direct connection, and the third direct connection or a subset thereof via at least one respective redundant direct connection. Method described. The gateway is a given gateway of a plurality of gateways, the core compute node is a given core compute node of a plurality of core compute nodes, and the sequencer is a given gateway of a plurality of sequencers. a sequencer, and the method includes:
enabling the plurality of gateways to communicate via a shared gateway network;
enabling the plurality of core compute nodes to communicate via a shared core compute node network;
enabling the plurality of sequencers to communicate via a shared sequencer network or via respective fourth direct connections;
17. The method of claim 16, further comprising: transmitting a system state log from the given sequencer to at least one other sequencer of the plurality of sequencers via the shared sequencer network, or storing the system state log by the given sequencer in a data store; 28. The method of claim 27, further comprising the step of: wherein the data store is accessible to the plurality of sequencers via the shared sequencer network. The electronic trading system is an active electronic trading system, and the method includes:
enabling at least one sequencer of the plurality of sequencers to communicate with a disaster recovery site, the disaster recovery site including a standby electronic trading system, the standby electronic trading system communicating with the active 28. The method of claim 27, further comprising the step of being a replica of an active electronic trading system and configured to allow electronic trading to continue if the active electronic trading system fails. A sequencer for an electronic trading system, comprising a non-transitory computer readable medium on which a series of instructions is encoded, and when the series of instructions is read into and executed by the sequencer, the sequencer:
communicating directly with a gateway via a first direct connection in a point-to-point mesh system of the electronic trading system;
direct communication with a core compute node via a second direct connection in the point-to-point mesh system of the electronic trading system, the sequencer connecting the first and second direct connections between the gateway and the core compute node; inserted via the first and second direct connections having respective non-shared bandwidths;
Marking a message or an indication thereof with a unique ordering identifier generates an order marked message, which message is received by the sequencer via the first or second direct connection, respectively, and the sequence of instructions The sequencer further causes the sequencer to send the order marked message to the gateway and the core compute node.