HK40081215B - Reconfigurable computing pods using optical networks with one-to-many optical switches - Google Patents

Description

Reconfigurable computing platform using optical networks with one-to-many optical switches

This application is a divisional application of the following application.

The original application number was 202010617949.1

The original application was filed on July 1, 2020.

Original application title: Reconfigurable computing platform using an optical network with one-to-many optical switches

Background Technology

Some computational workloads (such as machine learning training) require a large number of processing nodes to efficiently complete the task. These processing nodes can communicate with each other through an interconnected network. For example, in machine learning training, processing nodes can communicate with each other to converge to the optimal deep learning model. The interconnected network is crucial for the speed and efficiency of convergence for the processing units.

Because machine learning and other tasks vary in size and complexity, the rigid architecture of supercomputers, which include multiple processing nodes, can limit their availability, scalability, and performance. For example, if some processing nodes in a supercomputer with a rigid interconnect network connecting a particular arrangement of processing nodes fail, the supercomputer may not be able to replace those nodes, leading to reduced availability and performance. Certain arrangements may also result in higher performance than others, regardless of the failed nodes.

Summary of the Invention

This specification describes the technology involving a superpod with reconfigurable compute nodes, using an optical network to generate workload clusters from the superpod.

Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method including receiving request data for requested compute nodes specifying a computational workload. The request data specifies a target arrangement of compute nodes. A subset of the building blocks is selected from a superplatform comprising a set of building blocks, each building block comprising compute nodes arranged in m dimensions. Each building block is connected to an optical network comprising two or more optical path switches (OCS) for each of the m dimensions. For each of the m dimensions, each building block comprises one or more segments of compute nodes interconnected along the dimension. Each segment comprises a first compute node at a first end of the segment and a second compute node at a second end of the segment opposite the first end. For each of the m dimensions, a first portion of the first compute node is connected to a first OCS switch among the two or more OCS switches for that dimension, one or more additional portions of the first compute node are connected to corresponding additional OCS switches among the two or more OCS switches for that dimension, and the second compute node of each segment is connected to the input of a corresponding one-to-many optical switch having inputs and multiple outputs. A first output is connected to a first OCS switch, and for each additional portion of the first compute node, a corresponding additional output is connected to an additional OCS switch for the additional portion of the first compute node. A logical arrangement of a subset of compute nodes matching a target arrangement of compute nodes is determined. For each dimension in m dimensions, the logical arrangement defines the connection between a segment of each building block and a corresponding segment of one or more other building blocks. A workload cluster of compute nodes is generated, comprising a subset of building blocks and connected to each other based on the logical arrangement. Generation includes configuring corresponding routing data for each dimension of the workload cluster for each of two or more OCS switches for that dimension. The corresponding routing data for each dimension of the workload cluster specifies how the data of the computational workload is routed between compute nodes along the dimension of the workload cluster. The generation also includes configuring at least a portion of the one-to-many switches based on the logical arrangement such that a second compute node in each segment of compute nodes is connected to the same OCS switch as the corresponding first compute node in the corresponding segment to which the second compute node in the logical arrangement is connected. The compute nodes of the workload cluster are then made to perform the computational workload. Other implementations of this and other aspects include corresponding systems, methods, and computer programs configured to perform actions of methods encoded on a computer storage device. A system of one or more computers can be configured by means of software, firmware, hardware, or a combination thereof installed on the system to cause the system to perform actions during operation. One or more computer programs can be configured by instructions that, when executed by a data processing device, cause the device to perform the actions.

These and other implementations may each optionally include one or more of the following features. In some aspects, at least a portion of a one-to-many switch is configured based on a logical arrangement such that a second compute node in each segment of a compute node is connected to the same OCS switch as the corresponding first compute node in the corresponding segment to which the second compute node in the logical arrangement is connected. The configuration may include identifying second building blocks in a subset that are adjacent to the first building blocks along a specific dimension for a first building block in the subset, and for each segment of the first building block along the specific dimension, identifying the corresponding segment of the second building block, identifying the OCS switch to which the first compute node of the corresponding segment of the second building block is connected, and configuring the one-to-many switch to which the segment is connected to connect the second compute node of the segment to the identified OCS switch.

In some aspects, identifying the corresponding segment of the second building block includes identifying a segment of the second building block that is along the specific dimension in the logical arrangement, along the same logical axis as the segment of the first building block.

In some respects, one or more additional portions of the first computing node are an additional portion of the first computing node, and a one-to-many optical switch is a one-to-two optical switch having one input and two outputs; a first portion of the first computing node includes half of the first computing node, and an additional portion of the first computing node includes half of the first computing node.

In some aspects, the request data specifies different types of compute nodes, and the selection of a subset of building blocks includes selecting building blocks containing one or more compute nodes of the specified type for each type of compute node specified by the request data.

In some aspects, the corresponding routing data for each dimension of the super platform may include an OCS switch routing table for each of the two or more OCS switches in the dimension. In some aspects, each building block may include a three-dimensional torus of a compute node or one of the meshes of compute nodes.

In some respects, the super platform comprises multiple workload clusters, each of which includes a different subset of building blocks and performs workloads that are different from each of the other workload clusters.

Some aspects include receiving data indicating that a given building block of the workload cluster has failed, and replacing the given building block with an available building block. Replacing the given building block with an available building block may include updating the routing data of one or more optical path switches of the optical network to stop routing data between the given building block and one or more other building blocks in the workload cluster, and updating the routing data of the one or more optical path switches of the optical network to route data between the available building block and the one or more other building blocks in the workload cluster. In some aspects, the target arrangement of compute nodes includes an n-dimensional arrangement of compute nodes, where n is greater than or equal to 2.

The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. Clusters that dynamically configure compute nodes for workloads using optical networks result in higher availability of compute nodes because other compute nodes can easily replace failed or offline compute nodes. Workload clusters can be configured from a super platform that includes compute nodes connected to an optical network. The flexibility in compute node arrangement leads to higher performance and more efficient allocation of the appropriate number and arrangement of compute nodes optimized (or improved) for each workload. Utilizing a super platform that includes multiple types of compute nodes connected using optical networks, workload clusters can be generated that include not only an appropriate number and arrangement of compute nodes, but also appropriate types of compute nodes for each workload, e.g., and not only compute nodes physically close to each other in a data center or other location.

Using optical networks to configure workload clusters also provides fault isolation and enhanced security for workloads. For example, some traditional supercomputers route traffic between the various computers that make up the supercomputer. If one of these computers fails, the communication path is lost. With optical networks, data can be quickly rerouted and/or available compute nodes can replace (e.g., substitute for) a failed compute node. For example, by reconfiguring an Optical Path Switch (OCS), another compute node in a superplatform can be connected to other compute nodes in the workload cluster. Furthermore, the physical isolation between workloads provided by OCS switches (e.g., physical isolation between different optical paths) offers better security between various workloads performing within the same superplatform compared to managing separation using vulnerable software.

Compared to packet-switched networks, using optical networks to connect building blocks can also reduce latency in data transmission between building blocks. For example, in packet switching, there is additional latency because packets need to be received by a switch, buffered, and retransmitted on another port. Using OCS switches to connect building blocks provides a true end-to-end optical path without intermediate packet switching or buffering.

Optical networks can include one-to-many optical switches to increase the size of a superplatform for a given size of OCS switches. This, in turn, allows for the generation of larger workload clusters from the compute nodes of the superplatform for a given size of OCS switches. Similarly, for a given size of superplatform, using one-to-many optical switches can reduce the number of OCS ports used on each OCS switch.

The various features and advantages of the foregoing subject matter will now be described with reference to the accompanying drawings. Other features and advantages will be apparent from the subject matter and claims described herein.

Attached Figure Description

Figure 1 is a block diagram of an environment in which an exemplary processing system generates a workload cluster of compute nodes and uses the workload cluster to perform compute workloads.

Figure 2 illustrates an exemplary logical superplatform and an exemplary workload cluster generated from a portion of the building blocks in the superplatform.

Figure 3 illustrates an exemplary building block and an exemplary workload cluster generated using that building block.

Figure 4 shows an example optical link from a compute node to an optical path switching (OCS) switch.

Figure 5 shows the logic computation tray used to form building blocks.

Figure 6 shows a sub-block of an exemplary building block with one dimension omitted.

Figure 7 shows an exemplary building block.

Figure 8 shows the OCS structure topology used for the super platform.

Figure 9 shows the components of the example super platform.

Figure 10 is a flowchart illustrating an exemplary process for generating a workload cluster and using the workload cluster to perform computational workloads.

Figure 11 is a flowchart illustrating an exemplary process for reconfiguring an optical network to replace a failed building block.

Figure 12 shows a portion of an example super platform including building blocks and a 1×2 optical switch.

Figure 13 illustrates an exemplary workload cluster.

Figure 14 is a flowchart illustrating an exemplary process for generating a workload cluster and using the workload cluster to perform computational workloads.

In the various figures, the same reference numerals and designations denote the same elements.

Detailed Implementation

Typically, the systems and techniques described herein can be configured with optical network architectures to generate workload clusters of compute nodes from a superplatform comprising multiple building blocks of compute nodes connected via an optical network. For example, the superplatform may comprise a set of interconnected building blocks. Each building block may comprise multiple compute nodes arranged in m dimensions (e.g., two-dimensional or three-dimensional arrangement).

Users can specify a target arrangement of compute nodes for a given workload. For example, a user can provide a machine learning workload and specify a target arrangement of compute nodes for machine learning computations. The target arrangement can define the number of compute nodes in each of the n dimensions, for example, where n is greater than or equal to 2. That is, the target arrangement can define the size and shape of the workload cluster. For example, some machine learning models and computations perform better on non-square topologies.

Cross-sectional bandwidth can also be a limitation on the overall computation; for example, compute nodes waiting for data transfer leave idle computation cycles. Depending on how the work is distributed across compute nodes and how much data needs to be transferred over the network in different dimensions, the shape of the workload cluster can impact the performance of the compute nodes within it.

For workloads with all-to-all compute node data traffic, a cube-shaped workload cluster reduces the number of hops between compute nodes. If a workload has a large amount of local communication, then transfers data to adjacent groups of compute nodes in a specific dimension, and the workload invokes many of these adjacent communications linked together, the workload can benefit from having a larger arrangement of compute nodes in that specific dimension than in other dimensions. Therefore, enabling users to specify the arrangement of compute nodes in a workload cluster allows users to specify an arrangement that can lead to better performance for their workload.

If different types of compute nodes are included in the super platform, the request can also specify the number of each type of compute node to be included in the workload cluster. This allows users to specify the arrangement of compute nodes that perform better for a particular workload.

The workload scheduler can select building blocks for a workload cluster based, for example, on the availability of building blocks, the health status of building blocks (e.g., working or failed), and/or the priority of workloads in the superplatform (e.g., the priority of workloads performed by or to be performed by compute nodes of the superplatform). The workload scheduler can provide data to the Optical Path Switch (OCS) manager identifying the selected building blocks and their target arrangement. The OCS manager can then configure one or more OCS switches of the optical network to connect the building blocks together, thereby forming a workload cluster. The workload scheduler can then execute computational workloads on the compute nodes of the workload cluster.

If a building block in the workload cluster fails, it can be quickly replaced with another building block by simply reconfiguring the OCS switches. For example, the workload scheduler can select an available building block from the super platform to replace the failed one. The workload scheduler can then instruct the OCS manager to replace the failed building block with the selected building block. The OCS manager can then reconfigure the OCS switches so that the selected building block is connected to other building blocks in the workload cluster, and the failed building block is no longer connected to any other building blocks in the workload cluster.

Figure 1 is a block diagram of environment 100, in which an exemplary processing system 130 generates a cluster of computational nodes and performs computational workloads using the cluster of workloads. The processing system 130 can receive computational workloads 112 from user device 110 via a data communication network 120 (e.g., a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof). Example workloads 112 include software applications, machine learning models (e.g., training and/or using machine learning models), video encoding and decoding, and digital signal processing workloads, etc.

The user can also specify the requested cluster 114 of compute nodes for workload 112. For example, the user can specify the target shape and size of the requested cluster of compute nodes. That is, the user can specify the number and shape of compute nodes in multiple dimensions. For example, if the compute nodes are distributed across three dimensions x, y, and z, the user can specify multiple compute nodes in each dimension. The user can also specify one or more types of compute nodes to be included in the cluster. As described below, the processing system 130 may include different types of compute nodes.

As described below, the processing system 130 can use building blocks to generate a workload cluster that matches the target shape and size of the cluster. Each building block can include multiple computational nodes arranged in m dimensions (e.g., 3 dimensions or other suitable number of dimensions). Therefore, the user can specify the target shape and size based on the number of building blocks in each dimension. For example, the processing system 130 can provide the user device 110 with a user interface that allows the user to select the maximum number of building blocks in each dimension.

User equipment 110 can provide workload 112 and data specifying the requested cluster 114 to processing system 130. For example, user equipment 110 can provide requested data, including workload 112 and data specifying the requested cluster 114, to processing system 130 via network 120.

Processing system 130 includes a unit scheduler 140 and one or more units 150. A unit 150 is a group of one or more superplatforms. For example, the unit 150 shown includes four superplatforms 152-158. Each superplatform 152-158 includes a set of building blocks 160, also referred to herein as a building block pool. In this example, each superplatform 152-158 includes 64 building blocks 160. However, superplatforms 152-158 may include other numbers of building blocks 160, such as 20, 50, 100, or another suitable number. Superplatforms 152-158 may also include different numbers of building blocks 160. For example, superplatform 152 may include 64 building blocks, while superplatform 154 includes 100 building blocks.

As described in more detail below, each building block 160 may include multiple computation nodes logically arranged in two or more dimensions. For example, building block 160 may include 64 computation nodes arranged along three dimensions, with four computation nodes in each dimension. This arrangement of computation nodes is referred to herein as a 4×4×4 building block, with four computation nodes along the x-dimensional axis, four along the y-dimensional axis, and four along the z-dimensional axis. Other numbers of dimensions (e.g., two dimensions) and other numbers of computation nodes in each dimension are also possible, such as 3×1, 2×2×2, 6×2, 2×3×4, etc.

Building blocks can also include a single compute node. However, as described below, to generate a workload cluster, optical links between building blocks are configured to connect the building blocks together. Therefore, while smaller building blocks (e.g., those with a single compute node) may offer greater flexibility in generating workload clusters, smaller building blocks may require more OCS switch configurations and more optical network components (e.g., cabling and switches). The number of compute nodes in a building block can be chosen based on a trade-off between the desired flexibility of the workload cluster and the need to connect building blocks together to form a workload cluster, and the required number of OCS switches.

Each computing node in building block 160 may include an application-specific integrated circuit (ASIC), such as a Tensor Processing Unit (TPU), a Graphics Processing Unit (GPU), or other types of processing units for machine learning workloads. For example, each computing node may be a single processor chip that includes processing units.

In some implementations, all building blocks 160 in the superplatform have the same compute nodes. For example, superplatform 152 may include 64 building blocks, each with 64 TPUs arranged in a 4×4×4 pattern for performing machine learning workloads. The superplatform may also include different types of compute nodes. For example, superplatform 154 may include 60 building blocks with TPUs and 4 building blocks with dedicated processing units for performing tasks other than machine learning workloads. In this way, workload clusters for workloads may include different types of compute nodes. The superplatform may include multiple building blocks of each type of compute node in the superplatform for redundancy and/or to allow multiple workloads to run in the superplatform.

In some implementations, all building blocks 160 in the super platform have the same arrangement, such as the same size and shape. For example, each building block 160 of the super platform 152 may have a 4×4×4 arrangement. The super platform may also have building blocks with different arrangements. For example, the super platform 154 may have 32 building blocks arranged in a 4×4×4 arrangement and 32 building blocks arranged in a 16×8×16 arrangement. Different building block arrangements may have the same or different compute nodes. For example, building blocks with TPUs may have a different arrangement than building blocks with GPUs.

The superplatform can have building blocks of different levels. For example, superplatform 152 can include basic-level building blocks with a 4×4×4 arrangement. Superplatform 152 can also include intermediate-level building blocks with more compute nodes. For example, intermediate-level building blocks can have an 8×8×8 arrangement, for example, consisting of 8 basic-level building blocks. In this way, a larger workload cluster can be generated using intermediate-level building blocks with fewer links compared to basic-level building blocks being connected to generate a larger workload cluster. The presence of basic-level building blocks in the superplatform also provides flexibility for smaller workload clusters that may not require the number of compute nodes found in the intermediate-level building blocks.

The superplatforms 152-158 within unit 150 may have the same or different types of compute nodes in the building blocks. For example, unit 150 may include one or more superplatforms with TPU building blocks and one or more superplatforms with GPU building blocks. The size and shape of the building blocks may also be the same or different in the different superplatforms 152-158 of unit 150.

Each unit 150 also includes a shared data storage 162 and a shared auxiliary computing component 164. Each superplatform 152-158 in unit 150 may use the shared data storage 162 to, for example, store data generated by workloads performed in superplatform 152-158. The shared data storage 162 may include a hard disk drive, solid-state drive, flash memory, and/or other suitable data storage device. The shared auxiliary computing component 164 may include a CPU (e.g., a general-purpose CPU machine), GPU, and/or other accelerators (e.g., video decoder, image decoder, etc.) shared within unit 150. The auxiliary computing component 164 may also include storage devices, memory devices, and/or other computing components that can be shared by computing nodes via a network.

Unit scheduler 140 can select units 150 and/or superplatforms 152-158 of units 150 for each workload received from user equipment 110. Unit scheduler 140 can select superplatforms based on a target permutation specified for the workload, the availability of building blocks 160 in superplatforms 152-158, and the health status of the building blocks in superplatforms 152-158. For example, for a workload, unit scheduler 140 can select superplatforms that include at least a sufficient number of available and healthy building blocks to generate a workload cluster with a target permutation. If the request data specifies a type of compute node, unit scheduler 140 can select a superplatform with at least a sufficient number of available and healthy building blocks of the specified type of compute node.

As described below, each superplatform 152-158 may also include a workload scheduler and an OCS manager. When unit scheduler 140 selects a superplatform for unit 150, unit scheduler 140 may provide the workload and data specifying the requested cluster to the workload scheduler of that superplatform 150. As described in more detail below, the workload scheduler may select a set of building blocks from the superplatform's building blocks to connect to form a workload cluster based on the availability and health of the building blocks and optionally based on the priority of the workloads in the superplatform. For example, as described below, if the workload scheduler receives a request for a workload cluster that includes more building blocks than the number of healthy and available building blocks in the superplatform, the workload scheduler may reallocate the building blocks of lower-priority workloads to the requested workload cluster. The workload scheduler may provide data identifying the selected building blocks to the OCS manager. The OCS manager may then configure one or more OCS switches to connect the building blocks together to form a workload cluster. The workload scheduler may then execute the workload on the compute nodes of the workload cluster.

In some implementations, for example, when selecting superplatforms 152-158 for a workload, unit scheduler 140 balances the load between individual units 150 and superplatforms 152-158. For example, when selecting between two or more superplatforms with the capacity for building blocks for a workload, unit scheduler 140 may select the superplatform with the maximum capacity, such as the superplatform with the most available and healthy building blocks, or the superplatform with the unit with the maximum total capacity.

In some implementations, the unit scheduler 140 may also determine a target arrangement for the workload. For example, the unit scheduler 140 may determine the target arrangement of building blocks based on the estimated computational requirements of the workload and the throughput of one or more types of available compute nodes. In this example, the unit scheduler 140 may provide the determined target arrangement to the workload scheduler of the super platform.

Figure 2 illustrates an example logical superplatform 210 and example workload clusters 220, 230, and 240 generated from a subset of building blocks in superplatform 210. In this example, superplatform 210 comprises 64 building blocks, each arranged in a 4×4×4 pattern. Although many examples described in this document are based on 4×4×4 building blocks, the same techniques can be applied to other arrangements of building blocks.

In Super Platform 210, building blocks, indicated by shaded lines, are assigned to workloads as described below. Building blocks indicated by solid white lines are healthy and available. Building blocks indicated by solid black lines are unhealthy nodes, which, for example, cannot be used to generate workload clusters due to failures.

Workload cluster 220 is an 8×8×4 platform, comprising four of the four×4×4 building blocks from super platform 210. That is, workload cluster 220 has eight compute nodes along the x-dimensional axis, eight compute nodes along the y-dimensional axis, and four compute nodes along the z-dimensional axis. Since each building block has four compute nodes along each dimension, workload cluster 220 comprises two building blocks along the x-dimensional axis, two building blocks along the y-dimensional axis, and one building block along the z-dimensional axis.

The four building blocks of workload cluster 220 are shown with diagonal shading to indicate their positions within super platform 210. As shown, the building blocks of workload cluster 220 are not adjacent to each other. As described in more detail below, the use of optical networking enables the generation of workload clusters from any combination of workload clusters within super platform 210, regardless of their relative positions within super platform 210.

Workload cluster 230 is an 8×8×8 platform comprising eight building blocks from superplatform 210. Specifically, the workload cluster includes two building blocks along each dimension, providing eight compute nodes along each dimension to workload cluster 230. The building blocks of workload cluster 230 are shown with vertical shading to indicate their location within superplatform 210.

Workload cluster 240 is a 16×8×16 platform comprising 32 building blocks from superplatform 210. Specifically, workload cluster 240 includes four building blocks along the x-dimensional plane, two building blocks along the y-dimensional plane, and four building blocks along the z-dimensional plane, providing the workload cluster with 16 compute nodes along the x-dimensional plane, eight compute nodes along the y-dimensional plane, and 16 compute nodes along the z-dimensional plane. The building blocks of workload cluster 240 are shown with cross-hatching to indicate their location within superplatform 210.

Workload clusters 220, 230, and 240 are merely some examples of clusters that can be generated for the super platform 210 of the workload. Many other arrangements of workload clusters are also possible. Although the exemplary workload clusters 220, 230, and 240 have a rectangular shape, other shapes are also possible.

The workload clusters 220, 230, and 240 have logical rather than physical shapes. The optical network is configured so that the building blocks communicate along each dimension as if the workload clusters were physically connected in a logical configuration. However, the physical building blocks and their corresponding compute nodes can be physically arranged in the data center in various ways. The building blocks of workloads 220, 230, and 240 can be selected from any healthy available building blocks without restricting the physical relationships between building blocks in the super platform 210, except that the building blocks are all connected to the optical network of the super platform 210. For example, as described above and in Figure 2, workload clusters 220, 230, and 240 include physically non-adjacent building blocks.

Furthermore, the logical arrangement of workload clusters is not limited by the physical arrangement of the building blocks of the super platform. For example, building blocks can be arranged in eight rows and eight columns, with only one building block along the z-dimensional direction. However, by configuring the optical network to create this logical arrangement, workload clusters can be configured to include multiple building blocks along the z-dimensional direction.

Figure 3 illustrates an example building block 310 and example workload clusters 320, 330, and 340 generated using building block 310. Building block 310 is a 4×4×4 building block with four compute nodes along each dimension. In this example, each dimension of building block 310 comprises 16 segments, each containing four compute nodes. For example, there are 16 compute nodes at the top of building block 310. For each of these 16 compute nodes, there exists a segment along the y-dimensional dimension comprising the compute node and three other compute nodes, the three other compute nodes being the corresponding last compute node at the bottom of building block 310. For example, one segment along the y-dimensional dimension comprises compute nodes 301-304.

Each compute node is along a logical axis. For example, compute nodes 301-304 are along a logical axis, and the four compute nodes to the right of compute nodes 301-304 are along different logical axes. Compute nodes 305-308 are also along different logical axes. As shown in Figure 3, the 4×4×4 building block has 16 logical axes along each dimension. As described below, one or more OCS switches for logical axes can be used to connect compute nodes of different building blocks on the same logical axis.

Computational nodes within building block 310 can be connected to each other via internal links 318 made of a conductive material (e.g., copper cables). Internal links 318 can be used to connect computational nodes in each segment of each dimension. For example, there is an internal link 318 connecting computational node 301 to computational node 302. There is also an internal link 318 connecting computational node 302 to computational node 303, and another internal link 318 connecting computational node 303 to computational node 304. Computational nodes in each other segment can be connected in the same manner to provide internal data communication between the computational nodes of building block 310.

Building block 310 also includes external links 311-316 connecting building block 310 to the optical network. The optical network connects building block 310 to other building blocks. In this example, building block 310 includes 16 external input links 311 for the x-dimensional dimension. That is, building block 310 includes external input links 311 for each of the 16 segments along the x-dimensional dimension. Similarly, building block 310 includes external output links 312 for each segment along the x-dimensional dimension, external input links 313 for each segment along the y-dimensional dimension, external output links 314 for each segment along the y-dimensional dimension, external input links 315 for each segment along the z-dimensional dimension, and external output links 316 for each segment along the z-dimensional dimension. Since some permutations of building blocks can have more than three dimensions, such as tori, which can have any number of dimensions, building block 310 can include similar external links for each dimension of building block 310.

Each external link 311-316 can be an optical fiber link that connects a compute node on its corresponding segment to an optical network. For example, each external link 311-316 can connect its compute node to an OCS switch of the optical network. As described below, the optical network can include one or more OCS switches for each dimension, and for each dimension, building block 310 has segments. That is, external links 311 and 312 for the x-dimensional can be connected to different OCS switches than external links 313 and 314. The OCS switches can be configured to connect building blocks to other building blocks to form workload clusters, as described in more detail below.

Building block 310 is in the form of a 4×4×4 grid arrangement. Other arrangements are also possible for building blocks of 4×4×4 (or other sizes). For example, building block 310 can be in the form of a three-dimensional torus with surrounding toroidal links, similar to workload cluster 320. Workload cluster 320 can also be generated from a single grid building block 310 by configuring an optical network to provide surrounding toroidal links 321-323.

Toroidal links 321-323 provide wraparound data communication between one end of each segment and the other end of each segment. For example, toroidal link 321 connects a compute node at each end of each segment along the x-dimensional plane to a corresponding compute node at the other end of the segment. Toroidal link 321 may include a link connecting compute node 325 to compute node 326. Similarly, toroidal link 322 may include a link connecting compute node 325 to compute node 327.

The toroidal links 321-323 can be conductive cables, such as copper cables, or optical links. For example, the optical links of toroidal links 321-323 can connect their corresponding computer nodes to one or more OCS switches. The OCS switches can be configured to route data from one end of each segment to the other end of each segment. Building block 310 can include an OCS switch for each dimension. For example, toroidal link 321 can be connected to a first OCS switch that routes data between one end of each segment along the x-dimensional plane and the other end of each segment along the x-dimensional plane. Similarly, toroidal link 322 can be connected to a second OCS switch that routes data between one end of each segment along the y-dimensional plane and the other end of each segment along the y-dimensional plane. Toroidal link 322 can be connected to a third OCS switch that routes data between one end of each segment along the z-dimensional plane and the other end of each segment along the z-dimensional plane.

Workload cluster 330 comprises two building blocks 338 and 339 forming a 4×8×4 platform. Each building block 338 and 339 may be identical to building block 310 or workload cluster 320. The two building blocks are connected along the y-direction using external link 337. For example, one or more OCS switches may be configured to route data between the y-dimensional segments of building block 338 and building block 339.

Furthermore, one or more OCS switches can be configured to provide wraparound links 331-333 along all three dimensions between one end of each segment and the other end of each segment. In this example, wraparound link 333 connects one end of the y-dimensional segment of building block 338 to one end of the y-dimensional segment of building block 339, thereby providing full wraparound communication for the y-dimensional segment formed by the combination of two building blocks 338 and 339.

Workload cluster 340 comprises eight building blocks (one not shown) forming an 8×8×8 cluster. Each building block 348 may be identical to building block 310. Building block links connected along the x-dimensional plane use external links 345a-345c. Similarly, building block links connected along the y-dimensional plane use external links 344A-344C, and building blocks connected along the z-dimensional plane use external links 346A-346C. For example, one or more OCS switches may be configured to route data between x-dimensional segments, one or more OCS switches may be configured to route data between y-dimensional segments, and one or more OCS switches may be configured to route data between z-dimensional segments. Additional external links exist, with each dimension connecting the building block (not shown in Figure 3) to adjacent building blocks. Furthermore, one or more OCS switches may be configured to provide wraparound links 341-343 between one end of each segment and the other end of each segment along all three dimensions.

Figure 4 illustrates an example optical link 400 from a compute node to an OCS switch. The compute nodes of the super platform can be mounted in trays within a data center rack. Each compute node can include six high-speed electrical links. Two of these electrical links can be connected to the compute node's circuit board, and four electrical links can be routed to external electrical connectors, such as Octal Small Form Factor Pluggable (OSFP) connectors, which connect to port 410, for example, an OSFP port. In this example, port 410 is connected to optical module 420 via electrical contact 412. If needed, optical module 420 can convert the electrical links to optical links to extend the length of the external links, for example, to more than one kilometer (km), to provide data communication between compute nodes in a large data center. The type of optical module can vary depending on the required length between the building block and the OCS switch, as well as the required speed and bandwidth of the link.

Optical module 420 is connected to circulator 430 via fiber optic cables 422 and 424. Fiber optic cable 422 may include one or more fiber optic cables for transmitting data from optical module 420 to circulator 430. Fiber optic cable 424 may include one or more fiber optic cables for receiving data from circulator 430. For example, fiber optic cables 422 and 424 may include bidirectional or unidirectional TX/RX fiber pairs. Circulator 430 can reduce the number of fiber optic cables (e.g., from two pairs to a single pair of fiber optic cables 432) by converting from unidirectional to bidirectional fiber. This aligns well with the single OCS port 445 of OCS switch 440, which typically houses a pair of optical paths (2 fibers) switched together. In some implementations, circulator 430 may be integrated into optical module 420 or omitted from optical link 400.

Figure 5-7 illustrates how to form a 4×4×4 building block using multiple compute trays. Similar techniques can be used to form building blocks of other sizes and shapes.

Figure 5 illustrates a logical compute tray 500 used to form a 4×4×4 building block. The basic hardware block of the 4×4×4 building block is a single compute tray 500 with a 2×2×1 topology. In this example, the compute tray 500 has two compute nodes along the x-dimensional line, two nodes along the y-dimensional line, and one node along the z-dimensional line. For example, compute nodes 501 and 502 form an x-dimensional segment, while compute nodes 503 and 504 form an x-dimensional segment. Similarly, compute nodes 501 and 503 form a y-dimensional segment, and compute nodes 502 and 504 form a y-dimensional segment.

Each compute node 501-504 is connected to two other compute nodes via an internal link 510 (e.g., a copper cable or a trace on a printed circuit board). Each compute node is also connected to four external ports. Compute node 501 is connected to external port 521. Similarly, compute node 502 is connected to external port 522, compute node 503 is connected to external port 523, and compute node 504 is connected to external port 524. As mentioned above, external ports 521-524 can be OSFPs or other ports that connect compute nodes to the OCS switch. The ports can accommodate copper or fiber optic modules attached to fiber optic cables.

Each compute node 501-504 has external ports 521-524 with an x-dimensional port, a y-dimensional port, and two z-dimensional ports. This is because each compute node 501-504 is already connected to another compute node in the x and y dimensions via internal link 510. Having two z-dimensional external ports allows each compute node 501-504 to also connect to two compute nodes along the z-dimensional direction.

Figure 6 illustrates a sub-block 600 with the exemplary building block having its one-dimensional (z-dimensional) component omitted. Specifically, sub-block 600 is a 4×4×1 block formed by a 2×2 arrangement of computation trays (e.g., a 2×2 arrangement of computation tray 500 in Figure 5). Sub-block 600 includes four computation trays 620A-620D arranged in a 2×2 configuration. Each computation tray 620A-620D may be identical to computation tray 500 in Figure 5, including four computation nodes 622 arranged in a 2×2×1 configuration.

Computation nodes 622 of computation trays 620A-620D can be connected using internal links 631-634 (e.g., copper cables). For example, two computation nodes 622 of computation tray 620A can be connected along the y-axis to two computation nodes 622 of computation tray 620B using internal link 632.

The two compute nodes 622 of each compute tray 620A-620D are also connected to the external link 640 along the x-dimensional axis. Similarly, the two compute nodes of each compute tray 620A-620D are also connected to the external line 641 along the y-dimensional axis. Specifically, the compute nodes at the end of each x-dimensional segment and the end of each y-dimensional segment are connected to the external link 640. These external links 640 can be, for example, fiber optic cables using the optical link 400 of Figure 4 to connect the compute nodes to the OCS switch, thereby connecting the building blocks including the compute nodes to the OCS switch.

A 4×4×4 building block can be formed by connecting four sub-blocks 600 together along the z-dimensional axis. For example, compute nodes 622 of each compute tray 620A-620A can be connected to one or two corresponding compute nodes of other compute trays arranged in the z-dimensional axis using internal links. The compute nodes at the end of each z-dimensional segment can include external links 640 connected to the OCS switch, similar to the external links at the ends of the x-dimensional and y-dimensional segments.

Figure 7 illustrates an exemplary building block 700. Building block 700 includes four sub-blocks 710A-710D connected along the z-dimensional axis. Each sub-block 710A-710D may be identical to sub-block 600 of Figure 6. Figure 7 illustrates some connections between sub-blocks 710A-710D along the z-dimensional axis.

Specifically, building block 700 includes internal links 730-733 between corresponding computing nodes 716 of computing trays 715 along the z-dimensional sub-blocks 710A-710D. For example, internal link 730 connects a segment of computing node 0 along the z-dimensional. Similarly, internal link 731 connects a segment of computing node 1 along the z-dimensional, internal link 732 connects a segment of computing node 8 along the z-dimensional, and internal link 733 connects a segment of computing node 9 along the z-dimensional. Although not shown, similar internal links connect segments of computing nodes 2-7 and A-F.

Building block 700 also includes external links 720 at the end of each segment along the z-dimensional axis. Although external links 720 are only shown for segments of compute nodes 0, 1, 8, and 9, external links 720 are also included for each of the other segments of compute nodes 2-7 and A-F. External links can connect segments to OCS switches, similar to external links at the ends of x-dimensional and y-dimensional segments.

Figure 8 illustrates an OCS topology 800 for a super platform. In this example, the OCS topology includes individual OCS switches for each segment of each dimension along a 4×4×4 building block of the super platform, comprising 64 building blocks 805 (i.e., building blocks 0-63). The 4×4×4 building block 805 includes 16 segments along the x-dimensional, 16 segments along the y-dimensional, and 16 segments along the z-dimensional. In this example, the OCS topology includes 16 OCS switches for the x-dimensional, 16 OCS switches for the y-dimensional, and 16 OCS switches for the z-dimensional, for a total of 48 OCS switches, which can be configured to generate various workload clusters.

In other words, the OCS topology includes an OCS switch for each logical axis of a building block. Segments of building blocks on the same logical axis are connected to the same OCS switch. In this way, the OCS switches for logical axes can be configured to connect segments of compute nodes along the logical axis when creating a workload cluster, allowing compute nodes along the logical axis to communicate with each other via the OCS switches for the logical axis. If building block A is logically positioned to the right of building block B in the workload cluster, the OCS switches for the x-dimensional logical axis can be configured to route data between segments of building block A and segments of building block B on that logical axis.

For the x-dimensional dimension, the OCS topology 800 includes 16 OCS switches, including OCS switch 810. For each segment along the x-dimensional dimension, each building block 805 includes an external input link 811 and an external output link 812, which are connected to the OCS switch 810 for that segment. These external links 811 and 812 may be the same as or similar to the optical link 400 in Figure 4.

For the y-axis, the OCS topology 800 includes 16 OCS switches, including OCS switch 820. For each segment along the y-axis, each building block 805 includes an external input link 821 and an external output link 822, which are connected to the OCS switch 810 for that segment. These external links 821 and 822 may be the same as or similar to the optical link 400 in Figure 4.

For the z-dimensional dimension, the OCS topology 800 includes 16 OCS switches, including OCS switch 830. For each segment along the z-dimensional dimension, each building block 805 includes an external input link 831 and an external output link 832, which are connected to the OCS switch 810 for that segment. These external links 821 and 822 may be the same as or similar to the optical link 400 in Figure 4.

In other examples, multiple segments can share the same OCS switch, depending on the OCS cardinality and/or the number of building blocks in the superplatform. For instance, if the OCS switch has a sufficient number of ports for all x-dimensional segments of all building blocks in the superplatform, then all x-dimensional segments can be connected to the same OCS switch. In another example, if the OCS switch has a sufficient number of ports, two segments in each dimension can share the OCS switch. However, having corresponding segments of all building blocks of the superplatform connected to the same OCS switch makes it possible to use a single routing table for data communication between the compute nodes of these segments. Furthermore, using a separate OCS switch for each segment or each dimension simplifies troubleshooting and diagnostics. For example, if there is a problem with data communication on a particular segment or dimension, identifying the potentially faulty OCS is easier than if multiple OCSs are used for that particular segment or dimension.

Figure 9 illustrates the components of an example superplatform 900. For example, superplatform 900 could be one of the superplatforms in the processing system 130 of Figure 1. The example superplatform 900 includes 64 4×4×4 building blocks 960, which can be used to generate workload clusters that perform computational workloads (e.g., machine learning workloads). As described above, each 4×4×4 building block 960 includes 32 compute nodes, with four compute nodes arranged along each of the three dimensions. For example, building block 960 could be the same as or similar to building block 310, workload cluster 320, or building block 700 described above.

An exemplary super platform 900 includes an optical network 970 comprising 48 OCS switches 930, 940, and 950, which are connected to the building blocks 960 via 96 external links 931, 932, and 933 for each building block. Each external link may be a fiber optic link similar to or the same as optical link 400 in Figure 4.

Optical network 970 includes an OCS switch for each segment of each dimension of each building block, similar to the OCS topology 800 in Figure 8. For the x-dimensional dimension, optical network 970 includes 16 OCS switches 950, each OCS switch 950 for each segment along the x-dimensional dimension. For each building block 960, optical network 970 also includes input external links and output external links for each segment along the x-dimensional dimension of building block 960. These external links connect the compute nodes on that segment to the OCS switch 950 for that segment. Since each building block 960 includes 16 segments along the x-dimensional dimension, optical network 970 includes 32 external links 933 (i.e., 16 input links and 16 output links) that connect the x-dimensional segments of each building block 960 to the corresponding OCS switches 950 for those segments.

For the y-dimensional dimension, the optical network 970 includes 16 OCS switches 930, each OCS switch for each segment along the y-dimensional dimension. For each building block 960, the optical network 970 also includes input external links and output external links for each segment along the y-dimensional dimension of the building block 960. These external links connect the compute nodes on that segment to the OCS switch 930 for that segment. Since each building block 960 includes 16 segments along the y-dimensional dimension, the optical network 970 includes 32 external links 931 (i.e., 16 input links and 16 output links) that connect the y-dimensional segments of each building block 960 to the corresponding OCS switches 930 for those segments.

For the z-dimensional dimension, the optical network 970 includes 16 OCS switches 940, each OCS switch 940 for each segment along the z-dimensional dimension. For each building block 960, the optical network 970 also includes input external links and output external links for each segment along the z-dimensional dimension of the building block 960. These external links connect the compute nodes on that segment to the OCS switch 940 for that segment. Since each building block 960 includes 16 segments along the z-dimensional dimension, the optical network 970 includes 32 external links 932 (i.e., 16 input links and 16 output links) that connect the z-dimensional segments of each building block 960 to the corresponding OCS switches 940 for those segments.

Workload scheduler 910 can receive request data that includes workload and data specifying the building blocks 960 of the requested cluster for executing the workload. The request data may also include the priority of the workload. Priority can be represented horizontally, such as high, medium, or low, or numerically, such as in the range of 1-100 or another suitable range. For example, workload scheduler 910 may receive request data from a user equipment or a cell scheduler (e.g., user equipment 110 or cell scheduler 140 of FIG. 1). As described above, the request data may specify a target n-dimensional arrangement of compute nodes, such as a target arrangement of building blocks including compute nodes.

Workload scheduler 910 can select a set of building blocks 960 to generate a workload cluster that matches the target arrangement specified by the requested data. For example, workload scheduler 910 can identify a set of available healthy building blocks in super platform 900. Available healthy building blocks are those that do not perform another workload or are not part of a workload cluster and are not invalid.

For example, the workload scheduler 910 can maintain and update status data, such as in the form of a database, indicating the status of each building block 960 in the super platform. The availability status of a building block 960 can indicate whether it has been assigned to a workload cluster. The health status of a building block 960 can indicate whether it is active or inactive. The workload scheduler 910 can identify building blocks 960 with an availability status indicating that they are not assigned to a workload and are in a healthy active state. When a building block 960 is assigned to a workload (e.g., for generating a workload cluster for that workload), or when its health status changes (e.g., from active to inactive or vice versa), the workload scheduler can update the status data of the building block 960 accordingly.

From the identified building blocks 960, the workload scheduler 910 can select a number of building blocks 960 that match the number defined by the target arrangement. If the request data specifies one or more types of compute nodes, the workload scheduler 910 can select building blocks from the identified building blocks 960 that have the requested one or more types of compute nodes. For example, if the request data specifies a 2×2 arrangement of building blocks with two TPU building blocks and two GPU building blocks, the workload scheduler 910 can select two available and healthy building blocks with TPUs and two healthy and available building blocks with GPUs.

Workload scheduler 910 can also select building blocks 960 based on the priority of each workload currently running in the superplatform and the priority of the workloads included in the requested data. If the superplatform 900 does not have enough healthy available building blocks to generate a workload cluster for the requested workload, workload scheduler 910 can determine whether there are any ongoing workloads in the superplatform 900 with a lower priority than the requested workload. If so, workload scheduler 910 can reallocate building blocks from one or more workload clusters of lower priority workloads to the workload cluster of the requested workload. For example, workload scheduler 910 can terminate lower priority workloads, delay lower priority workloads, or reduce the size of the workload clusters used for lower priority workloads to free up building blocks for higher priority workloads.

Workload scheduler 910 can easily reassign building blocks from one workload cluster to another by reconfiguring the optical network (e.g., by configuring the OCS switch as described below), so that building blocks are connected to building blocks of higher-priority workloads instead of building blocks of lower-priority workloads. Similarly, if a building block of a higher-priority workload fails, workload scheduler 910 can reassign building blocks intended for lower-priority workload clusters to higher-priority workload clusters by reconfiguring the optical network.

The workload scheduler 910 can generate job configuration data 912 and provide it to the OCS manager 920 of the super platform 900. The job configuration data 912 can specify the selected building blocks 960 used for the workload and the arrangement of the building blocks. For example, if the arrangement is a 2×2 arrangement, then the arrangement includes four points for the building blocks. The job configuration data can specify which of the selected building blocks 960 enters each of the four points.

By configuring the job data 912, the selected building block 960 can be identified using a logical identifier for each building block. For example, each building block 960 can include a unique logical identifier. In a specific example, the 64 building blocks 960 can be numbered 0-63, and these numbers can be unique logical identifiers.

OCS Manager 920 uses job-specific configuration data 912 to configure OCS switches 930, 940, and/or 950 to generate workload clusters that match the arrangement specified by the job-specific configuration data. Each OCS switch 930, 940, and 950 includes a routing table for routing data between the physical ports of the OCS switch. For example, suppose the output external link of an x-dimensional segment for a first building block is connected to the input external link of a corresponding x-dimensional segment for a second building block. In this example, the routing table of the OCS switch 950 for that x-dimensional segment will indicate that data between the physical ports of the OCS switches to which these segments are connected will be routed to each other.

The OCS Manager 920 can maintain port data that maps each port of each OCS switch 920, 930, and 940 to each logical port of each building block. For each x-dimensional segment of a building block, this port data can specify which physical port of the OCS switch 950 an external input link is connected to and which physical port of the OCS switch 950 an external output link is connected to. The port data can include the same data for each dimension of each building block 960 for the Super Platform 900.

The OCS Manager 920 can use this port data to configure the routing tables of OCS switches 930, 940, and/or 950 to generate workload clusters for workloads. For example, suppose a first building block will be connected to a second building block in a 2×1 arrangement, where the first building block is to the left of the second building block in the x-dimensional direction. The OCS Manager 920 will update the routing table of the OCS switch 950 for the x-dimensional direction to route data between the x-dimensional segments of the first and second building blocks. Since each x-dimensional segment of the building block will need to be connected, the OCS Manager 920 can update the routing table of each OCS switch 950.

For each x-dimensional segment, the OCS Manager 920 can update the routing table of the OCS Switch 950 used for that segment. Specifically, the OCS Manager 920 can update the routing table to map the physical ports of the OCS Switch 950 to which the segments of the first building block are connected to the physical ports of the OCS Switch to which the segments of the second building block are connected. When each x-dimensional segment includes both input and output links, the OCS Manager 920 can update the routing table such that the input links of the first building block are connected to the output links of the second building block, and vice versa.

The OCS Manager 920 can update its routing table by obtaining the current routing table from each OCS switch. The OCS Manager 920 can then update the appropriate routing table and send the updated routing table to the appropriate OCS switch. In another example, the OCS Manager 920 can send update data specifying the update to the OCS switch, and the OCS switches can update their routing tables based on the update data.

After configuring the OCS switch with the updated routing table, a workload cluster is generated. The workload scheduler 910 then allows the workload to be executed by the compute nodes of the workload cluster. For example, the workload scheduler 910 can provide workloads to the compute nodes of the workload cluster for execution.

After completing the workload, the workload scheduler 910 can update the status of each building block used to generate the workload cluster back to available. The workload scheduler 910 can also instruct the OCS manager 920 to remove the connections between the building blocks used to generate the workload cluster. The OCS manager 920 can then update the routing table, thereby removing the mappings between the physical ports of the OCS switches used to route data between the building blocks.

Using OCS switches in this way to configure optical topology to generate workload clusters allows the super platform to host multiple workloads dynamically and securely. The workload scheduler 920 can dynamically create and terminate workload clusters as new workloads are received and completed. Compared to traditional supercomputers, the routing between segments provided by OCS switches offers better security between different workloads executed within the same super platform. For example, OCS switches utilize air gaps between workloads to physically decouple them from each other. Traditional supercomputers use software to provide isolation between workloads, which is more vulnerable to data corruption.

Figure 10 is a flowchart illustrating an exemplary process 1000 for generating a workload cluster and performing computational workloads using that workload cluster. The operation of process 1000 can be performed by a system including one or more data processing devices. For example, the operation of process 1000 can be performed by the processing system 130 of Figure 1.

The system receives request data (1010) for the requested cluster of specified compute nodes. For example, request data can be received from a user equipment. The request data may include computational workload and data on the target n-dimensional permutation of the specified compute nodes. For example, the request data may specify a target n-dimensional permutation including the building blocks of the compute nodes.

In some implementations, the request data may also specify the type of compute nodes for the building blocks. A super platform may include building blocks with different types of compute nodes. For example, a super platform may include 90 building blocks, each comprising a 4×4×4 arrangement of TPUs, and 10 dedicated building blocks comprising dedicated compute nodes arranged in a 2×1 pattern. The request data may specify the number of building blocks of each type of compute node and the arrangement of these building blocks.

The system selects a subset (1020) of building blocks for the requested cluster from a superplatform comprising a set of building blocks. As described above, the superplatform may comprise a set of building blocks having a three-dimensional arrangement of compute nodes (e.g., a 4×4×4 arrangement of compute nodes). The system can select a number of building blocks that matches the number defined by the target arrangement. As described above, the system can select healthy building blocks that are available for the requested cluster.

A subset of building blocks can be an appropriate subset of building blocks. An appropriate subset is a subset that does not include all members of the set. For example, generating a workload cluster that matches the target arrangement of compute nodes may require fewer than all building blocks.

The system generates a workload cluster (1030) that includes a selected subset of compute nodes. The workload cluster can have an arrangement of building blocks that matches the target arrangement specified in the request data. For example, if the request data specifies a 4×8×4 arrangement of compute nodes, the workload cluster can include two building blocks arranged similarly to workload cluster 330 in Figure 3.

To generate workload clusters, the system can configure routing data for each dimension of the workload cluster. For example, as described above, the super platform may include an optical network comprising one or more OCS switches for each dimension of the building block. The routing data for a dimension may include routing tables for one or more OCS switches. As illustrated above with reference to Figure 9, the routing tables of the OCS switches can be configured to route data along each dimension between compute nodes in the appropriate segments.

The system enables compute nodes in the workload cluster to perform compute workloads (1040). For example, the system can provide compute workloads to compute nodes in the workload cluster. While the compute workloads are being performed, the configured OCS switches can route data between building blocks of the workload cluster. Although the compute nodes are not physically connected in the target arrangement, the configured OCS switches can route data between compute nodes in the building blocks as if the compute nodes were physically connected in the target arrangement.

For example, compute nodes in each segment of a dimension can transmit data to other compute nodes in that segment via an OCS switch. These compute nodes are in different building blocks, as if the compute nodes in that segment were physically connected within a single physical segment. This differs from packet-switched networks because this configuration of workload clusters provides a true end-to-end optical path between the corresponding segments without intermediate packet switching or buffering. In packet switching, latency is increased because packets need to be received by the switch, buffered, and retransmitted on another port.

After completing the computational workload, for example by updating the state of the building blocks to an available state and updating the routing data to no longer route data between building blocks in the workload cluster, the system can release the building blocks for other workloads.

Figure 11 is a flowchart illustrating an exemplary process 1100 for reconfiguring an optical network to replace a failed building block. Operation of process 1100 can be performed by a system including one or more data processing devices. For example, operation of process 1100 can be performed by the processing system 130 of Figure 1.

This system enables the compute nodes of a workload cluster to perform computational workloads (1110). For example, the system can generate a workload cluster and enable the compute nodes to perform computational workloads using process 1000 of Figure 10.

The system receives data (1120) indicating that a building block of the workload cluster has failed. For example, if one or more compute nodes of a building block fail, another component (e.g., a monitoring component) can determine that the building block has failed and send data to the system indicating that the building block has failed.

The system identifies available building blocks (1130). For example, the system can identify available healthy building blocks in the same superplatform as other building blocks in the workload cluster. The system can identify available healthy building blocks based on, for example, the status data of the building blocks maintained by the system.

The system replaces the failed building block (1140) with the identified available building blocks. The system can update the routing data of one or more OCS switches in the optical network connecting the building block to replace the failed building block with the identified available building block. For example, the system can update the routing tables of one or more OCS switches to remove the connection between the failed building block and other building blocks in the workload cluster. The system can also update the routing tables of one or more OCS switches to connect the identified building block to other building blocks in the workload cluster.

The system can logically rank the identified building blocks within the logical points of the failed building block points. As mentioned above, the OCS switch's routing table can map the physical ports of an OCS switch connected to one segment of a building block to the physical ports of an OCS switch connected to the corresponding segment of another building block. In this example, the system can perform replacements by updating the mapping to the corresponding segment of the identified available building block (rather than the failed building block).

For example, suppose the input external link for a specific x-segment of a failed building block is connected to a first port of an OCS switch, and the input external link for the corresponding x-segment of an identified available building block is connected to a second port of the OCS switch. Also suppose the routing table maps the first port to a third port of the OCS switch, which is connected to the corresponding x-segment of another building block. To perform a replacement, the system can update the routing table mapping to map the second port to the third port, instead of mapping the first port to the third port. The system can do this for each segment of the failed building block.

As described above, the optical network architecture for the super platform can include one or more OCS switches for each logical axis of the building blocks. A 4×4×4 building block has 16 logical axes along each dimension. Therefore, the optical network architecture can include 48 OCS switches, which can be configured to connect building blocks with various logical arrangements.

Since each segment of each building block has input and output connections to the OCS switch corresponding to that segment's logical axis, the OCS switch for each axis will have 128 ports for a 64-building-block superplatform. Therefore, in this configuration, if one OCS switch is used per logical axis, each OCS switch will require at least 128 ports for 64 building-block superplatforms. For a given size of OCS switches used to connect building blocks (e.g., a given port count), a one-to-many switch can be used to increase the number of building blocks included in the superplatform, and/or for a given size of OCS switches, a one-to-many switch can be used to reduce the number of ports used on each OCS switch.

Figure 12 illustrates a portion of an exemplary super platform 1200 including building blocks 1211-1214 and 1×2 optical switches 1261-1264. A 1×2 optical switch is an exemplary one-to-many optical switch with one input and two outputs. Although the terms input and output are used, light can travel through the switch in either direction, e.g., from one output to one input and from one input to one output. As described below, other one-to-many switches with different numbers of outputs can be used, such as a 1×3 optical switch (one input and three outputs), a 1×4 optical switch (one input and four outputs), or other suitable one-to-many optical switches.

For clarity, this example illustrates the connection of segments 1221-1224 located on the same logical axis (along the x-dimensional) to two OCS switches 1271 and 1272. However, segments of each building block on each logical axis can be connected to the two OCS switches used for that logical axis. For example, segments 1231-1234 can be connected to two OCS switches (not shown), segments 1241-1244 can be connected to two OCS switches (not shown), and segments 1251-1254 can be connected to two OCS switches (not shown). Segments along each other logical axis in the x-dimensional, each other logical axis in the y-dimensional, and each logical axis in the z-dimensional can also be connected to the two OCS switches used for that logical axis in a similar manner.

The super platform 1200 may include other building blocks with the same configuration as building blocks 1211-1214. For example, the super platform may include 64 building blocks, of which only 4 are shown in Figure 1200. Segments of these building blocks may be connected to corresponding OCS switches for their logical axes.

For example, using fiber optic cables, one side of each segment 1221-1224 is connected to the input of the corresponding 1×2 switch 1261-1264 for that segment. For example, using fiber optic cables, one output of each 1×2 switch 1261-1264 is connected to OCS switch 1271, and the other output of each 1×2 switch 1261-1264 is connected to another OCS switch 1272. Each segment of each building block can be connected to the 1×2 switch for that segment on one side. The 1×2 switch for that segment can be selectively adjusted to connect that side of the segment to either OCS switch 1271 or OCS switch 1272 (for the logical axis shown). For a super platform comprising 64 building blocks with a 4×4×4 structure, the super platform can include 3,0721×2 switches, with one switch per segment of each building block.

The other side of each segment connects to either OCS switch 1271 or OCS switch 1272. For example, the other side of segments 1221 and 1223 connects to OCS switch 1271, while the other side of segments 1222 and 1224 connects to OCS switch 1272. In this way, building blocks 1211-1214 use fewer ports on OCS switches 1271 and 1272 compared to using a single OCS port and having the input and output connections of each building block connected to one OCS switch. For example, there are six connections to OCS switch 1271 (for the four building blocks shown). If OCS switch 1271 is the only switch used for the logical axis including segments 1221-1224, there will be eight connections to OCS switch 1271, with two connections for each of the four segments 1221-1224. OCS switch 1272 similarly has six connections instead of eight.

The workload scheduler can create workload clusters using building blocks 1221-1224 (and/or other building blocks of the super platform 1200) by configuring OCS switches 1271 and 1272 (and OCS switches for other segments on other logical axes) and 1×2 switches for each segment. As described above, the workload scheduler can select building blocks for the workload cluster and configure routing tables for the OCS switches to route data between segments of the building blocks in the workload cluster.

In this example, when each segment along a given logical axis is connected to a pair of OCS switches, the workload scheduler configures a routing table for each pair of OCS switches for each logical axis, enabling segments of building blocks in the workload cluster along that logical axis to communicate with each other. Similarly, the workload scheduler can configure each 1×2 switch based on segments of another building block with which the segment will communicate, to connect its corresponding segment to one of the two OCS switches.

For example, suppose building block 1211 will be logically arranged to the left of building block 1213, as shown in Figure 12. In this example, segment 1221 will need to be able to communicate with segment 1223; segment 1231 will need to communicate with segment 1233; segment 1241 will need to communicate with segment 1243; and segment 1251 will need to communicate with segment 1253.

Specifically, the compute nodes to the right of segments 1221, 1231, 1241, and 1251 should be connected to the compute nodes to the left of segments 1223, 1233, 1243, and 1253, respectively. For example, the compute nodes to the right of segment 1221 should be connected to the compute nodes to the left of segment 1223. The compute nodes to the right of segments 1221, 1231, 1241, and 1251 are connected to the input of their respective 1×2 switches (switch 1261 for segment 1221). The compute nodes to the left of segments 1223, 1233, 1243, and 1253 are connected to OCS switch 1271. Therefore, to connect the compute nodes on the right side of segments 1221, 1231, 1241, and 1251 to the compute nodes on the left side of segments 1223, 1233, 1243, and 1253 respectively, the workload scheduler can configure a 1×2 switch for each segment 1221, 1231, 1241, and 1251 to connect the input of the 1×2 switch to an output (O1) that is connected to the OCS switch 1271. For example, the 1×2 switch 1261 will be configured such that the input is routed to the output connected to the OCS switch 1271. Since light can propagate in both directions through the 1×2 switch, data can be routed in both directions between the compute nodes on the right side of segment 1221 and the compute nodes on the left side of segment 1223 via the 1×2 switch 1261 and the OCS switch 1271.

If the workload cluster has only two building blocks along the x-dimensional plane (e.g., a 2×4×4 arrangement), then the compute nodes to the right of segments 1223, 1233, 1243, and 1253 should also be connected to the compute nodes to the left of segments 1221, 1231, 1241, and 1251, respectively. For example, the compute nodes to the right of segment 1223 should be connected to the compute nodes to the left of segment 1221. The compute nodes to the right of segments 1223, 1233, 1243, and 1253 are connected to the input of their respective 1×2 switches (switch 1263 for segment 1223). The compute nodes to the left of segments 1221, 1231, 1241, and 1251 are connected to OCS switch 1271. Therefore, to connect the compute nodes on the right side of segments 1223, 1233, 1243, and 1253 to the compute nodes on the left side of segments 1221, 1231, 1241, and 1251, respectively, the workload scheduler can configure a 1×2 switch for each segment 1223, 1233, 1243, and 1253 to connect the input of the 1×2 switch to an output (O1) that is connected to OCS switch 1271. For example, 1×2 switch 1263 would be configured such that the input is routed to the output (O1) connected to OCS switch 1271.

The workload scheduler can also configure the OCS switch 1271 to route data between corresponding segments. For example, the workload scheduler can configure the routing table of the OCS switch 1271 to route data between segments 1221 and 1223. Specifically, the workload scheduler can configure the routing table to route data received at a port connected to output O1 of the 1×2 switch 1261 to a port of the compute node connected to the left of segment 1223. Similarly, the workload scheduler can configure the routing table to route data received at a port connected to output O1 of the 1×2 switch 1263 to a compute node to the left of segment 1221. The workload scheduler can configure the routing table of the OCS switch 1271 for each of the other segments 1231, 1241, 1251 and their corresponding segments 1233, 1243, and 1253 in a similar manner.

In another example, suppose that building block 1211 will be logically arranged to the left of building block 1212, rather than to the left of building block 1213. In this example, segment 1221 will need to be able to communicate with segment 1222. Segment 1231 will need to communicate with segment 1232, segment 1241 will need to communicate with segment 1242, and segment 1251 will need to communicate with segment 1252.

Specifically, the compute nodes to the right of segments 1221, 1231, 1241, and 1251 should be connected to the compute nodes to the left of segments 1222, 1232, 1242, and 1252, respectively. The compute nodes to the right of segments 1221, 1231, 1241, and 1251 are connected to the input of their respective 1×2 switches (switch 1261 for segment 1221). However, the compute nodes to the left of segments 1222, 1232, 1242, and 1252 are connected to OCS switch 1272. Therefore, in order to connect the compute nodes on the right side of segments 1221, 1231, 1241, and 1251 to the compute nodes on the left side of segments 1222, 1232, 1242, and 12532, respectively, the workload scheduler can configure a 1×2 switch for each segment 1221, 1231, 1241, and 1251 to connect the input of the 1×2 switch to the output (O2), which in turn connects to the OCS switch 1272. For example, the 1×2 switch 1261 would be configured such that the input is routed to the output (O2) connected to the OCS switch 1272.

If the workload cluster has only two building blocks along the x-dimensional plane (e.g., a 2×4×4 arrangement), then the compute nodes to the right of segments 1222, 1232, 1242, and 1252 should be connected to the compute nodes to the left of segments 1221, 1231, 1241, and 1251, respectively. For example, the compute nodes to the right of segment 1222 should be connected to the compute nodes to the left of segment 1221. The compute nodes to the right of segments 1222, 1232, 1242, and 1252 are connected to the input of their respective 1×2 switches (switch 1262 for segment 1222). The compute nodes to the left of segments 1221, 1231, 1241, and 1251 are connected to OCS switch 1271. Therefore, to connect the compute nodes on the right side of segments 1222, 1232, 1242, and 1252 to the compute nodes on the left side of segments 1221, 1231, 1241, and 1251, respectively, the workload scheduler can configure a 1×2 switch for each segment 1222, 1232, 1242, and 1252 to connect the input of the 1×2 switch to an output (O1) that is connected to OCS switch 1271. For example, 1×2 switch 1262 would be configured such that the input is routed to the output (O1) connected to OCS switch 1271.

The workload scheduler can also configure OCS switches 1271 and 1272 to route data between corresponding segments. For example, the workload scheduler can configure the routing table of OCS switch 1272 to route data received from the output (O2) of a 1×2 switch connected to compute nodes on the right side of segments 1221, 1231, 1241, and 1251 to compute nodes on the left side of segments 1222, 1232, 1242, and 1252, respectively. Specifically, the workload scheduler can configure the routing table to route data received at a port connected to the output (O2) of 1×2 switch 1261 to a port connected to a compute node on the left side of segment 1222. Similarly, the workload scheduler can configure the routing table of OCS switch 1271 to route data received at a port connected to the output (O1) of 1×2 switch 1262 to a compute node on the left side of segment 1221. The workload scheduler can configure the routing table of OCS switch 1271 for each of the other segments 1231, 1241, 1251 and their corresponding segments 1232, 1242 and 1252 in a similar manner.

As described above, other one-to-many switches can be used instead of 1×2 switches. For example, a super platform can include three OCS switches for each logical axis. In this example, one side of each segment of each building block can be connected to the input of a 1×3 switch with three outputs. The three outputs of the 1×3 switch for a segment can be connected to the three OCS switches corresponding to the logical axis of that segment. The 1×3 switches and OCS switches can be configured in a similar manner to those described above to connect segments of a building block to corresponding segments of other building blocks. Compared to using 1×2 switches and two OCS switches per logical axis, using 1×3 switches and three OCS switches per logical axis enables a larger super platform for a given size of OCS switches and/or uses fewer ports per OCS switch. However, this also results in more OCS switches per logical axis of the building block. Other one-to-many switches, such as 1×4, 1×5, etc., can also be used, where the number of OCS switches per logical axis equals the number of outputs of the one-to-many switch.

Figure 13 illustrates an exemplary workload cluster 1300. Workload cluster 1300 is an 8×8×8 cluster consisting of eight building blocks, 1311-1317 (one below building block 1315 is not shown). Each building block is a 4×4×4 building block with 16 segments of compute nodes along 16 logical axes in the x-dimensional direction, 16 segments of compute nodes along 16 logical axes in the y-dimensional direction, and 16 segments of compute nodes along 16 logical axes in the z-dimensional direction. For this example, it is assumed that the superplatform from which workload cluster 1300 is created includes two OCS switches for each logical axis and a corresponding 1×2 switch for each segment of each building block.

The workload scheduler can create workload cluster 1300 by configuring OCS switches and 1×2 switches to connect segments of building blocks to corresponding segments of other building blocks. For example, building block 1311 is logically above building block 1312. The workload scheduler can configure an OCS switch for each logical axis in the y-dimensional space, such that the OCS switch routes data in the y-dimensional space from the top compute node of each segment of building block 1312 to the bottom compute node of the corresponding segment of building block 1311. For example, the workload scheduler can configure an OCS switch for logical axis 1330 (along the leftmost and foremost segments in the y-dimensional space), such that the OCS switch routes data between compute node 1331 of building block 1312 and compute node 1332 of building block 1311. The workload scheduler can also configure 1×2 switches for each segment in the y-dimensional space of building blocks 1311 and 1312 to connect these segments to the appropriate OCS switches, as described above with respect to Figure 12.

Similarly, building block 1311 is logically located to the left of building block 1313. The workload scheduler can configure an OCS switch for each logical axis in the x-dimensional space, such that the OCS switch routes data in the x-dimensional space from the rightmost compute node of each segment of building block 1311 to the leftmost compute node of the corresponding segment of building block 1313. For example, the workload scheduler can configure an OCS switch for logical axis 1320 (along the topmost and foremost segments in the x-dimensional space), such that the OCS switch routes data between compute node 1321 of building block 1311 and compute node 1322 of building block 1313. The workload scheduler can also configure a 1×2 switch for each segment in the x-dimensional space of building blocks 1311 and 1313 to connect these segments to the appropriate OCS switch, as described above with respect to Figure 12.

Similarly, building block 1314 is logically located ahead of building block 1317 along the z-dimensional axis. The workload scheduler can configure an OCS switch for each logical axis in the z-dimensional axis, such that the OCS switch routes data along the z-dimensional axis from the last compute node of each segment of building block 1314 to the first compute node of the corresponding segment of building block 1317. For example, the workload scheduler can configure an OCS switch for logical axis 1340 (along the top and rightmost sides of the z-dimensional axis), such that the OCS switch routes data between compute node 1341 of building block 1314 and compute node 1342 of building block 1317. The workload scheduler can also configure a 1×2 switch for each segment in the z-dimensional axis of building blocks 1314 and 1317 to connect these segments to the appropriate OCS switch, as described above with respect to Figure 12.

The workload scheduler can configure an OCS switch for each logical axis and a 1×2 switch for each segment, enabling segments of each building block to communicate with corresponding segments of adjacent building blocks. The corresponding segment of a given segment is the segment on the same logical axis as the given segment.

Figure 14 is a flowchart illustrating an exemplary process 1400 for generating a workload cluster and performing computational workloads using that workload cluster. The operation of process 1100 can be performed by a system including one or more data processing devices. For example, the operation of process 1100 can be performed by the processing system 130 of Figure 1.

The system receives request data specifying the requested compute nodes (1410) for the computational workload. For example, the request data can be received from a user equipment. The request data may include the computational workload and data specifying a target n-dimensional permutation of the compute nodes. For example, the request data may specify a target n-dimensional permutation of building blocks including the compute nodes.

The system selects a subset (1420) of building blocks from a superplatform comprising a set of building blocks for the requested cluster. As described above, the superplatform may comprise a set of building blocks having a three-dimensional arrangement of compute nodes (e.g., a 4×4×4 arrangement of compute nodes). The system may select an amount of building blocks that matches the amount defined by the target arrangement. As described above, the system may select healthy building blocks that are available for the requested cluster.

In the super platform, each building block can be connected to an optical network comprising two or more OCS switches for each dimension in m dimensions. For example, the optical network could include two OCS switches for each logical axis of each dimension in m dimensions. For each dimension in m dimensions, each building block could include one or more segments of compute nodes interconnected along that dimension. For example, each building block could include segments along each logical axis of that dimension.

Each segment may include a first compute node at a first end of the segment and a second compute node at a second end of the segment opposite to the first end. If the segment includes more than two compute nodes, additional compute nodes may be located within the segment between the first and second compute nodes.

A first portion of the first compute node is connected to the first OCS switch among two or more OCS switches in that dimension. One or more additional portions of the first compute node are connected to corresponding additional OCS switches among two or more OCS switches in that dimension. As described above, the optical network may include two OCS switches for each logical axis of a building block. For a given logical axis, the first compute node of some building blocks may be connected to the first OCS switch among the two OCS switches. The first compute node of other building blocks may be connected to the second OCS switch among the two OCS switches. These connections may be direct connections without any intermediate one-to-many switches.

These segments can be allocated to each OCS switch to balance the OCS switch distribution. That is, if the optical network includes two OCS switches for each logical axis, half (or approximately half) of the segments on that logical axis can be allocated to the first OCS switch, and half (or approximately half) of the segments on that logical axis can be allocated to the second OCS switch.

The second compute node of each segment connects to the input of a corresponding one-to-many optical switch with inputs and multiple outputs. For example, a 1×2 optical switch has one input and two outputs. The first output of the one-to-many optical switch can be connected to a first OCS switch. Each additional output connects to an additional OCS switch. For example, if the optical network includes two OCS switches for each logical axis and a 1×2 optical switch connected to each segment, the second output of the 1×2 switch for that segment can be connected to a second OCS switch for that logical axis.

The system determines a logical arrangement (1430) of a subset of compute nodes that matches the target arrangement of compute nodes. The logical arrangement can be a memory-based model of the layout of building blocks. For each dimension in m dimensions, the logical arrangement can define the connections between segments of each building block and corresponding segments of one or more other building blocks. For example, the logical arrangement can specify which building block will enter which position in the target arrangement of compute nodes. In a specific example, if the target arrangement is an 8×8×8 arrangement similar to the workload cluster in Figure 13, the logical arrangement can specify which building block is at the top, front, and left; which building block is at the top, right, and front; which building block is at the bottom, left, and front; which building block is at the bottom, right, and front; and so on.

Based on these positions, segments of building blocks that are on the same logical axis and adjacent along that axis will be connected to each other. For example, if one building block is logically above another building block, then each segment along the y-dimensional axis of the top building block will be connected to the corresponding segment of the bottom building block on the same logical axis.

The system generates a workload cluster of compute nodes, which consists of a subset of building blocks and is interconnected based on a logical arrangement (1440). The system can use composition operations 1450 and 1460 to generate the workload cluster.

For each dimension of the workload cluster, the system configures corresponding routing data (1450) for each of the two or more OCS switches in the dimension. The corresponding routing data for each dimension of the workload cluster specifies how the data for computing workloads is routed between compute nodes along the dimension of the workload cluster.

For example, if an optical network includes two OCS switches for each logical axis in each dimension, the system can be configured to route data between segments along the logical axis. Routing data allows the OCS switches to route data between adjacent segments on the same logical axis.

The system configures at least a portion of the one-to-many switch based on a logical arrangement, such that a second compute node in each segment is connected to the same OCS switch (1460) as the corresponding first compute node in the segment to which the second compute node in the logical arrangement is connected. For example, if a first building block is on top of a second building block, a segment of the first building block will need to be connected to the corresponding segment of the second building block on the same logical axis. If the first compute node of that segment of the first building block is connected to the first OCS switch for that logical axis, a one-to-many switch for that segment of the second building block can be configured such that the input of the one-to-many switch is routed to the first OCS switch.

The system enables compute nodes in the workload cluster to perform compute workloads (1470). For example, the system can provide compute workloads to compute nodes in the workload cluster. While the compute workloads are being performed, the configured OCS switches and one-to-many optical switches can route data between building blocks of the workload cluster. Although the compute nodes are not physically connected in the target arrangement, the configured OCS switches and one-to-many optical switches can route data between compute nodes in the building blocks as if the compute nodes were physically connected in the target arrangement.

Embodiments of the subject matter and operations described herein can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiving device for execution by the data processing device. The computer storage medium can be or is included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or combinations thereof. Furthermore, when the computer storage medium is not a propagated signal, it can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or included therein.

The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or on data received from other sources.

The term "data processing apparatus" includes all types of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination of the foregoing. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but must not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit)), and the apparatus can also be implemented as special-purpose logic circuitry.

For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for receiving or transferring data to, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not necessarily have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or integrated into dedicated logic circuits.

To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface or web browser through which a user can interact with the implementation of the subject matter described in this specification. Or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), inter-network (e.g., the Internet) and peer-to-peer networks (e.g., hoc peer-to-peer networks).

A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The relationship between clients and servers is generated by computer programs running on separate computers that have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., for displaying data to a user interacting with the client device and receiving user input from the user interacting with the client device). Data generated at the client device (e.g., the result of user interaction) may be received at the server from the client device.

While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of specific features of specific embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes shown in the figures do not necessarily require the specific order or sequence shown to obtain the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims (20)

1. A method executed by one or more data processing devices, characterized in that the method comprises: Identify the target arrangement of computation nodes used to calculate workload; A subset of the building blocks used for the computational workload is selected from a set of building blocks, each building block in the set comprising computational nodes arranged in m dimensions, wherein: Each building block is connected to an optical network, which includes two or more optical switches for each dimension in the m-dimensional space and a set of one-to-many switches for each building block. For each of the m dimensions: Each building block includes one or more segments of compute nodes interconnected along the said dimension, each segment including a first compute node at a first end of the segment and a second compute node at a second end of the segment opposite to the first end; For a first subset of the set of building blocks, the first compute node along each segment of the dimension is connected to the first optical switch among the two or more optical switches in the dimension; For each of one or more second subsets of the set of building blocks, the first compute node along each segment of the dimension is connected to a corresponding additional optical switch among the two or more optical switches for the dimension; and The second compute node of each segment of each building block is connected to the input of a corresponding one-to-many optical switch for the building block and segment, each one-to-many optical switch having the input and a plurality of outputs, wherein a first output of the plurality of outputs is connected to the first optical switch for the dimension, and each additional output of the plurality of outputs is connected to a corresponding additional optical switch for the dimension. Determine the logical arrangement of a subset of the building blocks that match the target arrangement of the computing nodes; Generate a workload cluster of compute nodes comprising the subset of the building blocks, the generation comprising: For each dimension of the workload cluster, corresponding routing data is configured for each of the two or more optical switches in that dimension, the corresponding routing data for each dimension of the workload cluster indicating how the computational workload data is routed along that dimension of the workload cluster between compute nodes; and Based on the logical arrangement, at least a portion of the one-to-many switch is configured such that the second computing node of each segment is connected to the same optical switch as the corresponding first computing node of the segment to which the second computing node in the logical arrangement is connected. 2. The method according to claim 1, wherein each optical switch comprises an optical path switch (OCS). 3. The method according to claim 1, characterized in that it further includes causing the workload cluster to perform the computational workload. 4. The method according to claim 1, wherein the set of one-to-many switches for each building block comprises a corresponding one-to-many switch for each segment of each dimension of the building block. 5. The method according to claim 1, wherein each segment of each dimension includes a logical axis along said dimension. 6. The method according to claim 1, characterized in that configuring at least a portion of the one-to-many switch based on the logical arrangement, such that the second computing node of each segment is connected to the same optical switch as the corresponding first computing node of the corresponding segment to which the second computing node is connected in the logical arrangement, includes: For a first building block in the subset, identify a second building block in the subset that is logically adjacent to the first building block along a specific dimension; For each segment of the first building block along the specific dimension: Identify the corresponding segment of the second building block; Identify the optical switch, which is connected to the first computing node of the corresponding segment of the second building block; and Configure the corresponding one-to-many switch among at least a portion of the one-to-many switches to which the segment is connected, so as to connect the second computing node of the segment to the identified optical switch. 7. The method according to claim 1, characterized in that: The one-to-many optical switch is a one-to-two optical switch with one input and two outputs. 8. The method according to claim 1, wherein the set of building blocks comprises a plurality of workload clusters, and wherein each workload cluster comprises a different subset of the building blocks. 9. The method according to claim 1, characterized in that it further comprises: Receive data indicating that a given building block in the workload cluster has expired; and Replace the given building block with an available building block. 10. The method of claim 9, wherein replacing the given building block with an available building block comprises: Update the routing data of one or more optical switches in the optical network to stop routing data between the given building block and one or more other building blocks in the workload cluster; and Update the routing data of the one or more optical switches in the optical network to route data between the available building blocks in the workload cluster and the one or more other building blocks. 11. A system for configuring a super platform, characterized in that it comprises: A set of building blocks, each building block consisting of compute nodes arranged in m dimensions; and An optical network comprising two or more optical switches for each dimension in m dimensions, and a set of one-to-many switches for each building block, wherein: Each building block is connected to the optical network; For each of the m dimensions: Each building block includes one or more segments of compute nodes interconnected along the said dimension, each segment including a first compute node at a first end of the segment and a second compute node at a second end of the segment opposite to the first end; For a first subset of the set of building blocks, the first compute node along each segment of the dimension is connected to the first optical switch among the two or more optical switches in the dimension; For each subset of one or more second subsets of the set of building blocks, the first compute node along each segment of the dimension is connected to a corresponding additional optical switch among the two or more optical switches for the dimension; and The second compute node of each segment of each building block is connected to the input of a corresponding one-to-many optical switch for the building block and segment. Each one-to-many optical switch has the input and a plurality of outputs, wherein a first output of the plurality of outputs is connected to the first optical switch for the dimension, and each additional output of the plurality of outputs is connected to a corresponding additional optical switch for the dimension. 12. The system of claim 11, further comprising an OCS manager implemented on one or more computers, the OCS manager being configured to: Determine the logical arrangement of a subset of the building blocks that matches the target arrangement of the computation nodes used to compute the workload; and Generate a workload cluster of compute nodes comprising the subset of the building blocks, the generation comprising: For each dimension of the workload cluster, corresponding routing data is configured for each of the two or more optical switches in that dimension, the corresponding routing data for each dimension of the workload cluster indicating how the computational workload data is routed along that dimension of the workload cluster between compute nodes; and At least a portion of the one-to-many switch is configured based on the logical arrangement such that the second computing node of each segment is connected to the same optical switch as the corresponding first computing node of the corresponding segment to which the second computing node is connected in the logical arrangement. 13. The system according to claim 11, wherein each optical switch comprises an optical path switch (OCS). 14. The system of claim 11, wherein the set of one-to-many switches for each building block comprises a corresponding one-to-many switch for each segment of each dimension of the building block. 15. The system of claim 11, wherein each segment of each dimension comprises a logical axis along said dimension. 16. The system according to claim 12, characterized in that configuring at least a portion of the one-to-many switches based on the logical arrangement, such that the second computing node of each segment is connected to the same optical switch as the corresponding first computing node of the corresponding segment to which the second computing node is connected in the logical arrangement, includes: For a first building block in the subset, identify a second building block in the subset that is logically adjacent to the first building block along a specific dimension; For each segment of the first building block along the specific dimension: Identify the corresponding segment of the second building block; Identify the optical switch, which is connected to the first computing node of the corresponding segment of the second building block; and Configure the corresponding one-to-many switch among at least a portion of the one-to-many switches to which the segment is connected, so as to connect the second computing node of the segment to the identified optical switch. 17. The system according to claim 11, characterized in that: The one-to-many optical switch is a one-to-two optical switch with one input and two outputs. 18. The system of claim 11, wherein the set of building blocks comprises a plurality of workload clusters, and wherein each workload cluster comprises a different subset of the building blocks. 19. A non-transitory computer storage medium encoded by a computer program, characterized in that the program includes instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to perform an operation, the operation comprising: Identify the target arrangement of computation nodes used to calculate workload; A subset of the building blocks used for the computational workload is selected from a set of building blocks, each building block in the set comprising computational nodes arranged in m dimensions, wherein: Each building block is connected to an optical network, which includes two or more optical switches for each dimension in the m-dimensional space and a set of one-to-many switches for each building block. For each of the m dimensions: Each building block includes one or more segments of compute nodes interconnected along the said dimension, each segment including a first compute node at a first end of the segment and a second compute node at a second end of the segment opposite to the first end; For a first subset of the set of building blocks, the first compute node along each segment of the dimension is connected to the first optical switch of the two or more optical switches of the dimension; For each of one or more second subsets of the set of building blocks, the first compute node along each segment of the dimension is connected to a corresponding additional optical switch for the two or more optical switches of the dimension; and The second compute node of each segment of each building block is connected to the input of a corresponding one-to-many optical switch for the building block and segment, each one-to-many optical switch having the input and a plurality of outputs, wherein a first output of the plurality of outputs is connected to the first optical switch for the dimension, and each additional output of the plurality of outputs is connected to a corresponding additional optical switch for the dimension. Determine the logical arrangement of a subset of the building blocks that match the target arrangement of the computing nodes; Generate a workload cluster of compute nodes comprising the subset of the building blocks, the generation comprising: For each dimension of the workload cluster, corresponding routing data is configured for each of the two or more optical switches in that dimension, the corresponding routing data for each dimension of the workload cluster indicating how the computation workload data is routed along that dimension of the workload cluster between compute nodes; and Based on the logical arrangement, at least a portion of the one-to-many switch is configured such that the second computing node of each segment is connected to the same optical switch as the corresponding first computing node of the segment to which the second computing node in the logical arrangement is connected. 20. The non-transitory computer storage medium according to claim 19, wherein the set of one-to-many switches for each building block comprises a corresponding one-to-many switch for each segment of each dimension of the building block.