CN112131166B - Lightweight bridge circuit and operation method thereof - Google Patents

Abstract

Lightweight bridge circuits and methods of operating the same are disclosed. An endpoint of a lightweight bridge (LWB) may expose multiple Physical Functions (PFs) to a host. The root port of the LWB may be connected to the device and determine the PF and Virtual Function (VF) that are disclosed by the device. Application layer-endpoints (APP-EP) and application layer-root ports (APP-RP) may switch between PFs disclosed by the endpoints and PFs/VFs disclosed by the devices. APP-EP and APP-RP may implement a mapping between PFs disclosed by the endpoints and PFs/VFs disclosed by the devices.

Description

Lightweight bridge circuit and operation method thereof

This application claims the benefit of U.S. Provisional Patent Application No. 62/865,962, filed on June 24, 2019, and U.S. Provisional Patent Application No. 62/964,114, filed on January 21, 2020, both of which are incorporated herein by reference for all purposes.

Technical Field

The inventive concept relates generally to storage devices, and more particularly, to emulating a Peripheral Component Interconnect Express (PCIe) virtual function (VF) as a PCIe physical function (PF).

Background technique

A device, such as a Peripheral Component Interconnect Express (PCIe) device, exposes various functions that can be accessed by other components in a computer system. For example, a host processor can use such functions exposed by a solid-state drive (SSD) to perform various operations within the SSD. These functions can operate on data stored on the SSD, or can operate on data provided by the host. Typically, the functions exposed by the device relate to normal operation of the device, but such a limitation is not required: for example, although SSDs are traditionally used to store data, if the SSD includes a processor, the processor can be used to offload processing from the host processor.

The functions exposed by the device may be discovered by the host machine when the device is enumerated at startup (or, if hot installation of the device is supported, when installed). As part of discovery, the host machine may query the device for any exposed functions, which may then be added to the device's list of available functions.

Functions are divided into two categories: physical and virtual. A physical function (PF) can be implemented using hardware within the device. The resources of a PF can be managed and configured independently of any other PF provided by the device. A virtual function (VF) is a lightweight function that can be thought of as a virtualized function. Unlike a PF, a VF is usually associated with a specific PF and usually shares resources with the PF it is associated with (and may also share resources with other VFs associated with that PF).

Because PFs are independent of each other, and because VFs may need to use Single Root Input/Output Virtualization (SR-IOV) in order for the host to access the VFs, it is preferred that the device expose PFs. However, because PFs are independent, they may require separate hardware, increasing the space required within the device and the power consumption of the device. The SR-IOV protocol can impose complexity in the host system software stack, especially for virtualization use cases.

There remains a need to provide PF for devices without imposing cost requirements for PF so that system software complexity can be reduced.

Summary of the invention

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

An aspect of the embodiment relates to a lightweight bridge (LWB) circuit, the lightweight bridge circuit comprising: an endpoint for connecting to a host, the endpoint exposing a plurality of physical functions (PFs); a root port for connecting to a device, the device exposing at least one PF and at least one virtual function (VF) to the root port; and an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) for converting between the plurality of PFs exposed to the host and the at least one PF and the at least one VF exposed by the device. The APP-EP and the APP-RP implement mapping between the plurality of PFs exposed by the endpoint and the at least one PF and the at least one VF exposed by the device.

Another aspect of an embodiment relates to a method comprising: using a root port of a lightweight bridge (LWB) to enumerate at least one physical function (PF) exposed by a device; using the root port of the LWB to enumerate at least one virtual function (VF) exposed by the device; generating multiple PFs at an endpoint of the LWB to be exposed to a host; and using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB, mapping the multiple PFs at the endpoint of the LWB to the at least one PF and the at least one VF exposed by the device.

Yet another aspect of the embodiments relates to an article comprising a non-transitory storage medium having instructions stored on the non-transitory storage medium, which, when executed by a machine, causes: enumerating at least one physical function (PF) exposed by a device using a root port of a lightweight bridge (LWB); enumerating at least one virtual function (VF) exposed by the device using the root port of the LWB; generating multiple PFs at an endpoint of the LWB to be exposed to a host; and mapping the multiple PFs at the endpoint of the LWB to the at least one PF and the at least one VF exposed by the device using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB.

Other features and aspects will be apparent from the following detailed description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 illustrates a machine including a lightweight bridge (LWB) capable of emulating a physical function (PF) exposed by the LWB using a virtual function (VF) of a solid state drive (SSD) according to an embodiment of the inventive concept.

FIG. 2 shows additional details of the machine of FIG. 1 .

FIG. 3 shows details of the SSD of FIG. 1 .

4A to 4C illustrate the LWB of FIG. 1 according to various embodiments of the inventive concept.

FIG. 5 shows details of the configuration manager of FIGS. 4A-4C .

FIG. 6 shows a mapping between a PF disclosed by the LWB of FIG. 1 and a PF/VF disclosed by the SSD of FIG. 1 .

FIG. 7 illustrates the LWB of FIG. 1 processing a configuration write request from the host of FIG. 1 .

FIG. 8 illustrates the LWB of FIG. 1 processing a configuration read request from the host of FIG. 1 .

FIG. 9 shows that the Application Layer-Endpoint (APP-EP) and Application Layer-Root Port (APP-RP) of FIG. 4A to FIG. 4C handle address translation within the LWB.

FIG. 10 shows that the mapping of FIG. 6 is changed within the LWB of FIG. 1 .

FIG. 11 shows that the PF exposed by the LWB of FIG. 1 has an associated quality of service (QoS) policy.

12A-12B illustrate that the LWB of FIG. 1 performs bandwidth throttling.

FIG. 13 shows the LWB of FIG. 1 issuing credits to the SSD of FIG. 1 .

14A-14B illustrate a flowchart of an example process of the LWB of FIG. 1 identifying a PF/VF disclosed by the SSD of FIG. 1 , disclosing the PF from the LWB, and generating a mapping between the PF disclosed by the LWB of FIG. 1 and the PF/VF disclosed by the SSD of FIG. 1 , according to an embodiment of the inventive concept.

15A-15B illustrate a flow chart of an example process in which the LWB of FIG. 1 receives and processes a request from the host of FIG. 1 .

FIG. 16 is a flowchart illustrating an example process of converting addresses of the APP-EP and the APP-RP of FIGS. 4A to 4C between the host of FIG. 1 and the SSD of FIG. 1 .

FIG. 17 illustrates a flow chart of an example process by which the LWB of FIG. 1 issues credits to the SSD of FIG. 1 .

FIG. 18 is a flow chart showing an example process of the LWB of FIG. 1 processing a configuration write request.

FIG. 19 is a flow chart illustrating an example process of the LWB of FIG. 1 processing a configuration read request.

20 illustrates a flow chart of an example process by which the LWB of FIG. 1 associates a QoS policy with a PF published by the LWB of FIG. 1 .

21 illustrates a flow chart of an example process by which the LWB of FIG. 1 dynamically changes a mapping of a PF disclosed by the LWB of FIG. 1 to a PF/VF disclosed by the SSD of FIG. 1 .

22A-22B illustrate a flow chart of an example process in which the LWB of FIG. 1 performs bandwidth throttling.

Detailed ways

Reference will now be made in detail to embodiments of the invention, examples of which are illustrated in the accompanying drawings. In the following detailed description, many specific details are set forth to provide a thorough understanding of the invention. However, it should be understood that one of ordinary skill in the art may practice the invention without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail to avoid unnecessarily obscuring aspects of the embodiments.

It will be understood that although the terms first, second, etc. may be used herein to describe different elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the invention, a first module may be referred to as a second module, and similarly, a second module may be referred to as a first module.

The terms used in the description of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the invention. As used in the description of the invention and the appended claims, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that the term "and/or" as used herein represents and includes any and all possible combinations of one or more of the relevant listed items. It will also be understood that the terms "including" and/or "comprising" when used in this specification indicate the presence of the features, wholes, steps, operations, elements and/or components of the description, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and/or their groups. The components and features of the accompanying drawings do not have to be drawn to scale.

Embodiments of the inventive concept include methods and systems for emulating multiple physical functions (PFs) in a device using virtual functions (VFs) supported by a device, such as a solid-state drive (SSD). An SSD may have multiple non-volatile memory express (NVMe) controllers represented by multiple PCIe physical functions (PFs) or PCIe virtual functions (VFs). Each PF or VF essentially presents an NVMe controller that can be used by a host NVMe drive to perform data storage functions. The PF has actual physical resources, while the virtual functions share resources with the PFs. Embodiments of the inventive concept may use a collection of VFs in an SSD controller to expose (or expose) multiple PFs to a host. That is, multiple PFs may be emulated by a device, and the VFs may be used internally. From the perspective of the host, the host system software stack sees multiple PCIe PFs and the NVMe controllers behind those PFs. Embodiments of the inventive concept may map and convert all host accesses to PFs to VFs internally.

PCIe PFs are independent of any other functions (physical or virtual). That is, PFs have their own dedicated physical resources (such as memory buffers). For devices that support a large number of PFs, the logic area, power consumption, and complexity of the underlying device are increased. Therefore, in order to reduce the cost and complexity on the device side, the PCIe specification provides support for virtual functions (VFs). VFs share physical resources with PFs and rely on PFs for all physical aspects. These physical aspects include PCIe link control, device control, power management, etc.

Although VFs reduce the cost and complexity on the device side, VFs increase the complexity on the system software side. System software needs to support the Single Root Input/Output Virtualization (SR-IOV) protocol to be able to communicate with VFs. This added functionality sometimes reduces I/O performance in terms of additional latency. Therefore, from a system software perspective, it is desirable to have PFs.

Embodiments of the inventive concept allow a device to emulate PFs using VFs in an SSD controller. That is, from the perspective of the host, the storage device SSD appears to have multiple PFs. But on the device side, those PFs can be emulated using a collection of VFs. To reduce costs, a lightweight bridge (LWB) can be used to expose the functions of a device (such as an SSD) without implementing the various functions as PFs in an SSD controller ASIC.

In an example implementation of the LWB, a total of 16 functions may be exposed by the LWB as if they were physical functions, even though the underlying device may implement some or most of the functions as VFs instead of PFs. The LWB acts as a bridge between the host machine and the SSD controller itself: the LWB may be implemented as part of the entire SSD device, or as a separate component. The device may be an SSD with the functions (1 PF and 15 VFs) implemented as part of the endpoints of the SSD controller. (SSDs may also conventionally include a host interface layer (HIL), a flash translation layer (FTL), and a flash controller (FC) to access the flash memory.)

The LWB may use four lanes of a third generation (Gen3) PCIe bus to communicate with the host machine, while internally the LWB may use 16 lanes of the Gen3 PCIe bus to achieve communication. However, embodiments of the inventive concept may support the use of any particular version of the PCIe bus (or other bus type), and may support any desired speed or lane or bandwidth with and within the host machine without limitation. The PCIe lane widths and speeds in this description are examples only, and it should be understood that any combination may be achieved using the same concept.

Instead of the SSD controller communicating with a root port or root complex on a host machine, the SSD controller may communicate with a root port of the LWB. The SSD controller may be unaware of this change and may treat communications from the root port of the LWB as communications from the host machine. Similarly, the host machine may communicate with an endpoint of the LWB without knowing that the endpoint is not an SSD controller (implementing disclosed functionality). The party communicating with it (in embodiments of the inventive concept, the LWB) may be treated as a black box as far as the SSD controller or host machine is concerned.

The endpoints of the LWB may expose the same number of functions as the endpoints of the SSD controller. However, the endpoints of the LWB may expose all functions as PFs instead of exposing some of these functions as VFs. The LWB may also include a PCIe Application Layer-Endpoint (PAPP-EP) and a PCIe Application Layer-Root Port (PAPP-RP). The PAPP-EP and PAPP-RP may manage the mapping from PFs as exposed by the endpoints of the LWB to functions (physical or virtual) as exposed by the endpoints of the SSD controller. The PAPP-RP may include a configuration conversion table to assist in managing the mapping of the PCIe configuration space (Configuration Space) of the PF exposed to the host to the PCIe configuration space of the PF and/or VF of the SSD controller EP. This table may also indicate which PF exposed by the LWB endpoint maps to which function exposed by the SSD controller endpoint, as well as other information related to the mapping (e.g., what address in the SSD controller can store data for the exposed function). The PCIe configuration features and capabilities provided by the endpoints of the LWB may (and often do) differ from those provided by the endpoints of the SSD controller: the configuration translation table may help manage these differences. The PAPP-RP may also handle the translation of the memory base address register (BAR) addresses of the PF exposed to the host and the BAR addresses of the PF and/or VF of the SSD controller EP. The PAPP-EP and/or PAPP-RP may also include other tables as needed. The mapping of the PF exposed to the host and the PF/VF internal to the SSD controller may be flexible and dynamic in nature. The mapping may change during runtime based on certain events and/or policy changes issued by a management entity such as the host and/or baseboard management controller (BMC). Some examples of these events are virtual machine (VM) migration, changes in SLA, power/performance throttling, date, time, etc.

Although in some embodiments of the inventive concept the PAPP-EP and PAPP-RP may be separate components, other embodiments of the inventive concept may implement these components (and potentially the endpoints, root ports, and configuration manager of the LWB) in a single implementation. For example, the LWB may be implemented using a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), just two examples of possible implementations. The PAPP-EP and PAPP-RP may communicate using any desired mechanism. For example, an intermediate data format may be defined that may be used to exchange data between the PAPP-EP and the PAPP-RP to call a particular function and return a result.

The LWB may also include a configuration manager. The configuration manager may enumerate the functions (both PF and VF) provided by the endpoints of the SSD controller. The configuration manager may then use the information of both to "define" the functions exposed by the LWB's endpoints and help construct the configuration space translation table in the PAPP-RP. Typically, the configuration manager is used primarily during boot. However, the host machine may change the PCIe configuration space of the PF (such as interrupts for communicating with the (LWB's) endpoints for various functions), or the host machine may enable or disable specific functions. The configuration manager may be used to help make these changes as needed. The LWB may also need to ensure that similar PCIe configuration space changes are propagated to the SSD controller when appropriate. That is, the LWB performs PCIe configuration mirroring between the PF exposed to the host and the PF/VF exposed by the internal SSD controller EP. Therefore, the LWB may also manage the functions exposed by the SSD controller.

Because there are two endpoints in the entire device (one in the LWB and one in the SSD controller), each endpoint maintains its own configuration space. These configuration spaces should be synchronized to avoid potential problems or conflicts. When the host sends a configuration write command, the LWB can update the configuration space in the LWB endpoint. The configuration write command can also be forwarded to the endpoint of the SSD (via PAPP-EP, PAPP-RP, and root port) so that the endpoint of the SSD controller can also be updated. Not all configuration write commands initiated by the host can be reflected to the SSD controller. That is, some of the changes in the host configuration space may not be reflected to the back-end SSD controller as is. For example, the host may make power management changes to the LWB EP, but this change or similar changes may not be made by the LWB to the SSD controller EP.

When the host sends a configuration read command, the LWB's endpoint can satisfy the command. The LWB's endpoint can forward the configuration read command to the PAPP-EP, but the PAPP-EP can terminate the command: when the two configuration spaces are synchronized, the data in each configuration space should be the same. (Alternatively, the LWB's endpoint can respond to the configuration read command and terminate the command at this point without forwarding the configuration read command to the PAPP-EP.)

When the LWB receives a memory read or write transaction (from the host via an endpoint of the LWB or from the SSD controller via a root point of the LWB), the LWB may forward the transaction to the other party (host or SSD controller as appropriate). The LWB may use the BAR tables in the PAPP-EP and/or PAPP-RP to perform the appropriate address translation. An example of such an address translation for a transaction received from the host is that the PAPP-EP may subtract the address in the PF BAR from the received address, and the PAPP-RP may add the address in the VF BAR to the address to reach the correct address in the SSD controller.

Embodiments of the inventive concept may also include an optional intermediate element between the PAPP-EP and the PAPP-RP, which may be used for mapping, bandwidth control, and the like. This optional element may also be used to variably "throttle" bandwidth for a variety of reasons. For example, different functions exposed by the endpoints of the LWB may have different quality of service (QoS) requirements or service level agreements (SLAs). PFs with low bandwidth QoS may be throttled to ensure that there is sufficient bandwidth for PFs with higher QoS bandwidth requirements. Other reasons for throttling bandwidth may include power or temperature. The LWB may be used to perform bandwidth throttling on a per-PF basis, or it may perform bandwidth throttling for all PFs based on configuration settings or policy settings of a management entity (such as a host and/or BMC). The temperature or power throttling decision may be based on two thresholds, high_limit and low_limit. When power consumption or temperature exceeds high_limit, bandwidth throttling may be applied to all PFs or selective PFs based on policy. Bandwidth throttling may be applied until power or temperature drops below the respective low_limit thresholds.

In the example embodiments of the inventive concept above, the endpoints of the SSD controller are described as providing 1 PF and 15 VFs, which can be mapped to 16 different PFs at the endpoints of the LWB. These numbers are arbitrary: the endpoints of the SSD controller can provide any number of PFs and VFs, which can be mapped to 16 PFs on the endpoints of the LWB, or more or fewer PFs.

If necessary, the VFs on the SSD controller may be appropriately migrated within the LWB with respect to any appropriate changes made within the LWB and/or SSD controller. Thus, the mapping from the PFs exposed by the endpoints of the LWB to the functions exposed by the SSD controller may be modified at runtime, so the mapping is flexible.

Communications within the LWB may be performed using any desired technology. For example, information exchanged between endpoints, PAPP-EPs, PAPP-RPs, and root ports may use the Transaction Layer Packet (TLP) protocol.

The device itself may have any form factor. For example, an SSD may be packaged using a U.2 or Full Height Half Length (FHHL) form factor, among other possibilities. Other types of devices may similarly be packaged in any desired form factor.

In another embodiment of the inventive concept, multiple SSD controllers may be included, each providing any number of functions: for example, each SSD controller may provide a total of 8 functions. Incorporated between the endpoints of the LWB may be a multiplexer/demultiplexer that can direct data associated with a particular disclosed PF to the appropriate PAPP-EP/PAPP-RP/root port/SSD controller for final execution. In this way, a single LWB may expose more functions than a single SSD controller could possibly provide.

Each SSD controller can communicate with a separate root port in the LWB, and each SSD controller has its own slice consisting of PAPP-EP, PAPP-RP, and configuration translation table. In this way, operations involving one SSD controller may not have an impact on operations involving other SSD controllers within the LWB.

For example, embodiments of inventive concepts such as these may be used when the endpoints on a single SSD controller do not meet all the needs of a host machine. For example, if an SSD controller only supports 8 PCIe lanes but the host wants to support the bandwidth of 16 lanes, multiple SSD controllers may be used to support the required bandwidth. Or if an SSD controller only supports eight functions in total and the host wants to support 16 functions, multiple SSD controllers may be used to supply the required set of functions. That is, LWB may be used to connect any number of SSD controllers to a host as a single multi-function PCIe storage device using an appropriate number of PAPP-EP/PAPP-RP/RP slices.

As with other embodiments of the inventive concept, the numbers used are exemplary and not limiting. Thus, an endpoint of the LWB may use any type of bus, any version of that bus, and any number of lanes on that bus; an endpoint of the LWB may expose any number of PFs; the number of PFs exposed by each SSD controller may vary; the internal bus type, version, and speed may vary, as may the connections to the SSD controllers (and may also vary individually); the number of SSD controllers connected to the LWB may vary (and may be connected to different or the same routing points in the LWB); etc.

FIG1 illustrates a machine including a lightweight bridge (LWB) according to an embodiment of the inventive concept, which lightweight bridge is capable of emulating a physical function (PF) exposed by the LWB using a virtual function (VF) of a device. In FIG1 , a machine 105, which may also be referred to as a host, is shown. The machine 105 may include a processor 110. The processor 110 may be any kind of processor: for example, an Intel Xeon, Celeron, Itanium or Atom processor, an AMD Opteron processor, an ARM processor, etc. Although FIG1 illustrates a single processor 110 in the machine 105, the machine 105 may include any number of processors, each of which may be a single-core processor or a multi-core processor, and may be mixed in any desired combination.

The machine 105 may also include a memory 115. The memory 115 may be any of a variety of memories, such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory, ferroelectric random access memory (FRAM), or non-volatile random access memory (NVRAM) (such as magnetoresistive random access memory (MRAM), etc.). The memory 115 may also be any desired combination of different memory types. The machine 105 may also include a memory controller 120, which may be used to manage access to the memory 115.

The machine 105 may also include a solid state drive (SSD) 125. The SSD 125 may be used to store data, and may expose both physical functions (PFs) and virtual functions (VFs) for use by the processor 110 (and software executed thereon) to call various functions of the SSD 125. Although FIG. 1 shows an SSD 125, embodiments of the inventive concept may include any type of device that provides PFs and VFs. For example, other types of storage devices (such as hard disk drives) that provide PFs and VFs may be used in place of the SSD 125, and devices that provide basic functionality other than data storage may also replace the SSD 125. In the remainder of this document, any reference to the SSD 125 is intended to include references to other types of devices that can provide PFs/VFs. In some embodiments of the inventive concept, the SSD 125 may use a peripheral component interconnect express (PCIe) bus and provide PCIe PFs and VFs, but embodiments of the inventive concept may use other interfaces. The processor 110 may run a device driver 130, which may support access to the SSD 125.

Located between the SSD 125 and the rest of the machine 105 may be a lightweight bridge (LWB) 135. The LWB may act as a bridge between the SSD 125 and the rest of the machine 105, and may expose the functionality of the SSD 125 but as a PF rather than a VF.

Although FIG. 1 depicts machine 105 as a server (the server may be a stand-alone server or a rack server), embodiments of the inventive concept may include any desired type of machine 105 without limitation. For example, machine 105 may be replaced with a desktop computer or laptop computer or any other machine that may benefit from embodiments of the inventive concept. Machine 105 may also include dedicated portable computing machines, tablet computers, smart phones, and other computing machines. Additionally, applications that may access data from SSD 125 may be located in another machine that is separate from machine 105 and accesses machine 105 via a network connection across one or more networks of any type (wired, wireless, global, etc.).

FIG. 2 shows additional details of the machine 105 of FIG. 1 . In FIG. 2 , generally, the machine 105 includes one or more processors 110, which may include a memory controller 215 and a clock 205, which may be used to coordinate the operation of the components of the machine 105. The processor 110 may also be coupled to a memory 115, which may include, by way of example, a random access memory (RAM), a read-only memory (ROM), or other state-retaining medium. The processor 110 may also be coupled to a storage device 125, and to a network connector 210, which may be, for example, an Ethernet connector or a wireless connector. The processor 110 may also be connected to a bus 215, to which may be attached, among other components, input/output (I/O) interface ports that may be managed using an input/output engine 225, and a user interface 220.

FIG3 shows details of the SSD 125 of FIG1 . In FIG3 , the SSD 125 may include a host interface logic (HIL) 305, an SSD controller 310, and various flash memory chips 315-1 to 315-8 (also referred to as “flash memory”), which may be organized into various channels 320-1 to 320-4. The host interface logic 305 may manage communications between the SSD 125 and other components (such as the processor 110 of FIG1 and the LWB 135 of FIG1 ). The host interface logic 305 may also manage communications with devices that are remote from the SSD 125 (i.e., devices that are not considered part of the machine 105, but communicate with the SSD 125, for example, through one or more network connections). These communications may include read requests for reading data from the SSD 125, write requests for writing data to the SSD 125, and delete requests for deleting data from the SSD 125. Host interface logic 305 may manage interfaces through only a single port, or host interface logic 305 may manage interfaces through multiple ports. Alternatively, SSD 125 may include multiple ports, each of which may have a separate host interface logic 305 to manage interfaces through that port. Embodiments of the inventive concept may also mix possibilities (e.g., an SSD with three ports may have a first host interface logic for managing one port and a second host interface logic for managing the other two ports).

The SSD controller 310 may use a flash controller (not shown in FIG. 3 ) to manage read and write operations on the flash chips 315-1 to 315-8, as well as garbage collection and other operations. The SSD controller 310 may include a flash translation layer 325, a flash controller 330, and endpoints 335. The flash translation layer may manage the mapping of logical block addresses (LBAs) (such as those used by the host 105 of FIG. 1 ) to physical block addresses (PBAs) where data is actually stored on the SSD 310. By using the flash translation layer 325, the host 105 of FIG. 1 does not need to be notified when data is moved from one block to another within the SSD 125.

The flash controller 330 may manage writing of data to the flash chips 315-1 to 315-8 and reading of data from the flash chips 315-1 to 315-8. The endpoint 335 may act as an endpoint of the SSD 125, and the endpoint 335 may be connected to a root port on another device (such as the host 105 of FIG. 1 or the LWB 135 of FIG. 1).

Although FIG3 shows SSD 125 as including eight flash memory chips 315-1 to 315-8 organized into four channels 320-1 to 320-4, embodiments of the inventive concept may support any number of flash memory chips organized into any number of channels. Similarly, although FIG3 shows the structure of an SSD, other storage devices (e.g., hard disk drives) may be implemented using different (but similarly potentially beneficial) structures.

4A to 4C illustrate the LWB 135 of FIG. 1 according to various embodiments of the inventive concept. In FIG. 4A , the LWB 135 is shown as including an endpoint 405, an application layer-endpoint (APP-EP) 410, an application layer-root port (APP-RP) 415, and a root port 420. The endpoint 405 can be used to communicate with the root port of other devices (such as the host 105 of FIG. 1 ). In FIG. 4A , the endpoint 405 is shown as exposing 16 PFs (PF0-PF15) to the upstream device, and is shown as using a PCIe Gen 3 (Gen 3×4) interface with four lanes, and the endpoint 405 can be used to communicate with the upstream device. Therefore, as shown by interface 425, the interface to the host 105 of FIG. 1 can be a PCIe Gen 3 interface with four lanes. Similarly, the root port 420 can communicate with the endpoints of other devices (such as, SSD 125). Thus, as shown by interface 430, the interface to SSD 125 may be a PCIe Gen 3 interface with four lanes. In FIG4A, root port 420 is shown as using a PCIe Gen 3 interface with four lanes, which root port 420 may be used to communicate with downstream devices. In FIG4A, root port 420 is shown as having enumerated one PF and 15 VFs from SSD 125, and endpoint 405 is shown as exposing 16 PFs: one PF for each function (physical or virtual) exposed by endpoint 405.

Endpoint 405 may communicate with APP-EP 410, which in turn may communicate with APP-RP 415. APP-EP 410 and APP-RP 415 may manage the translation of information between endpoint 405 and root port 420. FIG4A does not show the details of how endpoint 405 communicates with APP-EP 410, how APP-EP 410 communicates with APP-RP 415, or how APP-RP 415 communicates with root port 420, because any desired communication mechanism may be used. For example, endpoint 405, APP-EP 410, APP-RP 415, and root port 420 may communicate using PCIe or another bus, such as an Advanced eXtensible Interface (AXI) bus, with any desired version of interface and including any desired number of data bus widths. Alternatively, endpoints 405, APP-EP 410, APP-RP 415, and root port 420 may communicate using a proprietary messaging scheme using any desired bus/interface. Embodiments of the inventive concept may also include other mechanisms for communication between endpoints 405, APP-EP 410, APP-RP 415, and root port 420. There may be no relationship between the communication method within LWB 135 and the communication method from LWB 135 to host 105 or SSD 125 of FIG. 1.

APP-RP 415 may include a configuration table 435. Configuration table 435 may store configuration information about LWB 135. Examples of such configuration information may include mappings between PFs exposed by endpoint 405 and PFs/VFs provided by SSD 125 (and determined via enumeration through root port 420). Configuration information stored in configuration table 435 may also include information about a Quality of Service (QoS) policy (which may also be referred to as a Service Level Agreement (SLA)) that may be associated with a particular PF exposed by endpoint 405. Configuration manager 440 may be used to configure LWB 135 storing information in configuration table 435. Configuration manager 440 may also be used to determine information about functions exposed by SSD 125, and may program endpoint 405 to provide similar (or identical) functionality (but using only PFs, rather than PFs and VFs). The details of what configuration manager 440 does may depend on the specific functionality provided by SSD 125 (because configuration manager 440 may configure endpoint 405 to provide a set of PFs that match the PFs and VFs provided by SSD 125), but the principles can generally be summarized as: determining the configuration of each PF/VF exposed by SSD 125, establishing an appropriate number of PFs to be exposed by endpoint 405, and configuring those PFs to match the configuration of the PFs/VFs exposed by SSD 125.

Although FIG. 4A shows APP-EP 410 and APP-RP 415 as separate components, embodiments of the inventive concept may combine these two components into a single component (responsible for handling the functions of both APP-EP 410 and APP-RP 415 as described). In fact, the entirety of LWB 135 may be implemented as a single unit, rather than as separate components that communicate with each other. LWB 135 (as well as endpoints 405, APP-EP 410, APP-RP 415, root port 420, and configuration manager 440) may be implemented in any desired manner. Among other possibilities, embodiments of the inventive concept may implement LWB 135 (and/or its components) using a general purpose processor, field programmable gate array (FPGA), application specific integrated circuit (ASIC), graphics processing unit (GPU), and general purpose GPU (GPGPU) with appropriate software (individually and/or collectively).

In FIG. 4A (and similarly to FIGS. 4B-4C below), any specific numbers shown are intended to be examples only, and embodiments of the inventive concept may include other numbers. For example, root port 420 is shown as having enumerated 1 PF and 15 VFs from SSD 125, but SSD 125 may expose any number (zero or more) of PFs and VFs (although since any exposed VF relies on hardware from a certain PF, there must be at least one exposed PF to provide hardware for any exposed VF). Similarly, endpoint 405 is shown as exposing 16 PFs, but endpoint 405 may expose any number (zero or more) of PFs to host 105 of FIG. 1. Note that while endpoint 405 exposes the same total number of functions as SSD 125, there need not be a one-to-one correspondence between the functions exposed by SSD 125 and the functions exposed by endpoint 405. For example, some functions exposed by SSD 125 may not have corresponding PFs exposed by endpoint 405, or a single PF exposed by endpoint 405 may map to multiple functions (physical and/or virtual) exposed by SSD 125. As an example of the former, write requests may be directed to only one SSD at a time, which would cause functions supporting write requests to the unselected SSDs to not have corresponding PFs exposed by endpoint 405. As an example of the latter, in an embodiment of the inventive concept where LWB 135 is connected to multiple SSDs (such as shown in FIG. 4C below), even though endpoint 405 may only expose a single PF to host 105 of FIG. 1 to request to perform garbage collection, the request for host 105 of FIG. 1 to perform garbage collection may be sent to all SSDs connected to LWB 135.

In a similar manner, any version of a particular product shown in FIG. 4A is intended to be merely an example, and embodiments of the inventive concept may use other products or other versions of products. For example, FIG. 4A shows LWB 135 communicating with host 105 (via interface 425) of FIG. 1 and with SSD 125 (via interface 430) using PCIe Gen 3 with four lanes. Other embodiments of the inventive concept may use other generations of PCIe buses/interfaces, or other numbers of lanes, or even other types of buses (such as, Advanced eXtensible Interface (AXI) buses). For example, FIG. 4B is similar to FIG. 4A except for the fact that interface 425 and interface 430 use PCIe Gen 3×8 with eight lanes instead of four lanes.

In some embodiments of the inventive concept, a single SSD 125 may not provide sufficient capabilities for the purposes of the host 105 of FIG1. For example, if the SSD 125 of FIGS. 4A-4B only provides 1 PF and 7 VFs, and therefore does not provide a particular functionality desired by the host 105 of FIG1, it would be possible to replace the SSD 125 with another device that provides the desired functionality. However, as an alternative, as shown in FIG4C, the LWB 135 may be connected to multiple devices.

In FIG. 4C , LWB 135 is shown connected to SSD 125-1 and SSD 125-2 via interfaces 430-1 and 430-2, respectively. Because SSD 125-1 and SSD 125-2 each provide 1 PF and 7 VFs (as can be seen in root ports 420-1 and 420-2), neither of SSD 125-1 and 125-2 can provide 16 functions individually. However, SSD 125-1 and 125-2 together provide 16 functions. Therefore, by aggregating the resources (such as the public functions) of SSD 125-1 and 125-2, LWB 135 can provide a total of 16 PFs without a single SSD providing all functions.

As can be seen in FIG. 4C , LWB 135 includes only one endpoint 405, but includes two APP-EPs 410-1 and 410-2, two APP-RPs 415-1 and 415-2, and two root ports 420-1 and 420-2. In this way, LWB 135 can expose all 16 functions from a single endpoint (endpoint 405) that aggregates the functions provided by SSDs 125-1 and 125-2. However, because different PFs exposed by endpoint 405 can be mapped to functions on SSDs 125-1 and 125-2, LWB 135 can also include a multiplexer/demultiplexer (MDM) 445. Multiplexer/demultiplexer 445 can route requests associated with a particular exposed PF of endpoint 405 to the appropriate APP-EP 410-1 or 410-2, whose communication path leads to SSD 125-1 or 125-2 that provides the corresponding PF or VF. Although FIGS. 4A-4B do not show a multiplexer/demultiplexer 445 (since there are only SSDs 125 connected to the LWB 135), in embodiments of the inventive concept where the LWB 135 is only connected to one SSD 125, the LWB 135 may still include a multiplexer/demultiplexer 445: the multiplexer/demultiplexer 445 may not add any functionality, but there is also no real cost associated with including the multiplexer/demultiplexer 445. (But if the LWB 135 is only capable of connecting to a single device, then including the multiplexer/demultiplexer also does not add any benefit.)

Note that embodiments of the inventive concept may aggregate resources other than just the disclosed functions. For example, in FIG. 4C , interfaces 430-1 and 430-2 may each include eight lanes of a PCIe Gen 3 bus/interface. However, because SSDs 125-1 and 125-2 may be used in parallel (i.e., both SSDs 125-1 and 125-2 may process requests simultaneously), LWB 135 may provide 16 lanes (Gen 3×16) of a PCIe Gen 3 bus/interface to host 105 of FIG. 1 via interface 425. In the same manner, the bandwidth provided by each of SSDs 125-1 and 125-2 may be aggregated so that LWB may advertise a bandwidth equal to the sum of the bandwidths provided by SSDs 125-1 and 125-2, respectively. Embodiments of the inventive concept may also aggregate other resources other than functions, PCIe lanes, or bandwidth.

As with Figures 4A to 4B, any number or version shown in Figure 4C is used as an example only, and embodiments of the inventive concept may support other numbers or versions. For example, SSDs 125-1 and 125-2 do not need to provide the same number of public functions (physical and/or virtual), nor do they necessarily need to use the same generation or number of PCIe bus/interface paths.

In FIGS. 4A to 4C , the LWB 135 is shown as being separate from the SSD 125. However, embodiments of the inventive concept may combine the two devices into a single unit. That is, the SSD 125 may include the LWB 135 within its housing, located before the host interface logic 305 of FIG. 3 (or at least before the SSD controller 310 of FIG. 3 ). Embodiments of the inventive concept are also not limited to including the LWB 135 within a single SSD: a single housing containing multiple SSDs may also include the LWB 135 to implement the implementation shown in FIG. 4C . For the purposes of this document, the term "connected" and similar terms are intended to represent any device with which the LWB 135 communicates, whether by virtue of physically sharing any hardware with the device or by virtue of being connected through some interface because the LWB 135 is physically different from the SSD 125.

As described above, one of the problems with devices that disclose VFs is that the host 105 of FIG. 1 may need to implement SR-IOV software to access the disclosed VFs. The SR-IOV software increases the delay in communicating with the device. However, since the SR-IOV sequence for managing access to the VFs of the device is known in advance, the SR-IOV sequence can be implemented using a state machine, alleviating the burden of the host 105 of FIG. 1 having to use SR-IOV to access the VFs of the device. In addition, because the state machine can be implemented using hardware, the use of a state machine within the LWB 135 instead of the SR-IOV software within the host 105 of FIG. 1 can also reduce the delay in processing requests. Using a state machine to implement the SR-IOV sequence within the LWB 135 essentially reduces and hides all the burdens of the SR-IOV protocol of the device from the host. In other words, the host 105 in FIG. 1 can be unaware of the existence of any implementation of SR-IOV within the device with which the host 105 of FIG. 1 is communicating.

FIG5 illustrates details of the configuration manager 440 of FIGS. 4A to 4C . In FIG5 , the configuration manager 440 may include a single root input/output virtualization (SR-IOV) sequence 505 and a state machine 510. The SR-IOV sequence 505 may be a sequence that would be implemented in SR-IOV software within the host 105 of FIG. 1 , and may be executed using the state machine 510. The SR-IOV sequence 505 may be stored in a read-only memory (ROM) within the configuration manager 440. Like the LWB 135 itself and its components shown in FIGS. 4A to 4C , the state machine 510 may be implemented using a general purpose processor, FPGA, ASIC, GPU, or GPGPU, among other possibilities.

FIG. 6 illustrates the mapping between the PFs disclosed by the LWB 135 of FIG. 1 and the PFs/VFs disclosed by the SSD 125 of FIG. 1 (or the SSDs 125-1 and 125-2 of FIG. 4C ). In FIG. 6 , PF 605 is the PF disclosed by the LWB 135 of FIG. 1 (more specifically, the PF disclosed by the endpoint 405 of FIG. 4A to FIG. 4C ). On the other hand, PF/VF 610 is the function (both physical and virtual) disclosed by the SSD 125 of FIG. 1 . Mapping 615 may then represent the mapping between PF 605 and PF/VF 610. The mapping shown in FIG. 6 is for illustrative purposes and is essentially arbitrary, however, the configuration manager 440 of FIG. 4A to FIG. 4C may use any desired mechanism to decide which PF in PF 605 maps to which PF/VF in PF/VF 610. The mapping may be implemented taking into account guidance and/or policies set by the host 105 of FIG. 1 (or a baseboard management controller (BMC) within the host 105 of FIG. 1 ) or the SSD 125 of FIG. 1 .

7 illustrates the LWB 135 of FIG. 1 processing a configuration write request from the host 105 of FIG. 7 , the LWB 135 of FIG. 1 (more specifically, the endpoint 405) may receive the configuration write request 705. The endpoint 405 may then process the configuration write request 705 locally to make any changes included in the configuration write request 705.

Note that the configuration write request 705 may not require changes to the SSD 125. For example, the QoS policy established by the host 105 of FIG. 1 may only affect the PF exposed by the endpoint 405 and may not require any changes within the SSD 125. However, in some cases, the configuration write request 705 may also require modifications to the SSD 125. For example, the configuration write request may change data about underlying functions on the SSD 125, rather than data that may be managed only by the LWB 135 of FIG. 1. In such a case, the endpoint 405 may propagate the configuration write request 705 to the SSD 125 via the root port 420 (as shown by the dashed arrows 710 and 715). The configuration write request 705 may be forwarded to the root port 420 via the APP-EP 410 of FIGS. 4A to 4C and the APP-RP 415 of FIGS. 4A to 4C, or by the configuration manager 440 of FIGS. 4A to 4C. Configuration table 435 of FIGS. 4A-4C may be used to determine whether configuration write request 705 should also be applied to SSD 125 (to ensure that the configurations of endpoint 405 and SSD 125 mirror each other).

4A-4C , the details of how endpoint 405 (and possibly SSD 125) is configured depend largely on the details of configuration write request 705. But once the details of configuration write request 705 are known, changes to endpoint 405 (and possibly SSD 125) can be performed unambiguously.

As described above, it is desirable that the configurations of endpoint 405 and SSD 125 mirror each other. However, as described above, there may be some configurations that apply only to endpoint 405 (or that do not matter to SSD 125 because endpoint 405 can handle those changes). In the case where configuration changes will not matter to SSD 125, those configuration changes do not need to be delivered to SSD 125.

In FIG. 7 , the configuration write request 705 is described as originating from the host 105 of FIG. 1 . However, embodiments of the inventive concept may support the SSD 125 requesting configuration changes in the endpoint 405 of FIGS. 4A to 4C . That is, the SSD 125 of FIG. 1 may dynamically request changes to the configuration, features, or capabilities of the PF exposed by the endpoint 405 of FIGS. 4A to 4C to the host 105 of FIG. 1 during runtime. For example, the SSD 125 of FIG. 1 may increase or decrease the number of interrupt vectors of the PF exposed by the endpoint 405 of FIGS. 4A to 4C . Such changes may be caused by changes in application requirements communicated by the host 105 of FIG. 1 to the SSD 125 of FIG. 1 using a higher-level storage communication protocol (such as Non-Volatile Memory Express (NVMe)) (via the LWB 125 of FIG. 1 ).

8 illustrates that LWB 135 of FIG. 1 processes a configuration read request from host 105 of FIG. 8 , endpoint 405 may receive configuration read request 805 from host 105 of FIG. Endpoint 405 may then read the requested configuration information and return it as configuration information 810 to host 105 of FIG. 1 .

As with the configuration write request 705 of FIG. 7 , the configuration read request 805 may be delivered to the SSD 125 to read from the SSD 125. Thus, the configuration read request 805 may be delivered to the root port 420 and the SSD 125 as configuration read requests 815 and 820, respectively, and the configuration information 810 is returned as configuration information 825 and 830. However, since the configuration of the endpoint 405 should mirror the configuration of the SSD 125, it may not be necessary to deliver the configuration read request 805 to the SSD 125 to determine the requested configuration information. Additionally, as with the configuration write request 705 of FIG. 7 , the configuration read request 805 may be delivered to the root port 420 via the APP-EP 410 of FIGS. 4A to 4C and the APP-RP 415 of FIGS. 4A to 4C , or by the configuration manager 440 of FIGS. 4A to 4C .

FIG. 9 illustrates that the application layer-endpoint (APP-EP) 410 of FIGS. 4A to 4C and the application layer-root port (APP-RP) 415 of FIGS. 4A to 4C handle address translation within the LWB 135. In FIG. 9, the address 905 may be an address received from the host 105 of FIG. 1 (e.g., an NVMe register that the host 105 of FIG. 1 requests to read). The APP-EP 410 may then subtract the host base address register (BAR) 910 for the PF called by the host 105 of FIG. 1. The APP-RP 415 may then add the SSD BAR 915 of the PF/VF to which the requested PF of the endpoint 405 of FIGS. 4A to 4C is mapped, thereby generating an SSD address 920. In this manner, the APP-EP 410 and the APP-RP 415 may translate addresses from the perspective of the host into addresses that can be processed by the SSD 125 of FIG. 1.

Note that this process can be reversed to convert the SSD address back to the host's perspective. Thus, APP-RP 415 can subtract address 915 in the SSD BAR from SSD address 920, and APP-EP 410 can add address 910 in the host BAR, thereby generating host address 905.

FIG. 10 shows the mapping of FIG. 6 being changed within the LWB 135 of FIG. 1 . There are a variety of reasons why the mapping 615 may be changed. For example, a storage parameter 1005 may trigger a change to the mapping 615: for example, if the available capacity on a particular SSD drops below a threshold amount of free space, new write requests may be directed to a different SSD (such as, SSD 125-2 of FIG. 4C ) connected to the LWB 135 of FIG. 1 . Alternatively, a day 1010 or a time of day 1015 may trigger a change to the mapping 615: for example, the peak times for a particular request at a particular host 105 may be between 6:30am and 8:00am and between 5:00pm and 9:00pm, so the bandwidth associated with the PF handling such requests may be increased during those times and decreased at other times. Even if not specified using conventional or known day/time values, the day 1010 and the time of day 1015 may be generalized to any trigger related to day and/or time. For example, at the end of a sporting event, it may be anticipated that fans returning home may request directions from a GPS application on their smartphones, even though the specific time at which such a request will begin may not be known (the duration of a sporting event is roughly known, but may run shorter or longer depending on the event itself). Bandwidth usage 1020 itself may also trigger changes to mapping 615: for example, to balance the load on the SSDs connected to LWB 135 of FIG. 1. Finally, changes 1025 in QoS policies may trigger changes to mapping 615: for example, the addition of a new QoS policy may require a change in how functions are mapped. Embodiments of the inventive concept may also include other triggers for changes to mapping 615. Regardless of the trigger, as a result of the change, mapping 615 may be replaced by mapping 1030, which still maps the PFs exposed by endpoint 405 of FIGS. 4A to 4C to the PFs/VFs exposed by SSD 125 of FIG. 1, but may be in a different arrangement.

FIG. 11 illustrates that the PFs disclosed by the LWB 135 of FIG. 1 have associated quality of service (QoS) policies. In FIG. 11 , one PF "PF1" is illustrated (as disclosed by the endpoint 405 of FIGS. 4A to 4C), with a single QoS policy 1105 associated with the disclosed PF. However, embodiments of the inventive concept may include disclosed PFs with any number of associated QoS policies. For example, some PFs disclosed by the endpoint 405 of FIGS. 4A to 4C may have no associated QoS policies; other PFs disclosed by the endpoint 405 of FIGS. 4A to 4C may have two or more associated QoS policies.

Given a QoS policy 1105 associated with a PF disclosed by endpoint 405 of FIGS. 4A to 4C , the LWB 135 of FIG. 1 may implement the QoS policy in accordance with its specification. Furthermore, because specifications may vary widely, the details of how the LWB 135 of FIG. 1 implements a particular QoS policy may not be set forth herein. However, as an example, the QoS policy 1105 may specify a maximum (and/or minimum) bandwidth for a particular PF (thereby ensuring that a particular PF does not prevent other PFs from receiving their fair share of bandwidth or ensuring that a particular PF receives its specified share of bandwidth). The LWB 135 of FIG. 1 may then ensure that the QoS policy 1105 is satisfied by preventing communications using the corresponding PF/VF on the SSD 125 of FIG. 1 from exceeding the maximum bandwidth, or by ensuring that communications using PF/VFs other than the PF/VF corresponding to the PF on the SSD 125 of FIG. 1 guarantee the minimum bandwidth of the disclosed PF.

Although the LWB 135 of FIG. 1 may implement the QoS policy 1105, it is up to the host 105 of FIG. 1 (or the SSD 125 of FIG. 1 if the SSD 125 of FIG. 1 requests that the policy be applied) to ensure that the policy is appropriate. For example, assume that the SSD 125 of FIG. 1 has a maximum bandwidth of 100 GB/sec and supports a total of 16 functions (PFs and VFs). If the host allocates a QoS policy that guarantees a minimum bandwidth of 10 GB/sec to each PF exposed by the LWB 135, the total allocated bandwidth will be 160 GB/sec, which exceeds the capabilities of the SSD 125 of FIG. 1. As a result, some PFs will not satisfy their associated QoS policies. Therefore, although the LWB 135 of FIG. 1 may implement the QoS policy 1105, the LWB 105 may not check that the QoS policy 1105 in combination with other QoS policies assigned to various PFs can all be implemented simultaneously.

Thus, there is a difference between enforcing a QoS policy and ensuring that a QoS policy is always enforceable. The LWB 135 of FIG. 1 may enforce the former; it may not enforce the latter. However, it is important to note that just because a set of QoS policies may not be enforceable simultaneously, this fact does not mean that any individual QoS policy will not be satisfied in the future. For example, consider again that the SSD 125 of FIG. 1 provides a total of 10 GB/sec of bandwidth, and assume that the host 105 of FIG. 1 specifies QoS policies for two different PFs exposed by the endpoints 405 of FIGS. 4A through 4C , guaranteeing a minimum bandwidth of 6 Gb/sec to each PF. Overall, since each PF is "guaranteed" 6 GB/sec of the available 10 GB/sec bandwidth of the SSD 125 of FIG. 1 , both QoS policies may not be satisfied. But if it turns out that invoking both functions simultaneously is logically (or physically) impossible (for example, if one function involves reading data from or writing data to SSD 125 of FIG. 1 , and the other function involves SSD 125 of FIG. 1 performing an internal consistency check that prevents SSD 125 of FIG. 1 from responding to any other function until the consistency check is complete), then host 105 of FIG. 1 may specify two QoS policies: even if they cumulatively exceed the bandwidth provided by SSD 125 of FIG. 1 , the two policies will not be implemented simultaneously, so that there is only a logical conflict, not an actual (or even potential) conflict.

As described above, sometimes the LWB 135 of FIG. 1 may need to perform bandwidth throttling: for example, to prevent a particular PF exposed by the endpoint 405 of FIG. 4A to FIG. 4C from using too much bandwidth, causing other functions of the SSD 125 of FIG. 1 to be overly limited. FIG. 12A to FIG. 12B illustrate the LWB 135 of FIG. 1 performing bandwidth throttling. In FIG. 12A , the LWB 135 is shown as receiving a "high" or "large" bandwidth 1205 from the host 105 of FIG. 1 . If this bandwidth is too large for any reason, the LWB 135 may throttle the bandwidth to the SSD 125 of FIG. 1 , as shown by the "low" or "small" bandwidth 1210. In FIG. 12B , the opposite situation is shown: the LWB 135 may have a "high" or "large" bandwidth 1215 with the SSD 125 of FIG. 1 , but may throttle the bandwidth 1220 of the host 105 of FIG. 1 . Therefore, when the available bandwidth of the SSD 125 of FIG. 1 is higher than the maximum bandwidth of the SLA set for the PF, the LWB 135 may throttle the bandwidth.

Embodiments of the inventive concept may perform bandwidth throttling for any number of reasons. In addition to the QoS policy 1105 of FIG. 11 (if the bandwidth for a particular PF exposed by the endpoint 405 of FIG. 4A to FIG. 4C is too large, the QoS policy 1105 may throttle the bandwidth associated with the PF), other reasons for throttling bandwidth may include temperature or power consumption. For example, if the temperature of the LWB 135 (or the SSD 125 of FIG. 1 ) becomes too high (i.e., exceeds a certain threshold), the LWB 135 may throttle the bandwidth to reduce the temperature. Then, once the temperature has dropped sufficiently (which may mean dropping below the original threshold or dropping below another threshold that may be lower than the original threshold), the LWB 135 may stop throttling the bandwidth. Power consumption may trigger bandwidth throttling in the same manner as temperature, except that the power consumption of the LWB 135 (or the SSD 125 of FIG. 1 ) may trigger bandwidth throttling, and a reduced power consumption threshold may release bandwidth throttling. These thresholds may be stored somewhere within the LWB 135. Yet another reason bandwidth may be throttled may be due to priorities established for PFs exposed by endpoint 405 of FIGS. 4A-4C : when different PFs are used, bandwidth on a PF with a lower priority may be throttled in favor of bandwidth for a PF with a higher priority.

Another reason to throttle bandwidth may be to manage QoSs or SLAs. For example, the host 105 of FIG. 1 may only pay for bandwidth at a lower level than the bandwidth that the LWB 135 is able to provide. Therefore, even though the host 105 of FIG. 1 and the LWB 135 could use a higher overall bandwidth for communication, the LWB 135 would limit the bandwidth of the host 105 of FIG. 1 so that the service provided is no greater than the service that has been guaranteed to the host 105 of FIG. 1. In other words, the LWB 135 may limit or restrict the maximum storage bandwidth of the PF based on an SLA or pricing plan.

Note that while Figures 12A-12B specifically discuss bandwidth, other resources may be similarly monitored and throttled for communications with the host 105 of Figure 1 or the SSD 125 of Figure 1. For example, latency may be controlled to some extent, favoring PFs with lower target latencies over PFs with higher target latencies.

LWB 135 can measure and monitor the bandwidth (or other resource) consumed by each PF individually exposed by endpoint 405 of Figures 4A to 4C. Bandwidth can be measured and monitored in both directions: namely, host to device (host write operations) and device to host (host read operations). Throttling bandwidth can also be performed independently in either direction. Bandwidth (or other resource) throttling can be performed as a result of both measured bandwidth and QoS policy settings.

FIG. 13 shows the LWB 135 of FIG. 1 issuing credits to the SSD 125 of FIG. 1 . Credits are one way that the LWB 135 can manage how much bandwidth the SSD 125 uses. Given a particular request, the LWB 135 can issue a certain number of credits 1305, each credit 1305 representing a certain amount of data that the SSD 125 can transfer. Thus, the number of credits 1305 issued by the LWB 135 can control the bandwidth of the SSD 125: the LWB 135 can issue just enough (or almost just enough, or any other desired number) of credits 1305 to cover a data transfer. When the SSD 125 transfers data, that data transfer uses credits 1305: if the SSD 125 does not have any available credits, then the SSD 125 cannot transfer any data until new credits are issued.

Credits 1305 may be issued in any desired manner. In some embodiments of the inventive concept, LWB 135 may send credits 1305 to SSD 125 via a message (which may be, for example, a proprietary message). In other embodiments of the inventive concept, LWB 135 may write credits 1305 to a specific address on SSD 125: for example, a reserved address in the NVMe address space. In other embodiments of the inventive concept, LWB 135 may write credits 1305 to an address within LWB 135 (again, possibly a reserved address): SSD 125 may then read the address to see what credits 1305 are available. The use of credits 1305 is a way to throttle the bandwidth of SSD 125. By limiting the number of credits 1305 issued to SSD 125, SSD 125 may be prevented from downloading all the data required for a particular transaction within a given unit of time. By reducing the bandwidth on SSD 125, SSD 125 may experience reduced power consumption and/or lower temperatures. Eventually, if power consumption and/or temperature drops to acceptable levels (which may be different than levels at which bandwidth throttling may begin, as discussed above with reference to FIG. 12 ), LWB 135 may stop throttling bandwidth to SSD 125 .

In another embodiment, LWB 135 may insert idle periods between data transmission packets to implement bandwidth throttling. By adjusting the inter-packet gap of data packets based on measured bandwidth (or other resources) and QoS policy settings, LWB 135 may implement desired bandwidth limits for individual PFs or all PFs in general.

14A-14B illustrate a flowchart of an example process of the LWB 135 of FIG. 1 identifying PF/VFs exposed by the SSD 125 of FIG. 1, exposing PFs from the LWB 135, and generating a mapping between the PFs exposed by the LWB 135 of FIG. 1 and the PF/VFs exposed by the SSD 125 of FIG. 1 according to an embodiment of the inventive concept. In FIG. 14A , at block 1405, the LWB 135 of FIG. 1 may enumerate the PFs and VFs exposed by the SSD 125 of FIG. 1. At block 1410, the LWB 135 of FIG. 1 may generate a PF to be exposed by the LWB 135 of FIG. 1 (more specifically, a PF to be exposed by the endpoint 405 of FIGS. 4A-4C ). Note that the number of PFs generated by the LWB 135 of FIG. 1 may not necessarily equal the number of PFs and VFs exposed by the SSD 125 of FIG. 1 : certain functions exposed by the SSD 125 of FIG. 1 may not obtain corresponding PFs exposed by the LWB 135 of FIG. 1 , and a single PF exposed by the LWB 135 of FIG. 1 may map to multiple PFs/VFs of the SSD 125 of FIG. 1 . At block 1415, the LWB 135 of FIG. 1 may check to see if there are any other connected devices to enumerate (such as, SSD 125-2 of FIG. 4C ). If so, the process may return to block 1405 to enumerate the PFs and VFs of the next device.

Assuming that all connected devices have been enumerated, at block 1420 (FIG. 14B), the LWB 135 of FIG. 1 may map the PFs exposed by the endpoints 405 of FIG. 4A-4C to the PFs and VFs exposed by the SSD 125 of FIG. 1 (and other connected devices). At block 1425, the LWB 135 of FIG. 1 may determine other resources (such as bandwidth) of the SSD 125 of FIG. 1 and any other connected devices (this determination may also be made as part of block 1405 of FIG. 14A), and at block 1430, the LWB 135 of FIG. 1 may aggregate the resources of all connected devices. In this manner, the LWB 135 of FIG. 1 may appear to provide higher overall resources than any of the individual devices.

At block 1435, the LWB 135 of Figure 1 may receive an enumeration request from the host 105 of Figure 1. At block 1440, the LWB 135 of Figure 1 may respond to the enumeration request of the host 105 of Figure 1 by exposing the PF of the endpoint 405 of Figures 4A-4C.

15A-15B illustrate a flow chart of an example process by which the LWB 135 of FIG. 1 receives and processes a request from the host 105 of FIG. 1. In FIG. 15A, at block 1505, the LWB 135 of FIG. 1 may receive a request for a certain PF exposed by the endpoint 405 of FIG. 4A-4C from the host 135 of FIG. 1: which PF the request is directed to and what function the PF represents are irrelevant to the analysis. At block 1510, the LWB 135 of FIG. 1 may map the PF to which the request is directed to a PF or VF of the SSD 125 of FIG. 1 (the mapping may be performed using, for example, the configuration table 435 of FIG. 4A-4C). At block 1515, the LWB 135 may select the SSD 125 of FIG. 1 as the device that includes the PF or VF to which the PF exposed by the endpoint 405 of FIG. 4A-4C is mapped. Note that, as shown in FIG4C, block 1515 may only be important if the LWB 135 of FIG1 is connected to multiple devices: if multiple devices may exist, multiple APP-EPs 410-1 and 410-2 of FIG4C may receive the request. In embodiments of the inventive concept where only one device may be connected to the LWB 135 of FIG1 (because the LWB 135 includes only one APP-EP 410 of FIGS. 4A-4B, or because only one SSD 125 of FIG1 is connected to the LWB 135 of FIG1), block 1515 may be omitted, as shown by dashed line 1520.

Once the appropriate device and the appropriate destination PF/VF have been identified, the LWB 135 of FIG. 1 may translate the request into a request for the identified PF/VF at block 1525. This may involve, for example, the address translation described above with reference to FIG. 9 (and described below with reference to FIG. 16). At block 1530 (FIG. 15B), the LWB 135 of FIG. 1 may send the translated request to the identified PF/VF on the identified device.

At block 1535, the LWB 135 of FIG. 1 may receive a response from the identified PF/VF of the identified device. At block 1540, the LWB 135 of FIG. 1 may map the PF/VF of the identified device to the PF exposed by the endpoint 405 of FIGS. 4A to 4C (the mapping may be performed using, for example, the configuration table 435 of FIGS. 4A to 4C). At block 1545, the LWB 135 of FIG. 1 may convert the response into a response to the host 135 of FIG. 1. Finally, at block 1550, the LWB 135 of FIG. 1 may send the response back to the host as the exposed PF from the endpoint 405 of FIGS. 4A to 4C.

FIG16 illustrates a flow chart of an example process of translating addresses between the host 105 of FIG1 and the SSD 125 of FIG1 by the APP-EP and APP-RP of FIG4A to FIG4C. In FIG16, at block 1605, the APP-EP 410 of FIG4A to FIG4C may subtract an address in the host BAR from an address included in a request from the host 105 of FIG1, and at block 1610, the APP-RP 415 of FIG4A to FIG4C may add the address in the device BAR to the address determined in block 1605.

Note that the reverse address translation from the SSD 125 of FIG. 1 to the host 105 of FIG. 1 may be omitted. The SSD 125 of FIG. 1 (like any such device) may receive a complete address for use by the host 105 of FIG. 1 . Thus, the address provided by the SSD 125 of FIG. 1 may be used to access a particular address within the host 105 of FIG. 1 without translation. However, if the SSD 125 of FIG. 1 does not have the complete address of the host 105 of FIG. 1 , the address translation may be performed in reverse, with the APP-RP 415 of FIGS. 4A to 4C subtracting the address in the device BAR from the address included in the request from the SSD 125 of FIG. 1 , and the APP-EP 410 of FIGS. 4A to 4C adding the address in the host BAR to the address.

FIG. 17 illustrates a flow chart of an example process for the LWB 135 of FIG. 1 to issue the credits 1305 of FIG. 13 to the SSD 125 of FIG. 1. In FIG. 17, at block 1705, the LWB 135 of FIG. 1 may measure or monitor the bandwidth consumed by the various PFs exposed by the endpoints 405 of FIG. 4A to FIG. 4C. At block 1710, the LWB 135 of FIG. 1 may determine the credits 1305 of FIG. 13 to issue to the SSD 125 of FIG. 1. These credits may be determined based on the bandwidth consumed by the PFs exposed by the endpoints 405 of FIG. 4A to FIG. 4C and the amount of data that the SSD 125 of FIG. 1 may need to transfer. The SSD 125 of FIG. 1 may use these credits to transfer data to or from the SSD 125 of FIG. 1. At this point, there are multiple options for delivering the credits 1305 of FIG. 13 to the SSD 125 of FIG. 1. At block 1715, the LWB 135 of FIG. 1 may write the credit 1305 of FIG. 13 to an address in the SSD 125 of FIG. 1. Optionally, at block 1720, the LWB 135 may write the credit 1305 of FIG. 13 to an address in the LWB 135 of FIG. 1, and the SSD 125 of FIG. 1 may read the credit 1305 of FIG. 13 from the address in the LWB 135 of FIG. 1. Optionally, at block 1725, the LWB 135 of FIG. 1 may send a message to the SSD 125 of FIG. 1, which may include the credit 1305 of FIG. 13.

FIG18 shows a flow chart of an example process for the LWB 135 of FIG1 to process the configuration write request 705 of FIG7. In FIG18, at block 1805, the LWB 135 of FIG1 may receive the configuration write request 705 of FIG7: the configuration write request 705 of FIG7 may be received from the host 105 of FIG1 or the SSD 125 of FIG1. At block 1810, the LWB 135 of FIG1 (more specifically, the configuration manager 440 of FIGS. 4A to 4C) may configure the PF exposed by the endpoint 405 of FIGS. 4A to 4C as specified by the configuration write request 705 of FIG7. At block 1815, the LWB 135 of FIG1 may perform the configuration using the state machine 510 of FIG5 and the SR-IOV sequence 505 of FIG5. Finally, at block 1820, LWB 135 of FIG. 1 may ensure that the configuration of the PF/VF exposed by SSD 125 of FIG. 1 mirrors the configuration of the PF exposed by endpoint 405 of FIGS. 4A-4C : as indicated by dashed arrows 710 and 715, this may involve sending configuration write request 705 of FIG. 7 to SSD 125 of FIG. 1. Since configuration write request 705 of FIG. 7 may not need to be sent to SSD 125 of FIG. 1, block 1820 is optional, as indicated by dashed line 1825.

FIG. 19 illustrates a flow chart of an example process by which the LWB 135 of FIG. 1 processes the configuration read request 805 of FIG. 8 . In FIG. 19 , at block 1905, the LWB 135 of FIG. 1 may receive the configuration read request 805 of FIG. 8 : the configuration read request 805 of FIG. 8 may be received from the host 105 of FIG. 1 or the SSD 125 of FIG. 1 (although the configuration read request 805 of FIG. 8 may generally be received from the host 105 of FIG. 1 ). At this point, there are some alternatives. At block 1910, the LWB 135 of FIG. 1 (more specifically, the configuration manager 440 of FIGS. 4A to 4C ) may determine the configuration of the PF exposed by the endpoint 405 of FIGS. 4A to 4C . In an embodiment of the inventive concept using block 1910, since the configuration of the PF disclosed by the SSD 125 of FIG. 1 should mirror the configuration of the PF disclosed by the endpoint 405 of FIGS. 4A to 4C, it may not be necessary to query the SSD 125 of FIG. 1 for the configuration of the PF/VF it discloses (although it may be possible to query the SSD 125 of FIG. 1 for the configuration of the PF/VF it discloses). Optionally, at block 1915, the LWB 135 of FIG. 1 may deliver the configuration read request 805 of FIG. 8 to the SSD 125 of FIG. 1 (as indicated by the dashed arrows 815 and 820 of FIG. 8), and may receive the configuration information 810 of FIG. 8 in return (as indicated by the dashed arrows 825 and 830 of FIG. 8). In either case, at block 1920, the LWB 135 of FIG. 1 may return the configuration information 810 of FIG. 8 to the requester.

FIG20 illustrates a flow chart of an example process by which the LWB 135 of FIG1 associates the QoS policy 1105 of FIG11 with a PF exposed by the LWB 135 of FIG1. In FIG20, at block 2005, the LWB 135 of FIG1 may receive the QoS policy 1105 of FIG11 from a source, which may be the host 105 of FIG1 or the SSD 125 of FIG1. Then, at block 2010, the LWB 135 of FIG1 may associate the policy with the identified PF exposed by the endpoint 405 of FIGS. 4A to 4C and implement the policy appropriately.

FIG21 illustrates a flow chart of an example process by which the LWB 135 of FIG1 dynamically changes a mapping of a PF exposed by the LWB 135 of FIG1 to a PF/VF exposed by the SSD 125 of FIG1. In FIG21, at block 2105, the LWB 135 of FIG1 may identify that the storage parameter 1005 of FIG10, the day 1010 of FIG10, the time of day 1015 of FIG10, the bandwidth usage 1020 of FIG10, or the QoS policy change 1025 of FIG10 has changed. At block 2110, the LWB 135 of FIG1 may dynamically change the mapping 615 of FIG6 between the PF 605 of FIG6 and the PF/VFS 610 of FIG6 in response to the detected change.

22A-22B illustrate a flow chart of an example process of performing bandwidth throttling by the LWB 135 of FIG. 22A , at block 2205, the LWB 135 of FIG. 1 may determine the QoS policy 1105 of FIG. 11 , or the temperature or power consumption of the LWB 135 of FIG. 1 or the SSD 125 of FIG. 1 . At block 2210, the LWB 135 of FIG. 1 may determine whether bandwidth throttling is applicable: for example, by comparing the temperature or power consumption of the LWB 135 of FIG. 1 or the SSD 125 of FIG. 1 with a threshold value stored in the LWB 135 of FIG. 1 .

If bandwidth throttling is applicable, then at block 2215, the LWB 135 of FIG. 1 may throttle the bandwidth of the SSD 125 of FIG. 1. The throttling may be achieved by limiting the number of credits 1305 of FIG. 13 issued to the SSD 125 of FIG. 1. Then, at block 2220 (FIG. 22B), the LWB 135 of FIG. 1 may determine the QoS policy 1105 of FIG. 11 (which may be the same QoS policy 1105 of FIG. 11 that triggered bandwidth throttling in block 2210 of FIG. 22A or a different QoS policy) or the temperature or power consumption of the LWB 135 of FIG. 1 or the SSD 125 of FIG. 1. At block 2225, the LWB 135 of FIG. 1 may determine whether the QoS policy 1105 of FIG. 11 or the new temperature or power consumption of the LWB 135 of FIG. 1 or the SSD 125 of FIG. 1 has reached a level where bandwidth throttling is no longer necessary. If bandwidth throttling is still applicable, control may return to block 2220 to check again whether a change has occurred that may end bandwidth throttling. Otherwise, at block 2230, the LWB 135 of FIG. 1 may stop bandwidth throttling.

In Figures 14A to 22B, some embodiments of the invention are shown. However, those skilled in the art will recognize that other embodiments of the invention are possible by changing the order of the blocks, by omitting blocks, or by including connections not shown in the drawings. All such variations of the flow charts are considered embodiments of the invention, whether explicitly described or not.

Embodiments of the inventive concept include technical advantages over conventional implementations. A lightweight bridge (LWB) can expose physical functions (PFs) to a host machine, which eliminates the need for the host machine to execute single root input/output virtualization (SR-IOV) software to access virtual functions (VFs) of a device (such as a solid state drive (SSD)). Since the LWB can be a separate device and the device does not require specific hardware to support multiple PFs, the device does not necessarily require any hardware modifications. In fact, even if a single housing contains both the LWB and the device, and even if the LWB and the device are implemented on a shared printed circuit board, the inclusion of the LWB may not require any hardware changes to the device, thereby allowing existing SSDs (and other devices) to be used with the LWB, and allowing future SSDs (and other devices) to continue to provide VFs rather than including hardware to support multiple PFs. (Devices may require some firmware updates—for example, to support the use of credits to manage data transfers—but such updates can be easily performed even in the field for existing devices.) In addition, because LWB can aggregate the resources of multiple devices, LWB can provide better overall performance than any individual device could provide.

The following discussion is intended to provide a brief, general description of one or more suitable machines in which certain aspects of the invention may be implemented. One or more machines may be controlled at least in part by input from conventional input devices (such as keyboards, mice, etc.) and by instructions received from another machine, interaction with a virtual reality (VR) environment, biometric feedback, or other input signals. As used herein, the term "machine" is intended to broadly include a single machine, a virtual machine, or a system of machines, virtual machines, or devices that are communicatively connected and operate together. Exemplary machines include computing devices (such as personal computers, workstations, servers, portable computers, handheld devices, phones, tablet computers, etc.) and transportation devices (such as private or public transportation (e.g., cars, trains, taxis, etc.)).

One or more machines may include embedded controllers (such as programmable or non-programmable logic devices or arrays, application specific integrated circuits (ASICs), embedded computers, smart cards, etc.). One or more machines may utilize one or more connections to one or more remote machines (such as through a network interface, modem, or other communication connection). The machines may be interconnected by way of a physical network and/or a logical network (such as an intranet, the Internet, a local area network, a wide area network, etc.). Those skilled in the art will appreciate that network communications may utilize a variety of wired and/or wireless short-range or long-range carriers and protocols, including: radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) 802.11, Bluetooth, optical, infrared, cable, laser, etc.

Embodiments of the present invention may be described by reference to or in conjunction with associated data, including functions, programs, data structures, applications, etc., which, when accessed by a machine, cause the machine to perform a task or define an abstract data type or a low-level hardware context. The associated data may be stored, for example, in volatile memory and/or non-volatile memory (e.g., RAM, ROM, etc.), or in other storage devices and their associated storage media (including hard disk drives, floppy disks, optical storage devices, magnetic tapes, flash memory, memory sticks, digital video disks, bio-memory, etc.). The associated data may be transmitted in a transmission environment (including a physical network and/or a logical network) in the form of packets, serial data, parallel data, propagation signals, etc., and may be used in a compressed format or an encrypted format. The associated data may be used in a distributed environment and stored locally and/or remotely for machine access.

Embodiments of the invention may include tangible, non-transitory machine-readable media that may include instructions executable by one or more processors, including instructions for performing elements of the invention as described herein.

The principles of the invention have been described and illustrated with reference to the illustrated embodiments, it will be appreciated that the illustrated embodiments may be modified in arrangement and detail without departing from such principles, and may be combined in any desired manner. Also, although the foregoing discussion has focused on specific embodiments, other configurations (constructions) are contemplated. Specifically, although expressions such as "embodiments according to the inventive concept" are used herein, these phrases are intended to refer to embodiment possibilities in general, and are not intended to limit the invention to specific embodiment configurations. As used herein, these terms may relate to the same or different embodiments that may be combined into other embodiments.

The foregoing illustrative embodiments should not be construed as limiting the invention. Although some embodiments have been described, it will be readily appreciated by those skilled in the art that many modifications are feasible for those embodiments without substantially departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims.

Embodiments of the invention extend to the following statements without limitation:

Statement 1. Embodiments of the inventive concept include a light weight bridge (LWB) circuit, the light weight bridge (LWB) circuit comprising:

Endpoints, used to connect to a host, the endpoints expose multiple physical functions (PF);

a root port for connecting to a device, the device exposing at least one PF and at least one virtual function (VF) to the root port; and

an application layer-end point (APP-EP) and an application layer-root port (APP-RP) for translating between the plurality of PFs exposed to the host and the at least one PF and the at least one VF exposed by the device,

The APP-EP and the APP-RP implement mapping between the multiple PFs disclosed by the endpoint and the at least one PF and the at least one VF disclosed by the device.

Statement 2. Embodiments of the inventive concept include the LWB circuit of statement 1, wherein the device comprises a solid state drive (SSD).

Statement 3. Embodiments of the inventive concept include the LWB circuit of statement 1, wherein an apparatus includes the LWB circuit.

Statement 4. An embodiment of the inventive concept comprises the LWB circuit of statement 1, wherein the LWB circuit is connected to a device.

Statement 5. Embodiments of the inventive concept include the LWB circuit of statement 1, wherein the APP-EP and the APP-RP are implemented as a single component.

Statement 6. An embodiment of the inventive concept includes the LWB circuit of statement 1, wherein the LWB circuit includes at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a general purpose processor, a graphics processing unit (GPU), and a general purpose GPU (GPGPU).

Statement 7. An embodiment of the inventive concept includes the LWB circuit of statement 1, wherein:

The plurality of PFs include a plurality of Peripheral Component Interconnect Express (PCIe) PFs;

The at least one PF comprises at least one PCIe PF; and

The at least one virtual function (VF) includes at least one PCIe VF.

Statement 8. An embodiment of the inventive concept includes the LWB circuit of statement 7, wherein:

APP-EP includes PCIe APP-EP (PAPP-EP); and

APP-RP includes PCIe APP-RP (PAPP-RP).

Statement 9. An embodiment of the inventive concept includes the LWB circuit of statement 1, further comprising: a configuration manager for configuring the APP-EP and the APP-RP.

Statement 10. An embodiment of the inventive concept includes the LWB circuit of statement 9, wherein the APP-RP may include a configuration table for storing a mapping between the plurality of PFs exposed by an endpoint and the at least one PF and the at least one VF exposed by a device.

Statement 11. An embodiment of the inventive concept includes the LWB circuit of statement 9, wherein the configuration manager may enumerate the at least one PF and the at least one VF disclosed by the device, and generate the plurality of PFs disclosed by the endpoint based at least in part on the at least one PF and the at least one VF disclosed by the device.

Statement 12. An embodiment of the inventive concept comprises the LWB circuit of statement 9, wherein the configuration manager can configure the device.

Statement 13. An embodiment of the inventive concept includes the LWB circuit of statement 12, wherein the configuration manager can configure the device based at least in part on a configuration write request receivable at the endpoint from the host.

Statement 14. An embodiment of the inventive concept comprises the LWB circuit of statement 9, wherein the APP-EP can intercept a configuration read request from the host and return configuration information to the host.

Statement 15. An embodiment of the inventive concept comprises the LWB circuit of statement 9, wherein the configuration manager is operable to ensure that a first configuration of the device mirrors a second configuration of the endpoint.

Statement 16. An embodiment of the inventive concept includes the LWB circuit of statement 9, wherein the configuration manager includes: a read only memory (ROM) and a state machine for implementing a single root input/output virtualization (SR-IOV) sequence.

Statement 17. An embodiment of the inventive concept includes the LWB circuit of statement 1, wherein:

The APP-EP may subtract the base address register (BAR) of the PF exposed by the endpoint from the address received in the request from the host; and

The APP-RP may add the BAR of the VF published by the device to the address received from the APP-EP.

Statement 18. An embodiment of the inventive concept includes the LWB circuit of statement 1, the LWB circuit further comprising:

a second APP-EP and a second APP-RP;

A second device, disclosing at least one second PF and at least one second VF; and

a multiplexer/demultiplexer arranged to be connected to the endpoint and to each of the APP-EP and the second APP-EP,

The second APP-EP and the second APP-RP implement a second mapping between the multiple PFs disclosed by the endpoint and the at least one second PF and the at least one second VF disclosed by the second device.

Statement 19. An embodiment of the inventive concept comprises the LWB circuit of statement 18, wherein the LWB circuit can provide aggregated resources of the device and a second device.

Statement 20. An embodiment of the inventive concept includes the LWB circuit of statement 1, wherein the mapping between the plurality of PFs exposed by an endpoint and the at least one PF and the at least one VF exposed by a device can be dynamically changed.

Statement 21. An embodiment of the inventive concept includes an LWB circuit according to statement 20, wherein the mapping between the multiple PFs disclosed by the endpoint and the at least one PF and the at least one VF disclosed by the device can be changed at least in part based on a change in a quality of service (QoS) policy for at least one of the multiple PFs disclosed by the endpoint, a storage parameter, a day, a time of day, or bandwidth usage.

Statement 22. An embodiment of the inventive concept comprises the LWB circuit of statement 1, wherein the LWB circuit can implement bandwidth throttling.

Statement 23. An embodiment of the inventive concept includes the LWB circuit of statement 22, wherein the LWB circuit is operable to measure a bandwidth used by at least one of the plurality of PFs exposed by an endpoint.

Statement 24. An embodiment of the inventive concept includes a LWB circuit as described in statement 22, wherein the LWB circuit can implement bandwidth throttling based at least in part on a policy set by a host, a policy set by a device, a temperature of the LWB circuit, power consumption of the LWB circuit, a temperature of an SSD, or power consumption of an SSD.

Statement 25. An embodiment of the inventive concept includes the LWB circuit of statement 24, wherein the LWB circuit further comprises: a memory for a threshold value for triggering bandwidth throttling.

Statement 26. An embodiment of the inventive concept includes the LWB circuit of statement 25, wherein the LWB circuit further comprises: a memory for a second threshold for triggering an end of bandwidth throttling.

Statement 27. An embodiment of the inventive concept comprises the LWB circuit of statement 22, wherein the LWB circuit can issue credits to a device for data transfer.

Statement 28. An embodiment of the inventive concept comprises the LWB circuit of statement 27, wherein the LWB circuit can write a credit to an address in a device.

Statement 29. An embodiment of the inventive concept comprises the LWB circuit of statement 27, wherein a device can read a credit from an address in the LWB circuit.

Statement 30. An embodiment of the inventive concept comprises the LWB circuit of statement 27, wherein the LWB circuit is operable to send the credit to the device in a message.

Statement 31. An embodiment of the inventive concept comprises the LWB circuit of statement 22, wherein the LWB circuit can implement bandwidth throttling by introducing an inter-packet gap between a first data packet and a second data packet.

Statement 32. An embodiment of the inventive concept comprises the LWB circuit of statement 1, wherein the LWB circuit is operable to enforce a QoS policy on a PF of the plurality of PFs exposed by a device.

Statement 33. An embodiment of the inventive concept comprises the LWB circuit of statement 32, wherein the policy is settable by one of the host, the BMC, and the device.

Statement 34. An embodiment of the inventive concept includes the LWB circuit of statement 1, wherein the LWB circuit is configurable, based at least in part on a configuration write request received from a device, of capabilities of at least one of the plurality of PFs exposed by an endpoint.

Statement 35. An embodiment of the inventive concept includes a method comprising:

enumerating at least one physical function (PF) exposed by the device using a root port of a lightweight bridge (LWB);

enumerating at least one virtual function (VF) exposed by the device using a root port of the LWB;

generating a plurality of PFs at the endpoints of the LWB to be exposed to the host; and

The plurality of PFs at an endpoint of the LWB are mapped to the at least one PF and the at least one VF exposed by the device using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB.

Statement 36. An embodiment of the inventive concept comprises the method according to statement 35, wherein:

The step of using a root port of a lightweight bridge (LWB) to enumerate at least one physical function (PF) exposed by the device comprises: using the root port of the LWB to enumerate at least one PF exposed by the device when the LWB is booted; and

The step of using a root port of the LWB to enumerate at least one virtual function (VF) exposed by the device includes using a root port of the LWB to enumerate at least one VF exposed by the device when the LWB is booted.

Statement 37. An embodiment of the inventive concept comprises the method according to statement 35, wherein:

The step of using a root port of a lightweight bridge (LWB) to enumerate at least one physical function (PF) exposed by the device comprises: when the device is connected to the LWB, using the root port of the LWB to enumerate the at least one PF exposed by the device; and

The step of enumerating at least one virtual function (VF) exposed by the device using a root port of the LWB includes enumerating at least one VF exposed by the device using a root port of the LWB when the device is connected to the LWB.

Statement 38. Embodiments of the inventive concept include the method of statement 35, wherein the device comprises a solid state drive (SSD).

Statement 39. An embodiment of the inventive concept comprises the method according to statement 35, wherein:

The plurality of PFs include a plurality of Peripheral Component Interconnect (PCIe) Express PFs;

The at least one physical function (PF) includes at least one PCIe PF; and

The at least one virtual function (VF) includes at least one PCIe VF.

Statement 40. An embodiment of the inventive concept comprises the method according to statement 39, wherein:

APP-EP includes PCIe APP-EP (PAPP-EP); and

APP-RP includes PCIe APP-RP (PAPP-RP).

Statement 41. An embodiment of the inventive concept comprises the method according to statement 35, wherein the method further comprises:

Receive enumeration from the host; and

The plurality of PFs at the endpoints of the LWB are exposed to the host.

Statement 42. An embodiment of the inventive concept comprises the method according to statement 35, wherein:

The step of using a root port of a lightweight bridge (LWB) to enumerate at least one physical function (PF) exposed by a device includes: using a second root port of the LWB to enumerate at least one second PF exposed by a second device;

The step of using a root port of the LWB to enumerate at least one virtual function (VF) exposed by the device includes: using a second root port of the LWB to enumerate at least one second VF exposed by a second device; and

Mapping the plurality of PFs to the at least one PF and the at least one VF includes mapping the plurality of PFs to the at least one PF, the at least one VF, the at least one second PF, and the at least one second VF.

Statement 43. An embodiment of the inventive concept comprises the method according to statement 42, wherein the method further comprises:

determining resources of the device and the second device; and

Aggregate the resources of the device and the second device.

Statement 44. An embodiment of the inventive concept comprises the method according to statement 35, wherein the method further comprises:

receiving, at a PF among the plurality of PFs exposed by the endpoint, a request from a host;

Mapping a PF of the plurality of PFs exposed by an endpoint of the LWB to a VF of the at least one VF exposed by the device;

converting a request from a PF of the plurality of PFs exposed by an endpoint to a VF of the at least one VF exposed by a device; and

A request is sent to a VF of the at least one VF exposed by the device.

Statement 45. An embodiment of the inventive concept comprises the method according to statement 44, the method further comprising:

receiving a response to the request from a VF of the at least one VF disclosed by the device;

Mapping a VF in the at least one VF exposed by the device to a PF in the plurality of PFs exposed by an endpoint of the LWB;

converting a response from a response of a VF in the at least one VF exposed by the device to a response of a PF in the plurality of PFs exposed by the endpoint; and

The response is sent to the host via a PF among the plurality of PFs exposed by the endpoint of the LWB.

Statement 46. An embodiment of the inventive concept includes a method according to statement 45, wherein the step of converting a response from a response of a VF in the at least one VF disclosed by the device to a response of a PF in the multiple PFs disclosed by the endpoint includes: modifying an address in the response based on a first base address register (BAR) of a PF in the multiple PFs disclosed by the endpoint and a second BAR of a VF in the at least one VF disclosed by the device.

Statement 47. An embodiment of the inventive concept includes a method according to statement 45, wherein the step of mapping a VF in the at least one VF disclosed by the device to a PF in the multiple PFs disclosed by the endpoint of the LWB includes: using a configuration table in the APP-RP of the LWB, mapping a VF in the at least one VF disclosed by the device to a PF in the multiple PFs disclosed by the endpoint of the LWB.

Statement 48. An embodiment of the inventive concept includes a method according to statement 44, wherein the step of converting a request from a PF among the multiple PFs exposed by the endpoint to a VF among the at least one VF exposed by the device includes: modifying an address in the request based on a first base address register (BAR) of a PF among the multiple PFs exposed by the endpoint and a second BAR of a VF among the at least one VF exposed by the device.

Statement 49. An embodiment of the inventive concept includes a method according to statement 44, wherein the step of mapping a PF among the multiple PFs disclosed by an endpoint of the LWB to a VF among the at least one VF disclosed by the device includes: using a configuration table in an APP-RP of the LWB, mapping a PF among the multiple PFs disclosed by an endpoint of the LWB to a VF among the at least one VF disclosed by the device.

Statement 50. An embodiment of the inventive concept comprises the method of statement 44, wherein the step of sending the request to a VF of the at least one VF exposed by the device comprises:

selecting between the device and a second device, the second device exposing at least one second PF and at least one second VF based at least in part on a mapping of PFs in the plurality of PFs exposed by an endpoint of the LWB to VFs in the at least one VF exposed by the device; and

The request is sent to the APP-EP associated with the device.

Statement 51. An embodiment of the inventive concept comprises the method according to statement 44, wherein the method further comprises:

determining a credit for a data transfer involving the device; and

Issues credit to the device.

Statement 52. An embodiment of the inventive concept comprises the method of statement 51 wherein the step of issuing a credit to the device comprises writing the credit to an address in the device.

Statement 53. An embodiment of the inventive concept includes the method of statement 51 wherein the step of issuing a credit to the device includes the device reading the credit from an address in the LWB.

Statement 54. An embodiment of the inventive concept includes the method of statement 51 wherein the step of issuing credit to the device includes sending a message from the LWB to the device, the message including the credit.

Statement 55. An embodiment of the inventive concept comprises the method according to statement 35, the method further comprising:

receiving a configuration write request; and

The plurality of PFs exposed by endpoints of the LWB are configured based at least in part on the configuration write request.

Statement 56. An embodiment of the inventive concept includes the method according to statement 55, the method further comprising: configuring the at least one PF and the at least one VF disclosed by the device to mirror the configuration of the multiple PFs disclosed by the endpoint of the LWB at least in part based on the configuration write request.

Statement 57. An embodiment of the inventive concept includes the method of statement 55 wherein the step of receiving a configuration write request includes receiving the configuration write request from a host.

Statement 58. An embodiment of the inventive concept includes the method of statement 55, wherein the step of receiving a configuration write request comprises: receiving the configuration write request from a device.

Statement 59. An embodiment of the inventive concept includes a method according to statement 58, wherein the step of configuring the multiple PFs exposed by the endpoint of the LWB based at least in part on a configuration write request includes: configuring the capabilities of at least one PF of the multiple PFs exposed by the endpoint based at least in part on a configuration write request received from the device.

Statement 60. An embodiment of the inventive concept includes the method of statement 55, wherein configuring the plurality of PFs exposed by an endpoint of the LWB based at least in part on a configuration write request comprises using a state machine to perform a single root input/output virtualization (SR-IOV) sequence.

Statement 61. An embodiment of the inventive concept comprises the method according to statement 35, wherein the method further comprises:

Receive a configuration read request;

determining a configuration of the at least one PF and the at least one VF of a device; and

The configuration read request is responded to with the configuration of the at least one PF and the at least one VF of the device.

Statement 62. An embodiment of the inventive concept includes the method of statement 61, wherein the step of determining the configuration of the at least one PF and the at least one VF of the device includes: determining the configuration of the plurality of PFs exposed by endpoints of the device.

Statement 63. An embodiment of the inventive concept includes the method of statement 61, wherein the step of determining the configuration of the at least one PF and the at least one VF of the device includes: querying the device for the configuration of the at least one PF and the at least one VF of the device.

Statement 64. An embodiment of the inventive concept comprises the method of statement 61 wherein the step of determining a configuration of the at least one PF and the at least one VF of the device comprises: not delivering a configuration read request to the device.

Statement 65. An embodiment of the inventive concept includes the method according to statement 35, the method further comprising: using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB to dynamically change the mapping of the multiple PFs at the endpoint of the LWB to the at least one PF and the at least one VF disclosed by the device.

Statement 66. An embodiment of the inventive concept includes a method according to statement 65, wherein the step of dynamically changing the mapping of the multiple PFs at the endpoint of the LWB to the at least one PF and the at least one VF disclosed by the device using the application layer-endpoint (APP-EP) and the application layer-root port (APP-RP) of the LWB includes: dynamically changing the mapping of the multiple PFs at the endpoint of the LWB to the at least one PF and the at least one VF disclosed by the device using the application layer-endpoint (APP-EP) and the application layer-root port (APP-RP) of the LWB based at least in part on a change in a quality of service (QoS) policy for at least one of the multiple PFs disclosed by the endpoint of the LWB, a storage parameter, a day, a time of day, or bandwidth usage.

Statement 67. An embodiment of the inventive concept comprises the method of statement 35, further comprising: throttling the bandwidth of the device.

Statement 68. An embodiment of the inventive concept includes the method of statement 67, further comprising: measuring a bandwidth used by at least one of the plurality of PFs exposed by an endpoint.

Statement 69. An embodiment of the inventive concept comprises the method of statement 67 wherein throttling the bandwidth of the device comprises introducing an inter-packet gap between the first data packet and the second data packet.

Statement 70. An embodiment of the inventive concept includes a method according to statement 67, wherein the step of throttling the bandwidth of the device includes throttling the bandwidth of the device based at least in part on a QoS policy set by the host, a QoS policy set by the device, a temperature of the LWB, power consumption of the LWB, a temperature of the SSD, and power consumption of the SSD.

Statement 71. An embodiment of the inventive concept includes a method according to statement 70, wherein the step of throttling the bandwidth of the device based at least in part on a QoS policy set by a host, a QoS policy set by a device, a temperature of the LWB, a power consumption of the LWB, a temperature of the SSD, and a power consumption of the SSD includes: throttling the bandwidth of the device based at least in part on one of the temperature of the LWB, the power consumption of the LWB, the temperature of the SSD, and the power consumption of the SSD exceeding a threshold.

Statement 72. An embodiment of the inventive concept includes the method of statement 71, further comprising ending bandwidth throttling of the device based at least in part on one of a temperature of the LWB, power consumption of the LWB, a temperature of the SSD, and power consumption of the SSD falling below a second threshold.

Statement 73. An embodiment of the inventive concept comprises the method according to statement 35, wherein the method further comprises:

receiving a QoS policy of a PF of the plurality of PFs disclosed by an endpoint of the LWB; and

A QoS policy of a PF in the plurality of PFs exposed by an endpoint of the LWB is implemented.

Statement 74. An embodiment of the inventive concept includes a method according to statement 73, wherein the step of receiving a QoS policy of a PF among the multiple PFs exposed by an endpoint of the LWB includes: receiving a QoS policy of a PF among the multiple PFs exposed by an endpoint of the LWB from one of a host and a device.

Statement 75. Embodiments of the inventive concept include an article comprising a non-transitory storage medium having stored thereon instructions that, when executed by a machine, cause:

enumerating at least one physical function (PF) exposed by the device using a root port of a lightweight bridge (LWB);

enumerating at least one virtual function (VF) exposed by the device using a root port of the LWB;

generating a plurality of PFs at the endpoints of the LWB to be exposed to the host; and

The plurality of PFs at an endpoint of the LWB are mapped to the at least one PF and the at least one VF exposed by the device using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB.

Statement 76. An embodiment of the inventive concept comprises an article according to statement 75, wherein:

The step of using a root port of a lightweight bridge (LWB) to enumerate at least one physical function (PF) exposed by the device comprises: using the root port of the LWB to enumerate the at least one PF exposed by the device when the LWB is started; and

The step of enumerating at least one virtual function (VF) exposed by the device using a root port of the LWB includes enumerating the at least one VF exposed by the device using a root port of the LWB when the LWB is booted.

Statement 77. An embodiment of the inventive concept comprises an article according to statement 75, wherein:

The step of using a root port of a lightweight bridge (LWB) to enumerate at least one physical function (PF) exposed by the device comprises: when the device is connected to the LWB, using the root port of the LWB to enumerate the at least one PF exposed by the device; and

The step of enumerating at least one virtual function (VF) exposed by the device using a root port of the LWB includes enumerating the at least one VF exposed by the device using a root port of the LWB when the device is connected to the LWB.

Statement 78. An embodiment of the inventive concept comprises the article of statement 75, wherein the device comprises a solid state drive (SSD).

Statement 79. An embodiment of the inventive concept comprises an article according to statement 75, wherein:

The plurality of PFs include a plurality of Peripheral Component Interconnect Express (PCIe) PFs;

The at least one physical function PF comprises at least one PCIe PF; and

The at least one virtual function (VF) includes at least one PCIe VF.

Statement 80. An embodiment of the inventive concept comprises the article of statement 79, wherein:

APP-EP includes PCIe APP-EP (PAPP-EP); and

APP-RP includes PCIe APP-RP (PAPP-RP).

Statement 81. An embodiment of the inventive concept comprises the article of statement 75, the non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

Receive enumeration from the host; and

The plurality of PFs at the endpoints of the LWB are exposed to the host.

Statement 82. An embodiment of the inventive concept comprises an article according to statement 75, wherein:

The step of using a root port of a lightweight bridge (LWB) to enumerate at least one physical function (PF) exposed by a device includes: using a second root port of the LWB to enumerate at least one second PF exposed by a second device;

The step of using a root port of the LWB to enumerate at least one virtual function (VF) exposed by the device includes: using a second root port of the LWB to enumerate at least one second VF exposed by a second device; and

Mapping the plurality of PFs to the at least one PF and the at least one VF includes mapping the plurality of PFs to the at least one PF, the at least one VF, the at least one second PF, and the at least one second VF.

Statement 83. An embodiment of the inventive concept comprises the article of statement 82, the non-transitory storage medium having further instructions stored thereon, which, when executed by a machine, cause:

determining resources of the device and the second device; and

Aggregate the resources of the device and the second device.

Statement 84. An embodiment of the inventive concept comprises the article of statement 75, the non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

receiving, at a PF among the plurality of PFs exposed by the endpoint, a request from a host;

Mapping a PF of the plurality of PFs exposed by an endpoint of the LWB to a VF of the at least one VF exposed by the device;

converting a request from a PF of the plurality of PFs exposed by an endpoint to a VF of the at least one VF exposed by a device; and

A request is sent to a VF of the at least one VF exposed by the device.

Statement 85. An embodiment of the inventive concept comprises the article of statement 84, a non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

receiving a response to the request from a VF of the at least one VF disclosed by the device;

Mapping a VF in the at least one VF exposed by the device to a PF in the plurality of PFs exposed by an endpoint of the LWB;

converting a response from a VF in the at least one VF exposed by the device to a PF in the plurality of PFs exposed by the endpoint; and

The response is sent to the host via a PF among the plurality of PFs exposed by the endpoint of the LWB.

Statement 86. An embodiment of the inventive concept includes an article according to statement 85, wherein the step of converting a response from a VF in the at least one VF disclosed by the device to a PF in the multiple PFs disclosed by the endpoint includes: modifying an address in the response based on a first base address register (BAR) of a PF in the multiple PFs disclosed by the endpoint and a second BAR of a VF in the at least one VF disclosed by the device.

Statement 87. An embodiment of the inventive concept includes an article according to statement 85, wherein the step of mapping a VF in the at least one VF disclosed by the device to a PF in the multiple PFs disclosed by the endpoint of the LWB includes: using a configuration table in the APP-RP of the LWB, mapping a VF in the at least one VF disclosed by the device to a PF in the multiple PFs disclosed by the endpoint of the LWB.

Statement 88. An embodiment of the inventive concept includes an article according to statement 84, wherein the step of converting a request from a PF in the multiple PFs exposed by the endpoint to a VF in the at least one VF exposed by the device includes: modifying an address in the request based on a first base address register (BAR) of a PF in the multiple PFs exposed by the endpoint and a second BAR of a VF in the at least one VF exposed by the device.

Statement 89. An embodiment of the inventive concept includes an article according to statement 84, wherein the step of mapping a PF in the multiple PFs disclosed by an endpoint of the LWB to a VF in the at least one VF disclosed by the device includes: using a configuration table in an APP-RP of the LWB, mapping a PF in the multiple PFs disclosed by an endpoint of the LWB to a VF in the at least one VF disclosed by the device.

Statement 90. An embodiment of the inventive concept comprises the article of statement 84, wherein the step of sending the request to a VF of the at least one VF exposed by the device comprises:

selecting between the device and a second device, the second device exposing at least one second PF and at least one second VF based at least in part on a mapping of PFs of the plurality of PFs exposed by an endpoint of the LWB to VFs of the at least one VF exposed by the device; and

The request is sent to the APP-EP associated with the device.

Statement 91. An embodiment of the inventive concept comprises the article of statement 84, a non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

determining a credit for a data transfer involving the device; and

Issues credit to the device.

Statement 92. An embodiment of the inventive concept comprises the article of statement 91 wherein the step of issuing credit to the device comprises writing the credit to an address in the device.

Statement 93. An embodiment of the inventive concept comprises the article of statement 91 wherein the step of issuing a credit to the device comprises the device reading the credit from an address in the LWB.

Statement 94. An embodiment of the inventive concept comprises the article of statement 91, wherein the step of issuing credit to the device comprises sending a message from the LWB to the device, the message comprising the credit.

Statement 95. An embodiment of the inventive concept comprises the article of statement 75, a non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

receiving a configuration write request; and

The plurality of PFs exposed by endpoints of the LWB are configured based at least in part on the configuration write request.

Statement 96. An embodiment of the inventive concept includes an article according to statement 95, a non-transitory storage medium having more instructions stored thereon, which, when executed by a machine, cause: at least in part based on a configuration write request, configure the at least one PF and the at least one VF disclosed by the device to mirror the configuration of the multiple PFs disclosed by an endpoint of the LWB.

Statement 97. An embodiment of the inventive concept comprises the article of statement 95, wherein the step of receiving the configuration write request comprises: receiving the configuration write request from a host.

Statement 98. An embodiment of the inventive concept comprises the article of statement 95, wherein the step of receiving the configuration write request comprises: receiving the configuration write request from the device.

Statement 99. An embodiment of the inventive concept includes an article according to statement 98, wherein the step of configuring the multiple PFs exposed by the endpoint of the LWB based at least in part on a configuration write request includes: configuring the capabilities of at least one PF of the multiple PFs exposed by the endpoint based at least in part on a configuration write request received from the device.

Statement 100. An embodiment of the inventive concept comprises an article according to statement 95, wherein the step of configuring the plurality of PFs exposed by the endpoints of the LWB based at least in part on the configuration write request comprises: using a state machine to perform a single root input/output virtualization (SR-IOV) sequence.

Statement 101. An embodiment of the inventive concept comprises the article of statement 75, a non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

Receive a configuration read request;

determining a configuration of the at least one PF and the at least one VF of a device; and

The configuration read request is responded to with the configuration of the at least one PF and the at least one VF of the device.

Statement 102. An embodiment of the inventive concept comprises the article of claim 101, wherein the step of determining the configuration of the at least one PF and the at least one VF of the device comprises: determining the configuration of the plurality of PFs exposed by endpoints of the device.

Statement 103. An embodiment of the inventive concept comprises the article of statement 101, wherein the step of determining the configuration of the at least one PF and the at least one VF of the device comprises: querying the device for the configuration of the at least one PF and the at least one VF of the device.

Statement 104. An embodiment of the inventive concept comprises the article of statement 101, wherein the step of determining a configuration of the at least one PF and the at least one VF of the device comprises: not delivering a configuration read request to the device.

Statement 105. An embodiment of the inventive concept includes an article according to statement 75, a non-transitory storage medium having more instructions stored thereon, which, when executed by a machine, cause: dynamically changing a mapping of the multiple PFs at an endpoint of the LWB to the at least one PF and the at least one VF exposed by the device using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB.

Statement 106. An embodiment of the inventive concept includes an article according to statement 105, wherein the step of dynamically changing the mapping of the multiple PFs at the endpoint of the LWB to the at least one PF and the at least one VF disclosed by the device using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB includes: dynamically changing the mapping of the multiple PFs at the endpoint of the LWB to the at least one PF and the at least one VF disclosed by the device using an application layer-endpoint (APP-EP) and an application layer-root port (APP-RP) of the LWB based at least in part on a change in a quality of service (QoS) policy for at least one of the multiple PFs disclosed by the endpoint of the LWB, a storage parameter, a day, a time of day, or bandwidth usage.

Statement 107. An embodiment of the inventive concept comprises the article of statement 75, the non-transitory storage medium having further instructions stored thereon, which when executed by the machine, cause: throttling bandwidth of the device.

Statement 108. An embodiment of the inventive concept comprises an article according to statement 107, a non-transitory storage medium having further instructions stored thereon, which when executed by a machine, cause: measuring a bandwidth used by at least one of the plurality of PFs exposed by an endpoint.

Statement 109. An embodiment of the inventive concept comprises the article of statement 107, wherein throttling the bandwidth of the device comprises introducing an inter-packet gap between the first data packet and the second data packet.

Statement 110. An embodiment of the inventive concept includes an article according to statement 107, wherein the step of throttling the bandwidth of the device includes: throttling the bandwidth of the device based at least in part on a QoS policy set by the host, a QoS policy set by the device, a temperature of the LWB, a power consumption of the LWB, a temperature of the SSD, and a power consumption of the SSD.

Statement 111. An embodiment of the inventive concept includes an article according to statement 110, wherein the step of throttling the bandwidth of the device based at least in part on a QoS policy set by a host, a QoS policy set by a device, a temperature of the LWB, a power consumption of the LWB, a temperature of the SSD, and a power consumption of the SSD includes: throttling the bandwidth of the device based at least in part on one of the temperature of the LWB, the power consumption of the LWB, the temperature of the SSD, and the power consumption of the SSD exceeding a threshold.

Statement 112. An embodiment of the inventive concept includes an article according to statement 111, a non-temporary storage medium having more instructions stored thereon, which, when executed by a machine, causes: ending bandwidth throttling of the device based at least in part on one of the temperature of the LWB, the power consumption of the LWB, the temperature of the SSD, and the power consumption of the SSD falling below a second threshold.

Statement 113. An embodiment of the inventive concept comprises the article of statement 75, a non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

receiving a QoS policy of a PF of the plurality of PFs disclosed by an endpoint of the LWB; and

A QoS policy of a PF in the plurality of PFs exposed by an endpoint of the LWB is implemented.

Statement 114. An embodiment of the inventive concept includes an article according to statement 75, wherein the step of receiving a QoS policy of a PF in the multiple PFs exposed by an endpoint of the LWB includes: receiving a QoS policy of a PF in the multiple PFs exposed by an endpoint of the LWB from one of a host and a device.

Therefore, in view of the wide variety of permutations of the embodiments described herein, this detailed description and accompanying materials are intended to be illustrative only and should not be taken as limiting the scope of the invention. Therefore, what is claimed is all such modifications that may fall within the scope and spirit of the claims and their equivalents.

Claims (20)

1. A lightweight bridge circuit comprising:

an endpoint for connection to a host, the endpoint disclosing a plurality of physical functions to the host,

A root port for connection to a device, the device exposing at least one physical function and at least one virtual function to the root port; and

An application layer-endpoint and an application layer-root port for converting between the plurality of physical functions disclosed by the endpoint to the host and the at least one physical function disclosed by the device to the root port and the at least one virtual function disclosed by the device to the root port,

Wherein the application layer-endpoint and the application layer-root port implement a mapping between the plurality of physical functions disclosed by the endpoint to the host and the at least one physical function disclosed by the device to the root port and the at least one virtual function disclosed by the device to the root port.

2. The lightweight bridge circuit of claim 1, further comprising: a configuration manager for configuring the application layer-endpoints and the application layer-root ports.

3. The lightweight bridge circuit of claim 2, wherein the application layer-root port comprises a configuration table for storing mappings between the plurality of physical functions disclosed by the endpoint and the at least one physical function and the at least one virtual function disclosed by the device.

4. The lightweight bridge circuit of claim 2, wherein the configuration manager enumerates the at least one physical function and the at least one virtual function disclosed by the device and generates the plurality of physical functions disclosed by the endpoint based at least in part on the at least one physical function and the at least one virtual function disclosed by the device.

5. The lightweight bridge circuit of claim 2, wherein the configuration manager ensures that the first configuration of the device mirrors the second configuration of the endpoint.

6. The lightweight bridge circuit of claim 2, wherein the configuration manager comprises: read-only memory and state machine for implementing a single root input/output virtualization sequence.

7. The lightweight bridge circuit of any one of claims 1 to 6, further comprising:

a second application layer-endpoint and a second application layer-root port;

A second device disclosing at least one second physical function and at least one second virtual function; and

A multiplexer and a demultiplexer arranged to be connected to the end points and to each of the application layer-end points and the second application layer-end points,

Wherein the second application layer-endpoint and the second application layer-root port implement a second mapping between the plurality of physical functions disclosed by the endpoint and the at least one second physical function and the at least one second virtual function disclosed by the second device.

8. The lightweight bridge circuit of claim 7, wherein the lightweight bridge circuit provides aggregate resources for the device and the second device.

9. The lightweight bridge circuit of claim 1, wherein the lightweight bridge circuit implements bandwidth throttling.

10. The lightweight bridge circuit of claim 9, wherein the lightweight bridge circuit implements bandwidth throttling based at least in part on at least one of a policy set by a host, a policy set by a device, a temperature of the lightweight bridge circuit, a power consumption of the lightweight bridge circuit, a temperature of a device, and a power consumption of a device.

11. The lightweight bridge circuit of claim 1, wherein the lightweight bridge circuit implements QoS policies on a physical function of the plurality of physical functions disclosed by a device.

12. A method of operating a lightweight bridge, comprising:

enumerating at least one physical function disclosed by the device using a root port of the lightweight bridge;

enumerating at least one virtual function disclosed by the device using a root port of the lightweight bridge;

generating a plurality of physical functions at endpoints of the lightweight bridge to be exposed to the host; and

The plurality of physical functions exposed to the host at the endpoint of the lightweight bridge are mapped to the at least one physical function exposed to the root port by the device and the at least one virtual function exposed to the root port by the device using the application-layer endpoint and the application-layer root port of the lightweight bridge.

13. The method according to claim 12, wherein:

the step of enumerating at least one physical function disclosed by the device using the root port of the lightweight bridge comprises: enumerating at least one second physical function disclosed by the second device using a second root port of the lightweight bridge;

The step of enumerating at least one virtual function disclosed by the device using the root port of the lightweight bridge comprises: enumerating at least one second virtual function disclosed by a second device using a second root port of the lightweight bridge; and

The step of mapping the plurality of physical functions to the at least one physical function and the at least one virtual function comprises: mapping the plurality of physical functions to the at least one physical function, the at least one virtual function, the at least one second physical function, and the at least one second virtual function.

14. The method of claim 13, further comprising:

determining resources of the device and the second device; and

The resources of the device and the second device are aggregated.

15. The method of any one of claims 12 to 14, further comprising:

receiving a request from a host for a physical function of the plurality of physical functions disclosed by the endpoint;

Mapping the physical function of the plurality of physical functions disclosed by the endpoint of the lightweight bridge to a virtual function of the at least one virtual function disclosed by the device;

Converting the request from a request for the physical function of the plurality of physical functions disclosed by the endpoint to a request for the virtual function of the at least one virtual function disclosed by the device; and

The request is sent to the virtual function of the at least one virtual function disclosed by the device.

16. The method of claim 15, further comprising:

determining a credit related to data transfer of the device; and

The credit is issued to the device.

17. The method of claim 12, further comprising: the bandwidth of the device is throttled.

18. The method of claim 17, wherein throttling the bandwidth of the device comprises: the bandwidth of the device is throttled based at least in part on at least one of a QoS policy set by the host, a QoS policy set by the device, a temperature of the lightweight bridge, a power consumption of the lightweight bridge, a temperature of the device, and a power consumption of the device.

19. An article comprising a non-transitory storage medium having instructions stored thereon that, when executed by a machine, cause:

enumerating at least one physical function disclosed by the device using a root port of the lightweight bridge;

enumerating at least one virtual function disclosed by the device using a root port of the lightweight bridge;

generating a plurality of physical functions at endpoints of the lightweight bridge to be exposed to the host; and

The plurality of physical functions exposed to the host at the endpoint of the lightweight bridge are mapped to the at least one physical function exposed to the root port by the device and the at least one virtual function exposed to the root port by the device using the application-layer endpoint and the application-layer root port of the lightweight bridge.

20. The article of claim 19, wherein:

the step of enumerating at least one physical function disclosed by the device using the root port of the lightweight bridge comprises: enumerating at least one second physical function disclosed by the second device using a second root port of the lightweight bridge;

The step of enumerating at least one virtual function disclosed by the device using the root port of the lightweight bridge comprises: enumerating at least one second virtual function disclosed by a second device using a second root port of the lightweight bridge; and

The step of mapping the plurality of physical functions to the at least one physical function and the at least one virtual function comprises: mapping the plurality of physical functions to the at least one physical function, the at least one virtual function, the at least one second physical function, and the at least one second virtual function.