WO2011078861A1

WO2011078861A1 - A computer platform providing hardware support for virtual inline appliances and virtual machines

Info

Publication number: WO2011078861A1
Application number: PCT/US2009/069386
Authority: WO
Inventors: Ravi L. Sahita; Raj K. Ramanujan; Arun Raghunath; Amir Zinaty; Parthasarathy Sarangam
Original assignee: Intel Corp
Current assignee: Intel Corp
Priority date: 2009-12-23
Filing date: 2009-12-23
Publication date: 2011-06-30
Anticipated expiration: 2012-06-23

Abstract

A computer platform may support hardware assisted packet handling to efficiently filter and route packets in an ordered sequence through a virtual network of inline appliances and virtual machines via network interface card (NIC) and other platform hardware. The hardware may add, update, and parse tags representing routing action into packets. The computer platform may support hardware assisted inter-VM communication without requiring additional processor cycles to switch packets between inline appliances in single root I/O virtualized (SR-IOV) and multiple root I/O virtualized (MR-IOV) scenarios. The computer platform may support hardware assisted VM migration technique to reduce the complexity and software overheads involved in migrating a running virtual machine from one platform to the other.

Description

A COMPUTER PLATFORM PROVIDING HARDWARE SUPPORT FOR VIRTUAL INLINE APPLIANCES AND VIRTUAL MACHINES

BACKGROUND

Hardware virtualization technology enables a single computer platform to support multiple virtual machines (VM) and each VM may support a guest operating system (OS). A virtual machine monitor (VMM) (hypervisor) may hide the hardware components of the computer platform from the guest OS (or VMs) and a guest OS on one VM may not know that the resources of the platform are being shared with other VMs. Such VMs communicate with each other using a standard networking protocols such as TCP/IP. To improve the efficiency, the inter- VM traffic may be avoided from entering the rest of the physical network and for security purposes the inter- VM traffic may be monitored transparently within the platform using monitoring appliances such as a virtual inline network appliance (VINA). In a software approach, VMMs use a network IOVM (I O virtual machine) to sort and copy data between the VM to VM and between VM and the VINAs. Unfortunately, a large portion of CPU resources assigned to the IOVM is used for switching and copying.

Supporting multiple VMs and guest operating systems on a computer platform has led to consolidation of many different servers on a single computer platform. A direct consequence of consolidation of servers on the single platform is that the communication (or packets exchanged) between the servers may not exit that computer platform. Hence, the network appliances that monitor the network traffic between the servers may reside in that computer platform. This scenario has led to virtualization of network appliances as well in which the network appliance functionality (e.g., firewall, virus checking, intrusion detection) may be run on virtual machines and such virtual machines may be referred to as virtual inline network appliance (VINA). A platform may comprise various VINAs with each VINA providing a function in monitoring the packets and the packets may also traverse multiple VINAs before reaching the destination VM. To minimize changes in the appliance, the VINAs may forward the received packets to a network interface card (NIC) after processing the received packets similar to a physical network appliance. Also, a specific VINA may examine the packets before other VINA implying an order in which the packets may be sent to VINAs. Given multiple VINAs and ordering constraints, the computer platform may have to make appropriate routing decisions to determine a next hop for the packet in the virtual network.

Also, to ensure high availability and optimum usage the virtual machines may be migrated from one computer platform to the other. Several approaches such as 'Save and Restore' migration and Live migration approach may be used. In the 'Save and Restore' migration approach, a current state of the VM is stored first and then the saved current state is used to restart/restore the VM on the destination platform. In Save and Restore approach, the VMs resources on the source platform is de-allocated and returned to the VMM causing disruption in the network connectivity and hence the save and restore approach is not seamless. In Live migration approach, the software VMMs transfer the VMs memory in an initial step and transfers only the changes made to the memory in subsequent steps. Such incremental updates to memory may need complex and cumbersome logic to be supported by the VMMs.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention described herein is illustrated by way of example and not by way of limitation in the accompanying figures. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements.

FIG. 1 illustrates an environment 100 comprising one or more multi-server systems each including a computer platform in accordance with one embodiment.

FIG. 2 illustrates a MSS 1 10, which may provide filtering of packets and routing the packets in an ordered sequence through the VINAs in accordance with one embodiment.

FIG. 3 is a flow-chart illustrating an operation of the MSS 1 10, which may provide filtering of packets and routing the packets in an ordered sequence through the VINAs in accordance with one embodiment.

FIG. 4 is a flow-chart illustrating a detailed operation of the MSS 110, which may provide filtering of packets and routing the packets in an ordered sequence through the VINAs using information in a layer-2 header in accordance with an embodiment.

FIG. 5 illustrates a format in which a tag may be inserted into the packet to enable a network interface card to filter and route the packet in an ordered sequence through the VINAs using layer-2 information in accordance with an embodiment.

FIG. 6 illustrates a network interface card in which a filter table and a matching table may be used to filter and route the packets in an ordered sequence through the VINAs in accordance with an embodiment.

FIG. 7 illustrates the MSS 110, which may support hardware assisted inter-VM communication without consuming additional processor cycles to redirect or switch packets between in-line appliances in SR-IOV scenario in accordance with an embodiment.

FIG. 8 is a flow-chart illustrating an operation of the MSS 1 10, which may support hardware assisted inter-VM communication conserving computational resources and interconnect bandwidth in accordance with an embodiment.

FIG. 9 is a flow-chart illustrating an operation of the MSS 1 10, which may support providing inter-VM traffic to a VINA using SR-IOV NIC in accordance with an embodiment.

FIG. 10 is a flow-chart illustrating an operation of the MSS 1 10, which may support hardware assisted inter-VM communication in response to receiving a packet from an external source and from a VM provisioned within the computer platform 700 in accordance with an embodiment.

FIG. 1 1 illustrates a source MSS 110-A and a destination MSS 1 10-B, which may together support hardware assisted migration of a virtual machine from the source to the destination MSS in accordance with an embodiment.

FIG. 12 is a flow-chart illustrating an operation of the source MSS 110-A, which may support hardware assisted migration of a virtual machine to the destination MSS 1 10-

B in accordance with an embodiment.

FIG. 13 illustrates a table 1300, which depicts management of memory pages of a virtual machine that is involved in hardware assisted migration in accordance with an embodiment.

FIG. 14 is a flow-chart illustrating an operation of the destination MSS 1 10-B, which may support hardware assisted migration of a virtual machine from the source to the destination platform in accordance with an embodiment.

DETAILED DESCRIPTION

The following description describes a computer platform, which may provide hardware support to filter and route the packets in an ordered sequence through the VINAs, hardware based inter- VM communication, and hardware assisted VM migration capabilities. In the following description, numerous specific details such as logic implementations, resource partitioning/sharing/duplication implementations, types and interrelationships of system components, and logic partitioning/integration choices are set forth in order to provide a more thorough understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. In other instances, control structures, gate level circuits, and full software instruction sequences have not been shown in detail in order not to obscure the invention. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

References in the specification to "one embodiment", "an embodiment", "an example embodiment", etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

Embodiments of the invention may be implemented in hardware, firmware, software, or any combination thereof. Further, firmware, software, routines, instructions may be described herein as performing certain actions. However, it should be appreciated that such descriptions are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

An embodiment of an environment 100 comprising one or more multi-server systems (MSS) 110 each including at least one computer platform (CP) 112 is illustrated in Fig. 1. In one embodiment, the environment 100 may comprise one or more multi- server systems 1 10-A to 110-N and a network 150. In one embodiment, the MSS 110-A to 110-N may each include a computer platform (CP) 112-A to 112-N. In one embodiment, each of the CP 1 12 may support multiple servers or multiple virtual machines with guest operating system (OS). In one embodiment, the environment 100 may refer to a data center in which the computer platforms 112 may support various features.

In one embodiment, the computer platform 1 12 may support hardware assisted packet handling to efficiently filter and route packets in an ordered sequence through an on-platform virtual network of inline appliances and virtual machines via network interface card (NIC) hardware. In one embodiment, the NIC hardware may include capabilities to add, update, and parse tags representing routing action into packets. In one embodiment, the computer platform 112 may support hardware assisted inter- VM communication technique without requiring additional processor cycles to switch packets between inline appliances in single root I/O virtualized (SR-IOV) scenarios and/or multi- root I/O virtualized (MR-IOV) without using extra interconnect bandwidth. In one embodiment the computer platform 1 12 may support hardware assisted VM migration technique to reduce the complexity and software overheads involved in migrating a running virtual machine from one platform to the other. In one embodiment, the hardware assisted VM migration technique may be used to increase the amount of memory available to a single VM as local memory and remote memory (used for swapping pages between computer platforms) may be available to the single VM. Also, reducing the complexity of the VMM may increase the security by reducing the software attack surface of the VMM.

An embodiment of a MSS 110, which may support efficient filtering of packets and routing the packets in an ordered sequence through the VINAs is illustrated in FIG. 2. In one embodiment, the MSS 110 may comprise virtual machines (VM) 210-A to 210-N, virtual inline network appliance (VINA) 220-A to 220-K, a virtual machine monitor (VMM) 230, and a computer platform 240. In one embodiment, the computer platform (CP) 240 may comprise a processor 250, a flash memory 255, a logic 260, a memory 280, an I/O device 290-A and 290-B, and a network interface card NIC 295.

The logic 260 may include controller hubs to support I O interconnect, for example. The logic 260 may comprise a chipset or any other such device to support the transfer of data and control information between the processor 250 and the I/O devices 290 and the NIC 295. The logic 260 may support coder-decoders, power control logic, and such other logic. The I/O devices 290 may comprise a key-board, mouse, a hard disk, and a monitor. The memory 280 may store data and/or software instructions that the processor 250 or any other device of the computing platform 240 or VMs 210 or VINAs 220 may access and perform operations. The memory 280 may comprise one or more different types of memory devices such as, for example, DRAM (Dynamic Random Access Memory) devices, SDRAM (Synchronous DRAM) devices, DDR (Double Data Rate) SDRAM devices, or other volatile and/or non-volatile memory devices used in MSS 1 10. The flash memory 255 may comprise devices of fast flash technology. In one embodiment, the flash memory 255 may comprise a NAND FLASH device.

In one embodiment, the CP 240 may support hardware assisted packet handling to efficiently filter and route packets in an ordered sequence through an on-platform virtual network of VINAs and virtual machines via network interface card (NIC) hardware. In one embodiment, the NIC hardware may include capabilities to add, update, and parse tags representing routing action into packets. In one embodiment, the CP 240 may support techniques to route the packets to VINAs based on layer-2 information. In one embodiment, routing the packets based on layer-2 (L2) information or parsing the L2 portion of the packet headers may avoid inefficiency associated with per hop L3/L4 filtering.

In one embodiment, the NIC 295 may comprise an interface 291 to couple the NIC 295 with other portions of the CP 240, a filter specification 292, a mapping table 293, virtual functions 294-A to 294-M, and a control logic 298. In one embodiment, the VINAs 220-A to 220-K may specify filters, which identify a subset of packets that may be sent to each VINA 220-A to 220-K. In one embodiment, the control logic 298 may consolidate the filters received from each VINA 220-A to 220-K and may generate a filter specification 292. In one embodiment, the filter specification 292 may comprise an association between a filter and a routing action. In one embodiment, the routing action may include a list of identifiers of the VINAs 220 that may process the packets matching the filter. In one embodiment, the routing action may comprise a simple bit vector with each bit vector representing a VINA. In one embodiment, a double vlan tag like feature (802. lq-in-q) may be utilized to add the tag into a packet header of a matching packet. In one embodiment, a set bit indicates that the packet is to be sent to the VINA.

In one embodiment, the control logic 298 may generate a mapping table 293, which may associate each of the virtual function 294 to one or more of VMs 210 and VINAs 220. In one embodiment, the control logic 298 may use the filter specifications 292 to match an incoming packet received from the interface 291. In one embodiment, the control logic 298 may use the mapping table 293 to determine a virtual function 294 (say VF 294-A) to which a filter matched packet may be sent. In one embodiment, the control logic 298 may include a tag comprising bytes, representing the routing action, into the filter matched packet and may send the filter matched packet to the virtual function 294. In one embodiment, the control logic 298 may include the tag as a part of the L2 header (or before the start of L3/L4 header) in the packet. In one embodiment, the control logic 298 may cause the virtual function 294 to forward the filter matched packet to a first VINA (say VINA 220-A) in the routing action. In one embodiment, the first VINA 220-A may process the filter matched packet to generate a processed packet and send the processed packet back to the VF 294-A.

In one embodiment, the NIC 295 may receive a processed packet from the first VINA via VF 294-A and forward the processed packet to the control logic 298. In one embodiment, the control logic 298 may determine the next hop (say VINA 220-B) using the routing action bytes, which may be provided as a part of the L2 header or interposed between L2 and L3/L4 header. In one embodiment, the control logic 298 may determine the next hop without using the L3/L4 information and such an approach may increase the efficiency of routing the packets to the VINAs 220 or VMs 210. In one embodiment, the control logic 298 may be aware that the processed packet was sent by the first VINA and the control logic 298 may reset the bit for the first VINA in the routing action.

In one embodiment, the control logic 298 may use the mapping table 293 to determine the virtual function 294 (say VF 294-B for the VINA 220-B) to which the processed packet may be forwarded. In one embodiment, the control logic 298 may forward the processed packet to the VF 294-B, which in turn may forward the processed packet to the next VINA (VINA 220-B) in the list of the VINAs indicated in the routing action. In one embodiment, the above described forwarding technique may be repeated until the packet is forwarded to all the VINAs indicated in the routing action and the control logic 298 may finally forward the packet to the destination VM (say VM 210-A). Such an approach may ensure efficient filtering and routing of the packets in an ordered sequence through a subset of VINAs that subscribe to process the packets before sending the packet to the destination VM.

An embodiment of an operation of the MSS 110, which may provide filtering and routing of the packets in an ordered sequence through the VINAs is illustrated in a flow- chart of FIG. 3. In block 310, the control logic 298 may generate routing actions by consolidating the filter specifications received from the VINAs 220-A to 220-K. In block 350, the control logic 298 may add a tag including the routing action into the layer-2 (L2) header of the packet to control the order in which the packet is sent to the VINAs 220. In block 380, the control logic 298 may route the packet based on the routing action information present in the layer-2 header.

An embodiment of a detailed operation of the MSS 110, which may provide filtering and routing of the packets in an ordered sequence through the VINAs is illustrated in a flow-chart of FIG. 4. In block 410, the control logic 298 may collect filter specifications from the VINAs 220-A to 220-K that identify the subset of incoming packets that may be sent to each VINA 220. In one embodiment, the VINA 220-A may provide a filter specification, which may indicate that the packets with a source destination equaling the address of a VM (say 210-A) may be processed by the VINA 220-A.

In block 415, the control logic 298 may generate a routing action and associate the routing action with each filter (such as those stored in the filter specification 292) and the routing action may include a list of VINA identifiers in an ordered sequence in which the incoming packet matching the filter may be processed. In one embodiment, the control logic 298 may generate a routing action and assuming that NIC 295 receives incoming packets P 1 and P2 and the matching filter may indicate that packet P 1 may match filter specification MF- 1 (620 of FIG. 6 described below) and the order in which the packet P 1 may be sent to VINAs may equal VINA 220-A, VINA-C, and VINA 220-G. In one embodiment, the routing action may comprise identifiers (A, C, G) of the VINAs 220-A, 220-C, and 220-G, respectively associated with the MF-1. Like wise, the incoming packet P2 may match filter specification MF-2 and the routing action associated with the MF-2 may equal a series of VINA identifiers equaling (B, C, H) of the VINAs 220-B, 220-C, and 220-H, respectively with the MF-2.

In block 420, the control logic 298 may generate a mapping function such as a mapping table 293, which may comprise a mapping between the VINA identifier and the virtual function. In one embodiment, the mapping function may include entries such as (VINA-A: VF 294-A). In one embodiment, the mapping of VINA id to VF may indicate that the packets destined to be sent to a VINA (say 220-A) may be sent to virtual function VF 294-A. In one embodiment, the virtual function 294-A may be coupled to the VINA- A and the packets received at the VF 294-A may be forwarded to the VINA-A.

In block 425, the NIC 295 may receive incoming packets PI and P2 from the network 150. In block 430, the control logic 298 of the NIC 295 may match the headers of the incoming packets PI and P2 with the filter specifications 292 such as MF-1 and MF-2 (entries in column 620 of FIG. 6). In block 440, the control logic 298 may tag the routing action associated with the match filter MF- 1 to the packet P 1 if the packet P 1 matches the filter specifications of MF- 1. In one embodiment, the routing action (A, C, G, i.e., the ids of the VINAs 220-A, 220- C, and 220-G) may be tagged at the end of the layer-2 header or before the layer-3/layer-4 headers. In one embodiment, the routing action bytes may be included into the packet between the L2 and L3/L4 headers. Likewise, if the packet P2 matches with the filter specifications MF-2, the NIC 295 may add the routing action (B, C, H, i.e., the identifiers of the VINA 220-B, 220-C, and 220-H) to the packet P2.

In block 450, the control logic 298 may identify the virtual function associated with a first VINA id in the routing action of the packet that matches the filter specifications. In one embodiment, the control logic 298 may use the mapping function 293 to determine a virtual function associated with the VINA-A for the first VINA id in the routing action associated with the packet PI. In one embodiment, the VINA-A may be associated with the virtual function VF 294-A, the VINA-C may be associated with VF 294-C, and the VINA-G may be associated with the virtual function VF 294-G.

In block 455, the control logic 298 may send the incoming packet to a VINA identified by the first VINA identifier through the identified virtual function. In one embodiment, the control logic 298 may send the packet PI to the VINA-A identified by the first VINA id (=A) in the routing action using the virtual function VF 294-A, which is associated with the VINA-A.

In block 460, the VINA identified by the first VINA id may process the packet received from the virtual function associated with the VINA identified by the first VINA id. In one embodiment, the VINA-A may receive the packet P 1 from the VF 294-A and may process the packet P 1.

In block 465, the VINA, which has processed the packet may send back the processed packet to the virtual function. In one embodiment, the VINA-A may send back the processed packet PI to the NIC 295 through the virtual function VF 294-A.

In block 470, the control logic 298 may update the routing action to indicate that the VINA identified by the first VINA id has completed processing of the packet. In one embodiment, the control logic 298 may reset the bit associated with the first VINA id to indicate that the VINA (i.e., VINA-A) identified by the first VINA id has completed processing of the packet PI.

In block 475, the control logic 298 may parse the tag in the layer-2 header of the processed packet and may read the routing action to retrieve the next VINA in the ordered sequence. In one embodiment, the control logic 298 may read the routing action and retrieve the next VINA (i.e., VINA 294-C) in the ordered sequence.

In block 480, the control logic 298 may check for presence of more VINA ids in the routing action and control passes to block 482 if a next VINA id in the ordered sequence is found and to block 495 if no more VINA ids are present in the routing action.

In block 482, the control logic 298 may identify the virtual function associated with a next VINA id in the routing action of the packet that matches the filter specifications. In one embodiment, the control logic 298 may use the mapping table 293 to determine a virtual function associated with the VINA-C for the next VINA id in the routing action associated with the packet PI. In one embodiment, the VINA-C may be associated with the virtual function VF 294-C and the VINA-G may be associated with the virtual function VF 294-G.

In block 484, the control logic 298 may send the packet processed by the VINA identified by the first VINA id to the next VINA identified by the next VINA id through the next VF. In one embodiment, the control logic 298 may send the packet processed by the VINA 220-A to the VINA 220-C identified by the next VINA id (C) through the VF 294-C. In block 486, the VINA identified by the next VINA id may process the packet sent by the VF 294-C.

In block 490, the VINA identified by the next VINA id may send the packet back to the NIC 295. In block 492, the control logic 298 may perform the blocks 470-490 until the packet is sent to all the VINAs identified by the VINA ids in the routing action. In block 495, the NIC 295 may send the packet PI to the destination VM (say VM 210-A). Such an approach may enable efficient filtering and routing of packets to the VINAs and VMs according to an ordered sequence.

An embodiment of a packet format in which a tag may be inserted into the packet to enable a network interface card to filter and route the packet in an ordered sequence through the VINAs using layer-2 information is depicted in FIG. 5. The packet format in which no tags are inserted is depicted in a packet 510. In one embodiment, the tag may be inserted into the packet between the layer-2 and layer-3 headers as depicted in a packet format 530. In one embodiment, the tag 560 may comprise a matching filter MF-1 and the associated VINA identifiers or the MF2 and associated VINA identifiers, or any other matching filter such as MFh. For example, the matching filter MF-1 may be associated with identifiers of (VINA-A, VI A-C, VINA-G), the MF-2 may be associated with identifiers of (VINA-B, VINA-C, VINA-H), the MF-h may be associated with identifiers of (VINA-C and VINA-L). In one embodiment, the control logic 298, as indicated in block 415, may generate the routing action that includes the matching filters and the associated routing action comprising identifiers of VINA and, as indicated in block 440, the control logic 298 inserts the matching filter identifier and the routing action into the tag 560.

An embodiment of a NIC 295 in which filter specifications and a mapping table may be used to filter and route the packets in an ordered sequence through the VINAs is illustrated in FIG. 6. In one embodiment, the NIC 295 may comprise a filter specification 292 and a mapping table 293. In one embodiment, the filter specification 292 may comprise two columns: match filter id 620 and ordered sequence of VINA id 640. In one embodiment, the match filter id 620 may comprise identifiers of the matching filters such as (MF-1, MF-2,..., MF-h) and the ordered sequence of VINA id 640 may comprise [(A, C, G) associated with MF-1; (B, C, H) associated with MF-2; (C, M) associated with MF- 3; ..., and (C,L) associated with MF-h].

In one embodiment, the incoming packet may be matched with a filter and for example, if the packet matches with a first filter with a match filter id of MF-1, the ordered sequence of VINAs that the packet may pass through is indicated by the VINA ids entry (A, C, G) in the ordered sequence of VINA ids 640. Likewise, if the incoming packet matches with a filter with a matching filter id MF-3, the ordered sequence of VINAs that the packet may pass through is indicated by the VINA id entry (C, M) in the ordered sequence of VINA ids 640. In one embodiment, the NIC 295 may determine, as in block 430, the filter that matches with the incoming packet and may look-up the match filter table 292 to identify the ordered sequence of VINAs that the packet may be sent through.

In one embodiment, the control logic 298 may tag the matching filter id and the associated routing action to the packet as indicated in block 440. In one embodiment, the control logic 298 may use the mapping table 293 to identify the virtual function associated with the VINA identified by the first VINA id in the ordered sequence as indicated in block 450. In one embodiment, the control logic 298 may send the packet on an identified virtual function VF 294 as depicted in block 455. In one embodiment, the packet may come back to the NIC 295 after the VINA identified by the first VINA identifier processes the packet. In one embodiment, the NIC 295 may parse the tag inserted at the end of the layer-2 header and then determine the next VINA to which the packet may be sent. As the NIC 295 performs a layer-2 look-up, the efficiency of filtering and routing the packet through an ordered sequence of VINAs may be enhanced.

In one embodiment, the NIC 295 may dynamically choose the order in which the VINAs receive the packets that match the filter. In one embodiment, the NIC 295 may assign each VINA a VINA identifier and an ordered list of ids may be generated to represent each permutation of VINAs. In one embodiment, the NIC 295 may assign each permutation (or ordered list) a permutation identifier and the permutation identifier may be inserted as a tag into the packet header. In one embodiment, for example, if the MSS 110 comprises three VINAs, VINA220-A, 220-B, and 220-C, the NIC 295 may generate 15 permutations and the NIC 295 may associate each permutation with a permutation identifier.

In one embodiment, if the identifier of VINA-A is 'a', VINA-B is 'b', and that of VINA-C is 'c', the following permutations associated with a permutation identifier may be available [(a,b,c)-Pidl; (a,c,b)-Pid2; (b,a,c)-Pid3; (b,c,a)-Pid4; (c,a,b)-Pid5; (c,b,a)-Pid6; (a,b)-Pid7; (a,c)-Pid8; (b,a)-Pid9; (b,c)-PidlO; (c,a)-Pidl l; (c,b)-Pidl2; (a)-Pidl3; (b)- Pidl4; (c)-Pidl5]. In one embodiment, the NIC 295 may insert the Permutaion id (Pid) number as the tag and may change the Pid number dynamically to change the ordered sequence that the packet may be sent through.

An embodiment of MSS 110, which may support hardware assisted inter-VM communication without consuming additional processor cycles to redirect or switch packets between in-line appliances in SR-IOV scenario, is illustrated in FIG. 7. In one embodiment, the MSS 110 may comprise virtual machines (VM) 710-A to 710-N, virtual inline network appliance (VINA) 720-A to 720-K, a virtual machine monitor (VMM) 730, and a computer platform 740. In one embodiment, the computer platform (CP) 740 may comprise a processor 750, a flash memory 755, a logic 760, a fast path hardware assist 770, a memory 780, I/O devices 790-A and 790-B, and a network interface card NIC 795. In one embodiment, the processor 750 may support VMM 730, VMs 710-A to 710-N, and VINAs 720-A to 720-K. In one embodiment, the computer platform 740 may support hardware assisted inter-VM communication technique without consuming additional hardware cycles to switch packets between inline appliances in a single root I/O virtualized (SR-IOV) or multi-root I/O virtualized (MR-IOV) scenarios and without using extra interconnect bandwidth. In one embodiment, the VMM 730, as a part of the set-up, may assign virtual functions such as VF 794-A to VF 794-M to VMs 710-A to 710-N and the VINAs 720-A to 720-K. In one embodiment, the VMM 730 may expose application program interface (APIs) to allow VINA 720-A to 720-K to subscribe to the packet flows. In one embodiment, the VMM 730 may program the NIC filters through a physical function PF 799 if the VINA 720-A to 720-K uses the software APIs to subscribe to the packet flows. In other embodiment, the VINAs 720-A to 720-K may program the NIC 795 using a hardware interface through the associated VFs 794-A to 794-M.

In one embodiment, the NIC 795 may be coupled to other blocks of the computer platform CP 740 using an interconnect 769. In one embodiment, the interconnect 769 may include peripheral component interconnect-express (PCIe), which may support transfer of data between the NIC 795 and other blocks (such as processor 750 and memory 780) of the CP 740. However, the number of packets that may be transferred between the NIC 795 (provisioned south of the interface 769) and the other blocks of the CP 740 such as the processor 750 and memory 780 (provisioned north of the interface 769) on the interconnect 769 may be limited by the bandwidth of the interconnect 769. To overcome the bandwidth limitation of the interconnect 769, in one embodiment, a fast path assist hardware 770 may be provisioned within the CP 740 in close proximity to the processor 750 or the memory 780 to the north of the interconnect 769 to reduce the traffic to the NIC 795 on the interconnect 769. In one embodiment, the fast path assist hardware 770 may perform a substantial portion of the packet processing that otherwise may be performed by the NIC 795 and as a result, the amount of packets transferred on the interconnect 769 to the NIC 795 may be reduced substantially.

In one embodiment, the NIC 795 may comprise an interface 791, a hardware filter policy 792, a routing policy 793, a control logic 798, and virtual functions VF 794-A to 794-M and a physical function PF 799. In other embodiment, the VFs 794-A to 794-M and PF 799 may be provisioned outside the NIC 795 as well. In one embodiment, the NIC 795 may send control information to the fast path assist hardware 770 to enable the fast past assist hardware 770 to use the VFs 794-A to 794-M and PF 799 if the VFs 794-A to 794- M and PF 799 are provisioned within the NIC 795. In other embodiment, the NIC 795 and fast path assist hardware 770 may access the VFs 794-A to 794-M and PF 799 if the VFs 794-A to 794-M and PF 799 are provisioned outside the NIC 795. In one embodiment, the interface 791 may receive packets from the network 150 that may be of broadcast type or the one that matches one of the L2 addresses of the VMs 210-A to 210-N. In one embodiment, the control logic 798 may use the hardware filter policy 792 to determine if the packet header matches with the filter policy and the control logic 798 may determine routing information using the routing policy 793. In one embodiment, the routing information may then be stored in the cache 752 provided in the fast path assist hardware 770. In one embodiment, the routing information may provide an identifier of the VINA 720-A (for example) to which the packet may be transferred. In one embodiment, the control logic 798 may determine that the VF 794-A may be coupled to VINA 720-A and may transfer the packet to the inward VINA queue 781 of the VINA packet memory (PM) 784. In one embodiment, the control logic 798 may then signal the VINA 720-A, which may retrieve the packet from the inward VINA queue 781 through the VF 794-A. However, the inter- VM communication (i.e., the next hop, VINA-720-K, for example, for the packet that is processed by the VINA-A) traffic may not reach the NIC 795 and such inter- VM communication traffic may be handled by the fast path assist hardware 770 that is described below.

In one embodiment, the VM 710-A to 710-N may each comprise a guest operating system OS 715-A to 715-N, respectively. In one embodiment, the VM 710-A to 710-N may each comprise a VF driver 716-A to 716-N, respectively. In one embodiment, the VF drivers 716-A to 716-N may couple the VM 710-A to 710-N with the virtual function 794- A to 794-M to provide a path between the NIC 795 and the VMs 710-A to 710-N and between the fast path assist hardware 770 and the VMs 710-A to 710-N. In one embodiment, the VINAs 720-A to 720-K may each comprise a slow path 721 -A to 721-K and a fast path 722-A to 722-K, respectively. In one embodiment, the slow path 721 and fast path 722 may be used for coupling the VINAs 720-A to 720-K to the NIC 795 using VFs 794-A to 794-M and to fast path assist hardware 770 using the using VFs 794-A to 794-M. In one embodiment, while the CP 740 supports inter- VM communications, the VMs 710-A to 710-N and VINAs 720-A to 720-K may send packets to NIC 795, however, such packets may be actually routed to (or sent) to the fast path assist hardware 770 instead of the NIC 795. In one embodiment, the fast path assist hardware 770 may be provisioned close to DMA engine 791 (i.e., to the north of interconnect 769). In one embodiment, the fast path assist hardware 770 may receive packets while performing inter- VM communications and perform packet processing tasks such as encryption, decryption, checksum, CRC stripping/adding and other computationally intensive tasks. As a result of the fast path assist hardware 770 performing such computationally intensive packet processing tasks before routing the packets to the next VINA 720 or VM 710, the packets that otherwise get transferred to the NIC 795 over the interconnect 769 may be reduced substantially.

In one embodiment, the fast path assist hardware 770 may determine the next

VINA 720 or the next VM in the path for using routing information stored in the cache 752. In one embodiment, the fast path assist hardware 770 may comprise hardware logic 774 to retrieve routing information and route packets to a next VINA 720 or VM in the inter- VM communication path and to perform computationally intensive packet processing tasks such as encryption and decryption.

In one embodiment, the hardware logic 774 may check the routing information stored in the cache 752 to determine the VINA 720 to which the packet may be redirected before sending the packet to the destination or final VM. In one embodiment, the hardware logic 774 may use the routing information stored in the cache 752 without having to perform the computationally intensive filtering task repeatedly. In one embodiment, the hardware logic 774 may use the routing information to determine or retrieve a bit field and a VINA id field that may correspond to the intermediate VM identified by the hardware logic 774 after performing packet filtering. In one embodiment, the bit field (equaling logic 0 or 1) may indicate whether the routing information is found and the VINA id field may include an identifier of a destination VINA to which the packet may be sent. In one embodiment, if the bit field is logic high (or logic 1), the hardware logic 774 may identify one of the virtual functions 794-A to 794-M that may be associated with the destination VINA identified by the entry in the VINA id and the hardware logic 774 may cause the packet to be sent to the identified virtual function. If the bit field is logic low or logic 0, the hardware logic 774 may cause the packet to be sent to the intermediate VM (one of 710- A to 710-N) identified by the hardware logic 774.

In one embodiment, the hardware logic 774 may determine the VINA queue id in the pool of VINA queues assigned to VINAs 720-A to 720-K in response to receiving a logic high on the bit field. In one embodiment, the pool of VINA queues may be represented as VINA packet memory VINA PM 784, which may be provisioned in the memory 780. In one embodiment, the hardware logic 774 may use the VINA queue id to identify the VINA queue within the pool of VINA queues VINA PM 784. In one embodiment, the hardware logic 774 may cause the packet to be stored in an inward VINA queue 781 identified by the VINA queue id within the VINA PM 784. In one embodiment, the hardware logic 774 may signal the identified VINA 720 using VF 794 to inform the availability of the packet in the inward VINA queue 781. In one embodiment, the identified VINA 720 may retrieve the packet stored in the inward VINA queue 781. After processing the packet, the VINA 720 may store the packet in an outward VINA queue 782 of the VINA PM 784. In one embodiment, the VINA 720 may send a signal to the hardware logic 774 through the associated VF 794. In one embodiment, the hardware logic 774 may retrieve the processed packet from the outward VINA queue 782 while receiving the packet from the VINA 720. In one embodiment, the hardware logic 774 may determine the next VINA 720 or VM 710 in the inter- VM communication path and store the packet in an inward VINA queue 781 or inward VM queue 786, which may be associated with the next VINA 720 or the VM 710. In one embodiment, the hardware logic 774 may use a DMA engine 791 to transfer packets from the outward VINA queue 782 to the inward queue 781 and to transfer the packets from the outward VM queue 782 to the inward VM queue 786. In one embodiment, the hardware logic 774 may repeat the above process until the packet may be sent to all the VINAs 720 and the destination VM 710 in the inter- VM communication path.

In one embodiment, the hardware logic 774 may determine a next VM after sending the packet to all the VINAs 720 in an identified path. In one embodiment, the hardware logic 774 may determine the next VM using the routing policy 793. In one embodiment, the hardware logic 774 may determine a VM field id that may identify the next VM to which the packet may be sent. In one embodiment, the VM id field may include an identifier of the next VM to which the packet may be sent if the packet header matches with the routing policy 793. In one embodiment, the pool of VM queues may be represented as VM PM 785, which may be provisioned in the memory 780. In one embodiment, the hardware logic 774 may use VM queue id to identify the VM queue within the pool of VM queues VM PM 785. In one embodiment, the hardware logic 774 may cause the packet to be stored in an inward VM queue 786 within the VM PM 785 identified by the VM queue id. In one embodiment, the hardware logic 774 may signal the identified VM 710 through one of the VFs 794 associated with the identified VM 710 to inform the availability of the packet in the inward VM queue 786 of the VM PM 785.

In one embodiment, the VM 710 may retrieve the packet from the inward queue 786, process the packet, and store the packet into an outward queue 787 of the VM PM 785. In one embodiment, the VM 710 may signal the hardware logic 774 to inform the availability of the packet in the outward queue 787. In one embodiment, the hardware logic 774 may receive the signal from the VM 710 through the associated VF 794 after the VM completes storing the packet in the outward VM queue 787 of the VM PM 785. In one embodiment, the hardware logic 774 may retrieve the packet from the outward VM queue 787 while receiving a packet and may transmit the packet to a next hop network device, for example. In one embodiment, the hardware logic 774 may use a DMA engine 791 to transfer packets from the outward VM queue 787 to the inward queue 786 while performing inter- VM communication and to transfer the packets from the outward queue 787 to the NIC 795 while transferring the packets to an external network.

An embodiment of an operation of the MSS 110, which may support hardware assisted inter-VM communication without consuming computational resources is illustrated in FIG. 8. In block 810, a fast path hardware component such as the fast path assist hardware 770 may be provisioned in a single route I/O virtualized (SR-IOV) NIC 795. In block 850, the SR-IOV NIC 795 may use the fast path hardware component to determine the inter-VM routing order, which may indicate the order in which a packet may be sent to VMs 710 or VINAs 720.

An embodiment of an operation of the SR-IOV NIC 795 in performing a VINA fast path routing policy look-up is illustrated in a flow-chart of FIG. 9. In block 910, the interface 791 may receive a packet either from a network.

In block 920, the control logic 798 of the NIC 795 may check the packet header for a match with the filter using a fast path filtering logic provisioned on the fast path hardware component. In one embodiment, the control logic 798 may match the packet information with the filter using H/W filtering policy 792. In block 930, the control logic 798 may check whether the packet information matches with the filter and control passes to block 940 if a filter match is not positive and to block 950 if the filter match is positive.

In block 940, the control logic 798 may set the destination VM identifier (id) to a default VM. In block 950, the control logic 798 may determine the destination VM id for the packet as described above. In one embodiment, the control logic 798 may use the H/W filter policy 792 to determine the destination VM id for the packet.

In block 960, the control logic 798 may check if the packet information matches with the routing policy 793 and control passes to block 965 if the packet information does not match with the routing policy 793 and to block 970 if the packet information matches with the routing policy 793. In block 965, the control logic 798 may send the packet to a VM identified by the destination VM id.

In block 970, the control logic 798 may determine a new destination VM or VINA id based on the routing policy 793 as described above. In block 975, the control logic 798 may check whether the fast path assist hardware 770 is to be used to route the packet based on either an original ordered sequence (determined before checking a match for routing policy) or a redirected ordered sequence. Control passes to block 980 if the fast past assist hardware 770 is to be used and to block 990 otherwise.

In block 980, the fast path assist hardware 770 may be used to continue to determine the next hop (i.e., the new destination VM or VINA) and send the packet to the next hop in the inter-VM communication path. In block 990, the control logic 798 may send the packet to the new destination VM or VINA in a redirected ordered sequence. In one embodiment, the new destination VM or VINA may not be the same as the destination VM determined before the match for routing policy 793 is performed. In one embodiment, the control logic 798 may store the routing information determined in block 960 in the cache 752 and may store the packet in the inward VINA queue 781 or the inward VM queue 786 and signal a corresponding VINA 720 or VM 710 as described above.

An embodiment of an operation of the MSS 110, which may support hardware assisted inter-VM communication is illustrated in FIG. 10. In block 1005, the NIC 795 may assign the virtual functions VF 794-A to 794-M to VMs 710-A to 710-N and the VINAs 720-A to 720-K.

In block 1010, the VINAs 720 may subscribe to packet flows by programming the NIC 795 via a hardware interface. In one embodiment, the subscription made by the VINAs 720 may be used by the NIC 795 to generate a filter policy.

In block 1020, the NIC 795 may receive a packet either from an external network, or VMs 710, or VINAs 720. In block 1030, the NIC 795 may check whether the packet is received from an external network and control passes to block 1035 if the packet is received from an external network and to block 1065 if the packet is received from one of the VMs 710 or VINAs 720. In one embodiment, if the packet is received from the VM 710 or VINA 720, the NIC 795 may handover the control to the fast path assist hardware 770 and the fast path assist hardware 770 may retrieve the packet stored in the outward VINA queue 782 or the outward VM queue 787. In one embodiment, the VM 710 or the VINA 720 may generate a signal to inform the NIC 795 after storing the processed packet in the outward queue 782 or 787. However, the signal generated by the VM 710 or VINA 720 may be handled by the fast path assist hardware 770 and the VM 710 and VINA 720 may not have information of the handover of control from the NIC 795 to the fast path assist hardware 770. In one embodiment, the handover of control from NIC 795 to fast path assist hardware 770 may be seamless and the VM 710 and VINA 720 may continue to operate without any changes.

In one embodiment, the block 1040 may be reached after the NIC 795 hands over the control to fast path assist hardware 770. In block 1040, the fast path assist hardware 770 may capture the source VM pool id (or VINA pool id) in the pseudo header of the packet. In one embodiment, the source VM or VINA pool id in the pseudo header of the packet may be used to determine a present position of the packet in a chain of VINAs in the routing order. For example, the routing information may indicate that the routing order or sequence through which the packet may traverse equals (VINA 720-A, VINA 720-C, and VINA 720-F). In one embodiment, the fast path assist hardware 770 may first send the packet to VINA 720-A and receive a processed packet back from VINA 720-A. In one embodiment, the fast path assist hardware 770 may use the source pool id value in the processed packet to determine that the processed packet was received from the VINA- A and the next hop in the routing order is VINA-C and not VINA-F. Such usage of source pool id may repeat while determining the next hop in the routing order.

In one embodiment, the fast path assist hardware 770 may use the source VM or VINA id in the pseudo header of the packet as a tag to locate the previously cached routing information. Such an approach may avoid the computationally intensive task of filter look-up that may have to be performed while receiving the packet for the first time.

In block 1050, the hardware logic 774 may use the source VM/VINA id to locate the routing information stored in the cache 752.

In block 1070, the hardware logic 774 may retrieve the routing information as described above. In one embodiment, the hardware logic 774 may use the routing information to determine a next VM or the VINA to which the packet may be sent. In one embodiment, the routing information may include an ordered list of VM ids or VINA ids to which the packet may be sent in that order.

In block 1080, the hardware logic 774 may determine the next VINA or next VM pool to store the packet based on the routing information and the source VM pool id captured in block 1050. In one embodiment, if there is a match in the routing information, the packet may be redirected to a next VM or next VINA other than the destination VM or VINA.

In block 1085, the hardware logic 774 may store the packet in a next inward VM or a next inward VINA queue associated with the next VM or next VINA. In one embodiment, the hardware logic 774 may store the packet in the inward VINA queue 781 if the routing information identifies a VINA as the next hop or in the inward VM queue 786 if the routing information identifies a VM as the next hop. In one embodiment, the next inward VINA or VM queue may be identified using the VINA or VM pool id in the routing information. In block 1090, the hardware logic 774 may send a signal to the next VM or the next VINA indicating the availability of the packet in the inward VM queue 786 or the inward VINA queue 781.

An embodiment of a source MSS 110-A and a destination MSS 110-K, which together may support hardware assisted migration of a virtual machine from the source MSS 110-A to the destination MSS 110-K is illustrated in FIG. 11. In one embodiment, the arrangement 1100 may comprise a source MSS 110-A and a destination MSS 110-K, and a remote memory store 1190. In one embodiment, the source MSS 110-A may comprise virtual machines (VM) 1110-A, 1110-B, a VINA 1120, a VMM 1130, and a computer platform CP 1140. In one embodiment, the CP 1140 may comprise a processor 1142, a memory 1143, logic 1144, a NIC 1145, a flash memory 1147, and a cache 1148. In one embodiment, the destination MSS 110-K may comprise a VM 1150-A, a VINA 1160, a VMM 1170, and a CP 1180. In one embodiment, the CP 1180 may comprise a processor 1182, a memory 1183, logic 1184, a NIC 1185, a flash memory 1187, and a cache 1188.

In one embodiment, the MSS 110-A and MSS 110-K may support hardware assisted migration to allow migration of the VM 1110-A from the MSS 110-A to MSS 110-K. In one embodiment, the migration of the VM 1110-A to the MSS 110-K may be symbolically denoted by a dotted line. In one embodiment, the flash memory 1147 and 1187 may support firmware layers that enable hardware assisted VM migration. In one embodiment, the firmware layer 1149 and 1189 may, respectively, monitor the processor events pertaining to hardware managed page tables of the CP 1140 and CP 1180. In one embodiment, the firmware layers 1149 and 1189 may, respectively, include VM migration services 1149-A and 1 189-A, which may be exposed to the VMMs 1 130 and 1 170. In one embodiment, the firmware layer 1 149 may use the NIC 1 145 to autonomously send traffic to destination server such as VMs 1 110 and VINA 1 120 and to the remote memory store 1 190. In one embodiment, the remote memory store 1 190 may serve as an intermediate memory store for the VM 1 1 10-A, which may be migrated to the MSS 110-K.

In one embodiment, as a part of the set-up, the CP 1140 may assign the hardware managed page tables such as extended page tables (EPT) to the firmware layer 1149. In one embodiment, the CP 1 140 may not expose the hardware managed page tables to the VMM 1130. In one embodiment, the CP 1 140 may not expose the hardware managed page tables to the VMM 1130 by virtualizing the processor identifier or cpu id, for example. In other embodiment, a firmware managed hardware page table may be used to support VM migration if the hardware managed page tables are exposed to the VMM 1 130.

In one embodiment, to initiate VM migration, the VMM 1 130 may send a migration ready signal to a VM migration service 1 149-A with a request to migrate context of the VM 1 110-A identified by the area of guest physical address (GPA) used by the VM 1 110-A. In one embodiment, the VMM 1 130 may identify the destination MSS (i.e., MSS 110-K, here) as well. In one embodiment, in response to receiving the migration ready signal, the VM migration services 1 149-A may confirm the participation of the destination MSS 110-K in the VM migration. In one embodiment, for the pages in the guest page address (GPA) range, the VM migration services 1149-A may mark the reserved bits on non-dirty hardware managed page entries and may also mark the page entries as 'not-present'. In one embodiment, the VM migration services 1 149-A may move the physical contents of the memory page referenced by the reserved bits on the non-dirty hardware page entries and the page entries to the remote memory store 1190.

In one embodiment, as the physical contents of the page are moved to the remote memory store 1 190, the VM migration services 1 149-A may update the hardware managed page entries (or EPT entries) for the page to include the address of the remote memory store 1 190 while maintaining the 'not-present' status for the page. In one embodiment, the VM migration services 1 149-A may configure the other accessed hardware managed page entries such that accesses to specific guest physical range may result in page fault and a callback may be made to the VM migration services 1149-A. In one embodiment, if the processor 1142 accesses the hardware managed page entries for a 'not-present' page, the VM migration services 1 149-A may get a callback. In one embodiment, in response to getting a callback, the VM migration services 1149-A may return the remote memory page contents to the cache 1148. In one embodiment, the processor 1 142 may retrieve the contents of the remote memory page from the cache 1 148. Also, in one embodiment, the processor 1142 may update the contents of the remote memory page in the cache 1148 and such updates to the contents to the remote memory page in the cache 1148 may be written back to the remote memory store 1 190.

In one embodiment, the VM migration services 1 189-A, on the destination MSS 1 10-K, may create a destination VM context, which may map the source VM guest physical address (GPA) to a destination VM GPA on the destination MSS 1 10-K. In one embodiment, the VM migration services 1 189-A may invoke the VMM 1 170 such that the VMM 1 170 may initiate preparation for the new VM context in the VMM 1 170. In one embodiment, for each page of the VM 11 10-A transferred to remote memory store 1 190, the firmware layer 1149 on the source MSS 1 10-A may send a control signal to the destination MSS 1 10-K with the context information for such pages. In one embodiment, the VM migration services 1 189-A provided in the CP 1 180 may create a hardware managed page context for the new VM 1 110-A' and the context of such pages may be marked as 'not-present'.

In one embodiment, the VM migration services 1189-A may start to map the hardware managed page entries for the new VM 11 10-A' to refer to the remote memory pages stored in the remote memory store 1190. In one embodiment, the hardware managed page entries may be marked as 'not-present'. As the pages are moved from the remote memory store 1190 to the memory 1183, the address reference in the hardware managed page entries may be updated and the page may be marked as 'present'. In one embodiment, the VM migration services 1189-A may be invoked by a hardware managed page event if the page is accessed by the new VM 1 110-A' before the page is moved into the memory 1183. In one embodiment, the VM migration services 1189-A may retrieve the contents of the pages stored in the memory 1 183 out of order and may store the contents of the pages into the cache 1188 to serve the request. In one embodiment, the VM migration services 1 189-A may retrieve most recently accessed pages first. In one embodiment, the pages of the VM 1 110-A may be moved to the destination MSS 1 10-K and an acknowledgement may be sent out to indicate completion of the VM migration.

An embodiment of an operation of the source MSS 1 10-A, which may support hardware assisted migration of a virtual machine from the source to the destination MSS 1 10-B is illustrated in flow-chart of FIG. 12. In block 1210, the VMM 1 130 on the source MSS 110-A may send a migration ready signal to a VM migration service 1149-A with a request to migrate a first VM (i.e., context of the VM 1 110-A, for example) identified by the GPA of the first VM. In block 1220, the VMM 1 130 may identify the destination MSS (i.e., MSS 110-K,) as well. In one embodiment, the VMM 1 130 may contact the VM migration services 1 149-A with a destination identify request to identify a destination MSS, which may participate in the VM migration.

In block 1230, in response to receiving the destination identify request, the VM migration services 1149-A may confirm whether the destination MSS 1 10-K may participate in the VM migration. Control passes to block 1240 in response to receiving a participation confirmation of the destination MSS 110-K and the search for a participating destination MSS may continue otherwise. In block 1240, for the memory pages in the guest page address (GPA) range, the VM migration services 1149-A may mark the reserved bits on non-dirty hardware managed page entries and may also mark the page entries as 'not-present'.

In one embodiment, the GPA range table 1300 is depicted in FIG. 13. In one embodiment, the table 1300 may comprise three columns - VM id 1310, page entries 1312, and reserved bits RB 1316 and rows 1320-A to 1320-L. In one embodiment, the first row 1320-A may comprise VM 11 10-A in column VM id 1310, a first page field (PF) equaling Al and an associated status field (SF) equaling NP, a second page field equaling A2 and an associated SF equaling NP, ... Nth page field equaling An and an associated SF equaling NP. In one embodiment, the VM 11 10-A may be migrated from the source MSS 1 10-A to MSS 110-K. As a result, the pages Al, A2,...An of the VM 11 10-A may be identified with a status of 'not present (NP)' until the migration is complete and the status of the pages Al to An of the VM 11 10-A may change to 'present (P)' after the migration is complete. However, the VM 1 110-B and 1 110-L may not be migrated and the pages of the VM 1 110-B and 1 110-L may be indicated as 'present' as depicted in rows 1320-B to 1320-L.

In block 1250, the VM migration services 1149-A may move the physical contents of the memory page referenced by the page reference (i.e., reserved bits on the non-dirty hardware page entries) and the page entries to the remote memory store 1190. In block 1260, as the physical contents of the page are moved to the remote memory store 1190, the VM migration services 1 149-A may update the hardware managed page entries (or EPT entries) for the page to include the address of the remote memory store 1190. In one embodiment, the page status may continue to be 'not-present (NP)' for the pages.

In block 1265, the VM migration services 1 149-A may configure the other accessed hardware managed page entries such that accesses to specific guest physical range may result in page fault and a callback may be made to the VM migration services 1 149-A. In block 1270, the VM migration services 1 149-A may check if the processor 1 142 may access the hardware managed page entries for a 'not-present' page and control passes to block 1275 if the processor 1 142 accesses the hardware managed page entries for a 'NP' page and to block 1290 otherwise.

In block 1275, in response to getting a callback, the VM migration services 1149-A may return the remote memory page contents to the local cache 1148. In one embodiment, the processor 1142 may retrieve the contents of the remote memory page from the cache 1 148. Also, in one embodiment, the processor 1 142 may update the contents of the remote memory page in the cache 1 148.

In block 1280, the VM migration services 1149-A may write back to the remote memory store 1 190 with the updated contents in the cache 1148. In block 1290, the VMM 1 130 may receive an acknowledgment from the destination MSS 1 10-K that may indicate the completion of hardware assisted VM migration.

An embodiment of an operation of the destination MSS 1 10-K, which may support hardware assisted migration of a virtual machine to the MSS 110-K is illustrated in FIG. 14. In block 1410, the VMM 1170, on the destination MSS 1 10-K, may send a request to a second migration service such as the VM migration services 1 189-A to create a destination VM context, which may map the guest physical address (GPA) of the source VM 11 10-A to a GPA of the destination VM 1 110-A' on the destination MSS 1 10-K.

In block 1420, the VM migration services 1 189-A may invoke the VMM 1170 such that the VMM 1 170 may initiate preparation for the new VM context in the VMM 1 170. In block 1430, for each page of the VM 1 110-A that may be transferred to remote memory store 1190 by the MSS 1 10-A, the VM migration services 1189-A may receive a control signal from the VM migration services 1149-A with the context information for such transferred pages.

In block 1440, the VM migration services 1 189-A may create a hardware managed page context for the new VM 11 10-A' and the context of such pages may be marked as 'not-present' as illustrated in FIG. 13.

In block 1450, the VM migration services 1 189-A may start to map the hardware managed page entries for the new VM 11 10-A' to refer to the remote memory pages stored in the remote memory store 1 190. In one embodiment, the hardware managed page entries may continue to be marked as 'not-present'.

In block 1460, the VM migration services 1189-A may move the pages from the remote memory store 1 190 to the local memory 1183 in the destination MSS 1 10-K. In block 1465, the VM migration services 1189-A may update the address reference in the hardware managed page entries and mark the page as 'present' (P).

In block 1470, the VM migration services 1189-A may check if the page is accessed by the new VM 11 10-A' before the page is moved into the local memory 1 183 and control passes to block 1475 if the page is accessed by the new VM 1110-A' and to block 1490 otherwise.

In block 1475, the VM migration services 1 189-A may retrieve the contents of the pages stored in the memory 1 183 out of order and may store the contents of the pages into the cache 1 188 to serve the request. In block 1 1480, the VM migration services 1189-A may retrieve most recently accessed pages first.

In block 1490, the VM migration services 1 189-A may check if more pages of the VM 1 110-A are available in the remote memory store to be moved to the local memory 1 183 of the destination MSS 1 10-K and control passes to block 1465 if more pages are available and to block 1495 if no more pages are available for transfer. In block 1495, the VM migration services 1 189-A may send an acknowledgement to indicate the completion of the VM migration.

Certain features of the invention have been described with reference to example embodiments. However, the description is not intended to be construed in a limiting sense. Various modifications of the example embodiments, as well as other embodiments of the invention, which are apparent to persons skilled in the art to which the invention pertains are deemed to lie within the spirit and scope of the invention.

Claims

What is claimed is:

1. A computer platform comprising:

a plurality of hardware components including a processor, a memory coupled to the processor, and a network interface card coupled to the processor and the memory, wherein the hardware components are to support a plurality of virtual machines and a plurality of virtual inline network appliances,

wherein the network interface card is to,

receive a packet from the interface and match the packet with a match filter, wherein the match filter is a consolidation of filter specifications provided by the plurality of virtual inline network appliances,

insert a tag within a layer-2 header of the packet, wherein the tag is to include a routing action generated using filter specifications, wherein the routing action includes an ordered sequence of virtual appliance identifiers,

route the packet to a first appliance of the plurality of virtual inline network appliances, wherein the first appliance is identified by a first identifier an ordered sequence of appliance identifiers within the routing action.

2. The computer platform of claim 1 the hardware components further comprise a fast path assist hardware, wherein the fast path assist hardware is to redirect the packet to a destination appliance of the plurality of virtual inline network appliances, wherein the destination appliance is different from an intermediate virtual machine identified during filter matching if a match between the packet and a routing policy is found, wherein the routing policy is stored in a cache by the network interface card, wherein the fast path assist hardware is provisioned in close proximity to the memory to avoid transfer of packets over an interconnect coupling the memory to the network interface card.

3. The computer platform of claim 1, wherein the network interface card is to support hardware assisted VM migration to migrate a first virtual machine of the plurality of virtual machines to a destination computer platform.

4. The computer platform of claim 1, wherein the network interface card is to identify a first virtual function associated with the first appliance before routing the packet to the first appliance through the first virtual function.

5. The computer platform of claim 1, wherein the network interface card is to identify a next appliance to which the packet is to be forwarded in response to receiving a processed packet from the first appliance.

6. The computer platform of claim 5, wherein the network interface card is to identify the next appliance by processing the layer-2 information of the processed packet.

7. The computer platform of claim 6, wherein the network interface card is to identify a next virtual function associated with the next appliance before sending the packet to the next appliance through the next virtual function.

8. The computer platform of claim 1, wherein the filter specifications provided by the plurality of virtual inline network appliance represent the subscriptions to desired packet flows.

9. The computer platform of claim 4, wherein the network interface card is to identify the first appliance as a sending source of the processed packet and reset the first appliance identifier in the ordered sequence of appliance identifiers in the routing action.

10. The computer platform of claim 6, wherein the network interface card is to generate a mapping table, wherein the mapping table is to provide an association between the first appliance and the first virtual function and the next appliance and the next virtual function.

1 1. The computer platform of claim 2 the fast path assist hardware further comprises a hardware logic,

wherein the hardware logic is to match the packet with the routing policy stored in a cache to determine an intermediate appliance to which the packet is to be sent, and

wherein the hardware logic is to generate a bit field and an identifier field, wherein the bit field is set to a first logic level and the identifier field includes an identifier of the intermediate appliance if the match is successful.

12. The computer platform of claim 11, wherein the hardware logic is to, store the packet in an inward appliance queue in the memory in response to receiving the identifier of the intermediate appliance, and

send a first available signal to the intermediate appliance, wherein the first available signal is to indicate the availability of the packet in the inward appliance queue, wherein the hardware logic is to perform packet processing tasks including encryption, decryption, checksum, cyclic redundancy code stripping and appending before sending the first available signal to the intermediate appliance.

13. The computer platform of claim 12, wherein hardware logic is to,

match the packet with the routing policy to determine the destination virtual machine to which the packet is to be sent after sending the packet to intermediate appliances listed in the routing policy, and

generate a bit field and an identifier field, wherein the bit field is set to a first logic level and the identifier field includes an identifier of the destination virtual machine if the match is successful.

14. The computer platform of claim 13, wherein the hardware logic is to, store the packet in an inward virtual machine queue in response to receiving the identifier of the destination virtual machine, and

send a second available signal to the destination virtual machine, wherein the second available signal is to indicate the availability of the packet in the inward virtual machine queue,

wherein the hardware logic is to perform packet processing tasks including encryption, decryption, check sum, cyclic redundancy check stripping and appending before sending the second available signal to the destination virtual machine.

15. The computer platform of claim 14 further comprises a direct memory access engine coupled to the fast path assist hardware, wherein the hardware logic is to transfer the packet to the inward appliance queue and the inward virtual machine queue using the direct memory access engine.

16. The computer platform of claim 1 further comprises,

a virtual machine monitor supported by the processor, wherein the virtual machine monitor is to support the plurality of virtual machines and the plurality of virtual inline network appliances, and

a flash memory coupled to the virtual machine monitor, wherein the flash memory is to support a hardware assisted virtual machine migration service,

wherein the virtual machine monitor is to send a request to the hardware assisted virtual machine migration service, wherein the request is to migrate the first virtual machine to a destination computer platform, wherein memory pages of the first virtual machine is identified by an area of guest physical address.

17. The computer platform of claim 16, wherein the hardware assisted virtual machine migration service is to,

mark page entries of the memory pages as 'not present',

move the memory pages to a remote memory store, and

update page entries of the memory pages moved to the remote memory with the address of the remote memory store while maintaining the 'not present' status.

18. The computer platform of claim 17, wherein the hardware assisted virtual machine migration service is to configure other page references of the first virtual machine to cause a call back to the hardware assisted virtual machine migration service if the processor accesses the other page references of the first virtual machine.

19. The computer platform of claim 17, wherein the hardware assisted virtual machine migration service is to,

return the memory pages from the remote memory store to a cache if the processor accesses the memory pages, and

write back updates to the memory pages in the cache to the remote memory store.

20. The computer platform of claim 17, wherein the virtual machine monitor is to receive an acknowledgement from the destination platform after the first virtual machine migration is complete.

21. A method in a virtualized computer system comprising:

receiving a packet from a interface,

matching the packet with a match filter, wherein the match filter is a consolidation of filter specifications provided by a plurality of virtual inline network appliances,

inserting a tag within a layer-2 header of the packet, wherein the tag is to include a routing action generated using filter specifications, wherein the routing action includes an ordered sequence of virtual appliance identifiers, and

routing the packet to a first appliance of the plurality of virtual inline network appliances, wherein the first appliance is identified by a first identifier an ordered sequence of appliance identifiers within the routing action

wherein a plurality of virtual machines and the plurality of virtual inline network appliances are supported on hardware components of the computer platform.

22. The method of claim 21 comprises redirecting the packet to an intermediate appliance of the plurality of virtual inline network appliances, wherein the intermediate appliance is different from a destination virtual machine identified during filter matching if a match between the packet and a routing policy is found,

wherein redirecting the packet to the intermediate appliance is performed by a fast path assist hardware is provisioned in close proximity to a processor to avoid transfer of packets over an interconnect coupling the processor to the network interface card,

wherein the routing policy is stored in a cache by the network interface card.

23. The method of claim 21 comprises supporting hardware assisted VM migration to migrate a first virtual machine of the plurality of virtual machines to a destination computer platform.

24. The method of claim 21 comprises identifying a first virtual function associated with the first appliance before routing the packet to the first appliance through the first virtual function.

25. The method of claim 21 comprises identifying a next appliance to which the packet is to be forwarded in response to receiving a processed packet from the first appliance.

26. The method of claim 25 comprises identifying the next appliance by processing the layer-2 information of the processed packet.

27. The method of claim 26 comprises identifying a next virtual function associated with the next appliance before sending the packet to the next appliance through the next virtual function.

28. The method of claim 21, wherein the filter specifications provided by the plurality of virtual inline network appliance represent the subscriptions to desired packet flows.

29. The method of claim 24 comprises identifying the first appliance as a sending source of the processed packet and reset the first appliance identifier in the ordered sequence of appliance identifiers in the routing action.

30. The method of claim 26 comprises generating a mapping table, wherein the mapping table is to provide an association between the first appliance and the first virtual function and the next appliance and the next virtual function.

31. The method of claim 22 further comprises,

matching the packet with a routing policy stored in a cache, wherein the matching of the packet is performed using a hardware logic to determine the intermediate appliance to which the packet is to be sent, and

generating a bit field and an identifier field, wherein the bit field is set to a first logic level and the identifier field includes an identifier of the intermediate appliance if the match is successful.

32. The method of claim 31 comprises,

storing the packet in an inward appliance queue in memory in response to receiving the identifier of the intermediate appliance, and

sending a first available signal to the intermediate appliance, wherein the first available signal is to indicate the availability of the packet in the inward appliance queue, wherein packet processing tasks including encryption is performed in the hardware logic before sending the first available signal to the intermediate appliance.

33. The method of claim 32 comprises,

matching the packet with the routing policy in the hardware logic to determine a destination virtual machine to which the packet is to be sent after sending the packet to intermediate appliances listed in the routing policy, and

generating a bit field and an identifier field, wherein the bit field is set to a first logic level and the identifier field includes an identifier of the destination virtual machine.

34. The method of claim 33 comprises,

storing the packet in an inward virtual machine queue in response to receiving the identifier of the destination virtual machine, and

sending a second available signal to the destination virtual machine, wherein the second available signal is to indicate the availability of the packet in the inward virtual machine queue

wherein packet processing tasks including encryption is performed in the hardware logic before sending the second available signal to the destination virtual machine.

35. The method of claim 34 further comprises transferring the packet to the inward appliance queue and the inward virtual machine queue using a direct memory access engine.

36. The method of claim 21 comprises,

provisioning a virtual machine monitor between the hardware components and the plurality of virtual machines and the plurality of virtual inline network appliances,

provisioning a hardware assisted virtual machine migration service on a flash memory, and

sending a request to the a hardware assisted virtual machine migration service from the virtual machine monitor, wherein the request is to migrate the first virtual machine to a destination computer platform, wherein memory pages of the first virtual machine is identified by an area of guest physical address.

37. The method of claim 36 comprises,

marking page entries of the memory pages as 'not present',

moving the memory pages to a remote memory store, and

updating the page entries of the memory pages moved to the remote memory with the address of the remote memory store while maintaining the 'not present' status.

38. The method of claim 37 comprises configuring other page references of the first virtual machine to cause a call back to the hardware assisted virtual machine migration service if the processor accesses the other page references of the first virtual machine.

39. The method of claim 37 comprises,

returning the memory pages from the remote memory store to a cache if the processor accesses the memory pages, and

writing back updates to the memory pages in the cache to the remote memory store.

40. The memory of claim 37 comprises receiving an acknowledgement from the destination platform after the first virtual machine migration is complete.