CN118414606A - Machine Learning Using Serverless Computing Architecture - Google Patents

Abstract

A serverless computing system is configured to provide access to a machine learning model by associating at least an endpoint including code to access the machine learning model with an extension that interfaces between a serverless computing architecture and the endpoint. A request to perform an inference is received by the system and processed by executing a compute function using the serverless computing architecture. The compute function causes the extension to interface with the endpoint to cause the machine learning model to perform the inference.

Description

Machine Learning Using Serverless Computing Architecture

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Indian patent application 202111054927, named “Liberty 1-IN”, filed on November 27, 2021, U.S. patent application 17/710,853, named “MACHINE LEARNING USING SERVERLESSCOMPUTE ARCHITECTURE”, filed on March 31, 2022, Indian patent application 202111054942, named “Liberty 2-IN”, filed on November 27, 2021, and U.S. patent application 17/710,864, named “MACHINELEARNING USING A HYBRID SERVERLESS COMPUTE ARCHITECTURE”, filed on March 31, 2022.

Background technique

Machine learning techniques are increasingly being used in a variety of industries. However, these techniques can be difficult to maintain and manage. The development and use of machine learning models can require significant computing resources, such as storage and processing time. These resources can be difficult to obtain and manage, which can present a significant barrier to the adoption of machine learning techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

Various techniques will be described with reference to the accompanying drawings, in which:

FIG1 illustrates a system for performing machine learning inference including a serverless computing architecture according to at least one embodiment;

FIG2 illustrates an example process for enabling a serverless computing architecture to perform machine learning inference according to at least one embodiment;

FIG3 illustrates an example of calling and executing a compute function to perform machine learning inference according to at least one embodiment;

4 illustrates an example of a hybrid system combining serverless processing and server-full processing with machine learning inference according to at least one embodiment;

5 illustrates an example process for configuring a serverless computing architecture to perform machine learning inference according to at least one embodiment;

6 illustrates an example process for configuring a hybrid computing architecture to perform machine learning inference according to at least one embodiment;

FIG7 illustrates an example process for performing machine learning inference using a serverless computing architecture according to at least one embodiment;

FIG8 illustrates an example process for performing machine learning inference using a hybrid computing architecture according to at least one embodiment; and

FIG. 9 illustrates a system in which various embodiments may be implemented.

Detailed ways

In an example, the system utilizes a serverless computing architecture to generate inferences using a machine learning model. A serverless computing architecture (which may also be referred to as a serverless computing system or a serverless computing subsystem) includes hardware and software that dynamically provides computing resources to perform computing functions. Access to the machine learning model is facilitated using a model server, which, while available on a dedicated server, can also utilize a serverless computing architecture by employing the techniques described herein.

In a server-based application, users of a machine learning service can create and train machine learning models hosted by the service. To use a hosted machine learning model or other computing service, a customer is assigned a dedicated server instance on which a model server is installed and activated. A model server is a unit of code that implements a Hypertext Transfer Protocol (“HTTP”) server that listens for requests to obtain inferences from a model and responds to those requests by interacting with the hosted model. A dedicated service is a computing device that is tasked with hosting a model server. If demand surges, using a dedicated server may not work well because the capacity of a dedicated server instance may be limited. In addition, this approach will typically require other overhead, such as management burdens.

To address these issues, users can request that access to the machine learning model be provided using a serverless configuration. In the implementation scheme of this example, this is accomplished by specifying in the endpoint configuration that a serverless configuration should be used. Additional parameters may also be supplied, such as the maximum amount of storage to be utilized or the maximum number of concurrent requests to be supported. These parameters can be used to help manage the capabilities utilized by the serverless environment. Here, an endpoint refers to an address or other means of identifying or accessing a machine learning model. An endpoint can act as an outward-facing interface to which users of a machine learning model or other computing service direct their requests.

When a serverless configuration is requested, the system configures the serverless computing architecture to use a model server to process the request to obtain an inference, and the router is configured to forward such requests to the serverless computing architecture. In order to be able to use the model server, the system generates a container including the model server and the extension, which is handed over between the serverless computing architecture and the extension. Generating the container may also include a cleanup process, which refers to the system editing or removing configuration data used by the model server. This configuration data may include configuration data that may have a negative impact if it is not edited or removed. For example, a model server may have configuration data that is applicable to a situation where the model server is installed on a dedicated server instance but is not applicable to a situation where the model server is executed by a serverless computing architecture. If the endpoint is configured to use a hybrid or full server configuration or when the endpoint is configured to use a hybrid or full server configuration, the removed or edited information can be stored and called for later use.

When a request to perform inference is received, this container is retrieved from storage. The serverless computing architecture dynamically allocates computing power to call and execute the computing function that is interfaced with the extension. The extension then activates the HTTP server implemented by the model server and calls the network-based method implemented by the model server. This in turn causes the model server to access the machine learning model and obtain the requested inference.

In another aspect of this example, a user can request to use a hybrid operating mode to provide access to a machine learning model. When configured to operate in hybrid mode, the system uses a dedicated server that has a model server installed to process a portion of the incoming inference requests. The size of this portion can be determined so that the size maximizes the use of the dedicated server. However, in order to cope with temporary surges in demand or to cope with increases in demand that have not been addressed by adding dedicated servers, the system employs a serverless computing architecture. Therefore, a serverless computing architecture is used to process requests that exceed the capabilities of a dedicated server, the serverless computing architecture comprising a container that includes a model server and an extension, as described in the preceding paragraph.

In the foregoing and following descriptions, various techniques are described. For purposes of explanation, specific configurations and details are set forth in order to provide a comprehensive understanding of possible ways to implement these techniques. However, it will also be apparent that the techniques described below can be practiced in different configurations without the specific details. In addition, well-known features may be omitted or simplified to avoid making the described techniques unclear.

1 illustrates a system for performing machine learning including a serverless computing architecture according to at least one embodiment. In the example system 100, a client 130 transmits a request to utilize a machine learning model 116, and this request is received by a router 104. The request is processed by the serverless computing architecture 102, which utilizes the machine learning model 116 in the requested manner and returns the result to the client 130.

Machine learning models, such as the depicted machine learning model 116, may include, but are not limited to, implementing supervised and unsupervised learning, reinforcement learning, linear regression, Naive Bayes ( The machine learning model includes data and code for any of various algorithms or techniques related to Bayesian networks, neural networks, deep learning, random forests, classification, regression, prediction, etc. In at least one embodiment, the machine learning model includes parameters for such a model or algorithm. These parameters may include, for example, various connection weights associated with the neural network. In at least one embodiment, the machine learning model includes a definition of the model architecture and a set of parameters representing the current state of the model. Typically, such parameters represent the current state of model training.

The serverless computing architecture 102 allows computing functions, such as the depicted computing function 110, to be executed using computing power assigned on an on-demand basis. The architecture 102 is described as serverless because, rather than dedicating a particular computing instance to executing a computing function, computing power is dynamically assigned to execute a computing function. Therefore, a serverless computing architecture (such as the architecture 102 depicted in FIG. 1 ) includes one or more computing systems that allocate computing power sufficient to call and execute the computing function in response to a request to call and execute a computing function, and then call and execute the function. In some embodiments, the serverless computing architecture also tracks the utilization of computing power based on the amount of power utilized by the client, rather than based on the number of server instances dedicated to client use. The power utilized by the client in the serverless computing architecture can be measured according to various metrics, which may include the number of invocations of the computing function, the time spent executing the computing function, or the size of the input operations or output operations performed by the computing function. In the example system 100, the serverless computing architecture includes additional features that utilize serverless computing, which utilizes machine learning models using the techniques described herein.

A computation function, such as the depicted computation function 110, includes an executable code unit. The code may include compiled instructions, source code, or intermediate code. A computation function may sometimes be referred to as a procedure, a routine, a method, an expression, a closure, a lambda function, etc. In a serverless computing architecture, the computation function may be provided by a client. However, in the example system 100, the computation function 110 may be automatically generated by the system 100 for performing machine learning functions utilizing the serverless computing architecture 102.

A request to utilize a machine learning model may include, but is not necessarily limited to, an electronic transmission or other communication that includes information indicating that a machine learning model (such as the depicted machine learning model 116) should perform operations. These operations may include, but are not necessarily limited to, inference operations. As used herein, inference operations may include any of a wide variety of machine learning tasks, such as classification, prediction, regression, clustering, segmentation, and the like.

The router 104 may include a network device or other computing device with network communication hardware that is configured to be able to communicate with the client 130 and various components of the serverless computing architecture 102. In at least one embodiment, the serverless computing architecture 102 includes the router 104, while in other embodiments, the router 104 is a front-end component that is separate from the serverless computing architecture but connected to the serverless computing architecture and capable of communicating with the serverless computing architecture. Although not depicted in Figure 1, the router 104 may also be connected to and communicate with a server instance hosted on behalf of the client 130, and the server instance hosts various model servers for utilizing machine learning models. An example of an embodiment including this configuration (sometimes referred to as a hybrid configuration) is further described with respect to Figure 4.

Upon receiving a request to perform inference, router 104 may determine that the request can be processed using serverless computing architecture 102. If so, router 104 may determine to utilize computing service 108 to obtain inferences, rather than directing the request directly to the model server hosted on the server. If a serverless architecture is to be used, router 104 may therefore convert the request into a suitable data format and use computing service 108 to call the appropriate computing function 110.

In at least one embodiment, the router 104 utilizes a role proxy service 106 so that the appropriate level of permissions or authorizations are used when processing requests. In at least one embodiment, the role proxy service 106 includes hardware and/or software that temporarily simulates certain computing roles. This can include temporarily adopting a role whose permissions are suitable for utilizing computing services 108 when these permissions are not associated with incoming requests. Requests directed to the router 104 may not necessarily have the appropriate permissions, because requiring incoming requests to have such permissions may make it more difficult to use serverless computing to set up or utilize machine learning models 116.

In at least one embodiment, the serverless computing architecture includes a computing service 108 that dynamically allocates computing power for invoking and executing a computing function 110. The computing service 108 (which may also be described as a serverless computing service) includes hardware that responds to requests to invoke and execute a computing function 110. This response includes allocating sufficient computing power and then invoking and executing the computing function 110.

The compute function 110 is designed such that when called and executed by the compute service 108 , it interfaces with the extension 112 to enable the model server 114 to obtain inferences from the machine learning model 116 or otherwise utilize the machine learning model.

In at least one embodiment, the machine learning model 116 is hosted by a machine learning service 118. The service may include computing devices and other computing resources configured to provide capabilities related to the configuration, training, deployment, and use of machine learning models. In some cases and embodiments, the service 118 may include storage for model parameters, while in other cases and embodiments, an external storage service may be used in addition to the machine learning service 118.

The model server 114 includes code for interfacing with the machine learning model 116. Here, interfacing can refer to interactions between modules, libraries, programs, functions, or other code units. Examples include, but are not necessarily limited to, a module calling a function of another module and obtaining a result, a module initiating a process on another module, a program accessing a property implemented by a class of a library, etc. It will be appreciated that these examples are intended to be illustrative and not limiting. In general, the model server 114 acts as a front end for the machine learning model 116 and can be used, for example, to train the model 116 or use the model 116 to obtain inferences.

In at least one embodiment, the model server 114 is compatible not only with the serverless computing architecture 102 depicted in FIG1 , but also with a server-based configuration in which clients directly access the model server 114. FIG1 depicts the use of the model server 114 within a serverless computing environment, while FIG4 depicts an example of the use of the model server 114 in a hybrid environment (involving both serverless and server-based configurations).

In at least one embodiment, model server 114 includes code that implements a Hypertext Transfer Protocol ("HTTP") server. In such embodiments, the server is implemented to receive HTTP-compatible messages requesting the use of a machine learning model to perform a machine learning task. For example, in at least one embodiment, the model server includes code that receives an HTTP-compatible request to perform an inference, and responds with an HTTP-compatible response that includes data obtained by performing the inference. The request may include various parameters or attributes associated with the request, such as a name of an endpoint, an identifier of a machine learning model to use, an identifier of an inference to be performed, inputs to the machine learning model, and/or other data.

When operating within a serverless computing architecture (such as the serverless computing architecture 102 depicted in FIG. 1 ), the model server 114 is not hosted on a dedicated instance of a server because the serverless computing architecture 102 dynamically allocates computing resources on a per-request basis. This may create various challenges. One such challenge is due to the different data formats used to call serverless computing functions in a serverless computing environment compared to the data formats used in HTTP-compatible requests. Another reason is that HTTP servers are typically long-running. Although an HTTP server can be activated once on a dedicated server and kept running, this approach may not be feasible in a serverless computing environment because the resources assigned to call and execute computing functions are dynamically assigned and may not last longer than the duration of a single request in some cases.

In an embodiment, extensions 112 are used to address these issues. Extensions 112 include code that interfaces with model servers 114. The interface may include operations that convert between data formats. For example, in at least one embodiment, compute services 108 may accept calls to compute functions 110 in a limited number of data formats. In one example, JavaScript Object Notation ("JSON") format is used, but it will be appreciated that this example is intended to illustrate rather than limit. However, it should be noted that it may be advantageous for system 100 to utilize pre-existing compute services 108 without modifications specifically for implementing machine learning. Therefore, it may be impractical to change the data format used to call compute functions 110, and to address this issue, extensions 112 convert between the data format used to call compute functions 110 and any data format expected by model servers 114.

In at least one embodiment, extension 112 also includes code for activating model server 114 for use within the context of compute function 110. This may include, for example, interfacing with model server 114 to initialize an HTTP server implemented by model server 114 so that extension 112 can then further utilize the HTTP server to access machine learning model 116.

In at least one embodiment, the system 100 includes various facilities for logging activities and recording metrics. These metrics can include metrics related to the operation of the router 104 and the serverless computing architecture 102. For example, the router 104 and the computing functions 120a, 120b can each output logs and metrics related to their respective activities. The client 130 can obtain these and other logs and metrics via the monitoring console 132.

2 illustrates an example process for enabling a serverless computing architecture to perform machine learning, according to at least one embodiment. In example 200, a client 204 requests access to machine learning capabilities (such as inference) provided to the client 204 via a serverless computing architecture.

In at least one embodiment, a serverless endpoint request 230 is provided by the client 204 to the control plane 220. The serverless endpoint request 230 includes data indicating that the client desires access to machine learning capabilities and that such access should be provided using a serverless computing architecture. The serverless endpoint request 230 may also include additional configuration information that may be used to limit the computing capabilities utilized by the serverless computing architecture on behalf of the client. Imposing such limits may help manage costs and improve the provision of computing capabilities.

Control plane 220 may include hardware and/or software to coordinate the execution of a workflow (such as the workflow described with respect to FIG. 2 ) to enable the creation of an endpoint to utilize a machine learning model via a serverless architecture. In response to receiving serverless endpoint request 230 , control plane 220 may send a command to initiate endpoint creation 232 to endpoint creation service 226 .

The endpoint creation service 226 may include hardware and/or software to generate a model container including a model server and extensions, initiate hosting of the model container 234, and, when appropriate, utilize the role proxy service 206 to assume the role 236 for generating the model container. In at least one embodiment, the task of creating the container is delegated to the hosting service 222. The endpoint creation service 226 may create a serverless compute function at 238 for later use by the compute service 208.

In at least one embodiment, the model container is a binary file that includes a model server 214 and an extension 212. The model server 214 and the extension 212 may correspond to the model server 114 and the extension 112 depicted in FIG. 1. The model server 214 includes code for interfacing with the machine learning model and may be compatible with use in a serverless configuration, a server-full configuration, and a hybrid configuration. The extension 212 includes code for interfacing between the serverless compute function 210 and the model server 214.

As used herein, a serverless configuration is a configuration that permits computing functions (such as those performed by model server 214 and extensions 212) to be performed without the need for dedicated servers. In contrast, a server-full configuration uses dedicated servers rather than a serverless computing architecture. A hybrid configuration uses at least some dedicated instances, but also employs a serverless computing architecture.

The hosting service 222 includes hardware and/or software for generating, validating, and/or storing model containers. The hosting service 222 can store the model container in a repository 224, which can then be downloaded by the computing service 208. At this point, the system 200 is configured for serverless provision of machine learning capabilities.

In at least one embodiment, the repository 224 includes a storage system storing containers. The repository 224 can store many such containers, and each container can be mapped to be associated with a different machine learning model or endpoint. As an illustrative example, the repository 224 can include three containers, the first container including a model server _M1 corresponding to the endpoint _E1 , the second container including a model server _M2 corresponding to the endpoint _E2 , and the third container including _a model server _M3 corresponding to the endpoint E3. Each of the endpoints _E1 , _E2 , and _E3 or the associated model servers can be associated with different IP addresses and machine learning models. These endpoints or associated model servers can also be associated with different clients.

In at least one embodiment, the system 200 can then respond to the request to perform the machine learning task by downloading the managed container 240 to the computing service 208 at step 240, and then calling the computing function 210 at step 242. In at least one embodiment, the computing function 210 is implemented by code automatically generated by the system 200 during the hosting process and included in the container. After being called by the computer service 208, the computing function 210 is handed over to the extension, which then converts the request to perform the machine learning task into a format that can be used by the model server 214, and the computing function is handed over to the model server 214 so that the model server obtains the inference results from the machine learning model. This process is explained in more detail with respect to FIG. 1.

FIG3 illustrates an example of calling and executing a compute function to perform machine learning inference according to at least one embodiment. In the example system 300, a router 304 receives a request to perform inference. The router 304 may be similar to the router 104 depicted in FIG1 . The router 304 determines that the request is associated with a model server 314 and that the model server has been configured to operate in a serverless configuration. This determination may be made in a variety of ways, which may include, but are not limited to, retrieving and inspecting configuration metadata associated with the model server to which the request is directed.

The system 300, having determined that the request will be executed using a serverless configuration, causes the compute service 308 to obtain the extension 312 and the model server 314 from the repository 324. This may be done in response to a message from the router 304, which may forward the request to the compute service 308 once the router has determined that the request to perform inference should be handled using a serverless configuration.

In at least one embodiment, the extension 312 and the model server 314 are stored within a container file, and the container file is retrieved from the repository 324. The container can be found in the repository 324 based on an index that associates the model server to which the request is directed (in this example, the model server 314) with the container. The container can also contain a specific implementation of the compute function 310, and in some embodiments, the compute function 310 is implemented by the extension 312. In other embodiments, the system 300 can obtain the specific implementation of the compute function independently of the container or the extension 312. In some embodiments, the model server 314 and the extension 312 can also be stored and retrieved separately, rather than being combined into a single container file.

The compute service 308 calls a compute function 310, which includes code for using an extension 312, causing the extension to ensure that the model server 314 is activated and the extension to interface with the model server 314 to obtain inferences. For example, in an embodiment where the model server 314 implements an HTTP server, the compute function causes the extension 312 to ensure that the HTTP server has been initialized. The extension is also caused to issue one or more HTTP requests to the HTTP server to perform inferences using the machine learning model 316. The extension 312 may include code for converting data provided via the compute function into a format compatible with the HTTP request. This may convey an advantage that this allows the use of pre-existing serverless computing architectures and pre-existing machine learning platforms and services without extensive modification.

The extension 312 may also retrieve data required by the model server 314. This may include, for example, model parameters 340 and various forms of metadata 342. The extension 312 may provide this data to the model server 314 as needed. In order to utilize the model server 314 in a serverless configuration, it may be necessary to provide the endpoint with data that should be normally available on any server instance where the endpoint is installed in a server-full implementation. However, due to the serverless architecture, this data cannot be pre-installed on a dedicated service instance. This technical challenge can be solved by using the extension 312 to load the model parameters, metadata, or other required information from their corresponding storage locations and provide the information to the model server 314 on an as-needed basis. This can be accomplished by the extension 312 interfacing with the model server 314 to provide the model server with parameters 340, metadata 342, or other information to be used by the machine learning model 316. In some cases, such as where the machine learning model is hosted on a machine learning service (such as the machine learning service depicted in FIG. 1 ), the extension 312 may trigger any interaction with the service necessary to make the model ready for use. In at least one embodiment, extension 312 stores log data in log 320 .

Examples of handover operations that can be performed between the extension 312 and the model server 314 can potentially include, but are not limited to, configuration operations, inference operations, training operations, debugging operations, data transformation operations, etc. The extension 312 can receive a request to perform one of these operations, which in turn can convert the request into a format compatible with the model server 314, and handover with the model server 314 to cause the model server to perform the requested operation and obtain any results of the operation.

The model server 314 performs these operations by interacting with the machine learning model 316. The model server 314 may include an HTTP server 330. The machine learning model may be hosted on a distributed, scalable service for configuring, training, deploying, and using the machine learning model. In such embodiments, the model server 314 may interface with the service by calling a network-based method provided by the service for hosting the machine learning model. The network-based method implemented by the service may potentially include, but is not limited to, configuration operations, inference operations, training operations, debugging operations, data transformation operations, and the like.

4 illustrates an example of a hybrid system combining serverless processing and server-full processing of machine learning inferences according to at least one embodiment. A hybrid system such as the depicted system 400 provides machine learning capabilities using one or more dedicated server instances on which machine learning operations are performed, while also incorporating a serverless computing architecture that provides additional capabilities for performing machine learning operations. In at least one embodiment, a baseline amount of machine learning operations are performed by dedicated instances, and a serverless computing architecture is employed to provide burst capabilities.

In at least one embodiment, hybrid system 400 includes router 404. Similar to router 104 depicted in FIG. 1 , router 404 may include a network device or other computing device having network communication hardware configured to enable communication with clients 430 and various components of hybrid system 400, including dedicated server instances, such as depicted server instance 450 and computing service 408.

The router 404 receives a certain amount of requests from the client 430 and distributes the requests between the model server 414a on the server instance 450 and the computing service 408. If there are more than one server instance 450, a certain proportion of requests can be divided between the server instance and the model server installed on the server instance by the router or load balancing component. In at least one embodiment, the proportion of requests routed to the server instance 450 is based on the utilization of the instance capacity. When the utilization exceeds the threshold amount, the router 404 begins to distribute a certain proportion of requests to the computing service 408. The proportion sent to the computing service 408 can be dynamically adjusted to maximize the utilization of the dedicated computing instance while also preventing the instance from being overloaded. The implementation scheme can also try to minimize the utilization of the computing service 408 in order to minimize the cost. For example, in at least one embodiment, the customer is assigned a fixed amount of fees for using the dedicated server instance 450, and is assigned a variable amount of fees for using the computing service 408. In such cases, embodiments can attempt to minimize the total cost allocation by maximizing the utilization of server instances 450 whose cost allocation is fixed and minimizing the utilization of computing services 408 whose costs are variable except for the fixed costs. In embodiments, this can be done while also avoiding over-utilization of dedicated server instances 450.

As depicted in Figure 4, server instance 450 represents a server that is assigned the task of continuously and indefinitely hosting an instance of model server 414a. There may be many such instances, each hosting one or more model servers, but for clarity of explanation, Figure 4 depicts only a single such instance. A server instance may include any computing device suitable for hosting model server 414a.

Compute service 408 provides serverless invocation of compute functions 410 , and an implementation of compute service 408 may correspond to the implementation described with respect to compute service 108 depicted in FIG. 1 .

Compute function 410 includes executable code units that are called and executed by compute service 408 using dynamically allocated compute capacity. Implementations of compute function 410 may correspond to implementations described with respect to compute function 110 depicted in FIG.

Extension 412 includes code for interfacing with model server 114, and an implementation of extension 412 may correspond to an implementation described with respect to compute function 110 depicted in FIG1. Similarly, model server 414 includes code for interfacing with machine learning model 416, and an implementation of model server 414 may correspond to an implementation described with respect to compute function 110 depicted in FIG1.

The implementation scheme of the machine learning service 418 and the machine learning model 416 may also correspond to the implementation scheme described with respect to the machine learning service 118 and the machine learning model 116 described with respect to FIG. 1 . It should be noted that here, the individual machine learning model 416 trained to perform a particular type of inference is used by both a model server 414a running on a dedicated server instance and another model server 414b executed via a serverless computing architecture including the computing service 408. In addition, the model servers 414a, 414b can be instances of the same model server, which means that the code constituting the two instances is the same. The technical advantage conveyed by this is that the customer only needs to provide or define a single endpoint, but the model server can be used in both architectures. In some embodiments consistent with FIG. 4 , the model server 414b is associated with the extension 412 so as to use the model server 414b within the serverless computing architecture.

FIG5 illustrates an example process for configuring a serverless computing architecture to perform machine learning inference according to at least one embodiment. Although the example process 500 is depicted as a series of steps or operations, it will be appreciated that the depicted embodiment of the process may include altered or reordered steps or operations, or may omit certain steps or operations, except where expressly indicated or logically required (such as where the output of one step or operation is used as input to another step or operation). In at least one embodiment, the example process 500 is implemented by a system incorporating a serverless computing architecture, such as any of the architectures depicted or described with respect to the figures.

At 502, the system receives a request to create an endpoint to provide a machine learning service. In an embodiment, endpoint creation refers to the system enabling itself to receive requests to interact with a machine learning model. In at least one embodiment, the endpoint is associated with a network address to which a request to access the machine learning model is directed. In other embodiments, a model server associated with the endpoint is associated with a network address.

At 504, the system determines that a serverless configuration has been requested. For example, in at least one embodiment, the request to create an endpoint can be accompanied by metadata that specifies the properties required for the endpoint, and the metadata can also include a flag or other value indicating that the endpoint should be hosted in a serverless configuration. Additional properties related to the serverless configuration can also be included in the request.

At 506, the system identifies parameters for maximum concurrency and memory utilization for the serverless computing architecture. These parameters may also be specified via metadata included in the request to create the endpoint. Concurrency refers to the number of outstanding requests directed to the endpoint at a given time. Memory utilization refers to the usage of system memory. It will be appreciated that these examples are intended to be illustrative and not limiting.

At 508, the system generates and stores a container including a model server associated with the requested endpoint and extension. Here, the model server refers to code and/or configuration data for implementing the model server, and the extension refers to code including at least instructions for interfacing with the model server. The container, the model server, and the extension may refer to the embodiments described in the accompanying drawings herein, including the embodiments described in relation to FIG. 1 .

Subsequently, when the system receives a request directed to the corresponding endpoint, the stored container can be located and called from the storage device. For example, the container can be stored in a repository indexed by a network address. A similar method can be used to store and index metadata associated with an endpoint. When the system receives a request directed to an endpoint, the system can use the index to determine that the request should be processed via a serverless computing architecture, load the container, and continue processing the request. Examples of implementations of processing requests are described herein with respect to the accompanying drawings (including with respect to FIG. 1).

FIG6 illustrates an example process for configuring a hybrid computing architecture to perform machine learning inference according to at least one embodiment. Although the example process 600 is depicted as a series of steps or operations, it will be appreciated that the depicted embodiment of the process may include altered or reordered steps or operations, or may omit certain steps or operations, except where expressly indicated or logically required (such as where the output of one step or operation is used as input to another step or operation). In at least one embodiment, the example process 600 is implemented by a system incorporating a serverless computing architecture, such as any of the architectures depicted or described with respect to the figures.

At 602, the system receives a request to enable a hybrid configuration to provide machine learning inference. As described above with respect to FIG. 5, a request to create an endpoint for communicating with a machine learning model may include information indicating how the endpoint should be configured. This may include information indicating that a hybrid configuration may be used. In a hybrid configuration, the system employs one or more dedicated servers to handle a portion of requests directed to a corresponding endpoint or model server, and employs a serverless computing architecture to handle the remainder. In some cases, this is done to respond to a surge in demand or to temporarily respond to an increase in demand until a new dedicated instance can be added.

At 604, the system identifies or obtains a dedicated server instance. The servers are referred to as dedicated because they are assigned the role of continuously processing requests directed to the endpoint or associated model server. This generally involves the model server being installed on the server and remaining active for a series of requests. In addition, a dedicated server can be assigned to the same user or account as the endpoint and not used by other users or accounts.

In some cases, the endpoint can be reconfigured so that the endpoint is converted from a server-full configuration to a hybrid configuration. In such cases, the system can identify any existing dedicated servers and continue to use those servers to handle a portion of the incoming requests, and configure the serverless computing architecture to handle the additional portion.

In other cases, such as when an endpoint is first created, the system can obtain access to one or more dedicated servers, configure the one or more dedicated servers to process a portion of requests directed to the endpoint, and configure the serverless computing architecture to process additional portions.

At 606, the system obtains parameters for operating a serverless computing architecture. These parameters may include the parameters described above with respect to FIG. 5. In addition, at 608, the system obtains service level parameters. These service level parameters may include parameters related to the expected utilization level of a dedicated server. To illustrate, in some embodiments, a higher service level may be achieved by keeping the utilization of the dedicated server relatively low and easily transferring the load to the serverless computing architecture when exceeding the utilization. On the other hand, this may result in the user being assigned an additional fee that exceeds and is higher than the fixed fee associated with the dedicated instance. The embodiment may solve this by allowing the user to indicate how the capacity utilization should be divided between the dedicated instance and the serverless computing architecture.

At 610, the system configures itself for hybrid operation. This may include generating and storing containers for model servers and extensions using steps similar or identical to those described with respect to FIG5. Configuration of the hybrid system may also include configuring a router (such as router 404 depicted in FIG4) to distribute workloads between one or more dedicated servers and the serverless computing architecture.

Once configured, a system operating in hybrid mode can achieve load balancing between dedicated server instances and serverless computing architectures. In at least one embodiment, this load balancing can include maximizing the utilization of dedicated servers and transferring the load to the serverless computing architecture when the utilization of dedicated servers exceeds a desired parameter.

In at least one embodiment, the system can generate recommendations for adjusting the number of dedicated servers based on usage patterns and costs associated with the serverless computing architecture. For example, if the utilization of the serverless computing architecture is consistently high, the recommendation may be to add additional dedicated servers; or if the system determines that the serverless computing architecture can be used to efficiently handle periodic demand surges, the recommendation may be to remove servers. It will be appreciated that these examples are intended to be illustrative and not limiting.

FIG7 illustrates an example process for performing machine learning inference using a serverless computing architecture according to at least one embodiment. Although the example process 700 is depicted as a series of steps or operations, it will be appreciated that the depicted embodiment of the process may include altered or reordered steps or operations, or may omit certain steps or operations, except where expressly indicated or logically required (such as where the output of one step or operation is used as input to another step or operation). In at least one embodiment, the example process 700 is implemented by a system incorporating a serverless computing architecture, such as any of the architectures depicted or described with respect to the figures.

At 702, the system receives a request to host a machine learning model using a serverless computing architecture. In at least one embodiment, the request includes information indicating which machine learning model to use, or includes information indicating an endpoint to be used to access the machine learning model.

At 704, the system identifies an endpoint associated with the request. The request may include data indicating an association between the machine learning model and the endpoint. The endpoint may be associated with a model server, as described herein with respect to various embodiments, which may be used to access the machine learning model and perform inference. The model server may include code for interfacing between the client and the machine learning model, such as code implementing an HTTP server whose methods may be used to perform inference using the model.

At 706, the system associates the endpoint with an extension that performs inference between the serverless compute function and the model server. This may include code for converting data provided by the compute function into a format compatible with the model server so that when the serverless compute architecture calls the compute function, the data can be made compatible with any format expected by the model server. For example, in an embodiment where the model server implements an HTTP server, the extension can convert the data into a data format compatible with a network-based approach of the HTTP server.

In at least one embodiment, the extension includes code that, when called by a compute function of a serverless architecture, makes the machine learning model accessible to a model server. In at least one embodiment, this is accomplished by calling an initialization function associated with the model server.

In at least one embodiment, the model server or associated endpoint is associated with the extension by creating a container file. For example, the system can generate a file including the model server and the extension, store the file, and store an association between the file and information identifying the endpoint. This information can then be used to locate the file based on information provided in a request to perform inference. In at least one embodiment, the information is a network address associated with the endpoint or model server.

At 708, the system receives a request to perform inference using a hosted machine learning model. The request can be received by the system as a network-based request directed to an endpoint or model server. The system can then determine that the request should be processed using a serverless computing architecture.

At 710, the system processes the request by executing a serverless compute function. The compute function, when executed, uses the extension to obtain inferences generated by the machine learning model via the model server. Generally, the control flow includes a compute service that calls the compute function, a compute function calling method of the extension, and an extension calling method of the model server. The handover between the extension and the model server can be performed after the extension has performed a suitable initialization process on the endpoint.

At 712, the system provides the requested inference in response to the request. In at least one embodiment, the extension calls one or more methods on the model server to cause the model server to access the machine learning model. The machine learning model performs the inference and returns data constituting the result of the performed inference via the model server.

FIG8 illustrates an example process for performing machine learning inference using a hybrid computing architecture according to at least one embodiment. Although the example process 600 is depicted as a series of steps or operations, it will be appreciated that the depicted embodiment of the process may include altered or reordered steps or operations, or may omit certain steps or operations, except where expressly indicated or logically required (such as where the output of one step or operation is used as input to another step or operation). In at least one embodiment, the example process 800 is implemented by a system incorporating a serverless computing architecture, such as any of the architectures depicted or described with respect to the figures.

At 802, the system recites a request to configure an endpoint to utilize a hybrid configuration. The system can then identify a model server associated with the endpoint, wherein the model server includes code for interfacing with a machine learning model. In some cases, the endpoint may have been previously created, such as when the system is transitioning from a configuration that relies only on dedicated instances to a configuration that relies on a hybrid configuration. In other cases, a new endpoint is created.

At 804, the system associates an endpoint or model server with an extension that includes code for interfacing with the model server. As described herein, for example, with respect to FIG. 4, this can include generating a container that includes code for both the model server and the extension, and storing information that can be used to determine that the serverless computing architecture has been configured to support serverless processing of inference requests.

At 806, the system receives a request to obtain an inference. The request can be received in various patterns over time, which can include a surge in demand or a steady increase in demand. It will be appreciated that these examples are intended to be illustrative and not limiting. The system can then divide the responsibility for processing these requests according to the expected pattern, for example, as described above with respect to FIG. 6. Thus, in at least one embodiment, the system divides the request between the two systems based on the capabilities of the dedicated servers.

At 808, the system responds to a first portion of the request using at least a first instance of a model server operating on at least one dedicated server instance. In some cases and embodiments, the first portion is determined by maximizing utilization of the dedicated instances up to a certain maximum utilization, and any remaining portion of the request is allocated to the serverless computing architecture.

At 810, the system responds to a second portion of the request using a second instance of the model server on the serverless computing architecture. The serverless computing architecture then dynamically allocates capacity to process this portion of the request based on the size of the second portion.

The systems, techniques, and methods described herein can be applied to a variety of computing services, including but not necessarily limited to machine learning models, simulations, network-based applications, or other code units. In general, the disclosed techniques can be applied to scenarios including but not necessarily limited to accessing software applications using an architecture including the disclosed model server.

In an embodiment, a system includes at least one processor and a memory including computer executable instructions, which in response to the execution of the at least one processor causes the system to configure a serverless computing architecture to a hosted computing service. The computing service may include a machine learning model, a computer-based simulation, a network-based application, or other computer services, provided that the computing service is accessed via a model server as described herein. The system configures the serverless computing architecture to a hosted computing service by associating at least an endpoint with an extension including a code for interfacing with a model server, wherein the model server includes a code for accessing the computing service. The system may then receive a request to obtain a result from the computing service, and respond to the request by executing a computing function on the serverless computing architecture. The computing function calls one or more functions implemented by the extension, and the one or more functions obtain the result generated by the computing service via the model server.

In at least one embodiment, one or more functions implemented by the extension, when invoked by the serverless compute function, prepare the compute service for use by the model server.

In at least one embodiment, the memory of the system also includes computer executable instructions that, in response to execution by the at least one processor, cause the system to intercept the request in response to determining that the request is directed to a network address associated with the model server and the endpoint has been configured to utilize a serverless computing architecture.

In at least one embodiment, the model server includes code for enabling hosting of the model server on an instance of a server that has been reserved for providing access to a computing service.

In at least one embodiment, the serverless computing architecture dynamically allocates computing capacity based on demand to process requests to obtain inferences using computing services.

In another example, a method for utilizing a hybrid configuration includes obtaining a model server including code for interfacing with a compute service, and associating the model server with an extension that interfaces with the model server. When a request to obtain a result using the compute service is received, the method includes responding to a first portion of the request by using a first instance of the model server installed on a server, and responding to a second portion of the request by using a serverless compute architecture. To use the serverless compute architecture, the method includes invoking a compute function that uses the extension and at least a second instance of the model server to obtain a result from the compute service.

Fig. 9 shows various aspects of the example system 900 for realizing various aspects according to the embodiment.As will be understood, although a network-based system is used for the purpose of explanation, various embodiments can be realized by using different systems appropriately.In an embodiment, the system includes an electronic client device 902, which includes any appropriate device that can be operated to send and/or receive a request, message or information through an appropriate network 904 and transmit the information back to the user of the device.The example of such client devices includes a personal computer, a mobile phone or other mobile phones, a handheld messaging device, a laptop computer, a tablet computer, a set-top box, a personal data assistant, an embedded computer system, an e-book reader, etc.In an embodiment, the network includes any appropriate network, including an intranet, the Internet, a cellular network, a local area network, a satellite network or any other such network and/or their combination, and the components for such a system depend at least in part on the type of the selected network and/or system.It is well known that many protocols and components for communicating via such a network are thus not discussed in detail herein.In an embodiment, the communication on the network is realized by wired and/or wireless connection and a combination thereof. In an embodiment, the network includes the Internet and/or other publicly addressable communications network, as the system includes a web server 906 for receiving requests and providing content in response to the requests, however for other networks, alternative devices serving similar purposes may be used, as will be apparent to one of ordinary skill in the art.

In an embodiment, the illustrative system includes at least one application server 908 and a data store 910, and it should be understood that there may be several application servers, layers or other elements, processes or components that can be linked or otherwise configured, which can interact to perform tasks such as obtaining data from appropriate data stores. In an embodiment, the server is implemented as a hardware device, a virtual computer system, a programming module executed on a computer system, and/or other devices configured with hardware and/or software to receive and respond to communications (e.g., web service application programming interface (API) requests) over a network. As used herein, unless otherwise specified or clearly visible from the context, the term "data store" refers to any device or device combination capable of storing, accessing and retrieving data, which may include any combination of and any number of data servers, databases, data storage devices and data storage media in any standard, distributed, virtual or clustered systems. In an embodiment, the data store communicates with a block-level and/or object-level interface. The application server may include any appropriate hardware, software and firmware for integrating with the data store as needed to perform various aspects of one or more applications for a client device, thereby processing some or all of the data access and business logic for the application.

In one embodiment, the application server cooperates with the data store to provide access control services and generate content, including but not limited to text, graphics, audio, video and/or other content provided by the network server to a user associated with the client device in the form of Hypertext Markup Language ("HTML"), Extensible Markup Language ("XML"), JavaScript, Cascading Style Sheets ("CSS"), JavaScript Object Notation (JSON) and/or another appropriate client-side or other structured language. In an embodiment, the content transmitted to the client device is processed by the client device to provide one or more forms of content, including but not limited to forms that the user can perceive through hearing, vision and/or other senses. In an embodiment, the processing of all requests and responses and the delivery of content between the client device 902 and the application server 908 are handled by the network server using PHP: Hypertext Preprocessor ("PHP"), Python, Ruby, Perl, Java, HTML, XML, JSON and/or another appropriate server-side structured language in this example. In an embodiment, the operations described herein as being performed by a single device are performed jointly by multiple devices forming a distributed and/or virtual system.

In an embodiment, data storage area 910 includes several separate data tables, databases, data documents, dynamic data storage schemes and/or other data storage mechanisms and media for storing data related to specific aspects of the present disclosure. In an embodiment, data storage area includes a mechanism for storing generated data 912 and user information 916, which are used to provide content for the generation end. The data storage area is also shown as including a mechanism for storing log data 914, which is used for reporting, computing resource management, analysis or other such purposes in an embodiment. In an embodiment, other aspects such as page image information and access rights information (e.g., access control policies or other permission encodings) are stored in the data storage area in any of the mechanisms listed above or in an additional mechanism in the data storage area 910 as appropriate.

In an embodiment, the data store 910 is operable, through logic associated with the data store 910, to receive instructions from the application server 908 and obtain, update or otherwise process data in response to the instructions, and the application server 908 provides static data, dynamic data or a combination of static data and dynamic data in response to the received instructions. In an embodiment, dynamic data (such as data used in web logs (blogs), shopping applications, news services and other such applications) is generated by a server-side structured language as described herein, or provided by a content management system ("CMS") operating on the application server or operating under the control of the application server. In an embodiment, a user submits a search request for a certain type of project through a device operated by the user. In this example, the data store accesses user information to verify the identity of the user, accesses directory details to obtain information about the type of project, and returns the information to the user, such as a web page viewed by the user via a browser on the user device 902 in a result list. Continuing with this example, the information of a specific project of interest is viewed in a dedicated page or window of the browser. However, it should be noted that the embodiments of the present disclosure are not necessarily limited to the context of web pages, but are more generally applicable to general processing requests, where the request is not necessarily a request for content. Example requests include requests to manage and/or interact with computing resources hosted by system 900 and/or another system, such as to launch, terminate, delete, modify, read, and/or otherwise access such computing resources.

In an embodiment, each server typically includes an operating system that provides executable program instructions for general management and operation of the server, and includes a computer-readable storage medium (e.g., a hard disk, random access memory, read-only memory, etc.) that stores instructions that, when executed by a processor of the server, cause or otherwise enable the server to perform its desired functions (e.g., such functions are implemented based on execution of instructions stored on the computer-readable storage medium by one or more processors of the server).

In an embodiment, system 900 is a distributed and/or virtual computing system utilizing several computer systems and components interconnected using one or more computer networks or direct connections via communication links (e.g., Transmission Control Protocol (TCP) connections and/or Transport Layer Security (TLS) or other cryptographically protected communication sessions). However, one of ordinary skill in the art will appreciate that such a system may operate in a system having fewer or greater numbers of components than those shown in FIG. 9 . Thus, the depiction of system 900 in FIG. 9 should be viewed as illustrative in nature and not limiting the scope of the present disclosure.

Various embodiments may further be implemented in a wide range of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that can be used to operate any of a plurality of applications. In an embodiment, user or client devices include any of a plurality of computers, such as desktop computers, laptop computers, or tablet computers running a standard operating system, and (mobile) cell phones, wireless and handheld devices running mobile software and capable of supporting a plurality of networking and messaging protocols, and such systems also include a plurality of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. In an embodiment, these devices also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and other devices capable of communicating via a network; and virtual devices, such as virtual machines, hypervisors, software containers utilizing operating system-level virtualization, and other virtual or non-virtual devices capable of communicating via a network that support virtualization.

In an embodiment, the system utilizes at least one network well known to those skilled in the art to support communications using any of a variety of commercially available protocols such as the Transmission Control Protocol/Internet Protocol ("TCP/IP"), the User Datagram Protocol ("UDP"), protocols operating in various layers of the Open Systems Interconnection ("OSI") model, the File Transfer Protocol ("FTP"), Universal Plug and Play ("UPnP"), the Network File System ("NFS"), the Common Internet File System ("CIFS"), and other protocols. In an embodiment, the network is a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, a satellite network, and any combination thereof. In an embodiment, a connection-oriented protocol is used to communicate between network endpoints, so that a connection-oriented protocol (sometimes referred to as a connection-based protocol) can transmit data in an ordered stream. In an embodiment, a connection-oriented protocol may be reliable or unreliable. For example, the TCP protocol is a reliable connection-oriented protocol. Asynchronous Transfer Mode ("ATM") and frame relay are unreliable connection-oriented protocols. Connection-oriented protocols are in contrast to packet-oriented protocols, such as UDP, which do not guarantee the order in which packets are transmitted.

In an embodiment, the system utilizes a web server running one or more of a variety of server or middle-tier applications, including a Hypertext Transfer Protocol ("HTTP") server, an FTP server, a Common Gateway Interface ("CGI") server, a data server, a Java server, an Apache server, and a commercial application server. In an embodiment, one or more servers are also capable of executing programs or scripts in response to requests from user devices, such as by executing a program implemented in any programming language (such as C, C# or C++) or any scripting language (such as Ruby, PHP, Perl, Python or TCL) and their combinations. In an embodiment, the one or more servers also include a database server, including but not limited to and Commercially available ones, as well as open source servers such as MySQL, Postgres, SQLite, MongoDB, and any other server capable of storing, retrieving, and accessing structured or unstructured data. In an embodiment, the database server includes a table-based server, a document-based server, an unstructured server, a relational server, a non-relational server, or a combination of these and/or other database servers.

In an embodiment, the system includes the various data stores discussed above, as well as other memories and storage media, which may reside in various locations, such as on storage media local to (and/or resident in) one or more computers, or remote from any or all of the computers on the network. In an embodiment, the information resides in a storage area network ("SAN") familiar to those skilled in the art, and similarly, any necessary files for performing the functions attributed to the computer, server, or other network device are stored locally and/or remotely as appropriate. In an embodiment where the system includes computerized devices, each such device may include hardware elements electrically coupled via a bus, including, for example, at least one central processing unit ("CPU" or "processor"), at least one input device (e.g., a mouse, keyboard, controller, touch screen, or keypad), at least one output device (e.g., a display device, printer, or speaker), at least one storage device such as a disk drive, an optical storage device, and a solid-state storage device such as a random access memory ("RAM") or a read-only memory ("ROM"), as well as removable media devices, memory cards, flash memory cards, etc., and various combinations thereof.

In an embodiment, such devices also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.), and a working memory as described above, wherein the computer-readable storage medium reader is connected or configured to receive a computer-readable storage medium, representing a remote, local, fixed and/or removable storage device and a storage medium for temporarily and/or more permanently containing, storing, transmitting and retrieving computer-readable information. In an embodiment, the system and various devices typically also include a plurality of software applications, modules, service systems or other elements located within at least one working memory device, including an operating system and applications, such as a client application or a web browser. In an embodiment, customized hardware is used, and/or specific elements are implemented in hardware, software (including portable software such as applets), or both. In an embodiment, a connection to other computing devices such as a network input/output device is employed.

In an embodiment, storage media and computer-readable media for containing code or code portions include any suitable media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and/or transmitting information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory ("EEPROM"), flash memory or other memory technology, read-only compact disk drive ("CD-ROM"), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the system device. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will understand other ways and/or methods to implement various embodiments.

Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims.

Other variations are also within the spirit of the present disclosure. Therefore, although the disclosed technology may have various modifications and alternative configurations, certain illustrated embodiments thereof are shown in the drawings and described in detail above. However, it should be understood that it is not intended to limit the present invention to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative configurations and equivalents that fall within the spirit and scope of the present invention as defined by the appended claims.

Additionally, embodiments of the present disclosure may be described in terms of the following:

1. A computer-implemented method comprising:

receiving a first request to host a machine learning model on a serverless computing architecture, the first request comprising an indication of an endpoint, the endpoint comprising code for accessing the machine learning model;

associating a model server with an extension, the model server being associated with the endpoint, the extension comprising code for interfacing with the model server;

receiving a second request to generate an inference using the machine learning model;

processing the second request by executing at least a compute function on the serverless computing architecture, wherein the compute function uses the extension to obtain inferences generated by the machine learning model via the model server; and

The inference is provided in response to the second request.

2. A computer-implemented method as described in clause 1, wherein the code in the extension enables the machine learning model to be accessible to the model server when called by the compute function.

3. The computer-implemented method of clause 1 or 2, further comprising:

Generate a file including the model server and the extension;

storing the file; and

An association between the file and the endpoint is stored, wherein the information is suitable for locating the file based on information included in the second request.

4. A computer-implemented method as described in any of clauses 1 to 3, wherein the extension includes code for converting data provided by the serverless computing function into a format compatible with a network server implemented by the model server.

5. A computer-implemented method as described in any of clauses 1 to 4, wherein the serverless computing architecture adds or removes computing power based at least in part on the amount of requests to generate inferences using the machine learning model.

6. A system comprising:

at least one processor; and

a memory including computer executable instructions that, in response to execution by the at least one processor, cause the system to:

configuring the serverless computing architecture to host a computing service by associating at least an endpoint with an extension including code for interfacing with a model server, the model server being associated with the endpoint, wherein the model server includes code for accessing the computing service;

receiving a request to obtain a result from the computing service; and

The request is responded to by executing at least a compute function on the serverless computing architecture, wherein the compute function calls one or more functions implemented by the extension, and the one or more functions obtain results generated by the compute service via the model server.

7. The system of clause 6, wherein the one or more functions implemented by the extension, when invoked by the serverless compute function, prepare the compute service for use by the model server.

8. The system of clause 6 or 7, wherein the memory further comprises computer executable instructions that, in response to execution by the at least one processor, cause the system to:

In response to determining that the request is directed to a network address associated with the model server and the endpoint has been configured to utilize the serverless computing architecture, the request is intercepted.

9. The system of clauses 6 to 8, wherein the memory further comprises computer executable instructions which, in response to execution by the at least one processor, cause the system to:

generating a file including the endpoint and the extension;

storing the file;

storing an association between the file and the endpoint; and

Information is used to locate the file based on information included in the request to generate an inference.

10. The system of any one of clauses 6 to 9, wherein the one or more functions implemented by the extension convert data provided by the serverless computing function into a format usable by the model server.

11. The system of any of clauses 6 to 10, wherein the model server comprises code for enabling hosting of the model server on an instance of a server reserved for providing access to the computing service.

12. The system of any of clauses 6 to 11, wherein the server implements an HTTP server, and wherein the extension interfaces with the endpoint to activate the HTTP server.

13. The system of any of clauses 6 to 12, wherein the serverless computing architecture dynamically allocates computing capacity based on demand to process requests to obtain inferences using the computing service.

14. A non-transitory computer-readable storage medium having executable instructions stored thereon, the executable instructions as a result of execution by one or more processors of a computer system causing the computer system to at least:

configuring a serverless computing architecture to host a machine learning model by associating at least an endpoint with an extension comprising code for interfacing with the endpoint, wherein a model server associated with the endpoint comprises code for interfacing with the machine learning model;

receiving a request to generate an inference using the machine learning model; and

The request is processed by executing at least a serverless computing function on the serverless computing architecture, wherein the serverless computing function uses the extension to obtain inferences generated by the machine learning model via the model server.

15. The non-transitory computer-readable storage medium of clause 14, wherein the extension enables the machine learning model to be accessible to the model server.

16. The non-transitory computer-readable storage medium of clause 14 or 15, wherein the instructions further comprise instructions that, as a result of execution by the one or more processors, cause the computer system to:

In response to determining that the request is directed to a network address associated with the model server and the endpoint has been configured to utilize the serverless computing architecture, the request is intercepted to generate inferences.

17. The non-transitory computer-readable storage medium of any one of clauses 14 to 16, wherein the instructions further comprise instructions that, as a result of execution by the one or more processors, cause the computer system to:

Generate a file including the model server and the extension;

storing the file; and

The file is retrieved from storage in response to receiving the request to generate an inference.

18. The non-transitory computer-readable storage medium of any of clauses 14 to 17, wherein the extension converts data provided by the serverless compute function into a format compatible with a network server implemented by the model server.

19. A non-transitory computer-readable storage medium as described in any of clauses 14 to 18, wherein the endpoint includes code for obtaining inferences using the machine learning model, and the extension includes code for obtaining the inferences using the model server.

20. A non-transitory computer-readable storage medium as described in any of clauses 14 to 19, wherein the machine learning model is hosted by a computing service accessed by the model server.

21. A system comprising:

at least one processor;

a memory including computer executable instructions that, in response to execution by the at least one processor, cause the system to:

obtaining a model server including code for interfacing with the machine learning model;

associating the model server with an extension including code for interfacing with the model server;

receiving a plurality of requests to obtain inferences using the machine learning model;

responding to a first portion of the plurality of requests using a first instance of the model server on a server configured to generate inferences using the model server; and

A second portion of the request is responded to by invoking a compute function using a serverless computing architecture, wherein the compute function generates inferences using the extension and at least a second instance of the model server, wherein a size of the second portion of the request is based at least in part on a capability of the server to respond to the first portion of the request.

22. The system of clause 21, wherein the serverless computing architecture allocates capacity for processing the second portion of the request based on the size of the second portion.

23. The system of clause 21 or 22, wherein the memory further comprises computer executable instructions that, in response to execution by the at least one processor, cause the system to:

A recommendation is generated to configure one or more additional servers to generate inferences using the model server.

24. The system of any of clauses 21 to 23, wherein the model server comprises an HTTP server activated by the extension.

25. A system as described in any of clauses 21 to 24, wherein the respective sizes of the first part and the second part are adjusted to maximize utilization of the server up to a threshold amount, and wherein requests that will cause the utilization to exceed the threshold amount are assigned to the second part.

26. A method comprising:

obtaining a model server including code for interfacing with a computing service;

Associating the model server with an extension that interfaces with the model server;

receiving a plurality of requests to obtain results by using the computing service;

responding to a first portion of the plurality of requests to obtain results from the computing service by using a first instance of the model server installed on a server; and

A second portion of the request is responded to by invoking a compute function using a serverless computing architecture, wherein the compute function uses the extension and at least a second instance of the model server to obtain a result from the compute service.

27. The method of clause 26, wherein the request is divided between the first portion and the second portion based at least in part on capabilities of the server.

28. The method of clause 26 or 27, wherein the serverless computing architecture allocates capacity for processing the second portion of the request based on a size of the second portion of the request.

29. The method of any one of clauses 26 to 28, further comprising:

A recommendation to add an additional server having the installed endpoint is generated, the recommendation based at least in part on utilizing the serverless computing architecture to obtain results from the computing service.

30. The method of any one of clauses 26 to 29, further comprising:

The first portion is sized to maximize utilization of the server without exceeding a threshold utilization amount.

31. The method of any one of clauses 26 to 30, further comprising:

receiving a request to enable a hybrid configuration for obtaining inferences using the machine learning model; and

In response to the request, the extension is generated and associated with the model server.

32. The method of any one of clauses 26 to 31, further comprising:

intercepting a first request and a second request to obtain a result from the computing service, the intercepting being based at least in part on determining that a hybrid configuration has been enabled;

processing the first request using the server; and

The second request is processed using the serverless computing architecture.

33. The method of any of clauses 26 to 32, wherein the workload comprising the request for obtaining the inference is transferable between instances of the server and the serverless computing architecture.

34. A non-transitory computer-readable storage medium having executable instructions stored thereon, the executable instructions as a result of execution by one or more processors of a computer system causing the computer system to at least:

obtaining a model server including code for interfacing with the machine learning model;

Associating the endpoint with an extension that interfaces with the model server;

receiving a plurality of requests to obtain inferences using the machine learning model;

responding to a first portion of the plurality of requests to obtain inferences by using a first instance of the model server installed on a server; and

A second portion of the request is responded to by invoking a compute function using a serverless computing architecture, wherein the compute function generates inferences using the extension and at least a second instance of the model server.

35. The non-transitory computer-readable storage medium of clause 34, wherein the instructions further comprise instructions that, as a result of execution by the one or more processors, cause the computer system to:

The router is configured to determine the first portion and the second portion based at least in part on capacity utilization of the server.

36. The non-transitory computer-readable storage medium of clause 34 or 35, wherein the serverless computing architecture allocates capacity for processing the second portion of the request based on a size of the second portion.

37. The non-transitory computer-readable storage medium of any of clauses 34 to 36, wherein the instructions further comprise instructions that, as a result of execution by the one or more processors, cause the computer system to:

A recommendation is generated to add an additional server having an installed model server, the recommendation based at least in part on generating inferences utilizing the serverless computing architecture.

38. The non-transitory computer-readable storage medium of any of clauses 34 to 37, wherein the instructions further comprise instructions that, as a result of execution by the one or more processors, cause the computer system to:

determining that processing a greater portion of the requests will cause the server to exceed a threshold utilization; and

Relative sizes of the first portion and the second portion are adjusted in response to the determination.

39. The non-transitory computer-readable storage medium of any of clauses 34 to 38, wherein the instructions further comprise instructions that, as a result of execution by the one or more processors, cause the computer system to:

receiving a request to enable a hybrid configuration for obtaining inferences using the machine learning model; and

generating the extension; and

An association between the extension and the model server is stored.

40. The non-transitory computer-readable storage medium of any of clauses 34 to 39, wherein the instructions further comprise instructions that, as a result of execution by the one or more processors, cause the computer system to:

The router is configured to intercept the request to obtain the inference, wherein the interception will be based at least in part on determining that the hybrid configuration has been enabled.

In the context of describing the disclosed embodiments (especially in the context of the following claims), the use of the terms "one" and "a kind of" and "the" and similar referents should be interpreted as covering both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. Similarly, the use of the term "or" should be interpreted as meaning "and/or" unless explicitly or contradicted by the context. The terms "include", "have", "include" and "contain" should be interpreted as open terms (i.e., meaning "including but not limited to") unless otherwise specified. The term "connected" should be interpreted as being partially or fully included in the following explanation when it is not modified and refers to a physical connection: attached to or combined together, even if there is an intervening object. Unless otherwise indicated herein, the enumeration of the value range herein is intended to be used only as a shorthand method to individually represent each individual value belonging to the range, and each individual value is incorporated into this specification as if it were individually described herein. Unless otherwise specified or contradicted by context, use of the term "set" (e.g., "a set of items") or "subset" should be interpreted as a non-empty set that includes one or more elements. Furthermore, unless otherwise specified or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, although the subset and the corresponding set may be equal. Unless explicitly stated otherwise or clear from context, use of the phrase "based on" means "based at least in part on," and is not limited to "based only on."

Unless specifically stated otherwise or otherwise clearly contradicted by context, conjunction phrases such as “at least one of A, B, and C” or “at least one of A, B and C” (i.e., the same phrase with or without the Oxford comma) are additionally understood within the context to be generally used to indicate that an item, term, etc. can be any non-empty subset of the set of A or B or C, A and B and C, or any set that includes at least one A, at least one B, or at least one C that is not contradicted by context or otherwise excluded. For example, in the illustrative example of a set with three members, the conjunction phrases "at least one of A, B, and C" and "at least one of A, B and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}, and, if not explicitly or contradicted by context, any set having {A}, {B}, and/or {C} as a subset (e.g., a set with multiple "A"s). Thus, such conjunction language is generally not intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C, respectively. Similarly, phrases such as "at least one of A, B, or C" and "at least one of A, B, or C" refer to any of the following sets: {A}, {B}, {C}, {A,B}, {A,C}, {B,C}, {A,B,C}, unless expressly specified or clearly indicated by context. In addition, the term "plurality" indicates plural state (e.g., "plurality items" indicates multiple items) unless otherwise specified or contradicted by context. The number of plural items is at least two, but the number can be more when expressly or by context indicates otherwise.

Unless otherwise noted herein or clearly contradictory to the context, the operations of the processes described herein can be performed in any suitable order. In an embodiment, processes such as those described herein (or variations and/or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions, and are implemented as codes (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processors by hardware or a combination thereof. In an embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program, which includes multiple instructions that can be executed by one or more processors. In an embodiment, a computer-readable storage medium is a non-transient computer-readable storage medium that excludes transient signals (e.g., propagated transient electricity or electromagnetic transmission) but includes non-transient data storage circuits (e.g., buffers, caches, and queues) in a transceiver of a transient signal. In an embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transient computer-readable storage media on which executable instructions are stored, and the executable instructions are executed by one or more processors of a computer system (i.e., due to being executed) to cause the computer system to perform the operations described herein. In an embodiment, the set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more individual non-transitory storage media of the plurality of non-transitory computer-readable storage media do not have all of the code, while the plurality of non-transitory computer-readable storage media collectively store all of the code. In an embodiment, the executable instructions are executed so that different instructions are executed by different processors—for example, in an embodiment, the non-transitory computer-readable storage medium stores instructions, and the main CPU executes some of the instructions, while the graphics processor unit executes other instructions. In another embodiment, different components of the computer system have separate processors and different processors execute different subsets of the instructions.

Thus, in an embodiment, the computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and/or software that enables the operations to be performed. Furthermore, in an embodiment of the present disclosure, the computer system is a single device, and in another embodiment, the computer system is a distributed computer system comprising multiple devices that operate in different ways such that the distributed computer system performs the operations described herein and such that a single device does not perform all the operations.

Unless otherwise claimed, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of the present invention, rather than to limit the scope of the present invention. Language in this specification should not be construed as indicating any non-claimed element as essential to the practice of the present invention.

Embodiments of the present disclosure are described herein, including the best mode known to the inventor for carrying out the present invention. After reading the above description, variations of those embodiments may become apparent to those of ordinary skill in the art. The inventor expects that skilled persons will adopt such variations when appropriate, and the inventor intends to practice the embodiments of the present disclosure in a manner different from that specifically described herein. Therefore, as permitted by applicable law, the scope of the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto. In addition, unless otherwise indicated herein or otherwise clearly contradictory to the context, the scope of the present disclosure covers any combination of the above-mentioned elements in all possible variations thereof.

All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

Claims (15)

1. A computer-implemented method, comprising:

receiving a first request to host a machine learning model on a serverless computing architecture, the first request comprising an indication of an endpoint, the endpoint comprising code for accessing the machine learning model;

Associating a model server with an extension, the model server associated with the endpoint, the extension comprising code for interfacing with the model server;

Receiving a second request to generate an inference using the machine learning model;

Processing the second request by executing at least a computing function on the serverless computing architecture, wherein the computing function uses the expansion to obtain inferences generated by the machine learning model via the model server; and

The inference is provided in response to the second request.

2. The computer-implemented method of claim 1, wherein the code in the extension, when invoked by the computing function, enables the machine learning model to be accessed by the model server.

3. The computer-implemented method of claim 1, further comprising:

Generating a file comprising the model server and the extension;

storing the file; and

An association between the file and the endpoint is stored, wherein information is adapted to locate the file based on information included in the second request.

4. The computer-implemented method of claim 1, wherein the extension includes code for converting data provided by the serverless computing function into a format compatible with a web server implemented by the model server.

5. The computer-implemented method of claim 1, wherein the server-less computing architecture adds or removes computing power based at least in part on an amount of requests to generate inferences using the machine learning model.

6. A system, comprising:

at least one processor; and

A memory comprising computer-executable instructions that, in response to execution by the at least one processor, cause the system to:

Configuring a serverless computing architecture to host a computing service by associating at least an endpoint with an extension that includes code for interfacing with a model server associated with the endpoint, wherein the model server includes code for accessing the computing service;

Receiving a request to obtain a result from the computing service; and

Responding to the request by executing at least a computing function on the serverless computing architecture, wherein the computing function invokes one or more functions implemented by the extension and the one or more functions obtain results generated by the computing service via the model server.

7. The system of claim 6, wherein the one or more functions implemented by the extension, when invoked by the serverless computing function, prepares the computing service for use by the model server.

8. The system of claim 6, the memory further comprising computer-executable instructions that, in response to execution by the at least one processor, cause the system to:

In response to determining that the request is directed to a network address associated with the model server and that the endpoint has been configured to utilize the serverless computing architecture, intercept the request.

9. The system of claim 6, wherein the model server comprises code for enabling hosting of the model server on an instance of a server reserved for providing access to the computing service.

10. The system of claim 6, wherein the serverless computing architecture dynamically allocates computing power according to demand to handle requests to obtain inferences using the computing services.

11. A method, comprising:

Obtaining a model server comprising code for interfacing with a computing service;

associating the model server with an extension interfacing with the model server;

Receiving a plurality of requests to obtain results by using the computing service;

Responding to a first portion of the plurality of requests to obtain results from the computing service by using a first instance of the model server installed on a server; and

Responding to the second portion of the request by invoking a computing function using a serverless computing architecture, wherein the computing function uses the extension and at least a second instance of the model server to obtain a result from the computing service.

12. The method of claim 11, wherein the serverless computing architecture allocates capacity for processing the second portion of the request according to a size of the second portion.

13. The method of claim 11, further comprising:

Receiving a request to enable a hybrid configuration for obtaining an inference using the machine learning model; and

In response to the request, the extension is generated and associated with the model server.

14. The method of claim 11, further comprising:

intercepting a first request and a second request to obtain results from the computing service, the intercepting based at least in part on determining that a hybrid configuration has been enabled;

Processing the first request using the server; and

The second request is processed using the serverless computing architecture.

15. The method of claim 11, wherein a workload including the request to obtain inferences can be transferred between an instance of the server and the serverless computing architecture.