WO2018207152A1

WO2018207152A1 - Systems and methods for generation and automated usage of characterization metadata for application deployment and scaling

Info

Publication number: WO2018207152A1
Application number: PCT/IB2018/053313
Authority: WO
Inventors: Rafi Rabipour; Cristina Badulescu; Claes Göran Robert EDSTRÖM
Original assignee: Telefonaktiebolaget LM Ericsson AB
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 2017-05-12
Filing date: 2018-05-11
Publication date: 2018-11-15
Anticipated expiration: 2019-11-12

Abstract

Systems and methods for generation and usage of characterization metadata for a virtualized deployment of a target application are disclosed. In some embodiments, a method comprises generating characterization data that characterizes one or more characteristics of a target application running as a virtualized function on an infrastructure, and generating metadata, based on the characterization data, that characterizes one or more performance aspects of the target application as a function of one or more configuration parameters for the target application and/or the infrastructure. In this manner, metadata is generated that can be used to orchestrate and manage a virtualized deployment of the target application to, e.g., optimize performance and efficiency.

Description

SYSTEMS AND METHODS FOR GENERATION AND AUTOMATED USAGE OF CHARACTERIZATION METADATA FOR APPLICATION DEPLOYMENT

AND SCALING Related Applications

[0001 ] This application claims the benefit of provisional patent application serial number 62/505,608, filed May 12, 2017, the disclosure of which is hereby incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates to virtualization and cloud computing environments.

Background

[0003] In existing cloud environments, the physical hardware platform typically features multiple cores on each Central Processing Unit (CPU) socket and each host may have one or more CPU sockets. In an environment where hardware virtualization or Operation System (OS) virtualization is used, one or more of these cores can be allocated to each virtual machine or container. Depending on how this allocation is done, along with the sharing of other resources such as memory, Input/Output (I/O), etc., different performance behavior can be experienced. For a given application, resource management functions generally assume a predictable and somewhat linear capacity model as a function of the quantity of resources dedicated to the application, without monitoring the behavior of the application for verification of the assumption.

[0004] The presumption of predictability and linearity of capacity, or other aspects of performance, as a function of the quantity of allocated and consumed resources has been the basis for the dimensioning of cloud based systems, e.g. for telecommunication systems. The industry trend towards the deployment of cloud technologies may diminish the criticality of pre-dimensioning of resource requirements. Rather, the "abundance" of Commercial Off The Shelf (COTS) servers combined with the ability for dynamic instantiation of virtual machines (or containers) in a cloud architecture is relied upon for on-demand adjustment of the system dimension to deliver a desired capacity or meet other performance targets for a given application. However, even in a cloud architecture, the cost- effective allocation and utilization of resources requires predictable and linear performance characteristics. Rigorous testing and dimensioning must be carried out for present systems (e.g., by vendors of (telecom) equipment) together with customization of the design in order to more-or-less guarantee these

characteristics.

[0005] However, going forward, it may be difficult if not impossible to guarantee predictable and linear performance characteristics when deploying a system, such as a telecommunications system, in a cloud-based architecture. As such, there is a need for systems and methods for addressing this issue.

Summary

[0006] Systems and methods for generation and usage of characterization metadata for a virtualized deployment of a target application are disclosed. In some embodiments, a method comprises generating characterization data that characterizes one or more characteristics of a target application running as a virtualized function on an infrastructure, and generating metadata, based on the characterization data, that characterizes one or more performance aspects of the target application as a function of one or more configuration parameters for the target application and/or the infrastructure. In this manner, metadata is generated that can be used to orchestrate and manage a virtualized deployment of the target application to, e.g., optimize performance and efficiency.

[0007] In some embodiments, generating the characterization data comprises generating a load on the target application, and collecting and storing at least some of the characterization data while the load is on the target application. Further, in some embodiments, the characterization data comprises data regarding the load on the target application, data that characterizes one or more characteristics of the target application while the load is on the target application, and data regarding configuration parameter values configured for the one or more configuration parameters for the target application and/or the infrastructure.

[0008] In some embodiments, generating the characterization data comprises configuring the one or more configuration parameters for the target application and/or the infrastructure with a first set of configuration values, subjecting the target application to a plurality of different load levels while the one or more configuration parameters for target application and/or the infrastructure are configured with the first set of configuration values, and collecting first

characterization data while subjecting the target application to the plurality of different load levels while the one or more configuration parameters for the target application and/or the infrastructure are configured with the first set of

configuration values. Generating the characterization data further comprises configuring the one or more configuration parameters for the target application and/or the infrastructure with a second set of configuration values, subjecting the target application to a plurality of different load levels while the one or more configuration parameters for the target application and/or the infrastructure are configured with the second set of configuration values, and collecting second characterization data while subjecting the target application to the plurality of different load levels while the one or more configuration parameters for the target application and/or the infrastructure are configured with the second set of configuration values. Further, in some embodiments, generating the

characterization data further comprises repeating the steps of configuring, subjecting, and collecting for one or more additional sets of configuration values for the one or more parameters for the target application and/or the infrastructure to thereby obtain additional characterization data.

[0009] In some embodiments, each of the first characterization data and the second characterization data comprises data regarding the plurality of different loads on the target application, data that characterizes one or more

characteristics of the target application while the plurality of different loads are on the target application, and data regarding the respective set of configuration parameter values for the one or more configuration parameters for the target application and/or the infrastructure.

[0010] In some embodiments, the one or more configuration parameters comprise one or more hardware configuration parameters and/or one or more software configuration parameters. Further, in some embodiments, the one or more hardware configuration parameters comprise a number of processor cores allocated for the target application, an amount of memory allocated for the target application, an amount of disk space allocated for the target application, and/or network bandwidth allocated for the target application. Further, in some embodiments, the one or more software configuration parameters comprise an operating system used in a virtualization environment in which the target application is running as a virtual function and/or one or more parameters of the virtualization environment.

[0011 ] In some embodiments, the metadata comprises one or more descriptor files that can be read by management and orchestration nodes of a cloud computing environment.

[0012] In some embodiments, the one or more performance aspects of the target application comprise maximum capacity, and the metadata comprises data that defines maximum capacity of the target application for two or more different sets of configuration parameter values for the one or more configuration parameters for the target application and/or the infrastructure. Further, in some embodiments, the two or more different sets of configuration parameter values for the one or more configuration parameters for the target application and/or the infrastructure comprise two or more different numbers of processor cores allocated for the target application.

[0013] In some embodiments, the one or more performance aspects of the target application comprise delay, and the metadata comprises data that defines delay of the target application as a function of capacity of the target application and two or more different sets of configuration values for the one or more configuration parameters for the target application and/or the infrastructure.

Further, in some embodiments, the two or more different sets of configuration values for the one or more configuration parameters for the target application and/or the infrastructure comprise two or more different numbers of processor cores allocated for the target application.

[0014] In some embodiments, the method further comprises utilizing the metadata. In some embodiments, utilizing the metadata comprises utilizing the metadata to orchestrate and/or manage the target application and/or an infrastructure of a cloud computing environment in which the target application is running. In some embodiments, the metadata is stored in one or more description files, and utilizing the metadata comprises utilizing the descriptor files to orchestrate and/or manage the target application and/or an infrastructure of a cloud computing environment in which the target application is running.

[0015] Embodiments of a node are also disclosed. In some embodiments, a node is adapted to generate characterization data that characterizes one or more characteristics of a target application running as a virtualized function on an infrastructure and generate metadata, based on the characterization data, that characterizes one or more performance aspects of the target application as a function of one or more configuration parameters for the target application and/or the infrastructure.

[0016] In some embodiments, a node comprises at least one processor and memory storing instructions executable by the at least one processor, whereby the node is operable to generate characterization data that characterizes one or more characteristics of a target application running as a virtualized function on an infrastructure and generate metadata, based on the characterization data, that characterizes one or more performance aspects of the target application as a function of one or more configuration parameters for the target application and/or the infrastructure.

[0017] In some embodiments, a node comprises one or more modules comprising a first generating module operable to generate characterization data that characterizes one or more characteristics of a target application running as a virtualized function on an infrastructure and a second generating module operable to generate metadata, based on the characterization data, that characterizes one or more performance aspects of the target application as a function of one or more configuration parameters for the target application and/or the infrastructure. Brief Description of the Drawings

[0018] The accompanying drawing figures incorporated in and forming a part of this specification illustrate several aspects of the disclosure, and together with the description serve to explain the principles of the disclosure.

[0019] Figure 1 depicts a system that includes elements of a framework, and their respective relationships, intended to extract characteristics of interest for a target application running on a specific infrastructure according to some embodiments of the present disclosure;

[0020] Figure 2 is a flow chart that illustrates a method of extracting and using characteristics of interest for a target application running on a specific infrastructure according to some embodiments of the present disclosure;

[0021] Figure 3 illustrates one example embodiment of the descriptor generator of Figure 1 ;

[0022] Figure 4 illustrates results of characterization of capacity of a target application as a function of the number of allocated processor cores according to some example embodiments of the present disclosure;

[0023] Figure 5 illustrates results of delay (or response time) of a target application as a function of capacity and the number of allocated processor cores according to some example embodiments of the present disclosure;

[0024] Figures 6 depicts an example mapping between delay (or response time) and the load level as a function of the number of allocated processor cores; and

[0025] Figures 7 and 8 illustrate example embodiments of a node

implementing at least some aspects of the present disclosure. Detailed Description

[0026] The embodiments set forth below represent information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the following description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure.

[0027] The presumption of predictability and linearity of capacity (or other performance dimensions) as a function of the quantity of allocated and utilized resources has been the basis for the dimensioning of systems designed to deliver services (e.g., telecommunications), including cloud based systems. The industry trend towards the deployment of cloud technologies may diminish the criticality of pre-dimensioning of resource requirements; the "abundance" of Commercial Off The Shelf (COTS) servers combined with the ability for dynamic instantiation of virtual machines (or containers) in a cloud architecture is relied upon for on-demand adjustment of the system dimension to deliver a desired capacity or meet other performance targets for a given application. However, even in a cloud architecture the cost-effective allocation and utilization of resources requires predictable and linear performance characteristics. Rigorous testing and dimensioning must be carried out for present systems (e.g., by vendors of (telecom) equipment) together with customization of the design in order to more-or-less guarantee these characteristics. Going forth, however, such may no longer be the case due to a number of factors such as:

· The increasing trend towards multicore and many-core systems make concurrency and parallelism increasingly necessary for meeting demanding performance requirements. However, the design and implementation of correct, efficient, and scalable concurrent software is often a very difficult task due to the fact that multicore processors typically rely on some form of synchronization to guarantee that the shared state information does not get corrupted. Current multithreaded software typically has a limitation on the number of cores of which it can make use. Past this point, performance does not improve and, in certain cases, performance can even get worse. This means that the increase in performance as a result of adding a core to a running application will largely depend on how the application software has been designed, particularly in the sense that the software implementation may or may not be compatible with the hardware platform constraints that may result from the sharing of resources among multiple cores.

• There is a strong desire on the part of service providers to build

systems with components from multiple vendors. This was cited as one of the motivations behind the launch of the Network Functions Virtualization (NFV) activity. The construction of systems with software and nodes from different vendors is likely to exacerbate the predictability and linearity of capacity behavior due to factors such as interoperability issues, improper configuration, faulty software, or corner-case performance problems.

• Current performance-sensitive applications, e.g. in telecommunication systems, are normally deployed over multiple redundant hosts and scaling is achieved by adding or removing hosts. Since these hosts are physically isolated and the application runs natively - and therefore utilizes all resources on the host - a high degree of performance linearity and predictability can be achieved. With the advent of cloud based systems this tight control of the execution environment may not be possible or even desired, yet the requirement for performance predictability remains.

[0028] The result of these is that it becomes very difficult to predict how a system will perform when additional resources are added for use by the software application. Aside from degrading operation efficiency, this becomes a significant issue with systems that must satisfy the terms of specific Service Level Agreements (SLAs), which for example could be a telecommunication system deployed in a cloud environment.

[0029] This results in problem areas such as:

1 . Non-linear capacity characteristics as a function of the number of cores or perhaps even processors/servers when shared resources are involved, amount of system memory, Input/Output (I/O) throughput, etc. This issue can make it very difficult to manage a system in an effective and/or efficient manner.

2. Lack of methods to address the bottlenecks caused by hardware

configuration that may not match the specific characteristics of the software applications that run on a specific cloud platform. An example of this could be when applications with similar characteristics (e.g., Central Processing Unit (CPU) -intensive) are co-located on the same hardware platform. In this case the CPU resource will be the bottleneck long before other resources such as memory and I/O are exhausted.

3. Standardized artifacts that support the configuration and enforcement during the Lifecycle Management (LCM) of Virtualized Network Functions (VNFs) and their components (VNF Components (VNFCs)) running on various runtime environments exist (Reference: "Network Functions Virtualization (NFV); Management and Orchestration; VNF Packaging Specification," ETSI GS NFV-IFA 01 1 V2.1 .1 , October 2016). However, there are not many known methods or algorithms that determine how to optimally select the virtual resources needed to offer optimal performance for a given VNFC of a VNF during the various traffic patterns in the operation cycle of the VNF.

[0030] For example, in the virtual compute space, the magnitude of the problems highlighted above is likely to grow further by the trend in technology towards significantly higher number of cores, since it can give rise to the need, especially in cloud architectures, to manage resource allocation at the granularity of cores (which, for the reasons described above, can lead to severe non- linearity in capacity performance of software applications).

[0031 ] The issue of performance optimization in the context of cloud technology has received attention in the industry, and a few methods have been proposed to deal with certain aspects of the problem. However, the known approaches are inadequate to deal with the full scope of the concerns highlighted above. A few such methods are listed below:

• US Patent No. 9,246,840 B2 proposes to deal with the issue of

performance optimization by scaling a workload up (i.e., adding resources) and out (i.e., creating new instances of the application), and then selecting the configuration that leads to the best result. This approach can improve performance efficiency, but at the expense of an initial duplication of resource consumption and associated overhead. Furthermore, it does not yield insight that can help with the predictability aspect. Consequently, it deprives the orchestration nodes from the knowledge that could help select scaling strategies with predictable outcomes in terms of targeted service-level objectives or co-location of a mix of workloads that might most effectively consume the allocated resources.

• International Application Publication No. WO 2012/165937 A1 defines a procedure to benchmark the performance of virtual machines of different flavors against specific physical resources. However, this approach is couched on the assessment of generic aspects of virtual machine performance. As such, it ignores rather than addresses the problem of the non-linearity and unpredictability of the performance of specific applications - that might be deployed on the virtual machine - as a function of the quantity of allocated resources; no action is taken to tailor the virtual machine to the characteristics of a specific software application and a target service-level objective.

• US Patent No. 9,124,498 B2 teaches the benchmarking of a software application, as well as candidate cloud infrastructure targets for the purpose of determining the cloud provider with the lowest cost for the intended throughput. However, the task of optimization is aimed at selecting from among fixed virtual machines with predetermined allocated resources, rather than determining the most effective/efficient combination of infrastructure resources for a target level of throughput for a given software application. Also, the described method explicitly uses a linear capacity model, thus ignoring the non-linearity of performance as a function of the quantity of resources.

[0032] Scientific articles can also be found on related topics, but they do not appear to consider the characteristics of the target application performance as a function of the configuration of a given computational platform and, more specifically, performance variation as a function of the number of allocated processor cores. Also, they assume a linear model of performance as a function of resources, failing to consider the cases where too many cores allocated to a task may contribute to the violation of the service-level objective. An example of such a paper is:

• S. Singh et al., "STAR: SLA-Aware Autonomic Management of Cloud Resources," IEEE Transactions on Cloud Computing, Vol. PP, Issue 99, January 5, 2017.

[0033] Systems and methods are disclosed herein for addressing the aforementioned problems. In some embodiments, the proposed solution comprises a method for systematic characterization of specific aspects of performance of target applications operating on specific infrastructure, and the generation of metadata to represent those characteristics. Furthermore, the metadata is to be captured in the form of descriptors (or appended to existing descriptors) that can be accessed by management and orchestration nodes in cloud computing environments to provide the guidance necessary to achieve efficient and optimal deployment and scaling of the target applications. This can be done using existing standard descriptors, such as the VNF Descriptor (VNFD) specifically by defining several deployment flavor templates of the VNF, to include the appropriate settings as determined by the methods disclosed herein. The appropriate deployment flavor template to be used would then be updated or a new one created during the VNF LCM triggered by different criteria such as threshold of traffic, threshold of CPU load in one Virtualization Deployment Unit (VDU), etc.

[0034] Alternatively, to support automated updates of the values in the descriptors, a new data model is possible and is more appropriate for the VNFD and the deployment flavor to allow a more flexible addition of new

characterization profiles as the resource usage refinement over time is possible based on the continually gathered data.

[0035] Accordingly, in some embodiments, the proposed solution introduces mechanisms to address the main problem areas comprising functions that provide:

• A characterization mechanism comprising functions that make it

possible to detect inconsistencies of the capacity performance of a target application as a function of the amount of resources that it consumes. The characterization mechanism is used to produce:

o Mapping between service-level objectives (e.g., capacity or service volume) and allocated resources (e.g., number of processor cores, memory, etc.).

o Mapping between selected performance metrics (e.g., delay, packet jitter) and allocated resources.

• A mechanism to automatically generate a descriptor containing

metadata derived from the characterization process to be used by orchestration/scheduling nodes for resource and/or performance optimization.

• A mechanism for the use of the descriptor metadata for deployment of specific applications in a manner that yields resource utilization efficiency and/or optimizes selected aspects of performance.

[0036] These functions can be used in combination to provide a solution for deployment and scaling of applications in a distributed cloud environment. [0037] The described mechanisms provide a solution to address nonlinear application performance characteristics in a cloud environment. The solution provides the means for an orchestration solution to use the metadata from the descriptor for guidance in the optimal deployment of an application on a specific cloud platform. Since the application characteristics are generally unknown to the orchestrator, it is impossible to know how many resources are required to perform at a specific level. The problem also occurs when scaling an application since there is typically a limit as to how many resources the application can make use of before hitting a bottleneck.

[0038] The solution described herein offers a number of advantages over existing methods, including:

• Rather than inefficient usage of fixed virtual machine (or containers) flavors (with predetermined allocation of resources), the solution allows the virtual machine (or container) to be tailored for a close fit to the application, leading to savings in energy consumption and minimization of the resource footprint.

• Provides the means for trading off efficiency versus performance in cases where service-level objectives require priority.

• Can be used to assist detection of performance issues and enabled to highlight deltas from the expected performance.

• Support the goal of reducing Operational Expenditure (OPEX) and Capital Expenditure (CAPEX) via an automated flow that determines when a different level of virtualized resources is needed by any of the VNFCs and then provides the adequate settings enabling optimal performance of the virtualized resources.

• Provide predictable performance.

[0039] Embodiments of the present disclosure provide a flow that includes gathering characterization data, composing updates of the appropriate metadata, inserting them in descriptors (such as VNFD, or at lower level), and lastly switching them into operation. [0040] Embodiments of the proposed solution include gathering

characterization data (e.g., Key Performance Indicator (KPI) data, e.g., for different load conditions and/or configuration parameter values) for a target application running on a specific infrastructure (i.e., hardware and software platform), generating metadata (e.g., descriptor files such as, e.g., VNFDs or lower level descriptors) based on the characterization data where the metadata characterize one or more aspects (e.g., capacity, delay, or the like) of the performance of the target application as a function of specific configuration parameters, and utilizing the descriptor files.

[0041] In this regard, Figure 1 illustrates one example of a system 10 implementing aspects of the present disclosure, using a single system in the examples. However, the same characterization method can be applied to multiple servers and the collective data can be used. As illustrated, the system 10 includes a characterization cradle 12, a characterization data generator 14, and a characterization test controller 16. The characterization cradle 12 includes a virtualization environment 18 running on a server 20 (i.e., a server computer). The server 20 is implemented in a combination of hardware and software, as will be appreciated by one of skill in the art. The virtualization environment 18 is implemented as software that is running on (i.e., executing on) the server 20. A target application 22 is running in the virtualization environment 18.

[0042] The characterization test controller 16 is implemented in a

combination of hardware and software (e.g., implemented as software and hardware including one or more processors that execute the software). The characterization test controller 16 includes a test environment configuration function 24, a target application configuration function 26, and a load generator configuration function 28, each of which is preferably implemented in software.

[0043] The characterization data generator 14 is implemented in hardware or a combination of hardware and software. The characterization data generator 14 includes a load generator 30 and a KPI data collector 32 that operates to provide characterization data 34. [0044] Note that, in some embodiments, the characterization test controller 16 and the characterization data generator 14 are implemented on the same node (e.g., the same network node) while in other embodiments they are implemented on separate nodes (e.g., separate network nodes). Further, any component of the characterization test controller 16 and the characterization data generator 14 may be implemented on the same node as the other components or on a different node than the other components. In other words, while the characterization data generator 14 and the characterization test controller 16 are shown in their own "blocks" in Figure 1 , they may be implemented in one or more nodes (e.g., network nodes) in any desired manner.

[0045] Figure 2 is a flow chart that illustrates a process for gathering characterization data for the target application 22 running on a specific infrastructure (i.e., hardware and software platform provided by the server 20 and the virtualization environment 18), generating metadata based on the characterization data where the metadata characterize one or more aspects (e.g., capacity, delay, or the like) of the performance of the target application 22 as a function of specific configuration parameters for the server 20, the virtualization environment 18, and/or the target application 22, and utilizing the descriptor files. The process is performed by the system 10 of Figure 1 .

[0046] As illustrated in Figure 2, the characterization data generator 14 generates characterization data for the target application 22 running on the hardware and software platform provided by the server 20 and the virtualization environment 18 (step 100). As discussed below, the characterization data includes characterization data (e.g., KPI values) for one or more characteristics of interest for the target application 22 running as a virtualized function on the specific infrastructure provided by the server 20 and the virtualization environment 18. As discussed below, the characterization data generator 14 analyzes, or processes, the generated characterization data to provide metadata (e.g., in the form of descriptor files) that characterize one or more aspects of the performance of the target application 22 as a function of specific configuration parameters for the server 20 and/or the virtualized environment 18 (step 102).

[0047] In automated deployments, the metadata (e.g., descriptor files) are utilized (step 104). For example, the metadata may be provided to and used by other servers and/or virtualized environments and used to improve performance for the target application 22 when running on those platforms. The metadata (e.g., descriptor files) can be read by a cloud computing environment's management and orchestration nodes and be used to optimize specific aspects of the target application performance, such as resource utilization efficiency and/or key performance indicators of interest (e.g., delay, packet jitter).

[0048] Each of the steps of the process of Figure 2 are discussed in more detail below.

Step 100: Generation of Characterization Data

[0049] Again, Figure 1 depicts the elements of a framework, and their respective relationships, intended to extract characteristics of interest for a target application running on a specific infrastructure. The characterization process does not assume any knowledge of the internal design of the target application 22. The main elements of the off-line characterization framework are described below. In addition to the off-line characterization, the solution also comprises an online characterization process. This process allows for dynamic changes to the characterization data due to, e.g., changes to the load profile or changes/updates to the application.

1 . Characterization cradle 12

The characterization cradle 12 is a system platform whose complete hardware and software specification (e.g., processor type, clock speed, operating system type and version, etc.) is especially selected and/or configured to match the type of the infrastructure, or one of a set of likely infrastructures, that the target application 22 will be deployed on. The target application 22 is deployed on the characterization cradle 12, configured, and subjected to a set of tests intended to exercise it in a manner simulating real in-service environment for the target application 22. The purpose of the exercise is to produce data to characterize aspects of performance that is required in order to properly dimension the application. Examples of such

characteristics include:

• Maximal capacity as a function of the type and quantity of resources allocated to the target application 22 (e.g., the number of cores, amount, and/or type of memory, network bandwidth, etc.).

• Specific KPIs such as response time and/or processing delay, packet jitter, etc.

Characterization data generator 14

The characterization data generator 14 performs two functions: The first is to emulate the source(s) of load that the target application 22 is designed to service. The second is to collect all relevant application performance data and store it for subsequent analysis. The characterization data generator 14 comprises the following components:

• Load generator 30

The load generator 30 emulates the service demand to which the target application 22 is subjected during regular service. Naturally, the type of load depends on the target application 22. Examples include audio and/or video streams for applications that perform channel processing, or a set of messages emulating service requests for specific protocols (e.g., Session Initiation Protocol (SIP)). In each case the load generator 30 is configured to generate loads with the appropriate service mix. Furthermore, depending on the desired characterization, the load level may be controlled according to a preset trajectory in order to obtain performance information as a function of the load level. While interacting to provide a load on the target application 22, the load generator 30 also captures performance data such as response time, processing delay, packet jitter, packet loss, etc. Since an important objective of the use of the load generator 30 is to exercise the target application 22 under stress, the load generator 30 shall be able to ensure that it does not introduce any bottleneck during the tests. This could for instance be insufficient power in the load generator platform which could require the usage of multiple load generator instances. It should also be able to detect bottlenecks between the load generator 30 and the target platform which could be the result of network issues and/or network equipment limitations.

• KPI data collector 32

The KPI data collector 32 accumulates all performance data of interest during the characterization exercises. The data, collected from the load generator 30, the target application 22, and the server/virtualization environment 20/18, is stored in a coherent manner so as to retain the relationship between the KPI data, performance data from the server/virtualization environment 20/18, and the specific load level. The target application 22 and the characterization cradle 12 configuration settings that produced the collected KPI are also captured. The collected data is stored in a characterization data file for further processing.

Characterization test controller 16

The characterization test controller 16 embodies the control element in the application characterization process, using the following components: • Test environment configuration function 24

This component handles the configuration of the

infrastructure, including both the hardware and the software environment in accordance with the objectives of the characterization. This is done to configure the server 20 for the desired parameters such as the number of allocated processor cores, the amount of memory, disk space, network bandwidth, etc. The configuration also includes the type of operating system and its relevant parameters, as well as the set-up of the virtualization environment.

• Target application configuration function 26

This component is intended to ensure the proper

configuration of the target configuration parameters in compliance with the requirements of the service and the application.

• Load generator configuration function 28

This component configures the load generator 30 to generate the characterization data of interest. Accordingly, the load generator 30 will be set up to generate the desired load profile or load profile mixtures, with the load level set to predetermined levels, or varied based on a predetermined trajectory designed to obtain the specific characterization objectives.

In order to generate the desired characterization, it may be necessary for the characterization test controller 16 to instigate several iterations of tests, varying the configuration parameters as necessary to obtain the characterization data over the complete range of desired parameters and variables. This could be performed, for example, by executing appropriate scripts.

[0050] Thus, as described above, the characterization data is generated as follows. The characterization test controller 16, and in particular the test environment configuration function 24 and/or the target application configuration function 26, configures one or more configuration parameters of the target application 22 and/or the infrastructure (e.g., the server 20) with a first set of configuration values. As discussed above, these configuration parameters can include parameters such as, for example, number of processor cores allocated to the target application 22, an amount of memory allocated for the target application 22, an amount of disk space allocated for the target application 22, and/or network bandwidth allocated for the target application 22. The

configuration parameters can additionally or alternatively include software parameters such as, for example, the particular operating system used in the virtualization environment 18 and/or one or more configuration parameters of the target application 22. As used herein, a "configuration parameter value" is a value for a particular configuration parameter (e.g., if the configuration parameter is the number of processor cores allocated to the target application 22, then the configuration parameter value is a positive integer number in the range of 1 to some maximum number of processor cores). Likewise, a "set of configuration parameter values" for the configuration parameters means one particular set of configuration values for the configuration parameters (e.g., number of processing cores allocated for the target application = X, amount of memory allocated for the target application 22 = Y, operating system = Windows 10, etc.). Different sets of configuration parameter values for the configuration parameters are sometimes referred to herein simply as "different configurations." While the first set of configuration parameter values are applied, the load generator 30 subjects the target application 22 to a number of different load levels (e.g., a load profile) and resulting characterization data is collected by the KPI data collector 32 and stored. In some embodiments, this process is repeated for one or more additional sets of configuration parameter values. In this manner,

characterization data is collected for each of a number of different sets of configuration parameter values for the configuration parameters of the target application 22 and/or the infrastructure over varying load levels on the target application 22. Step 102: Generating Metadata (e.g., Descriptor Files)

[0051] Once the characterization tests are completed, the descriptor generator (Figure 3) retrieves the raw characterization data 34, and processes it to produce a set of data records (metadata) that characterize the different aspects of the target application performance as a function of specific

configuration parameters, such as the characterization cradle 12 settings, the quantity and/or type of resources allocated for the service, etc.

[0052] In order to illustrate the generation of the metadata (e.g., descriptor files), two examples are provided. However, these examples are only two of the many possible examples and, as such, are not to be construed as limiting the scope of the present disclosure.

[0053] Example 1: Capacity as a Function of the Number of Allocated

Processor Cores. This example is selected due to the commercial importance of capacity. More specifically, in this case it is assumed that the objective of characterization is to obtain the maximum load level that can be handled by a target application running on a specific infrastructure, as a function of the number of allocated processor cores. The motivation would be to obtain data that can help improve the efficiency of providing the service that is realized by the target application 22.

[0054] In this case, the characterization cradle 12 will first be set up with the type of computing platform and virtualization environment that is used in the data center where the application is to be deployed. If the data center employs different servers, the exercise is to be repeated with each of the server types in order to ensure that the characterization data is available for any of the possible servers. For a given server, the test environment configuration function 24 configures the server 20 and the virtualization environment 18 in accordance with the expected operational parameters of the data center. Next, the target application 22 is deployed in the virtual environment 18 and configured for the intended service. Finally, the load generator configuration function 28 configures the load generator 30 in accordance with the specific type of desired characterization data. In this case, the load generator 30 has to be provided with the procedure to generate the type of load appropriate for the target application 22. The procedure, which is likely to be a computer program, is loaded onto the load generator 30. Furthermore, the program is configured to generate a load whose characteristics are predetermined to contain an appropriate profile (or a mix of profiles), and whose level is increased to span a range that is likely to include (and exceed) the maximal capability of the target application 22 as operated on the server 20 / virtualization environment 18 combination.

[0055] In the case of the present example, the test is started with a server configuration that has a single processor core allocated to the target application 22. During the test, the KPI data collector 32 collects and stores all relevant data obtained from the load generator 30, the target application 22, and the server 20 / virtualization environment 18. Once the tests are completed, the descriptor generator (Figure 3) processes the stored test results (i.e., the characterization data) to determine the maximum load level at which the target application 22 performed without failure, as determined by the examination of the combination of recorded performance data. This level is identified by the descriptor generator as the capacity of the target application 22 when a single processor core is allocated, and registered as such in the metadata.

[0056] The procedure above is repeated, this time with the server 20 / virtualization environment 18 configured to allocate two processor cores to the target application 22. The number of allocated processor cores is incremented in further iterations of the test to produce the capacity characteristics of the target application 22 as a function of the number of allocated processor cores.

[0057] Once the tests spanning the desired number of allocated processor cores are completed and the results are processed, the descriptor generator compiles the results in the form of records (e.g., descriptor files) that capture the essence of the characterization data.

[0058] The solid line in Figure 4 represents an example of the capacity of a target application 22 running on an 8-core server as a function of the number of allocated processor cores. The descriptor generator can convert these characteristics into data records (e.g., descriptor files) for subsequent usage by a data center's management and orchestration nodes. Such a "descriptor" could contain information such as:

1 . The identification of the processor used in the characterization process and its configuration information.

2. The operating system and the virtualization environment specifications.

3. The type of characterization information (e.g. "Capacity as a function of allocated processor cores").

4. A table to map the maximum capacity of the target application 22 for a different number of allocated processor cores.

Note that the terms "descriptor" and "metadata" are used interchangeably in this document, but they both refer to the artifact that contains the information elements that are set with values obtained from the generation of the

characterization data step.

[0059] The dashed and the dotted lines in Figure 4 are intended to highlight other types of potentially useful information that can be derived from the characterization data. The dashed line shows the trend of capacity as a function of the number of allocated processor cores derived from the first three data points: it shows that capacity goes up linearly for up to three allocated processor cores. However, starting with the fourth added processor core, the amount of the increase in capacity is diminished. Similarly, the dotted line shows that the incremental capacity increase due to the fifth added processor core is still less than the corresponding amount for the fourth core. Finally, the data shows that allocating more than five cores does not lead to any further capacity increase. This information can be inserted explicitly in the descriptor to indicate the points beyond which the usage of additional processor cores is less than 100% efficient (e.g., the fourth and the fifth cores), and the point beyond which added capacity requires scaling out rather than scaling up. This information can either be left to the management and orchestration nodes to derive from the descriptor information described above or, optionally, it can be added to the metadata records in order to reduce the on-line computational burden on those nodes. [0060] The example in this subsection used the number of allocated processor cores as the variable affecting capacity. In a similar manner, capacity may be characterized as a function of other resources such as the amount of memory, disk space, network bandwidth, etc. (or selected combinations thereof).

[0061 ] Example 2: Delay (or Response Time) as a Function of Capacity and Number of Allocated Processor Cores. Delay characteristics are used herein as an example of a metric that may affect performance-related service-level commitments. To be more specific, the objective is assumed to be the characterization of the target application delay (or response time, depending on the nature of the target application 22) as a function of the number of allocated processor cores for the entire capacity range that can be delivered by the selected server 20 / virtualization environment 18 combination.

[0062] The procedure to obtain the delay characterization data is somewhat similar to the previous example; the characterization cradle 12, the target application 22, and the load generator 30 are configured as before. Just as in the case of the previous example, the test is started with a server configuration that has a single processor core allocated to the target application 22. In iterative steps, the load generator 30 applies an increasing level of load to the target application 22, measuring the delay (or response time) and delivering it to the KPI data collector 32 for each load level. The process is repeated, each time allocating one more processor core to the target application 22.

[0063] Figure 5 illustrates the type of data that is generated as a result of this characterization activity for the same processor and the same target application 22 used in the previous example. For each of the total number of processor cores allocated to the target application 22, delay is produced for different load levels all the way up to the maximum load level that can be supported by the combination of the allocated number of processor cores.

[0064] A descriptor containing the characterization information can be generated in the form of the following records:

1 . The identification of the processor used in the characterization process and its configuration information. 2. The operating system and the virtualization environment 18

specifications.

3. The type of characterization information (e.g. "Delay as a function of allocated processor cores for different load levels").

4. A table for each of the possible total number of allocated processor cores to map the delay to the load level.

[0065] Figure 5 shows the mapping between delay (or response time) and the load level for the different number of cores allocated to the target application 22. The mapping reveals information that can be used to manage the trade-off between meeting target KPI (delay in this case) and achieving operational efficiency. For example, Figure 5 shows that the capacity of 900 can be obtained by allocating four, five, six, seven, or eight processor cores to the target application 22. The most efficient usage of the processor would favor the allocation of the smallest number of cores (four, in this case) to the target application 22. However, the most efficient configuration also gives rise to the highest delay. The lowest delay could be achieved by allocating seven or eight processor cores. However, this would clearly come at the cost of lower processor efficiency. Finally, it is noted that there is no delay performance advantage in allocating eight cores rather than seven. This type of information can be left to be derived from the delay mapping tables by the management and orchestration nodes. Alternatively, records could be added to the descriptor to point to the lowest number of processor cores to be allocated over the possible range of load levels, for each of a set of target delay ranges.

[0066] In order to highlight a different form of using this type of

characterization information, Figure 6 depicts the mapping between delay (or response time) and the load level as a function of the number of allocated processor cores, but this time for a processor with only four cores. As in the case of the example above, it is noted that the allocation of all four processor cores to the target application 22 will allow the accommodation of a capacity of 900. However, if the delay, as registered in the appropriate descriptor, is beyond the service-level objective, the accommodation of this level of capacity will require scaling out. The proposed method allows to optimally combine the results obtained from the characterization data gathering with the required service-level objective. Step 104: Using the Descriptor

[0067] The current LCM operations defined for the standard VNFD as defined in ETSI NFV specifications (ref: IFA007, IFA008, IFA01 1 ) can be used to use the updated metadata in the VNFD.

[0068] For an updated VNFD model that allows continual addition/removal of deployment flavors containing the different characterization parameter settings as needed, updated operations on the interfaces may be needed (ref.

IFA007/008 and IFA005/006).

[0069] Figure 7 illustrates one example of a node 36 in which the

characterization cradle 12, the characterization data generator 14, the characterization test controller 16, and/or the descriptor generator (Figure 3) may be implemented. As illustrated, the node 36 includes at least one processor 38 (e.g., CPU(s)), memory 40, and a network interface 42. In some embodiments, at least some of the functionality of the characterization cradle 12, the characterization data generator 14, the characterization test controller 16, and/or the descriptor generator (Figure 3) is implemented in software that is stored in the memory 40 and executed by the processor(s) 38.

[0070] Figure 8 illustrates one example of a node 36 in which the

characterization cradle 12, the characterization data generator 14, the characterization test controller 16, and/or the descriptor generator (Figure 3) may be implemented. As illustrated, the node 36 includes one or more modules 44, each of which is implemented in software. The module(s) 44 operate to provide at least some of the functionality of the characterization cradle 12, the characterization data generator 14, the characterization test controller 16, and/or the descriptor generator (Figure 3). [0071] Embodiments of the proposed solution comprise a method for systematic characterization of specific aspects of performance of target applications operating on an infrastructure, the generation of metadata to represent those characteristics, and reflecting the results in metadata such as descriptors that can be used by management and orchestration nodes in telecom and Information Technology (IT) cloud environments.

[0072] The purpose of the proposed method is to achieve efficient and optimal deployment and scaling of the target applications, while taking into account both service efficiency and the required service-level objectives.

Example Embodiments

[0073] While not being limited thereto, some example embodiments of the present disclosure are provided below.

[0074] Embodiment 1 : A method comprising: generating (100)

characterization data that characterizes one or more characteristics of a target application (22) running as a virtualized function on a specific infrastructure; and generating (102) metadata, based on the characterization data, that

characterizes one or more performance aspects of the target application (22) as a function of one or more configuration parameters for the target application (22) and/or the infrastructure.

[0075] Embodiment 2: The method of embodiment 1 wherein generating (100) the characterization data comprises: generating a load on the target application (22); and collecting and storing the characterization data while the load is on the target application (22).

[0076] Embodiment 3: The method of embodiment 2 wherein the

characterization data comprises data regarding the load on the target application (22), data that characterizes one or more characteristics of the target application (22) while the load is on the target application (22), and data regarding one or more configuration parameters configured for the target application (22) and/or the infrastructure. [0077] Embodiment 4: The method of any one of embodiments 1 to 3 wherein the metadata comprises one or more descriptor files that can be read by management and orchestration nodes of a cloud computing environment.

[0078] Embodiment 5: The method of any one of embodiments 1 to 4 wherein the one or more performance aspects of the target application (22) comprise maximum capacity, and the metadata comprises data that defines maximum capacity of the target application (22) for two or more different configurations of the infrastructure.

[0079] Embodiment 6: The method of embodiment 5 wherein the two or more different configurations of the infrastructure comprise two or more different numbers of processor cores allocated for the target application (22).

[0080] Embodiment 7: The method of any one of embodiments 1 to 4 wherein the one or more performance aspects of the target application (22) comprise delay (or response time), and the metadata comprises data that defines delay (or response time) of the target application (22) as a function of capacity of the target application (22) and two or more different configurations of the infrastructure.

[0081] Embodiment s: The method of embodiment 7 wherein the two or more different configurations of the infrastructure comprise two or more different numbers of processor cores allocated for the target application (22).

[0082] Embodiment 9: The method of any one of embodiments 1 to 8 further comprising utilizing (104) the metadata.

[0083] Embodiment 10: A node (36) adapted to perform the method of any one of embodiments 1 to 9.

[0084] Embodiment 1 1 : A node (36) comprising: at least one processor (38); and memory (40) storing instructions executable by the at least one processor (38) whereby the node (36) is operable to perform the method of any one of embodiments 1 to 9.

[0085] Embodiment 12: A node (36) comprising: one or more modules (44) operable to perform the method of any one of embodiments 1 to 9. [0086] Embodiment 13: A computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according to any one of embodiments 1 to 9.

[0087] Embodiment 14: A carrier containing the computer program of embodiment 13, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium.

[0088] The following acronyms are used throughout this disclosure.

• CAPEX Capital Expenditure

• COTS Commercial Off The Shelf

• CPU Central Processing Unit

• I/O Input/Output

• IT Information Technology

• KPI Key Performance Indicator

• LCM Lifecycle Management

• NFV Network Functions Virtualization

• OPEX Operational Expenditure

• OS Operating System

• SIP Session Initiation Protocol

• SLA Service Level Agreement

• VDU Virtualization Deployment Unit

• VNF Virtualized Network Function

• VNFC Virtualized Network Function Component

• VNFD Virtualized Network Function Descriptor

[0089] Those skilled in the art will recognize improvements and modifications to the embodiments of the present disclosure. All such improvements and modifications are considered within the scope of the concepts disclosed herein.

Claims

What is claimed is:

1 . A method comprising:

generating (100) characterization data that characterizes one or more characteristics of a target application (22) running as a virtualized function on an infrastructure; and

generating (102) metadata, based on the characterization data, that characterizes one or more performance aspects of the target application (22) as a function of one or more configuration parameters for the target application (22) and/or the infrastructure.

2. The method of claim 1 wherein generating (100) the characterization data comprises:

generating a load on the target application (22); and

collecting and storing at least some of the characterization data while the load is on the target application (22).

3. The method of claim 2 wherein the characterization data comprises data regarding the load on the target application (22), data that characterizes one or more characteristics of the target application (22) while the load is on the target application (22), and data regarding configuration parameter values configured for the one or more configuration parameters for the target application (22) and/or the infrastructure.

4. The method of claim 1 wherein generating (100) the characterization data comprises:

configuring the one or more configuration parameters for the target application (22) and/or the infrastructure with a first set of configuration values; subjecting the target application (22) to a plurality of different load levels while the one or more configuration parameters for the target application (22) and/or the infrastructure are configured with the first set of configuration values; collecting first characterization data while subjecting the target application (22) to the plurality of different load levels while the one or more configuration parameters for the target application (22) and/or the infrastructure are configured with the first set of configuration values;

configuring the one or more configuration parameters for the target application (22) and/or the infrastructure with a second set of configuration values;

subjecting the target application (22) to a plurality of different load levels while the one or more configuration parameters for the target application (22) and/or the infrastructure are configured with the second set of configuration values; and

collecting second characterization data while subjecting the target application (22) to the plurality of different load levels while the one or more configuration parameters for the target application (22) and/or the infrastructure are configured with the second set of configuration values. 5. The method of claim 4 wherein generating (100) the characterization data further comprises:

repeating the steps of configuring, subjecting, and collecting for one or more additional sets of configuration values for the one or more configuration parameters for the target application (22) and/or the infrastructure to thereby obtain additional characterization data.

6. The method of claim 4 or 5 wherein each of the first characterization data and the second characterization data comprises:

data regarding the plurality of different load levels on the target application (22), data that characterizes the one or more characteristics of the target application (22) while the plurality of different load levels are on the target application (22), and data regarding the respective set of configuration values for the one or more configuration parameters for the target application (22) and/or the infrastructure. 7. The method of any one of claims 1 to 6 wherein the one or more configuration parameters comprise one or more hardware configuration parameters and/or one or more software configuration parameters.

8. The method of claim 7 wherein the one or more hardware configuration parameters comprise a number of processor cores allocated for the target application (22), an amount of memory allocated for the target application (22), an amount of disk space allocated for the target application (22), and/or network bandwidth allocated for the target application (22). 9. The method of claim 7 or 8 wherein the one or more software

configuration parameters comprise an operating system used in a virtualization environment in which the target application (22) is running as a virtual function and/or one or more parameters of the virtualization environment. 10. The method of any one of claims 1 to 9 wherein the metadata comprises one or more descriptor files that can be read by management and orchestration nodes of a cloud computing environment.

1 1 . The method of any one of claims 1 to 10 wherein the one or more performance aspects of the target application (22) comprise maximum capacity, and the metadata comprises data that defines maximum capacity of the target application (22) for two or more different sets of configuration values for the one or more configuration parameters for the target application (22) and/or the infrastructure.

12. The method of claim 1 1 wherein the two or more different sets of configuration values for the one or more configuration parameters for the target application (22) and/or the infrastructure comprise two or more different numbers of processor cores allocated for the target application (22).

13. The method of any one of claims 1 to 10 wherein the one or more performance aspects of the target application (22) comprise delay, and the metadata comprises data that defines delay of the target application (22) as a function of capacity of the target application (22) and two or more different sets of configuration values for the one or more configuration parameters for the target application (22) and/or the infrastructure.

14. The method of claim 13 wherein the two or more different sets of configuration values for the one or more configuration parameters for the target application (22) and/or the infrastructure comprise two or more different numbers of processor cores allocated for the target application (22).

15. The method of any one of claims 1 to 14 further comprising utilizing (104) the metadata.

16. The method of claim 15 wherein utilizing (104) the metadata comprises utilizing (104) the metadata to orchestrate and/or manage the target application (22) and/or an infrastructure of a cloud computing environment in which the target application (22) is running.

17. The method of claim 15 wherein the metadata is stored in one or more description files, and utilizing (104) the metadata comprises utilizing the one or more description files to orchestrate and/or manage the target application (22) and/or an infrastructure of a cloud computing environment in which the target application (22) is running.

18. A node (36) adapted to:

generate characterization data that characterizes one or more

characteristics of a target application (22) running as a virtualized function on an infrastructure; and

generate metadata, based on the characterization data, that characterizes one or more performance aspects of the target application (22) as a function of one or more configuration parameters for the target application (22) and/or the infrastructure. 19. The node (36) of claim 18 wherein the node (36) is further adapted to perform the method of any one of claims 2 to 17.

20. A node (36) comprising:

at least one processor (38); and

memory (40) storing instructions executable by the at least one processor

(38) whereby the node (36) is operable to:

generate characterization data that characterizes one or more characteristics of a target application (22) running as a virtualized function on an infrastructure; and

generate metadata, based on the characterization data, that characterizes one or more performance aspects of the target application (22) as a function of one or more configuration parameters for the target application (22) and/or the infrastructure. 21 . A node (36) comprising:

one or more modules (44) comprising:

a first generating module operable to generate characterization data that characterizes one or more characteristics of a target application

(22) running as a virtualized function on an infrastructure; and

a second generating module operable to generate metadata, based on the characterization data, that characterizes one or more performance aspects of the target application (22) as a function of one or more configuration parameters for the target application (22) and/or the infrastructure. 22. A computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according to any one of claims 1 to 17.

23. A carrier containing the computer program of claim 22, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium.