WO2010034608A1

WO2010034608A1 - System and method for configuration of processing clusters

Info

Publication number: WO2010034608A1
Application number: PCT/EP2009/061494
Authority: WO
Inventors: Babbis Xagorarakis
Original assignee: Telefonaktiebolaget LM Ericsson AB
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 2008-09-24
Filing date: 2009-09-04
Publication date: 2010-04-01
Anticipated expiration: 2011-03-24

Abstract

A processing cluster (40) comprises a plurality of processor nodes (43a, 43b, 44a, 44b, 44c) and a configuration management system. Each processor node comprises a configuration agent operable to apply configuration parameters provided by the configuration management system. The configuration management system provides the same configuration parameters obtained from a common set of configuration parameters to all the processor nodes in the processing cluster (40). A node, configured for operation as a member of a processing cluster may function as either a master node (43b) or as a slave node (43a, 44a, 44b, 44c). A master node receives notifications of cluster configuration parameters from the common set of configuration parameters, and distribute these to other members of the processing cluster (40).

Description

System and Method for Configuration of Processing Clusters

Field of the Invention

The present invention relates to a system and method for configuring processing clusters, in particular, but not exclusively, clusters of Java EE based application servers.

Background

Application service providers (ASPs) such as telecom operators face the challenge of diversified resource allocation. This challenge includes the task of grouping application services in a manageable way. There are several reasons to strive for this, which include: Security - to control service access;

Maintenance - it is easier to maintain an organized group of resources; Performance - concentrating and coordinating the allocation of resources; and

Economics - grouping of resources makes it easier to reduce the operating costs through overhead elimination.

The most common way to group application services is through a computer cluster. A computer cluster is a group of computers usually linked over a fast, local computer network. The application software components are distributed across the computers in the cluster in a predefined way.

A typical application processing cluster consists of a number of application server working instances, each containing a predefined or dynamically allocated set of software components. A typical Java EE cluster consists of a number of Java EE application server instances, each containing the same set of software components. This simplifies the process of component deployment over the specified cluster. As a consequence, different applications, requiring different component sets, are usually arranged in different clusters. Complex Java EE application services, such as multimedia applications, can be provided from either homogenous or heterogeneous cluster configurations. A homogenous Java EE cluster is a cluster where all processing server instances operate on the same hardware, use the same operating system and have the same set of software components installed. Figure 1 is a schematic illustration of a group (or constellation) of homogeneous clusters 10a - 1 Od, all configured for providing the same application service via a service network 12. The clusters 10a-IOd are managed by operation and maintenance modules 0AM1 and 0AM2 via an internal network 14 and these modules are also linked into the operator's network 16.

A heterogeneous Java EE cluster is a cluster where server instances may operate on different hardware, use different operating systems and/or with different sets of components. Figure 2 is a schematic illustration of a constellation of heterogeneous clusters. Clusters 1 , 2 and 3 each include components that provide an application service, Service 1 , while clusters 1 and 2 include components that also provide a second application service, Service 2. From the operator's point of view there are effectively 5 clusters - Cluster 1 /Service 1 20a, Cluster 1 /Service 2 20b, Cluster 2/Service 1 20c, Cluster 2/Service 2 2Od, and Cluster 3/Service 1 2Oe. The clusters 20a-e are linked in the same manner to the networks and operator and maintenance modules as in Figure 1.

Service providers realize provided services as physical nodes in the operating network 16. Figure 3 illustrates schematically how the operator network is arranged for the configuration of the clusters of Figure 2. Linked into the operator network 16 is a configuration node 30 as well as configuration storage 32. Controlling the operation and maintenance of the service nodes is carried out using operation and maintenance (OAM) software. The OAM software operates through the configuration node 30 accessing a cluster through the OAM network (as depicted in the figures by OAM modules 18a, 18b). The OAM software is usually built upon configuration mechanisms such as SNMP, LDAP, SOAP or even provider propriety configuration mechanisms. Service providers that provide, for example, Java EE-based applications must create a configuration link between the OAM modules and each cluster's own operation and maintenance mechanisms. The configuration has to be initiated by the operator and propagated towards the application server instances in the clusters. The configuration of an application server instance in a cluster involves specifying configuration parameters. For example, for a service instance operating as an Http server, the configuration parameters might include the IP address and Port, but also other configuration parameters relating to the servlets running on the Http server. These are usually common for a specific Java EE cluster. The configuration parameter set of a software component, or a subset of it, may be shared with other components. Examples of shared configuration parameters are the I P addresses and Ports of Naming servers or databases.

The configuration of a typical Java EE cluster is vendor-specific. Each Java EE vendor defines its own way to configure a Java EE cluster. The mechanism employed varies between use of propriety protocol solutions and combinations of known technologies, such as Java MBeans, JNDI, EJBs and use of Preferences API. This means that, in the case of heterogeneous cluster constellations, the system integrator in the operator network has to customize the configuration management system depending on the individual configuration requirements of each of the participating clusters. Also, for all systems, the sanity of the configuration has to be monitored for configurational consistency. This is often a complex task due to the configuration differences between the participating clusters.

Another problem is that for the majority of existing solutions, the configuration of Java EE clusters must be decided statically. That is, the members of the cluster group must be known in advance, and the configuration applied solely for that membership. Such a static configuration restriction complicates the initial system start-up, but also affects system upgrades and failure recovery because a new static configuration must be applied each time. It would be advantageous if configuration could be performed dynamically so that each time a new member joins the group, or an upgrade is applied, membership of the group could be determined through ad-hoc identification, and would not need to be known in advance. In addition to this, some software components, typically in long-lived, non-stop commercial applications, have a runtime changeable nature, and this means that all runtime configuration changes have to be consistently spread among the service instances in the cluster.

The complexity of the configuration setup on many cluster constellations translates into high maintenance costs. The present invention has been conceived with the foregoing in mind.

Summary of the Invention

According to a first aspect of the present invention there is provided a processing cluster comprising a plurality of processor nodes and a configuration management system. Each processor node comprises a configuration agent operable to apply configuration parameters provided by the configuration management system. The configuration management system provides the same configuration parameters obtained from a common set of configuration parameters to all the processor nodes in the processing cluster.

The processing cluster may be a constellation of a plurality of server clusters, each server cluster comprising a plurality of processor nodes. The server clusters may be Java EE clusters. The plurality of server clusters may include one or more heterogeneous clusters.

It is an advantage that the processing cluster provides a flexible and reliable run-time configuration management system for a heterogeneous set of processor clusters, such as JavaEE clusters. It is a further advantage that these cluster collections can be handled as a single virtual Java EE application cluster. The configuration concept described can be regarded as dynamic, since it supports expansion of the overall number of Service I nstances and Java EE clusters at operation time. With the configuration parameters being obtained from a common set of parameters, configuration can be performed dynamically each time a new member joins the group, or an upgrade is applied.

In embodiments, the nodes may comprise a master node and one or more slave nodes, wherein the master node coordinates the provision of configuration parameters to the slave nodes. The master node may be elected from among the nodes in a cluster in accordance with an election algorithm.

In embodiments, the processing cluster is configured to provide a virtual configuration access area for JavaEE applications. The processing cluster may be configured to define rules for updates to, access to, and membership of the access area. The applications may share portions that have common configurations.

In embodiments, the processing cluster may comprise a cluster configuration agent. The cluster configuration agent may comprise a distribution service for distributing configuration parameters to nodes in a cluster.

In embodiments, the processing cluster may further comprise a persistent storage, wherein a current configuration set of configuration parameters is stored.

According to a second aspect of the present invention there is provided a server node configured for operation as a member of a processing cluster. The server node is operable to function as either a master node or as a slave node. When functioning as a master node, the server node is configured to receive notifications of cluster configuration parameters from a common set of configuration parameters, and to distribute the configuration parameters to other members of the processing cluster. When functioning as slave node, the server node is configured to receive configuration parameters from a master node and to implement the node's configuration in accordance with the configuration parameters.

In embodiments, the server node comprises a distribution service for distributing the configuration parameters, notifying configuration changes to the other members of the processing cluster, and receiving configuration parameters from a master node.

In embodiments, the server node may comprise a state machine that defines a logical state of each node and whereby state transitions define the protocol of the distribution service.

According to a third aspect of the present invention there is provided a method of configuring a processing cluster. The method comprises determining information relating to applications and configuration requirements for each of a plurality of nodes that make up the processing cluster. One of the nodes is elected to be a master node, in accordance with an election algorithm. The master node is notified of configuration parameters based on a common set of configuration parameters. The master node distributes the configuration parameters to all other nodes of the processing cluster. Every node in the processing cluster is configured in accordance with the configuration parameters.

Brief Description of the Drawings

Figure 1 is a schematic representation of a network that includes a constellation of homogeneous clusters.

Figure 2 is a schematic representation of a network that includes a constellation of heterogeneous clusters.

Figure 3 is a schematic representation of a network showing how the operator network is arranged for the constellation of clusters of Figure 2.

Figure 4 is a schematic representation of a constellation of a processing cluster.

Figure 5 is a schematic block diagram showing a component overview of a processing cluster constellation.

Figure 6 is a schematic block diagram illustrating the functional areas in a cluster. Figure 7 is a schematic illustration of a master node state machine. Figure 8 is a schematic illustration of a slave node state machine.

Figure 9 is a signal flow diagram for any early start-up procedure for creating a processing cluster.

Figure 10 is a signal flow diagram for a master node election process.

Figure 1 1 is a signal flow diagram for an application accessing a configuration parameter.

Figure 12 is a signal flow diagram showing a sequence of events for a notification of listener registration.

Figure 13 is a signal flow diagram for implementing configuration updates in a processing cluster.

Figure 14 is a signal flow diagram for implementing configuration updates in a slave node. Figure 15 is a signal flow diagram for update notifications that occur at a master node. Figure 16 is a signal flow diagram for update notifications that occur at a slave node.

Figure 17 is a signal flow diagram illustrating a sequence of events that take place after a master node failure.

Figure 18 is a flow chart illustrating the principle stages in a method of configuring a processing cluster.

Detailed Description

The following description relates specifically to examples using JavaEE clusters and the following are assumed to be provided by the infrastructure that will host the JavaEE clusters:

Persistent configuration management storage accessible from one or more of the individual server instances. Storage accesses are regarded as passive information accesses and the information that is accessed is controlled by a

JavaEE configuration management system. The persistent storage is interconnected to the northbound OAM interfaces. Configuration updates are controlled by the node administrator/operator.

A single point notification mechanism, which notifies a specified subscriber of configuration changes. This mechanism is the link between the JavaEE clusters and the external OAM infrastructure.

A well-defined and unambiguous interface with JavaEE applications. This interface will define the rules for the mapping of configuration parameters.

The following definitions are for terms used in the ensuing discussion.

Traffic or service network. A network intended to be used for application service traffic. Only customer services will be propagated through this network.

OAM network or Operator network. A network intended to be used for operation and maintenance of a processing cluster. The operator configuration directives will be propagated through this network. Internal network. A network intended for the internal communication traffic between instances of a processing cluster.

Server/Service instance. A Server can be seen as a collection of processing instances, or Service instances. Each service instance will be equivalent to a Java Virtual Machine (JVM) that executes Java EE cluster software. A Service instance is a member of a JavaEE cluster. A Server instance may be constituted of a single Service instance.

Node. A logical entity used for identification of a member in a processing cluster. A Node is the logical representation of a cluster entity. Its logic resides in a Server/Service instance, which contains the configuration software and is a recipient of a full copy of the configuration set. Each Node represents a member of a configuration cluster in the constellation. The realisation of a Node, a Server/Service instance will be resident on a Java Virtual Machine (JVM). There are two types of Node - the Master Node and Slave Nodes. The Master Node has the responsibility to coordinate configuration of all Slave Nodes. There is at most one Master Node in a processing cluster.

Processor or Payload Unit. A Processor will be a physical computer where one or several Service instances operate. For simplicity only a single Service instance will be presented in the diagrams. This is due to the complexity of the traffic when several service instances are allocated in a physical processor. The principles described herein are, however, valid for all such traffic cases.

Java EE Cluster. A collection of server/service instances where identical software is loaded and is operational.

Processing cluster. A collection of Java EE clusters. It is mainly a number of processors interconnected in a network. A processing cluster is defined by its application instance Node configuration.

Constellation. Consider a number of Java EE clusters that operate on a shared network. A constellation is the realisation of the software that defines one Master Node and several Slave Nodes sharing the same configuration. Each logical node is formed on top of a Server/Service instance from the participating Java EE Clusters. As a result of this, a constellation spans over a number of JavaEE clusters. One simple example is a constellation consisting of 2+2 heterogeneous clusters where the first 2 Nodes are handling Operation And Maintenance requests (members of the OaM JavaEE cluster) while the other 2 Nodes are handling payload traffic (such as Http traffic and are members of the Traffic JavaEE cluster). In that constellation, there is one Java VM per computer where the Application and Configuration software operates. However, more complex constellations could, for example, have 2+4+4+4+4 cluster configurations (the numbers being the number of nodes in each cluster.

Figure 4 shows a constellation of a processing cluster 40 that consists of two Java EE clusters 41 , 42 and a total of five processors 43a, 43b, 44a, 44b, 44c. Each processor is a node that includes a service instance where the configuration software is loaded and where a copy of the configuration will be allocated in accordance with the above definitions, Two processors 43a and 43b reside in one cluster 41 , while the other three processors 44a-c reside in the other cluster 42. Note that the Node on processor 43b has been elected as the Master Node while the other Nodes are all Slave Nodes. The election of a master Node will be described in more detail below.

In order to achieve a better understanding of the mechanisms that will be described, a number of functional primitives, used as building blocks, are presented below with a brief description of the functionality of each.

Configuration link against Operator network. The software that enables the communication between the processing cluster and the operator, and configures the cluster.

Node discovery system. Application Service instances will be automatically recognized after their appearance in the network regardless of their JavaEE cluster membership. All application server instances will be participating in the same virtual cluster defined by their Nodes.

Master Node election. An algorithm that elects a privileged application server instance to be the Master Node. One of the application server instances is elected as the controlling Master Node. The Master Node is responsible for coordination of configuration management. The Master Node subscribes for configuration changes to the OAM's propagation channel from the operator (northbound). In case of failure, a new Master Node is elected.

Passive initial configuration load. All application server instances load their initial configuration directly from a database.

Active configuration changes distribution. The elected Master Node distributes the configuration updates to all other application server instances.

Sanity checks on all server instances. The Master Node performs configuration sanity checks on all other nodes. In the case of a cluster configuration inconsistency, the Master Node will order all other nodes to compare their configuration storage against the persistent storage. If differences are found corrections are applied.

APIs against Java EE applications. JavaEE applications reside in cluster service instances. Their configuration is accessed by an API's configuration access.

The basic idea is to handle the configuration of a collection of related Java EE clusters with a common set of configuration parameters and provide the means of distribution and flexible data handling to make them available in all types of operational conditions. This way we can handle a collection of related Java EE clusters as a single processing area. The term related refers here to the product of such a constellation, where the Java EE application nodes share a specific task, such as a multimedia management system.

Grouping several Java EE clusters is effectively the same as constructing a single heterogeneous cluster, a cluster where its processing instances are not equivalent. The constellation, or processing cluster, described below is a heterogeneous Java EE cluster from the point of view of its deployment, but will be effectively a homogenous Java EE cluster from the point of view of its configuration.

Furthermore, in the embodiments described, both clustering (i.e. the creation, or definition of a cluster) and cluster configuration can be seen as dynamic in that alterations can be made during runtime. Clusters may expand or shrink at runtime without impact on the management the system.

In the embodiments described below, in addition to the primitives described above, the following components and roles contribute to the solution. These are described in relation to Figure 5, which shows a component overview of an exemplary constellation with three processors 5OX, 5OY, 5OZ, referred to as Processor X, Processor Y, and Processor Z respectively.

Cluster Configuration Service 51. This is the software that acts as a proxy for the operator configuration node. Its purposes are to access the static configuration storage, to distribute the configuration parameters to the service cluster, and to communicate, and accept, configuration changes. The configuration service will only be linked to the Master Node through specialized communication APIs. The configuration service is designed to operate in several processors independently.

Configuration Storage. This is a storage entity that holds the configuration parameters. The operator network includes a Persistent Configuration Storage 52, from which data and notifications are propagated to the processors 5OX, 5OY, 5OZ via the operator IP network 52a. In addition each of the processors 5OX, 5OY, 5OZ has local storage 53X, 53Y, 53Z on which configuration parameters are stored locally.

Configuration Lifecycle Module 54X, 54Y, 54Z. This is the software realized in all service instances in the cluster. Its purpose is to host the Distribution internal service, and also to spread the configuration parameters to its allocating service instance.

Distribution Service 55X, 55Y, 55Z. This is a service that links all service instances. Depending on the Node type, its role is to spread, accept or notify about configuration changes over the service cluster.

Communication primitives. These are basic communication primitives, used as building blocks for inter-cluster communications. They are used by communications software 56X, 56Y, 56Z in the Lifecycle Modules. Configuration state 57X, 57Y, 57Z. This is a state machine that defines the logical state of each node. The state transitions define the protocol of the distribution service.

Configuration mirror. This is a local copy of the configuration. It resides locally in the local storage 53X, 53Y, 53Z in the configuration lifecycle modules 54X, 54Y, 54Z.

Application APIs 58X, 58Y, 58Z. An application accesses its configuration through simplified APIs. The implementation of these APIs is a part of the Distribution Service.

Applications 59X, 59Y, 59Z. On each processor 5OX, 5OY, 5OZ, these application specific services are instantiated.

Figure 6 shows the functional areas in a cluster. The configuration client 61 , configuration storage 62, node configuration agent 67 in the OAM module 64, and communications over the operator network 65 between these areas, as well as application-specific software in each node, are either parts of commonly existing infrastructure or pure application software. The cluster configuration agent 68 in the OAM module 64, and the node configuration agents 67A, 67B and cluster configuration agents 68A, 68B in each node (payload) 69A, 69B are the functional areas that hold the key to the configuration solution and the following description will be based around these areas.

The node configuration agents 67, 67A, 67B act as proxies between the service provider's configuration infrastructure and the cluster's configuration system through the cluster configuration agents 68, 68A, 68B. Their purpose is to mediate between the infrastructure already defined in the service provider's network and the cluster that hosts the Java EE application. The protocols used by the service provider's infrastructure are expected to be a collection of known OAM protocols such as OMG Corba, SNMP, or Netconf. The node configuration agents 67, 67A, 67 B store configuration data in a persistent database placed in the internal network (see item 52 in Figure 5). Specific APIs are contacted by the cluster configuration agents 68, 68A, 68B to promote initial configuration and configuration updates. The persistent configuration storage 52 is a database that stores the exposed (i.e. currently active) configuration set. Its only direct communication link is with the node configuration agents 67, 67A, 67B.

The cluster configuration agents 68, 68A, 68B act as proxies between the node configuration agents 67, 67A, 67B and the configuration infrastructure. The cluster configuration agents 68A, 68B in the payloads 66A, 66B reside inside a Java EE lifecycle module and this is done for the following reasons:

To promote and control the infrastructure lifecycle as an integral part of the Java EE application server/service. Residing inside a Java EE lifecycle module, the cluster configuration agents control the start-up of the configuration but also ensure that the configuration is aligned with other built-in application services.

To prevent configuration access before a service state has been specified. All configuration parameters must be loaded before potential users, such as Java

EE applications, can access them. This is to avoid configuration errors arising due to absence of configuration data.

The cluster configuration agent makes use of the distribution service to spread the configuration set and its updates over the application cluster. The lifecycle module that contains the cluster configuration agent must be deployed in all service instances in the processing cluster.

The distribution service is the component that sets the grounds for the cluster configuration service. Its main task is to distribute the initial configuration and configuration updates in a reliable and efficient way. The distribution service is the component that embodies the role of a Node as previously described. It coordinates the Node behaviour depending on its type (Master/Slave), by following the Node State transitions which will be described below.

The distribution service has dependencies against the following components: Communication primitives. These provide the cluster communication basics to define a group membership function, broadcast a Node's group over the network and perform reliable multicasting of group messages.

Node state machine. This component defines a deterministic finite automat (DFA), which represents the different states of a Node. Its structure depends on the type of the Node (Master or Slave) while its state transitions define the behaviour of the Node.

Master node election algorithm. This component is used to decide if a

Node should be awarded the Master status or not, depending on certain parameters such as the number of known members in the cluster, start-up times and known IP addresses. The Master node will have the privilege of registering with the node configuration service as the main recipient of configuration updates. All incoming updates will be delivered from the Master

Node to all known slave nodes that appear in the cluster.

Local configuration storage. This component is basically a hash table that stores all configuration parameters in their final format, that is determined by the mapping within the Java EE application configuration API.

Java EE application configuration API. This is the API against Java EE applications. It consists of a parameter mapping part and a notification listener part. The parameter mapping API (not defined herein) will have to be defined to agree with conventional Java types (e.g. the C string type will have to be mapped to Java. lang. String object type). The distribution service provides the implementation of this API.

The node state machine provides two important logical entities of the cluster configuration agent, a condition indicator, the state, and the actions consumed that allow conditions to be altered, the state transitions. There are also actions that do not alter the condition of the state machine, the state consumptions. The distribution service completely relies on the Node state machine to handle the configuration of all Service instances in the processing cluster. The node state machine can be seen as the behaviour, the stack and the protocol of the distribution service.

For simplicity, there are two separate DFAs presented here to describe the state machine of a Node, a DFA for the Master Node and another one for the Slave node. In reality there is a single DFA for both, merging the two DFAs. This is because a Node is allowed to change states, a Slave Node can become a Master Node and vice versa.

Figures 7 and 8 are state machine diagrams for the Master Node State Machine, and the Slave Node State Machine respectively. Each depicts the same set of 5 states:

Init, or Start-up State 71 : The Node is initiated; Operational State 72: Applying updates locally and distributing; Alert State 73: Incoming update while processing other;

Master State 74: Receiving Mastership; Termination State 75: The Node terminates.

In addition both state machines involve the following state transitions and state consumptions, although certain of the transitions occur between different states and certain of the consumptions occur at different states.

State Transitions Tx

T1 : The node is not elected as Master T2: A new update arrive while still under update processing, the update is queued

T3: Update queue is empty

T4: Received Mastership.

T5: Received Mastership.. T6: Normal termination

T7: Termination under inconsistency.

T8: Termination from Mastership.

State Consumptions Cx

C1 : Update is applied locally C2: New updates are queued C3: Update queue too big, drop it and run Local consistency test. C4: Global Consistency checks.

The Master Node state machine represents the behaviour of a Node (Service Instance component) that is elected to be a Master Node. As a result of the influence of the node and its state machine in the distribution service, the following events can be observed:

• Startup state 71. The distribution service reads its initial configuration and loads the local storage with configuration data.

• Operational state 72. The Node is elected to be master. The distribution service will register to the node administration service in order to receive configuration updates. Consistency check messages are sent to all other service instances (slaves), who are forced to compare their local storage with the persistent storage for consistency.

• Distribution state. The cluster configuration agent receives updates from northbound; the updates are verified and loaded to the local configuration storage. Notifications are sent to application components that have registered as listeners for configuration updates. The distribution service forwards the updates to all other members in the cluster and waits for acknowledgement. When no outgoing messages exist, the Distribution state is transitioned to

Operational state. Note that the Distribution State is not shown as a separate state in Fig 7. This is because Fig 7 is only the state machine of a Node, which, for simplicity is being described as a separate DFA. The distribution state is a state of the overall DFA, and there is a state machine of the constellation but it is far too complex to show or describe.

• Alert state 73. Some or all of the update messages sent, either become timed out or failed (communication error). All timed out or failed update messages are resent. When all messages reach the destination, the Alert state is transitioned to Distribution state.

• Termination state 75. The Cluster configuration agent is terminated either normally (service instance termination) or abnormally (failure). The Slave Node state machine represents the behaviour of a Node that is not elected to be a Master Node. As a result of the influence of the node and its state machine in the distribution service, the following events can be observed:

• Operational state 72. The Node is not elected to be master. Configuration updates are received from the Master node. They are loaded to local storage and notifications are sent to application components that have registered as listeners for configuration updates.

• Alert state 73. While processing configuration updates, new updates arrive. These are queued in a FIFO pattern and processed when previous updates are applied. When the FIFO queue is empty (all updates are applied), the Alert state is transitioned to Operational state.

• Master state 74. The recent Master Node is terminated and the Node becomes elected as the new Master Node in the cluster. This state is identical to the Operation state of the Master Node. From now own the DFA is transformed to a Master Node state.

• Termination state 75. The Cluster configuration agent is terminated either normally (service instance termination) or abnormally (failure).

For present purposes, to avoid losing focus on the principles, the application configuration API may be considered as a black-box component. However, for illustrative purposes only, a recommendation of a possible way to construct an API against Java EE applications is provided below, with reservations regarding individual Java EE application needs.

A Java EE application should rely on standard API's to receive its configuration set. The building blocks for these API's should be based on commonly accepted technologies. There are several standard technologies which qualify for this, for example: • JMX Management Beans

• Enterprise Java Beans

• Java Preferences

• Java Persistence Management Each of these have both advantages and disadvantages.

The Java EE application API's should be implemented by the cluster configuration agent to promote the configuration set and its updates. One common technology used as a building block for configuration propagation is Enterprise Java Beans (EJBs). There are several advantages of this:

• Natural components. EJBs are native/natural building blocks for Java EE applications

• Context separation. Separation of business logic from user interface (Ul) is one of EJBs main features.

• Naming. Naming and registration primitives are available for EJBs as Java EE components.

• Security. Declarative security and access control.

The following paragraphs will provide a description of the most important use cases. In these use cases described, the components of a node are identified using reference numerals that correspond with numerals used in Figures 5 and 6.

Early startup At early startup, the configuration storage is mirrored locally to make it available for Java EE applications. It is important that this is done before startup of Java EE applications or even other subsystems that rely on the configuration data. If this convention is not followed, errors will occur by lack of configuration data. Figure 9 is a signal diagram illustrating the sequence of events. At steps 901 and 902, each cluster configuration agent 68 connects to the node configuration agent 67. The configuration data is actively loaded through the connection link between them. At step 903 the cluster configuration agent 68 creates, or initiates, the distribution service 55. At step 904 the cluster configuration agent 68 creates local storage 53. At steps 905 and 906 the Distribution service 55 sets the current state of the state machine to Init, and once this is done, at steps 907 to 910 the configuration data set is read from the Node Configuration Agent 67. At steps 911 and 912 the Distribution Service 55 maps and creates a data representation of the configuration set which is loaded to the local storage 53. The storage format complies with the data representation according to the provided application API's. Finally, at step 913 the Distribution Service is ready to initiate the configuration APIs.

Master node election

As previously mentioned, the configuration state of each service instance depends on the type of the node incorporated. Since there will be a single master node instance in the processing cluster, this must be done in an organized and secure way to avoid ambiguities. When a master node is elected, all other nodes in the cluster are considered as slave nodes.

This use case arises after early startup, when configuration data is locally mirrored. When the configuration is loaded, the distribution service subscribes to a predefined broadcast channel and sends broadcast messages over the internal network to announce its membership to the configuration group of the processing cluster. In parallel it listens on incoming messages on the same channel. All group membership messages from other nodes are listed to create a picture of the cluster. Figure 10 is a signal diagram illustrating the sequence of events when a node, at step 1001 , joins a cluster. Before it can declare its membership, it accesses the communication primitives 100 at steps 1002 and 1003 in order to create the resources (building blocks) that it will need, and at steps 1004 and 1005 to establish that it has joined the group. At this stage it does not have any information about other members of the cluster group and so, at step 1006 the communication primitives 100 respond with the information that there are no other members in the group. In general, steps 1004 to 1006 are the steps where members identify each other, through propagation of group membership.

Before it obtains information about any other members in the cluster group, the node assumes that it is itself a master node and registers to the node configuration agent 67 to receive configuration updates. This is shown at steps 1007 - 1012. The State Machine 70 is placed into the Master state. Then, at step 1009 the Distribution service 55 declares mastership of the cluster group to the cluster configuration agent 68, and at step 1010 the cluster configuration agent notifies the node configuration service 67 of the mastership. These notifications are confirmed back to the distribution service 55 at steps 1011 and 1012.

At step 1013, the node is informed of other nodes that join the cluster. For each such other node, the current master election algorithm is called to determine the cluster master node. The mechanism that calls the election algorithm is triggered when new members are visible in the network and is part of the cluster configuration software infrastructure. Thus, because all nodes in the processing cluster possess the same master node election algorithm, and will have the same information about the cluster membership, then when the nodes run the master election algorithm they will all determine the same node to be the master. Each time a master node is elected, it will reregister itself to the node configuration agent. As shown in Figure 10, at steps 1014 and 1015, in this example the node is still the master. Note that, although not shown in Figure 10, if some other node was master of an existing group, and a new member arrives, then the election algorithm could elect that the new member to be the master (or even elect another of the old members instead of the existing master). In such cases the existing master node will have to become a slave node. However, to illustrate this would require showing the two DFAs merged into one, which would be very complex both to illustrate and describe.

Configuration parameter access

After Java EE application startup, the application will be able to access the service instance configuration parameters though the appropriate APIs. The application can reach requested configuration data through programmatic access. Each access operation triggers its implementation. The requested data is accessed locally through the local configuration storage. It is important that configuration access will never take place before the configuration is loaded from persistent storage. This can only be guaranteed if all Java EE applications are allowed to start only after the local storage is fully loaded. The process is illustrated in Figure 11. At step 1101 , an application component 59a, requests provision of a configuration parameter X via its Configuration API 58a. Associated with the Configuration API 58a is a Configuration Implementation component 58x. At step 1 102, the configuration API 58a instructs the Configuration Implementation component 58x to fetch the parameter with key X. At step 1103, the Configuration Implementation component 58x contacts the Local storage 53 to get X, and at step 1104, the value Y is returned from storage 53. At step 1 105 the Configuration Implementation component 58x informs the configuration API 58 that Y is the value of parameter X, and at step 1106 the configuration API 58 provides the value Y to the application component 59a.

Notification of listener registration

The application components that wish to be notified of configuration updates will have to register as listeners for the configuration parameter(s) of interest. Application components may access configuration updates in two ways depending on the level of configuration state accuracy they desire: • Ad-hoc reads (dirty reads). If the application component wants to use configuration data only during short sessions and there is no need for long term accuracy.

• Polling. The application re-reads configuration data on a regular schedule. Polling is not recommended because it limits the accuracy of configuration data.

• Read once and get Updates. The application component may read the value of some configuration data once and keep it stored. When the configuration data is altered, an update will trigger further actions on the component.

The application components that wish to be notified of configuration updates will have to register as notification listeners of the configuration parameter(s) of interest. Figure 12 is a signal diagram illustrating the sequence of events. At steps 1201 to 1203 the application component 59a contacts its configuration API 58a to create and commence registration as a Listener of parameter X. At step 1204 the API 58a registers as a listener of X with the Cluster Configuration Agent 68. At step 1205 the Cluster Configuration Agent notifies the Distribution Service 55 of the registration. At step 1206, the Distribution Service adds the Listener and at steps 1207 to 1209 the successful registration is confirmed back to the application component 59a. There is no limitation as to when is the proper timeframe for an application component to register as a notification listener, as long as it is done before it processes the parameter values.

Configuration update As updates arrive from the northbound (OAM network), they are forwarded from the node configuration agent to the cluster configuration agent of the master node. The master node will then distribute these to all other members (slave nodes). A notification will be sent to all subscribers. Figure 13 is a signal diagram illustrating the sequence of events from the perspective of the Master Node.

At steps 1301 the operator 130 using the configuration client 61 updates a parameter X, setting a value Y on the Key X. At step 1302, the node configuration agent 67 receives the update, and at step 1303 sends the update, Set{X, Y} to the cluster configuration agent 68m, being the cluster configuration agent of the Master node, who is actually registered to receive updates. The update is verified (not shown in Figure 13) and at step 1304 the update is forwarded to the Distribution Service 55m on the Master node. At steps 1305 and 1306 the state in the Master Node's state machine 70 is changed to Operational (reference number 72 in Fig. 7) while the state consumption C1 - Applying Update Locally and Distributing - is performed. At step 1307 the Distribution Service 55 maps the value Y of parameter X to comply with its required representation Y' This is because the configuration set will not always contain the same parameter mapping as in JavaEE, and so the parameters have to be mapped to an application specific representation. At steps 1308 to 1313, the value Y' of parameter X is updated into the local storage 53m of the master node, by the illustrated procedure that involves setting and later releasing a Read Lock on the parameter X. At step 1314 notifications are then sent to all registered update notification listeners. The update will be finally sent at step 1315 for distribution over the rest of the service instances in the cluster. Successful distribution is notified back to the distribution service 55m at step 1316 and this is confirmed back to the cluster configuration agent 68m at step 1317. At step 1318, when the distribution service 55m determines that there are no more updates to be notified, at step 1319 it changes the state of the Master Node state machine 70 back to Operational. Finally, at steps 1320 to 1322 the successful completion of the configuration update is confirmed back to the Operator 130.

Figure 14 is a signal diagram illustrating the sequence of events from the perspective of a Slave Node. At step 1401 the cluster configuration agent 68m on the Master Node sends the update to the Communication Primitives 56s on the Slave Node. On receiving the update, it is forwarded by the Communication Primitives 56s to the Slave Node's distribution service 55s. At step 1403 the arrival of an update is notified to the Slave Node's state machine 80. However, this does not result in a state transition, but instead is handled as a state consumption (see Figure 8). The state remains operational (step 1404). At steps 1405 to 1410 the update is stored in the local storage 53s of the slave node, in the same manner as described above for the Master Node (steps 1308-1313 in Figure 13). At step 1411 the update is notified to all listeners, and at steps 1412 and 1413 the successful completion of the update is notified back to the Cluster configuration agent 58m on the master node.

Application receives update notification.

Regardless of the node type, when an update is loaded to the local storage, all application components that register as listeners to configuration updates will receive an update notification. After an update, the application components that operate on service instances over the processing cluster should be notified about the update of the configuration set, but only if the application has registered as a listener of a parameter or group of parameters that were part of the configuration update. On both Master and Slave Nodes, after a successful update is performed, notifications are sent to application components that have registered as listeners pointing out the updated parameters or groups of parameters. The notifications will contain the values that have been updated on the updated parameters.

The process is shown in Figure 15 for the Master node and in Figure 16 for a slave node. In Figure 15, steps 1501 to 1506 correspond to steps 1302, 1303, 1304, and 1308 to 1313 of Figure 13 in which the new value of the configuration parameter X has been updated to Y, and this has been stored in the local storage 53m of the master node. At step 1508 the distribution service 55m of the master node notifies the update to Application Component 1 59b, to indicate that the new value of parameter X is Y. The same occurs at step 1509 for Application Component 2 59c. Note that Application Components 1 and 2 59b, 59c must both have previously registered as listeners for updates to parameter X. At step 1510 the distribution service 55m on the master node then notifies the update to the slave nodes, as described above for Figure 13, and at steps 1511 to 1513 the successful update distribution is confirmed back to the operator. In figure 16 steps 1601 to 1605 correspond to steps 1401 , 1402, and 1405 to 1410 of Figure 14 in which the new value of the configuration parameter X has been updated to Y, and this has been stored in the local storage 53s of the slave node. At steps 1606 to 1608 the update is notified to Application Components 3 and 4 59d, 59e, which have registered as listeners for updates to parameter X, and at steps 1609 and 1610 the successful update distribution is confirmed back to the cluster configuration agent 68m on the master node.

Fai lover Failures can occur for several reasons, such as program or hardware failures. The most important failures will occur when a service instance fails. The actions to be taken depend on the type of node, i.e. if it is a master or a slave node. However, in case of failure of a Slave node there is no impact on the cluster logic. Failed slave nodes are either regarded as nodes that have disappeared from the cluster (and so no longer exist as member of the cluster). Such nodes are handled as new members when they reappear after a failure.

The Master node may be terminated due to a failure or for maintenance. When this occurs a new master is automatically elected from one of the previous Slave nodes. The new Master node will order consistency checks to ensure there are no missing updates. This might arise, for example, if a Master Node fails before it has managed to distribute an incoming update. After consistency resolution, all service instances of the processing cluster continue their normal operation. If the node that failed (old Master Node) reappears its reappearance is handled in the same way as a slave node joining the cluster, and it will to continue to operate as a Slave node to avoid disturbances.

Figure 17 shows the sequence of events that takes place after a Master node failure. One of the Slave nodes (in this example, node 1 ) becomes the new Master Node. At step 1701 the communication primitives on node 1 learn that the cluster's master node has failed. Meanwhile, at step 1702, the failure of the master node has simultaneously been learnt by Node 2 and the information has been received by the Cluster configuration agent on node 2. At step 1703, at node 1 , the master node failure is notified to the Distribution service. At step 1704 node 2 determines, after calling the master node election algorithm, that it is to remain a slave node. At the same time, at step 1705 the distribution service 55a on node 1 notifies the state machine 70a on node 1 of the master node failure. At step 1706, after calling the master node election algorithm, the distribution service 55a on node 1 is notified that node 1 is to be the new master (the state having been transitioned in the node 1 state machine).

Note that initially neither node 1 nor node 2 will know if it is to become the new master. Thus, although a different sequence is shown at node 1 and node 2, the overall initial sequence of signals will be the same on each node, at least until the node determines whether or not it is to become the new master. However, the signal sequences that follow will be different in each case and only the relevant initial signal sequence is shown for each case in figure 17.

At step 1707 node 1 , on becoming the new master, initiates consistency checks over all other Nodes. The consistency tests require comparing existing parameters in Persistent Storage with those found in Local storage. At steps 1708 and 1709 the consistency check instructions are sent to node 2 (as well as all other nodes in the cluster) via the node 1 communications primitives 56a. At step 1710, the cluster configuration agent 68b on node 2 , on receiving the consistency check instruction, requests the node configuration agent 67 to get the configuration set, and at step 1711 confirms receipt of the consistency check instruction back to node 1. At steps 1712 to 1714 node 2 obtains the configuration set from the persistent storage 52. At step 1715, the cluster configuration agent 68b on node 2 performs sanity checks by comparing the configuration set obtained from the persistent storage 52 with that in its local storage, and if there are any discrepancies, updates the configuration set in local storage.

Returning to node 1 , having become the master, at steps 1716 and 1717 the state machine 70a triggers the distribution service 55a to obtain the configuration set and any updates. This is done at steps 1718 to 1722 via the communication primitives 56a and node configuration agent 67 from the persistent storage 52. At step 1723, the configuration set obtained from the persistent storage 52 is compared with that the local storage and any required updates are made. Finally, at step 1724 the required sanity checks (as described above for node 2) are made. Regarding the Master Node election algorithm, there are several proven ways of ensuring that this produces a consistent result at every node. For example, a combination of consistent hashing algorithms may be employed in relation to existing members and their identification features. The algorithm and the criteria are identical for all nodes and will always result in the same election whichever node is using it.

Dynamic Cluster Expansion. The processing cluster may expand at runtime. This can be done either through addition of new Service Instances to existing Java EE clusters or through addition of new Java EE clusters to the processing cluster. The processing cluster is a virtual configuration cluster which may host Service Instances of several Java EE clusters. These instances share the same internal network and configuration set. The configuration set defines the information to be shared over the cluster and each the configuration of each service instance in the cluster is a subset of the common configuration set.

It is possible to expand the processing cluster to host additional Java EE clusters. This is feasible if the additional clusters deploy the cluster configuration agent as defined above, for all their Service Instances. However, the addition of a new cluster may require expansion of the configuration set as well, because the new cluster may require additional configuration parameters that are not part of the configuration set of the current processing cluster. The operator must allow the expansion of the configuration set to include this additional configuration data before the new Java EE clusters appear in the network.

The following table presents an example of an implementation of the system of the present innovation. The table lists components mentioned in the innovation described above against components or primitives used under development.

The approach to the configuration of processing clusters described above has several advantages.

Simplicity. Configuration software is reused for all related application server clusters, such as Java EE clusters. Initialization does not require static configuration of involved clusters. All involved cluster instances for all clusters, are considered as parts of a single processing cluster.

Fault tolerance. No single point of failure. Faults originating on both software and hardware are handled in a similar way. High order logic is used to decide system consistency. Application server instance start-up, shutdown or failure is handled in a similar way. There is no risk of fault propagation.

Expandable design. The overall number of the processing cluster instances may shrink or expand during runtime without additional measures having to be taken, (except where some additional data needs to be brought into the configuration set). Performance. Lightweight transactions - transactions are not as expensive as in a database management system, since no locks and expensive retransmissions are needed and no complex algorithms are used. This has a direct positive impact on performance. No complex strategy is required to recover from errors.

Economics. Simpler configuration of heterogeneous clusters leads to a reduction of operating costs on site.

Claims

CLAIMS:

1. A processing cluster comprising a plurality of processor nodes and a configuration management system, wherein each processor node comprises a configuration agent operable to apply configuration parameters provided by the configuration management system, and wherein the configuration management system provides the same configuration parameters obtained from a common set of configuration parameters to all the processor nodes in the processing cluster.

2. The processing cluster of claim 1 wherein the processing cluster is a constellation of a plurality of server clusters, each server cluster comprising a plurality of processor nodes.

3. The processing cluster of claim 2 wherein the server clusters are Java EE clusters.

4. The processing cluster of claim 2 or claim 3 wherein the plurality of server clusters includes one or more heterogeneous clusters.

5. The processing cluster of claim 4 wherein the nodes comprise a master node and one or more slave nodes, wherein the master node coordinates the provision of configuration parameters to the slave nodes.

6. The processing cluster of claim 5 wherein the master node is elected from among the nodes in a cluster in accordance with an election algorithm.

7. The processing cluster of any preceding claim configured to provide a virtual configuration access area for JavaEE applications.

8. The processing cluster of claim 7 configured to define rules for updates to, access to, and membership of the access area.

9. The processing cluster of claim 7 or claim 8, wherein the applications share portions that have common configurations.

10. The processing cluster of any preceding claim comprising a cluster configuration agent.

1 1. The processing cluster of claim 10 wherein the cluster configuration agent comprises a distribution service for distributing configuration parameters to nodes in a cluster.

12. The processing cluster of any preceding claim further comprising a persistent storage, wherein a current configuration set of configuration parameters is stored.

13. A server node configured for operation as a member of a processing cluster, the server node being operable to function as either a master node or as a slave node: wherein, when functioning as a master node, the server node is configured to receive notifications of cluster configuration parameters from a common set of configuration parameters, and to distribute the configuration parameters to other members of the processing cluster; and wherein, when functioning as slave node, the server node is configured to receive configuration parameters from a master node and to implement the node's configuration in accordance with the configuration parameters.

14. The server node of claim 12 comprising a distribution service for distributing the configuration parameters, notifying configuration changes to the other members of the processing cluster, and receiving configuration parameters from a master node.

15. The server node of claim 12, comprising a state machine that defines a logical state of each node and whereby state transitions define the protocol of the distribution service.

16. A method of configuring a processing cluster, the method comprising: determining information relating to applications and configuration requirements for each of a plurality of nodes that make up the processing cluster; electing one of the nodes to be a master node, in accordance with an election algorithm; notifying the master node of configuration parameters based on a common set of configuration parameters; the master node distributing the configuration parameters to all other nodes of the processing cluster; and configuring every node in the processing cluster in accordance with the configuration parameters.