HK1173867B - Globally sound and consistent configuration management for distributed datacenter components - Google Patents

Description

Globally robust and consistent configuration management for components of a distributed data center

Technical Field

The invention relates to data center configuration.

Disclosure of Invention

In a first aspect, a set of computer-useable instructions provides a method of updating a data center configuration. The data center configurations are organized into globally consistent configuration components. A developer may wish to alter the configuration data for a component. The developer downloads a copy of the active set of configurations and makes the required changes. A modified version of the configuration component is received from a developer. A determination is made whether the modified configuration component is globally consistent with the submitted version of the data center configuration. If the modified configuration component is consistent, it is accepted into the hosted set of data center configurations and if not, rejected. For more advanced operations, a developer may perform changes in a partial branch of configuration data before the changes are merged back into the main branch. When an active configuration in a data center changes, multiple versions of the configuration may be deployed simultaneously for a time window. During these transition windows, the hosted set of configurations is permitted to have multiple active versions to reflect the reality in the data center.

Drawings

Some illustrative embodiments of the invention are described in detail below with reference to the attached drawing figures, wherein:

FIG. 1 depicts an exemplary operating environment according to an embodiment of the invention;

FIG. 2 depicts an exemplary operating environment, according to an embodiment of the invention;

FIG. 3 depicts an exemplary configuration management environment, according to an embodiment of the invention;

FIG. 4 depicts exemplary configuration data according to an embodiment of the invention;

FIG. 5 depicts a method according to an embodiment of the invention;

FIG. 6 depicts a method according to an embodiment of the invention;

FIG. 7 depicts a method according to an embodiment of the invention; and

fig. 8-12 depict a data center configuration according to an embodiment of the present invention.

Detailed Description

The subject matter of some embodiments of the invention is described with specificity herein to meet statutory requirements. However, the description itself is not intended to necessarily limit the scope of the claims. Rather, the claimed subject matter might be embodied in other ways to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. The terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

The technique of the invention takes in particular the following form: a method, system, or set of instructions embodied on one or more computer-readable media. Computer-readable media include both volatile and nonvolatile media, removable and non-removable media, and contemplates media readable by a computing device. Computer readable media includes any medium that can be used to store information. Examples of stored information include computer-useable instructions, data structures, program modules, and other data representations. Examples of media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD), holographic media or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, and other magnetic storage devices. These techniques may store data instantaneously, temporarily, or permanently.

Referring initially to FIG. 1 in particular, an exemplary operating environment for implementing an embodiment of the present invention is shown and designated generally as computing device 100. Computing device 100 is but one example of a suitable computing environment.

An embodiment of the invention may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, etc., refer to code that perform particular tasks or implement particular abstract data types. An embodiment of the invention may be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general-purpose computers, more specialty computing devices, and the like. An embodiment of the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

Referring to FIG. 1, computing device 100 includes a bus 110 that directly or indirectly couples the following devices: memory 112, one or more processors 114, one or more presentation components 116, input/output ports 118, input/output components 120, and an illustrative power supply 122. Bus 110 may be one or more buses, such as an address bus, a data bus, or a combination thereof. Although the various blocks of FIG. 1 are shown with lines for the sake of clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would more accurately be grey and fuzzy. For example, a presentation component such as a display device may be considered an I/O component. Also, the processor has a memory. It is to be appreciated that such is the nature of the art, and reiterate that the diagram of FIG. 1 is merely illustrative of an exemplary computing device that can be used in connection with one or more embodiments of the present invention. There is no distinction between categories such as "workstation," server, "" laptop, "" handheld device, "etc., all of which are considered to be within the scope of fig. 1 and are referred to as" computing devices.

Computing device 100 typically includes a variety of computer-readable media. By way of example, and not limitation, computer-readable media may comprise Random Access Memory (RAM); read Only Memory (ROM); an Electrically Erasable Programmable Read Only Memory (EEPROM); flash memory or other memory technology; CDROM, Digital Versatile Disks (DVD), or other optical or holographic media; magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices; or any other medium that can be used to encode desired information and that can be accessed by computing device 100.

The memory 112 includes computer storage media in the form of volatile and/or nonvolatile memory. The memory may be removable, non-removable, or a combination of the two. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, and the like. Computing device 100 includes one or more processors that read data from various entities such as memory 112 or I/O components 120. The presentation component 116 presents data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, and the like.

I/O ports 118 allow computing device 100 to be logically coupled to other devices including I/O components 120, some of which may be built-in. Illustrative components include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, and the like.

Large-scale networked systems are common systems used in a variety of environments for running applications and maintaining data for business and operational functions, and may be referred to as data centers. The data center may provide various services (e.g., web applications, email services, search engine services, and others). These large-scale networking systems typically include a large number of nodes distributed throughout the data center, where each node resembles a physical machine or a virtual machine running on a physical host.

Referring now to FIG. 2, an exemplary operating environment for implementing an embodiment of the present invention is shown and designated generally as environment 200. The environment 200 is only one example of a suitable operating environment.

Environment 200 includes cluster 210 and cluster 212. In one embodiment, a cluster is a group of independent computers or servers that operate as a single unit. Cluster 210 includes a plurality of servers 214 and cluster 212 includes a plurality of servers 216. In one embodiment, environment 200 may include multiple clusters. In another embodiment, environment 200 may comprise a single cluster. In an embodiment, the number of clusters in environment 200 may change over time. Although clusters 210 and 212 are depicted as including multiple servers, in some embodiments, either or both of clusters 210 and 212 may each include only a single server.

In an embodiment, cluster 1210 and cluster 212 are located in the same geographic location. In another embodiment, cluster 210 and cluster 212 are geographically separated.

Data centers are large-scale networked systems that are commonly used to run applications and maintain business data and operational functions. A data center may provide various services (e.g., web applications, email services, search engine services, and others) to multiple customers simultaneously. These large-scale networking systems typically include a large number of resources distributed throughout the data center, with many physical machines and virtual machines hosted on these physical machines linked together through physical network devices such as routers, switches, and load balancers. When multiple tenants (e.g., clients) are hosted by a data center, these resources are optimally allocated to different tenants from the same data center.

A data center may include one or more clusters. In an embodiment, cluster 210 may constitute a first data center 210 and cluster 212 may constitute another data center 212. In another embodiment, cluster 210 and cluster 212 may together comprise a data center even though clusters 210 and 212 are geographically separated.

Data centers, such as data centers 210 and 212, require a large number of configurations that need to be managed. The configuration describes the physical and logical resources, the set of associated systems that need to interact to manage the data center, and the policies that should be enforced within the data center. As data centers grow in size and functionality therein, the required set of configurations also grows, spanning multiple components and formats, involving a wide variety of configuration artifacts (e.g., files containing values for configuration settings).

Providing uniformity across various policies and artifacts for different classes of data center assets (e.g., testing, pre-production, and production) reduces complexity and risk by simplifying the task of determining what other components are to be affected whenever the configuration functionality of a configuration artifact is extended or altered. Defining settings and values in a manner that is consistent across various configuration artifacts facilitates aspect-oriented (aspect-oriented) configuration audits (e.g., audits of global security policies).

Providing a format that is consistent across configuration artifacts minimizes the need for domain expertise in reviewing configurations and/or conducting global configuration reviews. When configuration settings are changed, it is important to manage the flow of these changes from developer-specific test environments to team-level test environments, to pre-production and production. It is also important to prevent changes from propagating unduly too quickly to the subsequent environment, as this would potentially introduce the risk of an untested configuration causing problems, or of developing a new version of a configuration that is incompatible with an earlier deployed configuration.

On the other hand, failure to propagate changes can result in the use of stale configurations or older versions of configurations that cannot be used for more recently deployed assets. If multiple teams make changes to configuration artifacts, a snapshot of these artifacts at a certain time may provide inconsistent data. When configuration artifacts are used from multiple locations, the risk of erroneous or malicious alteration or deletion of these artifacts increases. Inconsistent configurations may be deployed across multiple data center resources, leaving a window of time when components do not function properly.

Tracking the associated pairings between versions is beneficial as configurations are continually revised and deployed, making it possible to robustly roll back data center software to a previously known good state in a decoupled environment (mitigating problems in newer versions or replicating old environments for investigation).

With further reference to FIG. 2, a configuration engine 218 manages configuration data for the clusters 210 and 212. In some embodiments, the configuration engine 218 manages multiple clusters, while in other embodiments it manages only a single cluster. The configuration engine 218 may manage a single data center of a plurality of data centers. The configuration engine 218 may reside on a server, on a collection of servers, or on other types of network devices. The configuration engine 218 may be located at a particular facility, or may be distributed over a wider area. The configuration engine 218 allows users to check out, edit, and check in configuration values, which are organized in so-called configuration components. It also allows a user or other program to query the configuration values.

The configuration engine 218 supports constraint expressions that ensure global consistency across configuration components and versions, isolates changes in the data center from structural changes in configuration components, and enables partial configuration branching to allow active snapshots of configuration rollback to be enabled. For example, these constraints may prevent multiple organization (fabric) clusters from each listing the same node in their manifest. Some global constraints are fairly simple (e.g., if clusters are associated with disaster-recovery duplicate pairs, the partner cluster associated with each element of the pair must be the other element of the pair). The configuration engine also allows for transitional ambiguity (i.e., simultaneous implementation of different versions during the transition) to enable strict environment coordination even during non-atomic assembly (rollout).

The configuration engine 218 ensures that types are consistent and constraints are satisfied at the time of login to first eliminate the possibility of an unhealthy or inconsistent configuration in the system. It allows organized (i.e., coordinated updates) commit across configuration components with potentially different owners applying these rules across the global as well as component local constraints. Configuration data that is logged into or present in configuration engine 218 but not actually present in the data center is referred to as "committed," while "deployed" refers to configuration data that is actually used in the data center.

Referring now to FIG. 3, an exemplary configuration management environment for use in one embodiment of the present invention is shown and designated generally as configuration management environment 300. Configuration management environment 300 is only one example of a suitable configuration management environment.

The configuration engine 310 is accessed by a plurality of developers 312, 314, and 316. In one embodiment, the configuration engine 310 manages the configuration data set 318 and performs the functions described above. Configuration data 318 may include configuration settings for one or more clusters and one or more data centers. Developers 312, 314, and 316 can be engineers, technicians, and others responsible for maintaining and updating configuration data 318 for a data center and/or various data centers. For example, in one embodiment, the developer 312 maintains a portion of the configuration data 318 that is associated with a first data center, while the developer 314 maintains a portion of the configuration data 318 that is associated with a different data center.

Configuration engine 310 allows configuration owners (e.g., developers 312, 314, and 316) to decompose configuration data 318 into separate, aspect-oriented configuration components, such as component A320 and component B322. The configuration components are sets of configuration data corresponding to different aspects of the configuration, such as physical assets, logical assets, accounting policies, security policies, per-component policies, and other aspects of the data center configuration. This separation classifies ownership and reduces the complexity to consider when an owner makes changes. The major changes made by the owner are isolated to the aspects of the component. Those affected components evaluate and approve any chain reaction, while unaffected components are not updated at all and thus are not accidentally altered.

Versioned interfaces are placed between the configuration components through data contracts. For example, as shown in FIG. 3, component A320 includes interface version N324 and interface version N + 1326. Interface version N +1326 is the latest interface version. Each versioned interface describes named typed settings that a consumer of a particular configuration component can query. Producers (e.g., developers) do not affect consumers when they make interface changes because consumers migrate to newer interface versions at their own pace. Once all consumers migrate to the new version, the producer retires the old interface version over time.

Configuration data bound to an interface is also versioned. The use of typed settings may enable consistency checking when the data is provided to (or logged into) configuration engine 310 rather than when the data is consumed during deployment. This allows for early detection of configuration problems. The use of versioned configuration data eliminates timing and synchronization issues in the case of changes made by multiple developers. The configured consumers can retrieve the correct, versioned data at any time they want.

One feature of the interface is that the bound data can be represented in a manner that is independent of how the data is queried. In particular, it may be more compact and thus easier to review and thus ensure that it is correct. In particular, the configuration engine 310 allows various options, such as using inheritance, conditions, or default values for compact human-reviewable definitions. As an example, an aspect of the configuration may define a billing rate between data centers. The interface may specify:

DCxDC → the rate of billing,

that is, two data center names (e.g., DCA, DCB, DCC, or DCD) are provided and a billing rate is obtained, which is easy to query. One possible definition is the table:

	DCA	DCB	DCC	DCD
					DCA	0	4	4	7
DCB	4	0	4	7
					DCC	4	4	0	7
DCD	7	7	7	0

the table specifies the billing rate for any two data centers, but is difficult for humans to review because the rules it encodes are implicit and the table can become large. An example of a clearer, more compact definition based on conditions and defaults is provided with C-type syntax (note that inheritance is not shown in this particular example, and note that other representations such as XML may be used):

this definition is less error-prone and easier to expand when adding new data centers.

The configuration component may also contain multiple exact versions of data bound to an interface version. This allows values to be changed across exact versions as long as the data bound as each exact version is consistent with the interface (and global constraints). As shown in fig. 3, the interface version N324 includes binding data M332 and binding data M + 1334. The interface version and the binding exact version constitute a two-tier versioning scheme.

Component B322 shows two-level versioning with explicit versioned dependencies. As shown, configuration component B322 depends on interface version N324 and interface version N +1326 of configuration component a 320. This is illustrated by arrows 328 and 330, respectively (the arrows at least depend on the relationship, not the data flow). As such, component B322 has an explicit dependency on component a320 (e.g., by including an embedded query to configure component a 320), which means that the data bound to component B322 depends on the interface type or data of component a 320.

Configuration engine 310 ensures that queries are well-formed at login time and that global constraints and types across upstream components are consistent. In other words, the user cannot bind the new exact version unless the type of value it returns to each member in the interface is consistent with the interface declaration. It also ensures that the data bound to the exact version is consistent with global constraints. Implementing two-level versioning using explicit versioning dependencies (confirmed by the system at login time) enables well-defined exact versions to be retrieved. An example of an implicit constraint is that it is exaggerated that the same attribute of each interface version must have the same value (i.e., if the attribute "comp.

By making the dependencies explicit and versioned in the first of the two-tier versioning, the component owners can introduce versioned interface changes that can be employed by consumers at a pace that suits them-this ensures that interface changes do not cause failures when evaluating the relevant components. Because changes in the data bound to each exact version must be consistent with the type of interface and global constraints, the exact version changes do not cause downstream failures and thus can occur at any time. Because a constraint (which is part of the exact version) is limited by the dependencies of the interface versions of its components, it can only constrain what it can "see". Therefore, downstream constraints do not need to be re-evaluated, as they will not have changed. This is one benefit of having explicit dependencies.

Because the constraints are global, they may affect multiple configuration components. Thus, to make a particular global change, it is necessary to atomically update multiple configuration components in a single organized commit.

Environment coordination is the process by which a developer or program confirms that the configuration values actually deployed match the expected set of configuration values. This is easy when deploying a single configuration version-it is a straightforward checking procedure, assuming that expected and actual values can be obtained. However, when non-atomic assembly is done, it is more difficult to do so strictly — two or more different versions of the configuration may be expected, each with a potentially different set of values.

The configuration engine 310 enables strict environment coordination by using a feature called transition ambiguity. At any given time, in the configuration engine, a set of exact versions are marked as header versions (i.e., current, active data bindings for interface versions). Each exact version of a component may include constraints that need to be true. The configuration data is globally consistent if all constraints for all header exact versions of all components in all branches hold.

There may be zero header versions in the interface, in which case the interface cannot be queried at all. Most often, there will be one header version, in which case the values of the version are returned during evaluation unless the caller specifies a more specific version.

The configuration engine 310 supports two or more exact versions of the header. Referring to component A320, interface version N324 has a header version of binding data M +1334, as shown in FIG. 3. The interface version N +1326 includes binding data M336 and header binding data M +1338, which correspond to the binding data M332 and the binding data M +1334, respectively. But interface version N +1326 also includes binding data M +2340 as a header version.

When the interface version includes two header versions, the configuration engine 310 internally checks the contents of the binding data and returns a set of values when evaluated. A single common value is returned when these values are equivalent across the various head versions. Where these values differ across the respective head versions, a different set of possible values is returned.

Because the configuration components may depend on each other, this set of different possible values is concatenated across the components (folding the same values at each step) until the final result is returned to the caller. The minimum set of expected values is guaranteed back to the caller. This ensures strict environmental coordination (i.e., verification of the smallest set of possible expected values) even during development when the configuration values are in transition. This would occur in the case of fig. 3, where component B322 depends on interface version N +1326 of component a326, interface version N +1326 having two header versions.

When a developer wishes to modify one or more configuration components, those components are typically part of a larger set of configuration data. This larger set may be referred to as the primary or parent branch of configuration data. The developer may select components to be included in a partial branch (i.e., child branch) that contains a copy of the selected component from the parent branch, but does not include a copy of the entire set of configuration data.

A partial branch 342 is shown which includes a configuration component a 344. In this case, the developer has selected configuration component a320 in the main branch to be included in the partial branch 342. Configuration component A344 is a copy of configuration component A320 and includes copies of interface version N324 and interface version N +1326, as well as versions of binding data for each of these interface versions.

The partial branch 342 inherits the view of the non-dominant configuration components of the main branch, i.e. for other configuration components not included in the partial branch, the partial branch 342 points back to the main branch. In effect, components in the partial branch can "see" other components in the main branch, and thus need not include all of the configuration data of the main branch (again, need not replicate configuration data that models physical resources of the data center). Thus, the partial branch 342 enables a developer to modify the configuration component A344 without affecting the configuration components in the main branch, while still maintaining the ability to build and deploy the entire product.

For example, a developer may want to add a new version of binding data for an interface version. As shown, the configuration component a344 in the partial branch 342 has been modified by the developer to include binding data M +3346, which binding data M +3346 is a new header version of the binding data for interface version N + 1348.

When the configuration component a344 of the modified branch is ready, it is integrated (or copied) into the main branch. Configuration engine 310 simultaneously detects changes made in the primary branch and requires developers to understand and resolve these merging conflicts.

Referring now to FIG. 4, exemplary configuration data for use in one embodiment of the present invention is shown and designated generally as configuration data 400. Configuration data 400 is only one example of suitable configuration data.

Referring to fig. 4, a parent branch 410 is depicted. In one embodiment, the parent branch 410 itself may be part of a larger branch. In another embodiment, parent branch 410 may be the primary branch, in which case it includes configuration data for the entire data center or network managed by the configuration engine. Parent branch 410 includes configuration component ("component") 412, component 414, and other components 416, 418, and 420.

Component 412 includes dynamic enumeration (enum) 420. In one embodiment, dynamic enumeration is a type of constraint enforced by configuration engine 310 at login time. The dynamic enumeration type may be associated with a particular number of configuration components, or with parameters of functions (or mappings) within configuration components. In one embodiment, the dynamic enumeration type expresses a set of assets, such as a set of clusters (e.g., cluster A, cluster B, and cluster C) present in dynamic enumeration 420. In other embodiments, the dynamic enumeration may be a list of data centers, clusters, special tenants, and any other set of assets that are expected to grow or decrease over time as a cluster is built or retired. The concept of dynamic enumeration is that there is a particular class of things that can be queried and the membership in that class can vary, but it can be stated in advance that the selection will always be valid across all members of the class.

The dynamic enumeration type expresses a limited set of possible values. In addition, from a type perspective, the addition and subtraction of this set of possible values is opaque. Because dynamic enumeration is of an opaque type (i.e., the details of that type are hidden), the asset may be configured without requiring changes in the component interfaces. This isolates changes in the data center (e.g., addition of new assets, removal of old assets) from structural changes in the configuration component.

In other words, the dynamically enumerated content may be changed without affecting the interface version that owns its components or the binding of components that depend on the components that own it. Because the dynamic enumeration type is opaque, it is possible to grow or shrink the set of assets present in the configuration engine without affecting the configuration components that do not directly depend on the values within the enumeration.

Component 414 depends on interface version X (not shown) of component 412 because the queries available for component 414 are defined in interface version X of component 412. In an embodiment, component 414 can query component 412 to determine whether a given cluster in dynamic enumeration 420 is fast or slow.

Subcomponent 422 includes components 424 and 426 which are copies of components 412 and 414 in parent branch 410, respectively. Component 426 is modified to query whether a given cluster in dynamic enumeration 420 is fast, medium, or slow.

The configuration engine 310 of fig. 3 allows the limitations of dynamic enumeration in a subbranch to model the flow of configurations through a decoupled environment. That is, a parent branch may model all assets in the dynamic enumeration, while a child branch models only a subset of those assets. This enables the engineer to design and test configuration changes in the subbranches as follows: all global constraints are declared and enforced, but only on a subset of assets present in the branch. As shown in FIG. 4, the sub-branch 422 limits the scope of the dynamic enumeration 428 so that it lists only cluster A. Some global constraints are fairly simple (e.g., if clusters are associated with disaster-recovery duplicate pairs, the partner cluster associated with each element of the pair must be the other element of the pair).

In the case where the query is issued against a configuration component in the parent branch from a child branch, the values of the dynamic enumeration parameters are limited to matching a known set of values in the child branch, and the returned results must be present and correctly typed for that subset of assets. All global constraints are enforced, except that dynamic enumeration 424 only allows the definition of new configuration settings for cluster A (i.e., developers do not need to populate the new configuration settings with values for each cluster early in the deployment cycle). In one embodiment, the restrictions for dynamic enumeration are explicit. Explicit restrictions allow verifying that data not included in the restricted enumeration is still in full enumeration (as opposed to just ignoring everything outside the restriction), which allows typing errors to be more easily detected, among other things.

Thus, dynamic enumeration allows global constraints to be applied only to that set of clusters in the integration, which in turn allows deployment of new configuration versions in the integration before a complete set of required values is defined for all production clusters. Dynamic enumeration enables the configuration engine 310 to ensure that all required configurations have been provided to a new instance before the instance can be added to the enumeration, i.e., all required settings are filled with values before adding new assets to the data center.

This will ensure, for example, that each required configuration value for a newly built cluster has been prepared before the cluster is put into production. Dynamic enumeration isolates the cluster and tenant lifetimes from the lifetime of each component version, while still allowing the component owner to declare the specific configuration settings required per cluster.

Thus, a developer can independently validate the configuration (i.e., the addition of a new certificate), which can be very expensive in advance because there may be hundreds of production clusters in the parent branch 410.

As changes are integrated up through the parent branch, a new type of merge conflict may be detected (dynamic enumeration constraint validation fails), which as part of the integration will have to be corrected by the parent branch owner. In this way, configuration changes that progressively propagate from one decoupled environment to the next may be modeled in the configuration engine, enabling engineers to work efficiently with small collections of assets, while still allowing full assurance of consistency when produced using large numbers of assets.

In addition to ensuring that the global constraint is consistent with components in the parent branch, configuration engine 310 also ensures that the global constraint is consistent with components across all branches. This results in the following system: where the configuration logged into each branch is always valid-any changes that would violate this argument must be corrected, possibly across multiple components and branches that are organized commits. This allows developers to develop new configurations in the child branch without affecting other developers handling the parent branch.

As configuration changes are integrated across branches, dynamic enumeration grows to cover the clusters represented in each branch, ensuring global consistency independently in separate decoupled environments. The use of dynamic enumeration effectively decouples the configuration interface from asset management.

This also allows developers to branch to a particular set of configuration components and maintain those components as a class of active snapshots. That is, the set of configuration components is maintained in a version-specific subbranch that must always be consistent with the other components present in the parent branch. This means that the retained version can always be deployed to the cluster, while knowing that it will interoperate in a robust and consistent manner with other configuration versions that move forward or backward independently. The branched configuration version may have to be changed slightly, but only to ensure that global constraints are met — not new features or policies.

Referring now to FIG. 5, a method in accordance with an embodiment of the present invention is depicted and generally designated as method 500. Method 500 is only one example of a suitable method. At step 510, the data center configuration data is organized into configuration components that include configuration data corresponding to aspects of the data center configuration.

At step 512, the configuration component is provided to the developer, which enables the developer to modify the configuration component in a separate environment from the primary configuration data. At step 514, a modified version of the configuration component is received from the developer.

At step 516, a determination is made as to whether the query defined by the interface is well-formed, i.e., whether the type of values returned by the configuration component for each member of the interface are consistent with the interface declaration. At step 518, upon determining that the query is well-formed, the modified configuration component is accepted into the hosted set of datacenter configurations as a new exact version. At step 520, upon determining that the query is not well-formed, the modified configuration component is rejected.

Assuming that the modified configuration component is accepted, at step 522, a request is received from the developer to submit a new request version. At step 524, a determination is made as to whether the new exact version is globally consistent with the data center configuration. Any constraints that may have changed or that have been introduced are evaluated against the new data to verify that global consistency is maintained.

In step 526, upon determining that the new exact version is globally consistent, the request is accepted and the new exact version becomes the header exact version. At this point, any developer querying the data will receive the modified data. On the other hand, upon determining that the new exact version is not globally consistent, the request is denied at step 528.

In other embodiments, providing the configuration component to the developer at step 512 may include generating a partial branch of the configuration data that includes the configuration component. The partial branch references a data center configuration that is not included in the partial branch, thereby enabling a developer to confirm that the changes to the component are consistent with other portions of the data center configuration.

Referring now to FIG. 6, a method in accordance with an embodiment of the present invention is depicted and generally designated as method 600. Method 600 is only one example of a suitable method. At step 610, a primary branch of configuration data is provided that includes configuration data for the data center. At step 612, configuration components in the primary branch are provided. The configuration component is a configuration data set corresponding to an aspect of the data center configuration. The configuration component needs to specify a list of configuration values for the data center assets, and includes dependencies on other configuration components in the primary branch.

At step 614, the developer is provided (or logged out) with a partial branch of the configuration data. The partial branch is a copy of a portion of the main branch and includes a copy of the configuration component. Copies of the configuration components in the partial branch may be available for modification by a developer. At step 616, access is provided to the main branch to the partial branch such that dependencies of copies of the configuration component on other configuration components are implemented without including the other configuration components in the partial branch.

At step 618, the list of data center assets is limited to a subset of the list. If a data center asset in the list is not in the subset of the list, the configuration component in the partial branch need not specify a configuration value for the data center asset. As described above, this is a function of dynamic enumeration. In an embodiment, the restriction is explicit and validating includes checking for consistency of data not included in the restricted enumeration.

At step 622, a modified version of the configuration component is received (or logged in) from the developer. At step 624, a determination is made that the modified configuration component is globally consistent with the committed version of the data center configuration. At step 626, upon determining that the modified configuration component globally conforms to the committed version of the data center configuration, the modified configuration component is accepted. The modified configuration component may then be used for development by the same or other developers requesting data from the branch. In the case where changes are intended to become the default values for the data center, they must also be integrated into the primary branch before committing by the organization.

Referring now to FIG. 7, a method in accordance with an embodiment of the present invention is depicted and generally designated as method 700. Method 700 is only one example of a suitable method. At step 710, a parent branch of configuration data is provided that includes configuration data for the data center. At step 712, the configuration data is organized into components that include interfaces that enable one component to query another component for configuration settings. The data associated with the query needs to conform to the defined data type.

At step 714, a first sub-branch of configuration data is generated in response to a request from a first developer. The first child branch includes a first component that is a duplicate of a component in the parent branch. At step 716, the first sub-branch is provided to the first developer. Subsequently, the first developer can modify the first component.

At step 718, a second sub-branch of configuration data is generated in response to a request from a second developer. The second child branch includes a second component, where the second component is a duplicate of another component in the parent branch. At step 720, the second sub-branch is provided to the second developer.

At step 722, a modified version of the first component is received from the developer. At step 724, upon receiving the components, it is verified that (a) the data type of the data in the first component is consistent with the components in the parent branch, (B) the first component satisfies the global configuration constraint, (C) the query well-formed type associated with the first component, and (D) the data type of the data in the first component is consistent with the components in the second child branch.

8-12 illustrate real-world examples of component dependencies and how the dependencies relate to implementation of configuration changes according to an embodiment of the invention. In one embodiment, customers are charged according to the computation, storage, and network bandwidth they use when running their applications in the data center. If the connection is in the same data center as the application, the network bandwidth usage is free. However, if the network traffic goes outside of the data center, bandwidth usage can cause charges for the transmission of data in gigabytes. In addition, charging varies depending on the geographic source and destination of the traffic. For example, bandwidth is more expensive in asia than in north america. Thus, if the source or destination is in a data center in asia, the charge per gigabyte of transmission will be higher.

In an embodiment, a bandwidth tracker measures bandwidth used by applications running in a data center. In order to properly classify the source and destination of the network traffic it measures, a set of network ranges (i.e., IP address ranges) is used to program the bandwidth tracker. Each data center is listed and, for each data center, a set of IP addresses located in the data center is declared. In this way, when the bandwidth tracker observes traffic flowing into or out of hosted applications, it can record the necessary data to generate bandwidth-related charges on the customer's monthly bill.

The bandwidth tracker uses the network range in a file called the Machine Configuration File (MCF). The MCF has a particular format and the code of the bandwidth tracker knows how to use the data in that format. If there is a change in the format, the code of the bandwidth tracker must change immediately following the format change so that it can continue to utilize the MCF. MCFs contain various types of data including, but not limited to, IP ranges. The values of the data fields in the MCF change to reflect changes in the set of physical assets located in the data center and in the measurements governing the use of those assets.

The network scope is derived from a separate database managed by the network service group. The network service group is responsible for coordinating the purchase and management of all IP addresses used. The network service group updates this IP range database on its own schedule governed by budgets and operational procedures. There is no coordination between changes made in the IP range database and the periodic versions of the bandwidth tracker. However, whenever a new MFC is generated, it should contain the latest set of IP ranges to ensure billing accuracy.

The above discussion describes two configuration dependencies, which are shown in FIG. 8. Fig. 8 depicts a data center configuration according to an embodiment of the invention, and is generally designated as data center configuration 800. The data center configuration 800 includes bandwidth tracker code 810, MCF data 812, MCF interface version 1.0814, IP range data 816, and IP range interface version 1.0818.

In an embodiment, as described above, MCF data 812 is one configuration artifact that depends on data stored in a different configuration artifact (web service IP range data 816). In addition, the bandwidth tracker code 810 depends on the format of the MCF data 812. The bandwidth tracker code 810 declares dependencies on the MCF interface version 1.0814 (bold arrow 820). The interface version describes the format of the MCF data 812. The MCF interface version 1.0814 is in turn bound (dashed line 822) to several versions of MCF data 812 over time. The content of each version of MCF data 812 is different, but each version conforms to the format constraints in MCF interface version 1.0814, ensuring that bandwidth tracker code 810 will be able to use the data.

The MCF in turn depends (bold arrow 824) on the IP range interface version 1.0818, which in turn binds (dashed 826) to multiple versions of the IP range data 816 over time. The MCF data 812 contains an embedded query 828, the embedded query 828 automatically retrieving the latest updated version of the IP range data 816 from the IP range component via the mechanism stated in the IP range interface version 1.0818. Even if the IP Range data 816 changes frequently, a query in the MCF data 812 will always retrieve valid results in a format that it knows how to use, since the IP Range interface version 1.0818 is consistent

Because the interface versions are consistent, the specific values of the data bound to these interface versions may change according to different procedures and schedules for different owners of each category of data. At the same time, the user of the data does not have to do anything when these underlying values change, since they rely only on the format declared in the interface version.

Fig. 9 is a block diagram depicting a data center configuration according to an embodiment of the invention, and is generally designated as data center configuration 900. The data center configuration 900 includes bandwidth tracker code 810, MCF data 812, MCF interface version 1.0814, IP range data 816, and IP range interface version 1.0818, each of which is also depicted in fig. 8. Data center configuration 900 also includes IP range data 910 and IP range interface version 2.0912.

Referring to FIG. 9, in one embodiment, a developer decides to switch the data center from using the IPv4 protocol to the IPv6 protocol. IP range data 816 will then need to have a new format. The developer will declare a new IP scope interface version 2.0912. Initially it will be next to the old version as shown in fig. 9.

At this point, no component depends on the IP range interface version 2.0912. Finally, as the team owning the MCF data 812 begins considering it, the MCF component may generate a new MCF interface version that exposes IPv6 network scopes beyond those of IPv 4. Referring to fig. 10, a block diagram depicts a data center configuration according to an embodiment of the present invention, and is generally designated as data center configuration 1000. In addition to the items depicted in fig. 9, the data center configuration 1000 includes MCF data 1010 and an MCF interface version 2.01012.

At this point, a team of developers owning the bandwidth tracker modifies bandwidth tracker code 810 so that it stops relying on the version of the interface that exposes only the IPv 4-wide MCF (MCF interface version 1.0814) and starts relying on the version of the MCF interface that exposes the IP ranges of both IPv4 and IPv6 (MCF interface version 2.01012). This is shown in fig. 11. Referring to fig. 11, a block diagram depicts a data center configuration according to an embodiment of the present invention and is generally designated as data center configuration 1100. Data center configuration 1100 includes the items depicted in fig. 10, except that bandwidth tracker code 810 is depicted as relying on MCF interface version 2.01012 instead of MCF interface version 1.0814 as a result of being modified by a team of developers.

In one embodiment, to reduce their maintenance burden, the team responsible for the MCF may retire MCF version 1.0, as shown in fig. 12. Referring to fig. 12, a block diagram depicts a data center configuration according to an embodiment of the present invention and is generally designated as data center configuration 1200. The data center configuration 1200 includes the items depicted in fig. 11, except for the MCF interface version 1.0814 and its binding data (MCF data 812) as a result of having been taken charge of the team retirement of the MCF.

The above example shows the following points:

because different classes of data are maintained by different teams according to different processes and schedules, it is convenient to decompose a configuration into separate configuration components that can be maintained independently;

the configuration components may depend on each other, and the code may also depend on the configuration components;

a consistent, explicitly declared set of dependencies between interface versions may isolate different components (and the team responsible for maintaining them) from format changes to the data on which they depend, while still allowing the underlying data (or values) to change at any point in time;

the introduction of a new interface version allows for the gradual inclusion of format changes as follows: these changes do not interrupt any consumer; and

when the interface version is no longer relied upon by any component, it can be retired and no longer needs to be maintained.

Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the spirit and scope of the appended claims. The present technology has been described with an intent to be illustrative rather than restrictive. Alternative embodiments will become apparent to the reader of this invention after or as a result of reading this invention. Alternative means of accomplishing the foregoing may be accomplished without departing from the scope of the appended claims. Certain features and subcombinations are of utility and may be employed without reference to other features and subcombinations and are contemplated within the scope of the claims.

Claims (12)

1. A method of updating a data center configuration, the method comprising:

organizing (510) the data center configuration into configuration components, wherein the configuration components are configuration datasets corresponding to aspects of the data center configuration, and wherein each configuration component comprises a versioned interface, wherein for a given configuration component each version of the interface defines one or more configuration settings that can be queried by a user of the configuration component, and wherein each version of the interface defines one or more data types associated with configuration settings that are returned as a result of a query;

providing (512) a configuration component to a developer, wherein the configuration component is modifiable by the developer separately from the data center configuration;

receiving (514) a modified configuration component from the developer, wherein the modified configuration component is a modified version of the configuration component;

determining (516) whether an interface of the modified configuration component is well-formed, wherein the determining comprises determining whether a value returned by the modified configuration component is consistent with a configuration setting defined in the interface of the configuration component;

accepting (518) the modified configuration component into a hosted set of datacenter configurations when the interface is well-formed, wherein the modified configuration component is available for deployment,

receiving (522) a request to submit the modified configuration component,

determining (524) whether the modified configuration component is globally consistent with a committed version of the datacenter configuration,

accepting (526) the modified configuration component into a managed set of datacenter configurations when the modified configuration component is globally consistent with a submitted version of the datacenter configuration, wherein the modified configuration component is available for deployment by another developer, an

Rejecting (528) the modified configuration component when the modified configuration component is not globally consistent with the submitted version of the datacenter configuration;

rejecting (520) the modified configuration component when the modified configuration component is ill-formed.

2. The method of claim 1, wherein the method further comprises generating a partial branch of configuration data, wherein the partial branch includes the configuration component, wherein the partial branch references a data center configuration not included in the partial branch, and wherein the configuration component is provided to the developer by providing the partial branch to the developer.

3. The method of claim 1, wherein determining whether the modified configuration component is globally consistent with the submitted version of the datacenter configuration comprises determining whether the modified configuration component satisfies a configuration constraint that is enforced across configuration components.

4. The method of claim 1, wherein the method includes generating a second partial branch comprising a set of configuration components that constitute a retained version of configuration data, wherein the set of configuration components is maintained as a version-specific partial branch, wherein the version-specific partial branch needs to be consistent with each configuration component in a current version of a parent branch.

5. A method of updating data center configuration data, the method comprising:

providing (610) a primary branch of configuration data, the primary branch comprising the datacenter configuration data;

providing (612) a configuration component, wherein the configuration component is included in the primary branch, wherein a configuration component is a configuration dataset corresponding to an aspect of the datacenter configuration, wherein the configuration component requires specifying configuration values for a list of datacenter assets, and wherein the configuration component includes dependencies on other configuration components in the primary branch;

providing (614) a partial branch of configuration data to a developer, wherein the partial branch is a copy of a portion of the main branch, wherein the partial branch includes a copy of the configuration component, and wherein the copy of the configuration component is modifiable by the developer;

providing (616) access to the main branch to the partial branch, wherein dependencies of copies of the configuration components on other configuration components in the main branch are implemented during development without including the other configuration components in the partial branch;

in (618) the partial branch, limiting the list of data center assets to a subset of the list of data center assets, wherein if a data center asset in the list of data center assets is not in the subset of the list of data center assets, a configuration component in the partial branch need not specify a configuration value for the data center asset;

receiving (622) a modified configuration component from the developer, wherein the modified configuration component is a modified version of the configuration component;

prior to accepting the modified configuration component into a hosted set of data center configurations, determining (624) that the modified configuration component is globally consistent with a committed version of a data center configuration, wherein determining that the modified configuration component is globally consistent with a committed version of the data center configuration comprises verifying that the modified configuration component is consistent with data center assets that are not in the subset of the list of data center assets; and

accepting the modified configuration component into the hosted set of datacenter configurations when the modified configuration component is globally consistent with a submitted version of the datacenter configuration (626), wherein the modified configuration component is available for deployment.

6. The method of claim 5, wherein the configuration component includes an interface defining named, typed configuration settings that can be queried by a user of the configuration component.

7. The method of claim 5, wherein the developer's modifications to the partial branch are isolated from the data center configuration data.

8. The method of claim 5, wherein determining that the modified configuration component is globally consistent with the submitted version of the datacenter configuration comprises verifying that the modified configuration component is consistent with a configuration component in a second partial branch.

9. A method of managing independent, simultaneous modification of configuration data of a data center, the method comprising:

providing (710) a parent branch of configuration data, the parent branch comprising configuration data of the data center;

organizing (712) the configuration data into components, wherein the components include versioned interfaces that enable one component to query another component for configuration settings, wherein data associated with the queries conforms to data types, wherein for each component, each version of the included interfaces defines one or more configuration setting queries and one or more data types associated with configuration settings for that version of the interface;

generating (714) a first sub-branch of configuration data in response to a request from a first developer, wherein the first sub-branch comprises a first component, wherein the first component is a duplicate of a component in the parent branch, and wherein a modification to the first component in the sub-branch does not affect the parent branch;

providing (716) the first sub-branch to the first developer, wherein the first developer is enabled to modify the first component;

generating (718) a second child branch of configuration data in response to a request from a second developer, wherein the second child branch comprises a second component, wherein the second component is a copy of another component in the parent branch;

providing (720) the second sub-branch to the second developer, wherein the second developer is enabled to modify the second component;

receiving (722) the first component from the first developer for integration into the parent branch, wherein the first component has been modified; and

upon receiving the modified first component, and prior to integrating the modified first component into the parent branch,

(A) verifying (724) that the data type of the data in the modified first component is consistent with the components in the parent branch,

(B) verifying (724) that the modified first component satisfies a global configuration constraint,

(C) verifying (724) that a query associated with the modified first component is well-formed; and

(D) verifying (724) that a data type of data in the modified first component is consistent with components in the second subbranch.

10. The method of claim 9, wherein verifying that the data type is consistent with components in the parent branch comprises verifying that the data type is consistent across upstream components.

11. The method of claim 9, wherein verifying that a query is well-formed comprises verifying that the type of value returned by the query for each member of the interface is consistent with an interface declaration.

12. The method of claim 9, wherein a component is a configuration dataset corresponding to an aspect of the data center configuration, and wherein the aspect is one of a physical asset, a logical asset, and a policy.