CN115801833A - Enterprise-level public cloud resource management method and system - Google Patents

Abstract

The invention provides a method and a system for managing enterprise-level public cloud resources, wherein the method for managing the enterprise-level public cloud resources comprises the following steps: the management platform processes the resource data in a unified login account and third party storage mode by combining with STS service provided by the cloud platform; a resource timing synchronization mode is adopted; a thread pool technology is adopted, and a plurality of account resources are synchronized simultaneously; introducing a NoSql database, and storing large-data-volume resource data; introducing an organization relation of enterprise tenant isolation; the management platform analyzes the resource utilization rate according to the statistical condition of the resource utilization and generates a resource optimization scheme; and integrating various resource cost data of different cloud platforms. Promote enterprise's public cloud resource management fortune dimension efficiency, the expenditure of using manpower sparingly reduces fortune dimension cost, promotes resource utilization, improves the security, provides unified resource data simultaneously, makes things convenient for the function demand to expand, optimizes resource and expense input.

Description

An enterprise-level public cloud resource management method and system

technical field

The invention relates to the field of data supervision, in particular to an enterprise-level public cloud resource management method and system.

Background technique

When an enterprise uses a public cloud, it often uses organization management services to manage multiple accounts, such as AWSOrganizations, Alibaba Cloud Resource Management (Resource Management) service, Tencent Cloud Organization (TencentCloud Organization) service, Huawei Cloud Enterprise Management service, etc. For a large enterprise, there may be hundreds or even hundreds of accounts, and each account is assigned to a certain user, who will frequently operate common resources, mainly including basic computing resources (cloud server, mirroring, resource orchestration, etc.), Storage resources (data volumes, file storage, object storage, etc.), network resources (VPC, security groups, public network IP, routers, load balancing, etc.), and service resources such as databases, containers and middleware, security, monitoring and alarms, etc., usually The number of enterprise resources is as follows: more than 1,000 cloud hosts, more than 1,000 cloud data volumes, more than 500 networks, storage/database and other services are used indefinitely as needed. For end users, to operate these resources, they need to log in to the public cloud platform separately. If the enterprise uses multiple cloud platforms, such as AWS, Alibaba Cloud, Tencent Cloud, etc., they need to log in to the corresponding cloud platform to perform operations; for department or enterprise operation and maintenance Administrators need to manually count the resources and expenses of each account before collecting statistics on the overall resource and expense usage of the department or enterprise. This resource management method requires operation and maintenance personnel to record a large number of account information and manually maintain each account resource. The repetitive workload is heavy, the operation is cumbersome, and there is a risk of account password leakage in batches.

Contents of the invention

In view of the above problems, the present invention is proposed to provide an enterprise-level public cloud resource management method and system that overcomes the above problems or at least partially solves the above problems.

According to one aspect of the present invention, an enterprise-level public cloud resource management method is provided, the management method comprising:

The management platform combines the STS service provided by the cloud platform, and uses a unified login account + third-party storage to process resource data;

Using resource timing synchronization;

Use thread pool technology to synchronize multiple account resources at the same time;

Introduce NoSql database to save large amount of resource data;

Introducing organizational relationships for enterprise tenant isolation;

The management platform analyzes the resource usage rate according to the resource usage statistics and generates a resource optimization plan;

Integrate various resource cost data of different cloud platforms.

Optionally, the introduction of the organizational relationship of enterprise tenant isolation specifically includes: establishing a top-down management method, automatically corresponding to the organizational relationship of the cloud platform, automatically creating department and user information according to the cloud platform user organization, and automatically binding resources to which they belong organizations and users.

Optionally, the integration of various resource cost data on different cloud platforms specifically includes: according to different needs, counting the cost usage of each organization and each resource type from different dimensions, so as to provide data basis for resource optimization, resource allocation, and cost prediction.

Optionally, the management method also includes: creating a unified login account on the cloud platform and authorizing policies, specifically including:

Select the main account under the cloud platform organization to log in, create a sub-user under the main account, and generate AccessKeyId and AccessKeySecret, and use the main account as the unified login account of the organization;

Create a custom policy under the unified login account of the organization and authorize access to the AssumeRole of the STS;

Authorize the custom policy to a sub-account of the unified login account of the organization.

Optionally, the management method further includes: verifying the configuration;

After logging in to the sub-account of the organization's unified login account, use the remaining member accounts and roles to switch roles to verify whether the switch can be performed normally.

Optionally, the unified management of resources specifically includes:

Use AccessKeyId, AccessKeySecret and STS services, call the API provided by the cloud platform to obtain resource information, save it in the local database, perform secondary processing on resources, and realize unified management of resources through the management system function.

The present invention also provides an enterprise-level public cloud resource management system, the management system comprising:

The resource management module is used to manage the platform combined with the STS service provided by the cloud platform, and use the unified login account + third-party storage to process resource data;

A data synchronization module, configured to perform data synchronization in a resource timing synchronization manner;

The account resource synchronization module is used to synchronize multiple account resources at the same time using thread pool technology;

The resource data storage module is used to introduce the NoSql database and save large amount of resource data;

The organization relationship import module is used to introduce the organization relationship isolated by enterprise tenants;

The optimization plan generation module is used for the management platform to analyze the resource utilization rate and generate a resource optimization plan according to the resource usage statistics;

The data integration module is used to integrate various resource cost data of different cloud platforms.

An enterprise-level public cloud resource management method and system provided by the present invention, the management method includes: the management platform combines the STS service provided by the cloud platform, adopts a unified login account + third-party storage method to process resource data; adopts resource timing synchronization Method; use thread pool technology to synchronize multiple account resources at the same time; introduce NoSql database to save large amount of resource data; introduce organization relationship of enterprise tenant isolation; management platform analyzes resource usage rate according to resource usage statistics, and generates resource optimization plan ; Integrate various resource cost data on different cloud platforms. Improve the efficiency of enterprise public cloud resource management and operation and maintenance, save labor expenses, reduce operation and maintenance costs, improve resource utilization, improve security, and provide unified resource data to facilitate the expansion of functional requirements and optimize resource and cost investment.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

FIG. 1 is a schematic diagram illustrating the basic configuration of the public cloud STS provided by the embodiment of the present invention;

FIG. 2 is a unified management system for public cloud organization resources provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

The terms "comprising" and "having" and any variations thereof in the description, embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusion, for example, including a series of steps or units.

The technical solutions of the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

As shown in Figure 1, the management platform combines the STS service provided by the cloud platform, and uses a unified login account + third-party storage to process resource data to improve the efficiency and security of platform operation and maintenance; use resource timing synchronization to ensure management platform resources The synchronization between the state and the actual state of the cloud resources reduces the calling frequency of the cloud platform API interface and saves bandwidth usage; the thread pool technology is used to synchronize multiple account resources at the same time, improving the efficiency of data fetching and storage; the introduction of NoSql database, Save large amount of resource data, improve data storage and reading performance; introduce the organizational relationship of enterprise tenant isolation, establish a top-down management method, automatically correspond to the organizational relationship of the cloud platform, and automatically create departments and users according to the cloud platform user organization Information, automatically binds the organizations and users to which resources belong, reduces a large amount of manual operation and maintenance workload, and improves operation and maintenance efficiency; the management platform can analyze resource usage according to resource usage statistics, generate resource optimization plans, improve enterprise resource utilization, and reduce Cost: It can integrate various resource cost data of different cloud platforms, and count the cost usage of each organization and resource type from different dimensions according to different needs, and provide data basis for resource optimization, resource allocation, and cost forecasting. The above method is used to establish a resource management platform system to improve the efficiency of cloud platform enterprise-level operation and maintenance. By integrating data from different cloud platforms, enterprise resources can be managed in a unified, efficient and fast manner, and functions can be expanded based on unified data to improve resource utilization.

The present invention is a fast, efficient, convenient, unified and safe solution to improve enterprise-level public cloud resource management based on the existing technology of public cloud, so as to improve the efficiency of enterprise public cloud resource management operation and maintenance , save manpower expenses, reduce operation and maintenance costs, improve resource utilization, improve security, and provide unified resource data to facilitate the expansion of functional requirements, optimize resources and cost inputs, etc. This solution is applicable to all cloud platforms that support STS services.

The working principle provided by the present invention

Create a unified login account and authorize policies on the cloud platform

Select the main account of a cloud platform organization to log in, create a sub-user under this account, and generate AccessKeyId and AccessKeySecret, this account is used as the unified login account of the organization (hereinafter referred to as OUA (Organization unified login account));

Create a custom policy under the OUA sub-account and authorize access to the AssumeRole of the STS;

Authorize the policy in step 2 to the OUA sub-account.

Configure other accounts under the organization except the OUA account, including:

Create a policy under all member accounts except the OUA account, and authorize the policy authority for the entrusted management of the OUA account (that is, which resources under the specified account can be managed through the OUA account); or use an existing policy that meets the requirements;

Create a role under the account corresponding to step 1 (here, uniformly name the roles created under all accounts, such as AssumeRoleForCmp), and authorize the policy created in step 1 to this role;

Add a trust policy to the role created in step 2 to make it trust the delegated management of the OUA account.

Verify the configuration. After logging in to the OUA sub-account, use other member accounts and roles (AssumeRoleForCmp) to perform role switch operations to verify whether the switch can be performed normally.

The unified management of resources specifically includes: using AccessKeyId, AccessKeySecret and STS services, calling the API provided by the cloud platform to obtain resource information, saving it to the local database, performing secondary processing on resources, and realizing unified resource management through management system functions. The system provides two resource synchronization methods: manual synchronization and automatic synchronization. Automatic synchronization is scheduled synchronization every day when you are free, and the synchronization time can be configured.

Use AccessKeyId + AccessKeySecret + organization management account ID + entrusted management role (AssumeRoleForCmp) to perform temporary token security authentication, call the organization and account API interface, query the organization information and account list, save its data and relationships to the local database, and follow-up maintenance query account directly Select and obtain from the local database, no manual maintenance is required;

Use AccessKeyId+AccessKeySecret+organization member account ID+delegate management role (AssumeRoleForCmp) to perform temporary token security authentication for a specific account, call the resource query interface, query the required resources and save them in the local database, and save the corresponding relationship between resources and accounts. Saving resources to the local database is to reduce the frequency of calling the cloud platform interface. If this problem is not considered, resource information can be queried in real time; synchronization resources mainly include organization information, account, VPC, subnet, network card, public network IP, router , load balancing, security group, instance type, image, key pair, cloud host, data volume, data volume snapshot, RDS database, file storage, object storage, container service, monitoring alarm, cost information and other common resources.

Use AccessKeyId + AccessKeySecret + account ID to which the resource belongs + delegate management role (AssumeRoleForCmp) for temporary token security authentication, and use temporary security tokens for resource management, such as switching on and off the cloud host. When managing existing resources in the local database, the account to which the resource belongs has been bound to the resource, and there is no need to manually specify the account information; when creating a resource, the account information is pulled from the database for the operator to choose, and no manual filling is required.

Based on the data saved locally, flexible visual development is carried out according to specific needs, and all resources under the organization are displayed on the management platform, which facilitates the unified management of resources and saves operation and maintenance costs.

Data synchronization adopts thread pool technology, with account as the dimension, and multi-thread concurrent synchronization. Because it mainly involves resource network requests and data storage, the number of core threads here is determined by the IO-intensive calculation method, that is, the number of threads = the number of cpu cores/(1-0.8), such as the 4-core server running system, set the number of threads for 20. Assuming there are 200 accounts, using one-by-one synchronization, the synchronization takes about 40 minutes (the synchronization time is related to the amount of resources, the more resources the more time it takes); the multi-threaded concurrent synchronization only takes about 5 minutes. It can be completed, which greatly improves the resource reading efficiency.

Taking the AWS public cloud as an example, configure STS services and obtain public cloud resources through STS authentication information.

The present invention improves the resource management method based on the current technology of the public cloud, obtains the resources under all accounts of the organization through the API interface, stores them in the local database for unified management, and can integrate the resource data according to the requirements, extracting resources needed for resource optimization Data, thereby improving operation and maintenance efficiency, reducing manual maintenance resource costs, and improving resource utilization.

Cloud platform STS configuration, as shown in Figure 1, needs to pre-configure the organization account, as follows:

Log in to the organization management account, enter the organization management service, and select an account as the unified login account;

Log in to the OUA main account and create an AssuleRole permission policy for STS;

A. Locate the policy menu and click the "Create Policy" button;

B. Select configuration: STS service, write AssumeRole, all resources;

C. Set the label (optional);

D. Set the name and confirm the creation strategy;

E. Check the created policy to confirm whether it is correct.

Create a sub-user (unified login sub-user), generate AccessKeyId and AccessKeySecret, and authorize the AssumeRole of STS;

Enter the IAM user list, click the "Add User" button; set the user name, check (required) access key - programming access, check the console option as needed; set permissions, directly attach the existing policy, that is, use step 2 ) created the AssumeRole policy; add tags; confirm the creation of sub-users.

Create a role under all non-unified login member accounts, name it AssumeRoleForCmp, authorize the corresponding permissions for the role (the test can only authorize the read permission of all resources), and configure the trust relationship of the unified login account for the role;

Enter the role service, click the "Create role" button;

Select a trusted entity, use the account option here, trust another account, that is, "unified login account"; add permissions, set resource management permission policies as needed; set the name AssumeRoleForCmp, confirm the information; confirm the creation of the role.

Verify the configuration, that is, verify whether the "switch role" function is correct on the cloud platform;

Click the drop-down box in the upper right corner of the account information on the cloud platform, and click "Switch Role";

Enter the account and role to be switched (AssumeRoleForCmp);

After the switch is successful, the name filled in by the jump is displayed in the upper right corner.

Unified management of resources. The configured cloud organization accounts shown in Figure 2 perform unified management of resources.

Save the AccessKeyId and AccessKeySecret of the OUA account, as well as the organization management account ID and authorization role information of the entrustment letter to the management system; since the account information is sensitive information, data encryption is required for storage. The system adopts RSA+AES+Base64 multiple encryption methods to ensure account security.

According to the AccessKeyId and AccessKeySecret of the OUA account, the organization management account ID, and the authorization role, obtain the temporary security access token of the organization management account through the STS service.

Use temporary security tokens to get all organization member accounts and save them to the local database.

For each member account, use the corresponding temporary security token to call the API of the relevant resource service to obtain resources. Using thread pool technology, taking IO-intensive tasks as a reference, the number of threads = number of CPU cores/(1-0.8), a separate thread is opened for each account, and multiple threads pull data concurrently, which greatly improves resource synchronization efficiency.

Process the data synchronized to the management system.

Establish an organization management module based on the organizational relationship model of enterprises, multi-level departments, and users, divide resources into organizations, and realize resource tenant isolation management;

Use resource permission configuration + role authorization to realize resource access permission management, set different permissions for users with different roles, and realize hierarchical and hierarchical management of resources;

Divide resources into modules by type, mainly divided into computing, storage, network, container, database services and other resources, as well as extended functions such as resource overview and application management;

The system provides metering audit information, mainly including task execution records, audit lists (resource access records), login logs, resource metering statistics, etc.;

The billing management module performs multi-dimensional statistical display based on the synchronized expense bill data, combined with organizational relationships, counts enterprise expense bills, department expense bills, and single-account expense bills, checks expense income and expenditure details, and predicts expense usage based on historical expense information trend.

The scheduled task module mainly sets the scheduled startup, shutdown, and restart of the cloud host, and performs data backup and other operations on the data volume and cloud host, which can greatly reduce the cost of manual operation and maintenance.

The monitoring and alarm module uses ElasticSearch as a time-series database to store monitoring data obtained from the cloud platform, speed up retrieval efficiency, and facilitate data aggregation and statistics.

The log service mainly records system operation logs, mainly including the logs automatically printed by the system, system error logs and other information, which are used for system problem tracking and maintenance.

Function Expansion Description

Since there are many categories of public cloud resources and services, and each category or service has a different data structure and magnitude, here are some practical suggestions for this situation.

For general common resources, such as cloud hosts, cloud hard disks, networks and other information, from the perspective of order of magnitude, use relational databases (such as MySql) for data storage to facilitate business expansion;

For service resources such as object storage, since the number of objects may be large, if you use a relational database, you need to consider sub-database and sub-table storage; consider using non-relational databases (such as Hbase, MongoDB, etc.) for storage to improve data storage and read performance;

For monitoring data, it is recommended to consider using time-series database storage, such as InfluxDB, Prometheus, ElasticSearch, etc., to facilitate high-performance query and display of data.

Beneficial effects: Using the unified management method of organizational resources, it can automatically manage the information of multiple accounts of the organization, reduce the repetitive work of manually maintaining multiple accounts of the organization, reduce the risk of leakage of account key information, and improve the efficiency and security of resource management;

The resources under the organization are synchronously integrated into the local database, and the resource data can be flexibly customized and expanded, and resource usage can be optimized through the use of resources under the organization to effectively improve resource utilization;

After the unified processing of organizational resources, there is no need to log in to the cloud console one by one for resource management, which can simplify the steps of operation and maintenance personnel to operate resources, make the operation more convenient and fast, and improve the efficiency of operation and maintenance. If there are 100 enterprise accounts, it is necessary to shut down the cloud hosts under each account. Using the traditional management method, it takes about 50 minutes to log in to each account to perform the shutdown operation; The host performs power-on/off operations in batches, and it only takes about 1 minute or less to complete all operations.

There is a mandatory security policy on the public cloud platform. For example, the password or key must be changed every 90 days. If each account is changed once, it will take about 1 minute for each account to complete the operation; the management platform manages resources through STS. You only need to care about the unified login account and change the information of one account, which greatly reduces the maintenance cost.

The above specific implementation manners have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation modes of the present invention, and are not used to limit the protection scope of the present invention. Within the spirit and principles of the present invention, any modifications, equivalent replacements, improvements, etc., shall be included in the protection scope of the present invention.

Claims (7)

1. An enterprise-level public cloud resource management method, the management method comprising:

the management platform processes the resource data in a unified login account and third party storage mode by combining with STS service provided by the cloud platform;

a resource timing synchronization mode is adopted;

a thread pool technology is adopted, and a plurality of account resources are synchronized simultaneously;

introducing a NoSql database, and storing large-data-volume resource data;

introducing an organization relation of enterprise tenant isolation;

the management platform analyzes the resource utilization rate according to the statistical condition of the resource utilization and generates a resource optimization scheme;

and integrating various resource cost data of different cloud platforms.

2. The method according to claim 1, wherein the introducing of the organization relationship of enterprise tenant isolation specifically comprises: and establishing a top-down management mode, automatically corresponding to the organization relation of the cloud platform, and automatically binding the organization and the user to which the resource belongs according to the information of automatically creating departments and users of the user organization of the cloud platform.

3. The method according to claim 1, wherein the integrating of the resource charge data of different cloud platforms specifically comprises: according to different requirements, the cost use conditions of each organization and each resource type are counted from different dimensions, and data bases are provided for resource optimization, resource allocation and cost prediction.

4. The method of claim 1, wherein the method further comprises: the method for creating the unified login account and authorizing the strategy on the cloud platform specifically comprises the following steps:

selecting a primary account number under a cloud platform organization for login, creating a sub-user under the primary account number, and generating an Access KeyId and an Access KeySecret, wherein the primary account number is used as an organization unified login account number;

establishing a custom strategy under the organization unified login account, and authorizing access to the AssumeRole of the STS;

and authorizing the custom strategy to a sub-account of the organization unified login account.

5. The method of claim 4, wherein the method further comprises: verifying the configuration;

and after logging in the sub account of the organization unified login account, using other member accounts and roles to execute role switching operation, and verifying whether normal switching can be performed or not.

6. The method according to claim 1, wherein the unified resource management specifically comprises:

and calling an API (application programming interface) provided by the cloud platform to acquire resource information by using the Access KeyId, the Access KeySecret and the STS (service specification) service, storing the resource information into a local database, performing secondary processing on the resources, and realizing unified management of the resources through the functions of the management system.

7. An enterprise-level public cloud resource management system, the management system comprising:

the resource management module is used for processing resource data by combining the STS service provided by the cloud platform and the management platform in a unified login account and third party storage mode;

the data synchronization module is used for synchronizing data in a resource timing synchronization mode;

the account resource synchronization module is used for synchronizing a plurality of account resources simultaneously by adopting a thread pool technology;

the resource data storage module is used for introducing a NoSql database and storing large-data-volume resource data;

the organization relation introduction module is used for introducing an organization relation isolated by the enterprise tenants;

the optimization scheme generation module is used for analyzing the resource utilization rate and generating a resource optimization scheme by the management platform according to the resource utilization statistical condition;

and the data integration module is used for integrating various resource cost data of different cloud platforms.

Images