CN111813545A - A resource configuration method, apparatus, medium and device - Google Patents

Abstract

The present disclosure provides a resource configuration method, apparatus, medium and device. The method includes: acquiring a job task submitted by a user, wherein the job task includes a job label for marking a job type and a resource declaration, the resource declaration Including at least one resource information required to complete the job task; querying historical resource information of the job task based on the job label; Obtained by real-time monitoring and analysis; calculating the predicted resource requirement of the job task based on the historical resource information of the job task; allocating at least one resource based on the predicted resource requirement. The present disclosure can recommend and adjust the resources of the task according to the historical operation of the task, so that the user's resource application is more reasonable, thereby ensuring the reasonable and healthy use of the resources of the entire cluster.

Description

A resource configuration method, apparatus, medium and device

technical field

The present disclosure relates to the field of computer technology, and in particular, to a resource configuration method, apparatus, medium and device.

Background technique

In the process of processing the tasks entered by the user into the cluster system, the scheduling system needs to allocate system resources according to the submitted tasks, so as to reasonably use the system resources and efficiently process the user tasks. When a user submits a task, they need to judge and specify the size of the required resources, but they often cannot accurately estimate the resources required by the task. At the same time, in order to ensure the safe operation of the task, users often apply for larger resources. In the prior art, if the resources actually used during the task operation do not reach the size of the declared resources, it will cause a waste of resources. If the actual resources used by the user far exceed the size of the declared resources, when the machine resources are insufficient, the user tasks will be seriously affected. operation. The current scheduling system also cannot monitor and display the actual usage of real-time resources of queues, applications and containers, and cannot dynamically adjust the allocated resources according to the usage of real-time resources, which often leads to waste or shortage of resources.

Therefore, dynamic monitoring of the actual usage of system resources is of great significance for users to apply for resources more reasonably and for cluster resource tuning.

public content

The purpose of the present disclosure is to provide a resource configuration method, apparatus, medium and device, which can solve at least one of the above-mentioned technical problems. The specific plans are as follows:

According to specific embodiments of the present disclosure, in a first aspect, the present disclosure provides a resource configuration method, including: acquiring a job task submitted by a user, wherein the job task includes a job label and a resource declaration for marking the job type, and the The resource declaration includes at least one resource information required to complete the job task; the historical resource information of the job task is queried based on the job label; the historical resource information The job task is obtained by real-time monitoring and analysis; the predicted resource requirement of the job task is calculated based on the historical resource information of the job task; at least one resource is allocated based on the predicted resource requirement.

Optionally, the method further includes: monitoring in real time the running status of resources under each job label of the system and the relationship between the at least one resource allocated and the job task, and dynamically adjusting the at least one resource allocated.

Optionally, the real-time monitoring of the running status of resources under each job label of the system and the relationship between the at least one resource that has been allocated and the job task, and dynamically adjusting the at least one resource allocated include: The resource management module of the scheduling system obtains application meta information, the application meta information includes the application ID, application name, queue name, cluster name and application label when the task is running; the resource information of the container is obtained through the node management module of the scheduling system ; associate the application meta-information with the corresponding resource information of the container; monitor the associated application meta-information and the resource information of the container in real time, and analyze the resource application situation and actual resource usage of the container.

Optionally, the obtaining the resource information of the container through the node management module of the scheduling system includes: obtaining the resource information of the container through the node management module of the scheduling system, where the resource information of the container includes a process type and a disk type. ; Obtain resource information of each sub-process in the process type through the process tree module, and the resource information includes CPU information, memory information, input and output information or network interface information.

Optionally, the real-time monitoring of the associated application meta-information and the resource information of the container, and analyzing the resource application situation and actual resource usage of the container, includes: monitoring the CPU usage under the current task in real time, The usage rate of memory, the usage rate of input and output devices, or the usage frequency of network interfaces, and the resource application status and actual resource usage status of the container are analyzed according to the monitoring results.

Optionally, after analyzing the resource application situation and the actual resource usage situation of the container according to the monitoring result, it includes: when the usage rate of the CPU, the usage rate of the memory, the usage rate of the input and output devices, or the usage of the network interface. When the frequency exceeds the first threshold, a first alarm signal is sent; the scheduling system reallocates resource information for generating the first alarm signal according to the first alarm signal.

Optionally, after analyzing the resource application situation and the actual resource usage situation of the container according to the monitoring result, it includes: when the usage rate of the CPU, the usage rate of the memory, the usage rate of the input and output devices, or the usage of the network interface. When the frequency is lower than the second threshold, a second alarm signal is sent; the scheduling system reallocates the resource information for generating the second alarm signal according to the second alarm signal.

Optionally, the method further includes: recording the resource information for generating the first alarm signal and/or the second alarm signal into the historical resource information and updating the historical resource information.

According to a specific embodiment of the present disclosure, in a second aspect, the present disclosure provides a resource configuration apparatus, comprising: an acquisition unit configured to acquire a job task submitted by a user, wherein the job task includes a job label for marking a job type and a resource declaration, the resource declaration includes at least one resource information required to complete the job task; a query unit is configured to query the historical resource information of the job task based on the job tag; the historical resource information The completed job tasks with the same job label are obtained by real-time monitoring and analysis; a computing unit is used to calculate the predicted resource demand of the job task based on the historical resource information of the job task; an allocation unit is used to calculate the predicted resource demand of the job task based on the Forecasting resource requirements allocates at least one resource.

According to a specific embodiment of the present disclosure, in a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, the program implementing the method described in any one of the above when the program is executed by a processor.

According to specific embodiments of the present disclosure, in a fourth aspect, the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are The one or more processors, when executed, cause the one or more processors to implement a method as described in any of the above.

Compared with the prior art, the above solutions of the embodiments of the present disclosure have at least the following beneficial effects: the resource configuration method provided by the present disclosure dynamically understands the configuration of system resources by monitoring the running status of system resource information and the status of allocated resources in real time Is it reasonable? If it is unreasonable, adjust the allocation status in a timely manner. For tasks with application resources greater than the actual resources used, the task resource utilization can be optimized, thereby further optimizing the cluster resource utilization. Improve the running speed of user tasks and reduce the probability of user job failure due to OOM. For tasks that run periodically, the present invention can recommend and adjust the resources of the tasks according to the historical running conditions of the tasks, so that the user's resource application is more reasonable, thereby ensuring that the resource usage of the entire cluster is reasonable and healthy.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort. In the attached image:

FIG. 1 shows a structural diagram of a resource configuration system according to an embodiment of the present disclosure;

FIG. 2 shows a structural diagram of a resource configuration module according to an embodiment of the present disclosure;

FIG. 3 shows a flowchart of a resource configuration method according to an embodiment of the present disclosure;

FIG. 4 shows a flowchart of a data collection method according to an embodiment of the present disclosure;

FIG. 5 shows a schematic structural diagram of a resource configuration apparatus according to an embodiment of the present disclosure;

FIG. 6 shows a schematic diagram of a connection structure of an electronic device according to an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

The terms used in the embodiments of the present disclosure are for the purpose of describing particular embodiments only, and are not intended to limit the present disclosure. As used in the embodiments of the present disclosure and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise, "a plurality" Generally at least two are included.

It should be understood that the term "and/or" used in this document is only an association relationship to describe the associated objects, indicating that there may be three kinds of relationships, for example, A and/or B, which may indicate that A exists alone, and A and B exist at the same time. B, there are three cases of B alone. In addition, the character "/" in this document generally indicates that the related objects are an "or" relationship.

It should be understood that although the terms first, second, third, etc. may be used to describe . . . in the embodiments of the present disclosure, these . . . should not be limited to these terms. These terms are only used to distinguish ... For example, the first... may also be referred to as the second..., and similarly, the second... may also be referred to as the first... without departing from the scope of the embodiments of the present disclosure.

Depending on the context, the words "if", "if" as used herein may be interpreted as "at" or "when" or "in response to determining" or "in response to detecting". Similarly, the phrases "if determined" or "if detected (the stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detected (the stated condition or event)," depending on the context )" or "in response to detection (a stated condition or event)".

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion, such that a commodity or device comprising a list of elements includes not only those elements, but also those not explicitly listed other elements, or other elements inherent to such goods or devices. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the article or device that includes the element.

The optional embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG. 1 shows a structural diagram of a resource configuration system according to an embodiment of the present disclosure. With reference to FIG. 1 , the resource configuration system according to the embodiment of the present disclosure includes three modules, which are a data acquisition module, a data archiving module, and a data processing module. FIG. 2 shows a specific resource configuration module structure diagram corresponding to the structure diagram of the resource configuration system in FIG. 1 .

As shown in Figure 1, the data acquisition module is a module embedded in the scheduling system, for example, embedded in the scheduling system YARN (Yet Another Resource Negotiator). The data acquisition module is responsible for collecting the actual resource usage of each container in the cluster. And send the data to the data archiving module in real time. The user labels the completion of the current job task. For example, if the current job task is a retouching task, the retouching task is labelled. If the current job task is a table calculation, the table calculation task is labelled, and the data collection module is used for labeling. The resources required for the marked tasks are collected for subsequent analysis and monitoring.

The process of collecting data through the scheduling system, as shown in Figure 2, the scheduling system YARN includes a resource management module RM and a node management module NM, and the application meta information is obtained through the resource management module RM of the scheduling system, and the application meta information includes Application ID, application name, queue name, cluster name, and application label when the task is running; resource information of the container is obtained through the node management module NM of the scheduling system. The scheduling system YARN periodically transmits the APP status in the memory of the resource management module RM to the meta-information message queue, starts the stream processing job downstream of the meta-information message queue, and transfers the container information and application meta-information in real time through the stream processing job. The ID or higher dimensions of the APP are combined to form real-time resource usage information including various application meta-information and indicators.

The data archiving module is a stable storage system used to store the resource usage of all containers. The data archiving module can be the OLAP storage system as shown in Figure 2 or other forms of storage systems. The OLAP storage system includes, for example, the database Clickhouse, etc. . The data is sent to the OLAP storage system, and users can monitor the application and actual usage of resources in the OLAP storage system in real time through the monitoring system. Obtain the required data for analysis.

The data processing module includes a data monitoring module and a data analysis module, which are used to split the archived data according to the user's task label, and through the historical resource usage of related tasks, such as taking the maximum value, taking the average value or other automatic data. Define the model to get the most suitable resource for the user to submit the next task. In the next scheduling, the scheduling system sets the application resource amount in the task submitted by the user as the result calculated by the data processing module.

Specifically, as shown in FIG. 3, according to the specific implementation manner of the present disclosure, the present disclosure provides a resource configuration method, which includes the following method steps:

Step S304: Acquire a job task submitted by the user, wherein the job task includes a job label for marking the job type and a resource declaration, and the resource declaration includes at least one resource information required to complete the job task.

The cluster receives the job task submitted by the user through the resource scheduling system, wherein the job task includes a job label and at least one resource declaration. The job label is the task type information marked according to the job task, for example, a drawing task is marked as a drawing label, an audio task is marked as an audio label, and the like. The resource declaration includes various resource types and quantity information required to complete the job task corresponding to the job tag, such as CPU information, memory information, storage information, network interface information, input and output information, etc. The task obtains the required information. Resource information and then perform related jobs to complete the task.

Resource scheduling systems include but are not limited to YARN, Mesos, etc. The following uses YARN as an example to illustrate. The cluster receives job tasks submitted by users through the resource scheduling system YARN. The RM module of the resource scheduling system YARN is responsible for the resources and systems required by the job tasks. The resource status schedules the task to an NM module in the cluster, and the NM module assigns the resource corresponding to the job task to run.

The various resources required in the job tasks submitted by the user are divided by the data processing module, which can be divided according to the actual tasks. Different tasks have different division methods, and the same task can also be divided according to historical data analysis. There are different splitting methods, which are not specifically limited. As an example, including but not limited to the following splitting forms. For example, the user resource declaration for drawing tasks indicates that CPU, GPU, 100M memory, and network interface are required. etc., the system can split the task into CPU0, GPU0, 200M memory, network interface 1; CPU0, GPU0, GPU1, 100M memory 1, 100M memory 2, network interface 1; CPU0, CPU1, GPU0, GPU2, 100M memory 1 , 50M memory 3, network interface 2, etc., we will not do a group list here, as long as the split resources can meet the job tasks.

Step S306 : query historical resource information of the job task based on the job label; the historical resource information is obtained by performing real-time monitoring and analysis on completed job tasks with the same job label.

According to the label information of the job task, the resource information used in the corresponding historical task under the current task is queried, and the corresponding resource is allocated to the current task according to the resource information configured in the historical task. For example, if the current task is a drawing task, the historical resource usage of the task under the label related to the drawing task is calculated. For example, the historical resource usage is: 1 GPU, 100M memory, 1 GPU, 150M memory, etc. Perform subsequent resource analysis and allocation based on the query results.

Step S308: Calculate the predicted resource demand of the job task based on the historical resource information of the job task.

According to step S306, the historical resource information usage under the current task is obtained by query, and the predicted resource usage is given reasonably; the prediction methods include but are not limited to the mean method, the increment method, the shrinkage method, the minimum value method, the maximum value method, etc. As an example, it can be understood that, for example, when the system needs 100M of memory, according to the above historical data analysis, usually 100M of memory cannot meet the execution of the task, then the system will automatically allocate more than 100M of memory to the task, such as increasing 20 %-30% redundant space. According to the above historical data analysis, usually 100M of memory is far enough for the execution of the task and usually only 10M of memory is enough, then the system will automatically allocate 30M of memory to the task, which will increase by 20%-30 under the previous task data. % redundant space. For another example, if the memory usage under the current tag task given by the historical data is 100M, 150M, and 200M, then the average value method is used to allocate a content of 150M, or the maximum value method is used to allocate a 200M memory, or the minimum value method is used to allocate a memory. 100M of memory is possible.

Step S310: Allocate at least one resource based on the predicted resource demand.

After querying, analyzing, and calculating the status of the tasks to be allocated, the system allocates at least one resource according to the remaining state of the current resources. As an example, for example, when a user submits a task, it is considered that the task requires 100M memory. After query, analysis, and calculation After that, 130M of memory should be allocated normally, but at this time, in the case of ensuring system security, only 120M can be allocated, and the system will allocate 120M of memory to the current task.

Optionally, step S312 is further included: monitor the running status of resources under each job label of the system in real time and the relationship between the at least one resource that has been allocated and the job task, and dynamically adjust the at least one resource allocated.

As one embodiment, as shown in FIG. 2 , the real-time monitoring system for the resource running status under each job label includes: acquiring application meta information through the resource management module of the scheduling system, where the application meta information includes task running application ID, application name, queue name, cluster name, and application label at the time. Obtain the resource information of the container through the node management module of the scheduling system; collect the application meta-information and the resource information of the container on the same node; monitor the application meta-information and the container stored under the same node in real time The resource information of the container is analyzed, and the resource application status and the actual resource usage status of the container are analyzed.

As one embodiment, as shown in FIG. 4 , the obtaining the resource information of the container through the node management module of the scheduling system includes: obtaining the resource information of the container through the node management module of the scheduling system, the container The resource information includes process type and disk type; the resource information of each sub-process in the process type is obtained through the process tree module, and the resource information includes CPU information, memory information, input and output information or network interface information.

The node management module NM periodically obtains all container information in the memory. For each container, the node management module NM can obtain various container-level resource types such as the process tree and disk path belonging to the container. For various container-level resource types, it can be further collected to the lower layer. For example, for the process type, the information of the process can be collected for each process. For each process, you can use the process tree cgroup or directly read /proc and other statistical methods to obtain various resources used by the process, including but not limited to cpu, mem, io, and so on. The node management module NM aggregates the data from bottom to top to the container, and then sends it to the message queue.

Optionally, the real-time monitoring of the application meta information and the resource information of the container stored under the same node, and the analysis of the resource application situation and the actual resource usage of the container, includes: monitoring the CPU under the current task in real time. The usage rate, memory usage rate, input/output device usage rate, or network interface usage frequency, and analyze the resource application status and actual resource usage status of the container according to the monitoring results. The system call monitoring module monitors the allocated system resources in real time to make a reasonable analysis. For example, when the memory application index is 100M, the actual situation is usually kept at about 80M, which means that the system application index and the allocation index are relatively matched. Otherwise, the use remains in a low state or overflow occurs, which means that the allocation state has not achieved the expected effect. .

Optionally, after analyzing the resource application situation and the actual resource usage situation of the container according to the monitoring result, it includes: when the usage rate of the CPU, the usage rate of the memory, the usage rate of the input and output devices, or the usage of the network interface. When the frequency exceeds the first threshold, a first alarm signal is sent; the scheduling system reallocates resource information for generating the first alarm signal according to the first alarm signal. Similarly, as an example, for example, when the memory request index is 100M, and the actual situation usually reaches the threshold of 95M, the system will generate an alarm signal, indicating that there is a risk of overflow at this time, and informing the system that it needs to allocate larger memory to meet the task. needs. Similarly, a risk threshold can be set for the utilization rate of CPU, the utilization rate of input and output devices or the utilization rate of network interface. When the risk threshold is reached, the system generates an alarm signal, and the user can automatically or manually reset the risk resources. distribute.

Optionally, after analyzing the resource application situation and the actual resource usage situation of the container according to the monitoring result, it includes: when the usage rate of the CPU, the usage rate of the memory, the usage rate of the input and output devices, or the usage of the network interface. When the frequency is lower than the second threshold, a second alarm signal is sent; the scheduling system reallocates the resource information for generating the second alarm signal according to the second alarm signal. Similarly, as an example, for example, when the memory request index is 100M, the actual situation is usually 10M, which is far less than the usage threshold of 50M (50%), and the system will generate an alarm signal, indicating that the resource usage is insufficient at this time. The system needs to allocate smaller memory to meet the needs of the task. Similarly, a threshold can be set for the utilization rate of CPU, the utilization rate of input and output devices or the utilization rate of network interface. When the threshold is not reached, the system generates an alarm signal, and the user can automatically or manually reset the risk resources. Allocated to facilitate the release of resources and improve the overall resource utilization of the system.

As an implementation manner, the method further includes: recording the resource information generating the first alarm signal and/or the second alarm signal into the historical resource information and updating the historical resource information; according to the updated history The resource information and the job label allocates various resource information of the estimated size.

The system takes the resource allocation situation that generates the alarm as a record and inputs it to the training model for training. When such training records reach a certain number, more reasonable resource information will be allocated for such tasks to be generated again, so that the overall operation efficiency of the system will be improved. reach the highest.

The resource allocation method provided by the present disclosure dynamically understands whether the allocation of system resources is reasonable by monitoring the operation status of system resource information and the status of allocated resources in real time, and adjusts the allocation status in a timely manner when it is unreasonable. Resource tasks can optimize task resource utilization, thereby further optimizing cluster resource utilization. For tasks whose requested resources are less than the actual resources used, it can improve the running speed of user tasks and reduce the probability of user job failure due to OOM. For tasks that run periodically, the present invention can recommend and adjust the resources of the tasks according to the historical running conditions of the tasks, so that the user's resource application is more reasonable, thereby ensuring that the resource usage of the entire cluster is reasonable and healthy.

In addition, the present disclosure also provides device embodiments inherited from the above-mentioned embodiments, which are used to implement the method steps described in the above-mentioned embodiments. The explanations based on the same names and meanings are the same as those of the above-mentioned embodiments, and have the same technology as the above-mentioned embodiments. The effect will not be repeated here.

Specifically, as shown in FIG. 5 , according to a specific implementation manner of the present disclosure, the present disclosure provides a resource configuration device, including:

The obtaining unit 502 is configured to obtain a job task submitted by a user, wherein the job task includes a job label for marking the job type and a resource declaration, and the resource declaration includes at least one resource information required to complete the job task ;

A query unit 504, configured to query historical resource information of the job task based on the job label; the historical resource information is obtained by performing real-time monitoring and analysis on completed job tasks with the same job label;

a computing unit 506, configured to calculate the predicted resource demand of the job task based on the historical resource information of the job task;

An allocation unit 508, configured to allocate at least one resource based on the predicted resource demand.

Optionally, it also includes a processing unit (not shown) for real-time monitoring of the running status of the resources under each job label of the system and the relationship between the at least one resource that has been allocated and the job task, and dynamically adjusting the allocated value. at least one resource.

As one embodiment, as shown in FIG. 2 , the processing unit is further configured to obtain application meta information through a resource management module of the scheduling system, where the application meta information includes an application ID and an application name when the task is running , queue name, cluster name and application label; obtain the resource information of the container through the node management module of the scheduling system; collect the application meta-information and the resource information of the container in the same node; real-time monitoring is stored under the same node The application meta information and the resource information of the container are analyzed, and the resource application situation and the actual resource usage situation of the container are analyzed.

As one embodiment, as shown in FIG. 4 , the obtaining unit 502 is further configured to obtain resource information of a container through a node management module of the scheduling system, where the resource information of the container includes a process type and a disk type; The resource information of each sub-process in the process type is obtained through the process tree module, and the resource information includes CPU information, memory information, input and output information or network interface information.

As an embodiment, the processing unit is further configured to monitor the CPU usage, memory usage, input/output device usage, or network interface usage frequency under the current task in real time, and analyze the container according to the monitoring result. The application of resources and the actual use of resources.

As an implementation manner, the processing unit is further configured to issue a first alarm signal when the usage rate of the CPU, the usage rate of the memory, the usage rate of the input/output device, or the frequency of the network interface exceeds a first threshold value ; The scheduling system reallocates the resource information for generating the first alarm signal according to the first alarm signal.

As an implementation manner, the processing unit is further configured to issue a second alarm when the usage rate of the CPU, the usage rate of the memory, the usage rate of the input/output device, or the frequency of the network interface is lower than a second threshold signal; the scheduling system reallocates resource information for generating the second alarm signal according to the second alarm signal.

As an embodiment, the processing unit is further configured to record the resource information that generates the first alarm signal and/or the second alarm signal into the historical resource information and update the historical resource information.

The resource configuration device provided by the present disclosure dynamically understands whether the configuration of system resources is reasonable by monitoring the operating status of system resource information and the status of allocated resources in real time, and adjusts the allocation status in a timely manner when it is unreasonable. Resource tasks can optimize task resource utilization, thereby further optimizing cluster resource utilization. For tasks whose requested resources are less than the actual resources used, it can improve the running speed of user tasks and reduce the probability of user job failure due to OOM. For tasks that run periodically, the present invention can recommend and adjust the resources of the tasks according to the historical running conditions of the tasks, so that the user's resource application is more reasonable, thereby ensuring that the resource usage of the entire cluster is reasonable and healthy.

As shown in FIG. 6 , this embodiment provides an electronic device, the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores data that can be accessed by Instructions executed by the one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method steps described in the above embodiments.

Embodiments of the present disclosure provide a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, where the computer-executable instructions can execute the method steps described in the above embodiments.

Referring next to FIG. 6 , it shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. Terminal devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (eg, mobile terminals such as in-vehicle navigation terminals), etc., and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 6 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 6 , the electronic device may include a processing device (eg, a central processing unit, a graphics processor, etc.) 601 that may be loaded into a random access memory according to a program stored in a read only memory (ROM) 602 or from a storage device 608 The program in the (RAM) 603 executes various appropriate operations and processes. In the RAM 603, various programs and data necessary for the operation of the electronic device are also stored. The processing device 601 , the ROM 602 and the RAM 603 are connected to each other through a bus 605 . An input/output (I/O) interface 605 is also connected to the bus 605 .

Typically, the following devices can be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 605 of a computer, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 605. Communication means 605 may allow the electronic device to communicate wirelessly or by wire with other devices to exchange data. While FIG. 6 illustrates an electronic device having various means, it should be understood that not all of the illustrated means are required to be implemented or available. More or fewer devices may alternatively be implemented or provided.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 605 , or from the storage device 608 , or from the ROM 602 . When the computer program is executed by the processing apparatus 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. Among them, the name of the unit does not constitute a limitation of the unit itself under certain circumstances.

Claims (10)

1. a resource allocation method, is characterized in that, comprises: Acquiring a job task submitted by a user, wherein the job task includes a job label for marking a job type and a resource declaration, and the resource declaration includes at least one resource information required to complete the job task; Query the historical resource information of the job task based on the job label; the historical resource information is obtained by performing real-time monitoring and analysis on completed job tasks with the same job label; Calculate the predicted resource demand of the job task based on the historical resource information of the job task; At least one resource is allocated based on the predicted resource requirements. 2. The method of claim 1, further comprising: Real-time monitoring of the resource running status under each job label of the system and the relationship between the at least one resource allocated and the job task, and the dynamic adjustment of the allocated at least one resource. 3 . The method according to claim 2 , wherein the real-time monitoring system monitors the running status of resources under each job label and the relationship between the at least one resource that has been allocated and the job task, and dynamically adjusts the allocation. 4 . at least one resource, including: Obtain application meta information through the resource management module of the scheduling system, where the application meta information includes an application ID, an application name, a queue name, a cluster name, and an application label when the task is running; Obtain resource information of the container through the node management module of the scheduling system; associating the application meta information with the corresponding resource information of the container; The associated application meta information and the resource information of the container are monitored in real time, and the resource application situation and the actual resource usage situation of the container are analyzed. 4. The method according to claim 3, wherein the obtaining the resource information of the container through the node management module of the scheduling system comprises: Obtain resource information of the container through the node management module of the scheduling system, where the resource information of the container includes a process type and a disk type; The resource information of each sub-process in the process type is obtained through the process tree module, and the resource information includes CPU information, memory information, input and output information or network interface information. 5. The method according to claim 4, wherein the real-time monitoring of the associated application meta-information and the resource information of the container, and analyzing the resource application situation and actual resource usage of the container, comprises: Monitor the CPU usage, memory usage, input/output device usage, or network interface usage frequency in real time under the current task, and analyze the resource application and actual resource usage of the container according to the monitoring results. 6. The method according to claim 5, wherein after analyzing the resource application situation and the actual resource usage situation of the container according to the monitoring result, the method comprises: When the usage rate of the CPU, the usage rate of the memory, the usage rate of the input and output devices or the frequency of the network interface exceeds the first threshold, a first alarm signal is issued; The scheduling system reallocates resource information for generating the first alarm signal according to the first alarm signal. 7. The method according to claim 5, wherein after analyzing the resource application situation and the actual resource usage situation of the container according to the monitoring result, the method comprises: When the utilization rate of the CPU, the utilization rate of the memory, the utilization rate of the input and output devices or the frequency of the network interface is lower than the second threshold, a second alarm signal is issued; The scheduling system reallocates resource information for generating the second alarm signal according to the second alarm signal. 8. A resource configuration device, comprising: an obtaining unit, configured to obtain a job task submitted by a user, wherein the job task includes a job label for marking a job type and a resource declaration, and the resource declaration includes at least one resource information required to complete the job task; a query unit, configured to query historical resource information of the job task based on the job label; the historical resource information is obtained by performing real-time monitoring and analysis on completed job tasks with the same job label; a computing unit, configured to calculate the predicted resource demand of the job task based on the historical resource information of the job task; an allocation unit, configured to allocate at least one resource based on the predicted resource demand. 9. A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented. 10. An electronic device, comprising: one or more processors; A storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement any one of claims 1 to 7 one of the methods described.

Images