WO2022109932A1 - Multi-task submission system based on slurm computing platform - Google Patents

Abstract

A multi-task submission system based on the Slurm computing platform. The system comprises a running environment deployment module, a computing task scheduling plug-in module, a computing task running data storage management module, a unified API module and a task data unified management module. The multi-task submission system has a simple and convenient task submission check API, which can be easily accessed as a plug-in for serving task submission and monitoring. An NAS is used for managing a task running environment and task running data, such that a data management operation of a user can be simplified, data preparation and check can be efficiently completed, and the NAS can also be easily switched to an sge cluster or a cloud computing scheduler and an auxiliary storage service.

Description

Multi-task submission system based on slurm computing platform technical field

The invention belongs to the technical field of computer servers, in particular to a method for designing a free energy perturbation network based on machine learning.

Background technique

Slurm is a task scheduling tool for Linux and Unix kernel systems. It provides three key functions. First, users are allocated exclusive or non-exclusive resources (computer nodes) for a certain period of time for users to perform work. Second, it provides a framework for starting, executing, and monitoring tasks (usually parallel tasks, such as MPI) running on nodes, and third, assigning resources to task queues appropriately.

Python is an object-oriented, dynamic programming language. Has a very concise and clear syntax, suitable for various high-level tasks. It can be used to develop program scripts quickly, as well as large-scale software. Use Python for calculation script development, which is convenient and efficient.

Conda is an open source, cross-platform, language-agnostic package management and environment management system. Released under the BSD license by Continuum Analytics. Conda allows users to easily install different versions of binary packages and all the libraries needed for the computing platform. It also allows users to switch between different versions of packages, download packages from a software repository, and install them. Conda is developed in Python, but can manage projects in other programming languages (such as R), including multilingual projects. Conda installs packages in the Python language, similar to other Python-based cross-platform package managers (such as wheel or pip).

The running environment of tasks submitted by Slurm is determined by the environment of the computer node itself, while the actual running environment of tasks submitted by users often includes special libraries and software. Users usually only submit a bash script to Slurm, and define the actual python script in the script and specify the interpreter or script in other languages such as R and the specified interpreter.

This method itself is enough to meet the needs of users to use Slurm for massively parallel computing, but this method is not efficient. Mainly reflected in the following aspects:

1. Computational logic that is actually valuable to users is often defined in non-bash scripts (such as Python or R), but in order to run large-scale computing tasks in a Slurm cluster, it takes extra time to write support to run in Slurm bash script and corresponding script debugging, it will consume a certain amount of energy and time in the long run.

2. In the process of using slurm directly, users need to manage the data (files) such as input, output and errors of tasks by themselves, and increase the preparation work for using slurm.

Slurm runs tasks in an independent computing node. Users need to manually prepare the computing environment for each task and actively specify the computing environment when each task is delivered.

SUMMARY OF THE INVENTION

In view of the above-mentioned technical problems, the purpose of the present invention is to provide a multi-task submission system based on the slurm computing platform, which uses a server to perform task submission and management, and integrates data preparation and environment preparation required by different running tasks on the server, In this way, users can quickly complete the operation of submitting computing tasks to the slurm cluster using the same interface and different parameters.

To achieve the above object, the present invention provides the following technical solutions:

The multi-task submission system based on the slurm computing platform includes a running environment deployment module, a computing task scheduling plug-in module, a computing task running data storage management module, a unified API interface module, and a unified task data management module.

Specifically, in the operating environment deployment module, for the calculation script written based on python, the user uses Conda to deploy the environment. Prepare a path on the NAS to install Conda, create an env and install all dependencies. Finally provide the python interpreter under this env.

For scripts written in R language or other programming languages, use the corresponding package management tool to prepare the task execution environment and provide the interpreter path.

In summary, the environment needs to be deployed on the NAS, and the interpreter path needs to be accessible to the slurm cluster for calls.

For the computing task scheduling plug-in module, the common commands that users use slurm include: sbatch, squeue, sacct, scontrol, scancel, etc., which are respectively used to submit tasks, view task queues, view task running status, view tasks and rerun tasks, Cancel tasks, etc.

The computing task scheduling plug-in module mainly encapsulates these commands and forms high-level interfaces: submit, get_job_status, batch_get_jobs_status, get_job_detail, rerun_job, kill_job, dump_job and other interfaces.

For the computing task operation data storage management module, the user only needs to provide a task script (only a single file is supported) and an input data defined in json format; use the dump_file interface to copy the script to the task running path; use dump_json to convert the json format The data is written to the corresponding inputs.json file. The script needs to reload the inputs.json file to get the task specified input during the run. The output and error information of the task running, including the output of the script and the output defined by slurm, are placed in the task running path for viewing during and after the task is running.

The unified API interface module uniformly encapsulates the scheduling and storage management APIs of computing tasks, reduces the complexity of user understanding, and facilitates users to quickly submit tasks using the API interface.

In the unified management module for task data, all inputs and outputs are defined in the Log directory allocated by the NAS disk, and an independent ID is created for each task. Based on this ID, users can easily access all data of each task using the API .

Compared with the prior art, the beneficial effects of the present invention are:

1. Simple task submission and viewing API interface, which can be easily connected to the task submission and monitoring service as a plug-in.

2. Use NAS for task operation environment and task operation data management, simplify user data management operations, and efficiently complete data preparation collection and viewing.

3. The docking NAS+slurm is the solidified usage scheme of the plug-in. The user can easily transfer to the sge cluster or cloud computing scheduler and the complementary storage service based on the implementation logic of the present invention.

Description of drawings

Fig. 1 is a plug-in architecture design structural diagram of the present invention;

FIG. 2 is a flow chart of task management of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

The plug-in architecture design of the present invention is shown in FIG. 1 , and the specific task management process is shown in FIG. 2 .

1. Initialize the environment

1. First, create an exclusive permission directory on the NAS cluster, such as "/mydir/", all delivered task data, including input and output, algorithm data, task running environment and task running logs, etc., will be placed in this directory. Down.

(1) Create a conda path in the directory for installing the conda environment

(2) Create a scripts path in the directory for placing algorithm scripts

(3) Create a log path in the directory, similar to a sandbox environment, for placing computing runtime data (including logs)

2. In the created directory, create a conda environment, and install it by referring to the installation method provided by the official documentation.

3. Enter the conda environment, and according to the different dependencies of different algorithms on the environment, evaluate whether to distinguish the environment, and use the "conda create" command in conda to create the corresponding algorithm environment. Create method reference documentation.

Record the running environment, and the configuration information will be delivered as a task later.

2. Write a task delivery script

1. Write a shell script for the delivery task to perform the sbatch operation. The fixed variables include:

(1) The file name of the algorithm to be executed (.py file), such as "loader.py"

(2) The input file name (json format file) that the algorithm needs to read, such as "inputs.json"

(3) Other necessary environment variable information

Note: Uniform environment variables can be passed separately from input, but they need to be used with caution. Because improper handling of environment variables will affect the default environment.

Third, encapsulate the slurm command

1. Encapsulate the sbatch command into the submit interface, encapsulate the sbatch parameters, including cpu, memory and other information into the parameters of the submit interface, and separate a parameter of inputs={}, which is used to convert the input information to json (in python as dict) format. The interface needs to do the following things:

(1) Create a directory "/log/{id}" based on the task id identifier in the log path

(2) Copy the specified algorithm to "/log/{id}"

(3) Dump the inputs data to the input file

2. Encapsulate the squeue and sacct commands into the get_job_status interface and the batch_get_jobs_status interface. Squeue is to view the tasks in the queue, and scct can view the task status of the task that starts to run, including the status of success or failure after the running is completed. batch_get_jobs_status is used to view the status of multiple jobs in batches.

3. Encapsulate "scontrol show" into the get_job_detail command. For tasks that have entered or completed, sacct can obtain task running details. According to the user's needs, the task details are organized and provided to the user as the return information of get_job_detail.

4. Encapsulate the "scontrolrequeue" command into the rerun_job interface. Note that tasks that support rerun need to add the "--requeue" parameter when sbatch submits tasks.

5. Encapsulate the scancel command into a kill_job interface to cancel the task.

Independently encapsulate the dump_jobs interface, and provide user-accessible output (such as result data, operation logs, etc.) data in the "log/{id}" directory for download.

Example 2 Application scenarios and effects:

For engineers who use slurm clusters for computing task delivery, managing cluster data is tedious. In actual scenarios, a complete back-end service application is encapsulated based on this plug-in. In the application, the present invention is used for task delivery and task input and output management, and the algorithm service HTTP API is exposed externally. For business engineers, they can use client applications connected to back-end services to submit algorithm tasks anytime, anywhere, without the need to manage the operation of tasks in the slurm cluster, nor to manage the input and output storage of task calculations, just wait The result of the task execution can be obtained. The client can call the back-end API to obtain the result of the slurm cluster task operation, that is, you can know the success and failure of the task, and download the result of the successful task to the local, and the error task can also easily view the log. All of this is encapsulated into higher-level usage that is closer to the algorithm business using the client, while the back-end service undertakes the actual access to the slurm task and the NAS storage. The present invention provides this conversion service in the process.

After the product is launched, users can clearly manage their specific computing tasks, allocate CPU and memory for their tasks, and no longer pay attention to the scheduling situation of the slurm cluster and the storage method of computing results. Some users can do it in Without knowing the slurm cluster, you can still complete your own computing task delivery and computing result recovery.

Claims (2)

The multi-task submission system based on the slurm computing platform is characterized in that it includes a running environment deployment module, a computing task scheduling plug-in module, a computing task running data storage management module, a unified API interface module, and a unified task data management module. The multi-task submission system based on slurm computing platform according to claim 1, is characterized in that: For the operating environment deployment module, for computing scripts written based on python, users use Conda to deploy the environment; prepare a path on the NAS to install Conda, then create an env and install all dependent packages; finally, provide a python explanation under the env device; For scripts written in R language or other programming languages, use the corresponding package management tool to prepare the task running environment and provide the interpreter path; In short, the environment needs to be deployed on the NAS, and the interpreter path needs to be accessible to the slurm cluster; For the computing task scheduling plug-in module, the common commands that users use slurm include: sbatch, squeue, sacct, scontrol, scancel, which are respectively used to submit tasks, view task queues, view task running status, view tasks and rerun tasks, cancel tasks Task; The described computing task scheduling plug-in module mainly encapsulates these commands and forms high-level interfaces: submit, get_job_status, batch_get_jobs_status, get_job_detail, rerun_job, kill_job, dump_job interfaces; For the computing task operation data storage management module, the user only needs to provide a task script and an input data defined in json format; use the dump_file interface to copy the script to the task running path; use dump_json to write the json format data to the corresponding inputs .json file; the script needs to reload the inputs.json file during the running process to obtain the input specified by the task; the output and error information of the task operation, including the output of the script and the output defined by slurm, are placed in the task running path, so that the View the task running process and after the end; The unified API interface module uniformly encapsulates the scheduling and storage management API of computing tasks, reduces the complexity of user understanding, and facilitates users to quickly submit tasks using the API interface; In the unified management module for task data, all inputs and outputs are defined in the Log directory allocated by the NAS disk, and an independent ID is created for each task. Based on this ID, users can easily access all data of each task using the API .

Images