CN116192885A - High-availability cluster architecture artificial intelligence experiment cloud platform data processing method and system - Google Patents

Abstract

The application relates to a cloud platform technology and provides a high-availability cluster architecture artificial intelligent experimental cloud platform data processing method and system, wherein an artificial intelligent cloud platform comprises a plurality of master nodes and a plurality of slave nodes, and if a target slave node receives an experimental work task deployment instruction sent by the target master node, a target container is correspondingly created according to the experimental work task deployment instruction; if the target slave node receives the access request of the user terminal and passes the verification, connecting a target container instance corresponding to the target container with the user terminal; the target slave node receives target operation data of the user terminal, and stores the target operation data into a key value database corresponding to the target container; and if the target slave node receives the container operation instruction, correspondingly creating or deleting the container according to the container operation instruction. The cloud platform based artificial intelligent related experimental task processing method and system can process artificial intelligent related experimental tasks based on the cloud in the cloud platform, nodes can be added or subtracted from the cluster at any time, and the high availability and the load capacity of the cluster are improved.

Description

High-availability cluster architecture artificial intelligence experiment cloud platform data processing method and system

technical field

The present application relates to the field of cloud platform technology, and in particular to a data processing method and system for an artificial intelligence experiment cloud platform with a highly available cluster architecture.

Background technique

At present, when enterprises or universities conduct artificial intelligence-related experiments, some solutions using experimental platform clusters appear, that is, to put artificial intelligence-related experimental data on the cluster of the cloud platform for cloud experimental tasks. However, in the current cloud platform clusters, it is often impossible to add or delete nodes to the cluster at any time, which leads to a limited number of operators for artificial intelligence-related experiments, and cannot handle cloud experiment tasks with multi-scale personnel participation. Moreover, in the case of abnormal failures such as power outages in the cluster of the existing cloud platform, the experimental data cannot be automatically saved, and the data has a large security risk.

Contents of the invention

The embodiment of the present application provides a data processing method and system of a highly available cluster architecture artificial intelligence experiment cloud platform, aiming to solve the problem that the cloud platform cluster used for artificial intelligence related experiments in the prior art often cannot add or delete the cluster at any time This leads to the limited number of operators faced by artificial intelligence-related experiments, and only a small number of artificial intelligence-related experiments can be carried out.

In the first aspect, the embodiment of the present application provides a data processing method of a high-availability cluster architecture artificial intelligence experiment cloud platform, which is applied to the artificial intelligence experiment cloud platform, and the artificial intelligence cloud platform includes a plurality of master nodes and a plurality of slave nodes, The plurality of master nodes and the plurality of slave nodes are all communicatively connected; the method includes:

If the target slave node receives the experimental work task deployment instruction sent by the target master node, it will create a target container correspondingly according to the experimental work task deployment instruction; wherein, the target slave node is any one of the multiple slave nodes, The target master node is a currently active master node among the plurality of master nodes;

If the target slave node receives the access request of the user terminal and passes the verification, connects the target container instance corresponding to the target container with the user terminal;

The target slave node receives the target operation data of the user terminal, and stores the target operation data in a key-value database corresponding to the target container;

The target master node sends container operation instructions to the target slave node;

If the target slave node receives the container operation instruction, it creates or deletes a container correspondingly according to the container operation instruction.

In the second aspect, the embodiment of the present application provides a high-availability cluster architecture artificial intelligence experiment cloud platform data processing system, running on the artificial intelligence experiment cloud platform, which includes a plurality of master nodes and a plurality of slave nodes, the plurality of master nodes The node and the plurality of slave nodes are all communicatively connected; wherein, the target slave node is any one of the plurality of slave nodes, and the target master node is currently active among the plurality of master nodes master node;

The target slave node is configured to create a target container according to the experimental task deployment instruction corresponding to the experimental work task deployment instruction sent by the target master node; wherein the target slave node is any one of the plurality of slave nodes From the node, the target master node is a currently active master node among the plurality of master nodes;

The target slave node is further configured to connect the target container instance corresponding to the target container to the user terminal if the access request from the user terminal is received and passed the verification;

The target slave node is further configured to receive target operation data of the user terminal, and store the target operation data in a key-value database corresponding to the target container;

The target master node is used to send container operation instructions to the target slave node;

The target slave node is further configured to correspondingly create or delete a container according to the container operation instruction if receiving the container operation instruction sent by the target master node.

In the third aspect, the embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the computer program, the above-mentioned first aspect is realized. High-availability cluster architecture artificial intelligence experimental cloud platform data processing method.

In the fourth aspect, the embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the processor executes the high-availability cluster of the first aspect above Architect the artificial intelligence experimental cloud platform data processing method.

The embodiment of the present application provides a data processing method and system of a high-availability cluster architecture artificial intelligence experiment cloud platform. The artificial intelligence cloud platform includes multiple master nodes and multiple slave nodes, and the multiple master nodes and multiple slave nodes are all connected by communication. The method includes: if the target slave node receives the experimental work task deployment instruction sent by the target master node, then correspondingly create the target container according to the experimental work task deployment instruction; if the target slave node receives the access request of the user terminal and passes the verification, it will The target container instance corresponding to the target container is connected to the user terminal; the target slave node receives the target operation data of the user terminal, and stores the target operation data in the key-value database corresponding to the target container; the target master node sends the container operation command to the target slave node; If the slave node receives the container operation instruction, it will create or delete the container correspondingly according to the container operation instruction. In the artificial intelligence experiment cloud platform, the processing of artificial intelligence related experimental tasks can be carried out based on the cloud, and nodes can be added or deleted to the cluster at any time to improve the high availability and load capacity of the cluster.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application. Ordinary technicians can also obtain other drawings based on these drawings on the premise of not paying creative work.

Fig. 1 is a schematic diagram of an application scenario of a data processing method of a highly available cluster architecture artificial intelligence experiment cloud platform provided by an embodiment of the present application;

Fig. 2 is the schematic flow chart of the data processing method of the highly available cluster architecture artificial intelligence experiment cloud platform provided by the embodiment of the present application;

Fig. 3 is a schematic block diagram of a data processing system of a highly available cluster architecture artificial intelligence experiment cloud platform provided by an embodiment of the present application;

Fig. 4 is a schematic block diagram of a computer device provided by an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

It should be understood that when used in this specification and the appended claims, the terms "comprising" and "comprises" indicate the presence of described features, integers, steps, operations, elements and/or components, but do not exclude one or Presence or addition of multiple other features, integers, steps, operations, elements, components and/or collections thereof.

It should also be understood that the terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural referents unless the context clearly dictates otherwise.

It should be further understood that the term "and/or" used in the description of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations .

Please refer to Figure 1 and Figure 2, Figure 1 is a schematic diagram of the application scenario of the data processing method of the high-availability cluster architecture artificial intelligence experiment cloud platform provided by the embodiment of the application; Figure 2 is the high-availability cluster architecture artificial intelligence experiment provided by the embodiment of the application Schematic flow chart of the cloud platform data processing method. The high-availability cluster architecture artificial intelligence experiment cloud platform data processing method provided by the embodiment of the present application is applied to the artificial intelligence experiment cloud platform. As shown in Figure 1, the artificial intelligence experiment cloud platform includes multiple master nodes and multiple slave nodes. The plurality of master nodes and the plurality of slave nodes are all communicatively connected. The artificial intelligence experiment cloud platform can be regarded as a Kubernetes cluster including multiple master nodes and multiple slave nodes. It is a distributed system that can manage the orchestration and scheduling of a single container cluster resource.

Wherein, each of the plurality of master nodes includes an APIServer module (which can be understood as an interface module), a Scheduler module (which can be understood as a scheduling module), and a Controller-Manager module (which can be understood as a management control module) and a key-value Database (can be expressed as Etcd database). The APIServer module is used to notify the slave nodes to perform operations such as setting up, deleting and stopping cluster resources according to the decision of the master node; Pod scheduling according to the consumption situation (Pod is the smallest unit that can be created and managed in the Kubernetes system (that is, the K8S system), and is the smallest resource object model created or deployed by the user in the resource object model); the Controller-Manager module uses To detect whether the state of each master node and each slave node in the cluster corresponding to the artificial intelligence experiment cloud platform is healthy; the key-value database is used to store various important configuration information in the cluster corresponding to the artificial intelligence experiment cloud platform, and Persist various data resources in the cluster. Among the multiple master nodes, there is only one running and active master node at each moment and can be recorded as Leader-Master-Node (only Leader-Master-Node can provide external services), and other master nodes are in Inactive standby state. If the working master node (Leader-Master-Node) is abnormal, the cluster corresponding to the artificial intelligence experiment cloud platform will automatically select a master node from multiple master nodes in the standby state to immediately replace the master node in the abnormal state as the new master node. The running and active master node continues the current work.

Each of the plurality of slave nodes can be regarded as a working node in the cluster corresponding to the artificial intelligence experiment cloud platform, a node that actually executes artificial intelligence experiment tasks, and a running container responsible for running actual business and resources. In addition to providing the operating environment of the Pod, each slave node also has an infrastructure for management and communication. Specifically, each slave node communicates with each of the multiple master nodes through the Kubelet component (a proxy component on the slave node) Nodes interact with each other. The Kubelet component regularly receives work tasks from the API-Server module of the master node to process the entire life cycle related matters of the Pod on the master node; and Kubelet also regularly reports all the tasks to the master node through the API-Server module of the master node work info. Network proxy access is performed between different slave nodes through the Kube-proxy component (which is the network proxy component on the slave node of the Kubernetes cluster).

As shown in FIG. 2 , the data processing method of the high-availability cluster architecture artificial intelligence experiment cloud platform includes steps S101-S105.

S101. If the target slave node receives the experimental work task deployment instruction sent by the target master node, it will create a target container according to the experimental work task deployment instruction; wherein, the slave node is any one of the multiple slave nodes , the target master node is a currently active master node among the plurality of master nodes.

In this embodiment, when a cluster corresponding to the artificial intelligence experiment cloud platform is composed of multiple master nodes and multiple slave nodes, the artificial intelligence experiment cloud platform can be used as a cloud platform for artificial intelligence experiments. Specifically, the user of the target master node (for example, a platform administrator who logs in to the master node of the artificial intelligence experiment cloud platform through an administrator privilege user account) operates on the user interface to deploy experimental tasks on the slave nodes, which will trigger the deployment of experimental work tasks instruction. The experiment task deployment instruction generated in the target master node is sent to each slave node, and the target container is correspondingly created in the target slave node in each slave node based on the experiment task deployment instruction. Wherein, the target master node is the currently active master node among the multiple master nodes, so as to ensure that only one master node in the cluster is currently working and performing various data processing.

After the target container is created in the target slave node based on the experimental work task deployment instruction, it is necessary to add a target image corresponding to the experimental work task deployment instruction in each target container to obtain each target container instance. After the target container instance is obtained, each target container instance can correspond to the corresponding user terminal corresponding to the experiment participants, so that each experiment participant can use the user terminal to connect to the corresponding target container instance to process the artificial intelligence experiment task. Among them, data such as the container operating environment and model code have been deployed in each target container instance. The container engine of the target container instance can provide the container operating environment and make images for different requirements; the private image warehouse integrates framework images such as TensorFlow, Caffe, and PyTorch required for experiments in the field of artificial intelligence, and also supports deep neural network DNN, convolution Neural network CNN and target detection related YoLoV1~V5 models, etc.

It can be seen that the target container is created in the slave node instead of the master node, which ensures that the slave node is used as the cloud device that actually runs the container in the artificial intelligence experiment cloud platform, and the master node is used as a unified monitoring and management slave node cloud device. If a slave node fails, the artificial intelligence experimental cloud platform adopts a high-availability cluster and high-availability deployment of applications to reduce the damage caused by node failure and ensure the high reliability of the cloud platform.

In one embodiment, before step S101, it also includes:

The interface module in the target master node establishes a communication connection with the Kubelet agent component of the target slave node.

In this embodiment, when constructing a high-availability cluster architecture artificial intelligence experiment cloud platform, it is necessary to communicate and connect multiple master nodes and multiple slave nodes. Specifically, each slave node establishes a communication connection with the interface module in the target master node based on the Kubelet proxy component, so that the target slave node as one of the slave nodes also establishes a communication connection with the interface module in the target master node based on the Kubelet proxy component. Among them, the Kubelet proxy component can be vividly understood as the link for data interaction between the target master node and each slave node. The Kubelet component regularly receives work tasks from the API-Server module of the master node to process the entire life cycle related matters of the Pod on the master node; and Kubelet also regularly reports all the tasks to the master node through the API-Server module of the master node work info. Moreover, each slave node included in the artificial intelligence experiment cloud platform can access the Internet based on the Kube-proxy component, or communicate with the user terminal based on the Internet.

In one embodiment, the interface module in the target master node establishes a communication connection with the Kubelet agent component of the target slave node, including:

The Keepalived component of the target master node automatically configures the virtual IP address of the artificial intelligence experiment cloud platform through the virtual routing redundancy protocol;

The interface module of the target master node establishes a communication connection with the Kubelet agent component block of the target slave node based on the virtual IP address.

In this embodiment, each master node has a Keepalived component and a Haproxy component, wherein the Keepalived component is used to automatically configure the virtual IP address of the artificial intelligence experiment cloud platform through the virtual routing redundancy protocol (ie VRRP protocol), so as to ensure that the artificial intelligence The experimental cloud platform has a unified virtual IP for external access. The Haproxy component is used to provide load balancing services for slave nodes.

After the Keepalived component of the target master node obtains the virtual IP address of the artificial intelligence experiment cloud platform, automatic configuration is performed so that the target master node has the same virtual IP as the artificial intelligence experiment cloud platform. Moreover, except that the interface module of the target master node establishes a communication connection with the Kubelet proxy component block of the target slave node based on the virtual IP address, the remaining other master nodes are also based on the interface module based on the virtual IP address when switching from the standby state to the active state. IP address to establish a communication connection with the Kubelet proxy component block of the target slave node. It can be seen that based on this architectural approach, high availability and high load of the system are ensured.

In one embodiment, step S101 includes:

If the experimental work task deployment instruction is a unified experimental task deployment instruction, then the target obtains the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction from the node, and the target obtains from the node The node correspondingly creates a target container according to the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction;

If the experimental work task deployment instruction is a personalized container deployment instruction, the target slave node obtains the second target image resource corresponding to the personalized container deployment instruction, and the target slave node deploys the personalized container according to the The second target image resource corresponding to the instruction and the pre-stored data storage volume path correspond to the creation target container.

In this embodiment, user accounts with at least three types of authority are pre-divided in the artificial intelligence experiment cloud platform, which are administrator authority user accounts, first authority user accounts (such as teacher authority user accounts) and second authority user accounts (such as a user account with student permissions). Among them, the user account with administrator authority has the authority to manage all data of the entire artificial intelligence experiment cloud platform, for example, creating multiple namespaces (namely namespaces) in multiple slave nodes of the artificial intelligence experiment cloud platform is one of the permissions; The user account with the first authority has the authority to create multiple containers in the corresponding namespace for the login and use of the user account with the second authority; the user account with the second authority only has the corresponding container to log in the corresponding namespace to process the artificial intelligence experiment task permission.

Among them, the user account with administrator authority can receive the list of user accounts to be created provided by the user corresponding to a user account with the first authority. After the node or slave node, the combination name is obtained from the combination of the teacher's name and the student's class name corresponding to the user's bill list. Afterwards, the administrator authority user account corresponding to the administrator creates a namespace corresponding to the above-mentioned combined name from the node in the artificial intelligence experiment cloud platform. In this way, it can be vividly understood that the administrator has created a class-specific namespace for the class taught by the teacher corresponding to the teacher's name in the artificial intelligence experiment cloud platform. Then, the administrator authority user account corresponding to the administrator can also correspond to the corresponding number of second authority user accounts (that is, the student included in the user bill list) according to the student name list (or student student number list) included in the user bill list. The total number of names is the same as the total number of second-authority user accounts).

Of course, when multiple containers are created in each namespace according to actual needs, the user account with administrator authority can correspond to the administrator to create the corresponding number according to the requirements of artificial intelligence experiment tasks and the list of student names included in the user bill list. It may also be a corresponding number of containers created by the teacher corresponding to the first authority user account according to the requirements of the artificial intelligence experiment task and the list of student names included in the user bill list. In the artificial intelligence experiment cloud platform, the relevant information of the namespace is stored in the master node, and the container corresponding to each namespace is deployed in the slave node of the artificial intelligence experiment cloud platform.

After the creation of the namespace of a certain class such as class A is completed in the slave node in the artificial intelligence experiment cloud platform, the teacher corresponding to the first authority user account can continue to generate a unified experiment task according to the requirements of the artificial intelligence experiment task. Deployment instructions, and specifically set the first target image resources, GPU resources and data storage volume paths in the unified experimental task deployment instructions. Afterwards, the resource container layer creates a container corresponding to the first target image resource, GPU resource, and data storage volume path corresponding to the unified experimental task deployment instruction.

Wherein, the first target image resource can be mirrored from frameworks such as TensorFlow, Caffe, and PyTorch that are required for experiments in the field of artificial intelligence, or integrated with deep neural network DNN, convolutional neural network CNN, and target detection-related YoLoV1～ Choose one or more of these artificial intelligence-related neural network model images for target detection networks such as V5 to deploy. More specifically, the first target image resource may select the TensorFlow framework and deploy a convolutional neural network (CNN) on the TensorFlow framework.

When the teacher corresponding to the first authority user account generates a unified experimental task deployment instruction according to the requirements of the artificial intelligence experimental task, and correspondingly completes the creation of multiple containers corresponding to the namespace in the resource container layer , the initial environment construction of the artificial intelligence experiment task is completed.

Of course, when the target slave node in the artificial intelligence experiment cloud platform has completed the creation of a certain class such as the namespace of class A, the second authority user account can also be used by the corresponding student to perform the artificial intelligence experiment task according to their own Personalized requirements generate a personalized container deployment instruction, and the personalized container deployment instruction is sent by the slave node of the artificial intelligence experiment cloud platform to the user terminal used by the teacher corresponding to the first authorized user account , when the teacher corresponding to the user account with the first authority operates and approves the personalized container deployment instruction on the user terminal, then the target slave node of the artificial intelligence experiment cloud platform according to the second corresponding to the personalized container deployment instruction The target image resource and the path of the pre-stored data storage volume correspond to the creation of the container. Similarly, the second target image resource can be mirrored from frameworks such as TensorFlow, Caffe, and PyTorch that are required for experiments in the field of artificial intelligence, or integrated with deep neural network DNN, convolutional neural network CNN, and YoLoV1 related to target detection. ~V5 and other target detection networks can choose one or more of these artificial intelligence-related neural network model images for deployment. More specifically, the second target image resource can select the TensorFlow framework and deploy the YoLoV5 target detection network on the TensorFlow framework.

Moreover, when the student with the second-authority user account generates personalized container deployment instructions according to their own individual needs for artificial intelligence experiment tasks, GPU resources are not allocated by default, that is, based on the personalized container deployment instructions on the target slave node The containers created in are common server containers, not GPU server containers. Of course, if the container requirements corresponding to the personalized container deployment instruction have requirements for the use of the GPU server container, then the personalized container deployment instruction is sent to the user terminal used by the teacher corresponding to the first authorized user account. After the teacher corresponding to the user account operates and approves the personalized container deployment instruction on the user terminal, the target slave node of the artificial intelligence experiment cloud platform according to the second target image resource corresponding to the personalized container deployment instruction, GPU resources and pre-stored data storage volume paths correspond to the creation of containers.

In one embodiment, the target acquires the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction from the node, including:

If the target slave node detects a unified experimental task deployment instruction, then obtain the teaching progress information and teacher teaching tag set corresponding to the unified experimental task deployment instruction, and according to the teaching progress information, teacher teaching tag set and preset resources The calling strategy generates the first target image resource, GPU resource and data storage volume path corresponding to the unified experiment task deployment instruction.

In this embodiment, when the target slave node detects a unified experimental task deployment instruction, specifically, the target slave node can analyze the unified experimental task deployment instruction to determine whether it includes teaching progress information (for example, learned Section 5 of Chapter 1 of the AA artificial intelligence course, etc.) and teacher teaching label sets (such as including labels such as face recognition and convolutional neural networks). If the unified experimental task deployment instruction is parsed and the teaching progress information and the teacher's teaching label set are obtained, then according to the teaching progress information, the teacher's teaching label set and the preset resource call strategy, the first target image resource required for creating the container is generated, Container information such as GPU resources and data storage volume paths. Wherein, the resource calling strategy can be understood as a preset mapping table including several pieces of teaching progress information, teacher teaching label set and target image resource, GPU resource and data storage volume path, with teaching progress information, teacher teaching label set as The search conditions can query the corresponding target image resources, GPU resources, and data storage volume paths.

S102. If the target slave node receives the access request from the user terminal and passes the verification, connect the target container instance corresponding to the target container to the user terminal.

In this embodiment, after the target slave node completes the deployment of the target container and the target container instance, it can be provided to the user for access to process artificial intelligence-related experimental tasks. When a user needs to access a target container in a slave node, an access request with user account information is first sent to the target slave node. Afterwards, when the target slave node passes the verification of the access request, a communication connection between the target container instance corresponding to the target container and the user terminal is established. In this way, the user terminal can access the target slave node to process the experimental tasks related to artificial intelligence.

S103. The target slave node receives the target operation data of the user terminal, and stores the target operation data in the key-value database corresponding to the target container.

In this embodiment, after the user terminal is connected to the target container instance in the target slave node, the target container instance can receive the target operation data of the user terminal. In order to improve the data security of the target operation data, the target operation data can be stored in a key-value database corresponding to the target container in the target master node (such as Etcd database, which is a key-value database), In this way, even when the target slave node fails and stops running, the operation data of each container included therein is stored in the key-value database of the target master node. When the target slave node returns to normal after troubleshooting, it also calls all the data of the target slave node from the key-value database of the target master node for breakpoint recovery.

In one embodiment, after step S103, it also includes:

If the target master node detects that the current working state is abnormal, a master node is randomly selected from the remaining multiple master nodes as the target master node.

In this embodiment, currently only one master node in a general cluster is working and performing various data processing, and other master nodes are in an inactive standby state. If the working target master node (that is, Leader-Master-Node) is abnormal, the cluster corresponding to the artificial intelligence experiment cloud platform will automatically select a master node from multiple master nodes in the standby state to immediately replace the master node in the abnormal state as the new master node. The running and active master node continues the current work. It can be seen that no matter which master node in the cluster fails, it will not affect the work of the entire cluster. If the working master node is abnormal, the cluster will automatically select a master node from the standby master nodes to replace the abnormal node as the new master immediately. The control node Leader-Master-Node continues the current work.

In one embodiment, after step S103, it also includes:

If the target slave node detects that the current working state is an abnormal state, it obtains the current node container data and stores the current node container data in the key-value database corresponding to the target slave node in the target master node.

In this embodiment, when the current working state of the target slave node is an abnormal state, before it is restarted for troubleshooting, the current node container data of the target slave node can be obtained again in the target master node and the The current node container data is stored in the key-value database corresponding to the target slave node in the target master node for data backup. Create a persistent volume statement (PVC) and a persistent volume (PV) for each user in the key-value database. When the user stops and closes the operation of the target container instance in the target slave node or exits the target container instance due to a failure, The current node container data generated by the user's operation on the target container instance before this shutdown operation or exit operation will be automatically stored in the key-value database corresponding to the target slave node in the target master node for data backup. When the user enters the target container instance again next time, the artificial intelligence experiment cloud platform automatically calls the current node container data from the key-value database to restore the target container instance to the state when it exited last time, so as to For the user to continue to operate on the target container instance after re-entering this time.

Since all node container data of each container is persisted into a dynamically created persistent volume, whether the user actively exits the container instance or passively exits the container instance due to a failure, the artificial intelligence experiment cloud platform can automatically save the historical node container The data is stored in the persistent volume for the user to call the historical node container data and continue to operate on the container instance after re-entering the container.

S104. The target master node sends a container operation instruction to the target slave node.

In this embodiment, of course, in addition to operating the artificial intelligence-related experimental data in the container of the target slave node, the user can also access the target master node (for example, a user account with administrator authority logs in to the target master node), and then perform The triggering of the container operation instruction corresponding to operations such as adding or deleting a container generates an operation. After completing the container operation instruction that triggers adding or deleting containers, the target master node sends the container operation instruction to the target slave node.

S105. If the target slave node receives the container operation instruction, correspondingly create or delete a container according to the container operation instruction.

In this embodiment, the artificial intelligence experiment cloud platform can realize the addition and deletion of cluster nodes at any time based on container operation instructions, so as to facilitate the expansion and contraction of Pod through the cluster controller. More specifically, when the resources in the target slave nodes are tight, add more slave nodes and create containers in the slave nodes to achieve capacity expansion.

The method realizes the processing of artificial intelligence-related experimental tasks based on the cloud in the artificial intelligence experiment cloud platform, and can add or delete nodes to the cluster at any time, thereby improving the high availability and load capacity of the cluster.

The embodiment of the present application also provides a high-availability cluster architecture artificial intelligence experiment cloud platform data processing system, the highly available cluster architecture artificial intelligence experiment cloud platform data processing system is used to execute the aforementioned highly available cluster architecture artificial intelligence experiment cloud platform data processing method any of the examples. Specifically, please refer to FIG. 3 . FIG. 3 is a schematic block diagram of a data processing system 100 for an artificial intelligence experiment cloud platform with a high-availability cluster architecture provided by an embodiment of the present application.

Wherein, as shown in FIG. 3 , the data processing system 100 of the high-availability cluster architecture artificial intelligence experiment cloud platform includes multiple master nodes 101 and multiple slave nodes 102 . Wherein, the target slave node is any slave node in the plurality of slave nodes 102 , and the target master node is a currently active master node in the plurality of master nodes 101 .

The target slave node is configured to create a target container correspondingly according to the experimental work task deployment command if it receives the experimental work task deployment instruction sent by the target master node; wherein, the slave node is any one of the multiple slave nodes node, where the target master node is a master node that is currently active among the plurality of master nodes.

In this embodiment, when a cluster corresponding to the artificial intelligence experiment cloud platform is composed of multiple master nodes and multiple slave nodes, the artificial intelligence experiment cloud platform can be used as a cloud platform for artificial intelligence experiments. Specifically, the user of the target master node (for example, a platform administrator who logs in to the master node of the artificial intelligence experiment cloud platform through an administrator privilege user account) operates on the user interface to deploy experimental tasks on the slave nodes, which will trigger the deployment of experimental work tasks instruction. The experiment task deployment instruction generated in the target master node is sent to each slave node, and the target container is correspondingly created in the target slave node in each slave node based on the experiment task deployment instruction. Wherein, the target master node is the currently active master node among the multiple master nodes, so as to ensure that only one master node in the cluster is currently working and performing various data processing.

After the target container is created in the target slave node based on the experimental work task deployment instruction, it is necessary to add a target image corresponding to the experimental work task deployment instruction in each target container to obtain each target container instance. After the target container instance is obtained, each target container instance can correspond to the corresponding user terminal corresponding to the experiment participants, so that each experiment participant can use the user terminal to connect to the corresponding target container instance to process the artificial intelligence experiment task. Among them, data such as the container operating environment and model code have been deployed in each target container instance. The container engine of the target container instance can provide the container operating environment and make images for different requirements; the private image warehouse integrates framework images such as TensorFlow, Caffe, and PyTorch required for experiments in the field of artificial intelligence, and also supports deep neural network DNN, convolution Neural network CNN and target detection related YoLoV1~V5 models, etc.

It can be seen that the target container is created in the slave node instead of the master node, which ensures that the slave node is used as the cloud device that actually runs the container in the artificial intelligence experiment cloud platform, and the master node is used as a unified monitoring and management slave node cloud device. If a slave node fails, the artificial intelligence experimental cloud platform adopts a high-availability cluster and high-availability deployment of applications to reduce the damage caused by node failure and ensure the high reliability of the cloud platform.

In an embodiment, the target master node is further configured to establish a communication connection with the Kubelet proxy component of the target slave node through an interface module in the target master node.

In this embodiment, when constructing a high-availability cluster architecture artificial intelligence experiment cloud platform, it is necessary to communicate and connect multiple master nodes and multiple slave nodes. Specifically, each slave node establishes a communication connection with the interface module in the target master node based on the Kubelet proxy component, so that the target slave node as one of the slave nodes also establishes a communication connection with the interface module in the target master node based on the Kubelet proxy component. Among them, the Kubelet proxy component can be vividly understood as the link for data interaction between the target master node and each slave node. The Kubelet component regularly receives work tasks from the API-Server module of the master node to process the entire life cycle related matters of the Pod on the master node; and Kubelet also regularly reports all the tasks to the master node through the API-Server module of the master node work info. Moreover, each slave node included in the artificial intelligence experiment cloud platform can access the Internet based on the Kube-proxy component, or communicate with the user terminal based on the Internet.

In one embodiment, the interface module in the target master node establishes a communication connection with the Kubelet agent component of the target slave node, including:

The Keepalived component of the target master node automatically configures the virtual IP address of the artificial intelligence experiment cloud platform through the virtual routing redundancy protocol;

The interface module of the target master node establishes a communication connection with the Kubelet agent component block of the target slave node based on the virtual IP address.

In this embodiment, each master node has a Keepalived component and a Haproxy component, wherein the Keepalived component is used to automatically configure the virtual IP address of the artificial intelligence experiment cloud platform through the virtual routing redundancy protocol (ie VRRP protocol), so as to ensure that the artificial intelligence The experimental cloud platform has a unified virtual IP for external access. The Haproxy component is used to provide load balancing services for slave nodes.

After the Keepalived component of the target master node obtains the virtual IP address of the artificial intelligence experiment cloud platform, automatic configuration is performed so that the target master node has the same virtual IP as the artificial intelligence experiment cloud platform. Moreover, except that the interface module of the target master node establishes a communication connection with the Kubelet proxy component block of the target slave node based on the virtual IP address, the remaining other master nodes are also based on the interface module based on the virtual IP address when switching from the standby state to the active state. IP address to establish a communication connection with the Kubelet proxy component block of the target slave node. It can be seen that based on this architectural approach, high availability and high load of the system are ensured.

In one embodiment, the target slave node is also used to:

If the experimental work task deployment instruction is a unified experimental task deployment instruction, then the target obtains the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction from the node, and the target obtains from the node The node correspondingly creates a target container according to the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction;

If the experimental work task deployment instruction is a personalized container deployment instruction, the target slave node obtains the second target image resource corresponding to the personalized container deployment instruction, and the target slave node deploys the personalized container according to the The second target image resource corresponding to the instruction and the pre-stored data storage volume path correspond to the creation target container.

In this embodiment, user accounts with at least three types of authority are pre-divided in the artificial intelligence experiment cloud platform, which are administrator authority user accounts, first authority user accounts (such as teacher authority user accounts) and second authority user accounts (such as a user account with student permissions). Among them, the user account with administrator authority has the authority to manage all data of the entire artificial intelligence experiment cloud platform, for example, creating multiple namespaces (namely namespaces) in multiple slave nodes of the artificial intelligence experiment cloud platform is one of the permissions; The user account with the first authority has the authority to create multiple containers in the corresponding namespace for the login and use of the user account with the second authority; the user account with the second authority only has the corresponding container to log in the corresponding namespace to process the artificial intelligence experiment task permission.

Among them, the user account with administrator authority can receive the list of user accounts to be created provided by the user corresponding to a user account with the first authority. After the node or slave node, the combination name is obtained from the combination of the teacher's name and the student's class name corresponding to the user's bill list. Afterwards, the administrator authority user account corresponding to the administrator creates a namespace corresponding to the above-mentioned combined name from the node in the artificial intelligence experiment cloud platform. In this way, it can be vividly understood that the administrator has created a class-specific namespace for the class taught by the teacher corresponding to the teacher's name in the artificial intelligence experiment cloud platform. Then, the administrator authority user account corresponding to the administrator can also correspond to the corresponding number of second authority user accounts (that is, the student included in the user bill list) according to the student name list (or student student number list) included in the user bill list. The total number of names is the same as the total number of second-authority user accounts).

Of course, when multiple containers are created in each namespace according to actual needs, the user account with administrator authority can correspond to the administrator to create the corresponding number according to the requirements of artificial intelligence experiment tasks and the list of student names included in the user bill list. It may also be a corresponding number of containers created by the teacher corresponding to the first authority user account according to the requirements of the artificial intelligence experiment task and the list of student names included in the user bill list. In the artificial intelligence experiment cloud platform, the relevant information of the namespace is stored in the master node, and the container corresponding to each namespace is deployed in the slave node of the artificial intelligence experiment cloud platform.

After the creation of the namespace of a certain class such as class A is completed in the slave node in the artificial intelligence experiment cloud platform, the teacher corresponding to the first authority user account can continue to generate a unified experiment task according to the requirements of the artificial intelligence experiment task. Deployment instructions, and specifically set the first target image resources, GPU resources and data storage volume paths in the unified experimental task deployment instructions. Afterwards, the resource container layer creates a container corresponding to the first target image resource, GPU resource, and data storage volume path corresponding to the unified experimental task deployment instruction.

Wherein, the first target image resource can be mirrored from frameworks such as TensorFlow, Caffe, and PyTorch that are required for experiments in the field of artificial intelligence, or integrated with deep neural network DNN, convolutional neural network CNN, and target detection-related YoLoV1～ Choose one or more of these artificial intelligence-related neural network model images for target detection networks such as V5 to deploy. More specifically, the first target image resource may select the TensorFlow framework and deploy a convolutional neural network (CNN) on the TensorFlow framework.

When the teacher corresponding to the first authority user account generates a unified experimental task deployment instruction according to the requirements of the artificial intelligence experimental task, and correspondingly completes the creation of multiple containers corresponding to the namespace in the resource container layer , the initial environment construction of the artificial intelligence experiment task is completed.

Of course, when the target slave node in the artificial intelligence experiment cloud platform has completed the creation of a certain class such as the namespace of class A, the second authority user account can also be used by the corresponding student to perform the artificial intelligence experiment task according to their own Personalized requirements generate a personalized container deployment instruction, and the personalized container deployment instruction is sent by the slave node of the artificial intelligence experiment cloud platform to the user terminal used by the teacher corresponding to the first authorized user account , when the teacher corresponding to the user account with the first authority operates and approves the personalized container deployment instruction on the user terminal, then the target slave node of the artificial intelligence experiment cloud platform according to the second corresponding to the personalized container deployment instruction The target image resource and the path of the pre-stored data storage volume correspond to the creation of the container. Similarly, the second target image resource can be mirrored from frameworks such as TensorFlow, Caffe, and PyTorch that are required for experiments in the field of artificial intelligence, or integrated with deep neural network DNN, convolutional neural network CNN, and YoLoV1 related to target detection. ~V5 and other target detection networks can choose one or more of these artificial intelligence-related neural network model images for deployment. More specifically, the second target image resource can select the TensorFlow framework and deploy the YoLoV5 target detection network on the TensorFlow framework.

Moreover, when the student with the second-authority user account generates personalized container deployment instructions according to their own individual needs for artificial intelligence experiment tasks, GPU resources are not allocated by default, that is, based on the personalized container deployment instructions on the target slave node The containers created in are common server containers, not GPU server containers. Of course, if the container requirements corresponding to the personalized container deployment instruction have requirements for the use of the GPU server container, then the personalized container deployment instruction is sent to the user terminal used by the teacher corresponding to the first authorized user account. After the teacher corresponding to the user account operates and approves the personalized container deployment instruction on the user terminal, the target slave node of the artificial intelligence experiment cloud platform according to the second target image resource corresponding to the personalized container deployment instruction, GPU resources and pre-stored data storage volume paths correspond to the creation of containers.

In one embodiment, the target acquires the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction from the node, including:

If the target slave node detects a unified experimental task deployment instruction, then obtain the teaching progress information and teacher teaching tag set corresponding to the unified experimental task deployment instruction, and according to the teaching progress information, teacher teaching tag set and preset resources The calling strategy generates the first target image resource, GPU resource and data storage volume path corresponding to the unified experiment task deployment instruction.

In this embodiment, when the target slave node detects a unified experimental task deployment instruction, specifically, the target slave node can analyze the unified experimental task deployment instruction to determine whether it includes teaching progress information (for example, learned Section 5 of Chapter 1 of the AA artificial intelligence course, etc.) and teacher teaching label sets (such as including labels such as face recognition and convolutional neural networks). If the unified experimental task deployment instruction is parsed and the teaching progress information and the teacher's teaching label set are obtained, then according to the teaching progress information, the teacher's teaching label set and the preset resource call strategy, the first target image resource required for creating the container is generated, Container information such as GPU resources and data storage volume paths. Wherein, the resource calling strategy can be understood as a preset mapping table including several pieces of teaching progress information, teacher teaching label set and target image resource, GPU resource and data storage volume path, with teaching progress information, teacher teaching label set as The search conditions can query the corresponding target image resources, GPU resources, and data storage volume paths.

The target slave node is further configured to connect the target container instance corresponding to the target container with the user terminal if the access request from the user terminal is received and passed the verification.

In this embodiment, after the target slave node completes the deployment of the target container and the target container instance, it can be provided to the user for access to process artificial intelligence-related experimental tasks. When a user needs to access a target container in a slave node, an access request with user account information is first sent to the target slave node. Afterwards, when the target slave node passes the verification of the access request, a communication connection between the target container instance corresponding to the target container and the user terminal is established. In this way, the user terminal can access the target slave node to process the experimental tasks related to artificial intelligence.

The target slave node is further configured to receive target operation data of the user terminal, and store the target operation data in a key-value database corresponding to the target container.

In this embodiment, after the user terminal is connected to the target container instance in the target slave node, the target container instance can receive the target operation data of the user terminal. In order to improve the data security of the target operation data, the target operation data can be stored in a key-value database corresponding to the target container in the target master node (such as Etcd database, which is a key-value database), In this way, even when the target slave node fails and stops running, the operation data of each container included therein is stored in the key-value database of the target master node. When the target slave node returns to normal after troubleshooting, it also calls all the data of the target slave node from the key-value database of the target master node for breakpoint recovery.

In an embodiment, the target master node is further configured to randomly select a master node from the remaining multiple master nodes as the target master node if it is detected that the current working state is an abnormal state.

In this embodiment, currently only one master node in a general cluster is working and performing various data processing, and other master nodes are in an inactive standby state. If the working target master node (that is, Leader-Master-Node) is abnormal, the cluster corresponding to the artificial intelligence experiment cloud platform will automatically select a master node from multiple master nodes in the standby state to immediately replace the master node in the abnormal state as the new master node. The running and active master node continues the current work. It can be seen that no matter which master node in the cluster fails, it will not affect the work of the entire cluster. If the working master node is abnormal, the cluster will automatically select a master node from the standby master nodes to replace the abnormal node as the new master immediately. The control node Leader-Master-Node continues the current work.

In one embodiment, the target slave node is also used to obtain the current node container data and store the current node container data in the target master node corresponding to the target slave node if it detects that the current working state is an abnormal state. Key-value database.

In this embodiment, when the current working state of the target slave node is an abnormal state, before it is restarted for troubleshooting, the current node container data of the target slave node can be obtained again in the target master node and the The current node container data is stored in the key-value database corresponding to the target slave node in the target master node for data backup. Create a persistent volume statement (PVC) and a persistent volume (PV) for each user in the key-value database. When the user stops and closes the operation of the target container instance in the target slave node or exits the target container instance due to a failure, The current node container data generated by the user's operation on the target container instance before this shutdown operation or exit operation will be automatically stored in the key-value database corresponding to the target slave node in the target master node for data backup. When the user enters the target container instance again next time, the artificial intelligence experiment cloud platform automatically calls the current node container data from the key-value database to restore the target container instance to the state when it exited last time, so as to For the user to continue to operate on the target container instance after re-entering this time.

Since all node container data of each container is persisted into a dynamically created persistent volume, whether the user actively exits the container instance or passively exits the container instance due to a failure, the artificial intelligence experiment cloud platform can automatically save the historical node container The data is stored in the persistent volume for the user to call the historical node container data and continue to operate on the container instance after re-entering the container.

The target master node is used to send container operation instructions to the target slave node.

In this embodiment, of course, in addition to operating the artificial intelligence-related experimental data in the container of the target slave node, the user can also access the target master node (for example, a user account with administrator authority logs in to the target master node), and then perform The triggering of the container operation instruction corresponding to operations such as adding or deleting a container generates an operation. After completing the container operation instruction that triggers adding or deleting containers, the target master node sends the container operation instruction to the target slave node.

The target slave node is further configured to correspondingly create or delete a container according to the container operation instruction if the container operation instruction is received.

In this embodiment, the artificial intelligence experiment cloud platform can realize the addition and deletion of cluster nodes at any time based on container operation instructions, so as to facilitate the expansion and contraction of Pod through the cluster controller. More specifically, when the resources in the target slave nodes are tight, add more slave nodes and create containers in the slave nodes to achieve capacity expansion.

The system realizes the processing of artificial intelligence-related experimental tasks based on the cloud in the artificial intelligence experiment cloud platform, and can add or delete nodes to the cluster at any time to improve the high availability and load capacity of the cluster.

The data processing system of the artificial intelligence experimental cloud platform with a high-availability cluster architecture can be realized in the form of a computer program, and the computer program can run on a computer device as shown in FIG. 4 .

Please refer to FIG. 4 . FIG. 4 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 is a server, and may also be a server cluster. The server can be an independent server, or provide cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (ContentDeliveryNetwork, CDN) , and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

Referring to FIG. 4 , the computer device 500 includes a processor 502 connected through a device bus 501 , a memory and a network interface 505 , wherein the memory may include a storage medium 503 and an internal memory 504 .

The storage medium 503 can store an operating system 5031 and a computer program 5032 . When the computer program 5032 is executed, it can make the processor 502 execute the data processing method of the high-availability cluster architecture artificial intelligence experiment cloud platform.

The processor 502 is used to provide calculation and control capabilities and support the operation of the entire computer device 500 .

The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the data processing method of the high-availability cluster architecture artificial intelligence experiment cloud platform.

The network interface 505 is used for network communication, such as providing data transmission and the like. Those skilled in the art can understand that the structure shown in FIG. 4 is only a block diagram of a partial structure related to the solution of this application, and does not constitute a limitation to the computer device 500 on which the solution of this application is applied. The specific computer device 500 may include more or fewer components than shown, or combine certain components, or have a different arrangement of components.

Wherein, the processor 502 is used to run the computer program 5032 stored in the memory, so as to realize the data processing method of the high-availability cluster architecture artificial intelligence experiment cloud platform disclosed in the embodiment of the present application.

Those skilled in the art can understand that the embodiment of the computer device shown in FIG. 4 does not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or less components than those shown in the illustration. Or combine certain components, or different component arrangements. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structures and functions of the memory and the processor are consistent with those of the embodiment shown in FIG. 4 , and will not be repeated here.

It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (Central Processing Unit, CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits ( Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Wherein, the general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

In another embodiment of the present application a computer readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the data processing method of the high-availability cluster architecture artificial intelligence experiment cloud platform disclosed in the embodiment of the present application is realized.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the foregoing method embodiments, and details are not repeated here. Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the relationship between hardware and software Interchangeability. In the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are implemented by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In the several embodiments provided in this application, it should be understood that the disclosed devices, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only for logical function division. In actual implementation, there may be other division methods, and units with the same function can also be combined into one Units such as a plurality of units or components may be combined or integrated into another device, or some features may be omitted, or not implemented. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may also be electrical, mechanical or other forms of connection.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present application.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of software products, and the computer software products are stored in a storage medium Among them, several instructions are included to make a computer device (which may be a personal computer, a background server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media capable of storing program codes such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk, or an optical disk.

The above is only a specific embodiment of the application, but the scope of protection of the application is not limited thereto. Any person familiar with the technical field can easily think of various equivalents within the scope of the technology disclosed in the application. Modifications or replacements, these modifications or replacements shall be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (10)

1. A high-availability cluster architecture artificial intelligence experiment cloud platform data processing method, applied to the artificial intelligence experiment cloud platform, is characterized in that, the artificial intelligence cloud platform includes a plurality of master nodes and a plurality of slave nodes, and the plurality of The master node and the plurality of slave nodes are all communicatively connected; the method includes: If the target slave node receives the experimental work task deployment instruction sent by the target master node, it will create a target container correspondingly according to the experimental work task deployment instruction; wherein, the target slave node is any one of the multiple slave nodes, The target master node is a currently active master node among the plurality of master nodes; If the target slave node receives the access request of the user terminal and passes the verification, connects the target container instance corresponding to the target container with the user terminal; The target slave node receives the target operation data of the user terminal, and stores the target operation data in a key-value database corresponding to the target container; The target master node sends container operation instructions to the target slave node; If the target slave node receives the container operation instruction, it creates or deletes a container correspondingly according to the container operation instruction. 2. the high-availability cluster architecture artificial intelligence experiment cloud platform data processing method according to claim 1, is characterized in that, described corresponding creation target container according to institute's experiment task deployment instruction, comprises: If the experimental work task deployment instruction is a unified experimental task deployment instruction, then the target obtains the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction from the node, and the target obtains from the node The node correspondingly creates a target container according to the first target image resource, GPU resource and data storage volume path corresponding to the unified experimental task deployment instruction; If the experimental work task deployment instruction is a personalized container deployment instruction, the target slave node obtains the second target image resource corresponding to the personalized container deployment instruction, and the target slave node deploys the personalized container according to the The second target image resource corresponding to the instruction and the pre-stored data storage volume path correspond to the creation target container. 3. the high-availability cluster architecture artificial intelligence experiment cloud platform data processing method according to claim 1, is characterized in that, if described target slave node receives the experimental task deployment instruction that target master node sends, then according to the experimental work Before the task deployment instruction corresponds to creating the target container, it also includes: The interface module in the target master node establishes a communication connection with the Kubelet proxy component of the target slave node. 4. the high-availability cluster architecture artificial intelligence experiment cloud platform data processing method according to claim 3, is characterized in that, the interface module in the target master node and the Kubelet proxy component of the target slave node establish a communication connection, comprising: The Keepalived component of the target master node automatically configures the virtual IP address of the artificial intelligence experiment cloud platform through the virtual routing redundancy protocol; The interface module of the target master node establishes a communication connection with the Kubelet agent component block of the target slave node based on the virtual IP address. 5. the high-availability cluster architecture artificial intelligence experiment cloud platform data processing method according to claim 1, is characterized in that, described target receives the target operating data of described user terminal from node, and described target operating data is stored in the said target operating data After describing the key-value database corresponding to the target container, it also includes: If the target master node detects that the current working state is abnormal, a master node is randomly selected from the remaining multiple master nodes as the target master node. 6. The high-availability cluster architecture artificial intelligence experiment cloud platform data processing method according to claim 1, wherein the target slave node receives the target operation data of the user terminal, and stores the target operation data in the After describing the key-value database corresponding to the target container, it also includes: If the target slave node detects that the current working state is an abnormal state, it obtains the current node container data and stores the current node container data in the key-value database corresponding to the target slave node in the target master node. 7. the high-availability cluster architecture artificial intelligence experiment cloud platform data processing method according to claim 2, is characterized in that, described target obtains the first target image resource corresponding to described unified experimental task deployment instruction, GPU resource from node and datastore volume paths, including: If the target slave node detects a unified experimental task deployment instruction, then obtain the teaching progress information and teacher teaching tag set corresponding to the unified experimental task deployment instruction, and according to the teaching progress information, teacher teaching tag set and preset resources The calling strategy generates the first target image resource, GPU resource and data storage volume path corresponding to the unified experiment task deployment instruction. 8. A high-availability cluster architecture artificial intelligence experiment cloud platform data processing system, running on the artificial intelligence experiment cloud platform, is characterized in that it includes a plurality of master nodes and a plurality of slave nodes, and the plurality of master nodes and the plurality of Each slave node is connected by communication; wherein, the target slave node is any slave node in the plurality of slave nodes, and the target master node is a currently active master node in the plurality of master nodes; The target slave node is configured to create a target container according to the experimental task deployment instruction corresponding to the experimental work task deployment instruction sent by the target master node; wherein the target slave node is any one of the plurality of slave nodes From the node, the target master node is a currently active master node among the plurality of master nodes; The target slave node is further configured to connect the target container instance corresponding to the target container to the user terminal if the access request from the user terminal is received and passed the verification; The target slave node is further configured to receive target operation data of the user terminal, and store the target operation data in a key-value database corresponding to the target container; The target master node is used to send container operation instructions to the target slave node; The target slave node is further configured to correspondingly create or delete a container according to the container operation instruction if receiving the container operation instruction sent by the target master node. 9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, characterized in that, when the processor executes the computer program, the computer program according to claim 1 is realized. The data processing method of the high-availability cluster architecture artificial intelligence experiment cloud platform described in any one of 1 to 7. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes any one of claims 1 to 7. The high-availability cluster architecture artificial intelligence experimental cloud platform data processing method described in the item.

Images