CN114492844A - Method and device for constructing machine learning workflow, electronic equipment and storage medium - Google Patents

Abstract

The present disclosure provides a method, apparatus, electronic device and storage medium for constructing a machine learning workflow. The method includes: acquiring a target workflow template; generating a target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template; Parameter information, calculate the hash value of the target node included in the target hash calculation graph; verify the hash value of the target node, determine the workflow steps that need to be run and the workflow steps that have been successfully run, and then use the workflow steps that need to be run. The workflow steps that are running and the workflow steps that have run successfully, build the target workflow. The method can solve the problem of time and resource waste caused by repeated execution of the same workflow, improve the operation efficiency and performance of the workflow, and thus improve the user experience.

Description

Construction method, device, electronic device and storage medium of machine learning workflow

technical field

The present disclosure relates to the field of logistics technology, and in particular, to a method, apparatus, electronic device and storage medium for constructing a machine learning workflow.

Background technique

Machine learning workflow technology can solve the problems of low efficiency and uneven quality of application development. Machine learning workflow technology is mainly transformed from process management workflow. The current operation of workflow can be divided into data-dependent-driven and task-state-dependent driven. In a real application development scenario, the machine learning workflow may be modified and run repeatedly, and the directed acyclic graph structure of the workflow is complex. Developers need to locate the cause of the poor overall effect based on the running results of the steps. However, the prior art only pays attention to the top-down calculation correctness, data exchange correctness and running time when the workflow is running. There are time and resource waste problems caused by the repeated execution of the exact same workflow, which affects the workflow. The running performance is not good, and the user experience is not good.

It should be noted that the information disclosed in the above Background section is only for enhancement of understanding of the background of the present disclosure, and therefore may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

SUMMARY OF THE INVENTION

The purpose of the present disclosure is to provide a machine learning workflow construction method, device, electronic device and storage medium, which can solve the problem of time and resource waste caused by repeated execution of the exact same workflow, and improve the operation of the workflow efficiency and performance, thereby enhancing the user experience.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or be learned in part by practice of the present disclosure.

According to one aspect of the present disclosure, there is provided a method for constructing a machine learning workflow, including: acquiring a target workflow template; and generating a target hash corresponding to the target workflow template according to organizational structure information of the target workflow template Calculation graph; using the calculation method of the defined hash calculation graph, according to the parameter information corresponding to the target workflow template, calculate the hash value of the target node included in the target hash calculation graph; for the hash value of the target node The value is verified, and the workflow steps that need to be executed and the workflow steps that have been successfully executed are determined, and then the target workflow is constructed by using the workflow steps that need to be executed and the workflow steps that have been successfully executed.

In some exemplary embodiments of the present disclosure, the calculation method of the hash calculation graph includes: the calculation formula of the hash value of the node is: the hash value of the input parameter corresponding to the node and the hash value of the configuration parameter corresponding to the node Perform the summation, and then use the hash algorithm to calculate the summation result; the calculation formula of the hash value of the output parameter corresponding to the node is: sum the hash value of the node and the hash value of the output parameter name corresponding to the node.

In some exemplary embodiments of the present disclosure, the generating a target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template includes: The components are converted into target nodes included in the target hash calculation graph, the connection relationship between the components is converted into the connection relationship between the target nodes, and the target hash calculation graph is generated.

In some exemplary embodiments of the present disclosure, the hash calculation method of the defined hash calculation graph is used to calculate the hash of the target node included in the target hash calculation graph according to the parameter information corresponding to the target workflow template. value, including: determining the input parameter corresponding to the target node and the configuration parameter corresponding to the target node according to the connection relationship between the target nodes and the parameters of the component; according to the input parameter corresponding to the target node parameter information, calculate the hash value of the input parameter corresponding to the target node; query the parameter information of the configuration parameter corresponding to the target node, and calculate the hash value of the configuration parameter corresponding to the target node according to the query parameter information; The hash value of the input parameter corresponding to the target node and the hash value of the configuration parameter corresponding to the target node are substituted into the hash value calculation formula of the node, and the hash value of the target node is calculated.

In some exemplary embodiments of the present disclosure, the input parameters corresponding to the target node include: input parameters of components corresponding to the target node, output parameters corresponding to upstream nodes of the target node; and, the target node corresponds to The configuration parameters include: configuration parameters of the component corresponding to the target node.

In some exemplary embodiments of the present disclosure, the calculating the hash value of the input parameter corresponding to the target node according to the parameter information of the input parameter corresponding to the target node includes: acquiring the information of the component corresponding to the target node. The parameter information of the input parameter, the obtained parameter information is calculated by using a hash algorithm, and the hash value of the input parameter of the component corresponding to the target node is obtained; the hash value of the upstream node of the target node and the target node are obtained. The parameter name of the output parameter corresponding to the upstream node of the node is substituted into the hash value calculation formula of the output parameter corresponding to the node, and the hash value of the output parameter corresponding to the upstream node of the target node is calculated; The hash value of the input parameter of the component corresponding to the node and the hash value of the output parameter corresponding to the upstream node of the target node are summed to obtain the hash value of the input parameter corresponding to the target node.

In some exemplary embodiments of the present disclosure, the calculating the hash value of the configuration parameter corresponding to the target node includes: judging whether the parameter type of the configuration parameter corresponding to the target node is an index; if so, adopting an interface request method, the hash value of the file under the query parameter information is calculated, and the hash value of the file is determined to be the hash value of the configuration parameter corresponding to the target node.

In some exemplary embodiments of the present disclosure, the hash value of the target node is verified, the workflow steps that need to be executed and the workflow steps that have been executed successfully are determined, and then the work that needs to be executed is used. The flow steps and the workflow steps that have been successfully executed, and constructing the target workflow, include: querying whether the hash value of the target node exists in the step information database; if so, marking the workflow step corresponding to the target node as Workflow steps that have been successfully run, and read the running information of the workflow steps that have been run successfully from the step information database; if not, mark the workflow steps corresponding to the target node as needing to be run The workflow step; according to the operation information of the workflow step that has been successfully executed and the workflow step that needs to be executed, a target workflow configuration file is generated.

In some exemplary embodiments of the present disclosure, the method further includes: invoking an interface to run the target workflow configuration file; and monitoring the running status of each step in the target workflow configuration file, and executing the step in the step In the case of success, the running status of the step is updated to run successfully.

According to an aspect of the present disclosure, there is provided an apparatus for constructing a machine learning workflow, comprising: an acquisition module, used for acquiring a target workflow template; The target hash calculation graph corresponding to the target workflow template; the calculation module is used to utilize the calculation method of the defined hash calculation graph, and calculate the target hash calculation graph according to the parameter information corresponding to the target workflow template, including The hash value of the target node of The workflow steps and the workflow steps that have been successfully run build the target workflow.

According to one aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and storage means for storing at least one program, which, when the at least one program is executed by the at least one processor, makes the at least one program A processor implements any of the machine learning workflow construction methods described above.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, any one of the above-mentioned methods for constructing a machine learning workflow is implemented.

The method for constructing a machine learning workflow provided by the embodiment of the present disclosure firstly generates a target hash calculation graph corresponding to a target workflow template, and then can use the defined calculation method of the hash calculation graph to calculate the targets included in the target hash calculation graph The hash value of the node, and then by verifying the calculated hash value, the workflow steps that need to be run and the workflow steps that have been successfully run can be obtained, and then the workflow steps that have been successfully run can be pruned to construct a The target workflow can solve the problem of time and resources caused by the repeated execution of the same workflow in the prior art, improve the operation efficiency and performance of the workflow, and improve the user experience.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 shows a schematic diagram of an exemplary system architecture to which a method for constructing a machine learning workflow according to an exemplary embodiment of the present disclosure can be applied;

FIG. 2 shows a flowchart of a method for constructing a machine learning workflow according to an exemplary embodiment of the present disclosure;

FIG. 3 shows a schematic structural diagram of a workflow template according to an exemplary embodiment of the present disclosure;

FIG. 4 shows a schematic structural diagram of a hash calculation graph according to an exemplary embodiment of the present disclosure;

5 shows a flowchart of calculating the hash value of the target node included in the target hash calculation graph according to an exemplary embodiment of the present disclosure;

FIG. 6 shows an overall flow chart of machine learning workflow construction according to an exemplary embodiment of the present disclosure;

7 shows a flow chart of calculating the hash value of each target node included in the target hash calculation graph according to an exemplary embodiment of the present disclosure;

FIG. 8 shows a schematic structural diagram of an apparatus for constructing a machine learning workflow according to an exemplary embodiment of the present disclosure;

FIG. 9 shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the figures are functional entities that do not necessarily necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

FIG. 1 shows a schematic diagram of an exemplary system architecture to which a method for building a machine learning workflow according to an exemplary embodiment of the present disclosure may be applied.

As shown in FIG. 1 , the system architecture may include a server 101 , a network 102 and a client 103 . The network 102 may provide the medium of the communication link between the client 103 and the server 101 . The network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The server 101 may be a server that provides various services, for example, a background management server that provides support for devices operated by a user using a client. The background management server can analyze and process the received requests and other data, and feed back the processing results to the client.

The client 103 may be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, a smart home device, an AR (Augmented Reality, augmented reality) device, a VR (Virtual Reality, virtual reality) device, or the like, or, Client 103 may also be a personal computer, such as a laptop portable computer and desktop computer, among others.

In an exemplary embodiment of the present disclosure, a user can send a workflow processing instruction to a server through a client, and the server can, for example, obtain the workflow processing instruction, and obtain a target workflow template according to the pipeline processing instruction; the server can, for example, obtain the target workflow template according to the Organizational structure information to generate a target hash calculation graph corresponding to the target workflow template; the server may, for example, use the defined hash calculation graph calculation method to calculate the target included in the target hash calculation graph according to the parameter information corresponding to the target workflow template The hash value of the node; the server can, for example, verify the hash value of the target node, determine the workflow steps that need to be executed and the workflow steps that have been successfully executed, and then use the workflow steps that need to be executed and the workflow steps that have been successfully executed. Workflow steps that build the target workflow.

It should be understood that the numbers of clients, networks and servers in FIG. 1 are only schematic, and the server 101 may be an entity server, a server cluster composed of multiple servers, or a cloud server, depending on actual needs , can have any number of terminal devices, networks and servers.

Hereinafter, each step of the method for constructing a machine learning workflow (MachineLearning Pipelines) in an exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.

FIG. 2 shows a flowchart of a method for constructing a machine learning workflow according to an exemplary embodiment of the present disclosure. The method provided by the exemplary embodiment of the present disclosure may be executed by the server as shown in FIG. 1 , but the present disclosure is not limited thereto.

As shown in FIG. 2 , the method for constructing a machine learning workflow provided by an exemplary embodiment of the present disclosure may include the following steps.

Step S201: Obtain a target workflow template.

The target workflow template is a workflow template that needs to be run, and the workflow template can be obtained according to the unique identifier of the target workflow template provided by the user. For example, the user inputs the unique identifier of the workflow template to be run on the interface of the workflow system, and then, when the unique identifier of the target workflow template is determined, the target job can be queried in the workflow template database according to the unique identifier. flow template. In addition, the unique identifier of the workflow template can be set according to the actual situation, such as the workflow name, the business scenario name of the workflow, and so on. Among them, the workflow system can be regarded as a system for processing workflow, which can provide a system interface for the user to input information.

It should be noted that, in the method for constructing a machine learning workflow according to an exemplary embodiment of the present disclosure, a workflow template may be created in advance according to requirements, and then the created workflow template may be stored in the workflow template database, so as to facilitate the use of Unique identifier to query the required workflow template. Of course, the organizational structure information of the workflow template can also be stored in the workflow template database. The organizational structure information of the workflow template may include: components included in the template, connection relationships between components, and input parameters, output parameters, common configuration parameters and resource configuration parameters of each component. FIG. 3 shows a schematic structural diagram of a workflow template according to an exemplary embodiment of the present disclosure. The workflow template shown in Figure 3 includes read local files, data partitioning, file merging (by column), file merging (by row), and linear regression components. For the data partitioning component, it is used to partition a single dataset into training and test sets. In addition, the common configuration parameters of the data partitioning component are {parameter name: test_size, parameter type: Float} and {parameter name: target_column, parameter type: String}, and resource configuration parameters are {CPU: 0.5 core, memory: 1024M, GPU: 0}. It can also be seen from the workflow template shown in FIG. 3 that the connection relationship between the various components is specifically the connection relationship between the output parameters of the upstream components and the input parameters of the downstream components.

Therefore, after the unique identifier of the target workflow template is determined, the target workflow template can be obtained according to the unique identifier, and the components included in the target workflow template, the connection relationship between the components, and the parameters of the components can be obtained. The parameters of the component may include input parameters, output parameters, common configuration parameters, and resource configuration parameters.

Step S202: Generate a target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template.

According to the above step S201, after obtaining the target workflow template, the organizational structure information of the target workflow template can be obtained, that is, the components included in the target workflow template, the connection relationship between the components, and the input of each component parameters, output parameters, common configuration parameters, and resource configuration parameters. In this way, the target hash calculation graph corresponding to the target workflow template can be generated by using the organizational structure information of the target workflow template. The hash calculation graph refers to a calculation graph in which hash functions are added to each node.

In an exemplary embodiment of the present disclosure, generating a target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template may include: converting the components included in the target workflow template into the target hash calculation graph including: The target node, the connection relationship between components is converted into the connection relationship between target nodes, and the target hash calculation graph is generated.

In the process of generating the target hash calculation graph, the components included in the target workflow template can be converted into target nodes included in the target hash calculation graph, and the connection relationship between the components can be converted into the connection relationship between the target nodes. Specifically, each component in the target workflow template can be regarded as a target node of the target hash calculation graph, and the connection relationship between the components corresponds to the connection relationship between the target nodes. Additionally, the target workflow template contains one or more components. Since each component can be regarded as a node, the target hash calculation graph contains one or more target nodes. In addition, the target node is used to represent the node included in the target hash calculation graph in order to distinguish it from the node in the node hash value calculation formula.

FIG. 4 shows a schematic structural diagram of a hash calculation graph according to an exemplary embodiment of the present disclosure, and the hash calculation graph shown in FIG. 4 is generated according to the workflow template shown in FIG. 3 . As can be seen from Figure 4, the components of reading local files, data division, file merging (by column), file merging (by row) and linear regression in Figure 3 are converted into nodes A, B, C, D and E respectively. , according to the connection relationship between components, the relationship between nodes is set such that node A is the upstream node of node B, node B is the upstream node of node C and node D, and the upstream nodes of node E are node C and node D.

Step S203: Calculate the hash value of the target node included in the target hash calculation graph according to the parameter information corresponding to the target workflow template by using the defined calculation method of the hash calculation graph.

Different from the hash chain, the hash calculation graph can connect each node in the form of a graph. After splicing the values of all directly associated upstream nodes, the value calculated by the hash algorithm is the value of the current node, that is, using Define the calculation method of the hash calculation graph, and calculate the hash value of the node. Among them, the calculation method of the hash calculation graph can be regarded as a workflow pruning algorithm based on the hash calculation graph. Pruning is to avoid some unnecessary traversal processes through a certain judgment, that is, to cut off some "branches" in the search tree. The core problem of applying pruning optimization is to design a pruning judgment method, that is, a method to determine which branches should be discarded and which should be kept. The workflow pruning algorithm based on the hash calculation graph refers to the use of the hash calculation graph for pruning. Specifically, each node included in the hash calculation graph is analyzed, and it is judged that the workflow steps corresponding to the nodes should be discarded or retained. In this way, the repeatedly running workflow steps can be pruned, that is, when running the workflow, the pruned workflow steps can be skipped, so as to avoid the repeated execution of the exact same workflow, improve the running efficiency of the workflow, and improve the user experience. experience.

In an exemplary embodiment of the present disclosure, the calculation method of the hash calculation graph may include: (1) the hash value of the node=Hash (the hash value of the input parameter corresponding to the node+the hash value of the configuration parameter corresponding to the node) ), Hash() refers to using the hash algorithm to calculate the hash value of the content in (); (2) the hash value of the output parameter corresponding to the node = the hash value of the node + the hash value of the output parameter name Greek value. The hash algorithm guarantees that the same input can get the same output, and the hash algorithm can prevent different inputs from getting the same output. Currently, there are various hash algorithms available, such as md5, sha1, sha256, and sha512. Considering the computational efficiency, storage requirements, and collision probability, the md5 algorithm is preferred.

In the hash value calculation formula of the node, the hash value of the input parameter corresponding to the node = the hash value of all input parameters of the node + the hash value of the parameter names of all input parameters of the node, the corresponding configuration of the node The hash value of the parameter = the hash value of all configuration parameters of the node + the hash value of the parameter names of all configuration parameters of the node. Since the corresponding resource configuration parameters for different components included in the same workflow template are the same, the configuration parameters here can be common configuration parameters. In addition, considering that a node can have multiple output parameters, a hash value of the parameter name is introduced into the calculation formula for differentiation.

FIG. 5 shows a flowchart of calculating a hash value of a target node included in a target hash calculation graph according to an exemplary embodiment of the present disclosure. As shown in Figure 5, using the calculation method of the defined hash calculation graph, according to the parameter information corresponding to the target workflow template, calculate the hash value of the target node included in the target hash calculation graph, which may include:

Step S501, according to the connection relationship between the target nodes and the parameters of the components, determine the input parameters corresponding to the target node and the configuration parameters corresponding to the target node;

Step S502, according to the parameter information of the input parameter corresponding to the target node, calculate the hash value of the input parameter corresponding to the target node;

Step S503, query the parameter information of the configuration parameter corresponding to the target node, and calculate the hash value of the configuration parameter corresponding to the target node according to the queried parameter information;

Step S504: Substitute the hash value of the input parameter corresponding to the target node and the hash value of the configuration parameter corresponding to the target node into the hash value calculation formula of the node to calculate the hash value of the target node.

Among them, step S501 is to determine the input parameters and configuration parameters corresponding to the target node; step S502 is to calculate the hash value of the input parameters corresponding to the target node according to the information of the input parameters; step S503 is to calculate the corresponding input parameters of the target node according to the information of the configuration parameters. The hash value of the configuration parameter; step S504 is to substitute the hash value of the input parameter and the hash value of the configuration parameter calculated in the above steps into the hash value calculation formula of the defined node, so that the target node can be obtained. hash value. It should be noted that the number of target nodes included in the target hash calculation graph is also one or more, so it is necessary to use the methods described in steps S501 to S504 to calculate the hash value of each target node. Next, steps S501 to S504 will be described in detail.

First, in step S501, the input parameters corresponding to the target node may include: input parameters of the component corresponding to the target node, and output parameters corresponding to the upstream node of the target node; the configuration parameters corresponding to the target node may include: the component corresponding to the target node. Configuration parameters. Of course, if a node does not have an upstream node, the input parameter corresponding to the node is the input parameter of the component corresponding to the node.

For ease of understanding, nodes A, B, C, D, and E in FIG. 4 are used as examples for description. Node A corresponds to the read local file component, and node A has no upstream node, then the input parameter corresponding to node A is the input parameter a of the read local file component, and the configuration parameter corresponding to node A is the configuration parameter of the read local file component. Node B corresponds to the data partitioning component, and the upstream node of node B is node A, then the input parameters corresponding to node B are the input parameter b of the data partitioning component and the output parameter a1 corresponding to node A, and the configuration parameter corresponding to node B is data partitioning Configuration parameters for the component. Node C corresponds to the file merge (by column) component, and the upstream node of node C is node B, then the input parameter corresponding to node C is the input parameter c of the file merge (by column) component and the output parameter b1 corresponding to node B, node The configuration parameter corresponding to C is the configuration parameter of the file merge (by column) component. Node D corresponds to the file merge (by line) component, and the upstream node of node D is node B, then the input parameter corresponding to node D is the input parameter d of the file merge (by line) component and the output parameter b2 corresponding to node B, node The configuration parameter corresponding to D is the configuration parameter of the file merge (by line) component. Node E corresponds to the linear regression component, and the upstream nodes of node E are node C and node D, then the input parameters corresponding to node E are the input parameter e of the linear regression component, the output parameter c1 corresponding to node C, and the output parameter d1 of node D , the configuration parameters corresponding to node E are the configuration parameters of the linear regression component.

In step S502, the hash value of the input parameter corresponding to the target node is calculated according to the parameter information of the input parameter corresponding to the target node. The specific implementation can be as follows:

(1) Obtain the parameter information of the input parameters of the component corresponding to the target node, use the hash algorithm to calculate the obtained parameter information, and obtain the hash value of the input parameter of the component corresponding to the target node.

The user provides the parameter information of the input parameters of the component corresponding to the target node, such as the user inputs the actual operation parameter information of the input parameters of each component of the target workflow template on the interface of the workflow system, so that the parameter information provided by the user can be calculated, Get the corresponding hash value. For example, the specific parameter value m1 of the input parameter b of the data partition component corresponding to node B is obtained, and then the hash algorithm is used to perform the hash calculation on m1, and the hash algorithm is used to perform the hash calculation on the parameter name of b. The calculation results are summed to obtain the hash value of the input parameter b of the data division component corresponding to the node B.

It should be noted that if there are multiple input parameters of the component corresponding to the target node, the hash value of each input parameter needs to be calculated, and then summed to obtain the hash value of the input parameter of the component corresponding to the target node.

(2) Substitute the hash value of the upstream node of the target node and the parameter name of the output parameter corresponding to the upstream node of the target node into the hash value calculation formula of the output parameter corresponding to the node, and calculate the corresponding value of the upstream node of the target node. The hash value of the output parameter.

As described above, the input parameters corresponding to the target node include the output parameters corresponding to the upstream nodes of the target node. Therefore, the hash value of the output function corresponding to the upstream node can be calculated by using the hash value calculation formula of the output parameters corresponding to the node. For example, if the input parameter corresponding to node B includes the output parameter a1 corresponding to node A, then the hash value of node A and the hash value of the parameter name of a1 are summed to obtain the output parameter a1 corresponding to the upstream node A of node B hash value. Of course, if there are multiple output parameters corresponding to the upstream node of the target node, then the hash value of each output parameter needs to be calculated, and then summed to obtain the hash value of the output parameter corresponding to the upstream node of the target node. It should also be noted that the output parameter corresponding to the upstream node of the target node refers to the parameter output from the upstream node and input to the target node. For example, the output parameter b1 in Figure 4 is the output parameter corresponding to the upstream node B of the node C, which is the input parameter of the node C, and the parameter b2 is the output parameter of the node B, but it cannot be regarded as the input parameter of the node C.

(3) The hash value of the input parameter of the component corresponding to the target node and the hash value of the output parameter corresponding to the upstream node of the target node are summed to obtain the hash value of the input parameter corresponding to the target node. For example, the hash value of the input parameter corresponding to the node B = the hash value of the input parameter b + the hash value of the node A + the hash value of the parameter name of a1.

Step S502 is to calculate the hash value of the input parameter corresponding to the target node, and step S503 is to query the parameter information of the configuration parameter corresponding to the target node, and then calculate the hash value of the configuration parameter corresponding to the target node according to the queried parameter information.

In step S503, by querying the parameter configuration database, the actual operation parameter information of the configuration parameters of each component of the target workflow template is obtained, mainly including the actual values of the common configuration parameters of the components and the operation resource configuration parameters. Among them, the types of common configuration parameters of components include values and indexes, such as file paths; the configuration parameters of running resources include the number of CPUs, the number of GPU cores, and the memory size. In addition, if the type of the common configuration parameter of the component is index, you need to request the calculation of the hash value of the corresponding file through the interface request method, and the obtained hash value is used as the value of the parameter configuration. Therefore, in the exemplary embodiment of the present disclosure, calculating the hash value of the configuration parameter corresponding to the target node may include: judging whether the parameter type of the configuration parameter corresponding to the target node is an index; The hash value of the file under the parameter information is determined, and the hash value of the file is determined to be the hash value of the configuration parameter corresponding to the target node.

Through step S502, the hash value of the input parameter corresponding to the target node is obtained by calculation, and through step S503, the hash value of the configuration parameter corresponding to the target node is obtained by calculation, and then in step S504, the value obtained by calculation in step S502 can be The hash value of the input parameter corresponding to the target node and the hash value of the configuration parameter corresponding to the target node calculated in step S503 are substituted into the hash value calculation formula of the node, and finally the hash value of the target node can be obtained. It should be noted that, in the exemplary embodiment of the present disclosure, for the target nodes included in the target hash calculation graph, the hash value of each target node needs to be calculated from top to bottom starting from the target node without upstream nodes.

Step S204: Verify the hash value of the target node, determine the workflow steps that need to be executed and the workflow steps that have been successfully executed, and then construct the target using the workflow steps that need to be executed and the workflow steps that have been successfully executed. Workflow.

The hash value of each target node is obtained by calculating in step S203, and in step S204, the calculated hash value needs to be verified, and the specific verification process may be: query whether the hash value of the target node exists in the step information database; If yes, mark the workflow step corresponding to the target node as a successfully executed workflow step, and read the running information of the successfully executed workflow step from the step information database; if not, mark the work corresponding to the target node A flow step is a workflow step that needs to be executed; a target workflow configuration file is generated according to the running information of the successfully executed workflow step and the workflow step that needs to be executed.

After calculating and obtaining the hash value of a target node, it will query whether the hash value already exists in the step information database. Specifically, if the calculated hash value already exists in the step information database, it means that the step with the same parameter configuration and the same definition (that is, the exact same) has been successfully run at a certain time before, so only need The previous running status and information can be directly queried in the step information database and assigned to the workflow step corresponding to the target node, and the workflow step corresponding to the target node can be marked as a workflow step that has been successfully run. If the hash value of the target node is not in the step information database, it means that the same step as the workflow step corresponding to the target node has not been executed, so the workflow step corresponding to the target node needs to be marked as needing to be executed workflow steps.

After analyzing the hash value of each target node included in the target hash calculation graph, it can be determined that the target workflow template contains the workflow steps that have been successfully run and the workflow steps that need to be executed, and can obtain the workflow steps that have been successfully executed. Run information for the workflow steps that are run, which in turn can generate the target workflow configuration file. That is to say, after the hash values of all target nodes contained in the target hash calculation graph are calculated, the workflow steps that have been successfully run (that is, the workflow steps that do not need to be run again) will be skipped, that is The workflow steps that have been successfully run are pruned, and the workflow steps that need to be run can also be obtained. At this point, the target workflow configuration file can be reconstructed using the workflow steps that need to be executed and the running information of the workflow steps that have been successfully executed.

In addition, in an exemplary embodiment of the present disclosure, the workflow processing method may further include: calling an interface to run the target workflow configuration file; and monitoring the running status of each step in the target workflow configuration file, and if the step is successfully executed In this case, the run status of the update step is run successful.

An application program calling interface is established between the workflow system and the underlying workflow executor, and the workflow system calling interface runs the workflow configuration file. Moreover, the workflow system will always monitor the running status of each step of the workflow. When a step of the workflow runs successfully, the running information of the step will be updated in the step information database. In general, the step information database records all steps, the workflow to which the step belongs, and the operation information of the step, so that the step and the workflow to which the step belongs can be queried according to the operation information of a step.

In the workflow processing method provided by the embodiments of the present disclosure, a target hash calculation graph corresponding to the target workflow template is first generated, and then the hash calculation method of the defined hash calculation graph can be used to calculate the hash rate of the target node included in the target hash calculation graph. Then, by verifying the calculated hash value, the workflow steps that need to be run and the workflow steps that have been successfully run can be obtained, and then the workflow steps that have been successfully run can be pruned to construct the target workflow , which can solve the problem of time and resources caused by the repeated execution of the same workflow in the prior art, improve the running efficiency and performance of the workflow, and improve the user experience. In addition, all steps, the workflow to which the step belongs, and the operation information of the step are recorded in the step information database, and the workflow to which the step belongs can be found according to a certain step, and the operation result of the step and the connection with other steps can also be queried. , which realizes the traceability of workflow running results.

Fig. 6 shows an overall flow chart of machine learning workflow construction according to an exemplary embodiment of the present disclosure. As shown in Fig. 6, the overall flow of workflow construction is: according to the unique identifier of the workflow template provided by the user, from the workflow Read the target workflow template from the flow template database; convert the components contained in the target workflow template into the target nodes contained in the target hash calculation graph, convert the connection relationship between the components into the connection relationship between the target nodes, and generate the target Hash calculation graph; starting from the target node without an upstream node, calculate the hash value of each target node included in the target hash calculation graph from top to bottom, and then check whether the hash value of the target node exists in the step information database value; if so, mark the workflow step corresponding to the target node as a successfully executed workflow step, and read the running information of the successfully executed workflow step from the step information database; if not, mark the target The workflow step corresponding to the node is the workflow step that needs to be executed; according to the running information of the workflow step that has been successfully executed and the workflow step that needs to be executed, the target workflow configuration file is generated; the interface is called to run the target workflow configuration file ; Monitor the running status of each step in the target workflow configuration file. If the step runs successfully, update the running status of the step to run successfully, and store the running result of the step in the step information database.

FIG. 7 shows a flowchart of calculating the hash value of each target node included in the target hash calculation graph according to an exemplary embodiment of the present disclosure. As shown in Figure 7, the specific process of calculating the hash value of each target node may include:

Step S701, according to the connection relationship between the target nodes and the parameters of the components, determine the input parameters corresponding to the target node and the configuration parameters corresponding to the target node;

Step S702, obtaining parameter information of the input parameter of the component corresponding to the target node, using a hash algorithm to calculate the obtained parameter information, and obtaining the hash value of the input parameter of the component corresponding to the target node;

Step S703: Substitute the hash value of the upstream node of the target node and the parameter name of the output parameter corresponding to the upstream node of the target node into the hash value calculation formula of the output parameter corresponding to the node, and calculate the upstream node of the target node. The hash value of the output parameter corresponding to the node;

Step S704, summing the hash value of the input parameter of the component corresponding to the target node and the hash value of the output parameter corresponding to the upstream node of the target node to obtain the hash value of the input parameter corresponding to the target node;

Step S705, query the parameter information of the configuration parameter corresponding to the target node, and determine whether the parameter type of the configuration parameter corresponding to the target node is an index according to the parameter information inquired, if yes, execute step S706, if not, execute step S707 ;

Step S706, using the method of interface request, calculate the hash value of the file under the query parameter information, and determine that the hash value of the file is the hash value of the configuration parameter corresponding to the target node;

Step S707, using a hash algorithm to calculate the hash value of the query parameter information;

Step S708, the hash value of the input parameter corresponding to the target node and the hash value of the configuration parameter corresponding to the target node are substituted into the hash value calculation formula of the node, and the hash value of the target node is calculated.

The following are the apparatus embodiments of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

FIG. 8 shows a schematic structural diagram of an apparatus for constructing a machine learning workflow according to an exemplary embodiment of the present disclosure.

As shown in FIG. 8 , the apparatus 800 for constructing a machine learning workflow may include: an acquisition module 801 , a generation module 802 , a calculation module 803 and a construction module 804 .

Wherein, the obtaining module 801 can be used for: obtaining the target workflow template; the generating module 802 can be used for: generating the target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template; the calculating module 803 can be used for: using The calculation method of the defined hash calculation graph is to calculate the hash value of the target node included in the target hash calculation graph according to the parameter information corresponding to the target workflow template; the building module 804 can be used for: verifying the hash value of the target node , determine the workflow steps that need to be executed and the workflow steps that have been successfully executed, and then construct the target workflow using the workflow steps that need to be executed and the workflow steps that have been successfully executed.

The calculation method of the hash calculation graph may include: (1) The calculation formula of the hash value of the node is: sum the hash value of the input parameter corresponding to the node and the hash value of the configuration parameter corresponding to the node, and then use The hash algorithm calculates the summation result; (2) The calculation formula of the hash value of the output parameter corresponding to the node is: sum the hash value of the node and the hash value of the output parameter name corresponding to the node.

In an exemplary embodiment of the present disclosure, the generating module 802 is further configured to: convert the components included in the target workflow template into target nodes included in the target hash calculation graph, and convert the connection relationship between the components into the connection relationship between the target nodes Connect the relationship to generate the target hash calculation graph.

In an exemplary embodiment of the present disclosure, the calculation module 803 may be further configured to: determine the input parameters corresponding to the target node and the configuration parameters corresponding to the target node according to the connection relationship between the target nodes and the parameters of the components; according to the input parameters corresponding to the target node The parameter information of the parameter, calculate the hash value of the input parameter corresponding to the target node; query the parameter information of the configuration parameter corresponding to the target node, and calculate the hash value of the configuration parameter corresponding to the target node according to the queried parameter information; The hash value of the input parameter and the hash value of the configuration parameter corresponding to the target node are substituted into the hash value calculation formula of the node, and the hash value of the target node is calculated.

The input parameters corresponding to the target node may include: input parameters of components corresponding to the target node, and output parameters corresponding to upstream nodes of the target node; and configuration parameters corresponding to the target node may include: configuration parameters of components corresponding to the target node.

In an exemplary embodiment of the present disclosure, the calculation module 803 may be further configured to: obtain parameter information of the input parameters of the component corresponding to the target node, use a hash algorithm to calculate the obtained parameter information, and obtain the input parameters of the component corresponding to the target node The hash value of the target node; the hash value of the upstream node of the target node and the parameter name of the output parameter corresponding to the upstream node of the target node are substituted into the hash value calculation formula of the output parameter corresponding to the node, and the upstream node of the target node is calculated. Hash value of the corresponding output parameter; sum the hash value of the input parameter of the component corresponding to the target node and the hash value of the output parameter corresponding to the upstream node of the target node to obtain the hash value of the input parameter corresponding to the target node value.

In an exemplary embodiment of the present disclosure, the calculation module 803 is further configured to: determine whether the parameter type of the configuration parameter corresponding to the target node is an index; if so, calculate the hash of the file under the query parameter information by means of an interface request value, and determine the hash value of the file as the hash value of the configuration parameter corresponding to the target node.

In an exemplary embodiment of the present disclosure, the building module 804 is further configured to: query whether the hash value of the target node exists in the step information database; if so, mark the workflow step corresponding to the target node as a workflow step that has been successfully executed, And from the step information database, read the operation information of the workflow step that has been successfully run; if not, mark the workflow step corresponding to the target node as the workflow step that needs to be run; Run information and workflow steps that need to be run to generate the target workflow configuration file.

In an exemplary embodiment of the present disclosure, the above-mentioned apparatus may further include: an operation module. The running module is used for: calling the interface to run the target workflow configuration file; and, monitoring the running status of each step in the target workflow configuration file, if the step runs successfully, update the running status of the step to run successfully, and the step The running results are stored in the step information database.

It should be noted that the block diagrams shown in the above figures are functional entities, and do not necessarily necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

FIG. 9 shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. It should be noted that the electronic device shown in FIG. 9 is only an example, and should not impose any limitations on the function and scope of use of the embodiments of the present invention.

As shown in FIG. 9 , an electronic device 900 includes a central processing unit (CPU) 901 that can be loaded into a random access memory (RAM) 903 according to a program stored in a read only memory (ROM) 902 or a program from a storage section 908 Instead, various appropriate actions and processes are performed. In the RAM 903, various programs and data necessary for the operation of the system 900 are also stored. The CPU 901 , the ROM 902 , and the RAM 903 are connected to each other through a bus 904 . An input/output (I/O) interface 905 is also connected to bus 904 .

The following components are connected to the I/O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc. ; and a communication section 909 including a network interface card such as a LAN card, a modem, and the like. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I/O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 910 as needed so that a computer program read therefrom is installed into the storage section 908 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs according to embodiments of the present invention. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 909, and/or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-described functions defined in the system of the present invention are executed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The units involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described unit may also be provided in a processor, for example, it may be described as: a processor includes a sending unit, an obtaining unit, a determining unit and a first processing unit. Wherein, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the sending unit may also be described as "a unit that sends a request for obtaining pictures to the connected server".

As another aspect, the present disclosure also provides a computer-readable storage medium. The computer-readable storage medium may be included in the electronic device described in the above embodiments; in electronic equipment. The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device enables the electronic device to implement the methods described in the following embodiments. For example, the electronic device can implement the various steps shown in FIG. 2 .

According to one aspect of the present disclosure, there is provided a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods provided in various optional implementations of the above-described embodiments.

It should be understood that any number of elements in the drawings of the present disclosure is for illustration rather than limitation, and any designation is for distinction only and does not have any limiting meaning.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common general knowledge or techniques in the technical field not disclosed by this disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (12)

1. a construction method of machine learning workflow, is characterized in that, comprises: Get the target workflow template; generating a target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template; Using the calculation method of the defined hash calculation graph, according to the parameter information corresponding to the target workflow template, calculate the hash value of the target node included in the target hash calculation graph; Verify the hash value of the target node, determine the workflow steps that need to be executed and the workflow steps that have been successfully executed, and then use the workflow steps that need to be executed and the workflow steps that have been successfully executed. , build the target workflow. 2. The method according to claim 1, wherein the calculation method of the hash calculation graph comprises: The calculation formula of the hash value of the node is: sum the hash value of the input parameter corresponding to the node and the hash value of the configuration parameter corresponding to the node, and then use the hash algorithm to calculate the summation result; The calculation formula of the hash value of the output parameter corresponding to the node is: sum the hash value of the node and the hash value of the output parameter name corresponding to the node. 3. The method according to claim 2, wherein generating a target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template comprises: converting the target The components included in the workflow template are converted into target nodes included in the target hash calculation graph, the connection relationship between the components is converted into the connection relationship between the target nodes, and the target hash calculation graph is generated. 4. method according to claim 3, is characterized in that, described utilizing the calculation method of the hash calculation graph of definition, according to the parameter information corresponding to described target workflow template, calculates described target hash calculation graph comprises. The hash of the target node, including: According to the connection relationship between the target nodes and the parameters of the components, determine the input parameters corresponding to the target node and the configuration parameters corresponding to the target node; Calculate the hash value of the input parameter corresponding to the target node according to the parameter information of the input parameter corresponding to the target node; query the parameter information of the configuration parameter corresponding to the target node, and calculate the hash value of the configuration parameter corresponding to the target node according to the query parameter information; The hash value of the input parameter corresponding to the target node and the hash value of the configuration parameter corresponding to the target node are substituted into the hash value calculation formula of the node, and the hash value of the target node is calculated. 5. The method according to claim 4, wherein the input parameters corresponding to the target node comprise: input parameters of components corresponding to the target node and output parameters corresponding to upstream nodes of the target node; and, The configuration parameters corresponding to the target node include: configuration parameters of components corresponding to the target node. 6. The method according to claim 5, wherein, calculating the hash value of the input parameter corresponding to the target node according to the parameter information of the input parameter corresponding to the target node, comprising: Acquiring parameter information of the input parameters of the component corresponding to the target node, using a hash algorithm to calculate the obtained parameter information, and obtaining the hash value of the input parameter of the component corresponding to the target node; Substitute the hash value of the upstream node of the target node and the parameter name of the output parameter corresponding to the upstream node of the target node into the hash value calculation formula of the output parameter corresponding to the node, and calculate the target node The hash value of the output parameter corresponding to the upstream node of ; The hash value of the input parameter of the component corresponding to the target node and the hash value of the output parameter corresponding to the upstream node of the target node are summed to obtain the hash value of the input parameter corresponding to the target node. 7. The method according to claim 4, wherein the calculating the hash value of the configuration parameter corresponding to the target node comprises: Determine whether the parameter type of the configuration parameter corresponding to the target node is an index; If so, the method of interface request is used to calculate the hash value of the file under the parameter information of the query, and determine the hash value of the file as the hash value of the configuration parameter corresponding to the target node. 8. The method according to claim 1, characterized in that, verifying the hash value of the target node, determining the workflow steps that need to be executed and the workflow steps that have been successfully executed, and then using the The workflow steps that need to be run and the workflow steps that have been successfully run, build the target workflow, including: Whether there is a hash value of the target node in the query step information database; If so, mark the workflow step corresponding to the target node as a workflow step that has been successfully executed, and read the operation information of the workflow step that has been successfully executed from the step information database; If not, marking the workflow step corresponding to the target node as the workflow step that needs to be executed; A target workflow configuration file is generated according to the operation information of the workflow steps that have been successfully executed and the workflow steps that need to be executed. 9. The method according to claim 8, wherein the method further comprises: invoking an interface to run the target workflow configuration file; and, Monitor the running status of each step in the target workflow configuration file, and update the running status of the step to run successfully if the step runs successfully. 10. A device for constructing a machine learning workflow, comprising: Get module, used to get the target workflow template; a generating module, configured to generate a target hash calculation graph corresponding to the target workflow template according to the organizational structure information of the target workflow template; A calculation module, configured to calculate the hash value of the target node included in the target hash calculation graph according to the parameter information corresponding to the target workflow template by using the calculation method of the defined hash calculation graph; The building module is used to verify the hash value of the target node, determine the workflow steps that need to be executed and the workflow steps that have been successfully executed, and then use the workflow steps that need to be executed and the workflow steps that have been successfully executed. Workflow steps that run to build the target workflow. 11. An electronic device, characterized in that, comprising: at least one processor; A storage device for storing at least one program which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1 to 9. 12. A computer-readable storage medium on which computer-executable instructions are stored, characterized in that, when the executable instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images