[go: up one dir, main page]

CN111949832A - Method and device for analyzing dependency relationship of batch operation - Google Patents

Method and device for analyzing dependency relationship of batch operation Download PDF

Info

Publication number
CN111949832A
CN111949832A CN202010820319.4A CN202010820319A CN111949832A CN 111949832 A CN111949832 A CN 111949832A CN 202010820319 A CN202010820319 A CN 202010820319A CN 111949832 A CN111949832 A CN 111949832A
Authority
CN
China
Prior art keywords
job
dependency
instance
dependency relationship
batch
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010820319.4A
Other languages
Chinese (zh)
Inventor
郑旭然
王前程
魏雷
张凯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202010820319.4A priority Critical patent/CN111949832A/en
Publication of CN111949832A publication Critical patent/CN111949832A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/901Indexing; Data structures therefor; Storage structures
    • G06F16/9024Graphs; Linked lists
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/901Indexing; Data structures therefor; Storage structures
    • G06F16/9027Trees
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/904Browsing; Visualisation therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Stored Programmes (AREA)

Abstract

The invention provides a method and a device for analyzing a dependency relationship of batch operation, which relate to the technical field of big data, and the method for analyzing the dependency relationship comprises the following steps: abstracting each operation instance to obtain the characteristic attribute corresponding to each operation instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object; determining the dependency relationship between each job instance and other job instances; establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure; and analyzing the dependency model to obtain a job dependency graph of the batch job. The method and the device can intuitively master the background batch operation dependency relationship, reduce the system maintenance risk and improve the problem analysis efficiency.

Description

Method and device for analyzing dependency relationship of batch operation
Technical Field
The invention relates to the technical field of big data, in particular to a method and a device for analyzing a dependency relationship of batch operation.
Background
Batch job processing is suitable for background automation operations that require little user interaction, primarily running at a pre-scheduled time or centrally running when needed. The system can run according to a plan, intensively access a large amount of data, complete expected processing or generate a corresponding report. In recent years, with the rapid development of the internet, the daily average transaction amount of a bank system continues to increase greatly. In order to guarantee the service response speed, the online transaction only completes necessary real-time data updating operation, and the data updating operation which does not need real-time updating or user interaction is processed by background batch operation at the service peak.
The quantity of batch operations is increasing day by day, the dependency relationship between the front and back items is disordered and complex, and an intuitive visual batch operation relationship diagram is lacked, so that great inconvenience and risk are brought to system maintenance. Unnecessary dependency exists, which may cause delay of the whole batch operation time of the system and influence the service processing efficiency; even the cyclic dependence occurs, which results in the system not functioning properly.
In order to solve the problems of disordered and complex background batch operation relationships and high system maintenance cost of the existing large-scale enterprise, a background batch operation dependency relationship analysis method and a visual display method are urgently needed.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method and a device for analyzing the dependency relationship of batch operation, which can intuitively master the dependency relationship of background batch operation, reduce the risk of system maintenance and improve the problem analysis efficiency.
In order to solve the technical problems, the invention provides the following technical scheme:
in a first aspect, the present invention provides a method for analyzing a dependency relationship of a batch job, including:
abstracting each operation instance to obtain the characteristic attribute corresponding to each operation instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
determining the dependency relationship between each job instance and other job instances;
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure;
and analyzing the dependency model to obtain a job dependency graph of the batch job.
Wherein the feature attributes include: at least one of job English name, job function Chinese description, job trigger type, job group name and timing time.
Wherein the determining the dependency relationship between each job instance and other job instances comprises:
determining an opening operation example in each operation example;
determining the dependency relationship of a first operation example related to the opening operation example according to the operation execution sequence;
traversing dependencies of job instances associated with the first job instance based on a job execution order.
Wherein the data structure comprises: adjacency matrices and adjacency tables.
Optionally, the establishing a dependency model according to the dependency relationship of each job instance, the object corresponding to each job instance, and the preset data structure includes:
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and the adjacency matrix, wherein the dependency model comprises the following steps:
determining an adjacency matrix of the corresponding number of the operation instances;
and setting element positions corresponding to two objects with dependency relationship in the adjacency matrix as dependency marks.
Optionally, the establishing a dependency model according to the dependency relationship of each job instance, the object corresponding to each job instance, and the preset data structure includes:
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and the adjacency list;
determining an adjacency list formed by the list heads of the corresponding number of the operation examples;
storing the object corresponding to each operation instance to a header at one time;
and storing the operation examples corresponding to the table headers into a unidirectional chain structure or a tree structure in the adjacent table according to the dependency relationship of the operation examples.
Analyzing the dependency model to obtain a job dependency graph of the batch job, including:
and searching the upper layer and lower layer dependency relationship of the object corresponding to each operation instance in the dependency model by adopting a breadth-first traversal algorithm to generate an operation dependency relationship graph.
In a second aspect, the present invention provides an apparatus for analyzing a dependency relationship of a batch job, including:
the abstract unit is used for carrying out abstract processing on each operation example to obtain the characteristic attribute corresponding to each operation example; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
the dependency relationship unit is used for determining the dependency relationship between each job instance and other job instances;
the model unit is used for establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure;
and the analysis unit is used for analyzing the dependency model to obtain a job dependency graph of the batch job.
In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method for analyzing the dependency relationship of the batch job when executing the program.
In a fourth aspect, the present invention provides a computer readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the method for resolving dependency relationships of batch jobs.
According to the technical scheme, the invention provides the method and the device for analyzing the dependency relationship of the batch operation, and the characteristic attribute corresponding to each operation instance is obtained by performing abstract processing on each operation instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object; determining the dependency relationship between each job instance and other job instances; establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure; the dependency model is analyzed to obtain the operation dependency graph of the batch operation, the problem that the operation dependency graph is disordered and complex is effectively solved, the background batch operation dependency graph can be intuitively mastered, the system maintenance risk is reduced, and the problem analysis efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart illustrating a method for analyzing a dependency relationship of a batch job according to an embodiment of the present invention.
Fig. 2 is a flowchart illustrating steps 01 and 02 in the method for analyzing a dependency relationship of a batch job in the embodiment of the present invention.
FIG. 3 is a diagram of job dependencies in an embodiment of the present invention.
FIG. 4 is a diagram of a dependency model created based on an adjacency matrix in an embodiment of the present invention.
FIG. 5 is a diagram of a dependency model created based on an adjacency list in an embodiment of the present invention.
FIG. 6 is a diagram of a data structure for storing a dependency model in an embodiment of the present invention.
Fig. 7 is a logic diagram for searching the dependency relationship of the previous term based on the recursive traversal algorithm in the embodiment of the present invention.
FIG. 8 is a diagram illustrating dependency relationships of previous items of job 6 generated by taking FIG. 3 as an example in an embodiment of the present invention.
FIG. 9 is a logic diagram for finding dependencies between jobs in an embodiment of the present invention.
FIG. 10 is a logic diagram illustrating the example of FIG. 3 for generating the dependency relationship between job 2 and job 9 according to the embodiment of the present invention.
FIG. 11 is a diagram of the dependency relationship between job 2 and job 6 generated in the embodiment of the present invention.
FIG. 12 is a schematic structural diagram of an apparatus for analyzing dependency relationships of batch jobs according to an embodiment of the present invention.
Fig. 13 is a schematic structural diagram of an electronic device in an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention provides an embodiment of an analysis method of a dependency relationship of a batch job, which specifically includes the following contents with reference to fig. 1:
s101: abstracting each operation instance to obtain the characteristic attribute corresponding to each operation instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
in this step, the characteristic attributes include: at least one of job English name, job function Chinese description, job trigger type, job group name and timing time. In this embodiment, there are at least three characteristic attributes.
S102: determining the dependency relationship between each job instance and other job instances;
it should be noted that the object corresponding to each job instance corresponds to one point, and the dependency relationship corresponds to one line between two points.
S103: establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure;
in this step, the data structure includes: adjacency matrices and adjacency tables. When building the dependency model, one of the adjacency matrix and the adjacency list needs to be selected.
It should be noted that the selection is performed according to the dependency complexity of the actual scene and the time and space complexity of different data structure models. The dependency graph is typically a sparse graph. If the number of the operations is small and the dependency relationship changes frequently, the adjacency matrix can be selected and used, which is beneficial to the rapid establishment of the model. If the number of the operations is large and the dependency relationship is not changed frequently, the adjacency list can be selected to be used, and the storage space is saved.
It will be appreciated that the spatial complexity of the adjacency list is superior to that of the adjacency matrix. Since the operation dependency graph is normally a sparse graph, the use of the adjacency list can greatly reduce the memory usage. Based on actual working scenario considerations, the job dependency does not change frequently, and may change only once a month or a quarter, and therefore, it is more preferable to use an adjacency list-based mathematical model for modeling.
S104: and analyzing the dependency model to obtain a job dependency graph of the batch job.
In this step, an extent-first traversal algorithm is adopted to search the upper and lower dependency relationships of the object corresponding to each job instance in the dependency model, and a job dependency relationship graph is generated.
The breadth-first traversal algorithm is adopted to achieve the purpose of quickly searching the dependency relationship between the jobs by the user. For example: the user needs to obtain the upstream and downstream dependency relationship of a certain job: and inputting a job name, and respectively searching the upper layer dependency relationship and the lower layer dependency relationship of the model from the job node to finally generate a dependency relationship graph. For example: a user needs to check whether a direct or indirect dependency relationship exists between two jobs, inputs two job names, and the model automatically traverses from the job with a high hierarchy to the dependency relationship of a lower hierarchy for searching, and if the dependency relationship exists, a dependency link diagram between the two jobs is generated.
From the above description, the present invention provides an analysis method for dependency relationship of batch jobs, which obtains characteristic attributes corresponding to each job instance by performing abstraction processing on each job instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object; determining the dependency relationship between each job instance and other job instances; establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure; the dependency model is analyzed to obtain the operation dependency graph of the batch operation, the problem that the operation dependency graph is disordered and complex is effectively solved, the background batch operation dependency graph can be intuitively mastered, the system maintenance risk is reduced, and the problem analysis efficiency is improved.
In an embodiment of the present invention, referring to fig. 2, step S102 of the method for analyzing a dependency relationship of a batch job specifically includes:
s1021: determining an opening operation example in each operation example;
it should be noted that the opening operation may be performed manually, triggered manually, or performed at a timing point scheduled in advance.
S1022: determining the dependency relationship of a first operation example related to the opening operation example according to the operation execution sequence;
it should be noted that the first job instance related to the open job instance must wait for the open job instance to be executed before triggering, that is, the related job instance must wait for the previous dependent job to be executed before triggering automatically.
S1023: traversing dependencies of job instances associated with the first job instance based on a job execution order.
In this embodiment, step S102 of the method for analyzing the dependency relationship of the batch job is described as a specific example. Referring to fig. 3, job 1 is an opening job, independent of any other job; job 2 depends on job 1; job 3 depends on job 1; job 4 depends on job 1; job 5 depends on job 2; job 6 depends on job 2, job 3, and job 4; job 7 relies on job 4; job 8 relies on job 4; job 9 depends on jobs 5, 6, 7, and 8. Namely, determining that the open job instance is job 1, determining that the first job instance of job 1, job 2, job 3 and job 4;
determine the relevant job instances for job 2, job 3, and job 4, respectively; where job 2 corresponds to job 5 and job 6, job 3 corresponds to job 6, and job 4 corresponds to job 6, job 7 and job 8, and the traversal process is performed to determine the dependency relationship between each job instance and the other job instances.
In an embodiment of the present invention, an optional implementation manner of step S103 of providing the method for analyzing the dependency relationship of the batch job specifically includes:
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and the adjacency matrix, wherein the dependency model comprises the following steps:
(1) determining an adjacency matrix of the corresponding number of the operation instances;
(2) and setting element positions corresponding to two objects with dependency relationship in the adjacency matrix as dependency marks.
In this embodiment, an alternative implementation manner of step S103 of the method for analyzing the dependency relationship of the batch job is described as a specific example.
Referring to FIG. 4, a dependency model is created based on the adjacency matrix. Here, the number of job instances corresponding to 9 is explained.
The dependency relationship may exist between every two jobs, the dependency model is a 9 × 9 adjacency matrix, and the dependency relationship between jobs can be represented by this matrix. Defining the matrix as A, e.g., job 2 depends on job 1, then A [2 ]][1]When work 6 depends on work 2 and work 3, a [6 ]][2]=1、A[6][3]1. Obviously, the spatial complexity of the mathematical model created based on the adjacency matrix is O (V)2) And V is the total number of job instances. The time complexity for establishing the dependency model is O (E), and E is the total number of all the job dependencies in the whole graph.
In an embodiment of the present invention, another optional implementation manner of step S103 of providing the method for analyzing a dependency relationship of a batch job specifically includes:
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and the adjacency list;
(1) determining an adjacency list formed by the list heads of the corresponding number of the operation examples;
(2) storing the object corresponding to each operation instance to a header at one time;
(3) and storing the operation examples corresponding to the table headers into a unidirectional chain structure or a tree structure in the adjacent table according to the dependency relationship of the operation examples.
In this embodiment, an alternative implementation manner of step S103 of the method for analyzing the dependency relationship of the batch job is described as a specific example.
Referring to FIG. 5, a dependency model is created based on an adjacency list. Here, the number of job instances corresponding to 9 is explained.
The dependency model is a structure combining sequential allocation and chain allocation. And if the table head node has the corresponding antecedent dependent node, sequentially storing the antecedent dependent node in the unidirectional linked list or the tree structure pointed by the table head node. Dependencies between jobs can be represented by this adjacency list. For example, job 3 depends on job 1, the linked list to which the node of job 3 points includes job 1, job 6 depends on jobs 2 and 3, and the linked list or tree structure to which the node of job 6 points includes jobs 2 and 3. As can be seen from the analysis, the spatial complexity of the mathematical model created based on the adjacency list is O (V + E), the temporal complexity for building the mathematical model is O (E V), V is the total number of job instances, and E is the total number of job dependencies in the whole graph.
Referring to FIG. 6, a data structure diagram for storing a dependency model in a computer system is shown. A one-dimensional array composed of tree structures is used for storing the mathematical model. For example, job 9 is preceded by job 5, job 6, job 7, and job 8, and in this data structure, the tree structure to which job 9 points is the tree structure formed by job 5, job 6, and job 7, and so on. Similarly, if the dependency of job 1 depends on job 2, job 3, and job 4, the tree structure pointed by job 1 is the tree structure of job 2, job 3, and job 4.
To further explain the scheme, the detailed description is given to the operation dependency graph generated by searching the upper and lower dependency relationships of the object corresponding to each operation instance in the dependency model by using the breadth-first traversal algorithm.
FIG. 7 illustrates a logic diagram for searching a dependency relationship of a pre-job item based on a recursive traversal algorithm according to the present invention. The same principle is used for searching the dependency relationship logic of a certain operation back item and searching the dependency relationship logic of a front item, and the difference is that when a mathematical model is constructed, a back item dependency chain table or a back item dependency operation tree stored in a chain table or a tree structure pointed by an operation node is searched. The two search algorithms can be operated simultaneously, the result sets are merged, and the search of the dependency relationship of the front items and the dependency relationship of the back items of a certain operation is realized.
The following explains the search algorithm by taking the previous item dependency relationship search process as an example:
step 1: starting from a target node, finding a direct antecedent dependence tree structure pointed by the target node;
step 2: traversing all direct predecessor dependent nodes;
and step 3: and finding a antecedent dependency tree structure pointed by the direct antecedent dependency node, and recursively inquiring the indirect dependency node by analogy until the indirect dependency node has no antecedent dependency or reaches the preset maximum recursive depth, and finishing the search.
For example, look up the dependency graph of the previous item for job 6:
step 1: acquiring a previous item dependency tree structure of the operation 6;
step 2: the first node of the previous item dependency tree structure of the traversal job 6 is job 2;
and step 3: the former item of the recursive query operation 2 depends on the tree structure;
and 4, step 4: traversing the tree structure of the former item of the operation 2, wherein only one node is the operation 1;
and 5: the second node of the previous item dependency tree structure of the traversal job 6 is job 3;
step 6: recursively inquiring the previous item dependence tree structure of the job 3;
and 7: traversing the tree structure of the former item of the operation 3, wherein only one node is the operation 1;
and 8: traversing the third node of the previous item dependency tree structure of the job 6 as a job 4;
and step 9: the former item of the recursive query operation 4 depends on the tree structure;
step 10: traversing the tree structure of the former item of the operation 4, wherein only one node is the operation 1;
step 11: traversing the previous item dependency tree structure of the operation 6, finding no nodes which are not traversed, ending the traversal, and generating a previous item dependency relationship graph of the operation 6 in the figure 8.
FIG. 9 is a logic diagram for implementing a depth-first search algorithm to find a dependency relationship between two jobs according to the present invention. The following explains the logic of the dependency graph between the search job N and the job M, and by using the data structure-stack first-in first-out characteristic, the logic is as follows:
step 1: stacking the operation N object, and marking the operation N as accessed;
step 2: acquiring a consequent dependency tree structure of the operation N;
and step 3: traversing the back item of the operation N and depending on the tree structure, if the traversed operation object is not accessed, stacking the object and marking the object as accessed;
and 4, step 4: acquiring a stack top object, and judging whether the stack top object is equal to the operation M:
(1) if the stack top object is equal to the operation M, finding an operation dependency relationship path, recording the path, popping the stack top object, and marking the popped object as unaccessed;
(2) and if the stack top object is not equal to the operation M, acquiring a back item dependent tree structure of the stack top object. If the dependency tree structure of the latter item of the object is empty, the stack top object is popped; if the dependency tree structure of the latter item of the object is not empty, traversing the dependency tree structure of the latter item, stacking the traversed job pair with the first state as unaccessed, marking the job pair as accessed, and popping the top object of the stack if no unaccessed job object exists;
and 5: and 4, circulating the step 4 until no operation object exists in the stack.
FIG. 10 is a flowchart for searching for dependencies between job 2 and job 9 based on the job dependencies of FIG. 3.
Step 1: stack job 2 and mark as visited;
step 2: traversing the corresponding back item dependency relationship of the operation 2, finding the operation 5, and if the operation 5 is not accessed, stacking the operation 5 and marking the operation as accessed;
and step 3: acquiring a stack top object as operation 5;
and 4, step 4: traversing the corresponding back item dependency relationship of the operation 5, finding the operation 9, and if the operation 9 is not accessed, stacking the operation 9 and marking the operation as accessed;
and 5: at this point, the top object equals job 9, recording the first dependency path: operation 2, operation 5, operation 9;
step 6: pop job 9 off stack and mark as not visited;
and 7: acquiring a stack top object as operation 5;
and 8: a back item dependency relationship corresponding to the operation 5 is not provided with a back item dependency object which is not traversed, and the operation 5 is popped;
and step 9: acquiring a stack top object as operation 2;
step 10: traversing the corresponding back dependency relationship of the operation 2, finding the operation 6, and if the operation 6 is not accessed, stacking the operation 6 and marking the operation as accessed;
step 12: acquiring a stack top object as operation 6;
step 13: traversing the back item dependency corresponding to the operation 6, finding the operation 9, and if the operation 9 is not accessed, stacking the operation 9 and marking the operation as accessed;
step 14: at this point, the top object equals job 9, recording the second dependency path: operation 2, operation 6, operation 9;
step 15: pop job 9 off stack and mark as not visited;
step 16: acquiring a stack top object as operation 6;
and step 17: acquiring a back item dependency relationship corresponding to the operation 6, and popping the operation 6 without an unexplored back item dependency object;
step 18: acquiring a stack top object as operation 2;
step 19: acquiring a back item dependency relationship corresponding to the operation 2, and popping the operation 2 without an unexplored back item dependency object;
step 20: at this time, no job object exists in the stack, the search is finished, and two inter-job dependency relationship paths are generated. As shown in fig. 11.
An embodiment of the present invention provides a specific implementation manner of an analysis apparatus for batch job dependency relationships, which is capable of implementing the batch job dependency relationships of all contents in the analysis method for batch job dependency relationships, and referring to fig. 12, the analysis apparatus for batch job dependency relationships specifically includes the following contents:
the abstract unit 10 is configured to perform abstract processing on each job instance to obtain a feature attribute corresponding to each job instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
a dependency unit 20, configured to determine a dependency of each job instance with other job instances;
the model unit 30 is used for establishing a dependency model according to the dependency relationship of each job instance, the object corresponding to each job instance and a preset data structure;
and the analysis unit 40 is used for analyzing the dependency model to obtain a job dependency graph of the batch job.
The embodiment of the apparatus for analyzing a dependency relationship of a batch job provided by the present invention may be specifically used to execute the processing procedure of the embodiment of the method for analyzing a dependency relationship of a batch job in the foregoing embodiment, and the functions of the apparatus are not described herein again, and reference may be made to the detailed description of the embodiment of the method.
As can be seen from the above description, the apparatus for analyzing the dependency relationship of batch jobs according to the embodiment of the present invention obtains the feature attributes corresponding to each job instance by performing abstraction processing on each job instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object; determining the dependency relationship between each job instance and other job instances; establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure; the dependency model is analyzed to obtain the operation dependency graph of the batch operation, the problem that the operation dependency graph is disordered and complex is effectively solved, the background batch operation dependency graph can be intuitively mastered, the system maintenance risk is reduced, and the problem analysis efficiency is improved.
The application provides an embodiment of an electronic device for implementing all or part of contents in the analysis method for the dependency relationship of batch jobs, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between related devices; the electronic device may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the electronic device may refer to an embodiment of the method for analyzing the dependency relationship of the batch job and an embodiment of the apparatus for analyzing the dependency relationship of the batch job in this embodiment, which are incorporated herein, and repeated details are not repeated.
Fig. 13 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 13, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 13 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the function of resolving batch job dependencies may be integrated into central processor 9100. The central processor 9100 may be configured to control as follows:
abstracting each operation instance to obtain the characteristic attribute corresponding to each operation instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
determining the dependency relationship between each job instance and other job instances;
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure;
and analyzing the dependency model to obtain a job dependency graph of the batch job.
As can be seen from the above description, in the electronic device provided in the embodiment of the present application, by performing abstraction processing on each job instance, a feature attribute corresponding to each job instance is obtained; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object; determining the dependency relationship between each job instance and other job instances; establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure; the dependency model is analyzed to obtain the operation dependency graph of the batch operation, the problem that the operation dependency graph is disordered and complex is effectively solved, the background batch operation dependency graph can be intuitively mastered, the system maintenance risk is reduced, and the problem analysis efficiency is improved.
In another embodiment, the apparatus for analyzing the dependency relationship of the batch job may be configured separately from the central processor 9100, for example, the apparatus may be configured as a chip connected to the central processor 9100, and the function of analyzing the dependency relationship of the batch job may be realized by the control of the central processor.
As shown in fig. 13, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 13; in addition, the electronic device 9600 may further include components not shown in fig. 13, which can be referred to in the prior art.
As shown in fig. 13, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
An embodiment of the present invention further provides a computer-readable storage medium capable of implementing all the steps in the method for analyzing dependency relationships of batch jobs in the above embodiment, where the computer-readable storage medium stores a computer program, and the computer program implements all the steps of the method for analyzing dependency relationships of batch jobs in the above embodiment when executed by a processor, for example, the processor implements the following steps when executing the computer program:
abstracting each operation instance to obtain the characteristic attribute corresponding to each operation instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
determining the dependency relationship between each job instance and other job instances;
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure;
and analyzing the dependency model to obtain a job dependency graph of the batch job.
As can be seen from the above description, in the computer-readable storage medium provided in the embodiment of the present invention, by performing abstraction processing on each job instance, a feature attribute corresponding to each job instance is obtained; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object; determining the dependency relationship between each job instance and other job instances; establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure; the dependency model is analyzed to obtain the operation dependency graph of the batch operation, the problem that the operation dependency graph is disordered and complex is effectively solved, the background batch operation dependency graph can be intuitively mastered, the system maintenance risk is reduced, and the problem analysis efficiency is improved.
Although the present invention provides method steps as described in the examples or flowcharts, more or fewer steps may be included based on routine or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual apparatus or client product executes, it may execute sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the embodiments or methods shown in the figures.
As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, apparatus (system) or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention is not limited to any single aspect, nor is it limited to any single embodiment, nor is it limited to any combination and/or permutation of these aspects and/or embodiments. Moreover, each aspect and/or embodiment of the present invention may be utilized alone or in combination with one or more other aspects and/or embodiments thereof.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present invention, and they should be construed as being included in the following claims and description.

Claims (10)

1. A method for analyzing a dependency relationship of a batch job is characterized by comprising the following steps:
abstracting each operation instance to obtain the characteristic attribute corresponding to each operation instance; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
determining the dependency relationship between each job instance and other job instances;
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure;
and analyzing the dependency model to obtain a job dependency graph of the batch job.
2. The method for resolving the dependency of the batch jobs according to claim 1, wherein the feature attributes include: at least one of job English name, job function Chinese description, job trigger type, job group name and timing time.
3. The method for resolving the dependency relationship of the batch jobs according to claim 1, wherein the determining the dependency relationship of each job instance with other job instances comprises:
determining an opening operation example in each operation example;
determining the dependency relationship of a first operation example related to the opening operation example according to the operation execution sequence;
traversing dependencies of job instances associated with the first job instance based on a job execution order.
4. The method for resolving batch job dependencies of claim 1, wherein the data structure comprises: adjacency matrices and adjacency tables.
5. The method for analyzing the dependency relationship of the batch jobs according to claim 4, wherein the establishing of the dependency model according to the dependency relationship of each job instance, the object corresponding to each job instance and the preset data structure comprises:
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and the adjacency matrix, wherein the dependency model comprises the following steps:
determining an adjacency matrix of the corresponding number of the operation instances;
and setting element positions corresponding to two objects with dependency relationship in the adjacency matrix as dependency marks.
6. The method for analyzing the dependency relationship of the batch jobs according to claim 4, wherein the establishing of the dependency model according to the dependency relationship of each job instance, the object corresponding to each job instance and the preset data structure comprises:
establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and the adjacency list;
determining an adjacency list formed by the list heads of the corresponding number of the operation examples;
storing the object corresponding to each operation instance to a header at one time;
and storing the operation examples corresponding to the table headers into a unidirectional chain structure or a tree structure in the adjacent table according to the dependency relationship of the operation examples.
7. The method for analyzing the dependency relationship of the batch jobs according to claim 1, wherein analyzing the dependency model to obtain the job dependency relationship graph of the batch jobs comprises:
and searching the upper layer and lower layer dependency relationship of the object corresponding to each operation instance in the dependency model by adopting a breadth-first traversal algorithm to generate an operation dependency relationship graph.
8. An apparatus for analyzing a dependency relationship of a batch job, comprising:
the abstract unit is used for carrying out abstract processing on each operation example to obtain the characteristic attribute corresponding to each operation example; each job instance corresponds to a plurality of characteristic attributes, and the plurality of characteristic attributes form an object;
the dependency relationship unit is used for determining the dependency relationship between each job instance and other job instances;
the model unit is used for establishing a dependency model according to the dependency relationship of each operation instance, the object corresponding to each operation instance and a preset data structure;
and the analysis unit is used for analyzing the dependency model to obtain a job dependency graph of the batch job.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method for batch job dependency resolution of any one of claims 1 to 7 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of resolving batch job dependencies of any one of claims 1 to 7.
CN202010820319.4A 2020-08-14 2020-08-14 Method and device for analyzing dependency relationship of batch operation Pending CN111949832A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010820319.4A CN111949832A (en) 2020-08-14 2020-08-14 Method and device for analyzing dependency relationship of batch operation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010820319.4A CN111949832A (en) 2020-08-14 2020-08-14 Method and device for analyzing dependency relationship of batch operation

Publications (1)

Publication Number Publication Date
CN111949832A true CN111949832A (en) 2020-11-17

Family

ID=73342650

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010820319.4A Pending CN111949832A (en) 2020-08-14 2020-08-14 Method and device for analyzing dependency relationship of batch operation

Country Status (1)

Country Link
CN (1) CN111949832A (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112596878A (en) * 2020-12-24 2021-04-02 上海艾融软件股份有限公司 Batch processing method and device and electronic equipment
CN112597442A (en) * 2020-12-30 2021-04-02 南方电网数字电网研究院有限公司 Distributed-based electric power settlement calculation method and system
CN112837026A (en) * 2021-01-22 2021-05-25 山东浪潮通软信息科技有限公司 Method, device and equipment for analyzing delivery cycle dependence and storage medium
CN113886839A (en) * 2021-10-21 2022-01-04 中国工商银行股份有限公司 Method and device for identifying risk problems in batch job dependencies
CN114035769A (en) * 2021-10-20 2022-02-11 北京金山云网络技术有限公司 Job flow control method, device, electronic device and storage medium
JP2022094945A (en) * 2020-12-15 2022-06-27 インターナショナル・ビジネス・マシーンズ・コーポレーション Computer implementation method, system and computer program (optimization of batch job scheduling)
US20220334933A1 (en) * 2021-04-16 2022-10-20 International Business Machines Corporation Failover management for batch jobs
CN115269648A (en) * 2022-07-07 2022-11-01 浙江恒生长运网络科技有限公司 Work shift query management and control method for extracting work shift information in batches

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106293928A (en) * 2015-06-05 2017-01-04 阿里巴巴集团控股有限公司 A kind of overall situation task node dependence method for visualizing, device and system
CN107590592A (en) * 2017-08-31 2018-01-16 中国建设银行股份有限公司 Job dependence relation method for expressing, operation displaying and dispatch control method and device

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106293928A (en) * 2015-06-05 2017-01-04 阿里巴巴集团控股有限公司 A kind of overall situation task node dependence method for visualizing, device and system
CN107590592A (en) * 2017-08-31 2018-01-16 中国建设银行股份有限公司 Job dependence relation method for expressing, operation displaying and dispatch control method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
孙涵 等: "《数据结构 抽象建模、实现与应用》", 30 April 2020, 机械工业出版社, pages: 99 *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2022094945A (en) * 2020-12-15 2022-06-27 インターナショナル・ビジネス・マシーンズ・コーポレーション Computer implementation method, system and computer program (optimization of batch job scheduling)
CN112596878A (en) * 2020-12-24 2021-04-02 上海艾融软件股份有限公司 Batch processing method and device and electronic equipment
CN112597442A (en) * 2020-12-30 2021-04-02 南方电网数字电网研究院有限公司 Distributed-based electric power settlement calculation method and system
CN112837026A (en) * 2021-01-22 2021-05-25 山东浪潮通软信息科技有限公司 Method, device and equipment for analyzing delivery cycle dependence and storage medium
US20220334933A1 (en) * 2021-04-16 2022-10-20 International Business Machines Corporation Failover management for batch jobs
US11556425B2 (en) * 2021-04-16 2023-01-17 International Business Machines Corporation Failover management for batch jobs
CN114035769A (en) * 2021-10-20 2022-02-11 北京金山云网络技术有限公司 Job flow control method, device, electronic device and storage medium
CN113886839A (en) * 2021-10-21 2022-01-04 中国工商银行股份有限公司 Method and device for identifying risk problems in batch job dependencies
CN113886839B (en) * 2021-10-21 2025-02-28 中国工商银行股份有限公司 Method and device for identifying risk issues in batch job dependencies
CN115269648A (en) * 2022-07-07 2022-11-01 浙江恒生长运网络科技有限公司 Work shift query management and control method for extracting work shift information in batches
CN115269648B (en) * 2022-07-07 2023-08-18 浙江恒生长运网络科技有限公司 Shift inquiry control method for batch extracting shift information

Similar Documents

Publication Publication Date Title
CN111949832A (en) Method and device for analyzing dependency relationship of batch operation
US11741396B1 (en) Efficient command execution using aggregated compute units
US12423309B2 (en) Dynamic query processor for streaming and batch queries
US11782987B1 (en) Using an augmented process model to track process instances
US11567959B2 (en) Self-contained files for generating a visualization of query results
US10997192B2 (en) Data source correlation user interface
US11003682B2 (en) Metrics analysis workflow
US11188600B2 (en) Facilitating metric forecasting via a graphical user interface
US11960545B1 (en) Retrieving event records from a field searchable data store using references values in inverted indexes
US20200342068A1 (en) Two-tier capacity planning
US11841834B2 (en) Method and apparatus for efficient synchronization of search heads in a cluster using digests
KR101535813B1 (en) System and method for dynamic updating of event composition rule for complex event processing
US11729074B1 (en) Online data decomposition
US20180101622A1 (en) Perform graph traversal with graph query language
US12079233B1 (en) Multiple seasonality online data decomposition
JP5108011B2 (en) System, method, and computer program for reducing message flow between bus-connected consumers and producers
CN110222046B (en) List data processing method, device, server and storage medium
CN113296847A (en) Workflow engine implementation method based on self-analysis node
CN117312307A (en) Service data processing method, device, computer equipment and storage medium
CN111813769B (en) Data processing methods and devices
CN115878116A (en) Development method, terminal and storage medium
CN117971378A (en) Workflow execution method, workflow execution device, electronic equipment and storage medium
CN112035514A (en) Database statement processing method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20201117