CN114428608A - Task optimization method based on big data and related equipment - Google Patents

Abstract

The embodiment of the application belongs to the field of artificial intelligence and big data, and relates to a task optimization method based on big data, which comprises the steps of obtaining a task dependency set of each target task; classifying the target tasks to obtain a task overlap ratio set formed by the target tasks; calculating the contact ratio of any two target tasks in the task contact ratio set; obtaining the reuse degree; and when the reusability of the target tasks is low and the overlap ratio of the two target tasks is high, combining the two target tasks into the same target task. The application also provides a device, computer equipment and a storage medium. In addition, the application also relates to a block chain technology, and a task dependent set, a repetition dependent set, a coincidence degree and a reusability can be stored in the block chain. According to the method and the device, the target task pairs with low reuse degree and high coincidence degree are screened out, the target tasks are merged or replaced, chimney development is reduced, and the beneficial effect of optimizing task development is achieved.

Description

Task optimization method and related equipment based on big data

technical field

The present application relates to the technical field of artificial intelligence and big data, and in particular, to a task optimization method, device, computer equipment and storage medium based on big data.

Background technique

Data assets are the most important foundation for digital transformation of enterprises. With the rapid development of enterprises, the scale of data is unprecedented. In order to quickly meet business needs, the development process is often accompanied by the generation of a large number of chimneys. The application data between different business lines is fragmented, further increasing the number of chimney developments.

In the prior art, redundant tasks are manually sorted out from time to time, and tasks related to the same link are analyzed in the process of offline tasks, and there is a risk of missing; and the integration of data links, due to historical development Links with dozens of layers are often difficult to analyze and difficult to integrate. In addition, reducing the development of chimneys and improving task reuse requires that the existing data system must be strictly controlled in the requirements design stage, the optimal design scheme should be selected, and the requirements should be designed. Personnel requirements are high.

Therefore, chimney-style development results in a large number of repeated calculations and development in the system, which reduces R&D efficiency, leads to waste of computing and storage resources, and increases labor and computing costs.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to propose a task optimization method based on big data, so as to solve the technical problem of a large number of repeated calculations and development in the system caused by chimney development, waste of computing and storage resources, and increase of manpower.

In order to solve the above technical problems, the embodiment of the present application provides a task optimization method based on big data, which adopts the following technical solutions:

When receiving the optimization task instruction, obtain the task dependency set of each target task;

Classify each of the target tasks according to the optimization task instruction, and obtain a task coincidence degree set composed of the target tasks;

Calculate the coincidence degree of any two target tasks in the task coincidence degree set;

Obtain the degree of reuse of each target task in the set of task coincidence degrees;

If the reuse degree of the target task in the task coincidence degree set is lower than the preset reuse degree and the coincidence degree of the target task and another target task is higher than the preset coincidence degree, the target task is merged into the target task. another objective task.

In order to solve the above technical problems, the embodiment of the present application also provides a task optimization device based on big data, including:

The receiving module is used to obtain the task dependency set of each target task when receiving the optimization task instruction;

A classification module, configured to classify each of the target tasks according to the optimization task instruction, and obtain a task coincidence degree set composed of the target tasks;

A coincidence degree calculation module, used for calculating the coincidence degree of any two target tasks in the task coincidence degree set;

a reusability calculation module, used to obtain the reusability of each target task in the task coincidence degree set;

The merging module is configured to, if the degree of reuse of the target task in the set of task coincidence degrees is lower than the preset degree of reuse and the degree of coincidence between the target task and another target task is higher than the preset degree of coincidence, The target task is merged with the other target task.

In order to solve the above technical problem, the embodiment of the present application also provides a computer device, including:

A memory and a processor, where computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the steps of the above-mentioned big data-based task optimization method are implemented.

In order to solve the above technical problems, the embodiments of the present application also provide a computer-readable storage medium, including:

The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, implements the steps of the above-mentioned big data-based task optimization method.

Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

By finding the target task with low degree of reuse, and according to the degree of dependency overlap between the two target tasks, screen out the target task pair with low degree of reuse and high degree of overlap, and merge or replace the target tasks. The demand design prioritizes the use of tables with high reusability as the base. At the same time, the tasks after the reference design overlap with the existing tasks to reduce the development of chimneys and achieve the beneficial effect of optimizing task development.

Description of drawings

In order to illustrate the solutions in the present application more clearly, the following will briefly introduce the accompanying drawings used in the description of the embodiments of the present application. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is an exemplary system architecture diagram to which the present application can be applied;

2 is a flow chart of an embodiment of a task optimization method based on big data according to the present application;

3 is a schematic structural diagram of an embodiment of a task optimization device based on big data according to the present application;

FIG. 4 is a schematic structural diagram of an embodiment of a computer device according to the present application.

Detailed ways

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field of this application; the terms used herein in the specification of the application are for the purpose of describing specific embodiments only It is not intended to limit the application; the terms "comprising" and "having" and any variations thereof in the description and claims of this application and the above description of the drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the description and claims of the present application or the above drawings are used to distinguish different objects, rather than to describe a specific order.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings.

As shown in FIG. 1 , the system architecture 100 may include terminal devices 101 , 102 , and 103 , a network 104 and a server 105 . The network 104 is a medium used to provide a communication link between the terminal devices 101 , 102 , 103 and the server 105 . The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 101 , 102 and 103 , such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 101, 102, and 103 can be various electronic devices that have a display screen and support web browsing, including but not limited to smart phones, tablet computers, e-book readers, and MP3 players (Moving Picture E task optimization based on big data). perts GroupAudio Layer III, Motion Picture Expert Compression Standard Audio Layer 3), MP4 (Moving Picture E Big Data-Based Task Optimization perts GroupAudio Layer IV, Motion Picture Expert Compression Standard Audio Layer 4) Players, Laptops and Desktops computer, etc.

The server 105 may be a server that provides various services, such as a background server that provides support for the pages displayed on the terminal devices 101 , 102 , and 103 .

The server can be an independent server, or it can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and Content Delivery Network (CDN) , and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

It should be noted that the big data-based task optimization method provided by the embodiments of the present application is generally performed by a server/terminal device, and accordingly, the big data-based task optimization apparatus is generally set in the server/terminal device.

It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Continuing to refer to FIG. 2 , a flowchart of one embodiment of a method for task optimization based on big data according to the present application is shown. The method for task optimization based on big data includes the following steps:

Step S201, when receiving the optimization task instruction, obtain the task dependency set of each target task;

In this embodiment, the electronic device (for example, the server/terminal device shown in FIG. 1 ) on which the big data-based task optimization method runs can receive the optimization instruction sent by the client through a wired connection or a wireless connection. It should be pointed out that the above wireless connection methods may include but are not limited to 3G/4G connection, WiFi connection, Bluetooth connection, WiMA connection based on big data task optimization, Zigbee connection, UWB (ultra wideband) connection, and other known or Future development of wireless connectivity.

In this embodiment, the optimization task instruction may be triggered by human operation, or may be triggered by setting an alarm clock, or may be triggered according to the operating load of the system, so as to optimize the execution of tasks in the system. The optimization task instruction may contain the requirements of the optimization task, such as the scope of the target task, the optimization level, and so on. Task dependency means that a task a needs to be executed after another task b is completed, then task a depends on task b. The task dependency set refers to all upstream tasks that the final executed task a depends on. For example, task a depends on task b and task c, task b depends on task d and task e, and task c depends on task f, then the set {b ,c,d,e,f} is the set of all upstream dependencies of task a; the set {b,c} is called the set of direct upstream dependencies of task a; the set {b} is the set of direct downstream dependencies of d; the set {a ,b} is the set of all downstream dependencies of d.

In an optional embodiment, the refinement step of step S201 includes:

Step S2011, obtaining the configuration parameters of each of the target tasks from the task configuration system;

Step S2012, obtaining the upstream dependent tasks of each of the target tasks according to the configuration parameters;

Step S2013, obtaining the task dependency set according to the upstream dependent task.

The configuration parameters of each target task are stored in a task configuration system such as the linkdo task scheduling system, and the configuration parameters of each target task are obtained by calling the task configuration system. The configuration parameters include the tasks that the target task can execute after those tasks are completed. These tasks are the upstream dependent tasks of the target task, and the set of all upstream dependent tasks of a target task is the task dependency set of the target task.

By calling the configuration parameters in the task configuration system, trace all the upstream dependent tasks of the target task, integrate all the upstream dependent tasks, get the task dependency set of the target task, and accurately obtain the upstream dependent tasks according to the hierarchical relationship of the target task, so as to achieve fast and precise Get the intended effect of the target-dependent collection.

In yet another optional embodiment, the refinement step of step S201 further includes:

Step S2014, obtaining the operation log of each target task;

Step S2015, searching for the execution code of the target task in the running log;

Step S2016: Integrate the upstream dependent tasks of each of the target tasks in the source table according to the source table of the execution code of the target task to obtain the task dependency set.

In order to reduce the running pressure, a running log containing the target task within a certain period can be set, and the execution code of the target task can be recorded in the running log. Different target tasks can correspond to multiple running logs, and different target tasks can appear in the same running log. . In this embodiment, after obtaining the task dependency set of one target task, the task dependency set of the next target task is obtained. For example, if the target task is a, find the execution code of the target task a in the running log, trace the source table of the execution code according to the execution code, find the upstream execution code of the execution code, convert the upstream execution code into the upstream dependent task according to the upstream execution code, and convert the target task All upstream dependent tasks of a are integrated together to obtain a task dependency set.

In this embodiment, all upstream dependent tasks of the target task are traced back to all upstream dependent tasks of the target task through the running log of the target task and according to the execution code recorded in the running log, to obtain a task dependency set composed of all upstream dependent tasks.

Step S202, classify each of the target tasks according to the optimization task instruction, and obtain a task coincidence degree set composed of the target tasks;

Analyze the optimization task requirements in the optimization task instructions, wherein the optimization task instructions include classification by subject, data hierarchy (ODS, DWD, DWS, DMD, DMS, DMI) classification or cluster classification. Classify the target tasks according to the optimization task instructions, and obtain a task coincidence degree set, wherein the task coincidence degree set refers to classifying the target tasks according to different classification standards, and obtaining tasks with the same topic, the same data level or close clustering coincidence degree gather.

In an optional embodiment, the refinement step of step S202 includes:

Step S2021, parsing the optimization task instruction, wherein the optimization task instruction includes requirements classified according to the target data level;

Step S2022, acquiring target tasks whose level is the target data level, and generating the task coincidence degree set.

By parsing the optimization task instructions including the requirements for classification according to the target data level, the target data level to be classified is determined, the data level of each target task is obtained, the target task whose level is the target data level is obtained, and the task coincidence degree set is generated. The coincidence degree set is composed of target tasks whose level is the target data level.

In this embodiment, the target tasks are classified by means of the data link layer, which is beneficial for each department in the enterprise to optimize the department's tasks and achieve local optimization. The task coincidence degree set is obtained through fragmented processing, which improves the search efficiency. Efficiency of chimney development.

Step S203, calculating the coincidence degree of any two target tasks in the task coincidence degree set;

According to the classification requirements, a task coincidence degree set composed of target tasks is obtained, for example, the subject is the same, the data level is the same, or the cluster coincidence degree is similar. It is understandable that the target tasks of different classifications will not have repeated uses. In this case, the target tasks in the task-dependent set may be repeatedly developed. In order to optimize the task link, the tasks in the same classification are required. Perform iterative summary to get the task coincidence degree set. Obtain any two target tasks in the task dependency set, and determine the degree of coincidence between the two target tasks according to the task dependency sets corresponding to the two target tasks. The degree of overlap reflects the overlap rate between the upstream dependent tasks of the two target tasks. The higher the overlap rate, the higher the degree of overlap between the two target tasks and the higher the degree of repetitive development of tasks.

In an optional embodiment, the refinement step of step S203 includes:

Step S2031, acquiring any two target tasks in the task coincidence degree set;

Step S2032, respectively obtaining the number of elements of the respective task dependency sets of the two target tasks, the number of union elements and the number of intersection elements of the task dependency sets of the two target tasks;

Step S2033, according to the number of elements of the respective task dependency sets of the two target tasks, the number of union elements and the number of intersection elements of the task dependency sets of the two target tasks, calculate any two in the task coincidence degree set. The degree of coincidence of the target tasks.

Each time two target tasks in the task coincidence degree set are selected, step S2032 and step S2033 are performed. If the task coincidence degree set includes n target tasks, execute C _n ² times, that is, n(n-1)/2 times. If the selected target tasks are a and b, respectively, obtain the task dependency set A of target task a, obtain the task dependency set of target task b, obtain the number of elements n(A) of the task dependency set A of target task a, and the target The number of elements of task-dependent set B of task b is n(B). According to task-dependent sets A and B, the intersection and union of A and B can be obtained, and the number of intersection elements n(A∩B) and union elements can be obtained. Number n(A∪B).

According to n(A), n(B), n(A∩B) and n(A∪B), the coincidence degree of target task a and target task b is obtained. Coincidence is used to measure the overlap rate of upstream dependent tasks on which target task a and target task b depend, reflecting the redundancy rate between tasks, and is an important optimization indicator.

In this embodiment, the task dependency sets of the two target tasks are obtained, and the number of elements of the task dependency sets of the two target tasks, the number of intersection elements and the number of unions of the task dependency sets of the two target tasks are obtained respectively. , get the coincidence degree, the calculation process is simple, easy to operate, and can accurately reflect the coincidence degree between the two target tasks.

Further, the calculation formula of step S2033 is as follows:

或

or

或

or

Among them, S is the degree of coincidence; A is the task dependency set of the first target task, and the first target task is any one of the two target tasks; B is the other target task of the two target tasks The task dependent set of ; n(A∩B) the number of elements in the intersection of set A and set B; n(A∪B) is the number of elements in the union of set A and set B; n(A) is set A The number of elements of ; n(B) is the number of elements of set B.

Step S204, obtaining the reuse degree of each target task in the task coincidence degree set;

The degree of reuse refers to the number of elements contained in the direct downstream dependency set of the target task, which can represent how many downstream tables the task supports. It can be used to measure the quality of the design and reflect the sharing of data models. The higher the degree of reuse, the higher the degree of sharing, and the better the design. The set of direct downstream dependencies is the set of dependent target tasks. If the target task A can only be executed in A1 and A2 after the execution of the target task A is completed, the direct downstream dependency set of the target task A is {A1, A2}, and the reuse degree of the target task A is 2.

In an optional embodiment, the refinement step of step S204 includes:

Step S2041, obtaining the direct downstream dependency sets of each of the target tasks;

Step S2042: Determine the multiplexing degree of each of the target tasks according to the number of elements in the direct downstream dependency set.

By calling the configuration parameters of the target task in the task configuration system or the code that calls the target task recorded in the task operation log, according to all the calling relationships contained in the code, through task matching, the direct downstream tasks that depend on the target task can be obtained. All direct downstream tasks of the target task constitute the set of direct downstream dependencies. The direct downstream dependency set consists of the direct downstream tasks of the target task, and the number of elements is the number of direct downstream tasks, that is, the number of elements in the direct downstream dependency set is the reuse degree of the target task. The degree of reuse is used to measure the pros and cons of task design, reflecting the sharing of data models.

By obtaining the degree of overlap between the two target tasks, it is used to measure the dependency between the two target tasks, and the overlap rate between the dependent upstream tasks reflects the degree of redundancy, and the degree of reuse of the target task is directly obtained by obtaining The downstream dependency set reflects the data model sharing between target tasks, and screen out target tasks with high overlap rate and poor sharing from two perspectives, effectively reducing the beneficial effect of chimney development.

Step S205, if the degree of reuse of the target task in the set of task coincidence degrees is lower than the preset degree of reuse and the degree of coincidence of the target task and another target task is higher than the preset degree of coincidence, the target task is merged with the other target task.

The preset multiplexing degree and the preset overlapping degree can be set by the designer according to the operating conditions of the system, and can be adjusted according to the types of different target tasks. For example, the preset multiplexing degree is 3, and the preset overlapping degree is 90%. When the reuse degree of the target task is lower than the preset reuse degree and the coincidence degree between the target task and another target task is higher than the preset coincidence degree, it indicates the dependent task between the target task and another target task There is a high degree of overlap, the target task is being developed repeatedly with another target task, and the target task belongs to the chimney development. In order to optimize the redundant tasks and improve the overall operating efficiency of the system, the target task is merged into another target task with a high degree of coincidence.

In this embodiment, a target task with a low degree of reuse is found, and according to the degree of overlap of dependencies between the two target tasks, a pair of target tasks with a low degree of reuse and an excessively high degree of overlap are screened out, and the target task is evaluated. Combined or replaced, the follow-up requirement design preferentially uses the table with high reusability as the base. At the same time, the dependence of the task after the reference design and the existing task overlap, reduce the development of the chimney, and achieve the beneficial effect of optimizing the task development.

It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned candidate lexicon, the above-mentioned task dependency set, dependency repetition set, coincidence degree, and reuse degree can also be stored in a node of a blockchain.

The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

The embodiments of the present application may acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. .

The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

The present application may be used in numerous general purpose or special purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, including A distributed computing environment for any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. , when the program is executed, it may include the processes of the foregoing method embodiments. The aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM).

It should be understood that although the various steps in the flowchart of the accompanying drawings are sequentially shown in the order indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order and may be performed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution sequence is also It does not have to be performed sequentially, but may be performed alternately or alternately with other steps or at least a portion of sub-steps or stages of other steps.

The embodiments of the present application may acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. .

The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

Further referring to FIG. 3 , as an implementation of the method shown in FIG. 2 above, the present application provides an embodiment of a task optimization device based on big data, and the device embodiment corresponds to the method embodiment shown in FIG. 2 , Specifically, the device can be applied to various electronic devices.

As shown in FIG. 3 , the big data-based task optimization apparatus 300 in this embodiment includes: a receiving module 301 , a classification module 302 , a coincidence degree calculation module 303 , a reuse degree calculation module 304 , and a merging module 305 . in:

The receiving module 301 is configured to obtain the task dependency set of each target task when receiving the optimization task instruction;

A classification module 302, configured to classify each of the target tasks according to the optimization task instruction, and obtain a task coincidence degree set composed of the target tasks;

The coincidence degree calculation module 303 is used to calculate the coincidence degree of any two target tasks in the task coincidence degree set;

a reusability calculation module 304, configured to obtain the reusability of each target task in the task coincidence degree set;

The merging module 305 is configured to, if the degree of reuse of the target task in the set of task coincidence degrees is lower than the preset degree of reuse and the degree of coincidence between the target task and another target task is higher than the preset degree of coincidence, The target task is merged with the other target task.

In this embodiment, a target task with a low degree of reuse is found, and according to the degree of overlap of dependencies between the two target tasks, a pair of target tasks with a low degree of reuse and an excessively high degree of overlap are screened out, and the target task is evaluated. Combined or replaced, the follow-up requirement design preferentially uses the table with high reusability as the base. At the same time, the dependence of the task after the reference design and the existing task overlap, reduce the development of the chimney, and achieve the beneficial effect of optimizing the task development.

In some optional implementations of this embodiment, the receiving module 301 includes:

The parameter acquisition subunit 3011 is used to acquire the configuration parameters of each of the target tasks from the task configuration system;

A task acquisition subunit 3012, configured to acquire the upstream dependent tasks of each of the target tasks according to the configuration parameters;

The set obtaining subunit 3013 is configured to obtain the task dependency set according to the upstream dependent task.

In this embodiment, all upstream dependent tasks of the target task are traced back by calling the configuration parameters in the task configuration system, and all upstream dependent tasks are integrated to obtain the task dependency set of the target task, and the upstream dependencies are accurately obtained according to the hierarchical relationship of the target task. Tasks to achieve fast and precise acquisition of the intended effect of target-dependent collections.

In some optional implementations of this embodiment, the classification module 301 further includes:

The log acquisition subunit 3014 is used to acquire the operation log of each target task;

Search subunit 3015, used to search the execution code of the target task in the operation log;

The integration subunit 3016 is configured to integrate the upstream dependent tasks of each of the target tasks in the source table according to the source table of the execution code of the target task to obtain the task dependency set.

In this embodiment, all upstream dependent tasks of the target task are traced back to all upstream dependent tasks of the target task through the running log of the target task and according to the execution code recorded in the running log, to obtain a task dependency set composed of all upstream dependent tasks.

In some optional implementation manners of this embodiment, the receiving module 302 further includes:

A parsing subunit 3021, configured to parse the optimization task instruction, wherein the optimization task instruction includes requirements classified according to the target data level;

The set generating subunit 3022 obtains the target task whose level is the target data level, and generates the task coincidence degree set. .

In this embodiment, the target tasks are classified by means of the data link layer, which is beneficial for each department in the enterprise to optimize the department's tasks and achieve local optimization. The task coincidence degree set is obtained through fragmented processing, which improves the search efficiency. Efficiency of chimney development.

In some optional implementations of this embodiment, the coincidence degree calculation module 303 further includes:

The task acquisition subunit 3031 is used to acquire any two target tasks in the task coincidence degree set;

The counting subunit 3032 is used to obtain the number of elements of the respective task-dependent sets of the two target tasks, the number of union elements and the number of intersection elements of the task-dependent sets of the two target tasks;

The composite degree calculation subunit 3033 is used to calculate the task according to the number of elements of the respective task dependency sets of the two target tasks, the number of union elements and the number of intersection elements of the task dependency sets of the two target tasks The coincidence degree of any two target tasks in the coincidence degree set.

In this embodiment, the task dependency sets of the two target tasks are obtained, and the number of elements of the task dependency sets of the two target tasks, the number of intersection elements and the number of unions of the task dependency sets of the two target tasks are obtained respectively. , get the coincidence degree, the calculation process is simple, easy to operate, and can accurately reflect the coincidence degree between the two target tasks.

In some optional implementations of this embodiment, the multiplexing degree calculation module 304 further includes:

The reuse degree acquisition subunit 3041 is used to acquire the direct downstream dependency set of each of the target tasks;

The multiplexing degree calculation subunit 3042 is configured to determine the multiplexing degree of each of the target tasks according to the number of elements in the direct downstream dependency set.

In this embodiment, by invoking the configuration parameters of the target task in the task configuration system or the code for invoking the target task recorded in the task operation log, and according to all the invocation relationships contained in the code, by performing task matching, the dependent target task can be obtained. All direct downstream tasks of the target task constitute the set of direct downstream dependencies. The direct downstream dependency set consists of the direct downstream tasks of the target task, and the number of elements is the number of direct downstream tasks, that is, the number of elements in the direct downstream dependency set is the reuse degree of the target task. The degree of reuse is used to measure the pros and cons of task design, reflecting the sharing of data models.

By obtaining the degree of overlap between the two target tasks, it is used to measure the dependency between the two target tasks, and the overlap rate between the dependent upstream tasks reflects the degree of redundancy, and the degree of reuse of the target task is directly obtained by obtaining The downstream dependency set reflects the data model sharing between target tasks, and screen out target tasks with high overlap rate and poor sharing from two perspectives, effectively reducing the beneficial effect of chimney development.

To solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 4 for details. FIG. 4 is a block diagram of a basic structure of a computer device according to this embodiment.

The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that communicate with each other through a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure, but it should be understood that it is not required to implement all of the shown components, and more or less components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, special-purpose Integrated circuit (ApplicationSpecific Integrated Circuit, ASIC), programmable gate array (Field-Programmable GateArray, FPGA), digital processor (Digital Signal Processor, DSP), embedded equipment, etc.

The computer equipment may be a desktop computer, a notebook computer, a palmtop computer, a cloud server and other computing equipment. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touch pad or a voice control device.

The storage 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type storage (eg, SD or D task-optimized storage based on big data, etc.), random access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4 , such as a hard disk or a memory of the computer device 4 . In other embodiments, the memory 41 may also be an external storage device of the computer device 4 , such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, flash memory card (Flash Card) and so on. Of course, the memory 41 may also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4 , such as computer-readable instructions for a task optimization method based on big data. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

The processor 42 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. This processor 42 is typically used to control the overall operation of the computer device 4 . In this embodiment, the processor 42 is configured to execute computer-readable instructions stored in the memory 41 or process data, for example, computer-readable instructions for executing the big data-based task optimization method.

The network interface 43 may include a wireless network interface or a wired network interface, and the network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

By finding the target task with low degree of reuse, and according to the degree of dependency overlap between the two target tasks, screen out the target task pair with low degree of reuse and high degree of overlap, and merge or replace the target tasks. The demand design prioritizes the use of tables with high reusability as the base. At the same time, the tasks after the reference design overlap with the existing tasks to reduce the development of chimneys and achieve the beneficial effect of optimizing task development.

The present application also provides another embodiment, that is, to provide a computer-readable storage medium, where the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to The at least one processor is caused to perform the steps of the big data-based task optimization method as described above.

By finding the target task with low degree of reuse, and according to the degree of dependency overlap between the two target tasks, screen out the target task pair with low degree of reuse and high degree of overlap, and merge or replace the target tasks. The demand design prioritizes the use of tables with high reusability as the base. At the same time, the tasks after the reference design overlap with the existing tasks to reduce the development of chimneys and achieve the beneficial effect of optimizing task development.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of this application.

Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of the present application, but do not limit the scope of the patent of the present application. This application may be embodied in many different forms, rather these embodiments are provided so that a thorough and complete understanding of the disclosure of this application is provided. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. . Any equivalent structure made by using the contents of the description and drawings of the present application, which is directly or indirectly used in other related technical fields, is also within the scope of protection of the patent of the present application.

Claims (10)

1. A task optimization method based on big data is characterized by comprising the following steps:

when an optimization task instruction is received, a task dependency set of each target task is obtained;

classifying the target tasks according to the optimized task instruction to obtain a task overlap ratio set formed by the target tasks;

calculating the contact ratio of any two target tasks in the task contact ratio set;

acquiring the reusability of each target task in the task overlap ratio set;

and if the reusability of the target task in the task overlap ratio set is lower than a preset reusability and the overlap ratio of the target task and another target task is higher than a preset overlap ratio, merging the target task with the other target task.

2. The big-data-based task optimization method according to claim 1, wherein the step of obtaining the task dependency set of each target task comprises:

acquiring configuration parameters of each target task from a task configuration system;

acquiring an upstream dependent task of each target task according to the configuration parameters;

and obtaining the task dependency set according to the upstream dependency task.

3. The big data based task optimization method according to claim 1, wherein the step of obtaining the task dependency set of each target task further comprises:

acquiring running logs of each target task;

searching the execution code of the target task in the running log;

and integrating the upstream dependent tasks of the target tasks in the source table according to the source table of the execution codes of the target tasks to obtain the task dependent set.

4. The big data-based task optimization method according to claim 1, wherein the step of classifying each target task according to the optimization task instruction and obtaining a task overlap ratio set formed by the target tasks comprises:

analyzing the optimization task instruction, wherein the optimization task instruction comprises a requirement of classification according to a target data hierarchy;

and acquiring a target task with a level as the target data level, and generating the task overlap ratio set.

5. The big-data-based task optimization method according to claim 1, wherein the step of calculating the contact ratio of any two target tasks in the task contact ratio set comprises:

acquiring any two target tasks in the task overlap ratio set;

respectively acquiring the element number of the task dependency set of each of the two target tasks, the union element number and the intersection element number of the task dependency sets of the two target tasks;

and calculating the coincidence degree of any two target tasks in the task coincidence degree set according to the element number of the task dependency set of each of the two target tasks, the union element number and the intersection element number of the task dependency sets of the two target tasks.

6. The big-data-based task optimization method according to claim 5, wherein the overlap ratio of any two target tasks in the task overlap ratio set is calculated according to the number of elements in the task dependency set of each of the two target tasks, the number of union elements and the number of intersection elements in the task dependency set of the two target tasks, and the calculation formula is as follows:

or

Or

Wherein S is the contact ratio; a is a task dependency set of a first target task, wherein the first target task is any one of the two target tasks; b is a task dependency set of the other of the two target tasks; n (A ≈ B) the number of elements of the intersection of the set A and the set B; n (A U.B) is the number of elements of the union of the set A and the set B; n (A) is the number of elements in set A; n (B) is the number of elements in set B.

7. The big-data-based task optimization method according to claim 1, wherein the step of obtaining the reusability of each target task in the task overlap ratio set comprises:

acquiring a direct downstream dependency set of each target task;

and determining the multiplexing degree of each target task according to the number of elements of the direct downstream dependency set.

8. A big data based task optimization device, wherein the big data based task optimization device comprises:

the receiving module is used for acquiring a task dependency set of each target task when receiving the task optimization instruction;

the classification module is used for classifying the target tasks according to the optimization task instruction to obtain a task overlap ratio set formed by the target tasks;

the coincidence degree calculation module is used for calculating the coincidence degree of any two target tasks in the task coincidence degree set;

the reuse degree calculation module is used for acquiring the reuse degree of each target task in the task overlap degree set;

and the merging module is used for merging the target task into another target task if the reusability of the target task in the task overlap ratio set is lower than a preset reusability and the overlap ratio of the target task and the other target task is higher than a preset overlap ratio.

9. A computer device comprising a memory having computer readable instructions stored therein and a processor that when executed performs the steps of the big data based task optimization method of any of claims 1 to 7.

10. A computer-readable storage medium, wherein the computer-readable storage medium stores thereon computer-readable instructions, which when executed by a processor, implement the steps of the big data based task optimization method according to any one of claims 1 to 7.

Images