CN120011079A - Method, device, equipment, medium and product for determining operator parallelism - Google Patents
Method, device, equipment, medium and product for determining operator parallelism Download PDFInfo
- Publication number
- CN120011079A CN120011079A CN202510168173.2A CN202510168173A CN120011079A CN 120011079 A CN120011079 A CN 120011079A CN 202510168173 A CN202510168173 A CN 202510168173A CN 120011079 A CN120011079 A CN 120011079A
- Authority
- CN
- China
- Prior art keywords
- operator
- data stream
- parallelism
- data
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/042—Knowledge-based neural networks; Logical representations of neural networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
本公开涉及数据流处理技术领域，公开了算子并行度的确定方法、装置、设备、介质及产品。其中，该方法包括：获取当次迭代的数据流算子对应的邻居算子，所述邻居算子包括上游算子和/或下游算子；基于所述数据流算子与所述邻居算子之间的依赖关系，确定所述数据流算子的目标算子特征；按照所述目标算子特征对所述数据流算子的当前算子并行度进行调优处理，得到所述数据流算子的目标并行度。通过实施本公开技术方案，能够考虑数据流算子在结构上的依赖关系以提升并行度的自动调优能力，保证数据流算子的处理能力与其输入速率相匹配，从而保障了流处理系统的数据处理效率。
The present disclosure relates to the field of data stream processing technology, and discloses a method, device, equipment, medium and product for determining operator parallelism. Among them, the method includes: obtaining the neighbor operator corresponding to the data stream operator of the current iteration, and the neighbor operator includes an upstream operator and/or a downstream operator; based on the dependency relationship between the data stream operator and the neighbor operator, determining the target operator characteristics of the data stream operator; according to the target operator characteristics, the current operator parallelism of the data stream operator is tuned to obtain the target parallelism of the data stream operator. By implementing the technical solution of the present disclosure, the structural dependency of the data stream operator can be considered to improve the automatic tuning capability of the parallelism, ensure that the processing capability of the data stream operator matches its input rate, and thus ensure the data processing efficiency of the stream processing system.
Description
Technical Field
The disclosure relates to the technical field of data stream processing, in particular to a method, a device, equipment, a medium and a product for determining operator parallelism.
Background
Distributed stream processing systems find wide application in real-time data management and analysis in various industries. In practical applications, the data stream performs workload characteristics that should accommodate fluctuations, e.g., different data arrival rates and processing complexity. Currently, these fluctuations are mostly handled by system engineers manually adjusting the parallelism of the dataflow operators to match different workload demands. However, manual tuning is labor intensive and error-prone, which often results in poor resource allocation. Although some rule-based methods and model-based methods exist for automatic parallelism tuning in a stream processing system, the dependency relationship of a data stream operator on the structure is ignored, and effective tuning decision is difficult to make, so that the processing capacity of the data stream operator is not matched with the input rate of the data stream operator to cause bottleneck, and the data processing effect of the stream processing system is affected.
Disclosure of Invention
In view of this, the present disclosure provides a method, apparatus, device, medium and product for determining operator parallelism, so as to solve the problem of poor parallelism tuning capability.
In a first aspect, the disclosure provides a method for determining parallelism of operators, which comprises the steps of obtaining a neighbor operator corresponding to a current iterative data stream operator, wherein the neighbor operator comprises an upstream operator and/or a downstream operator, determining target operator characteristics of the data stream operator based on a dependency relationship between the data stream operator and the neighbor operator, and performing optimization processing on the current operator parallelism of the data stream operator according to the target operator characteristics to obtain the target parallelism of the data stream operator.
In a second aspect, the disclosure provides a determining device for operator parallelism, which comprises an obtaining module, an operator characteristic determining module and a parallelism adjusting module, wherein the obtaining module is used for obtaining a neighbor operator corresponding to a current iterative data stream operator, the neighbor operator comprises an upstream operator and/or a downstream operator, the operator characteristic determining module is used for determining a target operator characteristic of the data stream operator based on a dependency relationship between the data stream operator and the neighbor operator, and the parallelism adjusting module is used for adjusting and optimizing the current operator parallelism of the data stream operator according to the target operator characteristic to obtain the target parallelism of the data stream operator.
In a third aspect, the present disclosure provides a computer device, including a memory and a processor, where the memory and the processor are communicatively connected to each other, and the memory stores computer instructions, and the processor executes the computer instructions, thereby executing the method for determining operator parallelism in the first aspect or any implementation manner corresponding to the first aspect.
In a fourth aspect, the present disclosure provides a computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the method for determining operator parallelism of the first aspect or any one of its corresponding embodiments.
In a fifth aspect, the present disclosure provides a computer program product comprising computer instructions for causing a computer to perform the method for determining operator parallelism of the first aspect or any one of its corresponding embodiments.
According to the operator parallelism determination method, device, equipment, medium and product, the target operator characteristics of the data stream operator are determined by acquiring the neighbor operators corresponding to the data stream operator iterated at the present time and combining the dependency relationship between the data stream operator and the neighbor operators, so that the dependency relationship between the data stream operator and the neighbor operators can be captured through the target operator characteristics. And then, when the operator parallelism of the data stream operator is optimized according to the target operator characteristics, the self-optimizing capability of the parallelism can be improved by considering the structural dependency relationship of the data stream operator, and the parallelism is ensured to be the optimal parallelism, so that the processing capability of the data stream operator is matched with the input rate of the data stream operator, and the data processing efficiency of the stream processing system is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the related art, the drawings that are required to be used in the description of the embodiments or the related art will be briefly described below, and it is apparent that the drawings in the following description are some embodiments of the present disclosure, and other drawings may be obtained according to the drawings without inventive effort to those of ordinary skill in the art.
FIG. 1 is a flow diagram of a method of determining operator parallelism according to an embodiment of the disclosure;
FIG. 2 is a schematic diagram of a directed acyclic graph of an embodiment of the present disclosure;
FIG. 3 is a flow diagram of another method of determining operator parallelism in accordance with an embodiment of the disclosure;
FIG. 4 is a flow diagram of a method of determining parallelism of yet another operator according to an embodiment of the disclosure;
FIG. 5 is a block diagram of a determination apparatus of operator parallelism according to an embodiment of the disclosure;
fig. 6 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are some embodiments of the present disclosure, but not all embodiments. Based on the embodiments in this disclosure, all other embodiments that a person skilled in the art would obtain without making any inventive effort are within the scope of protection of this disclosure.
It will be appreciated that prior to using the technical solutions disclosed in the embodiments of the present disclosure, the user should be informed and authorized of the type, usage range, usage scenario, etc. of the personal information related to the present disclosure in an appropriate manner according to the relevant legal regulations.
For example, in response to receiving an active request from a user, a prompt is sent to the user to explicitly prompt the user that the operation it is requesting to perform will require personal information to be obtained and used with the user. Thus, the user can autonomously select whether to provide personal information to software or hardware such as a computer device, an application program, a server or a storage medium for executing the operation of the technical scheme of the present disclosure according to the prompt information.
As an alternative but non-limiting implementation, in response to receiving an active request from a user, the manner in which the prompt information is sent to the user may be, for example, a popup, in which the prompt information may be presented in a text manner. In addition, a selection control may be carried in the popup for the user to select "agree" or "disagree" to provide personal information to the computer device.
It will be appreciated that the above-described notification and user authorization process is merely illustrative and not limiting of the implementations of the present disclosure, and that other ways of satisfying relevant legal regulations may be applied to the implementations of the present disclosure.
It will be appreciated that the data (including but not limited to the data itself, the acquisition or use of the data) involved in the present technical solution should comply with the corresponding legal regulations and the requirements of the relevant regulations.
Distributed Stream Processing Systems (DSPSs), such as APACHE FLINK, timely Dataflow, etc., have found wide use in real-time data management and analysis in various industries. Unlike conventional batch processing systems, distributed stream processing systems are constructed for continuous execution and are capable of nearly instantaneous response to incoming data.
Distributed stream processing systems rely on data stream models as the underlying abstraction for defining and executing continuous stream processing tasks, and the data stream models organize computing tasks into Directed Acyclic Graphs (DAGs) to facilitate scalable execution. In this data flow model, nodes represent data flow operators that perform computations on incoming data, while edges characterize data dependencies between the data flow operators. The execution of the data streams relies on asynchronous messaging, allowing each data stream operator to process data independently, thereby achieving high throughput and fault tolerance.
However, in practical applications, the execution of the data stream should accommodate fluctuating workload characteristics such as different data arrival rates and processing complexity. Typically, distributed stream processing system engineers handle these fluctuations by manually adjusting the parallelism of the data stream operators to match different workload demands. This process includes increasing parallelism (i.e., longitudinal expansion) during peak hours to maintain performance and decreasing parallelism (i.e., longitudinal contraction) during off-peak hours to conserve resources. However, manual tuning is labor intensive and error-prone, which often results in poor resource allocation. Therefore, the automatic adjustment and optimization of the parallelism are generated, so that the problem of poor resource allocation caused by manual adjustment and optimization is solved, and the efficient utilization of resources and the reliability of a system can be ensured when unpredictable workload is handled.
At present, parallelism automatic tuning is mainly realized on the basis of rules and models, but still has some problems. In particular, the data flow operators of a flow processing task are typically organized in the form of directed acyclic graphs, with data flowing from upstream operators to downstream operators in sequence. When the upstream operator processes data faster than the downstream operator, bottlenecks occur, which cause the downstream operator to process more difficult due to excessive inputs. However, the existing methods for automatically tuning parallelism based on rules and models do not consider the interdependence between data stream operators, so that effective tuning decisions are difficult to make.
Based on the above, the technical scheme of the present disclosure uses the interdependence relationship between the data stream operators to perform effective automatic adjustment of parallelism, so as to ensure that the processing capability of each data stream operator is matched with the input rate thereof.
According to an embodiment of the present disclosure, there is provided an operator parallelism determination method embodiment, it being noted that the steps shown in the flowcharts of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.
In this embodiment, a method for determining operator parallelism is provided, which may be used in a computer device, such as a computer, a server, etc., and fig. 1 is a flowchart of a method for determining operator parallelism according to an embodiment of the disclosure, as shown in fig. 1, where the flowchart includes the following steps:
Step S101, a neighbor operator corresponding to the data stream operator of the current iteration is obtained, wherein the neighbor operator comprises an upstream operator and/or a downstream operator.
The data stream operator is a logic operator for executing stream processing tasks, and the data stream operator sequentially iterates according to the data stream topological order of the logic data stream directed acyclic graph (DIRECTED ACYCLIC GRAPH, DAG). The current iteration is the process that the data flow operator carries out the latest iteration according to the data flow topological sequence of the DAG.
The neighbor operator is a logic operator with a neighbor relation with the data stream operator, and the neighbor operator comprises an upstream operator of the data stream operator and a downstream operator of the data stream operator. Specifically, if the data stream operator is the first operator in the DAG, the neighbor operator is the downstream operator, if the data stream operator is the termination operator in the DAG, the neighbor operator is the upstream operator, and if the data stream operator is not the first operator and is not the termination operator, the neighbor operator comprises the upstream operator and the downstream operator. Taking fig. 2 as an example, for the data stream operator O1, its neighbor operators are downstream operators O2 and O3, for the data stream operator O2 or O3, its neighbor operators are upstream operators O1 and downstream operator O4, and for the data stream operator O4, its neighbor operators are upstream operators O2 and O3.
Each node in the DAG represents each logic operator for executing the stream processing task, the position of each data stream operator when executing the stream processing task can be determined by combining the graph structure of the DAG, and the neighbor operators corresponding to each data stream operator are determined by combining the position of each data stream operator and the data flow direction represented by the DAG.
Step S102, determining the target operator characteristics of the data stream operator based on the dependency relationship between the data stream operator and the neighbor operator.
The target operator characteristics represent characteristics of the data stream operator to perform stream processing tasks, such as inherent characteristics of the data stream operator (such as operator type, data length, etc.) and characteristics of the data stream operator that change over time (such as parallelism, source rate of external data sources, etc.).
The dependency relationship between the data stream operator and the neighbor operator determines the execution sequence of the data stream operator and the flow path of the task data. During each iteration, the task feature data of each operator can be iteratively propagated and aggregated according to the dependency relationship. Therefore, in the current iteration process, according to the dependency relationship between the data stream operator and the neighbor operator, the operator characteristics of the neighbor operator corresponding to the data stream operator are aggregated, and the aggregate result of the operator characteristics is combined to determine the target operator characteristics of the data stream operator, so that the target operator characteristics carry the characteristics of the data stream operator and the characteristics of the neighbor operator thereof, and the relationship between the data stream operator and the neighbor operator thereof and the structural characteristics of the directed acyclic graph where the data stream operator is located can be captured according to the target operator characteristics.
And step S103, performing optimization processing on the current operator parallelism of the data stream operator according to the target operator characteristics to obtain the target parallelism of the data stream operator.
The optimal parallelism for task processing generated by the current operator parallelism in the last iteration process of the data stream operator is the optimal parallelism for task processing of the data stream operator in the current iteration process of the data stream operator.
Because the target operator features aggregate the operator features of the current data stream operator and the neighbor operators thereof, the current real-time execution data can be combined, and whether an operator bottleneck exists in the data stream operator according to the current operator parallelism process task data processing can be determined. If operator bottlenecks exist, the parallelism of the current operator is adjusted and optimized according to the characteristics of the target operator, so that the optimal parallelism of the current real-time execution data, namely the target parallelism of the data flow operator in the current iteration process, is obtained.
According to the operator parallelism determination method provided by the embodiment, the target operator characteristics of the data stream operator are determined by acquiring the neighbor operators corresponding to the data stream operator iterated at the time and combining the dependency relationship between the data stream operator and the neighbor operators, so that the dependency relationship between the data stream operator and the neighbor operators can be captured through the target operator characteristics. And then, when the operator parallelism of the data stream operator is optimized according to the target operator characteristics, the self-optimizing capability of the parallelism can be improved by considering the structural dependency relationship of the data stream operator, and the parallelism is ensured to be the optimal parallelism, so that the processing capability of the data stream operator is matched with the input rate of the data stream operator, and the data processing efficiency of the stream processing system is ensured.
In this embodiment, a method for determining operator parallelism is provided, which may be used in a computer device, such as a computer, a server, etc., and fig. 3 is a flowchart of a method for determining operator parallelism according to an embodiment of the disclosure, as shown in fig. 3, where the flowchart includes the following steps:
Step S201, a neighbor operator corresponding to the data stream operator of the current iteration is obtained, wherein the neighbor operator comprises an upstream operator and/or a downstream operator.
Specifically, the step S201 includes:
and step S2011, obtaining a directed acyclic graph in which the data flow operator is located.
The directed acyclic graph is a graph structure formed by data flow operators through which a flow processing task starts to execute and ends to execute. The directed acyclic graph for executing the stream processing task is the same for the data stream processing system, and is stored in a database or memory space corresponding to the data stream processing system. As shown in fig. 2, the directed acyclic graph has nodes representing data stream operators for processing stream processing tasks, arrows representing data streams of stream processing tasks, and edges representing relationships between adjacent data stream operators.
Specifically, the data stream directed acyclic graph of the stream processing task is encoded by a graph neural network (Graph Neural Network, GNN) to capture features of its graph structure. Node dependencies within the directed acyclic graph are modeled by the graph neural network to learn node representations for iterative messaging between neighboring nodes to process the graph structure data. When the neighbor operator is searched for the data flow operator, the computer equipment can acquire the directed acyclic graph where the data flow operator is located and the position of the data flow operator in the directed acyclic graph by querying the database or the memory space.
Step S2012, determining an upstream operator and/or a downstream operator corresponding to the data stream operator based on the operator topology structure of the directed acyclic graph.
Different data stream operators have different operator identifications, i.e. each data stream operator in the directed acyclic graph is unique. The operator topological structure is used for representing the position of each data flow operator in the directed acyclic graph, so that the position of the data flow operator in the directed acyclic graph is combined, and an upstream operator, a downstream operator, or both the upstream operator and the downstream operator with the dependency relationship can be determined.
Step S202, determining target operator characteristics of the data stream operator based on the dependency relationship between the data stream operator and the neighbor operator.
Specifically, the step S202 includes:
in step S2021, the initial operator characteristics of the data stream operator are acquired.
The initial operator features are operator features that the dataflow operator possesses at the time of the iteration. Specifically, a directed acyclic graph is provided, denoted g= (V, E), where V is a node set and E is an edge set. For each node V e V in the set of nodes, there is a corresponding initialization feature
If the current iteration is the first iteration, the initial operator features are the initialization features of the dataflow operatorIf the iteration is not the first iteration, the initial operator characteristics are operator characteristics generated by the data stream operators after the last iterationWhere t represents the current iteration and t-1 represents the last iteration.
In some alternative embodiments, step S2021 described above comprises:
and a step a1, obtaining static characteristics and dynamic characteristics corresponding to the data flow operator.
And a2, constructing initial operator characteristics of the data flow operator according to the static characteristics and the dynamic characteristics.
Static features are fixed, upstream and downstream operator independent attribute features that remain unchanged during execution of a stream processing task, and may include operator type, data length, input data type, output data type, and the like. The inherent characteristics of the data stream operator are captured through static characteristics, and the inherent characteristics are migratable, so that the tuning knowledge of the parallelism can be transferred from the last iteration to the stream task processing process of the current iteration.
The dynamic characteristics are the characteristics of operator changes along with time in the process of processing the streaming task, and can specifically comprise the source rate controlled by a data source and the operator parallelism obtained in the iterative tuning process.
And connecting all static characteristics of the data stream operator v and the dynamic characteristic of the source rate to obtain initial operator characteristics corresponding to the data stream operator. Specifically, for the classification features in the static features, the classification features are expressed as a vector according to a preset encoding mode (such as single-hot encoding), and for the data type features, the numerical values are normalized to the continuous interval by adopting a preset normalization mode (such as minimum-maximum uniform scaling). And then, connecting the encoded static features and the source rate to form initial operator features.
In the embodiment, the initial operator characteristics of the data stream operator are constructed according to the static characteristics and the dynamic characteristics of the data stream operator, so that the attribute characteristics of the data stream operator can be accurately represented by the initial operator characteristics, and the improvement effect of parallelism is facilitated.
Step S2022, aggregating the operator features of the neighbor operators to generate operator aggregate features of the neighbor operators.
In each iteration process, all operator features of neighbor operators corresponding to the data stream operators are aggregated according to a preset aggregation mode (such as mean value aggregation, weighted mean value aggregation, maximum value aggregation and the like), so that operator aggregation features are obtained.
Step S2023, determining the data characteristics of the data flow operator in the current iteration according to the dependency relationship and the operator aggregation characteristics.
According to the dependency relationship between the data stream operator and the neighbor operator, calculating the data characteristics of the data stream operator in the current iteration process according to the operator aggregation characteristics corresponding to the neighbor operator, wherein the specific calculation mode is as follows:
wherein AGG () represents an aggregation function; Processing information for task data received by the data flow operator in the current iteration process, wherein the processing information is used for representing data characteristics of the data flow operator in the current iteration process; and N (v) represents operator aggregation characteristics of the neighbor operators.
Step S2024, updating the initial operator features according to the data features to obtain target operator features of the data stream operator in the current iteration.
In the current iteration, the data stream operator is combined with the data characteristics obtained by calculating the data stream operator according to the operator aggregation characteristics to update the initial operator characteristics corresponding to the data stream operator, so as to obtain corresponding target operator characteristics, wherein the specific calculation mode is as follows:
Wherein, Representing target operator characteristics of the data stream operator in the current iteration; UPDATE () represents an UPDATE function; Representing operator characteristics of the dataflow operator in the last iteration; Representing the data characteristics of the data stream operator in the current iteration, wherein the operator characteristics of the neighbor operator are fused.
Thus, after T iterations, the target operator characteristics of the final data stream operatorInformation from the data stream operator itself and its surrounding neighbor operators can be integrated so that the target operator features can effectively capture the structural characteristics of the directed acyclic graph and the relationships between the data stream operators.
And step S203, performing optimization processing on the current operator parallelism of the data stream operator according to the target operator characteristics to obtain the target parallelism of the data stream operator. Please refer to the related descriptions of the corresponding steps in the above embodiments, which are not repeated herein.
According to the operator parallelism determination method provided by the embodiment, the neighbor operators of the data stream operators are determined according to the operator topological structure of the directed acyclic graph by acquiring the directed acyclic graph where the data stream operators are located, so that the operator characteristics of the neighbor operators can be accurately aggregated later. And combining the dependency relation pair to iteratively propagate and aggregate the operator characteristics of the neighbor operators, combining the operator aggregation characteristics obtained by aggregation, and determining the target operator characteristics of the data stream operators in the current iteration process, so that the representation of the data stream operators can contain local characteristics of the data stream operators and context operator characteristic information from upstream operators and downstream operators of the data stream operators. Therefore, the target operator characteristics generated by aiming at the data flow operators can effectively capture the relation between the structural characteristics of the directed acyclic graph and the data flow operators, so that bottlenecks in the data processing process can be effectively perceived, and the parallelism can be effectively adjusted according to the bottleneck information.
In this embodiment, a method for determining operator parallelism is provided, which may be used in a computer device, such as a computer, a server, etc., and fig. 4 is a flowchart of a method for determining operator parallelism according to an embodiment of the disclosure, as shown in fig. 4, where the flowchart includes the following steps:
Step S301, a neighbor operator corresponding to the current iterative data stream operator is obtained, where the neighbor operator includes an upstream operator and/or a downstream operator. Please refer to the related descriptions of the corresponding steps in the above embodiments, which are not repeated herein.
Step S302, determining the target operator characteristics of the data stream operator based on the dependency relationship between the data stream operator and the neighbor operator.
Step S303, the current operator parallelism of the data stream operator is optimized according to the target operator characteristics, and the target parallelism of the data stream operator is obtained.
Specifically, the step S303 includes:
step S3031, the current operator parallelism of the data stream operator is fused with the target operator characteristics to obtain parallelism perception characteristics.
According to a preset feature fusion mode, the current parallelism required to be optimized and the target operator features of a data flow operator are integrated by utilizing nonlinear transformation, so that parallelism perception features with unchanged feature dimensions are obtained, and the parallelism perception features are ensured to participate in subsequent task data transfer iteration in a seamless mode, wherein the specific fusion mode is as follows:
Wherein, The feature fusion function is represented by FUSE (), and p v represents the current parallelism; Representing the target operator features. When iterative transfer is carried out on stream processing task data, target operator characteristics Is an operator feature independent of parallelism; is a feature with parallelism perception.
Step S3032, the parallelism of the current operator is optimized according to the parallelism perception characteristics, and the target parallelism of the data stream operator is obtained.
The target parallelism is the optimal parallelism generated in the current iteration. When iterative transfer is carried out on real-time task data generated by executing a stream processing task, the processing capacity of a data stream operator on the real-time task data is predicted by combining the real-time task data and the parallelism perception characteristic. If the processing capacity of the data flow operator on the real-time task data is insufficient, the current parallelism of the data flow operator is adjusted and optimized according to the parallelism perception characteristic, so that the target parallelism capable of meeting the real-time task data processing is obtained.
In some optional embodiments, step S3032 includes:
and b1, detecting whether an operator bottleneck exists in the data stream operator under the current operator parallelism.
And b2, if the bottleneck exists, adjusting the parallelism of the current operator according to the parallelism perception characteristic to obtain the target parallelism for eliminating the bottleneck of the operator.
The operator bottleneck represents the insufficient processing power of the data stream operator for the stream processing task. Specifically, when the current operator parallelism is utilized to execute the stream processing task, the task data received by the data stream operator is combined, and whether backlog of the task data exists when the task data received by the current operator parallelism is executed is predicted. If backlog of the task data exists, operator bottlenecks of the data stream operators can be judged. Therefore, the current parallelism can be determined not to meet the processing capability of task data, and the current parallelism can be adjusted (for example, the parallelism corresponding to the data flow operator is improved) according to the parallelism perception characteristic, so that the minimum parallelism which can eliminate operator bottlenecks and meet the processing capability of the task data is obtained, and the minimum parallelism is the target parallelism corresponding to the data flow operator.
In the above embodiment, by detecting the bottleneck of the operator level in the stream processing task, when it is determined that the operator bottleneck exists in the data stream operator under the current operator parallelism, the current operator parallelism is adjusted by combining the operator bottleneck and the parallelism perception feature until the operator bottleneck is eliminated, the corresponding target parallelism is determined, and the tuning capability of the operator parallelism is improved from the operator level.
In some alternative embodiments, step b1 includes:
and b11, identifying whether the data stream operator has a backpressure phenomenon under the parallelism of the current operator.
And b12, if the back pressure phenomenon does not exist, judging that the operator bottleneck does not exist in the data flow operator.
And b13, if the backpressure phenomenon exists, detecting whether the backpressure phenomenon exists in a downstream operator of the data flow operator.
And step b14, if the downstream operator of the data stream operator does not have the back pressure phenomenon, judging that the data stream operator has an operator bottleneck.
The backpressure phenomenon indicates that the data flow operator has backlog of task data when processing the task data. And in combination with the task data received by the data flow operator, predicting whether the task data received by the data flow operator is subjected to back pressure phenomenon according to the current parallelism degree. If the backpressure phenomenon does not exist, the execution level of the stream processing task does not observe the backpressure phenomenon, at the moment, the fact that the operator bottleneck does not exist in the data stream operator can be judged, and the data stream operator without the operator bottleneck can be marked with a label 0.
If the backpressure phenomenon exists, whether the downstream operator of the data flow operator can exist the backpressure phenomenon is continuously detected. If it is determined that the downstream operator of the data stream operator does not have the backpressure phenomenon, it may be determined that the downstream operator of the data stream operator is sufficient for executing the processing capability of the task data output by the data stream operator. At this time, it may be determined that the data stream operator has an operator bottleneck, and the data stream operator having the operator bottleneck may be labeled 1.
In a specific example, if the dataflow operator has back pressure at the execution level of the stream processing task, the cascading effect must be considered at this time. Firstly, identifying a data flow operator with a backpressure phenomenon and a downstream operator of the data flow operator without the backpressure phenomenon, and further checking the resource utilization rate of the downstream operator of the data flow operator. If the resource utilization rate of the downstream operator is higher (for example, the CPU load exceeds 60%), the operator bottleneck can be determined to exist in the data flow operator and marked as 1, otherwise, the upstream data rate of the data flow operator is influenced by the back pressure generated at the execution level of the flow processing task, and under the condition of the current parallelism, whether the processing capacity of the data flow operator to the task data is enough is difficult to determine, so that the data flow operator is marked temporarily.
In the above embodiment, the processing capability of the data stream operator on the task data is determined by identifying whether the data stream operator has a backpressure phenomenon in the execution of the stream processing task. And when determining that the back pressure phenomenon exists in the data stream operator, combining the back pressure phenomenon of a downstream operator of the data stream operator in the execution of the stream processing task to determine whether the data stream operator has an operator bottleneck. Therefore, the cascade effect between the data stream operators is considered in the process of determining the operator bottleneck, and the judgment accuracy of the operator bottleneck is improved.
In a specific embodiment, the operator parallelism is tuned by the pre-trained encoder according to the parallelism-aware feature. The training mode of the encoder comprises the following steps:
step c1, a data flow operator sample set is obtained, wherein the data flow operator sample set comprises a plurality of data flow operator samples, and each data flow operator sample is provided with a matched bottleneck label.
And c2, pre-training the encoder by using the bottleneck label, and performing bottleneck prediction on any data stream operator sample according to the pre-trained encoder to obtain a bottleneck prediction label of any data stream operator sample.
And step c3, training the encoder based on loss information between the bottleneck prediction tag and the real bottleneck tag.
The data flow operator sample set is a set of each logic operator (i.e. data flow operator sample) constituting the directed acyclic graph DAG, and the bottleneck label is used for representing whether the data flow operator has an operator bottleneck, for example, "1" indicates that the operator bottleneck exists, and "0" indicates that the operator bottleneck does not exist.
And (3) pre-training an encoder based on a Graph Neural Network (GNN) by using a bottleneck label, and predicting the parallelism of a data stream operator sample according to the pre-trained encoder to obtain a corresponding parallelism prediction value. For each data stream operator v with bottleneck labels, calculating loss information between the bottleneck prediction labels and the real bottleneck labels by using a binary cross entropy loss function (L v), and training the encoder by using the loss information so as to minimize the output loss information of the encoder.
Specifically, for a given set of operators O in the pre-trained data stream operator sample set, the operator set with the bottleneck label is denoted as O lable, and the total loss L total of the data stream operator set with the bottleneck label is the average loss of all the data stream operators with the bottleneck label, and the specific calculation mode is as follows:
The difference between the predicted probability distribution and the real label distribution of the bottleneck index (i.e., the loss information between the bottleneck predicted label and the real bottleneck label) is measured by using the loss function, and the encoder is trained by using the difference so as to minimize the loss function of the encoder.
In the embodiment, the encoder is trained according to the bottleneck label so as to accurately predict the bottleneck of the operator level in the stream processing task by combining the structural characteristics of the DAG through the encoder, thereby greatly improving the accuracy and efficiency of the automatic adjustment of the parallelism.
According to the operator parallelism determination method provided by the embodiment, the operator parallelism is fused in the target operator characteristics, so that the operator parallelism is combined with the target operator characteristics, and the prediction capability of operator bottlenecks is enhanced. Therefore, the precision prediction of the operator bottleneck can be combined, the current operator parallelism is optimized according to the parallelism perception characteristic, the accuracy of the parallelism automatic optimization is improved, the target parallelism obtained by the optimization is ensured to be capable of precisely eliminating the operator bottleneck, the bottleneck of the processing capacity of the data flow operator is avoided, and the data processing efficiency of the flow processing system is ensured.
The embodiment also provides a device for determining the operator parallelism, which is used for implementing the above embodiment and the preferred implementation, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
The embodiment provides a device for determining operator parallelism, as shown in fig. 5, including:
An obtaining module 401, configured to obtain a neighbor operator corresponding to a current iterative data stream operator, where the neighbor operator includes an upstream operator and/or a downstream operator.
An operator feature determination module 402, configured to determine a target operator feature of the data stream operator based on a dependency relationship between the data stream operator and the neighbor operator.
And the parallelism adjusting and optimizing module 403 is configured to perform adjusting and optimizing processing on the current operator parallelism of the data stream operator according to the target operator feature, so as to obtain the target parallelism of the data stream operator.
In some alternative embodiments, the acquisition module 401 includes:
And the diagram acquisition unit is used for acquiring the directed acyclic diagram where the data flow operator is located.
The neighbor operator determining unit is used for determining an upstream operator and/or a downstream operator corresponding to the data stream operator based on the operator topological structure of the directed acyclic graph.
In some alternative embodiments, the operator feature determination module 402 includes:
the initial information acquisition unit is used for acquiring initial operator characteristics of the data stream operator.
The feature aggregation unit is used for aggregating operator features of the neighbor operators to generate operator aggregation features of the neighbor operators.
And the data characteristic determining unit is used for determining the data characteristic of the data flow operator in the current iteration according to the dependency relationship and the operator aggregation characteristic.
And the characteristic updating unit is used for updating the initial operator characteristic according to the data characteristic to obtain the target operator characteristic of the data stream operator in the current iteration.
In some alternative embodiments, the initial information acquisition unit includes:
the feature acquisition subunit is used for acquiring static features and dynamic features corresponding to the data stream operators.
And the initial characteristic construction subunit is used for constructing initial operator characteristics of the data flow operator according to the static characteristics and the dynamic characteristics.
In some alternative embodiments, the parallelism adjusting module 403 includes:
And the feature fusion unit is used for fusing the current operator parallelism of the data stream operator with the target operator features to obtain parallelism perception features.
And the tuning unit is used for tuning the parallelism of the current operator according to the parallelism perception characteristic to obtain the target parallelism of the data stream operator.
In some alternative embodiments, the tuning unit comprises:
The bottleneck detection subunit is used for detecting whether an operator bottleneck exists in the data stream operator under the current operator parallelism.
And the parallelism adjustment subunit is used for adjusting the parallelism of the current operator according to the parallelism perception characteristic if the bottleneck exists, so as to obtain the target parallelism for eliminating the bottleneck of the operator.
In some optional embodiments, the bottleneck detection subunit is specifically configured to identify whether a backpressure phenomenon exists in the data stream operator under the current operator parallelism, determine that the data stream operator does not have an operator bottleneck if the backpressure phenomenon does not exist, detect whether a backpressure phenomenon exists in a downstream operator of the data stream operator if the backpressure phenomenon exists, and determine that the data stream operator has an operator bottleneck if the backpressure phenomenon does not exist in the downstream operator of the data stream operator.
In a specific embodiment, the apparatus further comprises:
the encoder training module is used for training the encoder, and the operator parallelism is adjusted and optimized according to the parallelism perception characteristic through the encoder which is trained in advance.
Specifically, the encoder training module includes:
The system comprises a sample acquisition unit, a data flow operator sample set and a data flow control unit, wherein the sample acquisition unit is used for acquiring a data flow operator sample set, the data flow operator sample set comprises a plurality of data flow operator samples, and each data flow operator sample is provided with a matched bottleneck label.
The pre-training unit is used for pre-training the encoder by utilizing the bottleneck label, and performing bottleneck prediction on any data stream operator sample according to the pre-trained encoder to obtain a bottleneck prediction label of any data stream operator sample.
And the training unit is used for training the encoder based on the loss information between the bottleneck prediction label and the real bottleneck label.
Further functional descriptions of the above respective modules and units are the same as those of the above corresponding embodiments, and are not repeated here.
The operator parallelism determination device in this embodiment is presented in the form of a functional unit, where a unit refers to an ASIC (Application SPECIFIC INTEGRATED Circuit) Circuit, a processor and a memory that execute one or more software or firmware programs, and/or other devices that can provide the above functions.
According to the operator parallelism determining device provided by the embodiment, the target operator characteristics of the data stream operator are determined by acquiring the neighbor operators corresponding to the data stream operator iterated at the present time and combining the dependency relationship between the data stream operator and the neighbor operators, so that the dependency relationship between the data stream operator and the neighbor operators can be captured through the target operator characteristics. And then, when the operator parallelism of the data stream operator is optimized according to the target operator characteristics, the self-optimizing capability of the parallelism can be improved by considering the structural dependency relationship of the data stream operator, and the parallelism is ensured to be the optimal parallelism, so that the processing capability of the data stream operator is matched with the input rate of the data stream operator, and the data processing efficiency of the stream processing system is ensured.
The embodiment of the disclosure also provides a computer device, which is provided with the operator parallelism determining device shown in the figure 5.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a computer device according to an alternative embodiment of the present disclosure, and as shown in fig. 6, the computer device includes one or more processors 10, a memory 20, and interfaces for connecting components, including a high-speed interface and a low-speed interface. The various components are communicatively coupled to each other using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the computer device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In some alternative embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple computer devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 10 is illustrated in fig. 6.
The processor 10 may be a central processor, a network processor, or a combination thereof. The processor 10 may further include a hardware chip, among others. The hardware chip may be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general-purpose array logic, or any combination thereof.
Wherein the memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods shown in implementing the above embodiments.
The memory 20 may include a storage program area that may store an operating system, application programs required for at least one function, and a storage data area that may store data created according to the use of the computer device, etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, memory 20 may optionally include memory located remotely from processor 10, which may be connected to the computer device via a network.
Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The memory 20 may comprise volatile memory, such as random access memory, or nonvolatile memory, such as flash memory, hard disk or solid state disk, or the memory 20 may comprise a combination of the above types of memory.
The computer device also includes a communication interface 30 for the computer device to communicate with other devices or communication networks.
The presently disclosed embodiments also provide a computer readable storage medium, and the methods described above according to the presently disclosed embodiments may be implemented in hardware, firmware, or as recordable storage medium, or as computer code downloaded over a network that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and is to be stored in a local storage medium, such that the methods described herein may be stored on such software processes on a storage medium using a general purpose computer, special purpose processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random-access memory, a flash memory, a hard disk, a solid state disk, or the like, and further, the storage medium may further include a combination of the above types of memories. It will be appreciated that a computer, processor, microprocessor controller or programmable hardware includes a storage element that can store or receive software or computer code that, when accessed and executed by the computer, processor or hardware, implements the methods illustrated by the above embodiments.
Portions of the present disclosure may be applied as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present disclosure by way of operation of the computer. Those skilled in the art will appreciate that the existence of computer program instructions in a computer-readable medium includes, but is not limited to, source files, executable files, installation package files, and the like, and accordingly, the manner in which computer program instructions are executed by a computer includes, but is not limited to, the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled programs, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed programs. Herein, a computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by a computer.
Although embodiments of the present disclosure have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the disclosure, and such modifications and variations are within the scope defined by the appended claims.
Claims (12)
1. A method for determining parallelism of operators, the method comprising:
Acquiring a neighbor operator corresponding to a current iterative data stream operator, wherein the neighbor operator comprises an upstream operator and/or a downstream operator;
determining target operator characteristics of the data stream operator based on the dependency relationship between the data stream operator and the neighbor operator;
and performing optimization processing on the current operator parallelism of the data stream operator according to the target operator characteristics to obtain the target parallelism of the data stream operator.
2. The method according to claim 1, wherein the obtaining the neighbor operator corresponding to the current iteration data stream operator comprises:
Acquiring a directed acyclic graph in which the data flow operator is positioned;
And determining an upstream operator and/or a downstream operator corresponding to the data stream operator based on the operator topological structure of the directed acyclic graph.
3. The method of claim 1, wherein the determining the target operator characteristics of the data stream operator based on the dependency relationship between the data stream operator and the neighbor operator comprises:
Acquiring initial operator characteristics of the data stream operator;
aggregating operator features of the neighbor operators to generate operator aggregation features of the neighbor operators;
Determining the data characteristics of the data flow operator in the current iteration according to the dependency relationship and the operator aggregation characteristics;
And updating the initial operator characteristics according to the data characteristics to obtain target operator characteristics of the data flow operator in the current iteration.
4. A method according to claim 3, wherein obtaining initial operator features of the data stream operator comprises:
acquiring static features and dynamic features corresponding to the data flow operators;
And constructing initial operator characteristics of the data stream operator according to the static characteristics and the dynamic characteristics.
5. The method according to claim 1, wherein said optimizing the current operator parallelism of the data stream operator according to the target operator feature to obtain the target parallelism of the data stream operator comprises:
Fusing the current operator parallelism of the data stream operator with the target operator characteristic to obtain parallelism perception characteristic;
And optimizing the parallelism of the current operator according to the parallelism perception characteristics to obtain the target parallelism of the data stream operator.
6. The method of claim 5, wherein optimizing the current operator parallelism according to the parallelism awareness feature results in a target parallelism for the data stream operator, comprising:
detecting whether an operator bottleneck exists in the data stream operator under the current operator parallelism;
if the bottleneck exists, adjusting the parallelism of the current operator according to the parallelism perception characteristic to obtain the target parallelism for eliminating the operator bottleneck.
7. The method of claim 6, wherein said detecting whether an operator bottleneck exists for the data stream operator at the current operator parallelism comprises:
identifying whether the data stream operator has a back pressure phenomenon under the parallelism of the current operator;
if the back pressure phenomenon does not exist, judging that the operator bottleneck does not exist in the data flow operator;
if the backpressure phenomenon exists, detecting whether the backpressure phenomenon exists in a downstream operator of the data flow operator;
and if the downstream operator of the data flow operator does not have the back pressure phenomenon, judging that the data flow operator has the operator bottleneck.
8. The method according to any of claims 5-7, wherein the operator parallelism is tuned according to the parallelism-aware feature by a pre-trained encoder, the training mode of the encoder comprising:
obtaining a data stream operator sample set, wherein the data stream operator sample set comprises a plurality of data stream operator samples, and each data stream operator sample is provided with a matched bottleneck label;
Pre-training the encoder by using the bottleneck label, and performing bottleneck prediction on any one of the data stream operator samples according to the pre-trained encoder to obtain a bottleneck prediction label of any one of the data stream operator samples;
and training the encoder based on loss information between the bottleneck prediction tag and the real bottleneck tag.
9. An apparatus for determining parallelism of operators, the apparatus comprising:
The acquisition module is used for acquiring a neighbor operator corresponding to the current iterative data stream operator, wherein the neighbor operator comprises an upstream operator and/or a downstream operator;
An operator characteristic determining module, configured to determine a target operator characteristic of the data stream operator based on a dependency relationship between the data stream operator and the neighbor operator;
And the parallelism adjusting and optimizing module is used for adjusting and optimizing the current operator parallelism of the data stream operator according to the target operator characteristics to obtain the target parallelism of the data stream operator.
10. A computer device, comprising:
A memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the method of determining operator parallelism of any one of claims 1 to 8.
11. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the method of determining operator parallelism according to any one of claims 1 to 8.
12. A computer program product comprising computer instructions for causing a computer to perform the method of determining operator parallelism of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510168173.2A CN120011079A (en) | 2025-02-14 | 2025-02-14 | Method, device, equipment, medium and product for determining operator parallelism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510168173.2A CN120011079A (en) | 2025-02-14 | 2025-02-14 | Method, device, equipment, medium and product for determining operator parallelism |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN120011079A true CN120011079A (en) | 2025-05-16 |
Family
ID=95661866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202510168173.2A Pending CN120011079A (en) | 2025-02-14 | 2025-02-14 | Method, device, equipment, medium and product for determining operator parallelism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN120011079A (en) |
-
2025
- 2025-02-14 CN CN202510168173.2A patent/CN120011079A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116438553A (en) | A Machine Learning Model for Probability Prediction of Operator Success in PAAS Cloud Environment | |
| Hong | A distributed, asynchronous and incremental algorithm for nonconvex optimization: An admm based approach | |
| CN116627630A (en) | Resource demand forecasting model training, demand forecasting and resource scheduling method and system | |
| CN119449630B (en) | Private domain flow peak value identification and route scheduling method and system based on time sequence prediction | |
| CN116841753B (en) | Stream processing and batch processing switching method and switching device | |
| CN111738474A (en) | Traffic state prediction method and device | |
| Rao et al. | Time Series Forecasting methods suitable for prediction of CPU usage | |
| CN117675615A (en) | A dynamic network traffic control method and system based on pre-training | |
| Huang et al. | A multi-modal attention neural network for traffic flow prediction by capturing long-short term sequence correlation | |
| Lian et al. | Conttune: Continuous tuning by conservative bayesian optimization for distributed stream data processing systems | |
| Kalantari et al. | Dynamic software rejuvenation in web services: a whale optimizationalgorithm-based approach | |
| CN113822356A (en) | A method, device, electronic device and storage medium for classifying electricity users | |
| KR102769259B1 (en) | Improved efficiency platform system for AI model management and deployment automation | |
| Mijumbi et al. | Design, implementation, and evaluation of learning algorithms for dynamic real‐time network monitoring | |
| CN120011079A (en) | Method, device, equipment, medium and product for determining operator parallelism | |
| CN115705479A (en) | Service execution system and related product | |
| CN109343952B (en) | Bayesian network determination method, Bayesian network determination device, storage medium and electronic device | |
| Tuli et al. | Optimizing the performance of fog computing environments using ai and co-simulation | |
| CN111309265A (en) | Node storage method, system, server and storage medium based on neural network | |
| Cooper | Proactive scaling of distributed stream processing work flows using workload modelling: doctoral symposium | |
| CN115081609A (en) | Acceleration method in intelligent decision, terminal equipment and storage medium | |
| CN112506652A (en) | Dynamic resource partitioning method | |
| Lei | Automated Machine Learning | |
| CN117150288B (en) | Container power consumption determination model training, determination methods, devices, equipment and media | |
| CN117762602B (en) | Deep learning cascade task scheduling method and device for edge heterogeneous hardware |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |