CN119576601A - Topic allocation method, device, server and medium applied to RocketMQ cluster - Google Patents

Abstract

The embodiment of the invention discloses a theme distribution method, a device, a server and a medium applied to RocketMQ clusters, wherein the method comprises the steps of acquiring data to be processed of all RocketMQ clusters under different evaluation dimensions when a theme distribution request is received, determining evaluation attributes under the evaluation dimensions by analyzing and processing the data to be processed under the evaluation dimensions for the different evaluation dimensions, obtaining target use attributes of each RocketMQ cluster by weighting the evaluation attributes of each RocketMQ cluster, determining target RocketMQ clusters corresponding to the theme distribution request according to all the target use attributes, and distributing target themes corresponding to the theme distribution request to target RocketMQ clusters for processing. The technical scheme provided by the embodiment of the invention realizes that the target RocketMQ cluster corresponding to the theme to be distributed is determined according to the target use attribute of each RocketMQ cluster, and achieves the technical effect of utilizing the RocketMQ cluster to the maximum extent.

Description

Theme distribution method and device applied to RocketMQ clusters, server side and medium

Technical Field

The embodiment of the invention relates to the technical field of data processing, in particular to a theme distribution method, device, server side and medium applied to RocketMQ clusters.

Background

Currently in large-scale RocketMQ cluster management, topic allocation and creation across clusters is important. The unreasonable theme distribution can cause the problems of overlarge load difference of different RocketMQ clusters, unbalanced utilization rate of the whole server resources and the like.

Currently, for the management of large-scale RocketMQ clusters, the assignment and creation of topics is specified by manually configured routing rules. That is, when assigning and creating the theme, the selection of RocketMQ clusters is completely analyzed and judged by the user on the operation status of all RocketMQ clusters, and finally, the appropriate RocketMQ clusters are selected.

The mode needs to be manually participated, has high labor cost, and can cause errors in manual distribution, so that the main distribution efficiency is greatly reduced, and the problem of overlarge resource utilization rate difference is caused.

Disclosure of Invention

The embodiment of the invention provides a theme distribution method, a device, a server and a medium applied to RocketMQ clusters, which realize the effect of effectively and conveniently determining a target RocketMQ cluster corresponding to a theme.

In a first aspect, an embodiment of the present invention provides a method for allocating a theme applied to RocketMQ clusters, where the method includes:

when a theme allocation request is received, acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions;

for different evaluation dimensions, determining evaluation attributes under the evaluation dimensions by analyzing and processing data to be processed under the evaluation dimensions;

Obtaining the target use attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ cluster;

And determining a target RocketMQ cluster corresponding to the theme allocation request according to all the target use attributes, and allocating the target theme corresponding to the theme allocation request to the target RocketMQ cluster for processing.

Further, the evaluation dimension comprises a central processing unit use dimension, a memory use dimension, a disk use dimension, a flow use dimension and an allocated theme property dimension;

correspondingly, the data to be processed comprises occupied amount data corresponding to a using dimension of a central processing unit, occupied memory data corresponding to a using dimension of a memory, disk using amount data corresponding to a using dimension of a disk, the flow using dimension comprises flow data occupied by the processed data, and the attribute dimension of the distributed subject comprises the number of the distributed subjects.

Further, the determining the evaluation attribute in the evaluation dimension through analyzing and processing the data to be processed in the evaluation dimension includes:

Determining an evaluation attribute of the central processing unit in the use dimension according to the occupied amount data of the central processing unit in the use dimension and the total amount data distributed for the RocketMQ clusters;

determining an evaluation attribute corresponding to the memory usage dimension according to the occupied memory data corresponding to the memory usage dimension and the total memory data allocated for the RocketMQ clusters;

Determining an evaluation attribute of the disk usage dimension according to the disk usage data corresponding to the disk usage dimension and the disk usage allocated for the RocketMQ clusters;

determining an evaluation attribute of the traffic usage dimension according to the traffic data corresponding to the traffic usage dimension and the total traffic data distributed to the RocketMQ clusters;

and determining the evaluation attribute of the distributed theme attribute dimension according to the distributed theme quantity of the distributed theme attribute dimension and the upper limit of the total theme quantity.

Further, before the target usage attribute of each RocketMQ cluster is obtained by weighting the evaluation attribute of each RocketMQ clusters, the method further includes:

And allocating corresponding use weights for the plurality of evaluation dimensions to determine the target use attribute of the RocketMQ cluster based on the use weight corresponding to each evaluation dimension.

Further, the obtaining the target usage attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ clusters includes:

Acquiring a use weight corresponding to each evaluation dimension;

For each RocketMQ cluster, determining the attribute to be superimposed of the RocketMQ cluster in each evaluation dimension according to the evaluation attribute of the RocketMQ cluster in each evaluation dimension and the corresponding use weight;

And obtaining the target use attribute of the RocketMQ cluster through superposition processing of the plurality of attributes to be superimposed of the RocketMQ cluster.

Further, the determining, according to all the target usage attributes, the target RocketMQ cluster corresponding to the topic allocation request includes:

And taking the RocketMQ cluster corresponding to the minimum target use attribute as a target RocketMQ cluster, so as to distribute a target theme corresponding to the theme distribution request to the target RocketMQ cluster, and enabling the RocketMQ cluster to process a message corresponding to the target theme.

In a second aspect, an embodiment of the present invention further provides a theme distribution apparatus applied to RocketMQ clusters, where the apparatus includes:

the to-be-processed data acquisition module is used for acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions when a theme allocation request is received;

The evaluation attribute determining module is used for determining evaluation attributes in different evaluation dimensions by analyzing and processing the data to be processed in the evaluation dimensions;

the target use attribute determining module is used for obtaining the target use attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ cluster;

And the theme distribution module is used for determining a target RocketMQ cluster corresponding to the theme distribution request according to all the target use attributes, and distributing the target theme corresponding to the theme distribution request to the target RocketMQ cluster for processing.

In a third aspect, an embodiment of the present invention provides a server, where the server includes:

One or more processors;

a memory for storing one or more programs;

The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the topic allocation method as provided by any embodiment of the present invention as applied to RocketMQ clusters.

In a fourth aspect, embodiments of the present invention further provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a topic allocation method as provided by any embodiment of the present invention for use with RocketMQ clusters.

In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, where the computer program when executed by a processor implements a topic allocation method as in any one of the embodiments of the present invention applied to RocketMQ clusters.

According to the technical scheme provided by the embodiment of the invention, after a theme allocation request is received, to-be-processed data of all RocketMQ clusters under different evaluation dimensions can be obtained, then for to-be-processed data under different evaluation dimensions, evaluation attributes under the evaluation dimensions are determined through analysis and processing of to-be-processed data under the evaluation dimensions, and then the evaluation attributes of each RocketMQ cluster are weighted to obtain target use attributes of each RocketMQ cluster, the target RocketMQ cluster corresponding to the theme allocation request is determined according to all the target use attributes, so that the problem that the cost is high due to the need of manual participation when the target RocketMQ cluster corresponding to the theme to be allocated is determined according to a manual mode in the prior art is solved, and the problem that the theme utilization rate is overlarge due to certain errors possibly generated when the theme is allocated based on the manual mode is realized, and the target RocketMQ clusters which are most suitable for the theme to be allocated are determined according to actual application data of each RocketMQ cluster, thereby improving the resource utilization rate of the RocketMQ clusters and the effect of automatic allocation.

Drawings

In order to more clearly illustrate the technical solution of the exemplary embodiments of the present invention, a brief description is given below of the drawings required for describing the embodiments. It is obvious that the drawings presented are only drawings of some of the embodiments of the invention to be described, and not all the drawings, and that other drawings can be made according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of a theme distribution method applied to RocketMQ clusters according to an embodiment of the present invention;

Fig. 2 is a flowchart of a theme distribution method applied to RocketMQ clusters according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a theme distribution apparatus applied to RocketMQ clusters according to an embodiment of the present invention;

Fig. 4 is a schematic structural diagram of a server according to an embodiment of the present invention.

Detailed Description

The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.

The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.

Before describing the technical scheme provided by the embodiment of the invention, an application scenario may be illustrated.

For large-scale RocketMQ clusters, if a corresponding target RocketMQ cluster needs to be allocated to the topic to be allocated, the solution provided in this embodiment may be adopted.

For each RocketMQ cluster, the number of topics that can be supported does not exceed a preset value, and based on this, the target RocketMQ cluster that is adapted to the topic to be allocated can be determined in the manner provided in this embodiment.

At present, the distribution and creation process of the topics in the large-scale RocketMQ clusters relies on manual analysis and judgment of the running conditions of the RocketMQ clusters to configure the routing rules for creating the topics, and the influence of human factors is relatively large. Meanwhile, in the use process, when hundreds of topics are accessed, the labor cost is increased, errors can be generated in manual distribution, the distribution efficiency of the topics is greatly reduced, and the problem of overlarge difference of resource utilization rates is caused.

Before describing the detailed scheme of the embodiment, a RocketMQ cluster may be described. Multiple RocketMQ clusters may be deployed on the same server, or one RocketMQ cluster may be deployed on different servers. For each RocketMQ cluster, its target usage attribute may be determined, and thus the target RocketMQ cluster to which the target topic corresponds.

Fig. 1 is a flowchart of a theme distribution method applied to RocketMQ clusters according to an embodiment of the present invention, where the embodiment may be applied to a scenario where a target RocketMQ cluster corresponding to a theme to be distributed is determined, and the theme distribution method applied to RocketMQ clusters according to the embodiment may be executed by a theme distribution device applied to RocketMQ clusters integrated in a client and/or a server.

As shown in fig. 1, the method specifically includes the following steps:

and S110, when a theme allocation request is received, acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions.

Among other things, a topic allocation request may be understood as a request that requires allocation RocketMQ of a cluster for a topic to be allocated. Optionally, when it is detected that a corresponding RocketMQ cluster needs to be allocated to the topic to be allocated, a corresponding control may be triggered to generate a topic allocation request.

For different RocketMQ clusters, where there are different usage data, the data to be processed of RocketMQ clusters under different evaluation dimensions may be obtained. That is, for RocketMQ clusters, the data to be processed in multiple evaluation dimensions may be acquired. The data to be processed at this time may be usage data when RocketMQ clusters process different topics.

Specifically, upon detecting a need to allocate RocketMQ clusters for a topic to be allocated, a display interface may be triggered or based on some program code to generate a corresponding topic allocation request. After receiving the theme allocation request, the server may acquire to-be-processed data of all RocketMQ clusters under different evaluation dimensions.

In this embodiment, the reason and benefit of acquiring the data to be processed in different evaluation dimensions is that whether the RocketMQ clusters are suitable for reassigning the topics can be determined in different evaluation dimensions, so that the effect of reasonably assigning the topics to the RocketMQ clusters is achieved.

S120, for different evaluation dimensions, determining evaluation attributes under the evaluation dimensions through analyzing and processing the data to be processed under the evaluation dimensions.

It should be noted that the number of evaluation dimensions may include a plurality of evaluation dimensions, and the specific number of evaluation dimensions is adapted to the resources occupied by RocketMQ clusters during operation.

For each RocketMQ cluster, the data to be processed in its different evaluation dimensions can be obtained. Then, for RocketMQ clusters, the data to be processed of the RocketMQ cluster under different evaluation dimensions can be analyzed and processed to obtain the evaluation attribute of the RocketMQ cluster under each evaluation dimension.

Wherein the evaluation attribute is used to characterize whether the RocketMQ clusters can be used in the evaluation dimension, and optionally, the higher the evaluation attribute is, the description RocketMQ clusters can be used in the evaluation dimension, and conversely, the description RocketMQ clusters cannot be used in the evaluation dimension.

It should be noted that, the manner of determining the evaluation attribute of each RocketMQ clusters is the same, and in this embodiment, the determination of the evaluation attribute of one RocketMQ cluster is taken as an example for illustration.

Specifically, after obtaining the data to be processed of the RocketMQ clusters under different evaluation dimensions, the data to be processed under different evaluation dimensions may be analyzed and processed to obtain evaluation data corresponding to the evaluation dimensions.

S130, obtaining the target use attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ cluster.

For different evaluation dimensions, the weights occupied by RocketMQ clusters are different, so that the weight values corresponding to the different evaluation dimensions have certain differences. Of course, if the importance of RocketMQ clusters in different evaluation dimensions is almost the same, the weight values corresponding to different evaluation dimensions are the same. The target usage attribute is used to evaluate the usage attribute of RocketMQ clusters, alternatively, the higher the target usage attribute, the higher the availability of RocketMQ clusters, and conversely, the lower the availability of RocketMQ clusters. That is, the target usage attribute is used to evaluate whether RocketMQ clusters can use the evaluation value.

Specifically, after obtaining the evaluation attribute corresponding to the RocketMQ clusters under each evaluation dimension, the weight value corresponding to each evaluation dimension may be invoked. And obtaining the target use attribute of RocketMQ clusters according to the evaluation attributes corresponding to different evaluation dimensions and the corresponding weighting processing of the weight values.

And S140, determining a target RocketMQ cluster corresponding to the theme distribution request according to all the target use attributes, and distributing the target theme corresponding to the theme distribution request to the target RocketMQ cluster for processing.

Wherein the target RocketMQ cluster is a cluster selected from a plurality of RocketMQ clusters. The target topic is a topic to be allocated corresponding to the topic allocation request. The target RocketMQ cluster is a RocketMQ cluster that carries the target topic.

Specifically, after obtaining the target usage attributes of all RocketMQ clusters, the target RocketMQ clusters satisfying the condition may be screened from all RocketMQ clusters. A target topic corresponding to the topic allocation request is allocated into the target RocketMQ cluster to execute data corresponding to the target topic based on the target RocketMQ cluster.

According to the technical scheme provided by the embodiment of the invention, after a theme allocation request is received, to-be-processed data of all RocketMQ clusters under different evaluation dimensions can be obtained, then for to-be-processed data under different evaluation dimensions, evaluation attributes under the evaluation dimensions are determined through analysis and processing of to-be-processed data under the evaluation dimensions, and then the evaluation attributes of each RocketMQ cluster are weighted to obtain target use attributes of each RocketMQ cluster, the target RocketMQ cluster corresponding to the theme allocation request is determined according to all the target use attributes, so that the problem that the cost is high due to the need of manual participation when the target RocketMQ cluster corresponding to the theme to be allocated is determined according to a manual mode in the prior art is solved, and the problem that the theme utilization rate is overlarge due to certain errors possibly generated when the theme is allocated based on the manual mode is realized, and the target RocketMQ clusters which are most suitable for the theme to be allocated are determined according to actual application data of each RocketMQ cluster, thereby improving the resource utilization rate of the RocketMQ clusters and the effect of automatic allocation.

Fig. 2 is a flowchart of a theme distribution method applied to RocketMQ clusters according to an embodiment of the present invention, on the basis of the foregoing embodiment, the foregoing steps may be further refined, and a specific implementation manner of the foregoing step may be described in detail in this embodiment, where technical terms that are the same as or corresponding to the foregoing embodiment are not repeated in this embodiment.

As shown in fig. 2, the method includes:

And S210, when a theme allocation request is received, acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions.

In this embodiment, the different evaluation dimensions may include at least five evaluation dimensions. Optionally, the at least five evaluation dimensions include a central processor usage dimension, a memory usage dimension, a disk usage dimension, a traffic usage dimension, and an assigned theme property dimension. Correspondingly, the data to be processed corresponding to different evaluation dimensions can be occupied amount data corresponding to a central processing unit usage dimension, occupied memory data corresponding to a memory usage dimension, disk usage amount data corresponding to a disk usage dimension, the flow usage dimension comprises flow data occupied by the processed data, and the distributed theme attribute dimension comprises the distributed theme quantity.

It should be noted that, since RocketMQ clusters are deployed in the server, the target usage attribute of RocketMQ clusters can be evaluated from the server's perspective. Further, when the RocketMQ cluster is deployed on the server, a corresponding usage parameter may be configured for the RocketMQ cluster, and optionally, the usage parameter may be a central processor usage parameter, a memory usage parameter, a disk usage parameter, a traffic usage parameter, and a theme number parameter. Based on this, the target usage attributes of RocketMQ clusters can be determined from these several dimensions.

The cpu dimension may be a cpu usage dimension of a cpu usage space allocated thereto, the memory usage dimension may be a dimension corresponding to a memory allocated thereto, the disk usage dimension may be a dimension corresponding to a disk allocated thereto, the traffic usage dimension may be a dimension of a traffic allocated thereto, and the allocated subject dimension is a dimension of a subject allocated thereto.

The data to be processed corresponding to different evaluation dimensions can be obtained. The central processor uses the occupied amount data corresponding to the central processor with dimension of RocketMQ clusters. The data to be processed in the memory usage dimension is used occupied memory data corresponding to the memory allocated for RocketMQ clusters. The data to be processed of the disk usage dimension is disk usage data corresponding to the disk data distributed for RocketMQ clusters. Traffic uses traffic data that is occupied when processing processed data for traffic allocated for RocketMQ clusters in the dimension. The assigned topic attribute dimension is the number of assigned topics assigned to RocketMQ clusters.

After the data to be processed corresponding to the evaluation dimension is obtained, the data to be processed can be analyzed and processed to obtain evaluation attributes corresponding to different evaluation dimensions.

S220, determining the evaluation attribute of the central processing unit in the use dimension according to the occupied amount data of the central processing unit in the use dimension and the total amount data distributed for RocketMQ clusters.

It will be appreciated that the number of occupied volumes in the dimension used by the central processor may be obtained, as well as the total volume data pre-assigned for RocketMQ clusters. And calculating the ratio between the occupied amount data and the total amount data to obtain the evaluation attribute of the central processing unit under the use dimension.

S230, determining an evaluation attribute corresponding to the memory usage dimension according to the occupied memory data corresponding to the memory usage dimension and the total memory data allocated for RocketMQ clusters.

It is understood that the current occupied memory data of the memory usage dimension and the total memory data allocated for RocketMQ clusters may be obtained. And calculating the ratio between the occupied memory data and the total memory data to obtain the evaluation attribute of the memory usage dimension.

S240, determining the evaluation attribute of the disk usage dimension according to the disk usage data corresponding to the disk usage dimension and the disk usage allocated for RocketMQ clusters.

It is understood that disk usage data for the disk usage dimension and disk usage data assigned to RocketMQ clusters may be obtained. And calculating the ratio between the disk usage data and the disk usable data to obtain the evaluation attribute of the disk usage dimension.

S250, determining the evaluation attribute of the traffic usage dimension according to the traffic data corresponding to the traffic usage dimension and the total traffic data distributed for RocketMQ clusters.

It is understood that traffic usage data, i.e., traffic data, for the traffic usage dimension may be obtained, as well as total traffic data allocated for RocketMQ clusters. And calculating the ratio between the flow usage data and the total flow data to obtain the evaluation attribute of the flow usage dimension.

S260, determining the evaluation attribute of the assigned theme attribute dimension according to the assigned theme quantity of the assigned theme attribute dimension and the upper limit of the total theme quantity.

It should be noted that, for each RocketMQ clusters, the number of topics that can be carried is limited, and may be taken as the upper limit of the total number of topics.

It will be appreciated that the number of assigned topics currently assigned to the RocketMQ cluster and the upper limit on the total number of topics that the RocketMQ cluster can carry may be obtained. And calculating the ratio of the number of the allocated topics to the upper limit of the total number of the topics, and taking the ratio as the evaluation attribute of the attribute dimension of the allocated topics.

And S270, distributing corresponding use weights for the plurality of evaluation dimensions so as to determine RocketMQ target use attributes of the cluster based on the use weights corresponding to each evaluation dimension.

A weight value assigned in advance to each evaluation dimension may be obtained, where the weight value is a usage weight corresponding to each evaluation dimension.

Optionally, determining the target usage attribute of RocketMQ clusters based on the usage weight corresponding to each evaluation dimension includes obtaining the usage weight corresponding to each evaluation dimension, determining the attribute to be superimposed of the RocketMQ clusters in each evaluation dimension according to the evaluation attribute of the RocketMQ clusters in each evaluation dimension and the corresponding usage weight for each RocketMQ cluster, and obtaining the target usage attribute of the RocketMQ clusters through superposition processing of the multiple attribute to be superimposed of the RocketMQ clusters.

For each evaluation dimension, a product between the evaluation attribute and the usage weight in the evaluation dimension can be calculated, and the value obtained at this time is taken as the attribute to be superimposed. And obtaining the target use attribute of the RocketMQ cluster after the attribute to be superimposed of all the evaluation dimensions under the RocketMQ cluster is superimposed.

The target usage attribute for each RocketMQ cluster may be determined based on the manner described above.

And S280, using the RocketMQ cluster corresponding to the minimum target use attribute as a target RocketMQ cluster to distribute a target theme corresponding to the theme distribution request to the target RocketMQ cluster, so that the RocketMQ cluster processes the message corresponding to the target theme.

It is understood that for each RocketMQ cluster there is a target usage attribute corresponding to it. The smaller the target usage attribute, the more topics it can carry. Based on this, rocketMQ clusters with the smallest target usage attribute can be taken as the final target RocketMQ cluster.

According to the technical scheme provided by the embodiment of the invention, after a theme allocation request is received, to-be-processed data of all RocketMQ clusters under different evaluation dimensions can be obtained, then for to-be-processed data under different evaluation dimensions, evaluation attributes under the evaluation dimensions are determined through analysis and processing of to-be-processed data under the evaluation dimensions, and then the evaluation attributes of each RocketMQ cluster are weighted to obtain target use attributes of each RocketMQ cluster, the target RocketMQ cluster corresponding to the theme allocation request is determined according to all the target use attributes, so that the problem that the cost is high due to the need of manual participation when the target RocketMQ cluster corresponding to the theme to be allocated is determined according to a manual mode in the prior art is solved, and the problem that the theme utilization rate is overlarge due to certain errors possibly generated when the theme is allocated based on the manual mode is realized, and the target RocketMQ clusters which are most suitable for the theme to be allocated are determined according to actual application data of each RocketMQ cluster, thereby improving the resource utilization rate of the RocketMQ clusters and the effect of automatic allocation.

As an optional embodiment of the foregoing embodiment, how to assign a corresponding target RocketMQ cluster to a target theme may be described by a specific example, and a specific implementation manner of the embodiment may be described in detail by referring to this technical solution, where the technical terms that are the same as or corresponding to the foregoing embodiment are not repeated in this embodiment.

Firstly, obtaining data to be processed under a monitoring index. I.e. collecting all RocketMQ clusters of data to be processed under different monitoring indexes. Optionally, the monitoring index corresponds to an evaluation dimension. The evaluation dimension (monitor metrics) includes a central processor usage dimension (CPU usage dimension), a memory usage dimension, a disk usage dimension, and an assigned theme property dimension. After the data to be processed in the plurality of dimensions can be obtained, the data to be processed can be analyzed and processed, so that the evaluation attribute in different evaluation dimensions can be obtained.

In this embodiment, the evaluation attributes corresponding to different evaluation dimensions may be represented by percentages. Alternatively, the evaluation attribute of the usage dimension of the central processing unit may be dividing the occupied amount data by the total occupied amount data allocated for the RocketMQ cluster to obtain the CPU _usage, the value range of which is between [0,1], the evaluation attribute of the usage dimension of the Memory may be dividing the occupied Memory data by the total Memory data allocated for the RocketMQ cluster to obtain the Memory _usage, the value range of which is between [0,1], the evaluation attribute of the usage dimension of the Disk may be dividing the Disk usage data by the Disk usable amount allocated for the RocketMQ cluster to obtain the Disk _usage, the value range of which is also between [0,1], the evaluation attribute of the usage dimension of the flow may be dividing the used amount data of the RocketMQ cluster by the total flow data allocated for the RocketMQ cluster to obtain the TX _usage, the value range of which is between [0,1], the evaluation attribute of the allocated theme dimension may be dividing the number of allocated theme number allocated for the RocketMQ cluster by the total theme number TC that can be borne by the RocketMQ cluster to obtain the theme load factor _usage, the value range of which is between [0,1 ].

Based on the mode, the evaluation attribute under different monitoring indexes can be obtained.

And configuring corresponding weight values for different monitoring indexes. Namely, the weight occupied by each monitoring index is set.

In this embodiment, the usage weight corresponding to the usage dimension of the central processing unit is W _cpu, the usage weight corresponding to the usage dimension of the memory is W _memory, the usage weight corresponding to the usage dimension of the disk is W _disk, the usage weight corresponding to the usage dimension of the flow is W _tx, and the usage weight of the assigned theme attribute dimension is W _tc. The above usage weights are all within the range of [0,1], and the sum of all the usage weights is a preset value of 1.

After obtaining the evaluation attribute and the usage weight corresponding to each evaluation dimension, the target usage attribute of each RocketMQ cluster may be determined in the following manner.

Alternatively, the formula may be ：EV＝CPU_usageW_cpu+Memory_usage W_memory+Disk_usageW_disk+TX_usage W_tx+TC_usageW_tc;

Where EV is the target usage attribute of RocketMQ clusters.

Finally, after obtaining the target usage attributes of all RocketMQ clusters, the RocketMQ cluster corresponding to the smallest target usage attribute may be taken as the target RocketMQ cluster. And distributing the target theme corresponding to the theme distribution request to the target RocketMQ cluster with the minimum target use attribute, and creating the theme.

According to the technical scheme provided by the embodiment of the invention, the optimal RocketMQ clusters can be automatically and efficiently selected based on the topic distribution method for comprehensive weighted evaluation of the plurality of monitoring index dimensions, errors possibly generated in the manual evaluation process are effectively avoided, the manual evaluation cost is reduced, the topic access efficiency of the large-scale RocketMQ clusters is improved, and the resource utilization rate of the whole server is balanced.

The following is an embodiment of a topic distribution device applied to RocketMQ clusters, which belongs to the same inventive concept as the topic distribution method applied to RocketMQ clusters in the above embodiments, and details of the topic distribution device applied to RocketMQ clusters, which are not described in detail, may refer to the above embodiment of the topic distribution method applied to RocketMQ clusters.

Fig. 3 is a schematic structural diagram of a topic distribution device applied to RocketMQ clusters according to an embodiment of the present invention, where the device specifically includes a data to be processed acquisition module 310, an evaluation attribute determination module 320, a target usage attribute determination module 330, and a topic distribution module 340.

The system comprises a data acquisition module 310 to be processed, an evaluation attribute determining module 320, a target usage attribute determining module 330 and a theme distribution module 340, wherein the data acquisition module 310 to be processed is used for acquiring data to be processed of all RocketMQ clusters under different evaluation dimensions when a theme distribution request is received, the evaluation attribute determining module 320 is used for determining evaluation attributes under the evaluation dimensions by analyzing and processing the data to be processed under the evaluation dimensions for the different evaluation dimensions, the target usage attribute determining module 330 is used for obtaining target usage attributes of each RocketMQ cluster by weighting the evaluation attributes of each RocketMQ cluster, and the theme distribution module 340 is used for determining target RocketMQ clusters corresponding to the theme distribution request according to all the target usage attributes and distributing target themes corresponding to the theme distribution request to the target RocketMQ clusters for processing.

Based on the technical scheme, the evaluation dimension comprises a central processing unit use dimension, a memory use dimension, a disk use dimension, a flow use dimension and an allocated theme attribute dimension;

correspondingly, the data to be processed comprises occupied amount data corresponding to a using dimension of a central processing unit, occupied memory data corresponding to a using dimension of a memory, disk using amount data corresponding to a using dimension of a disk, the flow using dimension comprises flow data occupied by the processed data, and the attribute dimension of the distributed subject comprises the number of the distributed subjects.

On the basis of the above technical solutions, the evaluation attribute determining module includes:

The first determining unit is used for determining an evaluation attribute of the central processing unit in the use dimension according to the occupied amount data of the central processing unit in the use dimension and the total amount data distributed for the RocketMQ clusters;

A second determining unit, configured to determine an evaluation attribute corresponding to the memory usage dimension according to occupied memory data corresponding to the memory usage dimension and total memory data allocated for the RocketMQ clusters;

A third determining unit, configured to determine an evaluation attribute of the disk usage dimension according to disk usage data corresponding to the disk usage dimension and a disk usable amount allocated to the RocketMQ clusters;

A fourth determining unit, configured to determine an evaluation attribute of the traffic usage dimension according to traffic data corresponding to the traffic usage dimension and total traffic data allocated to the RocketMQ clusters;

and a fifth determining unit, configured to determine an evaluation attribute of the assigned theme attribute dimension according to the assigned theme number of the assigned theme attribute dimension and an upper limit of the total theme number.

Based on the above technical solutions, the apparatus further includes a usage weight obtaining module, configured to allocate corresponding usage weights to the multiple evaluation dimensions, so as to determine a target usage attribute of the RocketMQ clusters based on the usage weight corresponding to each evaluation dimension.

On the basis of the above technical solutions, the target usage attribute determining module includes:

The using weight acquisition unit is used for acquiring the using weight corresponding to each evaluation dimension;

The superposition attribute determining unit is used for determining the attribute to be superimposed of the RocketMQ clusters in each evaluation dimension according to the evaluation attribute of the RocketMQ clusters in each evaluation dimension and the corresponding use weight for each RocketMQ cluster;

and the target use attribute determining unit is used for obtaining the target use attribute of the RocketMQ cluster through superposition processing of the plurality of attributes to be superposed of the RocketMQ cluster.

On the basis of the above technical solutions, the topic allocation module is further configured to use, as a target RocketMQ cluster, a RocketMQ cluster corresponding to a target usage attribute with a minimum size, so as to allocate a target topic corresponding to the topic allocation request to the target RocketMQ cluster, so that the RocketMQ cluster processes a message corresponding to the target topic. According to the technical scheme provided by the embodiment of the invention, after a theme allocation request is received, to-be-processed data of all RocketMQ clusters under different evaluation dimensions can be obtained, then for to-be-processed data under different evaluation dimensions, evaluation attributes under the evaluation dimensions are determined through analysis and processing of to-be-processed data under the evaluation dimensions, and then the evaluation attributes of each RocketMQ cluster are weighted to obtain target use attributes of each RocketMQ cluster, the target RocketMQ cluster corresponding to the theme allocation request is determined according to all the target use attributes, so that the problem that the cost is high due to the need of manual participation when the target RocketMQ cluster corresponding to the theme to be allocated is determined according to a manual mode in the prior art is solved, and the problem that the theme utilization rate is overlarge due to certain errors possibly generated when the theme is allocated based on the manual mode is realized, and the target RocketMQ clusters which are most suitable for the theme to be allocated are determined according to actual application data of each RocketMQ cluster, thereby improving the resource utilization rate of the RocketMQ clusters and the effect of automatic allocation.

The theme distribution device applied to RocketMQ clusters provided by the embodiment of the invention can execute the theme distribution method applied to RocketMQ clusters provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the theme distribution method applied to RocketMQ clusters.

It should be noted that, in the embodiment of the subject distribution device applied to RocketMQ clusters, each unit and module included are only divided according to the functional logic, but not limited to the above division, so long as the corresponding functions can be implemented, and the specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.

Fig. 4 is a schematic structural diagram of a server according to an embodiment of the present invention. Fig. 4 illustrates a block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention. The server 12 shown in fig. 4 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.

As shown in fig. 4, the server 12 is in the form of a general purpose computing device. Components of server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that connects the various system components, including system memory 28 and processing units 16.

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Server 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32. The server 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, commonly referred to as a "hard disk drive"). Although not shown in fig. 4, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In such cases, each drive may be coupled to bus 18 through one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of the embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored in, for example, system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The server 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the server 12, and/or any devices (e.g., network card, modem, etc.) that enable the server 12 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 22. Also, the server 12 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, via a network adapter 20. As shown, network adapter 20 communicates with the other modules of server 12 via bus 18. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with server 12 including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, for example, implementing the steps of the topic allocation method applied to RocketMQ clusters provided in the first embodiment of the present invention, the method includes:

when a theme allocation request is received, acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions;

for different evaluation dimensions, determining evaluation attributes under the evaluation dimensions by analyzing and processing data to be processed under the evaluation dimensions;

Obtaining the target use attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ cluster;

And determining a target RocketMQ cluster corresponding to the theme allocation request according to all the target use attributes, and allocating the target theme corresponding to the theme allocation request to the target RocketMQ cluster for processing.

Of course, it will be appreciated by those skilled in the art that the processor may also implement the technical solution of the topic allocation method applied to RocketMQ clusters provided in any embodiment of the present invention.

The present embodiment provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the steps of the topic allocation method applied to RocketMQ clusters as provided in the previous embodiment of the present invention, the method comprising:

when a theme allocation request is received, acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions;

for different evaluation dimensions, determining evaluation attributes under the evaluation dimensions by analyzing and processing data to be processed under the evaluation dimensions;

Obtaining the target use attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ cluster;

And determining a target RocketMQ cluster corresponding to the theme allocation request according to all the target use attributes, and allocating the target theme corresponding to the theme allocation request to the target RocketMQ cluster for processing.

The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims (10)

1. A theme distribution method applied to RocketMQ clusters, comprising:

when a theme allocation request is received, acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions;

for different evaluation dimensions, determining evaluation attributes under the evaluation dimensions by analyzing and processing data to be processed under the evaluation dimensions;

Obtaining the target use attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ cluster;

And determining a target RocketMQ cluster corresponding to the theme allocation request according to all the target use attributes, and allocating the target theme corresponding to the theme allocation request to the target RocketMQ cluster for processing.

2. The method of claim 1, wherein the evaluation dimensions include a central processor usage dimension, a memory usage dimension, a disk usage dimension, a traffic usage dimension, and an assigned topic attribute dimension;

correspondingly, the data to be processed comprises occupied amount data corresponding to a using dimension of a central processing unit, occupied memory data corresponding to a using dimension of a memory, disk using amount data corresponding to a using dimension of a disk, the flow using dimension comprises flow data occupied by the processed data, and the attribute dimension of the distributed subject comprises the number of the distributed subjects.

3. The method of claim 2, wherein the determining the evaluation attribute in the evaluation dimension by analyzing the data to be processed in the evaluation dimension comprises:

Determining an evaluation attribute of the central processing unit in the use dimension according to the occupied amount data of the central processing unit in the use dimension and the total amount data distributed for the RocketMQ clusters;

determining an evaluation attribute corresponding to the memory usage dimension according to the occupied memory data corresponding to the memory usage dimension and the total memory data allocated for the RocketMQ clusters;

Determining an evaluation attribute of the disk usage dimension according to the disk usage data corresponding to the disk usage dimension and the disk usage allocated for the RocketMQ clusters;

determining an evaluation attribute of the traffic usage dimension according to the traffic data corresponding to the traffic usage dimension and the total traffic data distributed to the RocketMQ clusters;

and determining the evaluation attribute of the distributed theme attribute dimension according to the distributed theme quantity of the distributed theme attribute dimension and the upper limit of the total theme quantity.

4. The method of claim 1, wherein prior to said deriving the target usage attribute for each RocketMQ cluster by weighting the evaluated attribute for each RocketMQ cluster, the method further comprises:

and allocating corresponding use weights for the plurality of evaluation dimensions to determine the target use attribute of the RocketMQ cluster based on the use weight corresponding to each evaluation dimension.

5. The method of claim 1, wherein the obtaining the target usage attribute of each RocketMQ cluster by weighting the evaluation attribute of each RocketMQ clusters comprises:

Acquiring a use weight corresponding to each evaluation dimension;

For each RocketMQ cluster, determining the attribute to be superimposed of the RocketMQ cluster in each evaluation dimension according to the evaluation attribute of the RocketMQ cluster in each evaluation dimension and the corresponding use weight;

And obtaining the target use attribute of the RocketMQ cluster through superposition processing of the plurality of attributes to be superimposed of the RocketMQ cluster.

6. The method of claim 1, wherein determining the target RocketMQ cluster corresponding to the topic allocation request based on all target usage attributes comprises:

And taking the RocketMQ cluster corresponding to the minimum target use attribute as a target RocketMQ cluster, so as to distribute a target theme corresponding to the theme distribution request to the target RocketMQ cluster, and enabling the RocketMQ cluster to process a message corresponding to the target theme.

7. A theme distribution apparatus applied to RocketMQ clusters, comprising:

the to-be-processed data acquisition module is used for acquiring to-be-processed data of all RocketMQ clusters under different evaluation dimensions when a theme allocation request is received;

The evaluation attribute determining module is used for determining evaluation attributes in different evaluation dimensions by analyzing and processing the data to be processed in the evaluation dimensions;

the target use attribute determining module is used for obtaining the target use attribute of each RocketMQ cluster through weighting the evaluation attribute of each RocketMQ cluster;

And the theme distribution module is used for determining a target RocketMQ cluster corresponding to the theme distribution request according to all the target use attributes, and distributing the target theme corresponding to the theme distribution request to the target RocketMQ cluster for processing.

8. A service end is characterized in that, the server side comprises:

One or more processors;

a memory for storing one or more programs;

When executed by the one or more processors, causes the one or more processors to implement the topic allocation method of any one of claims 1-6 as applied to RocketMQ clusters.

9. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a topic allocation method as claimed in any one of claims 1-6 for use with RocketMQ clusters.

10. A computer program product comprising a computer program which, when executed by a processor, implements the topic allocation method of any of claims 1-6 applied to RocketMQ clusters.