CN111800354A - Message processing method and device, message processing device and storage medium - Google Patents

Abstract

The embodiment of the invention discloses a message processing method and device, message processing equipment and a storage medium. The method comprises the following steps: receiving a target message; determining item information of the target message; according to the project information and the message strategy, transmitting the target message by using a message queue corresponding to the project information; therefore, when the middleware transmits the message between the producer and the consumer of the message by using the message queue, the message to be processed is pertinently pressed into the corresponding message queue according to the message strategy; therefore, when the message transmission fails, the message queue with the message failure can be positioned at least according to the item of the message with the current transmission failure, and the positioning and problem troubleshooting of the message error queue are facilitated to be simplified.

Description

Message processing method and device, message processing device and storage medium

technical field

The present invention relates to the field of information technology, and in particular, to a message processing method and device, a message processing device and a storage medium.

Background technique

It has become an inevitable trend for messages to realize distributed services. Under the Openstack cloud platform, there are many message brokers, including Rabbitmq, Qpid, etc. The Rabbitmq tool is used to elaborate. At present, the implementation of Rabbitmq, as shown in Figure 1, the producer (publisher) is responsible for generating messages, the messages will be routed to the specified queue according to the binding rules, and then the subscriber (Subscriber) of the consumer (consumer) is responsible for sending messages from The queue pulls the message and processes it. Middleware (broker) interacts between message producers and consumers through exchanges and using queues.

The services under the Openstack cloud platform are reflected in two aspects in the use of messages:

On the one hand, it is the monitoring of messages. In the process of starting the service, it will create a connection to the Rabbitmq server through the specified target object and the type of messages to be monitored, so that the service can monitor messages;

On the other hand, the client message is sent. The client obtains the channel connected to the Rabbitmq server by initializing the remote procedure call (RemoteProcedure Call, RPC) process, specifying the target object of the message, and the channel of the context message, and further according to the message type.

Routing rules send messages to the message queue broker (Broker).

From the above two aspects, we know the flow of messages in the Openstack cloud platform. First, the client obtains the RPC client connection channel, and then encapsulates the message and sends it to the broker according to the message mode the client wants to send, and then the consumer service When listening to the arrival of a message, it will immediately pull the message and process it. If it is a synchronous message, it will return the processing result, otherwise it will not return.

The types of messages under Openstack are divided into three types: direct, fanout, and topic. When the service is started, it generally listens to two types of messages, fanout and topic, which means monitoring matching patterns and all messages sent to this route. Direct The type of message, which is generally used in the process, is used to receive the message return result of the synchronization request.

Under the framework of the existing message queue, first of all, once there is a problem with the platform, the operation and maintenance personnel are often at a loss, and the cognitive ability of the back-end service has not reached the level of research and development. Still can't troubleshoot the problem.

Secondly, it is also difficult to determine the specific location of the error in the message.

Finally, the message queue is an important performance bottleneck of the cloud platform. This is not a problem with the capability of the message queue itself, but is caused by the insufficient service capability of consumers.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention are expected to provide a message processing method and apparatus, a message processing device, and a storage medium

The technical scheme of the present invention is realized as follows:

A message processing method, including:

receive the target message;

determining item information of the target message;

According to the item information and the message policy, the target message is transmitted by using the message queue corresponding to the item information.

Based on the above scheme, the method further includes:

obtain attribute information of the target message;

According to the attribute information, determine whether to track the target message;

If it is determined to track the target message, the message record and/or link track information of the target message is recorded in the metadata.

Based on the above solution, the acquiring attribute information of the target message includes:

According to the tracking mode, the attribute information of the target message is acquired.

Based on the above solution, the obtaining attribute information of the target message according to the tracking mode includes one of the following:

Obtain the item information of the target message according to the item mode;

Obtain the routing information of the target message according to the routing mode;

Obtain the queue information of the target message according to the queue mode.

Based on the above solution, determining whether to track the target message according to the attribute information includes:

matching the attribute information with the attribute information set by the tracking;

If the matching is successful, it is determined to track the message record and/or link track information of the target message.

Based on the above solution, if it is determined to track the target message, record the message record and/or link track information of the target message into metadata, including at least one of the following:

In the project mode, if it is determined to track the target message, it will include at least one of the routing information, queue information, routing key, message content, message producer, message receiver, and message sending and receiving information of the target message. One, recorded in the metadata;

In the routing mode, if it is determined to track the target message, at least one of the sender, receiver, message content and message sending and receiving information of the target message is recorded in the metadata;

In the queue mode, if it is determined to track the target message, it will include at least one of the message content of the message, the producer of the message, the receiver of the message, the information about the sending and receiving of the message, and the item information of the message. in the metadata.

Based on the above scheme, the method further includes:

determining the tracking priority of the target message according to the attribute information;

According to the tracking priority, a recording mode for tracking the target message is determined.

Based on the above solution, the recording method includes at least one of the following:

information info method;

error error method;

Debug debug mode.

Based on the above scheme, the method further includes:

If the first cluster where the message queue of the target message is located is abnormal, the message queue and/or the middleware for message processing is switched to the second cluster that is mutually backup with the first cluster.

Based on the above scheme, the method further includes:

If the first cluster is abnormal, according to the recovery mechanism of the project where the target message is located, the abnormal message of the project where the target message is located is recovered through the second cluster.

Based on the above solution, the metadata includes at least one of the following data objects:

queue object;

request object;

route object;

resource object.

Based on the above solution, the queue object includes at least one of the following fields:

Message ID, routing key, message header, message body, message producer, message purpose;

The request object includes at least one of the following fields:

An identifier, the identifier includes one of the following: a request identifier, an item identifier, a queue identifier, a routing identifier, and a message identifier;

creation time;

The routing object includes at least one of the following fields:

routing identifier;

route name;

project name;

queue name;

The resource object includes at least one of the following fields:

The resource identifier of the middleware;

The node location where the host resource is located;

resource information of the queue;

Service information where the service resource is located;

port information.

Based on the above scheme, the method further includes:

Based on the query request, query the metadata to obtain the message record and/or link track information of the message to be tracked;

Information processing is performed according to the message record and/or the link trace information.

Based on the above solution, performing information processing according to the message record and/or link track information includes at least one of the following:

According to the message record and/or the link track information, carry out statistics of the number of messages of a project, statistics of the number of messages delivered per unit time, and evaluation of the service capability of the message server that processes the messages;

According to the message record and/or the link track information, at least one of locating an erroneous node, locating an erroneous service, and locating an error cause is performed.

A message processing device, comprising:

a first receiving module, configured to receive a target message;

a first determining module for determining item information of the target message;

The transmission module is configured to use the message queue corresponding to the item information to transmit the target message according to the item information and the message policy.

Based on the above solution, the device further includes:

a first acquiring module, used for acquiring attribute information of the target message;

a second determining module, configured to determine whether to track the target message according to the attribute information;

A recording module, configured to record the message record and/or link track information of the target message into metadata if it is determined to track the target message.

Based on the above solution, the first acquisition module is configured to acquire attribute information of the target message according to the tracking mode.

Based on the above solution, the first acquisition module is specifically configured to execute one of the following:

Obtain the item information of the target message according to the item mode;

Obtain the routing information of the target message according to the routing mode;

Obtain the queue information of the target message according to the queue mode.

Based on the above solution, the second determination module is configured to match the attribute information with the attribute information set for tracking; if the matching is successful, determine the message record and/or link track information of the target message to be tracked.

Based on the above solution, the recording module is specifically configured to execute one of the following:

In the project mode, if it is determined to track the target message, it will include at least one of the routing information, queue information, routing key, message content, message producer, message receiver, and message sending and receiving information of the target message. One, recorded in the metadata;

In the routing mode, if it is determined to track the target message, at least one of the sender, receiver, message content and message sending and receiving information of the target message is recorded in the metadata;

In the queue mode, if it is determined to track the target message, it will include at least one of the message content of the message, the producer of the message, the receiver of the message, the information about the sending and receiving of the message, and the item information of the message. in the metadata.

Based on the above solution, the device further includes:

a third determining module, configured to determine the tracking priority of the target message according to the attribute information;

The fourth determining module is configured to determine a recording method for tracking the target message according to the tracking priority.

Based on the above solution, the device further includes:

A switching module, configured to switch the message queue and/or the middleware for message processing to a second cluster that is mutually backup with the first cluster if the first cluster where the message queue of the target message is located is abnormal.

Based on the above solution, the device further includes:

The recovery module is configured to, if the first cluster is abnormal, recover the abnormal message of the project where the target message is located through the second cluster according to the recovery mechanism of the project where the target message is located.

Based on the above solution, the device further includes:

a query module, configured to query the metadata to obtain the message record and/or link track information of the message to be tracked based on the query request;

A processing module, configured to perform information processing according to the message record and/or the link track information.

Based on the above solution, the processing module is configured to execute at least one of the following:

According to the message record and/or the link track information, carry out statistics of the number of messages of a project, statistics of the number of messages delivered per unit time, and evaluation of the service capability of the message server that processes the messages;

According to the message record and/or the link track information, at least one of locating an erroneous node, locating an erroneous service, and locating an error cause is performed.

A message processing device, comprising:

memory;

A processor, connected to the processor, is configured to implement the message processing method provided by one or more of the foregoing technical solutions by executing the computer-executable instructions stored in the memory.

A computer storage medium storing computer-executable instructions; after the computer-executable instructions are executed by a processor, the message processing method provided by one or more of the foregoing technical solutions can be implemented.

In the message processing method and device, message processing device, and storage medium provided by the embodiments of the present invention, different message queues are set in sub-projects, so that when the middleware uses the message queue to transmit messages between message producers and consumers, it will According to the message policy, the to-be-processed message is pushed into the corresponding message queue in a targeted manner; in this way, when a message delivery failure occurs, at least the message queue where the message has failed can be located according to the project where the message currently failed to transmit is located, which is conducive to simplifying Error location of the message, and troubleshooting.

Description of drawings

1 is a schematic diagram of a transmission path of a message;

2 is a schematic flowchart of a message processing method according to an embodiment of the present invention;

3 is a schematic flowchart of another message processing method provided by an embodiment of the present invention;

4 is a schematic structural diagram of a message processing apparatus according to an embodiment of the present invention;

5 is a schematic flowchart of still another message processing method provided by an embodiment of the present invention;

6 is a schematic diagram of a message transmission provided by an embodiment of the present invention;

FIG. 7 is a schematic diagram of a data structure of metadata according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a correspondence relationship among a cloud platform, a message server, and a cluster according to an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be further elaborated below with reference to the accompanying drawings and specific embodiments of the description.

As shown in FIG. 2, this embodiment provides a message processing method, including:

Step S110: receiving the target message;

Step S120: determine the item information of the target message;

Step S130: According to the item information and the message policy, use a message queue corresponding to the item information to transmit the target message.

In this embodiment, the message processing method can be applied to the middleware of the cloud platform and the message server.

The middleware can be embedded in the cloud platform and/or message server in the form of a plug-in; in this way, the physical architecture of the cloud platform and/or message server does not need to be modified, and the compatibility with the existing technology is strong, and at the same time, the project can be implemented according to the project. Distinguish message queues and implement project-level message forwarding and transmission processing.

The middleware is located between the message producer (Provider) and the consumer (Consumer). Middleware can at least be used to relay messages that need to be pushed into the queue or extracted from the queue. For example, the middleware will push the message provided by the producer into the corresponding message queue according to the project where the message is located, so as to facilitate the consumer of the message to pull the message from the corresponding message queue.

The items described in this embodiment may be composed of one or more services. If a project consists of multiple services, these services are related services. For example, a software management item may include: the software's certification service, the software's upgrade service, etc.; the correlation of these several services is reflected in that they correspond to the same software. For another example, taking a mobile communication charging item as an example, the communication charging item may include: a communication billing service, a charging management service, and the like. Of course the above are just examples.

In a specific implementation process, the setting of the item can be processed as required. In some embodiments, one such item may consist of one or more sub-services under a large service.

In this embodiment, the target message may be any message that needs to be transmitted between a message producer and a consumer, for example, a trigger message generated by a certain thread, and the like.

In this embodiment, a message policy is preset, and the message policy can be run by the middleware. If a piece of information is received and needs to be forwarded between the cloud platform and the message server, the item information of the message will be determined first, and the item information will indicate the item described in the target message. Different projects are provided with different message queues, which are called message queues in this embodiment.

For example, project A and project B are respectively set with various message queues for sending and receiving messages, the queue of project A may be the A queue, and the queue of project B may be the B queue. When the middleware performs message transfer in step S130, the A queue may be used to process the message of item A, and the B queue may be used to process the message of item B.

In this way, the messages transmitted between the cloud platform and the message server will use different message queues to realize message processing such as message transfer of different projects, and realize the distinction of message queues at the project level. In this way, when a fault is located or messages accumulate , which can be adjusted according to the project to achieve targeted performance optimization of the cloud platform and/or message service, and improve the quality and efficiency of message transmission. If an error occurs during message transmission, at least according to the project where the target message is located, it is possible to determine which message queues have errors, not all queues that can receive all messages, so that it is impossible to locate the message queues where the error causes the message to fail to be transmitted normally. It is a difficult problem, which is conducive to the simplification of the error location of the target message during the transmission process.

In some embodiments, the method further includes:

The queue status information of the message queue is recorded by item. The queue status information can be used to indicate the maximum length of the corresponding message queue settings, the average occupied queue length, etc.; in this way, the queue corresponding to the corresponding item can be adjusted according to the queue status information in the future. parameters, the queue parameters here include but are not limited to: the number of queues, the maximum length of a single message queue, etc., to meet the message queues required by different projects to transmit messages.

In some embodiments, the method further includes:

When the message transmission of an item is abnormal, it can be determined whether the abnormality occurs in the message queue or in the consumer receiving the message according to the queue status message of the message queue corresponding to the item. In this way, the queue status information can also be used In the positioning abnormal. If the queue status information indicates that the message queue corresponding to the item is normal, and a message transmission failure occurs when there is no overflow phenomenon, the abnormality may occur in the consumer instead of the message queue.

In some embodiments, as shown in Figure 3, the method further includes:

Step S210: acquiring attribute information of the target message;

Step S220: According to the attribute information, determine whether to track the target message;

Step S230: If it is determined to track the target message, record the message record and/or link track information of the target message into metadata.

The attribute information may be determined according to the message header of the target message, or may be determined according to the attribute of the last processing node of the target message. For example, the middleware can be determined according to the attributes of the producers, and different producers belong to different services or different projects. In this way, the attribute information of the target message can be determined according to the attributes of the producers.

In some embodiments, some messages need to be tracked, and some messages may not be tracked. For example, important information is tracked, and unimportant information may not be tracked, so compared with tracking all target messages, the tracking load can be reduced and processing can be simplified.

The message records include but are not limited to:

The update record of the message; the update record of the message here includes: the update record of the package body of the message and the update record of the content of the message. The update of the encapsulation body of the message includes, but is not limited to, related records of encapsulation and decapsulation of the message. The update record of the message content includes: a record of operations such as addition, deletion and/or replacement of the message content.

In some embodiments, the message record further includes:

The sending and receiving status of the message, for example, the transmission time of the message from one node to the next node, whether the transmission is successful or not, and other information.

The link trace information is used to indicate the path of message transmission; according to the link trace information, the producer, transmitter and receiver of the message can be known.

By recording the above message records and/or link track information, it is convenient for subsequent error analysis in the process of message transmission failure.

In this embodiment, the message record and/or the link track information are stored in the metadata, and the persistent memory of the message can be realized by using the recording function of the metadata.

In some embodiments, the step S210 may include:

According to the tracking mode, the attribute information of the target message is acquired.

The attribute information required by different tracking modules is different. The tracking mode can be set according to the user's instruction or according to the properties of the item.

Due to different tracking modes, attribute information of different target messages needs to be acquired to determine whether the target message needs to be tracked. Therefore, in step S210, the attribute information is obtained from the target message itself or the sending and receiving node according to the currently set tracking mode.

In some embodiments, the step S210 may specifically include one of the following:

Obtain the item information of the target message according to the item mode;

Obtain the routing information of the target message according to the routing mode;

Obtain the queue information of the target message according to the queue mode.

In this embodiment, the tracking modes may be divided according to different granularities. For example, in the project mode, all target messages of the project need to be tracked. At this time, when judging whether a target message needs to be tracked, the project information of the current target message needs to be obtained first.

The acquired item information is matched with the item information of the item that needs to be tracked, and if the matching is consistent, it can be determined that the current target message is the message that needs to be tracked.

Routing mode can be used to qualify traces that trace information that travels through a specific routing path. If the current tracking mode is the routing mode, the routing information of the target message needs to be acquired. After the routing information of the current target message is acquired, it is matched with the routing message that needs to be traced. If the matching is consistent, it can be determined that the current target message is the target message that needs to be traced.

If the current tracking mode is a queue mode with a queue as the granularity, you can obtain the queue information that the target message needs to be pushed into the queue, and match the queue information with the queue information that needs to be tracked in advance. If the matching is consistent, it can be considered that the current The target message is the target message that needs to be tracked.

In some embodiments, the tracking mode is not limited to the item mode, routing mode, and queue mode, and may also include a service-granularity service mode. In the service mode, the service information of the service where the target message is located can be obtained, and by matching with the service information of the service to be tracked, if the match is consistent, the corresponding target message is determined to be the target message to be tracked.

There are multiple tracking modes, and in some specific cases, multiple tracking modes can coexist, but if a message hits multiple tracking modes at the same time, only one tracking is performed, and repeated redundant tracking is not involved. For example, the current tracking mode includes both the queue mode and the item mode. If a target message needs to track both the item and the queue, it can be considered that the target message needs to be tracked, but it is only tracked once, and not tracked in both modes at the same time.

In a word, the step S220 may include: matching the attribute information with the attribute information set by the tracking; if the matching is successful, determining the message record and/or the link track information of the tracking target message.

The relevant description of the message record and the link track information can be found in the foregoing section for details, and will not be repeated here.

In this embodiment, various tracking information, such as message records and/or link track information obtained when tracking the target message, can be recorded in the metadata to achieve long-term records and facilitate query.

In some embodiments, the step S220 may specifically include at least one of the following:

In the project mode, if it is determined to track the target message, it will include at least one of the routing information, queue information, routing key, message content, message producer, message receiver, and message sending and receiving information of the target message. One, recorded in the metadata;

In the routing mode, if it is determined to track the target message, at least one of the sender, receiver, message content and message sending and receiving information of the target message is recorded in the metadata;

In the queue mode, if it is determined to track the target message, it will include at least one of the message content of the message, the producer of the message, the receiver of the message, the information about the sending and receiving of the message, and the item information of the message. in the metadata.

In the project mode, the routing information describing the routing path of the target message, the queue information of the target message being pushed into the queue, the routing key used in routing, the message content, the message producer, the message receiver, and each transit One or more of the information such as the sending and receiving situation information on the routing node are all recorded in the metadata.

The sending and receiving information may include at least one of the following:

Indicate whether the sending and receiving status information of the target message is successfully received;

Indicate the first time information for receiving the target message;

Indicates second time information for sending the target message.

In the routing mode, and the corresponding target message needs to be tracked, one or more of the producer, receiver, and sending and receiving information of the target message will be recorded in the metadata.

In the queue mode, at least one of the message producer, the message receiver and the item information of the message can be written into the metadata.

In still other embodiments, in the queue mode, the routing information and/or the service message of the target message may also be written into the metadata.

In some embodiments, in any one of the item mode, the routing mode and the queue mode, the identification information of the middleware that processes the target message, the resource information of the One or more kinds of information such as node information of processing information and port information of transmission ports are written into the metadata.

The resource information may include: resource identifiers of computing resources, storage identifiers of storage resources, and bandwidth identifiers of bandwidth resources. The node information is used to indicate a node identifier of a physical node and/or a virtual node that processes the to-be-processed information. The physical nodes include but are not limited to hosts or clusters. The virtual nodes include but are not limited to virtual machines or virtual machine clusters.

In some embodiments, the message record and/or link trace information of the target message stored in the metadata is at least used to determine the path information of the error-prone route segment, or may be used to determine the complete route of the target message Routing information for the path.

In this way, various information required for subsequent analysis based on error exclusion in error analysis is facilitated.

In some embodiments, the step S220 further includes:

determining the tracking priority of the target message according to the attribute information;

According to the tracking priority, a recording mode for tracking the target message is determined.

In this embodiment, the tracking priority of the target message is also determined according to the attribute information of the target message.

Target messages of different attribute information have different tracking priorities. The higher the tracking priority, the more detailed recording method is used to record the target message; in this way, the detailed tracking of the target message with high priority is realized.

In this way, according to the user settings or the importance of the message, a corresponding recording method can be used to record the tracking information, thereby reducing unnecessary acquisition and/or recording of the tracking information.

In some embodiments, there are multiple recording manners, and several optional manners are provided below:

information info method;

error error method;

Debug debug mode.

In this embodiment, the record detail level of the error mode is higher than that of the information method; the record detail level of the debug method is higher than that of the error method.

For example, in some cases, the information mode only records some general information in the process of tracking the target message; while the error mode records not only the general information, but also error information, such as the wrong routing node, error type and other error information. In addition to recording general information and error information, the debugging method also records some description information, which is a further detailed description of one or more information points in the general information and/or error information, or, general information and / or details of other information than error messages.

The general information includes but is not limited to: queue information, routing information, item information, message type, service where the message is located, and the like.

In other embodiments, the method further includes:

If the first cluster where the message queue of the target message is located is abnormal, the message queue and/or the middleware for message processing is switched to the second cluster that is mutually backup with the first cluster.

In this embodiment, in order to reduce the problem that information cannot be transmitted normally due to the exception of a single middleware or a single message queue, in this embodiment, backups are formed for both the middleware and the message queue. For example, configure the primary middleware and the primary message queue on the first cluster, and configure the backup middleware and backup message queue on the second cluster. If the first cluster is abnormal, it will automatically switch to the middleware and backup messages on the second cluster. Message processing on the queue.

In some embodiments, the backup middleware and the backup message queue are pre-configured on the second cluster, but in some cases, the backup middleware and the backup message queue may be temporarily pre-configured when an abnormality of the primary middleware and the main message queue is detected. configured on the second cluster.

In some embodiments, the backup of middleware and message queues can be at item granularity. For example, when different projects use different middleware and message queues, if a transmission exception occurs in project A, the message transmission of the entire project A can be migrated to the second backup cluster.

In still other embodiments, one middleware may process target messages of multiple items, and the message queue is divided into items. In this case, only the main-backup switching of the message queue and the middleware may be separated.

There are many ways to judge the abnormality of the first cluster. The following are several optional ways:

The message sent by the first node is not received when the second node times out, for example, the message sent by the client is not received by the receiver within a set time such as 60 seconds;

The message retransmission rate reaches the retransmission threshold.

In some embodiments, the method further includes:

If the first cluster is abnormal, according to the recovery mechanism of the project where the target message is located, the abnormal message of the project where the target message is located is recovered through the second cluster.

In this embodiment, in order to provide services normally, a recovery mechanism can be set for each project. If the message of a certain service in the project is not successfully transmitted, the recovery mechanism is automatically activated after waiting for a specific time or once it is determined that the transmission is abnormal. retransmission, etc. In this way, after migrating the middleware and/or the message queue to the second cluster, supplementary provision of services not provided during the abnormal time period can be implemented based on the recovery mechanism.

In this embodiment, the metadata sets different data objects. Different recording methods can record the message recording of the target message and/or the tracking information such as link track information from one or more data objects. The metadata includes at least one of the following data objects:

queue object;

request object;

route object;

resource object.

The data object includes at least one field.

Different data objects can include a different field.

For example, the queue object includes at least one of the following fields:

Message ID, routing key, message header, message body, message producer, message purpose.

The message identifier is used to record information such as the sequence number of the message that uniquely identifies the message.

A routing key may be a keyword for routing queries for messages.

The message header may include: a header of the target message.

The message body may include: the body of the target message.

The producer of the message may be the source of the message.

The destination of the message can be the recipient or consumer of the message.

The request object includes at least one of the following fields:

An identifier, the identifier includes one of the following: a request identifier, an item identifier, a queue identifier, a routing identifier, and a message identifier;

Creation time.

The creation time field here can be used to store the creation time of various data objects in the metadata.

The routing object includes at least one of the following fields:

routing identifier;

route name;

project name;

queue name;

The resource object includes at least one of the following fields:

The resource identifier of the middleware;

The node location where the host resource is located;

resource information of the queue;

Service information where the service resource is located;

port information.

In some embodiments, the method further includes: based on the query request, querying the metadata to obtain message records and/or link track information of the message to be tracked; and, according to the message records and/or link track information, performing Information processing.

After the tracking is performed, various tracking information is recorded using the metadata, and the metadata can be subsequently queried based on the query request to obtain the required information for information processing.

Specifically, the performing information processing according to the message record and/or the link track information includes: performing statistics on the number of messages in a project, unit time unit time, according to the message record and/or the link track information. Statistics on the number of transmitted messages, and evaluation of the service capability of the message server that processes the messages; at least one of faulty node location, faulty service location, and error cause location is performed according to the message record and/or link trajectory information .

As shown in FIG. 4 , this embodiment provides a message processing apparatus, including:

a first receiving module 110, configured to receive a target message;

a first determining module 120, configured to determine item information of the target message;

The transmission module 130 is configured to use the message queue corresponding to the item information to transmit the target message according to the item information and the message policy.

In some embodiments, the first receiving module 110, the first determining module 120, and the transmitting module 130 can all be program modules; after the program modules are executed by the processor, they can realize the reception of the aforementioned target message, the project information The determination of the target message and the transmission of the target message.

In other embodiments, the first receiving module 110 , the first determining module 120 and the transmitting module 130 may be software-hardware combination modules; the software-hardware combination modules may include various programmable arrays, the programmable arrays It can be a complex programmable array or a field programmable array, etc.

In still other embodiments, the first receiving module 110, the first determining module 120 and the transmitting module 130 may be pure hardware modules; the pure hardware modules include but are not limited to application specific integrated circuits.

In some embodiments, the apparatus further includes:

a first acquiring module, used for acquiring attribute information of the target message;

a second determining module, configured to determine whether to track the target message according to the attribute information;

A recording module, configured to record the message record and/or link track information of the target message into metadata if it is determined to track the target message.

In some embodiments, the first obtaining module is configured to obtain attribute information of the target message according to a tracking mode.

In some embodiments, the first obtaining module is specifically configured to execute one of the following:

Obtain the item information of the target message according to the item mode;

Obtain the routing information of the target message according to the routing mode;

Obtain the queue information of the target message according to the queue mode.

In some embodiments, the second determining module is configured to match the attribute information with the attribute information set for tracking; if the matching is successful, determine the message record and/or link track information of the target message to be tracked .

In some embodiments, the recording module is specifically configured to execute one of the following:

In the project mode, if it is determined to track the target message, it will include at least one of the routing information, queue information, routing key, message content, message producer, message receiver, and message sending and receiving information of the target message. One, recorded in the metadata;

In the routing mode, if it is determined to track the target message, at least one of the sender, receiver, message content and message sending and receiving information of the target message is recorded in the metadata;

In the queue mode, if it is determined to track the target message, it will include at least one of the message content of the message, the producer of the message, the receiver of the message, the information about the sending and receiving of the message, and the item information of the message. in the metadata.

In some embodiments, the apparatus further includes:

a third determining module, configured to determine the tracking priority of the target message according to the attribute information;

The fourth determining module is configured to determine a recording method for tracking the target message according to the tracking priority.

In some embodiments, the recording manner includes at least one of the following:

information info method;

error error method;

Debug debug mode.

In some embodiments, the apparatus further includes:

A switching module, configured to switch the message queue and/or the middleware for message processing to a second cluster that is mutually backup with the first cluster if the first cluster where the message queue of the target message is located is abnormal.

In some embodiments, the apparatus further includes:

The recovery module is configured to, if the first cluster is abnormal, recover the abnormal message of the project where the target message is located through the second cluster according to the recovery mechanism of the project where the target message is located.

In some embodiments, the metadata includes at least one of the following data objects:

queue object;

request object;

route object;

resource object.

In some embodiments, the queue object includes at least one of the following fields:

Message ID, routing key, message header, message body, message producer, message purpose;

The request object includes at least one of the following fields:

An identifier, the identifier includes one of the following: a request identifier, an item identifier, a queue identifier, a routing identifier, and a message identifier;

creation time;

The routing object includes at least one of the following fields:

routing identifier;

route name;

project name;

queue name;

The resource object includes at least one of the following fields:

The resource identifier of the middleware;

The node location where the host resource is located;

resource information of the queue;

Service information where the service resource is located;

port information.

In some embodiments, the apparatus further includes:

a query module, configured to query the metadata to obtain the message record and/or link track information of the message to be tracked based on the query request;

A processing module, configured to perform information processing according to the message record and/or the link track information.

In some embodiments, the processing module is configured to perform at least one of the following:

According to the message record and/or the link track information, carry out statistics of the number of messages of a project, statistics of the number of messages delivered per unit time, and evaluation of the service capability of the message server that processes the messages;

According to the message record and/or the link track information, at least one of locating an erroneous node, locating an erroneous service, and locating an error cause is performed.

Several specific examples are provided below in conjunction with any of the above-mentioned embodiments:

Example 1:

A message queue strategy proposed in this example is based on the combined use of middleware technology and the ability of middleware to notify messages.

The system architecture of the cloud platform message strategy proposed in this example is shown in Figure 5, which mainly includes four parts, namely cloud platform, message strategy, middleware, and message server.

A cloud platform is a platform that provides services, including two executive bodies: Consumer and Producer.

Middleware is the interface layer that encapsulates messages between cloud platforms and message servers.

A message server refers to a message server that provides message storage and forwarding.

In the initialization process, first the cloud platform service starts, acts as the role of Consumer, establishes a channel to the message server, and notifies the message server to establish routing, queue and other information; Provide services.

When a user initiates a request through the cloud platform, the client sends a message to the corresponding service queue with the help of the producer (Publisher) object in the cloud platform. Therefore, after the service startup is completed, the flow of messages in the above-mentioned framework is as shown in the steps in Figure 5; the specific steps can be as follows:

Step 1: The producer (Publisher) establishes a connection to the message server, and the message passes through the middleware interface;

Step 2: The middleware filters the message processing based on the message policy;

Step 3: Send to the message server again;

Step 4: The message server pushes the message into the message queue after routing;

Step 5: The cloud platform service listens to the arrival of a message in the queue, and the middleware pulls the message from the message queue, which will be filtered by the message policy;

In step 6, the message passes through the interface of the middleware and is finally transmitted to the corresponding cloud platform service, and the service completes the processing of the message.

The detailed design frame diagram of the message strategy proposed in this example is shown in Figure 6. The message policy part is added to cloud platforms such as the Openstack framework in the form of plug-ins, so that it is compatible with the previous version of the message usage, which is flexible and convenient.

The configurable plug-in is mainly designed with two parts, metadata and policy.

Metadata is used to collect metadata information in producer and consumer connections, and needs to be stored in persistent storage.

Policy is used to set policies, such as specifying items or specifying queues or specifying services.

The above two parts need to be based on the notification technology in the Openstack platform and the support of the Trace plug-in proposed in this example. Based on these two technologies, the purpose is to fully explore valuable information and realize the exploration of messages, so as to analyze the service capabilities of consumers.

Notification technology is a notification capability under the Openstack platform, which can record user operation behavior and classify it. The Trace plug-in will record all message links specified to the pattern behavior according to the pattern set by the user, including message routing, routing keys, sending queues, sending and receiving messages, etc.

In this example, the proposed Metadata design includes four objects, namely queue, request, router, and resource, as shown in Figure 7. Queue objects include: Id message id, routing_key message routing key, headers message header, body message body, producer of from_id message, message resource received by to_id, type received or sent message type.

The request object includes: the unique identifier of the id request object. This value corresponds to the request id in the openstack project. If not, it will be randomly generated. The queue id corresponding to queue_id corresponds to the id in the queue object, and the message unique id contained in the msg_id message. , the creation time of the create_time request, this value can analyze the message according to the time period.

The router object includes: the unique id of the id route, the exchange_name route name, the project_name project name, and the queue_name queue name. With these designs, the information of the queue resources can be counted according to the content of the project.

The resource object includes: the id flag of the id internal resource, the node location where the host resource appears, the service where the service resource is located, and the port on which the port service listens. With the design of this object, the complete link trace of the message can be counted at any time.

The concept of policy proposed in this example is the core content of the policy. Policy can be set according to the needs of users, and it is designed to count message queues, track the priority of Notifications, track the routing of specified items, track all messages sent and received at a certain time, query message links, and switch between active and standby message clusters.

The specific implementation steps of this example are to count the metadata information of the resource according to the policy policy set by the user, and then make a system response from the metadata information. The policy of the policy mentioned in this example is based on the collected valuable information, which is stored in the format designed by the metadata.

In this example, the statistical message queue in Policy, as shown in Figure 5, the existing technical solution does not include the message policy part, that is, after the service in the cloud platform is started, it listens to the queue in the message server. When the message arrives, it is processed. There are many projects on the cloud platform, and there are many services for each project. Therefore, the number of queues in the message server will reach a lot, the display is not intuitive, and the user cannot directly confirm the status of each queue.

The message strategy proposed in this example is mainly implemented by recording the router object in the metadata. The service in the cloud platform establishes a connection to the message queue through the consumer object, and performs initialization work, which is initiated by the service of the cloud platform. Operations for establishing queues, routes, and other information. The middleware extracts project information, routing information, and queue information from the context, and records it in the data of the router object, but does not record duplicate data. The queue information collected at this time is saved in the metadata layer, so that the system According to the persistent data storage of metadata, according to the project name or queue name, the corresponding information can be queried, that is, the system can have the ability to divide the queue, making the positioning of the message intuitive and simple.

In this example, the priority of tracking Notification in Policy is based on the record level set by the Openstack project, including info, error, debug and other parts. Users can set their own policies, such as the error level, so that All error message records in the system and complete link trace information are collected. These traces are completed by the trace module designed in this example. With these messages, users can quickly query the location of the error message, the service in error, and find out the root cause. . The specific implementation process is as follows:

Notification is an existing notification message capability of the cloud platform, and can be recorded according to the type of the message. For example, the message of the wrong operation will be sent to the error record queue, but the related technology will only store the content of the message in the queue. And messages lost due to other reasons will not be recorded or notified. The executor of the trace module designed in this example is middleware, and three modes are set, including three modes: project, message routing, and queue name. If the project mode is set, the message will be sent according to the publisher object of the current cloud platform. The project information in the file is compared with the currently set trace project. If so, continue to track the content of the middleware, and send the publisher to the message routing, message queue, routing key, message content, and message production in the message server. The complete path of the sender, the receiver of the message, and the sending and receiving status is recorded in the metadata layer, otherwise, it is not recorded.

If the queue mode is set, the message sent by the publisher object will be distributed to the queue of the specified message, and check whether the currently set trace queue and the message queue sent by the publisher are consistent. If they are consistent, then through the middleware, in the metadata layer, Extract the relevant content of the message, including the content of the message, the producer of the message, the receiver of the message, the sending and receiving of the message, and the project to which the message belongs.

If the queue routing mode is set, the messages sent by the pulisher object will have corresponding message routing information in the message server. Whether this routing information is the same as the currently set trace queue routing, if it is the same, record all the routing information that passes through The message of this route, and the sender, receiver, message content, and sending and receiving of the message are recorded. Otherwise, it is not recorded.

With the record of the message in the metadata layer, the user can query the location of the message error, the error service, and find out the root cause.

For example, in the metadata, according to the trace mode, the specific message content is recorded in several objects of the metadata, and the user can query the link of the message and the service location of the error according to the request id or the id of the message. For example, if the user queries the id of the specified message, the corresponding queue object is found through the content of msg_id in the request object in the metadata, and the routing key, message content, message source id and other information of the message are recorded in the queue object. According to the message source id, the corresponding To the id in the resource object, you can find the service of the message, so as to know the node, port and other information where the service is located, which is convenient for locating the problem.

In this example, Policy tracks the sending and receiving of all messages at a certain time. This is based on the trace module, which counts the sending and receiving of all messages and the link of messages. It is mainly used to count the upper limit of the pressure of the system, which is convenient for architects to make reasonable arrangements for deployment. Nodes and Services. For example, in the Openstack project, before the project goes live, a system stress test needs to be done. This function is useful. You can design 8-hour time-dimension statistics. All messages passing through the message server are recorded and stored in the metadata. Among several objects of the layer, you can perform an interval screening based on the create_time field in the request object of the metadata to find the number of messages in this interval, and according to the message id, correspond to the queue object in the metadata object to get the source id, and then corresponding to the resource object, find the corresponding service, so as to know the pressure time, pressure location and other information of the project.

In this example, the query message link in Policy refers to finding the queue object according to the message record collected by the metadata layer, according to the message ID or request ID, and further finding the resource according to the message producer ID The service information in the object is presented to the user. Through this function, it is convenient for the user to locate the cause of the request error.

In this example, the route of the specified item is tracked in Policy, and the relevant routing information is found according to the router object in the metadata, and then these routing messages are monitored through the trace module, and the content, route, queue, and related information in the message are recorded. The information of the service and the content of the service are recorded in the resource object of the metadata layer. With this data, the ability detection and statistics of the project can be realized, and valuable analysis data can be provided for users or developers, and the information detected by trace, Big data analysis can be done in time series. The following example illustrates the specific implementation process. It is set to trace all the routes of the nova project in the Openstack platform. The trace module will record all the message links passing through all the routes of this project. With the content of these messages, the core services of the nova project can be counted. The number of messages can be quantified as how many messages per hour. With this data, the capacity of the service can be roughly estimated, which is reflected in the number of requests processed by each service per unit time.

Finally, this example proposes that the policy provides cluster master-slave switching. In the Openstack project, the message queue cluster is divided into multiple resource pools, which are masters and backups of each other, and the control of these resource pools is handed over to the policy. The message queue parameter is configured as the VIP address of the cluster. Whenever there is a problem with the cluster in use, it can automatically switch to the standby cluster, and then the faulty cluster can complete self-repair. The ability of these repairs depends on some monitoring scripts. Wait until The cluster returns to normal, and it joins the cluster again, waiting for the next use. The specific implementation method is as follows:

In the Openstack cloud platform project, the service of each project can set the address of the message queue server. The solution proposed in this example, as shown in Figure 8, adds policy control to the address of the message server, and combines multiple active and standby clusters. It is designed with a virtual IP (Virtual Internet Protocol, VIP) address, and when a certain type of service in the cloud platform cannot be connected or the message is blocked in a certain cluster, it can switch to another cluster in real time. The method of determination If the service fails to connect to the cluster during the startup process or the client connects to the message queue, or the client times out after sending a message (the default is to wait for 60s), it is considered that there is a problem with the cluster connected to the service. The Policy policy can detect problems in real time and re-assign a cluster to the service from the cluster. However, it should be noted that the connection between the cloud platform and the message server needs to be reconnected due to cluster switching. However, the content of the queues in the previous cluster does not need to be synchronized for the following reasons: the project on the cloud platform has a retry mechanism, that is, lost messages can be sent repeatedly; the queue does not need to be synchronized, and the process of rebuilding the connection is It is equivalent to the process of service reconstruction, which will automatically create queues, routes and other related information. In addition, the policy policy needs to clear the problem queue in the previous cluster. In order to ensure that the previous cluster can continue to be recycled in the future. The specific cleaning method needs to use the API interface of the back-end message server. In Fig. 8, there are clusters 1 to n, the primary backup middleware and message queue of a project, which can be arranged on any two clusters from cluster 1 to cluster n.

This embodiment provides a message processing device, including:

memory;

A processor, connected to the processor, configured to implement the message processing method provided by any of the foregoing technical solutions by executing the computer-executable instructions stored in the memory, for example, as shown in FIG. 2 , FIG. 3 or FIG. 5 The message processing method shown.

An embodiment of the present invention provides a computer storage medium, where computer-executable instructions are stored in the computer storage medium; after the computer-executable instructions are executed by a processor, the message processing method provided by any of the foregoing technical solutions can be implemented, for example, One or more of the message processing methods shown in FIG. 2 , FIG. 3 and FIG. 5 can be performed.

The computer storage medium provided in this embodiment may be a non-transitory storage medium.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.

The unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may all be integrated into one processing module, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integration The unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, execute Including the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk and other various A medium on which program code can be stored.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (27)

1. a message processing method, is characterized in that, comprises: receive the target message; determining item information of the target message; According to the item information and the message policy, the target message is transmitted by using the message queue corresponding to the item information. 2. The method according to claim 1, wherein the method further comprises: obtain attribute information of the target message; According to the attribute information, determine whether to track the target message; If it is determined to track the target message, the message record and/or link track information of the target message is recorded in the metadata. 3. The method according to claim 2, wherein the acquiring the attribute information of the target message comprises: According to the tracking mode, the attribute information of the target message is acquired. 4. The method according to claim 3, wherein the acquiring attribute information of the target message according to the tracking mode comprises one of the following: Obtain the item information of the target message according to the item mode; Obtain the routing information of the target message according to the routing mode; Obtain the queue information of the target message according to the queue mode. 5. The method according to claim 2, wherein the determining whether to track the target message according to the attribute information comprises: matching the attribute information with the attribute information set by the tracking; If the matching is successful, it is determined to track the message record and/or link track information of the target message. 6. The method according to any one of claims 2 to 5, wherein, if the target message is determined to be tracked, the message record and/or link track information of the target message is recorded in metadata , including at least one of the following: In the project mode, if it is determined to track the target message, it will include at least one of the routing information, queue information, routing key, message content, message producer, message receiver, and message sending and receiving information of the target message. One, recorded in the metadata; In the routing mode, if it is determined to track the target message, at least one of the sender, receiver, message content and message sending and receiving information of the target message is recorded in the metadata; In the queue mode, if it is determined to track the target message, it will include at least one of the message content of the message, the producer of the message, the receiver of the message, the information about the sending and receiving of the message, and the item information of the message. in the metadata. 7. The method according to any one of claims 2 to 5, wherein the method further comprises: determining the tracking priority of the target message according to the attribute information; According to the tracking priority, a recording mode for tracking the target message is determined. 8. The method of claim 7, wherein The recording method includes at least one of the following: information info method; error error method; Debug debug mode. 9. The method of claim 1, wherein the method further comprises: If the first cluster where the message queue of the target message is located is abnormal, the message queue and/or the middleware for message processing is switched to the second cluster that is mutually backup with the first cluster. 10. The method according to claim 9, wherein the method further comprises: If the first cluster is abnormal, according to the recovery mechanism of the project where the target message is located, the abnormal message of the project where the target message is located is recovered through the second cluster. 11. The method according to any one of claims 2 to 5, wherein, The metadata includes at least one of the following data objects: queue object; request object; route object; resource object. 12. The method of claim 11, wherein the queue object includes at least one of the following fields: Message ID, routing key, message header, message body, message producer, message purpose; The request object includes at least one of the following fields: An identifier, the identifier includes one of the following: a request identifier, an item identifier, a queue identifier, a routing identifier, and a message identifier; creation time; The routing object includes at least one of the following fields: routing identifier; route name; project name; queue name; The resource object includes at least one of the following fields: The resource identifier of the middleware; The node location where the host resource is located; resource information of the queue; Service information where the service resource is located; port information. 13. The method according to any one of claims 2 to 5, wherein the method further comprises: Based on the query request, query the metadata to obtain the message record and/or link track information of the message to be tracked; Information processing is performed according to the message record and/or the link trace information. 14. The method of claim 13, wherein The performing information processing according to the message record and/or the link track information includes at least one of the following: According to the message record and/or the link track information, carry out statistics of the number of messages of a project, statistics of the number of messages delivered per unit time, and evaluation of the service capability of the message server that processes the messages; According to the message record and/or the link track information, at least one of locating an erroneous node, locating an erroneous service, and locating an error cause is performed. 15. A message processing device, comprising: a first receiving module, configured to receive a target message; a first determining module for determining item information of the target message; The transmission module is configured to use the message queue corresponding to the item information to transmit the target message according to the item information and the message policy. 16. The apparatus of claim 15, wherein the apparatus further comprises: a first acquiring module, used for acquiring attribute information of the target message; a second determining module, configured to determine whether to track the target message according to the attribute information; A recording module, configured to record the message record and/or link track information of the target message into metadata if it is determined to track the target message. 17 . The apparatus according to claim 16 , wherein the first obtaining module is configured to obtain attribute information of the target message according to a tracking mode. 18 . 18. The apparatus according to claim 17, wherein the first obtaining module is specifically configured to execute one of the following: Obtain the item information of the target message according to the item mode; Obtain the routing information of the target message according to the routing mode; Obtain the queue information of the target message according to the queue mode. 19 . The device according to claim 16 , wherein the second determining module is configured to match the attribute information with the attribute information set by tracking; if the matching is successful, determine whether to track the target message. 20 . Message records and/or link trace information. 20. The device according to any one of claims 16 to 19, wherein the recording module is specifically configured to execute one of the following: In the project mode, if it is determined to track the target message, it will include at least one of the routing information, queue information, routing key, message content, message producer, message receiver, and message sending and receiving information of the target message. One, recorded in the metadata; In the routing mode, if it is determined to track the target message, at least one of the sender, receiver, message content and message sending and receiving information of the target message is recorded in the metadata; In the queue mode, if it is determined to track the target message, it will include at least one of the message content of the message, the producer of the message, the receiver of the message, the information about the sending and receiving of the message, and the item information of the message. in the metadata. 21. The device according to any one of claims 16 to 19, wherein the device further comprises: a third determining module, configured to determine the tracking priority of the target message according to the attribute information; The fourth determining module is configured to determine a recording method for tracking the target message according to the tracking priority. 22. The apparatus of claim 15, wherein the apparatus further comprises: A switching module, configured to switch the message queue and/or the middleware for message processing to a second cluster that is mutually backup with the first cluster if the first cluster where the message queue of the target message is located is abnormal. 23. The apparatus of claim 22, wherein the apparatus further comprises: The recovery module is configured to, if the first cluster is abnormal, recover the abnormal message of the project where the target message is located through the second cluster according to the recovery mechanism of the project where the target message is located. 24. The device according to any one of claims 16 to 19, wherein the device further comprises: a query module, configured to query the metadata to obtain the message record and/or link track information of the message to be tracked based on the query request; A processing module, configured to perform information processing according to the message record and/or the link track information. 25. The apparatus of claim 24, wherein The processing module is configured to execute at least one of the following: According to the message record and/or the link track information, carry out statistics of the number of messages of a project, statistics of the number of messages delivered per unit time, and evaluation of the service capability of the message server that processes the messages; According to the message record and/or the link track information, at least one of locating an erroneous node, locating an erroneous service, and locating an error cause is performed. 26. A message processing device, comprising: memory; A processor, connected to the processor, configured to implement the method provided by any one of claims 1 to 14 by executing computer-executable instructions stored on the memory. 27. A computer storage medium storing computer-executable instructions; after the computer-executable instructions are executed by a processor, the method provided by any one of claims 1 to 14 can be implemented.

Images