CN114461413A - Message track information processing method, system, terminal and storage medium of message queue - Google Patents

Abstract

The invention provides a message track information processing method, a system, a terminal and a storage medium of a message queue, wherein the method comprises the following steps: packaging the message tracks of the message queue into track information, and coding and serializing the track information; sending the track information after the coding and the serialization to a message queue for caching; deserializing and decoding the track information cached in the message queue, and analyzing the track information into objects which can be identified by a storage module; and calling an application program interface of the storage module and storing the object for query.

Description

Method, system, terminal and storage medium for processing message track information of message queue

technical field

The present application relates to a message track information processing method, system, terminal and storage medium of a message queue.

Background technique

The message queue is an important component in the distributed system. At present, RabbitMQ, Kafka, ZeroMQ, ActiveMQ, etc. are widely used. In the existing application of producers and consumers based on message queues, upstream message producers send messages and send them to message queues, and downstream message consumers consume messages from consumption queues for data processing.

However, in real applications, abnormal scenarios are prone to occur. For example, the upstream producer sends a message, but the downstream consumer does not consume it, or the downstream consumer consumes the same message multiple times. This will lead to inaccurate and unstable message queue delivery and invocation. When such an abnormal scenario occurs, the upstream and downstream message records are particularly necessary and important.

Based on this, the message queue provides a tracking plugin (Tracing plugin), which can be opened on the message queue cluster through commands, and the tracer log file is stored on the disk of the message queue cluster. After the background is configured through the page, You can query the message traces of Exchange and Queue. However, the functions provided by the trace plug-in are too simple, the content of the trace cannot be customized or searched, and the tracer log file is saved on the cluster disk, which will bring risks to the stability of the cluster. If there are many messages in the cluster and many tracks are monitored, the Tracer log file will occupy a large amount of disks, and it needs to be cleaned regularly, which brings great challenges to the maintenance work. Tracer log files are plain text, which is inconvenient to query historical messages. The content format is fixed and lacks key information, such as Message ID, RequestID and other information that users are concerned about, and cannot be accurately searched through key information.

SUMMARY OF THE INVENTION

An overview of various exemplary technical solutions is presented below. Some simplifications and omissions may be made in the following summary, which is intended to highlight and introduce some aspects of various exemplary solutions, but not to limit the scope of the invention. A detailed description of exemplary solutions sufficient to allow one of ordinary skill in the art to make and use the inventive concepts will be presented in subsequent sections.

The technical scheme of the present invention provides a message track information processing method, system, terminal and storage medium of a message queue, which can provide large-capacity and stable track information storage, and realize accurate and convenient search of track information content.

The technical solution provides a message track information processing method of a message queue, comprising: encapsulating the message track of the message queue into track information and encoding and serializing it; sending the encoded and serialized track information to the message queue for processing Cache; deserialize and decode the track information cached in the message queue, and parse it into an object that can be identified by the storage module; call the application program interface of the storage module and store the object for query.

In one technical solution, the method further includes querying the object through at least one of the following object parameters: resource type, resource name, message ID, message routing key, message Tag, Request ID, Polaris trace ID, message time range .

In a technical solution, the message track of the message queue includes at least one of the following: message sending time, producer IP, sending time, consumer IP, message id, message key, time when the message was received, and start consumption time , consumption completion time, consumption result status, consumption time.

In a technical solution, the storage module includes at least one of the following: Elasticsearch, HIVE, Mysql, and DB.

In a technical solution, the message queue includes at least one of the following: ActiveMQ, RabbitMQ, ZeroMQ, and Kafka.

In a technical solution, the encapsulating the message trace of the message queue into trace information includes encapsulating the message trace into at least one of the following: Java objects, C language objects, C++ objects, and Python objects.

In a technical solution, the coding is performed through a development kit. The data is compressed by encoding, thereby reducing the occupation of the bandwidth of the service end by the trajectory information.

The technical solution also provides a message track information processing system for a message queue, comprising: a message processing module configured to encapsulate the message of the message queue into track information, encode and serialize it, and encode and serialize the encoded and serialized messages. The track information is sent to the message queue; the message queue module is configured to receive and cache the encoded and serialized track information; the track service module is configured to decode and reverse the track information cached in the message queue. Serialize and parse into an object that can be recognized by the storage module, call the application program interface of the storage module, and send the object to the storage module; the storage module stores the object for query.

In a technical solution, the message track information processing system of the message queue further comprises: a query module, configured to call the application program interface of the storage module to query the object.

In a technical solution, the message processing module is a software tool development kit.

In one technical solution, the trajectory service module is an executable archive file.

The technical solution also provides a terminal, comprising: a processor, a memory, and a computer program stored and run on the memory, and the processor implements the steps of any of the foregoing technical solutions when the processor executes the computer program.

The technical solution also provides a computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above-mentioned technical solutions are implemented.

The invention has the following beneficial effects: providing more stable and reliable storage for message track information, without affecting the message queue cluster, and providing more convenient query and positioning.

The track information of the technical solution is not directly sent to the message queue for storage, nor is it directly sent to the storage module for storage. Instead, it is sent to a separate message queue cluster with better performance for caching, and then decoded and sent to the storage module. In this way, the message is decoupled from the storage module and from other message queue clusters. The advantage of this is that even if the type of the storage module is changed, the message processing module is not affected, and operations such as upgrading and changing settings are not required, which is convenient for future expansion. Storage through a separate storage module is more stable and reliable than storing in a message queue, data is not easily lost, the storage capacity is larger, and it does not occupy the resources of the message queue cluster, thereby improving the performance of the message queue cluster. The technical solution also provides a more powerful search function, which can search for trajectory information more conveniently and quickly through key information.

Description of drawings

For a better understanding of the various exemplary embodiments, reference may be made to the accompanying drawings, in which:

1 shows a schematic flowchart of a method for processing message track information of a message queue provided by an embodiment;

FIG. 2 shows a schematic structural diagram of a message track information processing system of a message queue provided by an embodiment; and

3A and 3B show a schematic diagram of a query interface of a message track information processing system of a message queue provided by an embodiment;

FIG. 4 shows a schematic diagram of a query result of a message track information processing system of a message queue provided by an embodiment.

To facilitate understanding, the same reference numerals have been used to refer to elements having substantially the same or similar structure and/or substantially the same or similar function.

Detailed ways

The description and drawings illustrate the principles of the invention. Accordingly, it will be appreciated that those skilled in the art will be able to devise various arrangements that, although not expressly described or illustrated herein, embody the principles of and are included within the scope of the invention. Furthermore, all examples cited herein are primarily intended to be expressly used for teaching purposes to assist the reader in understanding the principles of the invention and the concepts provided by the inventors to deepen the art, and all examples should be construed as not limiting Examples and conditions of such specific citations. Also, as used herein, the term "or" refers to a non-exclusive or (ie, and/or) unless otherwise indicated (eg, "or otherwise" or "or in the alternative"). Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

FIG. 1 shows a method for processing message trace information of a message queue provided by an embodiment, including the following steps:

Step S101, encapsulate the message track of the message queue into track information, and encode and serialize it.

In this embodiment, a software tool development kit (SDK) or other suitable tools can assemble the message traces into Java objects or other suitable objects, such as C language, C++, Python, and the like. The software tool development kit (SDK) may include a producer software kit (Producer SDK) and a consumer software kit (Consumer SDK). The messages referred to in this embodiment include messages sent by upstream producers and messages consumed by downstream consumers. The message trace includes at least one of the following: message sending time, producer IP, sending result status, sending time, receiving message time, consumer IP, message id, message key, start consumption time, consumption completion time, consumption result status, Consumption time etc. In this embodiment, processing can be performed at the sending end and the consuming end, and the whole process can be recorded before the message is sent, after the message is sent, before the message is consumed, and after the message is consumed, so that the specific situation of the message sending and consumption can be clearly known. The encapsulated track information Java object can be encoded by using a Java Development Kit (Java Development Kit (JDK)) or other suitable methods, such as using other development kits (Development Kit). The encoding process can compress the track information, thereby reducing the occupation of the bandwidth of the service end by the track information. After the encoding is completed, the encoded trajectory information is serialized.

Step S102, sending the encoded and serialized trajectory information to a message queue for buffering.

In this embodiment, the message queue includes at least one of the following: ActiveMQ, RabbitMQ, ZeroMQ, and Kafka. Preferably, the trajectory information can be sent to a separate MQ cluster, so that the trajectory information can be decoupled from other MQ clusters. Trajectory information can be sent to the message queue for storage on a separate server on the switch. The switch can be named production_default_subscription.SYS.MERCURY.TRACE.DATA.rabbitmq.

Step S103, deserialize and decode the track information cached in the message queue, and parse it into an object that can be identified by the storage module.

In this embodiment, the track information cached in the message queue can be deserialized and decoded by using an executable Java Archive File or other suitable tools, such as an executable C language archive file, etc., And parse it into an object that the storage module can recognize, then call the storage module application programming interface (API) and send the object to the storage module.

Step S104, calling the application program interface API of the storage module to store the object.

In this embodiment, the object can be stored in the storage module by invoking the application program interface of the storage module through the executable Java archive file for query. The storage module can choose Elasticsearch, HIVE, Mysql, DB, etc. Preferably, Elasticsearch (ES) can be used. ES data will be persistent and backed up, so there is no need to worry about the loss of trajectory information or affecting the resources of the MQ cluster. Storage through a separate storage module is more stable and reliable than storage in a message queue, less likely to be lost, and has a larger storage capacity, and does not occupy the resources of the message queue cluster, thereby improving the performance of the message queue cluster.

Preferably, step S105 may also be included, calling the application program interface API of the storage module to query the object. Specifically, the object can be queried by calling the API of the storage module through the background interface (Portal). Taking ES as an example of a storage module, ES has a powerful query and retrieval function, which allows us to query historical trajectory data in seconds based on parameters such as message ID or request ID, and quickly locate problems. In addition, if you cannot find valid information, you can also query based on resource type, resource name, message ID, message routing key, Request ID, Polaris trace ID, time range, etc. to narrow the query scope.

In this embodiment, the trajectory information is not directly sent to the MQ for storage, nor directly sent to the storage module for storage. Instead, it is sent to a separate MQ cluster with better performance for caching, and then decoded and sent to the storage module. In this way, the message is decoupled from the storage module and from other MQ clusters. The advantage of this is that even if ES is replaced with another storage engine, it is unaware of the SDK and does not require users to perform upgrades and other operations.

FIG. 2 shows a message track information processing system of a message queue provided by an embodiment, including: a message processing module 201 , a message queue module 202 , a track service module 203 , a storage module 204 and a query module 205 .

The message processing module 201 encapsulates the message trace into trace information. Preferably, the message processing module 201 may include a software tool development kit (SDK) or other suitable tools. The software tool development kit (SDK) may include a producer software kit (Producer SDK) and a consumer software kit (Consumer SDK). The message processing module 201 can assemble the message traces into Java objects or other suitable objects, such as C language, C++, Python and so on. The messages referred to in this embodiment include messages sent by upstream producers and messages consumed by downstream consumers. The message trace includes at least one of the following: message sending time, producer IP, sending result status, sending time, receiving message time, consumer IP, message id, message key, start consumption time, consumption completion time, consumption result status, Consumption time etc. In this embodiment, processing can be performed at the sending end and the consuming end, and the whole process can be recorded before the message is sent, after the message is sent, before the message is consumed, and after the message is consumed, so that the specific situation of the message sending and consumption can be clearly known. The message processing module 201 encodes the encapsulated track information Java object, and the encoding method may adopt JDK or other suitable methods. The encoding process can compress the track information, thereby reducing the occupation of the bandwidth of the service end by the track information. After the encoding is completed, the message processing module 201 serializes the encoded trajectory information and sends it to the message queue module 202 .

The message queue module 202 is configured to receive and buffer the encoded and serialized track information. The message queue includes at least one of the following: ActiveMQ, RabbitMQ, ZeroMQ, and Kafka. Preferably, the trajectory information can be sent to a separate MQ cluster, so that the trajectory information can be decoupled from other MQ clusters. Trajectory information can be sent to the message queue for storage on a separate server on the switch. The switch can be named production_default_subscription.SYS.MERCURY.TRACE.DATA.rabbitmq.

The track service module 203 is configured to deserialize and decode the track information cached in the message queue, parse it into an object that can be recognized by the storage module, call the storage module application programming interface (API) and store the object in the storage module. in the storage module. Preferably, the track service module 203 may be an executable Java archive file (Java ArchiveFile), that is, a JAR file.

The storage module 204 is configured to store the received object. The storage module can choose Elasticsearch, HIVE, Mysql, DB, etc. Preferably, Elasticsearch (ES) can be used. ES data will be persistent and backed up, so there is no need to worry about the loss of trajectory information or affecting the resources of the MQ cluster.

Query module 205. The query module may be a background interface (Portal) or other suitable tools. The query module 205 is configured to call the API of the storage module to query the object. Taking ES as an example of a storage module, ES has powerful query and retrieval functions.

Referring to the schematic diagram of the query interface of a message track information processing system of a message queue provided by the embodiment shown in FIGS. 3A and 3B , the query can be performed in two ways: message ID query ( FIG. 3A ) and general query ( FIG. 3B ). The ID query includes: resource type, resource name, and message ID. Common queries include resource type, resource name, message routing key, message Tag, Request ID, Polaris trace ID, message time range, etc. According to the parameters in the message ID, the historical trajectory data can be queried in seconds for quick positioning. In addition, if you cannot find valid information, you can also query according to the parameters of the general query to narrow the query scope. The query results include: message ID, message routing key, message Tags, Request ID, Polaris trace ID, Client ID, sending status, sending time, consumption status, operation, number of queries, etc.

Referring to FIG. 4 , a schematic diagram of a query result of a message track information processing system of a message queue provided by an embodiment is shown. According to the query request initiated by the user, the query result can be obtained, and after assembly, the track details of the producer, server and consumer can be displayed to the user. The producer trace information can include message ID, sender IP, sending time, sending time, sending status, Request ID, Polaris trace ID, etc. The server track information can include cluster information, topics, routing keys, storage time, etc. Consumer trace information includes subscription, number of times of consumption, consumer IP, delivery time, consumption time, consumption status, Request ID, Polaris trace ID, Client ID, etc.

The embodiment further provides a terminal, including: a processor, a memory, and a computer program stored on the memory to run, and the processor implements the steps of the method in any of the foregoing embodiments when the processor executes the computer program, for example, steps S101 to S104, or When the processor executes the computer program, the functions of the modules/units in the above embodiments, such as the functions of the units 201 to 205 shown in FIG. 1 , are implemented. The computer program may be divided into one or more modules/units, which are stored in the memory and executed by the processor. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal.

The terminal may be a mobile terminal such as a smart phone, or a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The terminal may include, but is not limited to, a processor and a memory, may include more or less components, or combine certain components, for example, the terminal may also include an input and output device, a network access device, a bus, and the like. The processor may be a central processing unit (Central Processing Unit, CPU), other general-purpose processors, a digital message queue manager (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) , Off-the-shelf programmable gate array (FieldProgrammable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The memory may be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal. The memory may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a Secure Digital (Secure Digital, SD) card, a flash memory card ( Flash Card), etc. Further, the memory may also include both an internal storage unit of the terminal and an external storage device.

The embodiment also provides a computer-readable storage medium, when the computer program is executed by a processor, the steps of the method of any of the above-mentioned embodiments are implemented.

Each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units. The integrated modules/units, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer When the program is executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM) , Random Access Memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable media Electric carrier signals and telecommunication signals are not included.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above-mentioned system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed system, terminal and method may be implemented in other manners. For example, the above-described system and terminal embodiments are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or Components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of systems or units, and may be in electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above-mentioned embodiments, those of ordinary skill in the art should understand that: it can still be used for the above-mentioned implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be included in the within the scope of protection of this application.

Claims (13)

1. A message track information processing method of a message queue is characterized by comprising the following steps:

packaging the message tracks of the message queue into track information, and coding and serializing the track information;

sending the track information after the coding and the serialization to a message queue for caching;

deserializing and decoding the track information cached in the message queue, and analyzing the track information into objects which can be identified by a storage module;

and calling an application program interface of the storage module and storing the object for query.

2. The message track information processing method of the message queue according to claim 1, further comprising: querying the object with object parameters of at least one of: resource type, resource name, message ID, message routing key, message Tag, RequestID, Polaris trace ID, message timeframe.

3. The message track information processing method of the message queue according to claim 1, wherein the message track of the message queue comprises at least one of: message sending time, producer IP, sending time consumption, consumer IP, message ID, message Key, time of receiving message, consumption starting time, consumption finishing time, consumption result state and consumption time consumption.

4. The message track information processing method of the message queue according to claim 1, wherein the storage module comprises at least one of: elasticisearch, HIVE, Mysql, DB.

5. The message track information processing method of the message queue according to claim 1, wherein the message queue comprises at least one of: ActiveMQ, RabbitMQ, ZeroMQ, Kafka.

6. The method of claim 1, wherein encapsulating message tracks of a message queue into track information comprises encapsulating message tracks into at least one of: java objects, C language objects, C + + objects, Python objects.

7. The message track information processing method of the message queue according to claim 1, wherein the encoding is performed by a development kit.

8. A message track information processing system for a message queue, comprising:

the message processing module is configured to encapsulate the messages of the message queue into track information, encode and serialize the track information, and send the encoded and serialized track information to the message queue module;

a message queue module configured to receive and buffer the encoded and serialized track information;

the track service module is configured to decode, deserialize and analyze the track information cached in the message queue into an object which can be identified by the storage module, call an application program interface of the storage module and send the object to the storage module;

a storage module to store the object for querying.

9. The message track information processing system of the message queue of claim 8, further comprising:

and the query module is configured to call an application program interface of the storage module to query the object.

10. The message track information processing system of a message queue of claim 8, wherein the message processing module is a software tool development kit.

11. The message track information handling system of a message queue of claim 8, wherein the track service module is an executable archive file.

12. A terminal, comprising: a processor, a memory, and a computer program stored for execution on the memory, the processor when executing the computer program implementing the method of any one of claims 1-7.

13. A computer-readable storage medium, characterized in that the computer program, when being executed by a processor, implements the method of any one of claims 1-7.

Images