CN118631918B - Dynamic parsing method and related device based on protobuf protocol - Google Patents
Dynamic parsing method and related device based on protobuf protocol Download PDFInfo
- Publication number
- CN118631918B CN118631918B CN202411096731.0A CN202411096731A CN118631918B CN 118631918 B CN118631918 B CN 118631918B CN 202411096731 A CN202411096731 A CN 202411096731A CN 118631918 B CN118631918 B CN 118631918B
- Authority
- CN
- China
- Prior art keywords
- message
- file
- protobuf
- class
- name
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4488—Object-oriented
- G06F9/449—Object-oriented method invocation or resolution
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
Abstract
本申请公开了一种基于protobuf协议的动态解析方法及相关装置。方法包括：根据业务需求配置protobuf文件，创建消息结构的描述文件，将描述文件转换成C++代码可调用的类函数；发送端通过调用protobuf文件的类结构生成消息数据流；发送端将消息数据流、protobuf文件的文件名、消息名保存在消息对象中；接收端在接收到消息对象后，通过protobuf文件的文件名和消息名动态加载消息结构的描述文件，再基于描述文件生成descriptor对象，并基于descriptor对象构造相应的动态消息类；接收端完成动态解析protobuf的转换。可以减少消息结构描述文件的数据流量，从而提高传输速度。
The present application discloses a dynamic parsing method and related device based on protobuf protocol. The method includes: configuring protobuf file according to business needs, creating a description file of message structure, converting the description file into a class function callable by C++ code; the sending end generates a message data stream by calling the class structure of protobuf file; the sending end saves the message data stream, the file name of protobuf file, and the message name in the message object; after receiving the message object, the receiving end dynamically loads the description file of the message structure through the file name and message name of protobuf file, and then generates a descriptor object based on the description file, and constructs a corresponding dynamic message class based on the descriptor object; the receiving end completes the conversion of dynamic parsing protobuf. The data flow of the message structure description file can be reduced, thereby improving the transmission speed.
Description
Technical Field
The invention relates to the technical field of computer data processing, in particular to a dynamic analysis method based on a protobuf protocol and a related device.
Background
In data processing services, one of the following traffic conditions is often encountered: the upstream business data collection module collects data according to a certain format and transmits the data to the downstream data processing module. The downstream data processing module must recognize the data format transmitted from the upstream to further process, convert, calculate, etc. the data.
Protobuf is an open source serialization framework provided by google (google) corporation, is a lightweight and efficient binary structured data storage format, and can be used for serializing or serializing structured data. It is well suited for data exchange formats for data storage or remote procedure calls (Remote Procedure Call, RPC). The method can be used for language-independent, platform-independent and extensible serial structure data formats in the fields of communication protocols, data storage and the like. With the recent widespread use of the protobuf format, another scheme has emerged: first, the upstream application and the downstream application are both well defined in the protobuf format and share a structural description file (proto file), and then the tool is used to generate the cc and h files, in which a description class is automatically generated from the structural description file. The description class may then be conveniently used in both the upstream data collection module code and the downstream data processing module code for set or get operations. The method still actually operates at the code level.
In the current scheme of adopting protobuf protocol transmission, data sent by a sending end to a receiving end comprises message content and description file content of a message structure. The amount of data transmitted in this way is relatively large.
Disclosure of Invention
The invention aims to provide a dynamic analysis method based on a protobuf protocol and a related device. Therefore, the method and the device realize dynamic analysis of the content of the message bus, analyze the operation interaction condition of each module in real time, monitor the data flow of the whole message bus, and provide a scheme for dynamically analyzing the message based on protobuf protocol. The data flow of the message structure description file can be reduced, and the transmission speed is improved.
In view of this, the first aspect of the present application provides a dynamic parsing method based on protobuf protocol, which is characterized in that the method includes: configuring a protobuf file according to service requirements, creating a description file of a message structure, and converting the description file into a class function which can be called by C++ codes; the sending end generates a message data stream by calling a class structure of the protobuf file; the sending end stores the message data stream, the file name of the protobuf file and the message name in a message object; after receiving the message object, the receiving end dynamically loads a description file of a message structure through the file name and the message name of the protobuf file, generates a descriptor object based on the description file, and constructs a corresponding dynamic message class based on the descriptor object, wherein the descriptor object is in a class structure; the receiving end completes the conversion of dynamic analysis protobuf.
With reference to the first aspect, in one possible implementation manner, the storing, by the sending end, a message data stream, a file name of a protobuf file, and a message name in a message object specifically includes: the sending end calls the class generated by protobuf conversion to process, converts the message class into msg data stream and sends the msg data stream to a message bus.
With reference to the first aspect, in one possible implementation manner, after the receiving end receives the message object, the description file of the message structure is dynamically loaded by using the file name and the message name of the protobuf file specifically includes: the receiving end designates a file directory of protobuf and creates an import class; creating a file description object of protobuf according to the file name under the file directory; generating a message content description class according to the message content name; and identifying the message according to the message content description class and the descriptor object.
With reference to the first aspect, in one possible implementation manner, the identifying a message according to the message content description class and the descriptor object specifically includes: judging that if the msg data flow is equal to null, determining that the analysis fails; if the msg data flow is not null, the automatic analysis is successful, and the msg data flow is the analyzed message.
The second aspect of the present application provides a dynamic parsing system based on protobuf protocol, which is characterized in that the system comprises: the system comprises a configuration module, a sending end and a receiving end, wherein the configuration module is used for configuring a protobuf file according to service requirements, creating a description file of a message structure and converting the description file into a class function which can be called by C++ codes; the sending end is used for generating a message data stream by calling a class structure of the protobuf file; the sending end is also used for storing the file name and the message name of the message data stream and the protobuf file in the message object; the receiving end is used for dynamically loading the description file of the message structure through the file name and the message name of the protobuf file after receiving the message object, generating a descriptor object based on the description file, and constructing a corresponding dynamic message class based on the descriptor object, wherein the descriptor object is in a class structure; the receiving end is used for completing conversion of dynamic analysis protobuf.
With reference to the second aspect, in one possible implementation manner, the sending end is specifically configured to call a class generated by protobuf conversion to process, convert a message class into an msg data stream, and send the msg data stream to a message bus.
With reference to the second aspect, in one possible implementation manner, the receiving end is specifically configured to: designating a file directory of protobuf and creating an import class; creating a file description object of protobuf according to a file name under the file directory; generating a message content description class according to the message content name; and identifying the message according to the message content description class and the descriptor object.
With reference to the second aspect, in one possible implementation manner, the receiving end is specifically configured to: judging that if the msg data flow is equal to null, determining that the analysis fails; if the msg data flow is not null, the automatic analysis is successful, and the msg data flow is the analyzed message.
A third aspect of the present application provides a computer device, characterized in that computer readable instructions are stored in the computer device, which when executed by one or more processors, cause the one or more processors to perform the steps in a protobuf protocol based dynamic parsing method as described in any of the possible implementations of the first aspect to the first aspect.
A fourth aspect of the present application provides a computer-readable storage medium, wherein computer-readable instructions are stored in the computer-readable storage medium, and when executed by one or more processors, cause the one or more processors to perform the steps in the protobuf protocol-based dynamic parsing method according to any one of the first aspect to the first aspect.
The application provides a dynamic analysis method and a related device based on a protobuf protocol, which can enable data sent by a sending end to a receiving end to be only a message data stream, a protobuf file name and a message name, and enable the receiving end to dynamically analyze message contents after receiving the message data stream, the protobuf file name and the message name. The scheme can reduce the data flow of the message structure description file, thereby improving the transmission speed.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
Fig. 1 is a schematic flow chart of a dynamic analysis method based on a protobuf protocol provided by the application;
Fig. 2 is a schematic flow chart of a dynamic analysis method based on a protobuf protocol provided by the application;
Fig. 3 is a schematic diagram of a dynamic analysis system based on a protobuf protocol according to the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b or c may represent: a, b, c, a and b, a and c, b and c or a and b and c, wherein a, b and c can be single or multiple. It is noted that "at least one" may also be interpreted as "one (a) or more (a)".
In the present application, the words "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
The first, second, etc. descriptions in the embodiments of the present application are only used for illustrating and distinguishing the description objects, and no order is used, nor is the number of the devices in the embodiments of the present application limited, and no limitation on the embodiments of the present application should be construed.
Protobuf is an open source serialization framework provided by google (google) corporation, is a lightweight and efficient binary structured data storage format, and can be used for serializing or serializing structured data. It is well suited for data exchange formats for data storage or remote procedure calls (Remote Procedure Call, RPC). The method can be used for language-independent, platform-independent and extensible serial structure data formats in the fields of communication protocols, data storage and the like. With the recent widespread use of the protobuf format, another scheme has emerged: first, the upstream application and the downstream application are both well defined in the protobuf format and share a structural description file (proto file), and then the tool is used to generate the cc and h files, in which a description class is automatically generated from the structural description file. The description class may then be conveniently used in both the upstream data collection module code and the downstream data processing module code for set or get operations. The method still actually operates at the code level.
In the current scheme of adopting protobuf protocol transmission, data sent by a sending end to a receiving end comprises message content and description file content of a message structure. The amount of data transmitted in this way is relatively large.
Therefore, the present application provides a dynamic parsing method based on protobuf protocol, please refer to fig. 1, which includes:
step S1: and configuring a protobuf file according to service requirements, creating a description file of a message structure, and converting the description file into a class function which can be called by C++ codes.
And configuring a protobuf file according to the service requirement, and creating a description file of the message structure, wherein the description file is in a proto format.
And converts the description file into class functions that c++ code can call.
Step S2: and the sending end generates a message data stream by calling the class structure of the protobuf file.
And the sending end adopts C++ codes to generate a message data stream by calling the class structure of the protobuf file.
Step S3: the sending end stores the message data stream, the file name of the protobuf file and the message name in the message object.
The sending end stores the message data stream, the file name of the protobuf file and the message name in the message object.
Step S4: after receiving the message object, the receiving end dynamically loads the description file of the message structure through the file name and the message name of the protobuf file, generates a descriptor object based on the description file, and constructs a corresponding dynamic message class based on the descriptor object.
After receiving the message object, the receiving end dynamically loads the description file of the message structure through the file name and the message name of the protobuf file, generates a descriptor object based on the description file, and constructs a corresponding dynamic message class based on the descriptor object, wherein the descriptor object is in a class structure.
Step S5: the receiving end completes the conversion of dynamic analysis protobuf.
The receiving end completes the conversion of dynamic analysis protobuf.
It should be noted that, the step S2 specifically includes:
The sending end calls the class generated by protobuf conversion to process, converts the message class into msg data stream and sends the msg data stream to a message bus. It should be noted that msg is a binary stream, and is a binary data stream transmitted internally by protobuf.
Illustratively, the pre-service collects device data via a protocol and sends the device data to the bus. After the data processing service receives the data, alarm information is generated to the alarm service, and change data is generated to the disk storage service for disk storage.
In step S4, after receiving the message object, the receiving end dynamically loads the description file of the message structure through the file name and the message name of the protobuf file, where the description file specifically includes:
Step S41: the receiving end designates a file directory of the protobuf and creates an import class.
In a specific case, the receiving end designates a file directory of protobuf, and creating an import class specifically includes:
google::protobuf::compiler::DiskSourceTree sourceTree;
sourceTree.MapPath("","E:\\RelyEZ\\ems_new\\product\\common\\sysconfig\\potobuf");
google::protobuf::compiler::Importer importer(&sourceTree, NULL);
Step S42: and creating a file description object of protobuf according to the file name under the file directory.
And the receiving end creates a file description object of protobuf according to the file name under the file directory. Specifically, the method comprises the following steps:
const google::protobuf::FileDescriptor *file = importer.Import(fileName.toStdString());
step S43: a message content description class is generated from the message content name.
The receiving end generating the message content description class according to the message name specifically may include:
const Descriptor* descptor = file->FindMessageTypeByName(msgStr.toStdString());
Step S44: the message is identified based on the message content description class and the descriptor object.
The receiving end identifies the message according to the message content description class and the descriptor object.
Specifically, the receiving end may determine that the parsing fails if the msg data stream is equal to null, and the message is not the message; if the msg data flow is not null, the automatic analysis is successful, and the msg data flow is the analyzed message.
Further, the receiving end identifies the message according to the message content description class and the structure format, which may specifically include:
google::protobuf::DynamicMessageFactory message_factory;
const Message* msg = message_factory.GetPrototype(descptor);
The application provides a dynamic analysis method and a related device based on a protobuf protocol, which can enable data sent by a sending end to a receiving end to be only a message data stream, a protobuf file name and a message name, and enable the receiving end to dynamically analyze message contents after receiving the message data stream, the protobuf file name and the message name. The scheme can reduce the data flow of the message structure description file, thereby improving the transmission speed.
The application further provides a dynamic analysis system based on the protobuf protocol. The system comprises: configuration module, transmitting end and receiving end. The configuration module is used for configuring a protobuf file according to service requirements, creating a description file of a message structure, and converting the description file into a class function which can be called by C++ codes; the sending end is used for generating a message data stream by calling a class structure of the protobuf file; the sending end is also used for storing the file name and the message name of the message data stream and the protobuf file in the message object; the receiving end is used for dynamically loading the description file of the message structure through the file name and the message name of the protobuf file after receiving the message object, generating a descriptor object based on the description file, and constructing a corresponding dynamic message class based on the descriptor object, wherein the descriptor object is in a class structure; the receiving end is used for completing conversion of dynamic analysis protobuf.
The sending end is specifically used for calling the class generated by protobuf conversion for processing, converting the message class into msg data stream and sending the msg data stream to the message bus.
The receiving end is specifically configured to: designating a file directory of protobuf and creating an import class; creating a file description object of protobuf according to a file name under the file directory; generating a message content description class according to the message content name; and identifying the message according to the message content description class and the descriptor object.
The receiving end is specifically configured to: judging that if the msg data flow is equal to null, determining that the analysis fails; if the msg data flow is not null, the automatic analysis is successful, and the msg data flow is the analyzed message.
The present application also provides a computer device having stored therein computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps in a protobuf protocol based dynamic parsing method as described in the present application.
The present application also provides a computer-readable storage medium having stored therein computer-readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps in a protobuf protocol-based dynamic parsing method as described in the present application.
The technical means disclosed by the scheme of the application is not limited to the technical means disclosed by the embodiment, and also comprises the technical scheme formed by any combination of the technical features. It should be noted that modifications and adaptations to the application may occur to one skilled in the art without departing from the principles of the present application and are intended to be within the scope of the present application.
The above description is made in detail of a dynamic analysis method and related device based on protobuf protocol provided by the embodiment of the present application, and specific examples are applied to describe the principle and implementation of the present application, and the description of the above embodiment is only used to help understand the method and core idea of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above. Although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (8)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411096731.0A CN118631918B (en) | 2024-08-12 | 2024-08-12 | Dynamic parsing method and related device based on protobuf protocol |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411096731.0A CN118631918B (en) | 2024-08-12 | 2024-08-12 | Dynamic parsing method and related device based on protobuf protocol |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118631918A CN118631918A (en) | 2024-09-10 |
| CN118631918B true CN118631918B (en) | 2024-11-22 |
Family
ID=92608924
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411096731.0A Active CN118631918B (en) | 2024-08-12 | 2024-08-12 | Dynamic parsing method and related device based on protobuf protocol |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118631918B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113746842A (en) * | 2021-09-03 | 2021-12-03 | 唯品会(广州)软件有限公司 | Message sending method based on Protobuf protocol dynamic analysis |
| CN117369828A (en) * | 2023-09-26 | 2024-01-09 | 山东新一代信息产业技术研究院有限公司 | Protobuf-based IROS message conversion method, device and medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113742231A (en) * | 2021-09-03 | 2021-12-03 | 唯品会(广州)软件有限公司 | Protobuf protocol dynamic analysis-based number making method and device |
-
2024
- 2024-08-12 CN CN202411096731.0A patent/CN118631918B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113746842A (en) * | 2021-09-03 | 2021-12-03 | 唯品会(广州)软件有限公司 | Message sending method based on Protobuf protocol dynamic analysis |
| CN117369828A (en) * | 2023-09-26 | 2024-01-09 | 山东新一代信息产业技术研究院有限公司 | Protobuf-based IROS message conversion method, device and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118631918A (en) | 2024-09-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8375094B2 (en) | Creating a message readable by a plurality of heterogeneous recipients | |
| US8655941B2 (en) | Service oriented architecture enterprise service bus with universal ports | |
| US7894431B2 (en) | System and method for communicating asynchronously with web services using message set definitions | |
| US20030009539A1 (en) | Distributed object middleware connection method | |
| CN112114979A (en) | Remote procedure call method and device | |
| CN110943911A (en) | High-efficiency data transmission method for Internet of things based on protobuf | |
| EP1916600A1 (en) | Network robot system and method of communication therein | |
| CN112214967A (en) | Message format conversion method and device | |
| JP2005174120A (en) | Web service connection processing method, system, and program | |
| CN113472889A (en) | Scheduling system and method of microservice | |
| CN112084042A (en) | Message processing method and device | |
| KR20110065448A (en) | Message processing pipeline configuration | |
| CN115550438A (en) | Internet of things message processing method, device, system, equipment and storage medium | |
| US7739389B2 (en) | Providing web services from a service environment with a gateway | |
| US20140215011A1 (en) | Message exchange via generic tlv generator and parser | |
| WO2013016994A1 (en) | Message sending method and system | |
| CN113556399B (en) | Message pushing device, system, method, electronic equipment and storage medium | |
| CN118631918B (en) | Dynamic parsing method and related device based on protobuf protocol | |
| CN112866182B (en) | Data docking method, device and computer readable storage medium | |
| CN113779018B (en) | A data processing method and device | |
| CN112769741A (en) | Message communication method and electronic equipment | |
| CN112445850B (en) | Data conversion method, device, electronic device and readable medium | |
| CN120541149B (en) | Data synchronization method and electronic equipment | |
| KR100395501B1 (en) | Apparatus and method of interface between SDL system and CORBA | |
| CN119105887B (en) | Message processing method, device, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |