CN113032492B - Method and device for transmitting and storing data at edge end - Google Patents
Method and device for transmitting and storing data at edge end Download PDFInfo
- Publication number
- CN113032492B CN113032492B CN202110431076.XA CN202110431076A CN113032492B CN 113032492 B CN113032492 B CN 113032492B CN 202110431076 A CN202110431076 A CN 202110431076A CN 113032492 B CN113032492 B CN 113032492B
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- forwarding
- information
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention is applicable to the technical field of data transmission, and provides a method and a device for transmitting and storing data at an edge end, wherein the method comprises the following steps: the data acquisition module creates a transmission initial message queue to a first message middleware according to the acquired edge equipment data; the data processing module acquires and processes an initial message queue in the first message middleware, acquires and forwards message queue information and stores the message queue information to the second message middleware; the data storage module acquires and stores the information of the storage message queue; the data forwarding module acquires forwarding message queue information. The invention can prevent the situation that data accumulation easily occurs when the data volume is increased or the frequency is increased in the data transmission process, avoid the problem that the data loss is easy to occur when the service of the edge equipment is down, and can also inquire under the condition of large data volume and quickly return the inquiry result.
Description
Technical Field
The invention belongs to the technical field of data transmission, and particularly relates to a method and a device for transmitting and storing edge data.
Background
In the related art, data transmission is mostly performed through Http protocol ((Hypertext Transfer Protocol, hypertext transfer protocol) or Grpc protocol (Google Remote Procedure Call ), and as the data volume increases or the frequency increases, the situation that data accumulation easily occurs, and even one service or several services at the edge end are down, the situation that data is lost occurs.
The edge uses a common embedded database, such as Sqlite (a software library) and H2 (a database), to store and query large data volume, and queries under large data volume, so that on one hand, the memory and cpu (central processing unit ) overhead is increased, on the other hand, the performance of the database is also a serious test, and the query result return time is also long. Mysql is far more unsupported than large databases, which have relatively small memory, CPU, and disk.
Disclosure of Invention
In view of the above, the embodiments of the present invention provide a method and an apparatus for transmitting and storing data at an edge, so as to solve the problems in the prior art that data accumulation easily occurs along with an increase in data volume or a frequent increase in data transmission process, data loss occurs due to a service downtime of the edge, and support is difficult for large database memory, cpu, and relatively small hardware of a disk.
In a first aspect of the embodiment of the present invention, a method for transmitting and storing data at an edge is provided, including: the data acquisition module creates an initial message queue according to the acquired edge equipment data, and transmits the initial message queue to a first message middleware;
When the data processing module acquires an initial message queue in the first message middleware, the data processing module processes the initial message queue according to the requirements of the data storage module and the data forwarding module to acquire forwarding message queue information and storage message queue information, and transmits the forwarding message queue information and the storage message queue information to the second message middleware;
when the data storage module acquires the information of the storage message queue, the information of the storage message queue is stored;
And when the data forwarding module acquires the forwarding message queue information, forwarding the forwarding message queue information.
In a second aspect of the embodiment of the present invention, there is provided an apparatus for transmitting and storing data at an edge, including:
the data acquisition module is used for creating an initial message queue according to the acquired edge equipment data and transmitting the initial message queue to the first message middleware;
a first message middleware for storing the initial message queue;
The data processing module is used for processing the initial message queue according to the requirements of the data storage module and the data forwarding module when the data processing module acquires the initial message queue in the first message middleware, acquiring forwarding message queue information and storage message queue information, and transmitting the forwarding message queue information and the storage message queue information to the second message middleware;
the second message middleware is used for storing the forwarding message queue information and the storage message queue information;
the data storage module is used for storing the stored message queue information when the data storage module acquires the stored message queue information;
and the data forwarding module is used for forwarding the forwarding message queue information when the data forwarding module acquires the forwarding message queue information.
In a third aspect of an embodiment of the present invention, there is provided a terminal device, including: the edge data transmission and storage method comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the steps of the edge data transmission and storage method when executing the computer program.
In a fourth aspect of the embodiments of the present invention, there is provided a computer readable storage medium storing a computer program, which when executed by a processor, implements the steps of the method for edge data transmission and storage described above.
Compared with the prior art, the embodiment of the invention has the beneficial effects that: the method for transmitting and storing the edge data can prevent the situation that data accumulation easily occurs due to the increase of data volume or frequency in the transmission process, avoid the problem that the data of the edge equipment is easy to lose when one service or a plurality of services of the edge are down, solve the problem that compared with a large database, the data are difficult to support for hardware with relatively small memory, cpu and magnetic disk, and can query under the condition of large data volume and quickly return query results.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of a method for transmitting and storing data at an edge according to an embodiment of the present invention;
Fig. 2 is a schematic diagram of an implementation flow of creating and transmitting an initial message queue to a first message middleware by a data acquisition module in a method for transmitting and storing edge data according to an embodiment of the present invention;
Fig. 3 is a schematic flow chart of a method for transmitting an initial message queue in a first message middleware to a second message middleware in a method for transmitting and storing edge data according to an embodiment of the present invention;
fig. 4 is an exemplary diagram of storing information of a storage message queue in a method for transmitting and storing edge data provided by an embodiment of the present invention;
Fig. 5 is an exemplary diagram of forwarding message queue information in a method for transmitting and storing edge data provided by an embodiment of the present invention;
Fig. 6 is a schematic diagram of an apparatus for transmitting and storing data at an edge according to an embodiment of the present invention;
Fig. 7 is a schematic diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
In order to illustrate the technical scheme of the invention, the following description is made by specific examples.
Fig. 1 is a flowchart of a method for transmitting and storing data at an edge according to an embodiment of the present invention. As shown in fig. 1, the method includes:
Step S11: the data acquisition module creates an initial message queue according to the acquired edge equipment data, and transmits the initial message queue to the first message middleware.
The data acquisition module acquires the data of the edge equipment, creates an initial message queue according to the data of the edge equipment, and transmits the message queue to the message middleware RabbitMQ, and the RabbitMQ is open source message proxy software for realizing an advanced message queue protocol, is an open standard of an application layer protocol, is designed for the message-oriented middleware, and is mainly used for decoupling among components, so that the message is stored in the middleware, and the loss of the message in the transmission process is avoided.
The message queue is chosen in the embodiments of the invention because of its own capabilities, which is the container that holds messages during their transmission, the message queue manager acts as a man-in-the-middle in transmitting messages to the target, the main purpose of the message queue being to provide routing and to ensure delivery of the message, and if the recipient is not available when the message is sent, the message queue will hold the message until it is successfully delivered.
Fig. 2 is a schematic flow chart of an implementation of the data acquisition module according to the embodiment of the present invention for creating and transmitting an initial message queue to the first message middleware. As shown in fig. 2, step S11 specifically includes the following steps:
Step S111: the data acquisition module acquires edge equipment data through a transmission protocol, wherein the transmission protocol comprises Modbus-RTU, modbus-TCP, S7 or BACnet;
The Http protocol (HyperText Transfer Protocol ) or the Rpc protocol (Remote Procedure Call Protocol, remote call protocol) is adopted to transmit the data easily accumulated, so that the embodiment of the invention selects Modbus-RTU (an industrial communication protocol with an encoding format of RTU), modbus-TCP (an industrial communication protocol with an encoding format of TCP), S7 (SIEMENS 7, siemens S7 communication protocol) or BACnet ((A Data Communication Protocol for Building Automation and Control Networks, building automatic control network data communication protocol), and the defect of data accumulation is effectively avoided.
Step S112: the data acquisition module creates an initial message queue based on the data of the edge terminal equipment;
step S113: the data acquisition module transmits the initial message queue to a first message middleware.
Step S12: when the data processing module acquires an initial message queue in the first message middleware, the data processing module processes the initial message queue according to the requirements of the data storage module and the data forwarding module to acquire forwarding message queue information and storage message queue information, and transmits the forwarding message queue information and the storage message queue information to the second message middleware. Wherein the stored message queue information includes a real-time data message queue and a historical data consumption queue.
In the embodiment of the invention, the initial message queue is stored in the first message middleware, the data processing module acquires the initial message queue, and the data processing module acquires the information of the forwarding message queue and the information of the storage message queue after specifically processing the initial message queue according to the requirements of the data storage module and the data forwarding module, and transmits the information of the forwarding message queue and the information of the storage message queue to the second message middleware.
Fig. 3 is a flowchart of a method for transmitting an initial message queue in a first message middleware to a second message middleware according to an embodiment of the present invention. As shown in fig. 3, the processing procedure of the data processing module is specifically as follows:
Step S121: the data processing module acquires the storage period set by the data storage module, and processes the initial message queue according to the storage period and the principle of first-in first-out of the message queue to obtain the information of the stored message queue.
The data storage module sets a storage period for the information of the stored message queue, and the data processing module acquires the storage period and processes the initial message queue according to the principle of first-in first-out of the message queue. The storage period can be flexibly set according to the requirement.
Step S122: the data processing module acquires the forwarding period set by the data forwarding module, and processes the initial message queue according to the forwarding period and the principle of message queue first-in first-out, so as to obtain forwarding message queue information.
The data forwarding module sets a forwarding period for the information of the forwarding message queue, and the data processing module acquires the forwarding period and processes the initial message queue according to the principle of first-in first-out of the message queue. The forwarding period can be flexibly set according to requirements.
Step S13: and when the data storage module acquires the information of the storage message queue, the information of the storage message queue is stored.
The storage message queue information is stored in the second middleware, when the data storage module acquires the storage message queue information, a storage period is set for the storage message queue information, and the storage message queue information is stored according to the requirement of the data storage module.
Fig. 4 is an exemplary diagram of storing stored message queue information provided by an embodiment of the present invention. As shown in fig. 4, the steps for storing the stored message queue information are specifically as follows:
Step S131: when the data storage module acquires the real-time data message queue, the data of the real-time data message queue is cached into a specified hash class.
The data storage module acquires real-time message queue information, and a high-speed memory inside the hard disk, so that the real-time data can be used for providing reading and re-reading services, and temporarily stored in a pre-designated hash class.
Step S132: and when the data storage module acquires the historical data message queue, the historical data message queue is stored in a time sequence database according to the number of data points.
The data storage module acquires the historical message queue information, a high-speed memory inside the hard disk, temporarily stores the historical data and stores the historical data into a time sequence database according to the number of data points in order to use the historical data for providing reading and re-reading services. Because TDengine is a time series database, different tables can be created according to different data points, so that the data points can return results faster when the historical data is queried, and TDengine supports simultaneous storage of multiple tables. TDengine the query storage consumes relatively less memory, CPU (central processing unit ), disk usage.
Wherein the number of points refers to the number of data points.
After the storing step of the information of the stored message queue, the method further comprises:
calling a metadata module interface through a browser;
And acquiring data points through a data storage module according to the data point identifiers stored by the metadata module so as to display the data.
In the embodiment of the invention, the data point identification is stored in the metadata module, and the data storage module acquires the data point, so that corresponding service can be provided when a user needs to inquire the data.
Where the data point identification is a unique identity of the data point itself.
Step S14: and when the data forwarding module acquires the forwarding message queue information, forwarding the forwarding message queue information.
In the embodiment of the invention, the information of the stored message queue is stored in the second middleware, and when the data processing module acquires the information of the forwarded message queue, the information of the stored message queue is forwarded according to the requirement of the data forwarding module.
Fig. 5 is an exemplary diagram for forwarding message queue information according to an embodiment of the present invention. As shown in fig. 5, the specific steps are as follows:
step S141: the data forwarding module acquires the information of the forwarding message queue and sets a forwarding period;
the forwarding message queue information is stored in the second message middleware, and when the data forwarding module acquires the forwarding message queue information, a forwarding period is set for the forwarding message queue information.
Step S142: based on the forwarding period, the data forwarding module transmits the data of the forwarded message queue information to a large data platform through a message queue telemetry transmission protocol so as to provide display and query services.
In the embodiment of the invention, based on the forwarding period, the data forwarding module transmits data through the MQTT (Message Queuing Telemetry Transport, message queue telemetry transmission), and transmits the message queue information to the cloud big data platform, so that the function of displaying and inquiring the user is provided when the user needs the inquiring function.
According to the embodiment of the invention, the data processing module acquires the message queue transmitted to the first middleware by the data acquisition module, processes the message queue according to the need, and stores the message queue by the data storage module and forwards the message queue to the second middleware by the data forwarding module so as to provide the functions of user display and query. The invention can prevent the situation that data accumulation easily occurs when the data volume is increased or the frequency is increased in the data transmission process, avoid the problem that the data is easy to be lost when the service of the edge equipment is down, solve the problem that hardware is difficult to support, and also can inquire under a large data volume and quickly return the inquiry result.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention. Based on the same conception, the embodiment of the invention also provides a device for transmitting and storing the edge data. Fig. 6 is a schematic diagram of an apparatus for transmitting and storing data at an edge according to an embodiment of the present invention, as shown in fig. 6, including: the system comprises a data acquisition module 21, a first message middleware 22, a data processing module 23, a second message middleware 24, a data storage module 25 and a data forwarding module 26. Wherein:
The data acquisition module 21 is configured to create an initial message queue according to the acquired edge device data, and transmit the initial message queue to the first message middleware.
A first message middleware 22 for storing the initial message queue;
The data processing module 23 is configured to, when the data processing module obtains an initial message queue in the first message middleware, process the initial message queue according to requirements of the data storage module and the data forwarding module, obtain forwarding message queue information and storage message queue information, and transmit the forwarding message queue information and the storage message queue information to the second message middleware;
a second message middleware 24 for storing the forwarding message queue information and the storage message queue information;
The data storage module 25 is configured to store the stored message queue information when the data storage module acquires the stored message queue information;
the data forwarding module 26 is configured to forward the forwarding message queue information when the data forwarding module obtains the forwarding message queue information.
Specifically, the data acquisition module 21 acquires data of the edge device through a transmission protocol Modbus-RTU, modbus-TCP, S7 or BACnet, creates an initial message queue based on the data of the edge device, and transmits the initial message queue to the first message middleware 22. The data processing module 23 obtains the storage period set by the data storage module 25, and processes the initial message queue according to the storage period and the principle of message queue first-in first-out, so as to obtain the information of the stored message queue. The data processing module 23 obtains the forwarding period set by the data forwarding module 26, and processes the initial message queue according to the principle of message queue first-in-first-out according to the forwarding period, so as to obtain the forwarding message queue information. Store message queue information and forward message queue information is transmitted to the second message middleware 24. The data storage module 25 stores the stored message queue information when it acquires the stored message queue information, and forwards the forwarded message queue information when it acquires the forwarded message queue information.
Fig. 7 is a schematic diagram of a terminal device according to an embodiment of the present invention. As shown in fig. 7, the terminal device 3 of this embodiment includes: a processor 30, a memory 31 and a computer program 32 stored in the memory 31 and executable on the processor 30, such as an edge data transmission and storage program. The processor 30 executes the computer program 32 to implement the steps of the above-described embodiments of the edge data transmission and storage method, such as steps S11 to S14 shown in fig. 1. Or the processor 30, when executing the computer program 32, performs the functions of the modules of the apparatus embodiments described above, such as the functions of the modules 21 to 26 shown in fig. 6.
Illustratively, the computer program 32 may be partitioned into one or more modules/units that are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions for describing the execution of the computer program 32 in the terminal device 3.
The terminal device 3 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The terminal device may include, but is not limited to, a processor 30, a memory 31. It will be appreciated by those skilled in the art that fig. 7 is merely an example of the terminal device 3 and does not constitute a limitation of the terminal device 3, and may include more or less components than illustrated, or may combine certain components, or different components, e.g., the terminal device may further include an input-output device, a network access device, a bus, etc.
The Processor 30 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or a memory of the terminal device 3. The memory 31 may also be an external storage device of the terminal device 3, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the terminal device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the terminal device 3. The memory 31 is used for storing the computer program as well as other programs and data required by the terminal device. The memory 31 may also be used for temporarily storing data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.
Claims (8)
1. A method for transmitting and storing data at an edge, comprising:
The data acquisition module creates an initial message queue according to the acquired edge equipment data and transmits the initial message queue to a first message middleware, wherein the method comprises the following steps: the data acquisition module acquires edge equipment data through a transmission protocol, wherein the transmission protocol comprises Modbus-RTU, modbus-TCP, S7 or BACnet; the data acquisition module creates an initial message queue based on the data of the edge terminal equipment; the data acquisition module transmits the initial message queue to a first message middleware; the first message middleware is RabbitMQ based on an advanced message queue protocol;
When the data processing module acquires an initial message queue in the first message middleware, the data processing module processes the initial message queue according to the requirements of the data storage module and the data forwarding module to acquire forwarding message queue information and storage message queue information, and transmits the forwarding message queue information and the storage message queue information to the second message middleware;
when the data storage module acquires the information of the storage message queue, the information of the storage message queue is stored;
When the data forwarding module acquires the forwarding message queue information, forwarding the forwarding message queue information;
When the data processing module obtains an initial message queue in the first message middleware, the data processing module processes the initial message queue according to the requirements of the data storage module and the data forwarding module to obtain forwarding message queue information and storage message queue information, and transmits the forwarding message queue information and the storage message queue information to the second message middleware, and the method comprises the following steps:
The data processing module acquires a storage period set by the data storage module, and processes the initial message queue according to the storage period and the principle of first-in first-out of the message queue to obtain information of the stored message queue;
the data processing module acquires the forwarding period set by the data forwarding module, and processes the initial message queue according to the forwarding period and the principle of message queue first-in first-out, so as to obtain forwarding message queue information.
2. The method of edge data transmission and storage of claim 1, wherein the stored message queue information includes a real-time data message queue and a historical data consumption queue.
3. The method for transmitting and storing data at an edge of claim 2, wherein when the data storage module obtains the stored message queue information, the step of storing the stored message queue information includes:
When the data storage module acquires the real-time data message queue, caching the data of the real-time data message queue into a specified hash class;
and when the data storage module acquires the historical data message queue, the historical data message queue is stored in a time sequence database according to the number of data points.
4. The method for transmitting and storing data at an edge of claim 1, wherein when the data forwarding module obtains the forwarding message queue information, forwarding the forwarding message queue information includes:
the data forwarding module acquires the information of the forwarding message queue and sets a forwarding period;
Based on the forwarding period, the data forwarding module transmits the data of the forwarded message queue information to a large data platform through a message queue telemetry transmission protocol so as to provide display and query services.
5. The method for transmitting and storing data at an edge of claim 4, wherein when the data storage module obtains the stored message queue information, after the step of storing the stored message queue information, the method further comprises:
calling a metadata module interface through a browser;
and acquiring data points through the data storage module according to the data point identifiers stored by the metadata module so as to display the data.
6. An apparatus for transmitting and storing data at an edge, comprising:
The data acquisition module is used for creating an initial message queue according to the acquired edge equipment data and transmitting the initial message queue to the first message middleware, and comprises the following steps: the data acquisition module acquires edge equipment data through a transmission protocol, wherein the transmission protocol comprises Modbus-RTU, modbus-TCP, S7 or BACnet; the data acquisition module creates an initial message queue based on the data of the edge terminal equipment; the data acquisition module transmits the initial message queue to a first message middleware;
The first message middleware is used for storing the initial message queue, and is RabbitMQ based on an advanced message queue protocol;
The data processing module is used for processing the initial message queue according to the requirements of the data storage module and the data forwarding module when the data processing module acquires the initial message queue in the first message middleware, acquiring forwarding message queue information and storage message queue information, and transmitting the forwarding message queue information and the storage message queue information to the second message middleware;
the second message middleware is used for storing the forwarding message queue information and the storage message queue information;
the data storage module is used for storing the stored message queue information when the data storage module acquires the stored message queue information;
The data forwarding module is used for forwarding the forwarding message queue information when the data forwarding module acquires the forwarding message queue information;
The data processing module is specifically configured to: the data processing module acquires a storage period set by the data storage module, and processes the initial message queue according to the storage period and the principle of first-in first-out of the message queue to obtain information of the stored message queue; the data processing module acquires the forwarding period set by the data forwarding module, and processes the initial message queue according to the forwarding period and the principle of message queue first-in first-out, so as to obtain forwarding message queue information.
7. Terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the edge data transmission and storage method according to any of claims 1 to 5 when the computer program is executed.
8. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor performs the steps of the edge data transmission and storage method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110431076.XA CN113032492B (en) | 2021-04-21 | 2021-04-21 | Method and device for transmitting and storing data at edge end |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110431076.XA CN113032492B (en) | 2021-04-21 | 2021-04-21 | Method and device for transmitting and storing data at edge end |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113032492A CN113032492A (en) | 2021-06-25 |
| CN113032492B true CN113032492B (en) | 2024-07-09 |
Family
ID=76457218
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110431076.XA Active CN113032492B (en) | 2021-04-21 | 2021-04-21 | Method and device for transmitting and storing data at edge end |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113032492B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114595080A (en) * | 2022-03-02 | 2022-06-07 | 平凯星辰(北京)科技有限公司 | Data processing method, apparatus, electronic device, and computer-readable storage medium |
| CN114741391A (en) * | 2022-04-13 | 2022-07-12 | 山东浪潮科学研究院有限公司 | Time sequence data storage optimization method and system |
| CN116016267B (en) * | 2023-01-05 | 2026-02-03 | 明阳产业技术研究院(沈阳)有限公司 | Network IP monitoring method and system based on message transmission |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1937550A (en) * | 2006-07-25 | 2007-03-28 | 中山大学 | Message middleware based on digital home network |
| CN110365802A (en) * | 2019-08-26 | 2019-10-22 | 北京奇艺世纪科技有限公司 | A kind of method for message transmission, message forwarding device and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105187327A (en) * | 2015-08-14 | 2015-12-23 | 广东能龙教育股份有限公司 | Distributed message queue middleware |
| CN106534338B (en) * | 2016-12-05 | 2019-05-21 | 东北大学 | A method for realizing cloud robot |
| CN106878473B (en) * | 2017-04-20 | 2021-03-30 | 腾讯科技（深圳）有限公司 | Message processing method, server cluster and system |
| CN107347030B (en) * | 2017-06-29 | 2020-07-14 | 重庆邮电大学 | Message management device based on V2X communication |
| CN107835203B (en) * | 2017-08-09 | 2020-07-10 | 平安壹钱包电子商务有限公司 | Message packet delivery method, device, storage medium and terminal |
| CN111510474B (en) * | 2020-03-26 | 2023-09-29 | 中国平安人寿保险股份有限公司 | Data transmission method based on message middleware and related equipment |
-
2021
- 2021-04-21 CN CN202110431076.XA patent/CN113032492B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1937550A (en) * | 2006-07-25 | 2007-03-28 | 中山大学 | Message middleware based on digital home network |
| CN110365802A (en) * | 2019-08-26 | 2019-10-22 | 北京奇艺世纪科技有限公司 | A kind of method for message transmission, message forwarding device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113032492A (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230145267A1 (en) | Maintaining clinical messaging during an internet outage | |
| CN113032492B (en) | Method and device for transmitting and storing data at edge end | |
| CN106850402B (en) | Message transmission method and device | |
| CN114090366B (en) | Method, device and system for monitoring data | |
| CN111163159B (en) | Message subscription method, device, server and computer readable storage medium | |
| CN112948138B (en) | A method and device for processing messages | |
| CN113157625B (en) | Data transmission method, device, terminal equipment and computer-readable storage medium | |
| CN112433920B (en) | A method, device, equipment and medium for reporting buried data | |
| CN107613529B (en) | Message processing method and base station | |
| CN104243610A (en) | Distributed file transmission service method | |
| CN116938848A (en) | Message uploading method and device, electronic equipment and storage medium | |
| CN113094002B (en) | Message processing method, device, electronic equipment and computer medium | |
| CN110309028B (en) | Monitoring information acquisition method, service monitoring method, device and system | |
| CN112804301B (en) | Method and device for determining equipment state, gateway and storage medium | |
| CN114500036A (en) | Method, device and medium for transmitting data | |
| CN112865927B (en) | Message delivery verification method, device, computer equipment and storage medium | |
| CN111291127A (en) | Data synchronization method, device, server and storage medium | |
| CN113114611A (en) | Method and device for managing blacklist | |
| CN114331261B (en) | Data processing method and device | |
| CN113726885B (en) | Flow quota adjusting method and device | |
| CN113778909B (en) | Method and device for caching data | |
| CN111581244B (en) | Method, system and equipment for orderly synchronizing service transaction data of heterogeneous system | |
| CN112783716B (en) | Monitoring method and device | |
| CN104270444A (en) | A method for configurable mobile phone application menu | |
| CN115220794A (en) | Data processing method, apparatus, electronic device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |