CN108667725B - Industrial soft gateway based on multiple accesses and edge calculation and implementation method - Google Patents
Industrial soft gateway based on multiple accesses and edge calculation and implementation method Download PDFInfo
- Publication number
- CN108667725B CN108667725B CN201810379840.1A CN201810379840A CN108667725B CN 108667725 B CN108667725 B CN 108667725B CN 201810379840 A CN201810379840 A CN 201810379840A CN 108667725 B CN108667725 B CN 108667725B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- sending
- message queue
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 38
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000004891 communication Methods 0.000 claims abstract description 57
- 238000013480 data collection Methods 0.000 claims abstract description 54
- 230000005540 biological transmission Effects 0.000 claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 22
- 230000003993 interaction Effects 0.000 claims abstract description 20
- 230000008569 process Effects 0.000 claims description 15
- 238000002360 preparation method Methods 0.000 claims description 13
- 230000007246 mechanism Effects 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 7
- 230000010354 integration Effects 0.000 claims description 7
- 238000007405 data analysis Methods 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 5
- 230000009467 reduction Effects 0.000 claims description 5
- 230000003068 static effect Effects 0.000 claims description 2
- 238000005096 rolling process Methods 0.000 claims 1
- 238000012544 monitoring process Methods 0.000 abstract description 6
- 230000000007 visual effect Effects 0.000 abstract description 5
- 238000005192 partition Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 3
- 238000004806 packaging method and process Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000012800 visualization Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000010453 quartz Substances 0.000 description 1
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N silicon dioxide Inorganic materials O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/66—Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention provides an industrial soft gateway based on multiple accesses and edge calculation and an implementation method thereof, wherein the industrial soft gateway comprises a configuration interaction module, a data collection module, a data edge calculation module and a data transmission control module; the configuration interaction module comprises a connection configuration module and a data standardization module; the data collection module is used for collecting a plurality of access methods; the data edge calculation module is used for performing real-time calculation processing on the data collected by the data collection module; the data transmission control module is used for caching all data to be transmitted and scheduling and distributing tasks to be transmitted externally. The invention can provide a visual interface to provide connection configuration and standardized operation for various communication protocols and databases, and simultaneously provides a graphical monitoring interface of a data collection state, an edge calculation result and a forwarding state. The data can be standardized, the edge computing module can eliminate data noise and extract data characteristics, the data cloud platform network transmission data volume is reduced, and the data transmission efficiency is improved.
Description
Technical Field
The invention relates to the field of industrial data analysis and establishment, in particular to an industrial soft gateway based on multiple accesses and edge calculation and an implementation method.
Background
The monitoring of the operating state of industrial equipment and fault diagnosis have been a research hotspot in the professional field, and the basis of modeling, analyzing and diagnosing the system equipment is to collect and acquire the operating state data of the equipment. In the data collection of the field industrial equipment, since the applied data collection equipment is usually from different suppliers, various data communication protocols and different kinds of data storage and management systems may be included, so that the collected data has the characteristics of diversity, complexity and non-standardization, and the use, processing, statistics and analysis of the later data are inconveniently affected. Therefore, how to collect data from various communication types in a centralized manner and achieve data uniformity and standardization has important practical significance on the analysis and processing of the operation data of the industrial equipment.
Disclosure of Invention
Based on the background research, the invention aims at common communication protocols in various industrial fields and common database management systems, and realizes an intelligent soft gateway which supports the functions of multi-network communication mode, integrated communication connection of various communication protocols and various databases, data collection, data standardization, edge calculation processing, data sending, edge caching, configuration interaction and remote upgrading. The core content realized by the invention is an industrial intelligent soft gateway based on various access and edge computing technologies, the system supports data collection of Modbus protocol, OPC protocol, Profibus, Profibet protocol, MySql database, Oracle database and SQLServer database, can process in parallel, and provides a collection method of TCPIP or serial port aiming at different network communication protocols. And providing a visualization interface to provide connection configuration and standardized operation for the various communication protocols and databases, and simultaneously providing a graphical monitoring interface of a data collection state, an edge calculation result and a forwarding state. And through the standardization of data, the data is converted into a data format convenient for the unified management of a remote data cloud platform. The data transmission strategy allocation is utilized to realize the control and scheduling of the transmitted data and ensure the optimal transmission of the data. Meanwhile, in order to reduce the transmission pressure of a remote data network, the raw data which needs to be preprocessed and characterized is subjected to local edge computing processing according to the requirement of a remote data cloud platform after data collection, data noise is eliminated, and the characteristic data is extracted and then transmitted, so that the data transmission amount of the data cloud platform network is reduced, and the data transmission efficiency is improved. And the soft gateway distributes the edge cache size according to the hard disk size of the deployed hardware and stores the collected data in a short time. According to the increase and modification of the service requirements, network remote upgrading can be carried out, and errors can be repaired to improve the performance.
The invention can be realized by adopting the following system:
an industrial soft gateway based on multiple accesses and edge calculation comprises a configuration interaction module, a data collection module, a data edge calculation module and a data sending control module;
the configuration interaction module comprises a connection configuration module and a data standardization module; the configuration interaction module is used for performing connection configuration of the integrated data collection module and the field data source through a configuration interaction visual interface; the connection configuration module is used for connecting and configuring a plurality of communication protocols and a plurality of databases, wherein the connecting and configuring of the plurality of communication protocols and the plurality of databases is to establish TCP or serial port connection according to the configuration of connection parameters of different communication protocol types or different database types, configure the connection parameters of the TCP or serial port connection through a client, store corresponding parameter configuration information in an XML file form, and simultaneously store a predicted data address or a scanned data address into the XML file through a visual client interface. And according to the information of the XML file, performing connection with the protocol server, the slave station and the database server, returning the connection state and the reason of connection failure, and manually starting acquisition after the connection is successful.
The data standardization module is used for standardizing various communication protocol types and data of various database types;
the data collection module includes: the scheduling submodule, the collecting submodule and the exception handling submodule;
the data edge calculation module is used for performing real-time calculation processing on the data collected by the data collection module, and comprises: the data integration sub-module, the data analysis sub-module, the filtering noise reduction sub-module and the sending preparation sub-module;
the data transmission control module is used for caching all data to be transmitted and scheduling and distributing tasks to be transmitted; the data sending module comprises a data preparation sub-module, a cycle sending module and an exception resending module. Further, the data sending module sends data according to data requirements of different communication protocols or different types of databases and sends the data to the internet remote data cloud platform through a remote communication network, the sending period depends on the pushing speed of the data collection module to the data sending control module, and the data format is determined when the data collection module packages the data.
Furthermore, the multiple communication protocols comprise three data communication protocols of Modbus, OPC and Profibus, wherein the Modbus protocol comprises a Modbus serial port and Modbus TCP, and the Profibus protocol comprises a Profibus serial port and Profinet.
Further, the database comprises MySql, Oracle and SQLServer databases.
Furthermore, the attribute Modbus protocol comprises an IP address, a serial port name, a port number, slave station information, a register type and a data reading address, the OPC comprises the IP address, a server user name, a password and a device name, the Profibus and Profibet protocol data comprises the IP address, the serial port name, the port number and the data address, and the database comprises the IP address, the port number, a source database table and a timing event.
Further, the standardizing the data of various communication protocol types and various database types specifically includes: respectively establishing block partitions aiming at slave stations established by the Modbus, and then standardizing data in the partitions; aiming at establishing connection with an OPC server, scanning the packet through a scanning device, and then standardizing data; and analyzing the Profibet and Profibus protocols based on a protocol standard message format to obtain analyzed data, and then performing data standardization matching. Further, the normalizing the data of the various communication protocol types and the corresponding database types further comprises: establishing connection aiming at MySql, Oracle and SQLServer type databases, firstly scanning a data source table and a timing event, and collecting and standardizing according to a source data point, a table structure and a newly-built table event, wherein the database table structure is complex and various. The invention also provides a method for realizing the industrial soft gateway based on various accesses and edge calculation, which comprises the following steps:
firstly, performing connection configuration of a data collection module and a corresponding data source through a configuration interaction module;
standardizing data to be collected, and starting a data collection program corresponding to a data source after standardization;
step three, judging whether the collected data needs to be subjected to edge calculation processing, if so, automatically sending the collected data to an edge calculation module for data integration, data analysis, filtering and noise reduction and sending preparation;
step four, sending the data to a data sending control module after the data preparation is finished, and storing the data into an edge cache for backup storage; and the data transmission control module receives the data to be transmitted, and transmits the data according to the request of the data cloud platform according to the transmission processing and scheduling method of the cache message queue.
Further, the data source includes: modbus serial port, ModbusTCP, OPC, Profibus, MySql, Oracle, SQLServer data source.
Aiming at common communication protocols in various industrial fields and common database management systems, the invention realizes an intelligent soft gateway which supports the functions of multi-network communication mode, integrated communication connection of various communication protocols and various databases, data collection, data standardization, edge calculation processing, data transmission, edge caching, configuration interaction and remote upgrading.
The industrial intelligent soft gateway system has the beneficial effects that the core content realized by the system is the industrial intelligent soft gateway based on various access and edge computing technologies, the system supports data collection of Modbus protocol, OPC protocol, Profibus, Profibet protocol, MySql database, Oracle database and SQLServer database, parallel processing can be realized, and a TCPIP or serial port collection method is provided for different network communication protocols. And providing a visualization interface to provide connection configuration and standardized operation for the various communication protocols and databases, and simultaneously providing a graphical monitoring interface of a data collection state, an edge calculation result and a forwarding state. And through the standardization of data, the data is converted into a data format convenient for the unified management of a remote data cloud platform. The data transmission strategy allocation is utilized to realize the control and scheduling of the transmitted data and ensure the optimal transmission of the data. Meanwhile, in order to reduce the transmission pressure of a remote data network, the raw data which needs to be preprocessed and characterized is subjected to local edge computing processing according to the requirement of a remote data cloud platform after data collection, data noise is eliminated, and the characteristic data is extracted and then transmitted, so that the data transmission amount of the data cloud platform network is reduced, and the data transmission efficiency is improved. And the soft gateway distributes the edge cache size according to the hard disk size of the deployed hardware and stores the collected data in a short time. According to the increase and modification of the service requirements, network remote upgrading can be carried out, and errors can be repaired to improve the performance.
Drawings
FIG. 1 is a block diagram of an embodiment of an industrial soft gateway based on multiple access and edge computing according to the present invention;
FIG. 2 is a flow chart of an implementation of the method for an industrial soft gateway based on multiple accesses and edge computations according to the present invention;
FIG. 3 is a flow chart of the core function of the industrial soft gateway provided by the present invention;
FIG. 4 is a schematic diagram illustrating an implementation of an operating program of a configuration interaction module according to the present invention;
FIG. 5 is a schematic diagram of a data collection procedure connected via a communication network according to the present invention;
FIG. 6 is a schematic diagram of a data collection procedure implemented via a database connection according to the present invention;
FIG. 7 is a schematic diagram illustrating an implementation of an operation procedure of the data edge calculation module according to the present invention;
fig. 8 is a schematic view illustrating an implementation of an operation program of the data transmission control module according to the present invention;
Detailed Description
In order to make the technical solutions in the embodiments of the present invention better understood and make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention provides an industrial soft gateway based on multiple accesses and edge computations and an implementation method thereof, and the technical solutions in the present invention are further described in detail below with reference to the accompanying drawings:
the invention can be realized by adopting the following system:
an industrial soft gateway based on multiple accesses and edge calculation comprises a configuration interaction module 101, a data collection module 102, a data edge calculation module 103 and a data transmission control module 104;
the configuration interaction module 101 comprises a connection configuration module 1011 and a data standardization module 1012; the configuration interaction module 101 is configured to perform connection configuration between the integrated data collection module 102 and the field data source through a configuration interaction visualization interface; the connection configuration module 1011 is configured to connect multiple communication protocols with corresponding databases, where the connection configuration of multiple communication protocols with corresponding databases is performed according to respective attributes of the slave stations or clients of each communication protocol type, and when configuration of one or more clients or slave stations of all types is completed, corresponding parameter configuration information is stored in an XML file form, so as to complete data collection connection configuration;
specifically, if the connection fails, three times of automatic reconnection are performed until the connection is successful, if the three times of connection fail, the connection state is returned, the original connection state monitoring program is returned, and the operator determines when to perform the connection again. And if the connection is successful, returning to the connection success state.
The data standardization module 1012 is used for standardizing data of various communication protocol types and corresponding database types;
the data collection module 102 includes: a collection submodule 1021 and an exception handling submodule 1022;
the data collection module runs a data collection subprogram, the data collection subprogram mainly comprises a ModbusTCP, a Modbus serial port, Profibus, Profibet, an OPC communication data collection subprogram and MySql, Oracle and SQLServer database data collection subprograms, each collection subprogram respectively comprises a collection submodule and an exception handling submodule, the collection frequency and thread allocation are realized through a scheduling submodule 1021, the Quartz technology is used for realizing the collection, the conditions such as deadlock and the like are avoided, exceptions are processed as soon as possible, and the collection is carried out orderly according to the frequency. The ModbusTCP, Profibet and OPC communication data collection subprograms are only differences in protocol analysis processing in implementation, main communication modes of the ModbusTCP, Profibet and OPC communication data collection subprograms conform to a TCP/IP communication protocol, and the Modbus serial port and Profibus serial port collection subprograms are only differences in connection attributes, so that the ModbusTCP communication data collection subprograms are taken as an example to explain working mechanisms of the ModbusTCP communication data collection subprograms. The MySql, Oracle and SQLServer database collection subprograms are only different in implementation of the loaded drive in the JDBC data connection process, so the MySql database collection subprogram is taken as an example to explain the working mechanism of the MySql database collection subprogram.
In the development process of the Modbus collection subprogram, through a large number of simulation tests, the optimal read quantity of single-thread serial is measured to be 120 according to the reading speed and stability as measurement indexes, namely if the Modbus collection program is single-thread reading, the conditions that the program is stable for a long time and does not throw out abnormity, data loss or collection errors are caused by breakpoints and the like can be guaranteed through one-time Modbus TCP connection of at most 120 item serial reading, and therefore the Modbus collection subprogram is used as priori knowledge for Modbus collection of the system. The Modbus collection subprogram mainly comprises a Modbus collection part and an exception handling part, and the specific program flow is as follows, wherein the Modbus collection part program firstly obtains standardized configuration information, then adopts single thread serial reading optimal quantity to be 120 continuous data of a single BLOCK partition, if the data in the partition is more than 120, multithreading is automatically started, Modbus TCP connection is established and collected according to the optimal reading quantity of 120 as a unit, all data reading of the single BLOCK partition is completed in a circulating mode, asynchronous multithreading is automatically distributed, each thread is enabled to accord with the quantity of the single thread serial optimal reading data, and each thread is read by adopting a Jamod technology. If the number of the data in the partition is less than 120, the serial reading is finished. After all reading operations are finished, all data integration of a single BLOCK partition is carried out in a data reading result buffer pool, data serialization is carried out after all BLOCK partitions are respectively subjected to data integration, then data is stamped and packaged, and a program enters a data sending preparation state. Meanwhile, in the whole data collection process, thread exception handling mechanisms such as thread blocking, thread waiting and the like are designed to handle thread exception, and meanwhile, a reconnection mechanism is started when connection is interrupted in the program design, so that the normal communication connection state in the data collection process and the condition of handling connection interruption are ensured.
The MySql database data source collection subprogram is composed of a data reading program and an exception handling program and is divided into real-time data reading and historical data reading, and the working mechanism of a real-time data reading thread is as follows: the data reading program mainly comprises three threads, and the functions of the data reading program are a real-time data reading thread and a historical data reading thread and the function of identifying the change frequency of a real-time database. The identification of the real-time database change frequency thread provides assistance for the real-time data reading thread, the identification can be carried out at regular time, if the database on the site is collected and can be used for outsiders to write interface programs in an individualized way, the identification can be designed and developed as blocking monitoring identification, namely, the change frequency of the real-time database does not need to be actively identified at regular time, but the thread is activated to actively access by waiting for an interface instruction, so that the computing resources are saved. The working mechanism of the real-time data reading thread is as follows: firstly, acquiring standardized configuration information, reading data such as a database, a data source table, a target data point, a data packet and the like according to real-time data change frequency, wherein the collected database table on site has various designs, and can be personalized and modified according to the site condition, then, an SQL query statement is established, data query is carried out, a query result is returned, after a timestamp is added to the query data point or data packet return time, the data is standardized according to the data standardized configuration requirement, data serialization and packaging are carried out, and the data enters the final data sending preparation state. The working mechanism of the historical data reading thread is as follows: firstly, manually selecting a data source and a data table according to standardized configuration information, judging reading frequency and reading size according to current network pressure and the size of a source data table, establishing an SQL query statement according to collection frequency, executing query and returning a reading result, then, similarly performing data standardization, serialization and packaging, and entering a final data sending preparation state. In the thread execution process of the database source data collection process, thread exception handling programs such as thread waiting and thread blocking are also set, and meanwhile, mechanisms for handling the exception of database connection interruption and reconnection are also considered and designed, so that the normal connection state of the database is ensured.
The data edge calculation module 103 is configured to perform real-time calculation processing on the data collected by the data collection module, where the data edge calculation module 103 includes: data integration sub-module 1031, data analysis sub-module 1032, filtering noise reduction sub-module 1033, and sending preparation sub-module 1034;
specifically, after real-time collected data needing real-time processing is acquired, the collected data are classified according to rules and algorithms in an edge calculation rule base and an algorithm base, then the data are preprocessed, data feature extraction processing, data aggregation and classification and the like according to specific rules and algorithms, the processed data are filtered and denoised according to business knowledge, namely noise data caused by calculation or collection errors are deleted according to the business features of the data, and finally the data enter a data sending process through serialization, time stamping and packaging.
The data transmission control module 104 is configured to cache all data to be transmitted and schedule and allocate tasks to be forwarded to the outside; the data sending module includes a data preparation sub-module 1041, a loop sending module 1042, and an exception resending module 1043.
Preferably, the data sending module sends data according to data requirements of different communication protocols or different types of databases and sends the data to the internet remote data cloud platform through a remote communication network, the sending period depends on the pushing speed of the data collection module to the data sending control module, and the data format is determined when the data collection module packages the data.
The data sending module can send data in real time, and meanwhile, if the connection with the remote data platform is interrupted, the data is queued in the message queue to be sent, and the message is still in the memory after the network connection is disconnected, but is not immediately stored in the file, so that the reading speed is high, and hard disk resources are not consumed; and if the message queue is full, locally caching the unsent data to an Avro file, and automatically resending the unsent data after connection is recovered. The Avro format is adopted because the Json format is a form corresponding to key values, when one message sends a plurality of groups of data with the same source data, the standardized data information can be repeatedly sent for a plurality of times, and the formatted data can only record the standardized data information once during network transmission, and meanwhile, the data of the type is convenient for a message receiver to analyze, and can be serialized into binary data to realize data compression, thereby solving the problem of low network transmission efficiency caused by the increase of network transmission quantity due to the transmission of repeated information.
The data sending module mainly completes the data caching and sending functions of the module by the parallel operation of three threads, wherein the first thread is a data receiving thread, and the specific working process is as follows: before data enters a message queue, whether the data is encrypted or not is selected by a user, if the data is encrypted, the data is encrypted by adopting an AES or RSA algorithm, then the data enters the message queue, and if the data is not encrypted, the data directly enters the message queue. And after the data enters the message queue, acquiring the current latest storage bit. The current storage bit is a static variable and is stored by adopting a circular queue, if the current storage bit is not processed, the message queue is full, if the current storage bit is processed, the message queue is used for indicating that remote transmission is realized, if the unprocessed data is covered by new data, the unprocessed data can be immediately stored in a cache file to be transmitted, and when network connection is available, the data can be retransmitted and deleted from the cache file to be transmitted. And if the message queue is full, establishing a corresponding Avro file, and copying the message queue data to the Avro file one by one. The second thread is used for circularly sending the message queue, and the work flow is as follows: reading a message queue, acquiring a current latest reading position, judging whether a current storage bit of the current message queue is processed or not, if the current storage bit of the current message queue is not processed, sending data, if communication is interrupted in the sending process, releasing connection for resending, if the communication is normal, sending the data by adopting an MQTT protocol, after the sending is finished, updating the processing state of the current data reading position to be processed, updating the current latest reading position to a next queue space, if the data at the current position is processed, indicating that no data waiting for sending exists in the queue, and stopping a reading position pointer at the position at the moment. The third thread is used for resending the data backed up to the data file in a cache manner, the Avro file is read firstly, data sending is carried out according to the MQTT protocol, if sending is finished, the sent data is deleted in the file, if sending is not finished, resending is carried out until sending is successful, and the specific implementation mode is as follows: if the transmission is not finished, the data reading pointer is rolled back, which is equivalent to that the whole reading and transmitting process is repeated in the next transmission, so that the problem of data loss to be transmitted caused by unexpected power failure, program quitting and the like is avoided. And after all the data in the current Avro file are sent, automatically searching the next Avro file, and sending the data according to the same sending mechanism. In the whole data sending process, all data entering a sending queue has a local storage mechanism, which is irrelevant to whether the data is sent successfully or not, such as all data are stored, each queue stores the first data, and the like.
Preferably, the plurality of communication protocols comprise three data communication protocols of Modbus, OPC and Profibus, wherein the Modbus protocol comprises a Modbus serial port and Modbus TCP, and the Profibus protocol comprises a Profibus serial port and Profinet.
Preferably, the database comprises a MySql, Oracle, SQLServer database.
Preferably, the attribute Modbus protocol includes an IP address, a serial port name, a port number, slave station information, a register type, and a data reading address, the OPC includes an IP address, a server user name, a password, and a device name, the Profibus, ProfiNet protocol data includes an IP address, a serial port name, a port number, and a data address, and the database includes an IP address, a port number, a source database table, and a timing event.
Specifically, Modbus protocol data are collected through connection of ModbusTCP communication or serial port communication and a field master station, OPC protocol data are collected through connection of OPC communication and a field OPC server, Profibet protocol data are collected through IT communication through IEEE 802.3 Ethernet standard and TCP/IP, Profibus protocol data are collected through communication through a serial port, database data such as MySql, Oracle and SQLServer are collected through JDBC connection established with the field database server on the basis of TCP/IP communication protocols.
Preferably, the standardizing the data of various communication protocol types and various database types is specifically: respectively establishing block partitions aiming at slave stations established by the Modbus, and then standardizing data in the partitions; aiming at establishing connection with an OPC server, scanning the packet through a scanning device, and then standardizing data; according to the Profibet and Profibus protocols, the protocol standard message format is analyzed, the analyzed data is subjected to data standardization matching to complete the standardization operation process, then the connection test is carried out, the data collection state is entered after the connection is successful, the data collection state can be manually or automatically operated, namely, the data collection module is started immediately after standardization or the data collection is started manually by an operator.
Preferably, the normalizing the data of the various communication protocol types and the corresponding database types further comprises: establishing connection aiming at MySql, Oracle and SQLServer type databases, firstly scanning a data source table and a timing event, and collecting and standardizing according to a source data point, a table structure and a newly-built table event, wherein the database table structure is complex and various. The invention also provides a method for realizing the industrial soft gateway based on various accesses and edge calculation, which comprises the following steps:
s201, performing connection configuration of a data collection module and a corresponding data source through a configuration interaction module;
s202, standardizing the data to be collected, and starting a plurality of data collection programs corresponding to the standardized data sources after connection testing;
s203, judging whether the collected data need to be subjected to edge calculation processing or not, if so, automatically sending the collected data to an edge calculation module for data integration, data analysis, filtering and noise reduction and sending preparation;
s204, step four, sending the data to a data sending control module after the data preparation is finished, and storing the data into an edge cache for backup storage; and the data transmission control module receives the data to be transmitted, and transmits the data according to the request of the data cloud platform according to the transmission processing and scheduling method of the cache message queue.
Preferably, the data source comprises: ModbusTCP, Modbus serial port, OPC, Profibus, Profibet, MySql, Oracle and SQLServer data source.
The invention provides an industrial soft gateway based on multiple accesses and edge calculation and an implementation method thereof, wherein the industrial soft gateway comprises a configuration interaction module, a data collection module, a data edge calculation module and a data transmission control module;
the configuration interaction module comprises a connection configuration module and a data standardization module; the configuration interaction module is used for performing connection configuration of the integrated data collection module and the field data source through a configuration interaction visual interface; the connection configuration module is used for connecting and configuring a plurality of communication protocols and a plurality of databases, and the data standardization module is used for standardizing data of various communication protocol types and the plurality of database types; the data collection module includes: the scheduling submodule, the collecting submodule and the exception handling submodule; the data edge calculation module is used for performing real-time calculation processing on the data collected by the data collection module, and the data transmission control module is used for caching all data to be transmitted and scheduling and distributing tasks to be transmitted externally. The invention can provide a visual interface to provide connection configuration and standardized operation for various communication protocols and databases, and simultaneously provides a graphical monitoring interface of a data collection state, an edge calculation result and a forwarding state. And through the standardization of data, the data is converted into a data format convenient for the unified management of a remote data cloud platform. Meanwhile, the edge computing module can eliminate data noise, reduce the data transmission amount of the data cloud platform network and improve the data transmission efficiency. And the soft gateway distributes the edge cache size according to the hard disk size of the deployed hardware and stores the collected data in a short time. According to the increase and modification of the service requirements, network remote upgrading can be carried out, and errors can be repaired to improve the performance.
The above examples are intended to illustrate but not to limit the technical solutions of the present invention. Any modification or partial replacement without departing from the spirit and scope of the present invention should be covered in the claims of the present invention.
Claims (2)
1. A method for realizing an industrial soft gateway based on multiple accesses and edge calculation is characterized by comprising the following steps:
firstly, performing connection configuration of a data collection module and a corresponding data source through a configuration interaction module;
standardizing data to be collected, and starting a data collection program corresponding to a data source after standardization;
step three, judging whether the collected data needs to be subjected to edge calculation processing, if so, automatically sending the collected data to an edge calculation module for data integration, data analysis, filtering and noise reduction and sending preparation;
step four, sending the data to a data sending control module after the data preparation is finished, and storing the data into an edge cache for backup storage; the data transmission control module receives data to be transmitted, and the data is transmitted according to the requirements of the data cloud platform according to the transmission processing and scheduling method of the cache message queue;
the data transmission control module mainly completes the data caching and transmitting functions of the module by the parallel operation of three threads;
the first thread is a data receiving thread, and the specific work flow is as follows: before data enters a message queue, whether the data is encrypted or not is selected by a user, if the data is encrypted, the data is encrypted by adopting an AES (advanced encryption standard) or RSA (rivest-Shamir-Adleman) algorithm, then the data enters the message queue, if the data does not are encrypted, the data directly enters the message queue, after the data enters the message queue, the current latest storage bit is obtained, the current storage bit is a static variable and is stored by adopting a circular queue, if the current storage bit is not processed, the message queue is full, if the current storage bit is processed, the remote sending is realized, if the unprocessed data is covered by new data, the data is immediately stored into a cache file to be sent, and when network connection is available, the data is retransmitted and deleted from the cache file to; if the message queue is full, establishing a corresponding Avro file, and copying the message queue data to the Avro file one by one;
the second thread is used for circularly sending the message queue, and the work flow is as follows: reading a message queue, acquiring a current latest reading position, judging whether a current storage bit of the current message queue is processed or not, if the current storage bit of the current message queue is not processed, sending data, if communication is interrupted in the sending process, releasing connection for resending, if the communication is normal, sending the data by adopting an MQTT protocol, after the sending is finished, updating the processing state of the current data reading position to be processed, updating the current latest reading position to a next queue space, if the data at the current position is processed, indicating that no data waiting for sending exists in the queue, and stopping a reading position pointer at the position;
the third thread is used for resending the data backed up to the data file in a cache manner, the Avro file is read firstly, data sending is carried out according to the MQTT protocol, if sending is finished, the sent data is deleted in the file, if sending is not finished, resending is carried out until sending is successful, and the specific implementation mode is as follows: if the transmission is not finished, rolling back the data reading pointer, which is equivalent to that the whole reading and transmitting process is repeated in the next transmission, so that the problem of data to be transmitted loss caused by unexpected power failure and program quitting is avoided; and after all the data in the current Avro file are sent, automatically searching the next Avro file, and sending the data according to the same sending mechanism.
2. The method for implementing an industrial soft gateway based on multiple access and edge computing according to claim 1, wherein the data source comprises: modbus serial port, ModbusTCP, OPC, Profibus, Profibet, MySql, Oracle, SQLServer data source.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810379840.1A CN108667725B (en) | 2018-04-25 | 2018-04-25 | Industrial soft gateway based on multiple accesses and edge calculation and implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810379840.1A CN108667725B (en) | 2018-04-25 | 2018-04-25 | Industrial soft gateway based on multiple accesses and edge calculation and implementation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108667725A CN108667725A (en) | 2018-10-16 |
| CN108667725B true CN108667725B (en) | 2021-06-11 |
Family
ID=63780904
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810379840.1A Active CN108667725B (en) | 2018-04-25 | 2018-04-25 | Industrial soft gateway based on multiple accesses and edge calculation and implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108667725B (en) |
Families Citing this family (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109672723A (en) * | 2018-10-24 | 2019-04-23 | 国网河北省电力有限公司雄安新区供电公司 | Comprehensive energy application service system |
| CN111290879A (en) * | 2018-11-21 | 2020-06-16 | 上海安浦鸣志自动化设备有限公司 | Modbus protocol dynamic link library implementation method and system with automatic reconnection mechanism |
| CN110032535B (en) * | 2018-12-28 | 2021-06-29 | 北京润科通用技术有限公司 | Serial port data analysis method and device |
| CN109491301A (en) * | 2019-01-23 | 2019-03-19 | 东莞固高自动化技术有限公司 | Industrial internet intelligent controller based on edge computing system architecture |
| CN109885566A (en) * | 2019-02-25 | 2019-06-14 | 南京世界村云数据产业集团有限公司 | A kind of acquisition of data and edge calculations system |
| CN109933004B (en) * | 2019-03-27 | 2022-05-24 | 苏芯物联技术(南京)有限公司 | Machine tool fault diagnosis and prediction method and system based on edge computing and cloud cooperation |
| CN110113257B (en) * | 2019-05-14 | 2021-06-08 | 北京京投亿雅捷交通科技有限公司 | Unified data access gateway based on big data and implementation method |
| CN110221558B (en) * | 2019-06-05 | 2020-09-01 | 镇江四联机电科技有限公司 | Electro-hydraulic servo valve online fault diagnosis gateway based on edge computing technology |
| CN110191025B (en) * | 2019-06-06 | 2021-06-08 | 国家工业信息安全发展研究中心 | Intelligent gateway data acquisition and test method and device |
| CN110209383B (en) * | 2019-06-12 | 2025-05-13 | 广东外语外贸大学南国商学院 | Ubiquitous object interaction interface method for pan-manufacturing and edge computing |
| CN110266677A (en) * | 2019-06-13 | 2019-09-20 | 广州中国科学院沈阳自动化研究所分所 | An edge computing intelligent gateway for industrial manufacturing and its implementation method |
| CN110191138A (en) * | 2019-06-13 | 2019-08-30 | 广州中国科学院沈阳自动化研究所分所 | An edge computing intelligent adapter for industrial manufacturing and its implementation method |
| CN110365529B (en) * | 2019-07-10 | 2022-03-22 | 广州博依特智能信息科技有限公司 | A kind of edge computing intelligent gateway business processing method and edge computing intelligent gateway |
| CN110398264A (en) * | 2019-07-31 | 2019-11-01 | 联想(北京)有限公司 | A kind of equipment condition monitoring method and system |
| CN110502719A (en) * | 2019-08-06 | 2019-11-26 | 上海展湾信息科技有限公司 | Edge calculations model management system and method |
| CN110579998A (en) * | 2019-10-24 | 2019-12-17 | 苏州良将智能科技有限公司 | Data acquisition method |
| CN110661699A (en) * | 2019-10-31 | 2020-01-07 | 朗狮(深圳)科技有限公司 | Gateway for multi-protocol connection based on edge calculation |
| CN110888354A (en) * | 2019-11-21 | 2020-03-17 | 深圳赛动生物自动化有限公司 | Method and device for realizing automatic identification of access equipment, computer equipment and medium |
| CN110989440A (en) * | 2019-12-06 | 2020-04-10 | 福建和盛高科技产业有限公司 | Portable anti-theft industrial gateway |
| CN111031034A (en) * | 2019-12-11 | 2020-04-17 | 研祥智能科技股份有限公司 | Multi-protocol convergence edge computing gateway |
| CN111083047B (en) * | 2019-12-17 | 2022-02-01 | 首都航天机械有限公司 | Gateway based on internet of things multi-protocol communication |
| CN111224840A (en) * | 2019-12-31 | 2020-06-02 | 江苏南高智能装备创新中心有限公司 | Gateway system with fault diagnosis performance and method thereof |
| CN113094292B (en) | 2020-01-09 | 2022-12-02 | 上海宝存信息科技有限公司 | Data storage device and non-volatile memory control method |
| CN111371661B (en) * | 2020-03-23 | 2022-03-25 | 许昌中科森尼瑞技术有限公司 | System and method for realizing communication of multiple monitoring backgrounds based on CAN bus |
| CN111478960A (en) * | 2020-04-03 | 2020-07-31 | 河海大学常州校区 | Data acquisition and edge calculation system based on edge calculation |
| CN111523782B (en) * | 2020-04-14 | 2023-04-28 | 杭州迪普科技股份有限公司 | Industrial control asset management method, device, equipment and storage medium |
| US11159449B1 (en) * | 2020-07-09 | 2021-10-26 | International Business Machines Corporation | Dispatching tasks and data using multi-access edge computing |
| CN111954318B (en) * | 2020-07-20 | 2022-06-10 | 广东工贸职业技术学院 | Equipment interconnection method, device and system |
| CN114169365B (en) * | 2020-11-25 | 2025-07-15 | 摩赛恩科技(苏州)有限公司 | Mass spectrometry data processing system and method |
| CN112688997B (en) * | 2020-12-17 | 2022-07-22 | 重庆邮电大学 | RPA robot-based universal data acquisition and management method and system |
| CN113242247B (en) * | 2021-05-17 | 2022-08-26 | 佳木斯大学 | Industrial intelligent Internet of things module based on edge calculation |
| CN113364674B (en) * | 2021-05-26 | 2022-04-26 | 博依特(广州)工业互联网有限公司 | Edge computer network relation system |
| CN114024801B (en) * | 2021-10-26 | 2024-03-01 | 江苏中安智信通信科技股份有限公司 | Industrial edge computing service gateway |
| CN114900570B (en) * | 2022-07-13 | 2022-10-18 | 江西联创精密机电有限公司 | Standardized data acquisition and transmission method and system |
| CN115514786A (en) * | 2022-09-09 | 2022-12-23 | 通号（长沙）轨道交通控制技术有限公司 | A method for constructing a simplified data set on the edge side of a gateway |
| CN115801884B (en) * | 2022-09-22 | 2025-06-20 | 国网河北省电力有限公司经济技术研究院 | Communication network master and slave station data connection method |
| CN115658647A (en) * | 2022-09-28 | 2023-01-31 | 国能信控互联技术有限公司 | ODBC (open data base) specification-based universal relational database edge data acquisition method and system |
| CN116112357B (en) * | 2023-02-14 | 2024-08-13 | 中电华瑞电气有限公司 | Visual configuration soft gateway system and implementation method thereof |
| CN119094636B (en) * | 2024-10-28 | 2025-04-15 | 雷玺智能科技(上海)有限公司 | Fast docking method and system for power monitoring equipment based on edge computing gateway |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203057061U (en) * | 2012-11-13 | 2013-07-10 | 山东御馨生物科技有限公司 | A Novel Motor Control Loop Applicable to Protein Production DCS System |
| CN104486783A (en) * | 2014-12-17 | 2015-04-01 | 广州中国科学院沈阳自动化研究所分所 | Multi-state wireless gateway system for multi-state wireless monitoring network and control method |
| CN104954242A (en) * | 2014-03-26 | 2015-09-30 | 洛克威尔自动控制技术股份有限公司 | Unified data ingestion adapter for migration of industrial data to a cloud platform |
| CN105530240A (en) * | 2015-11-27 | 2016-04-27 | 机械工业仪器仪表综合技术经济研究所 | Method for automatically mapping multiple industrial protocols to OPC UA address space |
| CN106685815A (en) * | 2017-01-04 | 2017-05-17 | 上海建坤信息技术有限责任公司 | Multi - protocol hardware acquisition gateway system |
| CN106936838A (en) * | 2017-03-29 | 2017-07-07 | 苏州中材建设有限公司 | It is applied to the control system and control method of industrial equipment communication Protocol Conversion |
| CN107770263A (en) * | 2017-10-16 | 2018-03-06 | 电子科技大学 | A kind of internet-of-things terminal safety access method and system based on edge calculations |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104679596A (en) * | 2013-12-02 | 2015-06-03 | 航天信息股份有限公司 | Message processing method and system for improving concurrence performance of server-side |
| US9614646B2 (en) * | 2015-03-20 | 2017-04-04 | Vmware, Inc. | Method and system for robust message retransmission |
| CN105138410A (en) * | 2015-08-31 | 2015-12-09 | 北京锐安科技有限公司 | Message queue achievement method and device based on disk buffer |
| CN105634984A (en) * | 2015-12-29 | 2016-06-01 | 重庆安碧捷科技股份有限公司 | Multithread many-to-many medical data transmission method based on TCP protocol |
| US10445141B2 (en) * | 2016-08-18 | 2019-10-15 | Honeywell International Inc. | System and method supporting single software code base using actor/director model separation |
| CN107291865A (en) * | 2017-06-13 | 2017-10-24 | 苏州弘铭检测科技有限公司 | A kind of workshop appliance monitoring system describes method |
| CN107370667B (en) * | 2017-07-31 | 2021-05-07 | 北京北信源软件股份有限公司 | Multithreading parallel processing method and device, readable medium and storage controller |
| CN107895176B (en) * | 2017-11-13 | 2021-08-24 | 国网湖南省电力有限公司 | A fog computing system and method for wide-area monitoring and diagnosis of hydroelectric cluster |
-
2018
- 2018-04-25 CN CN201810379840.1A patent/CN108667725B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203057061U (en) * | 2012-11-13 | 2013-07-10 | 山东御馨生物科技有限公司 | A Novel Motor Control Loop Applicable to Protein Production DCS System |
| CN104954242A (en) * | 2014-03-26 | 2015-09-30 | 洛克威尔自动控制技术股份有限公司 | Unified data ingestion adapter for migration of industrial data to a cloud platform |
| CN104486783A (en) * | 2014-12-17 | 2015-04-01 | 广州中国科学院沈阳自动化研究所分所 | Multi-state wireless gateway system for multi-state wireless monitoring network and control method |
| CN105530240A (en) * | 2015-11-27 | 2016-04-27 | 机械工业仪器仪表综合技术经济研究所 | Method for automatically mapping multiple industrial protocols to OPC UA address space |
| CN106685815A (en) * | 2017-01-04 | 2017-05-17 | 上海建坤信息技术有限责任公司 | Multi - protocol hardware acquisition gateway system |
| CN106936838A (en) * | 2017-03-29 | 2017-07-07 | 苏州中材建设有限公司 | It is applied to the control system and control method of industrial equipment communication Protocol Conversion |
| CN107770263A (en) * | 2017-10-16 | 2018-03-06 | 电子科技大学 | A kind of internet-of-things terminal safety access method and system based on edge calculations |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108667725A (en) | 2018-10-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108667725B (en) | Industrial soft gateway based on multiple accesses and edge calculation and implementation method | |
| CN112671560B (en) | High-availability distributed real-time alarm processing method and system | |
| CN105074698B (en) | It is performed in parallel continuous events processing (CEP) inquiry | |
| CN100568193C (en) | Systems and methods for performance management in a multi-tier computing environment | |
| CN114745295A (en) | Data acquisition method, device, equipment and readable storage medium | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| CN112313627A (en) | Mapping mechanism of events to serverless function workflow instances | |
| CN117573680A (en) | Positioning data transmission management system and method based on big data | |
| CN112817539A (en) | Industrial data storage method and system, electronic device and storage medium | |
| CN113570347A (en) | RPA operation and maintenance method for micro-service architecture system | |
| CN119557052A (en) | Virtual machine indicator data monitoring method, device, equipment and medium | |
| CN112260902B (en) | Network equipment monitoring method, device, equipment and storage medium | |
| CN105446707B (en) | Data conversion method | |
| CN117992522A (en) | Standardized processing method and system for multi-source heterogeneous data | |
| CN116016117A (en) | Network equipment operation and maintenance data acquisition method and system, electronic equipment and storage medium | |
| CN109921963B (en) | Network state inspection method and system | |
| CN112817613A (en) | Plug-in data deleting method, server, electronic equipment and storage medium | |
| CN116126619A (en) | Method and device for processing database node information | |
| CN108805741B (en) | A method, device and system for fusion of power quality data | |
| CN116737834B (en) | Real-time synchronization system for dealer multi-tenant database and table division scene data | |
| CN117609315B (en) | Data processing method, device, equipment and readable storage medium | |
| CN112818059B (en) | Information real-time synchronization method and device based on container release platform | |
| CN116737274A (en) | Data processing method and device and computing equipment | |
| CN111654410B (en) | Gateway request monitoring method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |