CN103731298A - Large-scale distributed network safety data acquisition method and system - Google Patents
Large-scale distributed network safety data acquisition method and system Download PDFInfo
- Publication number
- CN103731298A CN103731298A CN201310572103.0A CN201310572103A CN103731298A CN 103731298 A CN103731298 A CN 103731298A CN 201310572103 A CN201310572103 A CN 201310572103A CN 103731298 A CN103731298 A CN 103731298A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- acquisition
- transmission
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 231100000279 safety data Toxicity 0.000 title abstract 2
- 230000005540 biological transmission Effects 0.000 claims abstract description 66
- 238000009826 distribution Methods 0.000 claims abstract description 54
- 238000004458 analytical method Methods 0.000 claims abstract description 44
- 238000007405 data analysis Methods 0.000 claims abstract description 25
- 230000007246 mechanism Effects 0.000 claims abstract description 5
- 238000012544 monitoring process Methods 0.000 claims description 32
- 239000003795 chemical substances by application Substances 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 18
- 238000004519 manufacturing process Methods 0.000 claims description 14
- 238000003860 storage Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 230000010076 replication Effects 0.000 claims description 6
- 230000006978 adaptation Effects 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 3
- 238000005538 encapsulation Methods 0.000 claims description 3
- 230000003213 activating effect Effects 0.000 claims description 2
- 230000003139 buffering effect Effects 0.000 claims description 2
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 239000000284 extract Substances 0.000 claims description 2
- 230000002085 persistent effect Effects 0.000 claims description 2
- 238000011084 recovery Methods 0.000 claims description 2
- 238000012546 transfer Methods 0.000 claims description 2
- 238000013480 data collection Methods 0.000 claims 4
- 238000012549 training Methods 0.000 claims 1
- 238000005516 engineering process Methods 0.000 abstract description 8
- 238000007726 management method Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 7
- 238000004140 cleaning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a large-scale distributed network safety data acquisition method and system. The method comprises the steps of multimode data acquisition, data analysis and standardization and data distribution and transmission. The system comprises an acquisition agent module, a data acquisition module, a data analysis module and a data distribution and transmission module. With respect to data acquisition, multiple modes such as an active mode, a passive mode and a data stream mirror image mode are adopted, and comprehensive acquisition of various types of data is realized; with respect to data analysis, a data analysis and standardization mechanism based on strategies is adopted, original data are extracted, mapped, replaced, supplemented and the like by means of writing analysis strategies, and therefore quick analysis of a newly added data format and data standardization oriented to multiple application systems are realized; with respect to transmission, the multi-stage connection technology and the multi-path distribution technology are adopted, elastic combination, cascading deployment and multi-path distribution between acquisition systems are realized, and the requirements for vertical and horizontal expansion of a network environment and acquisition of mass data information are met.
Description
Technical Field
The invention belongs to the field of network security management, and relates to a large-scale distributed network security data acquisition method and system.
Background
Data acquisition is a precondition for network security management and operation and maintenance. The data acquisition tool realizes the comprehensive acquisition, analysis, cleaning and standardization of security events, log information, running states, system configuration, security strategies and the like generated by various software and hardware resources in the network system, and converts the security events, the log information, the running states, the system configuration, the security strategies and the like into usable data which can be identified, managed, exchanged and shared by an upper-layer system.
At present, in the field of network security management, there are a plurality of data acquisition tools, which can effectively solve data acquisition and cleaning tasks in certain specific environments, but with the continuous improvement of informatization level, a large number of information systems and advanced information technologies are deployed and used on a large scale, which poses a serious challenge to security management, and simultaneously, new requirements are also raised for data acquisition technologies: the method supports diversified acquisition modes, and in one set of network system, one set of data acquisition tools are adopted to realize comprehensive acquisition of different network security data, rather than a plurality of sets of acquisition tools are deployed to realize acquisition requirements of different data sources; and the fast analysis of multi-source and heterogeneous data and the multi-standard-oriented data reconstruction capability are supported. The traditional data acquisition tool can only analyze one or more data sources with specific formats, frequently needs secondary customized development for newly added data formats, does not support the requirement of constructing different data standards for different upper application systems, and cannot meet the requirement of safety management on quick information deployment and dynamic change. And thirdly, a dynamic extensible deployment mode is supported, the data acquisition tools can be dynamically combined and adjusted along with the expansion of the network scale, the derivation of the network hierarchy and the dynamic increase of the data volume, and the requirements of complex and variable network environment and massive security event information acquisition are met.
Disclosure of Invention
Aiming at the problems, the invention provides a large-scale distributed network security data acquisition method and system. In the aspect of data acquisition, 4 modes of active and passive data sent by a previous-stage acquisition system and data stream mirror image are adopted, so that all kinds of data are comprehensively acquired; in the aspect of data analysis, a data analysis and standardization mechanism based on a strategy is adopted, and the operations of extraction, mapping, replacement, completion and the like are carried out on original data by compiling an analysis strategy, so that the rapid analysis of a newly added data format and the data standardization facing a multi-application system are realized; in the aspect of transmission, a multi-level connection and multi-path distribution technology is adopted, so that elastic combination, cascade deployment and multi-path distribution among acquisition systems are realized, and the requirements of longitudinal and transverse expansion of a network environment and massive acquisition of data information are met.
A large-scale distributed network security data acquisition method comprises the following steps:
step one, multi-mode data acquisition.
And step two, analyzing and standardizing the data.
And step three, distributing and transmitting data.
The method is characterized in that:
step one the multimode data acquisition adopts the following 4 modes:
the first method is as follows: active mode. A method of deploying an acquisition agent at a data production site acquires specified data. The collection agent working mechanism is as follows: monitoring files in an appointed directory, performing incremental reading on the files updated in the directory according to a configurable time interval, and updating and maintaining the latest reading position of the files to avoid repeated reading of data in the files. For the original data stored in the form of a database, the original event is acquired through universal protocols such as ODBC/JDBC and the like, and an acquisition agent does not need to be deployed in a data production place.
The second method comprises the following steps: a passive mode. After the original data are generated, the original data are sent to a designated data receiver in a Syslog mode, a Snmp mode, a WebService mode and the like. For a data acquisition system, only passive reception of data is required.
The third method comprises the following steps: data sent from the upper-level acquisition system (the cascade of the acquisition systems).
The method is as follows: mirror mode. Any network access stream transmitted from the network is received through a mirror port of the network switching device.
And the data analysis in the step two adopts a mode based on a strategy document to analyze the log, so as to realize the support of the newly added equipment. And aiming at each specific device or system, typical examples of the types of the logs with important attention are collected to make a resolution strategy. When the log format of the newly added equipment or the original equipment in the information system is changed, the log format of the newly added equipment or the changed log format of the original equipment is collected to make or modify an analysis strategy.
The data distribution and transmission in the third step further comprises the following steps:
(1) setting a distribution strategy: two modes of data replication and data routing are provided.
(2) Establishing a data cache unit, distributing various data: and a data cache unit is established for each type of upper-layer application, and the standardized data is cached, so that the data is prevented from being lost due to the fact that the data is generated too fast and the receiving rate of an upper-layer application system is too slow.
(3) Data transmission: the function of transmitting data by multiple targets such as a relational database RDB, a distributed storage system HDFS/HBASE, a memory database Redis, a subordinate data acquisition system and the like is provided, and the requirements of different service application systems are met.
The analysis strategies in the second step are different according to different forms of data sources, and mainly include two types: one is to store or transmit the primitive journal in the form of the data flow in the form of the file, the tactics file is a series of XML documents that analyze the expression of the positive value to form; the other is that aiming at the original log stored in the form of a database, the strategy document is an XML document formed by a series of SQL statements and an analysis positive value expression;
the analysis strategy described in the step two has two traditional analysis functions, and provides the following three functions:
(1) and replacing the special symbols or fields of the original log according to the dictionary table. The special symbols and fields are analyzed in a contrast mode by allowing a dictionary table which is proprietary to the manufacturer to be placed in an analysis strategy file.
(2) And the original log is supplemented and perfected according to a preset value. Parsing the policy file allows additional information to be added to the original log by way of source IP address association.
(3) And (4) multi-standard data reconstruction. The data standards with different formats are allowed to be described in the analysis strategy file, and different standardized data are constructed for different upper-layer application systems.
A system for realizing the large-scale distributed network security data acquisition method is characterized by comprising the following steps: the system comprises an acquisition agent module, a data acquisition module, a data analysis module and a data distribution and transmission module. Wherein,
and the acquisition agent module is deployed in a data production place and acquires and transmits the specified data.
And the data acquisition module is used for acquiring original data from various data sources scattered in the network and forwarding the original data to the data analysis module.
And the data analysis module is used for analyzing and standardizing the original data according to a preset analysis strategy after the original data are acquired from the data acquisition module, and distributing the standardized data to the distribution transmission module in combination with the distribution strategy.
And the data distribution transmission module is used for performing grouping cache on the standardized data according to the distribution strategy and performing data transmission to a plurality of different transmission targets according to the forwarding strategy.
The data acquisition system adopts a system architecture of cascade deployment and multi-path distribution under a multi-stage complex network environment.
The acquisition agent module is deployed in a data production place to acquire specified data and comprises the following steps:
and the directory monitoring unit is responsible for monitoring the file change condition under the specified directory in real time and reporting the change condition to the reading module. The method mainly comprises the steps of monitoring the newly added file and monitoring the newly added content of the file.
And the configuration file unit is used for recording parameters such as a source file storage path, a source file reading time interval, a sending target address, a target port and the like.
And the configuration loading unit loads the configuration file and provides service for other units.
And the reading unit reads the newly added data of the source file in the specified directory according to the file change condition reported by the directory monitoring unit, and packages and sends the newly added data to the sending unit. The reading unit is responsible for identifying and maintaining the latest reading position.
And the sending unit is used for sending the source data according to the specified target address and the specified port.
The data acquisition module mainly comprises the following 7 processing units:
and the proprietary protocol monitoring unit is used for monitoring data streams sent by the acquisition agent or other acquisition systems.
And the Syslog monitoring unit is used for starting the Syslog service and receiving the Syslog data stream.
And the Snmp monitoring unit is used for starting the Snmp service and receiving the Snmp data stream.
And the WebService calling unit acquires the original data by calling a WebService interface provided by the data source.
And the network flow monitoring unit is used for capturing the data stream forwarded by the image port of the switching equipment.
And the database middleware unit is used for establishing connection with the specified database and acquiring data information from the formulated table.
And the configuration file unit is used for recording basic information required by the normal work of each monitoring unit.
The data analysis module comprises:
the analysis strategy document unit is responsible for recording analysis related strategies and comprises the following steps: positive expression description in a source log format, a key field extraction strategy, a special character or field conversion table, standardized format description and a data completion strategy;
and the analysis engine unit is used for analyzing and standardizing the original event according to the analysis strategy document.
And the data encapsulation and adaptation unit serves the distribution control unit and is responsible for encapsulating and adapting the standardized data to form event data. Each event consists of two parts: head and body.
And the distribution control unit distributes the event based on the header information in the event, and supports two data distribution modes of data replication and data routing.
The data distribution transmission module comprises:
and the buffer unit is used for grouping and buffering the events sent from the data analysis module. The data analysis module determines which events are sent to which buffer unit, the sending frequency and the number of sending each time. The number of cache units may be dynamically increased or decreased depending on the distribution policy. Because the cache unit plays a cache function, in order to ensure the rapid transmission of events, the invention adopts a data structure based on a memory type as a storage container of the cache unit, so that the cache unit does not provide persistent data storage, and the data in the cache unit can be lost once power is cut off and equipment fails.
And the transmission unit extracts the events from the cache unit according to the configuration information and transmits the events to the appointed next hop or the final target. And when the transmission is completed, deleting the events from the buffer unit. The plurality of transmission units constitute a transmission unit group. The transmission unit supports various transmission targets, mainly including: a next-stage data acquisition system; distributed storage systems, such as HDFS, HBASE, etc.; a relational database RDB; and the memory type database Redis supports real-time display.
And the transmission control unit is responsible for activating one transmission unit from a designated transmission unit group and controlling the load balance or fault recovery of the transmission unit. The transmission unit group can realize load balance through all transmission units in the group; or may be transferred to another transmission unit when one transmission unit fails. The supported load balancing algorithm comprises a random algorithm, a round-robin algorithm or a self-defined selection algorithm and the like.
And the triggering unit is used for monitoring and managing the running condition of the transmission unit in real time and triggering the transmission control unit to carry out load balancing or fault transfer on the transmission unit.
Compared with the prior art, the invention has the following advantages:
(1) the invention adopts multi-mode data acquisition, and the provided data acquisition mode can cover the current mainstream data acquisition mode and meet the requirement of comprehensive acquisition of various data sources.
(2) The invention adopts a comprehensive data analysis mode. On the basis of the original analysis of special field extraction, format conversion and the like, the analysis functions of special symbols or fields such as replacement according to a dictionary table, field completion, multi-standard-oriented data reconstruction and the like are added, and the requirement of constructing different standardized data for different upper-layer application systems is met.
(3) The system architecture adopting cascade deployment and multi-path distribution mainly has the following advantages:
under a large-scale complex network environment, data sources are more and scattered, and a plurality of sets of acquisition systems are often deployed to meet the data acquisition requirement. The acquisition system adopts cascade deployment to reduce the management difficulty of data acquisition.
In a large network environment, many business systems and management systems have gradually tended to use MapReduce, hdfs, and other distributed big data computing and storing technologies that are adept at handling large files or large data streams, rather than numerous small files or small data streams. The acquisition system can just collect the scattered small data streams or small files and then submit the small data streams or small files to the distributed computing and storing system, so that the distributed computing and storing system is in line with the application scene of big data and is beneficial to improving the processing efficiency.
At present, data acquisition of various application systems presents a chimney structure, own data acquisition tools are respectively deployed, mutual data cannot be shared, resources are wasted, and management difficulty is increased. By adopting a multi-path distribution technology, each system can share one set of data acquisition technology, and data distribution is carried out according to the requirements of each service system after data are comprehensively acquired. Can better adapt to business application and safety management requirements.
Drawings
FIG. 1 is a schematic diagram of data acquisition according to an embodiment of the present invention;
FIG. 2 is a block diagram of a data acquisition system according to the present invention;
FIG. 3 is a diagram of a collection agent module according to an embodiment of the present invention;
fig. 4 is a schematic diagram of multi-system cascade deployment according to an embodiment of the present invention.
Detailed Description
The invention is further illustrated with reference to the figures and the specific embodiments.
A large-scale distributed network security data acquisition method comprises the following steps:
step one, multi-mode data acquisition.
Considering that the network security data sources have various forms and wide distribution, the invention realizes the comprehensive acquisition of various data sources in the aspect of data acquisition through 4 modes, and a data acquisition schematic diagram is shown in fig. 1 and comprises the following 4 modes:
the first method is as follows: active mode. The original data is stored in a data production place in the form of files, databases and the like, and the data cannot be actively sent to third-party equipment or a system, and unsafe services such as file sharing, super permission, FTP and the like cannot be started in the data production place. Aiming at the application scene, for the original data stored in the form of files, the invention adopts a method of deploying collection agents in a data production place to collect the specified data. The collection agent working mechanism is as follows: monitoring files in an appointed directory, performing incremental reading on the files updated in the directory according to a configurable time interval, and updating and maintaining the latest reading position of the files to avoid repeated reading of data in the files. Aiming at the original data stored in the form of a database, the invention acquires the original event through universal protocols such as ODBC/JDBC and the like without deploying an acquisition agent in a data production place.
The second method comprises the following steps: a passive mode. After the original data are generated, the original data are sent to a designated data receiver in a Syslog mode, a Snmp mode, a WebService mode and the like. For a data acquisition system, only passive reception of data is required.
The third method comprises the following steps: data sent from the upper-level acquisition system (the cascade of the acquisition systems).
The method is as follows: mirror mode. Any network access stream transmitted from the network is received through a mirror port of the network switching device.
And step two, analyzing and standardizing the data.
The safety management operation and maintenance system needs to manage a large number of heterogeneous safety devices, and various log information needs to be collected from the devices. In order to perform secondary processing such as unified retrieval query, cross correlation, analysis statistics and the like on various types of log information, format unification of various types of logs is often required in advance. However, a unified log format standard is not established in the industry at present, the log formats of different manufacturers are different, even the log formats of different products of the same manufacturer or different versions of the same product are different, and product logs of some manufacturers contain numerous non-universal numbers and symbols, which need to be converted through a special dictionary table of the manufacturer to become information recognizable to an administrator. In order to solve the problems, the traditional solution is to customize and develop an analysis algorithm for each log format, and the method is time-consuming and labor-consuming and cannot meet the requirement of safety management on rapid acquisition of log data of newly-added equipment.
The invention analyzes the log by adopting a mode based on the strategy document, overcomes the defect of the traditional method that customization and development are needed, and simply and quickly realizes the support of the newly added equipment.
Through research, although the log formats generated by manufacturers and devices are different, the types and formats of logs generated by each specific device or system are relatively fixed, that is, each device basically comprises logs in several fixed formats, such as a system log, a management log, a security log, a service log, and the like. Therefore, for each specific device or system, only typical examples of the types of the logs with important attention need to be collected, and the resolution strategy can be formulated. When the log format of the newly added equipment or the original equipment in the information system is changed, the analysis strategy can be formulated or modified only by collecting the log format of the newly added equipment or the log format changed by the original equipment.
The analysis strategies are different according to different forms of data sources, and there are two main strategies: one is to store or transmit the primitive journal in the form of the data flow in the form of the file, the tactics file is a series of XML documents that analyze the expression of the positive value to form; the other is that aiming at the original log stored in the form of a database, the strategy document is an XML document formed by a series of SQL statements and an analysis positive value expression; conventional parsing strategies provide only two aspects of parsing functionality:
(1) the specified fields are extracted from the original log.
(2) And carrying out format normalization on the original field which does not conform to the unified format.
On the basis of providing the two analysis functions, the invention also provides the following three functions:
(1) and replacing the special symbols or fields of the original log according to the dictionary table. The special symbols and fields are analyzed in a contrast mode by allowing a dictionary table which is proprietary to the manufacturer to be placed in an analysis strategy file.
(2) And the original log is supplemented and perfected according to a preset value. Basic management information such as the name, code, type, and department of the device is not reflected in the original log information generated by the device, and the basic management information is usually required to be known through the subsequent processing such as alarm and response generated by analyzing the original log. The parsing policy file allows additional information to be added to the original log in a source IP address association manner.
(3) And (4) multi-standard data reconstruction. The data acquisition method of the invention allows for providing data acquisition functions for a plurality of upper-level application systems. Different application systems have different data standard formats, so that the data standards with different formats are allowed to be described in the analysis policy file, and different standardized data are constructed for different upper-layer application systems.
And step three, distributing and transmitting data.
At present, data acquisition of various application systems presents a chimney structure, own data acquisition tools are respectively deployed, mutual data cannot be shared, resources are wasted, and management difficulty is increased. Therefore, the data acquisition method designs a data distribution transmission mode. The data distribution transmission comprises the following steps:
(1) and setting a distribution strategy. Two modes of distribution are provided:
data replication: a part of data with standardized format can be shared by a plurality of upper-layer application systems at the same time, and aiming at the data, a part of data copy is provided for respective upper-layer application and is sent to a specified data cache unit;
data routing: the original event of a certain type is only used by an application system at the upper layer, and only a routing function is provided for the data, and the data is routed to a specified data cache unit.
(2) And establishing a data cache unit and distributing various types of data. Various application systems considering data generation rate of data source and upper layer
The data cache function is designed in the data acquisition method by integrating the mismatching factors of the receiving data rate, and the data cache function is used for each type of upper layer
The standardized data is cached by establishing a data cache unit, so that the situation that the data is generated too fast and an upper application system is connected is prevented
Too slow a rate of reception results in data loss.
(3) And (5) data transmission. The function of transmitting data by multiple targets such as a relational database RDB, a distributed storage system HDFS/HBASE, a memory database Redis, a subordinate data acquisition system and the like is provided, and the requirements of different service application systems are met.
A system for implementing the large-scale distributed network security data acquisition method, as shown in fig. 2, includes: the system comprises an acquisition agent module, a data acquisition module, a data analysis module and a data distribution and transmission module. Wherein,
the collection agent module: aiming at the application scenes that original data are stored in a data production place in a file form, and do not support actively sending the data to third-party equipment or a system, and do not support opening unsafe services such as file sharing, super permission and FTP in the data production place, the invention adopts the mode of deploying a collection agent in the data production place to collect specified data. The collection agent module is composed of a block diagram as shown in fig. 3, including: the device comprises a directory monitoring unit, a configuration file, a configuration loading unit, a reading unit and a sending unit.
And the data acquisition module is responsible for acquiring original data from various data sources scattered in the network and forwarding the original data to the data analysis module. The data acquisition module includes: the system comprises a proprietary protocol monitoring unit, a Syslog monitoring unit, a Snmp monitoring unit, a WebService calling unit, database middleware and a configuration file.
And the data analysis module is used for analyzing and standardizing the original data according to a preset analysis strategy after the original data are acquired from the data acquisition module, and distributing the standardized data to the distribution transmission module in combination with the distribution strategy. The data analysis module comprises:
(1) and analyzing the strategy document. And the system is responsible for recording and analyzing related strategies.
(2) And the analysis engine unit is used for analyzing and standardizing the original event according to the analysis strategy document.
The working process of the analysis engine unit mainly comprises the following steps: firstly, matching original data; secondly, extracting key fields; thirdly, replacing special fields; and fourthly, completing the specified fields.
(3) And the data encapsulation and adaptation unit serves the distribution control unit and is responsible for encapsulating and adapting the standardized data to form event data. Each event consists of two parts: head and body. And the subpackaging function constructs the body according to the standard format appointed in the analysis strategy document. The body carries the standard data which is packaged according to the target requirement. The adaptation function completes the header according to a preset distribution strategy. The header is mainly composed of a built-in field timestamp and a custom field, such as: ID. Grouping flag bits, etc. Through the adaptation function, information such as host names, static identifiers and the like can be added in the header, and distribution bases are provided for the distribution control unit.
(4) And the distribution control unit is used for distributing the event based on the header information in the event. And two data distribution modes of data replication and data routing are supported.
And the data distribution transmission module is used for performing grouping cache on the standardized data according to the distribution strategy and performing data transmission to a plurality of different transmission targets according to the forwarding strategy. The module comprises: the device comprises a buffer unit, a transmission control unit and a trigger unit.
The data acquisition system of the invention adopts a system architecture of cascade deployment and multi-path distribution under a multi-stage complex network environment, and a schematic diagram of a cascade deployment mode of a plurality of acquisition systems is shown in fig. 4. The acquisition system-1, the acquisition system-2 and the acquisition system-3 respectively receive various data sources deployed in a large-scale network system, analyze, normalize and standardize the data sources, and gather the dispersed data to form a large data stream. The acquisition system-4 distributes the summarized data stream according to a pre-configured strategy, for example, the data from the data source-1 and the data source-2 are distributed to the first service application; distributing data from a data source-1, a data source-2, a data source-3 and a data source-4 to a service application II; and summarizing all data sources, distributing the data sources to a third service application, copying the data sources at the same time, and transmitting the copied data sources to a next-level acquisition system. The distributed targets support various types, including a memory type database redis for real-time display, a relational database based on structured storage, a distributed storage system for big data storage, and the like.
Claims (9)
1. A large-scale distributed network security data acquisition method comprises the following steps:
step one, multi-mode data acquisition;
step two, data analysis and standardization;
step three, data distribution and transmission;
the method is characterized in that:
step one the multimode data acquisition adopts the following 4 modes:
the first method is as follows: an active mode; a method for deploying an acquisition agent in a data production place acquires specified data; the collection agent working mechanism is as follows: monitoring files under an appointed directory, performing incremental reading on the files updated under the directory according to a configurable time interval, and updating and maintaining the latest reading position of the files to avoid repeated reading of data in the files; aiming at original data stored in a database form, obtaining an original event through an ODBC/JDBC universal protocol without deploying an acquisition agent in a data production place;
the second method comprises the following steps: a passive mode; after the original data are generated, the original data are sent to a specified data receiver in a Syslog, Snmp and WebService mode; for a data acquisition system, only passive reception of data is required;
the third method comprises the following steps: data sent by a previous-stage acquisition system;
the method is as follows: a mirror mode; receiving any network access stream transmitted from the network through a mirror port of the network switching device;
secondly, the data analysis adopts a mode based on a strategy document to analyze the log, so as to realize the support of the newly added equipment; aiming at each specific device or system, collecting typical samples of the log types with important attention to make an analysis strategy; when the log format of the newly added equipment or the original equipment in the information system is changed, collecting the log format of the newly added equipment or the log format changed by the original equipment to make or modify an analysis strategy;
step three, the data distribution transmission further comprises the following steps:
(1) setting a distribution strategy: providing two distribution modes of data replication and data routing;
(2) establishing a data cache unit, distributing various data: establishing a data cache unit for each type of upper-layer application, caching the standardized data, and preventing data from being lost due to the fact that the data is generated too fast and the receiving rate of an upper-layer application system is too slow;
(3) data transmission: the function of target data transmission for a relational database RDB, a distributed storage system HDFS/HBASE, a memory database Redis and a subordinate data acquisition system is provided, and the requirements of different service application systems are met.
2. The large-scale distributed network security data acquisition method according to claim 1, wherein the parsing strategies are different according to different forms of data sources, and there are two types: one is to store or transmit the primitive journal in the form of the data flow in the form of the file, the tactics file is a series of XML documents that analyze the expression of the positive value to form; the other is that aiming at the original log stored in the form of a database, the strategy document is an XML document formed by a series of SQL statements and an analysis positive value expression.
3. The large-scale distributed network security data collection method according to claim 1 or 2, wherein the parsing strategy has two conventional parsing functions, and provides the following three functions:
(1) replacing special symbols or fields of the original log according to the dictionary table; allowing a dictionary table specially owned by a manufacturer to be put into an analysis strategy file, and carrying out comparison analysis on special symbols and fields;
(2) the original log is supplemented and perfected according to a preset value; analyzing the strategy file to allow additional information to be added to the original log in a source IP address association mode;
(3) reconstructing multi-standard data; the data standards with different formats are allowed to be described in the analysis strategy file, and different standardized data are constructed for different upper-layer application systems.
4. A large-scale distributed network security data acquisition system is characterized by comprising: the system comprises an acquisition agent module, a data acquisition module, a data analysis module and a data distribution and transmission module; wherein,
the acquisition agent module is deployed in a data production place and acquires and transmits specified data;
the data acquisition module is used for acquiring original data from various data sources scattered in a network and forwarding the original data to the data analysis module;
the data analysis module is used for analyzing and standardizing the original data according to a preset analysis strategy after the original data are acquired from the data acquisition module, and distributing the standardized data to the distribution transmission module in combination with the distribution strategy;
and the data distribution transmission module is used for performing grouping cache on the standardized data according to the distribution strategy and performing data transmission to a plurality of different transmission targets according to the forwarding strategy.
5. The system for securely data collection in a large-scale distributed network according to claim 4, wherein the data collection system employs a cascade deployment and multi-path distribution architecture in a multi-stage complex network environment.
6. The large scale distributed network security data acquisition system of claim 4, wherein the acquisition agent module comprises:
the directory monitoring unit is responsible for monitoring the file change condition under the specified directory in real time and reporting the change condition to the reading module; the method mainly comprises the steps of monitoring a newly added file and monitoring newly added content of the file;
the configuration file unit is responsible for recording a source file storage path, a source file reading time interval, a sending target address and target port parameters;
the configuration loading unit loads the configuration file and provides service for other units;
the reading unit reads the newly added data of the source file in the appointed directory according to the file change condition reported by the directory monitoring unit, and packages and sends the newly added data to the sending unit; the reading unit is responsible for identifying and maintaining the latest reading position;
and the sending unit is used for sending the source data according to the specified target address and the specified port.
7. The large scale distributed network security data acquisition system of claim 4, wherein the data acquisition module comprises:
the special protocol monitoring unit is used for monitoring data streams sent by an acquisition agent or other acquisition systems;
the system comprises a Syslog monitoring unit, a Syslog data processing unit and a Syslog data processing unit, wherein the Syslog monitoring unit is used for starting a Syslog service and receiving a Syslog data stream;
the Snmp monitoring unit is used for starting the Snmp service and receiving the Snmp data stream;
the WebService calling unit acquires original data by calling a WebService interface provided by the data source;
the network flow monitoring unit is used for capturing the data stream forwarded by the mirror image port of the switching equipment;
the database middleware unit is used for establishing connection with the specified database and acquiring data information from the formulated table;
and the configuration file unit is used for recording basic information required by the normal work of each monitoring unit.
8. The large scale distributed network security data collection system of claim 4, wherein the data parsing module comprises:
the analysis strategy document unit is responsible for recording analysis related strategies and comprises the following steps: positive expression description in a source log format, a key field extraction strategy, a special character or field conversion table, standardized format description and a data completion strategy;
the analysis engine unit analyzes and standardizes the original event according to the analysis strategy document;
the data encapsulation and adaptation unit serves the distribution control unit and is responsible for encapsulating and adapting the standardized data to form event data; each event consists of a header part and a body part;
and the distribution control unit distributes the event based on the header information in the event, and supports two data distribution modes of data replication and data routing.
9. The large-scale distributed network security data acquisition system according to claim 4, wherein the data distribution transmission module comprises:
the buffer unit is used for grouping and buffering events sent from the data analysis module; the data analysis module determines which events are sent to which cache unit, sending frequency and the number of sending each time; the number of the cache units is dynamically increased or decreased according to a distribution strategy; a data structure based on a memory type is adopted as a storage container of a cache unit, the cache unit does not provide persistent data storage, and once power failure and equipment failure occur, data in the cache unit is lost;
the transmission unit extracts the events from the cache unit according to the configuration information and transmits the events to a specified next hop or a final target; deleting the events from the buffer unit after the transmission is finished; the plurality of transmission units form a transmission unit group; the transmission unit supports a plurality of transmission targets;
the transmission control unit is responsible for activating one transmission unit from a designated transmission unit group and controlling the load balance or fault recovery of the transmission unit; the transmission unit group can realize load balance through all transmission units in the group; or may be transferred to another transmission unit when one transmission unit fails; the supported load balancing algorithm comprises a random algorithm, a round training algorithm or a self-defined selection algorithm;
and the triggering unit is used for monitoring and managing the running condition of the transmission unit in real time and triggering the transmission control unit to carry out load balancing or fault transfer on the transmission unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310572103.0A CN103731298A (en) | 2013-11-15 | 2013-11-15 | Large-scale distributed network safety data acquisition method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310572103.0A CN103731298A (en) | 2013-11-15 | 2013-11-15 | Large-scale distributed network safety data acquisition method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103731298A true CN103731298A (en) | 2014-04-16 |
Family
ID=50455233
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310572103.0A Pending CN103731298A (en) | 2013-11-15 | 2013-11-15 | Large-scale distributed network safety data acquisition method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103731298A (en) |
Cited By (65)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104486157A (en) * | 2014-12-16 | 2015-04-01 | 国家电网公司 | Information system performance detecting method based on deep packet analysis |
| CN104599042A (en) * | 2014-12-26 | 2015-05-06 | 国家电网公司 | Agreement-based data management and service system management method |
| CN105007278A (en) * | 2015-07-31 | 2015-10-28 | 中国电建集团成都勘测设计研究院有限公司 | Automatic real-time acquisition system and acquisition method for network safety log |
| CN105207852A (en) * | 2015-10-09 | 2015-12-30 | 西安未来国际信息股份有限公司 | Method for directionally acquiring network data based on distributed mode |
| CN105391584A (en) * | 2015-11-30 | 2016-03-09 | 用友网络科技股份有限公司 | Abnormity early warning system for use in distributed environment |
| CN105447146A (en) * | 2015-11-26 | 2016-03-30 | 陕西艾特信息化工程咨询有限责任公司 | Massive data collecting and exchanging system and method |
| CN105959151A (en) * | 2016-06-22 | 2016-09-21 | 中国工商银行股份有限公司 | High availability stream processing system and method |
| CN106100999A (en) * | 2016-08-28 | 2016-11-09 | 北京瑞和云图科技有限公司 | Image network flow control protocol in a kind of virtualized network environment |
| CN106209420A (en) * | 2016-06-27 | 2016-12-07 | 瑞斯康达科技发展股份有限公司 | A kind of method positioning data forwarding service fault and electronic equipment |
| CN106294644A (en) * | 2016-08-02 | 2017-01-04 | 山东鲁能软件技术有限公司 | A kind of magnanimity time series data collection and treatment device based on big data technique and method |
| WO2017008598A1 (en) * | 2015-07-10 | 2017-01-19 | 中兴通讯股份有限公司 | Big data exchange method and device |
| CN106534257A (en) * | 2016-09-29 | 2017-03-22 | 国家电网公司 | Multi-level cluster-type construction multi-source safety log collection system and method |
| CN106817693A (en) * | 2015-11-27 | 2017-06-09 | 国网智能电网研究院 | A kind of distributed network security control system and method |
| CN106897159A (en) * | 2017-01-20 | 2017-06-27 | 武汉华信联创技术工程有限公司 | A kind of system and method for gathering Data of Automatic Weather |
| CN107169854A (en) * | 2016-03-07 | 2017-09-15 | 阿里巴巴集团控股有限公司 | A kind of method and device of data processing |
| CN107247721A (en) * | 2017-04-24 | 2017-10-13 | 江苏曙光信息技术有限公司 | Visualize collecting method |
| CN107317838A (en) * | 2017-05-24 | 2017-11-03 | 重庆邮电大学 | A kind of astronomical metadata archiving method and system based on stream data processing framework |
| CN107453946A (en) * | 2017-07-20 | 2017-12-08 | 阿里巴巴集团控股有限公司 | Field management method and device and electronic equipment |
| CN107463610A (en) * | 2017-06-27 | 2017-12-12 | 北京小度信息科技有限公司 | A kind of data storage method and device |
| CN107679544A (en) * | 2017-08-04 | 2018-02-09 | 平安科技（深圳）有限公司 | Automatic data matching method, electronic equipment and computer-readable recording medium |
| CN107864056A (en) * | 2017-11-04 | 2018-03-30 | 公安部第三研究所 | A kind of distributed event acquisition probe, distributed event high speed acquisition system and method |
| WO2018072158A1 (en) * | 2016-10-19 | 2018-04-26 | 达闼科技（北京）有限公司 | Method, device and system for remote control, and cloud-based intelligent robot |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN108073620A (en) * | 2016-11-14 | 2018-05-25 | 北京航天长峰科技工业集团有限公司 | A kind of method for quickly retrieving based on graph data structure |
| CN108133017A (en) * | 2017-12-21 | 2018-06-08 | 广州市申迪计算机系统有限公司 | A kind of multi-data source acquisition configuration method and device |
| CN108173674A (en) * | 2017-12-11 | 2018-06-15 | 西安优卓软件有限公司 | A kind of network-based service data acquisition delivery system and method |
| CN108241528A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of User Defined mass network secure data dynamic collecting method |
| CN108429755A (en) * | 2018-03-21 | 2018-08-21 | 深圳天源迪科信息技术股份有限公司 | Basic network security information dynamic management platform and method |
| CN108540513A (en) * | 2017-03-03 | 2018-09-14 | 中国移动通信集团福建有限公司 | Ask the determination method and device of Replay Attack |
| CN108614820A (en) * | 2016-12-09 | 2018-10-02 | 腾讯科技（深圳）有限公司 | The method and apparatus for realizing the parsing of streaming source data |
| CN108717391A (en) * | 2018-05-16 | 2018-10-30 | 平安科技（深圳）有限公司 | Monitoring device, method and the computer readable storage medium of test process |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of big data log intelligent analysis system and method |
| CN109005083A (en) * | 2018-07-17 | 2018-12-14 | 千寻位置网络有限公司 | The method and system of large scale collection base station data |
| CN109086195A (en) * | 2018-08-02 | 2018-12-25 | 四川长虹电器股份有限公司 | Log statistic and analysis system and method based on log versatility regulation engine |
| CN109271349A (en) * | 2018-09-29 | 2019-01-25 | 四川长虹电器股份有限公司 | A kind of rules process method based on log versatility regulation engine |
| CN109660620A (en) * | 2018-12-20 | 2019-04-19 | 北京树根互联科技有限公司 | Data distribution frame |
| CN109684291A (en) * | 2018-12-21 | 2019-04-26 | 北京奇安信科技有限公司 | A kind of data collector file method, system, electronic equipment and medium |
| CN109753502A (en) * | 2018-12-29 | 2019-05-14 | 山东浪潮商用系统有限公司 | A kind of collecting method based on NiFi |
| CN110430158A (en) * | 2019-06-13 | 2019-11-08 | 中国科学院信息工程研究所 | Collection agent dispositions method and device |
| CN110944025A (en) * | 2020-01-16 | 2020-03-31 | 四川天翼网络服务有限公司 | Multi-protocol video data acquisition system and implementation method thereof |
| CN110995538A (en) * | 2019-12-03 | 2020-04-10 | 北京博睿宏远数据科技股份有限公司 | Network data collection method, device, system, equipment and storage medium |
| CN111061807A (en) * | 2019-11-23 | 2020-04-24 | 方正株式(武汉)科技开发有限公司 | Distributed data acquisition and analysis system and method, server and medium |
| CN111080840A (en) * | 2019-12-04 | 2020-04-28 | 中国直升机设计研究所 | Helicopter flight control system data sending and reproducing method |
| CN111104397A (en) * | 2019-11-19 | 2020-05-05 | 浙江工业大学 | Flume-based configurable data integration method |
| CN111581170A (en) * | 2020-04-17 | 2020-08-25 | 上海中通吉网络技术有限公司 | Distributed intelligent data acquisition method, device, equipment and storage medium |
| CN111967850A (en) * | 2020-08-19 | 2020-11-20 | 支付宝(杭州)信息技术有限公司 | Data reporting monitoring method and device and electronic equipment |
| CN112307064A (en) * | 2020-10-29 | 2021-02-02 | 上海达梦数据库有限公司 | Data management system, method and storage medium |
| CN112579675A (en) * | 2019-09-29 | 2021-03-30 | 西门子（中国）有限公司 | Data processing method and device |
| CN112749065A (en) * | 2021-01-22 | 2021-05-04 | 大连高德瑞信科技有限公司 | Application system performance data acquisition method |
| CN112783728A (en) * | 2021-01-28 | 2021-05-11 | 杉德银卡通信息服务有限公司 | Data automation processing method and system |
| CN112925689A (en) * | 2021-01-22 | 2021-06-08 | 复旦大学 | Multi-channel monitoring data transmission optimization method |
| CN113127413A (en) * | 2021-05-12 | 2021-07-16 | 北京红山信息科技研究院有限公司 | Operator data processing method, device, server and storage medium |
| CN113485894A (en) * | 2021-07-14 | 2021-10-08 | 深信服科技股份有限公司 | Data acquisition method, device and equipment and readable storage medium |
| CN113553093A (en) * | 2020-04-24 | 2021-10-26 | 上海颢联数字科技有限公司 | Method and system for parallel acquisition and integration of multi-source transaction data |
| CN113572780A (en) * | 2021-07-28 | 2021-10-29 | 中国南方电网有限责任公司 | Equipment security policy configuration method |
| CN114546519A (en) * | 2022-01-26 | 2022-05-27 | 华北电力大学 | An industrial control safety data acquisition system and method |
| CN114760150A (en) * | 2022-06-13 | 2022-07-15 | 交通运输通信信息集团有限公司 | Network security protection method and system based on big data |
| CN115186016A (en) * | 2021-04-01 | 2022-10-14 | 中核武汉核电运行技术股份有限公司 | Data integration method based on nuclear power industry Internet platform |
| CN115269694A (en) * | 2022-05-20 | 2022-11-01 | 南京迪塔维数据技术有限公司 | A kind of data unified real-time acquisition device and acquisition method |
| CN115480998A (en) * | 2021-06-16 | 2022-12-16 | 深圳富桂精密工业有限公司 | Log parsing system and log parsing method |
| WO2022261810A1 (en) * | 2021-06-15 | 2022-12-22 | 中国科学技术大学 | Method and apparatus for constructing data acquisition system |
| CN115658637A (en) * | 2022-12-26 | 2023-01-31 | 北京六方云信息技术有限公司 | Log normalization processing method and device, storage medium and processor |
| CN116860861A (en) * | 2023-09-05 | 2023-10-10 | 杭州瞬安信息科技有限公司 | ETL data management system |
| CN117520597A (en) * | 2023-09-11 | 2024-02-06 | 北京国卫星通科技有限公司 | Data record implementation method of inertial navigation data acquisition and analysis system |
| CN117609175A (en) * | 2024-01-24 | 2024-02-27 | 锱云(上海)物联网科技有限公司 | Configurable industrial control file acquisition and analysis method and system |
-
2013
- 2013-11-15 CN CN201310572103.0A patent/CN103731298A/en active Pending
Cited By (96)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104486157A (en) * | 2014-12-16 | 2015-04-01 | 国家电网公司 | Information system performance detecting method based on deep packet analysis |
| CN104599042A (en) * | 2014-12-26 | 2015-05-06 | 国家电网公司 | Agreement-based data management and service system management method |
| WO2017008598A1 (en) * | 2015-07-10 | 2017-01-19 | 中兴通讯股份有限公司 | Big data exchange method and device |
| US10706062B2 (en) | 2015-07-10 | 2020-07-07 | Xi'an Zhongxing New Software Co., Ltd. | Method and system for exchanging data from a big data source to a big data target corresponding to components of the big data source |
| CN105007278A (en) * | 2015-07-31 | 2015-10-28 | 中国电建集团成都勘测设计研究院有限公司 | Automatic real-time acquisition system and acquisition method for network safety log |
| CN105207852A (en) * | 2015-10-09 | 2015-12-30 | 西安未来国际信息股份有限公司 | Method for directionally acquiring network data based on distributed mode |
| CN105447146A (en) * | 2015-11-26 | 2016-03-30 | 陕西艾特信息化工程咨询有限责任公司 | Massive data collecting and exchanging system and method |
| CN106817693B (en) * | 2015-11-27 | 2020-10-27 | 国网智能电网研究院 | A distributed network security control system and method |
| CN106817693A (en) * | 2015-11-27 | 2017-06-09 | 国网智能电网研究院 | A kind of distributed network security control system and method |
| CN105391584A (en) * | 2015-11-30 | 2016-03-09 | 用友网络科技股份有限公司 | Abnormity early warning system for use in distributed environment |
| CN107169854B (en) * | 2016-03-07 | 2021-04-02 | 创新先进技术有限公司 | Data processing method and device |
| CN107169854A (en) * | 2016-03-07 | 2017-09-15 | 阿里巴巴集团控股有限公司 | A kind of method and device of data processing |
| CN105959151A (en) * | 2016-06-22 | 2016-09-21 | 中国工商银行股份有限公司 | High availability stream processing system and method |
| CN105959151B (en) * | 2016-06-22 | 2019-05-07 | 中国工商银行股份有限公司 | A kind of Stream Processing system and method for High Availabitity |
| CN106209420B (en) * | 2016-06-27 | 2019-03-26 | 瑞斯康达科技发展股份有限公司 | A kind of method and electronic equipment of location data forwarding service failure |
| CN106209420A (en) * | 2016-06-27 | 2016-12-07 | 瑞斯康达科技发展股份有限公司 | A kind of method positioning data forwarding service fault and electronic equipment |
| CN106294644A (en) * | 2016-08-02 | 2017-01-04 | 山东鲁能软件技术有限公司 | A kind of magnanimity time series data collection and treatment device based on big data technique and method |
| CN106294644B (en) * | 2016-08-02 | 2019-06-14 | 山东鲁能软件技术有限公司 | A kind of magnanimity time series data collection and treatment device and method based on big data technology |
| CN106100999B (en) * | 2016-08-28 | 2019-05-24 | 北京瑞和云图科技有限公司 | Image network flow control methods in a kind of virtualized network environment |
| CN106100999A (en) * | 2016-08-28 | 2016-11-09 | 北京瑞和云图科技有限公司 | Image network flow control protocol in a kind of virtualized network environment |
| CN106534257B (en) * | 2016-09-29 | 2019-09-27 | 国家电网公司 | A multi-source security log collection system and method with a multi-level cluster architecture |
| CN106534257A (en) * | 2016-09-29 | 2017-03-22 | 国家电网公司 | Multi-level cluster-type construction multi-source safety log collection system and method |
| WO2018072158A1 (en) * | 2016-10-19 | 2018-04-26 | 达闼科技（北京）有限公司 | Method, device and system for remote control, and cloud-based intelligent robot |
| CN108073620A (en) * | 2016-11-14 | 2018-05-25 | 北京航天长峰科技工业集团有限公司 | A kind of method for quickly retrieving based on graph data structure |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN108076111B (en) * | 2016-11-15 | 2021-07-09 | 亿阳安全技术有限公司 | System and method for distributing data in big data platform |
| CN108614820A (en) * | 2016-12-09 | 2018-10-02 | 腾讯科技（深圳）有限公司 | The method and apparatus for realizing the parsing of streaming source data |
| CN108241528A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of User Defined mass network secure data dynamic collecting method |
| CN108241528B (en) * | 2017-01-19 | 2020-10-09 | 上海直真君智科技有限公司 | Dynamic acquisition method for mass network security data customized by user |
| CN106897159A (en) * | 2017-01-20 | 2017-06-27 | 武汉华信联创技术工程有限公司 | A kind of system and method for gathering Data of Automatic Weather |
| CN108540513B (en) * | 2017-03-03 | 2021-08-13 | 中国移动通信集团福建有限公司 | Determination method and device for request replay attack |
| CN108540513A (en) * | 2017-03-03 | 2018-09-14 | 中国移动通信集团福建有限公司 | Ask the determination method and device of Replay Attack |
| CN107247721A (en) * | 2017-04-24 | 2017-10-13 | 江苏曙光信息技术有限公司 | Visualize collecting method |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of big data log intelligent analysis system and method |
| CN107317838A (en) * | 2017-05-24 | 2017-11-03 | 重庆邮电大学 | A kind of astronomical metadata archiving method and system based on stream data processing framework |
| CN107317838B (en) * | 2017-05-24 | 2020-11-17 | 重庆邮电大学 | Astronomical metadata filing method and system based on streaming data processing architecture |
| CN107463610B (en) * | 2017-06-27 | 2021-01-26 | 北京星选科技有限公司 | Data warehousing method and device |
| CN107463610A (en) * | 2017-06-27 | 2017-12-12 | 北京小度信息科技有限公司 | A kind of data storage method and device |
| CN107453946A (en) * | 2017-07-20 | 2017-12-08 | 阿里巴巴集团控股有限公司 | Field management method and device and electronic equipment |
| CN107453946B (en) * | 2017-07-20 | 2020-07-17 | 阿里巴巴集团控股有限公司 | Field management method and device and electronic equipment |
| CN107679544A (en) * | 2017-08-04 | 2018-02-09 | 平安科技（深圳）有限公司 | Automatic data matching method, electronic equipment and computer-readable recording medium |
| CN107864056A (en) * | 2017-11-04 | 2018-03-30 | 公安部第三研究所 | A kind of distributed event acquisition probe, distributed event high speed acquisition system and method |
| CN108173674A (en) * | 2017-12-11 | 2018-06-15 | 西安优卓软件有限公司 | A kind of network-based service data acquisition delivery system and method |
| CN108133017A (en) * | 2017-12-21 | 2018-06-08 | 广州市申迪计算机系统有限公司 | A kind of multi-data source acquisition configuration method and device |
| CN108429755B (en) * | 2018-03-21 | 2021-02-05 | 深圳天源迪科信息技术股份有限公司 | Dynamic management platform and method for network security basic information |
| CN108429755A (en) * | 2018-03-21 | 2018-08-21 | 深圳天源迪科信息技术股份有限公司 | Basic network security information dynamic management platform and method |
| CN108717391A (en) * | 2018-05-16 | 2018-10-30 | 平安科技（深圳）有限公司 | Monitoring device, method and the computer readable storage medium of test process |
| CN108717391B (en) * | 2018-05-16 | 2021-09-28 | 平安科技（深圳）有限公司 | Monitoring device and method for test process and computer readable storage medium |
| CN109005083A (en) * | 2018-07-17 | 2018-12-14 | 千寻位置网络有限公司 | The method and system of large scale collection base station data |
| CN109086195A (en) * | 2018-08-02 | 2018-12-25 | 四川长虹电器股份有限公司 | Log statistic and analysis system and method based on log versatility regulation engine |
| CN109271349A (en) * | 2018-09-29 | 2019-01-25 | 四川长虹电器股份有限公司 | A kind of rules process method based on log versatility regulation engine |
| CN109660620B (en) * | 2018-12-20 | 2021-08-03 | 北京树根互联科技有限公司 | Data distribution system |
| CN109660620A (en) * | 2018-12-20 | 2019-04-19 | 北京树根互联科技有限公司 | Data distribution frame |
| CN109684291A (en) * | 2018-12-21 | 2019-04-26 | 北京奇安信科技有限公司 | A kind of data collector file method, system, electronic equipment and medium |
| CN109684291B (en) * | 2018-12-21 | 2021-05-14 | 奇安信科技集团股份有限公司 | A file data acquisition method, system, electronic device and medium |
| CN109753502A (en) * | 2018-12-29 | 2019-05-14 | 山东浪潮商用系统有限公司 | A kind of collecting method based on NiFi |
| CN109753502B (en) * | 2018-12-29 | 2023-05-12 | 浪潮软件科技有限公司 | Data acquisition method based on NiFi |
| CN110430158B (en) * | 2019-06-13 | 2020-07-03 | 中国科学院信息工程研究所 | Collection agent deployment method and device |
| CN110430158A (en) * | 2019-06-13 | 2019-11-08 | 中国科学院信息工程研究所 | Collection agent dispositions method and device |
| CN112579675A (en) * | 2019-09-29 | 2021-03-30 | 西门子（中国）有限公司 | Data processing method and device |
| CN111104397A (en) * | 2019-11-19 | 2020-05-05 | 浙江工业大学 | Flume-based configurable data integration method |
| CN111104397B (en) * | 2019-11-19 | 2021-10-15 | 浙江工业大学 | A Flume-based Configurable Data Integration Method |
| CN111061807A (en) * | 2019-11-23 | 2020-04-24 | 方正株式(武汉)科技开发有限公司 | Distributed data acquisition and analysis system and method, server and medium |
| CN110995538B (en) * | 2019-12-03 | 2022-01-07 | 北京博睿宏远数据科技股份有限公司 | Network data acquisition method, device, system, equipment and storage medium |
| CN110995538A (en) * | 2019-12-03 | 2020-04-10 | 北京博睿宏远数据科技股份有限公司 | Network data collection method, device, system, equipment and storage medium |
| CN111080840A (en) * | 2019-12-04 | 2020-04-28 | 中国直升机设计研究所 | Helicopter flight control system data sending and reproducing method |
| CN111080840B (en) * | 2019-12-04 | 2022-02-18 | 中国直升机设计研究所 | Helicopter flight control system data sending and reproducing method |
| CN110944025B (en) * | 2020-01-16 | 2022-04-15 | 四川天翼网络服务有限公司 | Multi-protocol video data acquisition system and implementation method thereof |
| CN110944025A (en) * | 2020-01-16 | 2020-03-31 | 四川天翼网络服务有限公司 | Multi-protocol video data acquisition system and implementation method thereof |
| CN111581170A (en) * | 2020-04-17 | 2020-08-25 | 上海中通吉网络技术有限公司 | Distributed intelligent data acquisition method, device, equipment and storage medium |
| CN111581170B (en) * | 2020-04-17 | 2024-04-09 | 上海中通吉网络技术有限公司 | Distributed intelligent data acquisition method, device, equipment and storage medium |
| CN113553093B (en) * | 2020-04-24 | 2023-05-02 | 上海颢联数字科技有限公司 | Method and system for parallel acquisition and integration of multi-source transaction data |
| CN113553093A (en) * | 2020-04-24 | 2021-10-26 | 上海颢联数字科技有限公司 | Method and system for parallel acquisition and integration of multi-source transaction data |
| CN111967850A (en) * | 2020-08-19 | 2020-11-20 | 支付宝(杭州)信息技术有限公司 | Data reporting monitoring method and device and electronic equipment |
| CN112307064A (en) * | 2020-10-29 | 2021-02-02 | 上海达梦数据库有限公司 | Data management system, method and storage medium |
| CN112749065A (en) * | 2021-01-22 | 2021-05-04 | 大连高德瑞信科技有限公司 | Application system performance data acquisition method |
| CN112925689A (en) * | 2021-01-22 | 2021-06-08 | 复旦大学 | Multi-channel monitoring data transmission optimization method |
| CN112783728A (en) * | 2021-01-28 | 2021-05-11 | 杉德银卡通信息服务有限公司 | Data automation processing method and system |
| CN115186016A (en) * | 2021-04-01 | 2022-10-14 | 中核武汉核电运行技术股份有限公司 | Data integration method based on nuclear power industry Internet platform |
| CN113127413A (en) * | 2021-05-12 | 2021-07-16 | 北京红山信息科技研究院有限公司 | Operator data processing method, device, server and storage medium |
| CN113127413B (en) * | 2021-05-12 | 2024-03-01 | 北京红山信息科技研究院有限公司 | Operator data processing method, device, server and storage medium |
| WO2022261810A1 (en) * | 2021-06-15 | 2022-12-22 | 中国科学技术大学 | Method and apparatus for constructing data acquisition system |
| CN115480998A (en) * | 2021-06-16 | 2022-12-16 | 深圳富桂精密工业有限公司 | Log parsing system and log parsing method |
| CN113485894A (en) * | 2021-07-14 | 2021-10-08 | 深信服科技股份有限公司 | Data acquisition method, device and equipment and readable storage medium |
| CN113572780A (en) * | 2021-07-28 | 2021-10-29 | 中国南方电网有限责任公司 | Equipment security policy configuration method |
| CN114546519A (en) * | 2022-01-26 | 2022-05-27 | 华北电力大学 | An industrial control safety data acquisition system and method |
| CN114546519B (en) * | 2022-01-26 | 2023-10-03 | 华北电力大学 | An industrial control safety data collection system and method |
| CN115269694A (en) * | 2022-05-20 | 2022-11-01 | 南京迪塔维数据技术有限公司 | A kind of data unified real-time acquisition device and acquisition method |
| CN114760150A (en) * | 2022-06-13 | 2022-07-15 | 交通运输通信信息集团有限公司 | Network security protection method and system based on big data |
| CN115658637A (en) * | 2022-12-26 | 2023-01-31 | 北京六方云信息技术有限公司 | Log normalization processing method and device, storage medium and processor |
| CN116860861A (en) * | 2023-09-05 | 2023-10-10 | 杭州瞬安信息科技有限公司 | ETL data management system |
| CN116860861B (en) * | 2023-09-05 | 2023-12-15 | 杭州瞬安信息科技有限公司 | ETL data management system |
| CN117520597A (en) * | 2023-09-11 | 2024-02-06 | 北京国卫星通科技有限公司 | Data record implementation method of inertial navigation data acquisition and analysis system |
| CN117520597B (en) * | 2023-09-11 | 2024-04-26 | 北京国卫星通科技有限公司 | Data record implementation method of inertial navigation data acquisition and analysis system |
| CN117609175A (en) * | 2024-01-24 | 2024-02-27 | 锱云(上海)物联网科技有限公司 | Configurable industrial control file acquisition and analysis method and system |
| CN117609175B (en) * | 2024-01-24 | 2024-04-05 | 锱云(上海)物联网科技有限公司 | Configurable industrial control file acquisition and analysis method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103731298A (en) | Large-scale distributed network safety data acquisition method and system | |
| US12050696B2 (en) | Agent-based vulnerability management | |
| CN109245931B (en) | Log management and monitoring alarm realization method of container cloud platform based on kubernets | |
| CN104111983B (en) | A kind of open multi-source data acquiring system and method | |
| US9009139B2 (en) | Query pipeline | |
| US10528599B1 (en) | Tiered data processing for distributed data | |
| CN110650038B (en) | Security event log collecting and processing method and system for multiple classes of supervision objects | |
| US11290367B2 (en) | Hierarchical network configuration | |
| EP2051183B1 (en) | Query processing system and method for database with encrypted column by query encryption transformation | |
| CN103152352A (en) | Perfect information security and forensics monitoring method and system based on cloud computing environment | |
| WO2020186807A1 (en) | System and method for power data linking based on blockchain technology | |
| CN103548022A (en) | Systems and methods of UTF-8 pattern matching | |
| US9992269B1 (en) | Distributed complex event processing | |
| CN104333512A (en) | Distributed memory database access system and method | |
| CN104838620A (en) | Event management in telecommunications networks | |
| CN111740868A (en) | Alarm data processing method and device and storage medium | |
| US11921602B2 (en) | Edge-based data collection system for an observability pipeline system | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| EP3672158B1 (en) | Network slice management | |
| CN116980475B (en) | Data pushing system based on binlog and double annular buffer areas | |
| US20170149892A1 (en) | Large data set updating for network usage records | |
| US8782079B2 (en) | Configuration information management device, distributed information management system and method | |
| CN113742382B (en) | Synchronous cache-based power grid monitoring system information cross-region method | |
| CN117234739B (en) | Methods, devices, systems and storage media for industrial data analysis | |
| Nie et al. | IoTPass: IoT Data Management System for Processing Time-series Data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140416 |