CN116775424B - Audit log analysis method, device and storage device - Google Patents
Audit log analysis method, device and storage device Download PDFInfo
- Publication number
- CN116775424B CN116775424B CN202310505782.3A CN202310505782A CN116775424B CN 116775424 B CN116775424 B CN 116775424B CN 202310505782 A CN202310505782 A CN 202310505782A CN 116775424 B CN116775424 B CN 116775424B
- Authority
- CN
- China
- Prior art keywords
- database
- local
- log
- information
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种审计日志分析方法、设备及存储设备，方法具体如下：日志文件及数据库信息采集：将用户各类sql数据库存储于本地磁盘的日志文件中；读取本地日志文件，将本地日志文件推送至kafka平台对应的主题；直接拉取用户各类sql数据库中的表信息，将表中字段信息持久化到本地；日志文件及数据库信息处理：将本地日志文件解析后的第一操作数据持久化到本地；将表信息进行字段碰撞检测后，再经解析，得到第二操作数据，并持久化到本地；操作数据分析：对第一操作数据和第二操作数据进行统计分析，并通过统一接口服务对外展示分析结果。有益效果是：提高了日志解析的兼容性，提升了处理大量级数据下的处理性能，提升系统整体安全性。
The present invention discloses an audit log analysis method, device and storage device, and the method is as follows: log file and database information collection: storing various sql databases of users in log files of local disks; reading local log files, and pushing local log files to topics corresponding to kafka platforms; directly pulling table information in various sql databases of users, and persisting field information in the tables locally; log file and database information processing: persisting the first operation data after parsing the local log file locally; performing field collision detection on the table information, and then parsing to obtain the second operation data, and persisting it locally; operation data analysis: statistically analyzing the first operation data and the second operation data, and displaying the analysis results externally through a unified interface service. The beneficial effects are: improving the compatibility of log parsing, improving the processing performance under large-scale data processing, and improving the overall security of the system.
Description
Technical Field
The invention relates to the field of audit log analysis, in particular to an audit log analysis method, audit log analysis equipment and audit log storage equipment.
Background
At present, the importance of various industries on data security is higher and higher, the network environment is more complex, operations on a database come from various different entities inside and also come from outside, thousands of times of inquiry modification operations are generated, and some operations may be sensitive or potential safety hazards exist (such as that a password field of a certain user table is accessed for many times, or the transaction money of an order is modified abnormally, a certain unknown ip accesses the database, and a table which should not be accessed is accessed for many times), so that various safety-oriented mechanisms are needed to realize monitoring on the operation statement of the current database, and the operation statement is presented to a user after statistical analysis, so that the user can more intuitively find potential safety risks.
The prior art generally has no universality for various databases, and the analysis engine of the databases is often relied on to analyze the sql of the databases, so that the analysis of the sql statement of a certain specific database can be realized, but the complex environment is often more than one database, only one database is often analyzed, the requirements cannot be met, and the scheme of whether corresponding characters are contained or not is relatively low according to the conventional field collision logic.
Disclosure of Invention
In order to solve the technical problems of poor analysis compatibility and low efficiency of the existing audit log, the invention provides an audit log analysis method, equipment and storage equipment, wherein the method specifically comprises the following steps:
S1, acquiring log files and database information:
For the log file, storing various sql databases of the user in the log file of a local disk, reading the local log file, and pushing the local log file to a theme corresponding to the kafka platform;
for database information, directly pulling table information in various sql databases of users, and persisting field information in the table to the local;
S2, processing log files and database information:
persistence of the first operation data after the analysis of the local log file to the local;
After field collision detection is carried out on the table information, analyzing the table information to obtain second operation data, and lasting the second operation data to the local;
And S3, analyzing the operation data, namely performing statistical analysis on the first operation data and the second operation data, and displaying analysis results outwards through unified interface service.
A storage device stores instructions and data for implementing an audit log analysis method.
An audit log alarming device comprises a processor and a storage device, wherein the processor loads and executes instructions and data in the storage device to realize an audit log analysis method.
The method has the advantages that log analysis compatibility is improved, processing performance under large-magnitude data processing is improved, alarming can be timely carried out after sensitive data are collided, and overall safety of the system is improved.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a schematic diagram of stepwise data fetching from the map;
FIG. 3 is a diagram of the operation of the hardware device of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be further described with reference to the accompanying drawings.
The related technical concept in the application is as follows:
(1) Kafka is a distributed, partition-supporting and multi-copy (replica) distributed message system based on zookeeper coordination, and has the biggest characteristic of being capable of processing a large amount of data in real time to meet various requirement scenes;
(2) Persistence is a mechanism by which program data transitions between a persistent state and an instantaneous state. Colloquially, instant data (such as data in memory that cannot be permanently stored) persists as persistent data (such as persisted into a database that can be permanently stored);
Referring to fig. 1, fig. 1 is a flow chart of the method of the present invention.
The invention provides an audit log analysis method, audit log analysis equipment and audit log storage equipment, wherein the audit log analysis method specifically comprises the following steps:
S1, acquiring log files and database information:
For the log file, storing various sql databases of the user in the log file of a local disk, reading the local log file, and pushing the local log file to a theme corresponding to the kafka platform;
for database information, directly pulling table information in various sql databases of users, and persisting field information in the table to the local;
It should be noted that, in step S1, the process of persisting the fields in the table to the local is as follows:
Reading a local database field table database_field and loading the database_field to a local FIELDLIST;
reading a local database information table table_info and loading the table_info to a local tableList;
The local database table database_info is read and loaded to the local databaseList.
S2, processing log files and database information:
persistence of the first operation data after the analysis of the local log file to the local;
After field collision detection is carried out on the table information, analyzing the table information to obtain second operation data, and lasting the second operation data to the local;
It should be noted that, the conventional method receives the log collector normative and sends the normative log to the database operation log of kafka, and analyzes sql according to the table field information, the table information and other information stored in the data, performs enhancement operation on the original log data, and then stores the enhanced log data in the local database for later analysis and use.
Different from the conventional scheme, the method converts the related information of the queried database into a Map structure, caches the Map structure in a local cache system, divides sql of a user operation log into individual words, fetches data step by step from the Map according to the words, finally acquires field classes containing all information of the words to perform attribute enhancement and other operations, and the method can greatly reduce time complexity and predict that O (mn) is reduced to O (n) -O (nlogm), n is the number of the sql divided words, and m is the number of database fields.
Referring to fig. 2, fig. 2 is a schematic diagram of stepwise data fetching from the map;
Specifically, the log and database information processing in step S2 is as follows:
constructing database information baseMap according to the field table database_field, the information table table_info and the database table database_info;
Caching database information baseMMap;
dividing sql sentences in the local log into sqlWords lists by taking spaces, ", and";
For example, select id, date_ time from network _match will be segmented into [ "select", "id", "date_time", "from", "networ k _match" ];
traversing sqlWords the list, and obtaining a database dMap corresponding to the database table database_info through a basemap. Get (sqlWord) function;
wherein sqlWord is a specified sql statement;
Obtaining a corresponding information table tMap through a dMap.get () function;
obtaining a corresponding field table fMap through a tmap.get () function;
obtaining a corresponding field class file through an fMap.get () function;
data enhancement is carried out on the field class flied;
Adding enhancement data to the operation table and the inner surface of the association table of the database;
In the invention, a database field table (database_field), an operation table and an association table are respectively shown in tables 1,2 and 3;
Table 1 database field table
Table 2 database operating table (data_ manipulate)
TABLE 3 operation-field association table (manipulate _relation)
And if the enhancement data are sensitive data, directly generating alarm data to push to the kafka platform, and executing corresponding notification or blocking operation.
For example, in the sql collision process, when a sensitive field (such as a password) marked by a user in advance is touched, or a sensitive operation is combined, and an update operation is performed on a field, such as a name, which should not be modified, alarm data is directly generated and pushed to kafka, and the alarm program judges that a corresponding operation (e.g. sending a mail or a short message, or directly blocking the work of a corresponding ip through a firewall according to the operation ip of a log) is performed.
As an embodiment, for example, the log after normalization collected by the present invention:
{
"host": "192.168.184.131",// operations ip
"Data_type": "mysql",// database type
"Manipulation": SELECT id, sql FROM @ mysql @ general_log @ LIMIT 0,1000",// operation statement
}
The enhanced log processed by the steps is as follows:
Step 1, decomposing manipulation into word list wordList ([ 'SELECT', 'id', 'sql' FROM 'mysql', 'general_log', 'LIMIT' ]
Step 2, obtaining baseMap a library list of mysql in the database type field data_type, taking wordList to collide with the library information mysql one by one, and storing the library information mysql in a databases field
Step 3, inquiring a list of table information under the mysql library according to the mysql library, obtaining the table information of general_log by carrying out collision one by using wordList, and storing the table information into tables field
Step 4, inquiring a field list under the table according to the general_log, obtaining field information, and storing the field information in fields
Step 5, judging the sentence operation type according to wordList first words are select, alter, update, delete, and storing in the type field
Step 6, judging whether the field is sensitive or not, storing the sensitive operation into the alert field, and then deciding whether to generate an alarm and a corresponding operation according to the field, and ending the operation
After analysis:
{
"host": "192.168.184.131",// operations ip
"Data_type": "mysql",// database type
"Manipulation": SELECT id, sql FROM @ mysql @ general_log @ LIMIT 0,1000",// operation statement
"Databases" [ "mysql" ],// database
"Tables" [ "general_log" ]// table
"Fields" [ "id", "sql" ]// fields
"Type": "select",// type of operation
"Alert": "false"// whether sensitive data is manipulated
}
And S3, analyzing the operation data, namely performing statistical analysis on the first operation data and the second operation data, and displaying analysis results outwards through unified interface service.
For example, the storage statement database after storage analysis is queried, a certain ip address (192.168.182.131) is counted to perform 20 update operations on a mysql database user information table user_info, 120 password fields password of the table are checked, and because the password fields are sensitive fields, an alarm is generated and a system administrator user is notified.
Referring to fig. 3, fig. 3 is a schematic working diagram of a hardware device according to an embodiment of the present invention, where the hardware device specifically includes an audit log alarm device 401, a processor 402, and a storage device 403.
An audit log alarming device 401, wherein the audit log alarming device 401 implements the audit log analysis method.
Processor 402 the processor 402 loads and executes instructions and data in the storage device 403 for implementing the one audit log analysis method.
The storage device 403 is used for storing instructions and data, and the storage device 403 is used for realizing the audit log analysis method.
The method has the advantages of improving log analysis compatibility, improving processing performance under large-magnitude data processing, timely alarming after collision with sensitive data, and improving overall safety of the system.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.
Claims (4)
1. The audit log analysis method is characterized by comprising the following steps of:
S1, acquiring log files and database information:
For the log file, storing various sql databases of the user in the log file of a local disk, reading the local log file, and pushing the local log file to a theme corresponding to the kafka platform;
for database information, directly pulling table information in various sql databases of users, and persisting field information in the table to the local;
the process of persisting the fields in the table to the local in step S1 is as follows:
Reading a local database field table database_field and loading the database_field to a local FIELDLIST;
reading a local database information table table_info and loading the table_info to a local tableList;
reads the local database table database_info and loads it to the local databaseList
S2, processing log files and database information:
persistence of the first operation data after the analysis of the local log file to the local;
After field collision detection is carried out on the table information, analyzing the table information to obtain second operation data, and lasting the second operation data to the local;
the log and database information processing in step S2 is as follows:
constructing database information baseMap according to the field table database_field, the information table table_info and the database table database_info;
Caching database information baseMap;
dividing sql sentences in the local log into sqlWords lists by taking spaces, ", and";
traversing sqlWords the list, and obtaining a database dMap corresponding to a database table_info through a basemap. Get (sqlWord) function, wherein sqlWord is a specified sql statement;
Obtaining a corresponding information table tMap through a dMap.get () function;
obtaining a corresponding field table fMap through a tmap.get () function;
obtaining a corresponding field class file through an fMap.get () function;
data enhancement is carried out on the field class flied;
Adding enhancement data to the operation table and the inner surface of the association table of the database;
And S3, analyzing the operation data, namely performing statistical analysis on the first operation data and the second operation data, and displaying analysis results outwards through unified interface service.
2. The audit log analysis method according to claim 1, wherein if the enhanced data is sensitive data, alarm data is directly generated and pushed to the kafka platform, and a corresponding notification or blocking operation is performed.
3. A storage device is characterized in that the storage device stores instructions and data for implementing an audit log analysis method according to any one of claims 1-2.
4. An audit log alarm device is characterized by comprising a processor and a storage device, wherein the processor loads and executes instructions and data in the storage device to realize the audit log analysis method according to any one of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310505782.3A CN116775424B (en) | 2023-05-06 | 2023-05-06 | Audit log analysis method, device and storage device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310505782.3A CN116775424B (en) | 2023-05-06 | 2023-05-06 | Audit log analysis method, device and storage device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116775424A CN116775424A (en) | 2023-09-19 |
| CN116775424B true CN116775424B (en) | 2025-03-25 |
Family
ID=88008933
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310505782.3A Active CN116775424B (en) | 2023-05-06 | 2023-05-06 | Audit log analysis method, device and storage device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116775424B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119336758A (en) * | 2024-10-15 | 2025-01-21 | 北京鸿鹄元数科技有限公司 | Data-level lineage analysis method and system based on system log |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20220087408A (en) * | 2021-06-25 | 2022-06-24 | 아폴로 인텔리전트 커넥티비티 (베이징) 테크놀로지 씨오., 엘티디. | Log audit method, log audit device, electronic equipment, storage medium and computer program |
| CN115509995A (en) * | 2022-08-24 | 2022-12-23 | 智网安云(武汉)信息技术有限公司 | Address processing method based on flow log matching |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050114505A1 (en) * | 2003-11-26 | 2005-05-26 | Destefano Jason M. | Method and apparatus for retrieving and combining summarized log data in a distributed log data processing system |
| CN103729442B (en) * | 2013-12-30 | 2017-11-24 | 华为技术有限公司 | Record the method and database engine of transaction journal |
-
2023
- 2023-05-06 CN CN202310505782.3A patent/CN116775424B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20220087408A (en) * | 2021-06-25 | 2022-06-24 | 아폴로 인텔리전트 커넥티비티 (베이징) 테크놀로지 씨오., 엘티디. | Log audit method, log audit device, electronic equipment, storage medium and computer program |
| CN115509995A (en) * | 2022-08-24 | 2022-12-23 | 智网安云(武汉)信息技术有限公司 | Address processing method based on flow log matching |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116775424A (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11741089B1 (en) | Interactive location queries for raw machine data | |
| CN111881011B (en) | Log management method, platform, server and storage medium | |
| CN110347716B (en) | Log data processing method, device, terminal equipment and storage medium | |
| US9355152B2 (en) | Non-exclusionary search within in-memory databases | |
| CN112463800A (en) | Data reading method and device, server and storage medium | |
| WO2020134684A1 (en) | Information retrieval method, apparatus, device and medium | |
| CN112948396A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN107733902A (en) | A kind of monitoring method and device of target data diffusion process | |
| CN116775424B (en) | Audit log analysis method, device and storage device | |
| CN114610754B (en) | SQL log anomaly detection method, device, storage medium and electronic device | |
| CN110110153A (en) | A kind of method and apparatus of node searching | |
| CN119088676A (en) | Method, device, electronic device and storage medium for generating program optimization suggestions | |
| US20100138795A1 (en) | Managing advisories for complex model nodes in a graphical modeling application | |
| CN114547406A (en) | Data monitoring method, system, storage medium and electronic device | |
| CN107920067B (en) | Intrusion detection method on active object storage system | |
| CN114168616A (en) | Data acquisition method, device, electronic device and storage medium | |
| CN118860378A (en) | API management method, system, device, storage medium and program product | |
| CN116881962B (en) | Security monitoring system, method, device and storage medium | |
| CN117421640A (en) | API asset identification method, device, equipment and storage medium | |
| CN117349307A (en) | Method, device, equipment and storage medium for acquiring associated data of slow query | |
| CN117591477A (en) | A log aggregation query method for massive data | |
| CN117938428A (en) | A method, device, electronic device and storage medium for reporting alarm logs | |
| CN117093555A (en) | Method, device, equipment and readable storage medium for acquiring equipment state information | |
| CN113641702B (en) | Method and device for interactive processing with database client after statement audit | |
| CN115509995A (en) | Address processing method based on flow log matching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |