CN114595197B - Abnormal data management method and device - Google Patents
Abnormal data management method and device Download PDFInfo
- Publication number
- CN114595197B CN114595197B CN202210186388.3A CN202210186388A CN114595197B CN 114595197 B CN114595197 B CN 114595197B CN 202210186388 A CN202210186388 A CN 202210186388A CN 114595197 B CN114595197 B CN 114595197B
- Authority
- CN
- China
- Prior art keywords
- abnormal data
- file
- block
- abnormal
- block file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/122—File system administration, e.g. details of archiving or snapshots using management policies
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Retry When Errors Occur (AREA)
Abstract
The invention discloses an abnormal data management method which comprises the following steps of S1, recording basic information of abnormal data in an abnormal occurrence time period, creating a block file in a local file, writing abnormal data records in the block file, S2, starting a corresponding thread to read the abnormal data records in the block file according to the abnormal data type, and S3, carrying out recovery retry on the read abnormal data according to an abnormal data recovery request. The abnormal monitoring module detects abnormal conditions, establishes a block file, stores abnormal data into a local file when the abnormal data occurs, reads the recovered abnormal data from the local module when the abnormal information needs to be recovered, avoids the condition that the data is not processed due to unstable network, downtime of a database and the like, ensures that the abnormal data is not lost when the system is recovered to be normal, and improves the recovery efficiency of the abnormal data.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and apparatus for managing abnormal data.
Background
In a service-oriented data system architecture, in order to avoid excessive coupling of each functional module, data is usually transferred between modules to push a business process or interaction through data communication, but certain risks often exist in transferring data, so that abnormal data transmission and even data transmission failure are caused, for example, database downtime, unstable network and the like.
When a data anomaly occurs, the prior art generally resorts to retrying the transmission of the data. However, there are problems in practice that retransmission of data cannot be repeatedly attempted for a long time, and in the case of a large amount of data and a complex data type, the data recovery efficiency is low or the difficulty is large.
Disclosure of Invention
The invention provides an abnormal data management method and device capable of efficiently recovering abnormal data, aiming at the problem that the abnormal data in the data system architecture in the prior art is difficult to recover under the condition of huge data volume and complex data types.
In one aspect, the present application provides an abnormal data management method, including the steps of:
step S1, recording basic information of abnormal data in an abnormal occurrence time period, creating a block file in a local file, and writing abnormal data record in the block file;
Step S2, starting a corresponding thread for the same type of abnormal data according to the type of the abnormal data to read the abnormal data record in the block file;
and step S3, according to the abnormal data recovery request, carrying out recovery retry on the read abnormal data. The block file is used for recording the abnormal data, so that the processing speed of the abnormal data can be greatly improved, and the abnormal data can be rapidly recorded into the local file when the abnormality occurs, so that the abnormal data can be directly read and recovered from the local file after the abnormality occurs. The situation that data is not processed due to unstable network, downtime of a database and the like is avoided, abnormal data is not lost when the system is recovered to be normal, and the recovery efficiency of the abnormal data is improved.
Preferably, creating the block files in the local file in step S1 includes creating first block areas with consecutive addresses in the local disk, each first block area corresponding to one block file, writing start position information and end position information of the block file in the head of each block file, and writing length and end identifier of the block file in the tail of each block file. To support reverse order query reading of block files.
Preferably, the writing of the abnormal data records in the block file in step S1 includes writing the abnormal data in each record and writing the length data of each record in the head and tail of the record. To support reading the record in reverse order.
Preferably, the step S1 further includes respectively creating second block areas in the local file according to the type of the abnormal data, each second block area corresponds to a second block file, writing a position index of the abnormal data recorded in the local file in the second block file, updating the length of the file and the end position identifier of the file at the tail of the second block file at the end position of the new file and at the head of the second block file.
Preferably, the step S2 of reading the abnormal data record in the block file includes obtaining abnormal data position index information from the second block area, finding the abnormal data record in the first block file according to the abnormal data position index information, reading the abnormal data, and recording the current read abnormal data number. And the abnormal data can be conveniently retrieved.
Preferably, step S3 further includes cleaning up the abnormal data record that has been restored to completion according to the current abnormal data processing progress.
In another aspect, the present application provides an abnormal data management apparatus, in which the abnormal data management method of the present application is incorporated.
Preferably, the system comprises an abnormality monitoring module, a data processing module and an abnormality data recording module, wherein the abnormality monitoring module monitors and detects abnormal occurrence, and when the abnormal occurrence is determined, signals with abnormality occurrence information are transmitted to the abnormality data recording module;
The abnormal data recording module is used for receiving the signal with the abnormal occurrence information sent by the abnormal monitoring module and storing the abnormal data in the local file when the abnormality occurs;
The abnormal data reading module is provided with multi-thread communication, and after receiving an abnormal data recovery request signal, the abnormal data reading module starts a corresponding thread to read abnormal data records in the block files for the same type of abnormal data;
and the abnormal data recovery module is used for receiving an abnormal data recovery request and carrying out recovery retry on the read abnormal data.
Preferably, the abnormal data recording module includes:
The generation unit is used for respectively creating first block areas for storing abnormal data in the local disk, and respectively creating second block areas for writing position index information in the local file according to the type of the abnormal data;
The writing unit is used for writing abnormal data records in the first block file, writing abnormal data in each record, writing length data of the record in the head and tail of each record, writing a position index of the abnormal data record in the local file in the second block file, updating the length of the file and the end position identifier of the file in the tail of the second block file at the end position of the head of the second block file and the end position of the new file. And the block files are conveniently queried in reverse order, and the recovery efficiency of abnormal data is further improved.
Preferably, the generating unit is further configured to create a new first block file after the available capacity of the first block file reaches the lower limit, and create a new second block file after the available capacity of the second block file reaches the lower limit. Ensuring that there is always sufficient bulk document capacity ready.
The abnormal data management method and device have the advantages that the abnormal data management method and device are provided, the abnormal monitoring module detects abnormal conditions and builds a block file, when abnormal data occurs, the abnormal data are stored in the local file, when the abnormal information needs to be restored, the abnormal data are read from the local module, the situation that the data are not processed due to unstable network, downtime of a database and the like is avoided, abnormal data are not lost when the system is restored, and the abnormal data restoration efficiency is improved.
Drawings
FIG. 1 is a flowchart of an abnormal data management method according to an embodiment of the present invention.
Fig. 2 is a flowchart illustrating a subdivision step of the step S1 of the abnormal data management method according to an embodiment of the present invention.
FIG. 3 is a schematic diagram of a first block file structure according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a single abnormal data record structure according to an embodiment of the present invention.
Fig. 5 is a block diagram of an abnormal data management apparatus according to an embodiment of the present invention.
In the figure, 1, an abnormal monitoring module 2, an abnormal data recording module 3, an abnormal data reading module 4, an abnormal data recovery module 5, an abnormal data recording 6, file header information 7, file tail information 8, recording header information 9, recording tail information 10 and abnormal data are shown.
Detailed Description
The invention is further described below with reference to the drawings and detailed description.
Examples:
The abnormal data management method as shown in fig. 1 comprises the following steps:
step S1, recording basic information of abnormal data in an abnormal occurrence time period, creating a block file in a local file, and writing abnormal data record 5 in the block file;
Step S1 comprises the following subdivision steps as shown in fig. 2:
s1001 creating a Block File
And creating a block area with a certain size of continuous addresses in the local disk, and corresponding to a block file for recording abnormal data. After the accommodable data of each block file reaches an upper limit, a new block file is created. Recording the abnormal data by the block file can greatly improve the reading and writing speed of the abnormal data.
S1002 initializing a block file
And writing a starting position and an ending position of the file in the head part of the block file to determine the storage space position of the block file.
The length of the file and the end identifier of the file are written at the tail of the block file to support reverse order query reading of the block file.
S1003 write abnormal data record 5
And when the occurrence of the abnormality is monitored, recording the abnormality into a block file.
The length of the record is written in the head of the record to determine the storage space of the record.
And writing the abnormal data of the record in the middle of the record.
The length of the record is written at the end of the record to support reading the record in reverse order.
S1004, storing abnormal data record index
According to the type of the abnormal data, block areas are respectively created, and the position indexes of the abnormal data records 5 are written, so that the abnormal data can be conveniently retrieved.
S1005, updating head and tail information of block files
The end position of the file is updated at the head of the block file.
The length of the file is updated at the end of the block file and the end identifier of the file.
And S2, starting a corresponding thread to read the abnormal data record 5 in the block file according to the abnormal data type, wherein reading the abnormal data record 5 in the block file comprises the steps of acquiring abnormal data position index information from a second block area, finding the abnormal data record 5 in the first block file according to the abnormal data position index information, reading the abnormal data, and recording the current read abnormal data number.
And step S3, according to the abnormal data recovery request, carrying out recovery retry on the read abnormal data. And cleaning up the abnormal data record 5 which is already recovered and completed according to the current abnormal data processing progress.
The first block file header information 6 comprises a start position and an end position as shown in fig. 3 and the file trailer information 7 comprises a length of the file and an end identifier of the file, between which the exception data record 5 is written.
As shown in fig. 4, the recording header information 8 of the single abnormal data record 5 includes the length of the record, the recording trailer information 9 includes the length of the record, and abnormal data 10 is written between the headers and between the trailers.
Figure 5 is a schematic diagram of an exception data processing apparatus according to one embodiment of the present application,
The system comprises an abnormality monitoring module 1, an abnormality data recording module 2 and a control module, wherein the abnormality monitoring module monitors and detects abnormal occurrence, and when the abnormal occurrence is determined, signals with abnormality occurrence information are transmitted to the abnormality data recording module 2;
the abnormal data recording module 2 is used for receiving the signal with the abnormal occurrence information sent by the abnormal monitoring module 1 and storing the abnormal data in the local file when the abnormality occurs;
An abnormal data recording module 2 includes:
The generation unit is used for respectively creating first block areas for storing abnormal data in the local disk, respectively creating second block areas for writing position index information in the local file according to the type of the abnormal data, and also used for creating a new first block file after the available capacity of the first block file reaches the lower limit and creating a new second block file after the available capacity of the second block file reaches the lower limit.
The writing unit is used for writing abnormal data records in the first block file, writing abnormal data in each record, writing length data of the record in the head and tail of each record, writing a position index of the abnormal data record in the local file in the second block file, updating the length of the file and the end position identifier of the file in the tail of the second block file at the end position of the head of the second block file and the end position of the new file.
The abnormal data reading module 3 is provided with multi-thread communication, and after receiving an abnormal data recovery request signal, the abnormal data reading module opens a corresponding thread for the same abnormal data to read the abnormal data record in the block file;
And the abnormal data recovery module 4 receives the abnormal data recovery request and performs recovery retry on the read abnormal data.
When abnormal data occurs, the abnormal data is stored in the local file, when the abnormal information needs to be recovered, the abnormal data is read from the local module, the situation that the data is not processed due to unstable network, downtime of a database and the like is avoided, the abnormal data is not lost when the system is recovered, and the recovery efficiency of the abnormal data is improved.
The specific embodiments described herein are offered by way of example only to illustrate the spirit of the invention. Those skilled in the art may make various modifications or additions to the described embodiments or substitutions thereof without departing from the spirit of the invention or exceeding the scope of the invention as defined in the accompanying claims.
Although the terms anomaly data, block areas, block files, logos, etc. are used more herein, the possibility of using other terms is not precluded. These terms are only used to more conveniently describe and explain the nature of the invention and should be construed in a manner consistent with their spirit and scope.
Claims (8)
1. An abnormal data management method is characterized by comprising the following steps:
Step S1, recording basic information of abnormal data in an abnormal occurrence time period, creating first block areas with continuous addresses in a local disk, wherein each first block area corresponds to a first block file, writing initial position information and end position information of the first block file in the head of each first block file, writing abnormal data record in the first block file;
step S2, starting a corresponding thread to read the abnormal data record in the first block file according to the abnormal data type, wherein the step comprises the steps of acquiring abnormal data position index information from a second block area, finding the abnormal data record in the first block file according to the abnormal data position index information, reading the abnormal data, and recording the current read abnormal data number;
and step S3, according to the abnormal data recovery request, carrying out recovery retry on the read abnormal data.
2. The method of claim 1, wherein the step S1 includes creating first block areas with consecutive addresses in the local disk, each first block area corresponding to a first block file, and writing a length and an end identifier of the first block file at the end of each block file.
3. The method of claim 2, wherein writing the abnormal data record in the first block file in step S1 includes writing the abnormal data in each record and writing the length data of each record in the head and tail of the record.
4. A method of managing abnormal data according to claim 3, wherein said step S1 further comprises creating second block areas in the local file according to the type of abnormal data, respectively, each second block area corresponding to a second block file, writing a location index of the abnormal data recorded in the local file in the second block file, updating an end location of the file at the head of the second block file, updating a length of the file and an end location identifier of the file at the tail of the second block file.
5. The method according to claim 1, wherein step S3 further comprises clearing the abnormal data record that has been restored according to the current abnormal data processing progress.
6. An abnormal data management apparatus employing the abnormal data management method according to any one of claims 1 to 5, comprising:
The abnormality monitoring module monitors and detects abnormal occurrence conditions, and transmits signals with abnormality occurrence information to the abnormality data recording module when the abnormal occurrence conditions are determined;
The abnormal data recording module is used for receiving the signal with the abnormal occurrence information sent by the abnormal monitoring module and storing the abnormal data in the local file when the abnormality occurs;
The abnormal data reading module is provided with multi-thread communication, and after receiving an abnormal data recovery request signal, the abnormal data reading module starts a corresponding thread to read abnormal data records in the first block file for the same type of abnormal data;
and the abnormal data recovery module is used for receiving an abnormal data recovery request and carrying out recovery retry on the read abnormal data.
7. The apparatus according to claim 6, wherein the abnormal data recording module comprises a generation unit for creating first block areas for storing abnormal data in the local disk, respectively, and creating second block areas for writing position index information in the local file according to the type of the abnormal data, respectively;
The writing unit is used for writing abnormal data records in the first block file, writing abnormal data in each record and writing length data of the record in the head and tail of each record, writing a position index of the abnormal data record in the local file in the second block file, updating the end position of the file in the head of the second block file, and updating the length of the file and the end position identifier of the file in the tail of the second block file.
8. The apparatus according to claim 7, wherein the generating unit is further configured to create a new first block file after the available capacity of the first block file reaches a lower limit, and create a new second block file after the available capacity of the second block file reaches a lower limit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210186388.3A CN114595197B (en) | 2022-02-28 | 2022-02-28 | Abnormal data management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210186388.3A CN114595197B (en) | 2022-02-28 | 2022-02-28 | Abnormal data management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114595197A CN114595197A (en) | 2022-06-07 |
| CN114595197B true CN114595197B (en) | 2025-04-01 |
Family
ID=81806957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210186388.3A Active CN114595197B (en) | 2022-02-28 | 2022-02-28 | Abnormal data management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114595197B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103150226A (en) * | 2013-04-01 | 2013-06-12 | 山东鲁能软件技术有限公司 | Abnormal dump and recovery system for computer model and dump and recovery method thereof |
| CN103186461A (en) * | 2011-12-30 | 2013-07-03 | 重庆重邮信科通信技术有限公司 | Storage method and recover method for field data, and related devices |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103107905B (en) * | 2011-11-14 | 2017-08-04 | 华为技术有限公司 | Abnormality eliminating method, device and client |
| CN107908495B (en) * | 2017-11-15 | 2021-12-10 | 南京南瑞继保电气有限公司 | Embedded system abnormal record display method |
| CN111338583B (en) * | 2020-05-19 | 2020-08-25 | 北京数字绿土科技有限公司 | High-frequency data storage method and device and computer |
-
2022
- 2022-02-28 CN CN202210186388.3A patent/CN114595197B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103186461A (en) * | 2011-12-30 | 2013-07-03 | 重庆重邮信科通信技术有限公司 | Storage method and recover method for field data, and related devices |
| CN103150226A (en) * | 2013-04-01 | 2013-06-12 | 山东鲁能软件技术有限公司 | Abnormal dump and recovery system for computer model and dump and recovery method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114595197A (en) | 2022-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101809558B (en) | Remote asynchronous data replication system and method | |
| CN100562858C (en) | Method, device and system for remote disaster recovery of memory database | |
| US20040044705A1 (en) | Optimized disk repository for the storage and retrieval of mostly sequential data | |
| JP2011123752A (en) | Device for collecting log, program for collecting log, method for collecting log, and system for collecting log | |
| US20050267916A1 (en) | Data backup system and method | |
| CN112380260B (en) | Broken line caching method based on different acquisition scenes | |
| CN104991747A (en) | Method and system for data management | |
| JPH04360227A (en) | Magnetic tape control device | |
| WO2010080781A1 (en) | Systems and methods for monitoring archive storage condition and preventing the loss of archived data | |
| CN114595197B (en) | Abnormal data management method and device | |
| CN108234465B (en) | Redundancy method and device for handling exception in distributed file system | |
| US7913109B2 (en) | Storage control apparatus and storage control method | |
| CN112181973B (en) | Time sequence data storage method | |
| JP2009017634A (en) | Train data recording system | |
| CN101169956A (en) | Disc device | |
| CN103207916A (en) | Metadata processing method and device | |
| CN100426246C (en) | Method for protecting cached data of storage system | |
| CN103137159A (en) | Direct-to-disc fault-tolerant method for videos in hard-disk based digital video recorder and device thereof | |
| CN112711497A (en) | Recovery method and system for unfolded faults of container deployment Ceph cluster object | |
| US8811133B2 (en) | Writing system, writing device, and writing method | |
| CN117851511A (en) | A data synchronization method and system for multiple data centers | |
| CN116700610A (en) | Data processing method, device and system | |
| KR102003432B1 (en) | Video managing apparatus and method of surveillance systems | |
| CN113886352A (en) | Metadata recovery method, device, equipment and medium for distributed file system | |
| US20030033440A1 (en) | Method of logging message activity |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |