CN119690909A

CN119690909A - A FUSE-based file system data migration method, system, device and medium

Info

Publication number: CN119690909A
Application number: CN202510213099.1A
Authority: CN
Inventors: 黎静; 赵亚娥; 强奕铭; 郭奕汐
Original assignee: Shanghai Grid Electric Technology Co ltd
Current assignee: Shanghai Grid Electric Technology Co ltd
Priority date: 2025-02-26
Filing date: 2025-02-26
Publication date: 2025-03-25

Abstract

The present invention relates to a FUSE-based file system data migration method, system, device and medium, belonging to the field of data migration protection technology. The system includes a data protection file system module, a FUSE file system module, a detection and feedback module, and a system configuration and maintenance management module; the data protection file system module includes a metadata submodule and a data snapshot submodule, the metadata submodule is used for data mapping and related metadata management, and the data snapshot submodule is used for data backup, unique ID management, and verification management; the FUSE file system module is used for mixed storage, compression, and transmission of data; the detection and feedback module is used for recording file operations and statistical data flow; the system configuration and maintenance management module is used for managing system configuration and daily maintenance tasks; the file system is realized without intruding the kernel, reducing the switching between user state and kernel state, improving the efficiency of file system data migration, and realizing real-time data backup protection.

Description

File system data migration method, system, equipment and medium based on FUSE

Technical Field

The invention belongs to the technical field of data migration protection, and particularly relates to a file system data migration method, system, equipment and medium based on FUSE.

Background

Data protection is the process of copying all or part of a data set from the hard disk or array of an application host to other storage media in order to prevent data loss due to system operational errors or system failures. The main stream data protection method comprises the steps of backing up related data of a tape drive, backing up main database data of a standby database, storing backup data of a local network and storing the backup data in different places.

Currently, in the field of data migration, various backup software is commonly used to realize redundant backup of data. However, the simple file copy mode of the third party backup software as a stand-alone application program is not suitable for applications such as databases, and when the opened file has a limitation that the opened file cannot be copied, the database must be closed first for backup, which is obviously impractical. In addition, the third party backup software has the problem of insufficient real-time performance in the data transmission process, namely, the data is written into the host firstly and then is synchronized to the standby machine, which may cause that the standby machine data is not updated timely, and if the data fails at this time, the standby machine data may not be complete and continuous enough. At the same time, the use of third party backup software may also cause degradation in server performance.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides a file system data migration method, a system, equipment and a medium based on FUSE.

The aim of the invention can be achieved by the following technical scheme:

A file system data migration system based on FUSE comprises a data protection file system module, a FUSE file system module, a detection and feedback module and a system configuration and maintenance management module;

The data protection file system module comprises a metadata sub-module and a data snapshot sub-module, wherein the metadata sub-module is used for data mapping and related metadata management, and the data snapshot sub-module is used for data backup, unique ID management and verification management;

The FUSE file system module is used for carrying out mixing, compression and transmission on the acquired data through a file system kernel mode interface and a file system user mode interface, and comprises a file system kernel mode interface submodule and a file system user mode interface submodule;

The detection and feedback module is used for recording file operation and statistical data flow and comprises a file operation recording submodule and a data flow statistical submodule;

The system configuration and maintenance management module is used for managing system configuration and daily maintenance tasks and comprises a file system mounting and dismounting sub-module, a remote storage fault short-circuit switch sub-module and a log management sub-module.

Preferably, the data map includes a local Super block data map, dentry data map, inode data map, and the related metadata management includes, but is not limited to, namespace management, storage path management, file attribute management, quota management, and file check management.

Preferably, the local Super block data map is used to manage global information of the file system, including but not limited to block size and inode number;

The Dentry data mapping is used for maintaining the corresponding relation between the file name and the inode so as to quickly locate the file;

The Inode data map is used to store metadata information for the file including, but not limited to, file size, creation time.

Preferably, the data backup comprises snapshot creation, idempotent assurance and consistency assurance;

the unique ID management distributes unique identification for each file operation and prevents the data from being repeatedly written in the related backup operation;

The check management comprises data check and interface check, wherein the data check is used for carrying out data check after the snapshot is created, so that the data in the snapshot is consistent with the original data, and the interface check is used for relevant interface check, so that the input parameters and the execution result are correct and consistent when the backup operation is called.

Preferably, the snapshot creation is used for creating a snapshot of the data of each operation, the snapshot comprises the state of the data at a specific time point so as to ensure that the data can be restored to the previous state when the data is lost or damaged, the idempotent guarantee is used for designing the snapshot operation into idempotent so as to ensure that the result is still the same after the operation is repeatedly executed, and the consistency guarantee is used for ensuring the consistency of the data during the snapshot operation.

Preferably, the kernel mode interface submodule of the file system processes data through a kernel mode interface, directly intercepts data in a VFS layer, realizes kernel mode data reading and writing, and maintains kernel mode data cache;

the file system user mode interface sub-module realizes file system operation in a user mode, and realizes user mode data reading and writing through interaction between FUSE and a kernel mode interface.

The remote storage fault short-circuit switch submodule is used for monitoring the remote storage state and providing a short-circuit switch to quickly switch a storage path to standby storage when the remote storage fails;

The log management submodule comprises log grade, switch configuration and log data rolling deletion, wherein the log data rolling deletion is carried out according to file time, size and the like, and the log comprises system fault log records and data operation log records and is used for recording log information in the running process of a system.

A file system data migration method based on FUSE is applied to the file system data migration system based on FUSE, and comprises the following steps:

When application is mounted on FUSE and system call is executed, the operation of the system call is routed to a FUSE driver through VFS, wherein the FUSE comprises a kernel module and a FUSE daemon;

The fuse driver creates a fuse request structure body and stores the request in a request queue, and the process of executing the operation is blocked;

The fuse daemon reads a fuse request of a kernel queue in the kernel module through/dev/fuse block equipment and submits operation to an underlying file system, wherein the underlying file system comprises EXT4 and F2FS;

After the bottom file system processes the request, the fuse daemon writes the reply back to the/dev/fuse block device, and the fuse driver marks the request as completed and wakes up the user process.

An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the FUSE-based file system data migration system described above when executing the computer program.

A storage medium containing computer executable instructions that when executed by a computer processor are for performing the FUSE-based file system data migration system described above.

The beneficial effects of the invention are as follows:

(1) The non-invasive kernel interaction is realized by introducing the FUSE, the file system is realized under the condition of not invading the kernel, the data migration of the file system is carried out in the user space, the incremental backup is directly carried out on the bottom layer by using the file system, the kernel code is not modified, the switching between the user mode and the kernel mode when the data is read and transmitted is reduced, and the data migration efficiency is effectively improved.

(2) The snapshot creation ensures that the data can be restored to the previous state when the data is lost or damaged, the idempotent property ensures that the repeated execution is the same as the result obtained by the repeated execution, the consistency of the data during the snapshot operation is ensured through the consistency guarantee, and the problem that the data is not updated timely is solved.

(3) By designing the buffer area, the data is accumulated to a certain amount or is sent in batches for a certain time, so that unnecessary memory allocation and copying are avoided as much as possible, the requirement of local data backup to a far end is met, the system call cost caused by frequent small data amount multi-group number sending is reduced, and the data sending speed is greatly improved.

Drawings

The present invention is further described below with reference to the accompanying drawings for the convenience of understanding by those skilled in the art.

FIG. 1 is a schematic diagram of a FUSE-based file system data migration system of the present invention.

Detailed Description

In order to further describe the technical means and effects adopted by the invention for achieving the preset aim, the following detailed description is given below of the specific implementation, structure, characteristics and effects according to the invention with reference to the attached drawings and the preferred embodiment.

Referring to fig. 1, a file system data migration system based on FUSE includes a data protection file system module, a FUSE file system module, a detection and feedback module, and a system configuration and maintenance management module;

In this embodiment, the data map includes a local Super block data map, dentry data map, inode data map, where the related metadata management includes, but is not limited to, namespace management, storage path management, file attribute management, quota management, and file check management;

the local Super block data map is used for managing global information of a file system, wherein the global information comprises, but is not limited to, block sizes and inodes;

the Inode data map is used for storing metadata information of a file, and the metadata information comprises, but is not limited to, file size and creation time;

It should be noted that, the local Super block, denry, inode are all file organization forms of linux vfs.

In this embodiment, the data backup includes snapshot creation, idempotent assurance, and consistency assurance, where the snapshot creation is used to create a snapshot for the data of each operation, where the snapshot includes the state of the data at a specific time point, and ensures that the state can be restored to the previous state when the data is lost or damaged, the idempotent assurance is used to design the snapshot operation into idempotent, and ensures that no matter how many times the operation is executed, the result is the same, no side effect is generated due to repeated execution, and the consistency assurance is used to ensure that the consistency of the data in the snapshot operation, that is, the state of the data reflected by the snapshot is consistent, and no partial update or no update occurs;

Specifically, a 64-bit ID is adopted, the unique ID is indexed through index, the index is organized according to a tree structure and comprises a father catalog ID and a file ID, the backup file is indexed and searched through the index, and the unique ID is used for ensuring that the backup operation cannot cause data to be written for multiple times due to system errors or repeated requests, so that the problem of data repetition or inconsistency is avoided.

The check management comprises data check and interface check, wherein the data check is used for carrying out data check by a system after the snapshot is created, ensuring that the data in the snapshot is consistent with the original data and is not damaged or tampered, and the interface check is used for relevant interface check so as to ensure that the input parameters and the execution result are correct and consistent when the backup operation is invoked.

In this embodiment, the kernel mode interface submodule of the file system processes data through a kernel mode interface, directly intercepts data in the VFS layer, realizes kernel mode data reading and writing, and maintains kernel mode data cache;

It should be noted that, the mixing is to cache data in the memory, speed up the reading operation, the compressing is to reduce the volume of data transmission and improve the transmission efficiency, and the data is transmitted to the target storage location by the transmission.

In this embodiment, the file operation recording submodule is used to record detailed information of file operations, including but not limited to reading, writing, and modifying.

The data flow statistics sub-module is used for counting flow information of data, in particular, reading flow and writing flow of the data.

In this embodiment, the file system mounting and dismounting submodule is used for providing a mounting and dismounting function of the data protection file system;

the remote storage fault short-circuit switch submodule is used for monitoring a remote storage state, and providing a short-circuit switch to rapidly switch a storage path to standby storage when the remote storage fails;

A file system data migration method based on FUSE includes:

When application is installed on a FUSE and a system call is executed, the operation of the system call is routed to a FUSE driver through a VFS, wherein the FUSE comprises a kernel module and a user space daemon (FUSE daemon);

In this embodiment, the application mounting on the FUSE includes mounting the specified fuse_path to the/dev/FUSE device via a mount function.

In this embodiment, when the FUSE reads data through the/dev/FUSE block device, the FUSE uses the splice block of the FUSE to perform memory zero copy, where the splice block allows the user space to transfer data between two kernel memory buffers, without copying the data to the user space.

Specifically, the buffer area form data containing the file descriptor is processed by the write_buf () method of the fuse daemon, so that one memory application and copy are omitted.

In this embodiment, the multithreaded mode read request is implemented by fuse_session_loop_mt, which is more efficient in request processing than a single thread.

Specifically, the Fuse daemon processes file operation requests based on libfuse libraries, and in a multithreading mode, the Fuse daemon starts with one thread, if more than two requests exist in a kernel queue, other threads can be automatically generated, and 10 threads are supported simultaneously by default to process the requests.

In this embodiment, for data to be protected, non-invasive kernel interaction is implemented through FUSE, and file system data migration is performed in user space without modifying kernel codes;

The method comprises the steps of intercepting data in a VFS file system layer, and mixing, compressing and transmitting the acquired data through a kernel mode interface of the file system and a user mode interface of the file system;

It should be noted that, when the kernel module is loaded, the kernel module is registered as a fuse file system driver of the Linux virtual file system, and the fuse file system driver mainly realizes that codes are located in a user space, so that the kernel is not required to be recompiled, and the own file system can be realized under the condition of not invading the kernel;

Specifically, a user-defined file system is realized in a user space through FUSE, remote call between a client and a server is processed through an RPC interface, so that remote backup and recovery of data are realized, FUSE backup file metadata are stored in a key-value form, the file metadata key is a compression code generated according to a father directory and a file ID, the file metadata value comprises file attribute information and file block information, the file metadata key is organized and stored through an index tree of the FUSE backup file system, the file metadata value of backup is obtained through the file metadata key index, and storage of file actual data is performed through the file block information.

In order to reduce the read speed gap between the processor and the memory, a cache memory is generally used to increase the speed of request response and data transmission, i.e. the data is first stored in a buffer area and then accessed and transmitted.

In this embodiment, in order to avoid the system call overhead caused by frequent transmission of small data volume and multiple sets of data, a buffer module is designed to avoid unnecessary memory allocation and duplication as much as possible by designing a buffer area, where the buffer area is a character queue, and includes sender buffering and receiver buffering;

the sender buffers additional data from the tail of the queue, takes out the data from the head of the queue and sends the data to the socket;

and the receiver buffers and takes out data from the head of the queue, and receives data in the socket from the tail of the queue.

Specifically, the buffer module stores the data in the buffer area before the data is transmitted, and the data is accumulated to a certain amount or is transmitted in batches after a certain time is reached, so that the data transmission speed is greatly improved.

The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The present invention is not limited in any way by the above-described preferred embodiments, but is not limited to the above-described preferred embodiments, and any person skilled in the art will appreciate that the present invention can be embodied in the form of a program for carrying out the method of the present invention, while the above disclosure is directed to equivalent embodiments capable of being altered or modified in a slight manner, any and all concise modifications, equivalent variations and alterations of the above embodiments are still within the scope of the present disclosure, all as may be made without departing from the scope of the present disclosure.

Claims

1. A FUSE-based file system data migration system, characterized by comprising a data protection file system module, a FUSE file system module, a detection and feedback module, and a system configuration and maintenance management module;

The data protection file system module includes a metadata submodule and a data snapshot submodule, wherein the metadata submodule is used for data mapping and related metadata management, and the data snapshot submodule is used for data backup, unique ID management, and verification management;

The FUSE file system module is used to store, compress and transmit the acquired data through the file system kernel state interface and the file system user state interface, and includes a file system kernel state interface submodule and a file system user state interface submodule;

The detection and feedback module is used to record file operations and statistical data traffic, including a file operation recording submodule and a data traffic statistics submodule;

The system configuration and maintenance management module is used to manage system configuration and daily maintenance tasks, and includes a file system mounting and unmounting submodule, a remote storage fault short-circuit switch submodule, and a log management submodule.

2. According to the FUSE-based file system data migration system of claim 1, it is characterized in that the data mapping includes local Super block data mapping, Dentry data mapping, and Inode data mapping, and the related metadata management includes but is not limited to namespace management, storage path management, file attribute management, quota management, and file verification management.

3. The FUSE-based file system data migration system according to claim 2, wherein the local Super block data mapping is used to manage the global information of the file system, and the global information includes but is not limited to the block size and the number of inodes;

The Dentry data mapping is used to maintain the correspondence between the file name and the inode so as to quickly locate the file;

The Inode data mapping is used to store metadata information of the file, and the metadata information includes but is not limited to the file size and creation time.

4. The FUSE-based file system data migration system according to claim 1, wherein the data backup includes snapshot creation, idempotence guarantee, and consistency guarantee;

The unique ID management assigns a unique identifier to each file operation to prevent duplicate writing of data in related backup operations;

The verification management includes data verification and interface verification. The data verification is used to perform data verification after creating a snapshot to ensure that the data in the snapshot is consistent with the original data. The interface verification is used to verify the relevant interfaces to ensure that when calling the backup operation, the parameters passed in and the execution results are correct and consistent.

5. According to the FUSE-based file system data migration system of claim 4, it is characterized in that the snapshot creation is used to create a snapshot of the data for each operation, and the snapshot contains the state of the data at a specific point in time to ensure that the previous state can be restored when data loss or damage occurs. The idempotence guarantee is used to design the snapshot operation as idempotent to ensure that the result remains the same after repeated execution of the operation. The consistency guarantee is used to ensure the consistency of data during the snapshot operation.

6. The FUSE-based file system data migration system according to claim 1, characterized in that the file system kernel state interface submodule processes data through the kernel state interface, directly intercepts data at the VFS layer, realizes kernel state data reading and writing, and maintains kernel state data cache;

The file system user state interface submodule implements file system operations in the user state, interacts with the kernel state interface through FUSE, and implements data reading and writing in the user state.

7. The FUSE-based file system data migration system according to claim 1, characterized in that the file system mounting and unmounting submodule is used to provide the mounting and unmounting functions of the data protection file system; the remote storage fault short-circuit switch submodule is used to monitor the remote storage status, and when a remote storage fails, a short-circuit switch is provided to quickly switch the storage path to the backup storage;

The log management submodule includes log level and switch configuration and log data rolling deletion; the log data rolling deletion is performed according to file time, size, etc. The log includes system fault log records and data operation log records, which are used to record log information during system operation.

8. A FUSE-based file system data migration method, characterized in that it is applied to the FUSE-based file system data migration system as claimed in any one of claims 1 to 7, comprising:

When the application is mounted on FUSE and executes a system call, the system call operation is routed to the fuse driver through VFS. The FUSE includes a kernel module and a fuse daemon.

The fuse driver creates a fuse request structure and saves the request in the request queue. The process executing the operation will be blocked.

The fuse daemon reads the fuserequest of the kernel queue in the kernel module through the /dev/fuse block device, and submits the operation to the underlying file system, wherein the underlying file system includes EXT4 and F2FS;

When the underlying file system completes processing the request, the fuse daemon writes a reply back to the /dev/fuse block device, and the fuse driver marks the request as completed and wakes up the user process.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the FUSE-based file system data migration system as described in any one of claims 1 to 7 is implemented.

10. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute the FUSE-based file system data migration system as described in any one of claims 1 to 7 when executed by a computer processor.