CN1317662C - A Distributed File Access Method - Google Patents

Abstract

A distributed file access method, comprising: constructing initial file information in a distributed file system, establishing a file information database; providing a file access interface, and submitting a file access request through the access interface; obtaining file information; File information, determine the specific location of the file; perform file access operations according to the type of file access request and the specific location of the file; locally image the remote node file; synchronize the file information during the operation; in the file information database Real-time persistence; the method of the present invention eliminates the adjustment of this file organization structure, no matter how the application moves in the distributed system, the file system of the application itself remains unchanged, the method of the present invention adopts real-time distributed directory information management, and the file The distributed access of files is completed in a peer-to-peer manner between the client and the server, thus truly providing the application with a file access method that has nothing to do with the actual operating environment.

Description

A Distributed File Access Method

technical field

The present invention relates to a method for accessing files in a distributed processing system, and in particular to an access method that enables access to files on any processing node in the distributed processing system to be the same as accessing local files. The file access processing application does not need to care about the actual storage location of the file. At the same time, the method provides a flexible access method, and uniformly organizes the files managed by the method in the distributed system into a file directory structure to realize the same file management as the centralized file system.

Background technique

The file system is a key component of any distributed processing system. The main function of the file system is to provide access to files, and DFS (Distributed File System) needs to provide the ability to access files scattered on different physical nodes in the system. A good distributed file system can ensure that the application does not need to care about the actual storage location of the file. At the same time, when the running location of the application changes, it does not need to modify the structure of the local file system, and can still ensure that the application can access files correctly.

The current distributed file system basically uses the form of file server and directory server to provide the realization of a unified file system, such as a file array architecture with a file system distributed on a data processing platform disclosed in US Patent No. 6,219,693 (File arraystorage architecture having file system distributed across a dataprocessing platform), when an application requests a file in the distributed file system, the client requests the directory server to find the file, and at the same time, the directory information is cached locally on the client. Then request the file from the file server. Therefore, the client gradually builds an image of directory information, and the directory information is organized in a special naming manner. Usually, the directory information contains information about actual network nodes. At the same time, a factor that cannot be ignored is the coexistence of distributed file systems and local file systems. Any distributed file system needs to achieve real file access through the local file system. Those non-local directory structures are usually accessed through A certain method is attached to the local file system, such as the NFS (Network File System) provided by the famous SUN (Sun Microsystems). This mounting method brings certain constraints to the deployment of applications in distributed systems. When an application is moved from one node to another, the file system may change to a certain extent, so it is necessary to mount the local file system. Certain adjustments must be made directly, and this adjustment is related to the application itself, which obviously violates the principle that the distributed system provides an operating system that has nothing to do with the actual physical distribution for the applications on it. Patent No. 6,078,929 also has the same problem in the Internet file system (Internet filesystem), which uses a unified file location description, but contains physical information of distributed nodes in the description, so once the location of the file changes, the unified The file location information must also change, so that the application that accesses the file must understand this change, display and adjust its access to the file. The file server and directory server are distributed in all nodes, and the file request client is also the server. This structure solves the relationship between file distribution and application to a certain extent, but it does not solve it completely. At the same time, file operations between the remote file system and the local file system, such as file copy, move, etc., also have certain incompatibility. This incompatibility requires applications to participate in management, which also violates the application and distributed The isolation of the file system itself.

The current distributed application is relatively fixed for the application running location, and the system itself is used for system load sharing, which can be used relatively well; however, it needs to be developed for occasions where the application moves frequently. In short, the current distributed file system can well complete the distributed access of files when the location of files and distributed applications is relatively fixed, but if the file location or the location of the distributed application changes, the storage Otherwise, it is difficult to be independent of the operating logic of the application, and the application must intervene in the adjustment of this change.

Contents of the invention

The technical problem to be solved by the present invention is to provide a distributed file access method, which overcomes the shortcomings of the current distributed file system for applications that need to frequently move the running location and adjust the deployment of the file system. When the location is specified, the files in the distributed file system can still be read and written correctly without caring about the actual storage location of the files. At the same time, the method can achieve real-time update of file directory information while accessing files to ensure the consistency of the file system.

The above-mentioned technical problems of the present invention are solved in this way, and a distributed file access method is provided for a distributed system with relatively unified file directory information for applications with a relatively high possibility of mobility. The method adopts real-time distributed directory information management, and the file client and server are peer-to-peer to complete the distributed access of files.

File access mainly includes file opening/creation, file reading/writing, file copying/moving, file deletion, and file closing. This method provides these file access methods in the distributed file system. Accessibility. It should be pointed out that the distributed file system is a part of the distributed processing system. It provides file-related services for the distributed processing system. Some methods in the file processing process are carried out through existing mechanisms in the distributed processing system. For example, communication, remote procedure call, etc., these means do not belong to the category of this method.

This method adopts following steps to realize above-mentioned function:

1. Initial file information construction

The distributed file system needs to maintain a unified file directory information, and this method also needs this information. When the system is initialized, each node in the system constructs local file directory information. The information is stored in a special local file, called the file information library. The system maps the file to the system memory, and the mapped memory The operation is reflected in the file in real time at the same time, realizing the persistence of information.

This file initially stores file and directory information locally participating in the distributed file system. If there is no local file to participate in distributed file access initially, the file content can be empty.

2. File access request reception

This method provides file access interfaces. The application sends file access requests to the processing unit of the distributed file system by calling these interfaces. The processing unit generates a processing subunit for each processing request, and the subunit implements specific file Access the request and return the processing result to the application. The file access interface can be synchronous or asynchronous, depending on the mechanisms provided by the distributed processing system itself, such as remote procedure calls.

3. File information acquisition

After receiving the request, the processing subunit of the file access request determines whether to obtain file information according to the type of the request. If the file information is needed, first search in the local memory image, if there is no such file information locally, then query other processing nodes through the communication mechanism of the distributed processing system, and wait for information feedback. If the requested file information is obtained, proceed to the next step, otherwise return failure to the application.

4. File location

Once the file information is obtained, the specific location of the file can be obtained to prepare for the next operation; the location information of the file includes the relative path of the file relative to the root of the distributed file system, and node information.

5. File operation

According to the type of file access request and the specific location of the file, the operation on the file can be divided into four types, local file operation, remote file operation, local image operation of remote file, and remote image of image file. For local file operation, modify the state of the file in the local file information image, and use the file access method provided by the local file system to operate on the file, and the subsequent operations of the local file are completely localized. For remote file operations, the local processing subunit initiates a file access request to the node where the file is located. At this time, the local access subunit is in the position of the file access initiator. File access requests from native applications are handled in the same way. If the file is on a remote node, but the file must be operated locally, most of the file access in this method needs to be completed in this way. See the description in Table 1 for details. At this time, a local image of the remote file is required. When the image file is changed during the operation, after the file operation is completed, the image file needs to be imaged back to the original file. This operation is called remote image of the image file.

6. File local image

The local image of the file is to map the remote node to the local, and complete the subsequent file operations locally. The local image of the file is also carried out through the operation request, which is initiated by the local operation subunit to the remote operation unit. This request is an internal request in the distributed file system and is not provided to the upper layer application. The request is combined with the type of the local original file access request, and after receiving the file image request, the memory image status of the file information is modified, and the access count of the file is increased. After the operation on the local image file is completed, if the file changes, the local processing subunit needs to synchronize the operated file back to the original node (equivalent to the reverse image of the file).

7. File information synchronization

During the operation of the above-mentioned files, each operation action is recorded in the local file information memory image and persisted in the file information database in real time. Every recording operation needs to be synchronized to the relevant distribution nodes to ensure the consistency of the entire distributed file system. Since the file information is updated and persisted at the same time as the file access operation, the distributed file access provided by this method belongs to the persistence level of the overall file system, and the failure of a single node will not affect the integrity of the entire file system. The damage of the local file system of the node will only affect some files, and it can even be recovered through the imaged files.

Implementing the method provided by the present invention, compared with the file access method used in the existing distributed file system, has the following advantages: the impact on the distributed file system when the application is moved in the distributed system is emphatically considered. There are methods. In this case, it is generally necessary to adjust the organization of the file system, or the application itself for consistency coordination. This method eliminates the adjustment of this file organization structure. No matter how the application moves in the distributed system, the file system of the application itself will not change. This method adopts real-time distributed directory information management, and the file client and server and so on to complete the distributed access of files. Therefore, a file access method that has nothing to do with the actual operating environment is truly provided to the application.

Description of drawings

Fig. 1 is a structural diagram of a distributed file system;

Fig. 2 is a local directory structure diagram of node 1 in the directory structure;

The local directory structure diagram of node 2 in the directory structure of Fig. 3;

Fig. 4 is the virtual directory structural diagram (stand-alone and comprehensive) of distributed system;

Fig. 5 is a file operation category table;

Fig. 6 is a flowchart of the distributed file access method of the present invention.

Detailed ways

Figure 1 shows the overall management structure of the distributed file system. The distributed file system manages files through the control process located on each node of the distributed system. There is a memory image of the structure of the distributed file system on each node. DFS, the memory image holds the overall virtual directory shown in Figure 2. The application process on each node interacts with the DFS control process of the node through the file access interface provided by the distributed file system to complete the file operation. The DFS control process completes the file access, search and image operations of the entire distributed file system through the communication mechanism of the distributed system.

2, 3 and 4 show the file directory structure of the distributed file system of two nodes, each node has a root directory of the distributed file system, this directory can be any directory in the node local directory, in All directories below the root directory can be included in the management of the distributed file system. The local directories of nodes 1 and 2 are shown in Figure 2 and Figure 3, respectively, and the entire distributed file directory is shown in Figure 4. It performs an OR operation on the directories in Figure 2 and Figure 3, and merges the same directories. Add different directories to each other.

FIG. 5 shows the file operation method defined in this paper used by all file access interfaces in the distributed file system. Where the files mentioned in Figure 5 exist refers to the storage location of the original copy of the file. They are described as follows:

1. File creation

Create the file locally if it does not exist in DFS;

If the file exists in DFS and exists locally, the success of the file creation is determined according to the requirements of the file creation and the usage of the current file. If the creation is allowed, the local file is overwritten or opened. If the file exists in the remote place, the file creation permission is judged in the remote place, and if the file is allowed to be created, the file is mapped to the local.

2. Open the file

If the original copy of the file is local, it is a local operation; if the image file is local, it is a local image operation

If the file is in a remote place, the file will be imaged locally before operation.

3. File close

After the original copy of the file is closed locally, if there are other remote images, remote image operations need to be performed;

If the closed image file is a local one, and the content of the file to be closed has changed, remote image of the file is required.

4. File copy

For copying local files, only local file operations are involved;

If you copy a remote file, you need to map the new file to the local after the remote file copy is completed.

5. File movement

Regardless of whether the file is moved locally or remotely, due to the location change of the source file, the file needs to be deleted on all nodes where the file image exists.

6. File deletion

Regardless of whether the file is deleted locally or remotely, after the file is deleted, it needs to be notified in the relevant nodes of DFS for synchronization.

7. File explicit image

The explicit image of the file is to map the remote file to the local. This operation is mainly used when interacting with non-DFS systems.

8. File reading and writing

Since the reading and writing of the file is always performed after the file is opened or created, and the previous operation completes the image of the file (if necessary), the reading and writing of the file is always performed on the local file.

Fig. 6 is a schematic flow diagram of a distributed file access process using the method of the present invention. First, the distributed file processing unit constructs and synchronizes initial file information when starting to form a file information library, and then waits for a file access request state, when a file access request arrives, the processing unit obtains the file information in the file information base, thereby completing the file location, and then determines whether it is necessary to image the file. If necessary, the original file is imaged to the local, and then the actual The file access operation is implemented. When the file access operation is completed, the modified file and file information must be synchronized.

Claims (9)

1, a distributed file access method, is characterized in that, comprises the steps: 1) Construct initial file information in the distributed file system and establish a file information library; 2) Provide a file access interface, and submit a file access request through the access interface; 3) Obtain file information; 4) Determine the specific location of the file according to the obtained file information; 5) perform file access operations according to the type of file access request and the specific location of the file; 6) Locally image the remote node files; 7) During the file access operation, each operation action is recorded in the local file information memory image, and is persisted in the file information database in real time; each recording operation is synchronized to the relevant distribution nodes , to ensure the consistency of the entire distributed file system. 2. The access method according to claim 1, characterized in that said establishing a file information library comprises the following steps: when performing system initialization, constructing local file directory information for each node in the system, and saving the directory information In a local file information base, the system maps the file information base into the system memory, and any operation on the mapped memory is simultaneously reflected in the file information base in real time. 3. The access method according to claim 1, characterized in that said providing a file access interface, and submitting a file access request through the access interface, comprises the following steps: sending the request for file access by calling the access interface In the processing unit of the distributed file system, the processing unit generates a processing subunit for implementing a specific file access request for each file access request, and returns the processing result to the application. 4. The access method according to claim 3, wherein said obtaining file information comprises the following steps: after receiving the request, the processing subunit of the file access request determines whether to obtain file information according to the type of request, If the file information is needed, first search in the local memory image, if there is no such file information in the local memory image, it means that the file information in each distributed node may be inconsistent, then through the communication mechanism of the distributed processing system , query other processing nodes and wait for information feedback; if the requested file information is obtained, proceed to the next step, otherwise return failure to the application. 5. The access method according to claim 1, wherein said obtaining file information comprises obtaining location information including the relative path and/or node information of the file relative to the root of the distributed file system. 6. The access method according to any one of claims 1-5, characterized in that said performing file access operations includes local file operations, remote file operations and local image operations of remote files. 7. The access method according to claim 6, wherein the local file operation is to modify the file state in the local file information image, and use the file access method provided by the local file system to operate the file, Subsequent operations on local files are fully localized; operations on files at remote nodes are also included locally. 8. The access method according to claim 6, characterized in that the remote file operation is initiated by the local processing subunit to the node where the file is located, and at this time the local access subunit is in the process of file access In the status of the originator, the remote processing unit receives the request and processes it the same as it receives the file access request from the local application. 9. The access method according to claim 1, characterized in that said locally mapping the remote node file comprises the following steps: Map the files in the remote node to the local, and complete the subsequent file operations locally; the local image of the file is performed through the operation request, which is initiated by the local operation subunit to the remote operation unit, and the request is distributed The internal request in the file system is not provided to the upper layer application; the request is combined with the type of the local original file access request, after receiving the file image request, modify the memory image state of the file information, and increase the access count of the file; After the operation on the local image file is completed, if the file changes, the local processing subunit will synchronize the operated file back to the original node.

Images