CN110263005B - Storage system management system for realizing data content locality read-write optimization - Google Patents

Abstract

The invention relates to a storage system management system for realizing data content locality read-write optimization, which is characterized by comprising the following steps: on the basis of the traditional data space-time locality, the concept of content locality is provided, a design scheme of a file system middle layer capable of clustering and storing labeled data is provided, the data analysis module is in butt joint with a user to capture data label information, the back-end interface module is in butt joint with a bottom layer parallel or distributed storage system API (application program interface), clustering and storing according to labels are achieved, and therefore a subsequent data access mode meeting the content locality is converted into space locality. The invention comprises the following steps: 1) The performance of tagged data access is improved significantly. 2) The sequentiality of tagged data access is optimized. 3) The original structured data format is not destroyed, and the correctness of the data is ensured.

Description

A storage system management system that realizes local read and write optimization of data content

technical field

The invention relates to a storage management scheme for large-scale structured data, in particular to a storage system management system for realizing localized reading and writing optimization of data content.

Background technique

With the advent of the era of big data, the front-end information collection and calculation processing capabilities of the computer system have been greatly improved, and larger data files can be generated in a shorter period of time, which has brought enormous pressure to the rear storage system, making the file System input/output (Input/Output, I/O) has become one of the important bottlenecks limiting the performance of computer systems.

One of the ways for traditional file storage systems to improve performance is to exploit the locality of data access. Data locality can be divided into the following two types:

1. Temporal Locality: During a period of time, the application often accesses the data at the same location of the file multiple times in a row, such as reading and writing a certain calculation result multiple times;

2. Spatial Locality (Spatial Locality): After accessing the data at a certain location in the file, the application will often continue to access related data near the storage space at that location, such as sequentially reading a string of arrays;

Through the disk-to-memory data caching mechanism, after the user application accesses a certain location of the file, it prefetches a piece of nearby data into the memory, which can significantly reduce the number of times that file access requests actually reach the disk, thereby improving the performance of interaction with the storage system.

In scientific computing applications, different applications have their specific structured data file formats, such as the Bag file (.bag) used in the robot operating system ROS and the Xtc file (.xtc) used in the biomolecular dynamics visualization software VMD, etc. . The data file contains a large number of data entries, which can be clustered according to certain characteristics, for example, according to data sources (such as infrared sensors, position sensors, etc.), or according to data attributes (such as water molecules, protein molecules, etc.); The class feature is called the "label" of the data item. It can be foreseen that when extracting and analyzing data, users need to access and process certain types of data related to specific labels within the same period of time. At this time, if the data items are clustered and stored according to the labels, then their spatio-temporal locality can play a role.

However, as the amount of data collected by the front-end continues to expand, they often use a log-type writing method to aggregate data with multiple tags into data files in pure time order. Limited by this storage optimization method for the writing process, in most current structured data file formats, data entries of different label categories are often not clustered together, but complexly intertwined in time order, so that The user's subsequent data access no longer satisfies the characteristics of spatio-temporal locality.

In this case, the traditional file system based on the assumption of temporal and spatial locality not only fails to improve performance through the caching mechanism, but also introduces a large amount of useless data prefetching, which seriously drags down the performance of the storage link. In addition, such frequent, small-volume, and random data access patterns are unfriendly to existing storage hardware (especially mechanical hard disks), resulting in high I/O latency, small bandwidth, and frequent head addressing. Bad and other disadvantages.

Contents of the invention

The purpose of the present invention is to provide a management system of a storage system based on content locality.

In order to achieve the above purpose, the technical solution of the present invention is to provide a storage system management system that realizes localized read and write optimization of data content. The unique data access mode is transformed into spatial locality, and content locality means that after the user application accesses the data of a certain label content, there is a tendency to continue to access the data of the same or related content. It is characterized in that the management system includes The data analysis module connected with the user program, the back-end interface module connected with the underlying parallel storage system, and the middle layer of the virtual file system transparent to the underlying storage system and translucent to the user application program;

The data analysis module is used to capture the user's request to write the tagged data file to the file system, then parse the data records in the tagged data file, sequentially extract the tag category Topic of each data record to be written, and obtain the data in the original Offset in the tagged data file;

The back-end interface module realizes the interface call to the back-end parallel or distributed storage system, and is used to actually write the data processed by the middle layer of the virtual file system into the storage hardware in a way that conforms to the selected back-end file system;

The middle layer of the virtual file system is implemented in the form of a user state library. It is connected to the user program on the upper side, and the default POSIX read and write system call function is packaged, and it is connected to the back-end storage system on the lower side; the user's write request All are captured by the data analysis module, and the extracted data records and their label categories Topic and offset are handed over to the middle layer of the virtual file system for processing, and the middle layer of the virtual file system clusters and stores the data of the same label category Topic into the same sub-data file, and at the same time, the offset of each record in the original tagged data file and its offset in the sub-data file are one-to-one, and stored in the accompanying sub-index file; the user for a specific tag The read request of the data of the category Topic is implemented through the function interface provided by the middle layer of the virtual file system, and directly reads the corresponding sub-data file in the backend.

Preferably, the data analysis module regards metadata that is not clearly classified in the tagged data file as a special tag category Topic.

Preferably, the middle layer of the virtual file system includes a set of scalability solutions for the middle layer of the file system, by unifying the interface of the data analysis module to the middle layer of the virtual file system, and the middle layer of the virtual file system The standard of the interface of the back-end interface module makes it only necessary to provide the corresponding data analysis module and the back-end interface module for different user applications and different back-end storage systems, which can be plugged The corresponding data analysis module and the back-end interface module are linked to realize compatibility and expansion.

Preferably, for the user's normal read request and file traversal operation to the tagged data file, the index table stored in the sub-index file is used for translation and merging operations, which can provide the user with an illusion that the file is still in accordance with the original The structure is preserved so as not to affect the original storage structure logic that may be required by the algorithm of the user program.

The present invention proposes the idea of "content locality" on the basis of the traditional space-time locality. The so-called content locality of data access means that after a user application accesses the data of a certain label content, there is a tendency to continue to access the data of the same or related content. Content locality is a data access mode that appears in structured and labeled data in today's big data environment, and it is an extension of traditional spatiotemporal locality.

The present invention has following beneficial effect:

1) Significantly improved the performance of tagged data access. Through the reorganization of the data storage structure, the content locality of user data access is transformed into spatial locality, and the contradiction between large-scale tagged data reading and writing and the traditional space-time locality caching mechanism is reconciled. Taking the data message Bag file in the robot operating system ROS as an example, this solution improves the file opening efficiency to 5 times that of the original API; in the file reading operation, the reading of the relevant message records is guided to the reading of the entire piece of data, Make the traditional data caching mechanism work, and the read latency is reduced by 15%.

2) Optimize the sequence of tagged data access. For the data reading and collection of related tags, the original interleaved and random access mode is changed to a sequential and block access mode, so as to maximize the performance of physical storage hardware, especially HDD mechanical hard disks , and reduce the chance of failure.

3) Ensure the correctness of the data. The present invention only classifies and organizes data records, and implements it in the form of a user-mode middle layer. On the one hand, it does not intervene in the workflow of the mature parallel or distributed storage system used by the bottom layer, and the back-end storage still exerts its parallelization Read and write, consistent redundancy, fault recovery and other functions ensure the correctness of the data; on the other hand, it does not break the requirements of the user program for the data format, and can still provide the user with the original format of the file.

Description of drawings

Fig. 1 is a structural flow chart of the present invention.

Detailed ways

Below in conjunction with specific embodiment, further illustrate the present invention. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

The present invention provides a design scheme for the middle layer of the file system for clustering and storing labelable data, and transforms the data access mode satisfying content locality into spatial locality. Specifically, a storage system management system that realizes localized read and write optimization of data content provided by the present invention includes:

A data analysis module that interfaces with user programs. The data analysis module can capture the user's request to write the tagged data file to the file system, then parse the data records in the tagged data file, and sequentially extract the tag category of each data record to be written, which is called Topic. At the same time, the offset of the data in the original tagged data file is also obtained. Since not all data in the original tagged data file is real content data, in particular, metadata that is not clearly classified, such as the version number of the data file, etc., are uniformly treated as a special category of Topic .

A back-end interface module that interfaces with the underlying parallel storage system. The back-end interface module realizes the interface call to the back-end parallel or distributed storage system, and is used to actually write the data processed by the middle layer of the virtual file system into the storage hardware in a way that conforms to the selected back-end file system.

The middle layer of the virtual file system that is transparent to the underlying storage system and semi-transparent to user applications. The middle layer of the virtual file system is implemented in the form of a user state library. It connects with the user program on the top, wraps the default POSIX read and write system call functions, and connects with the back-end storage system on the bottom. The user's write request is captured by the above-mentioned data analysis module, and the extracted data record and its label category and original offset are handed over to the middle layer of the virtual file system for processing, and then the middle layer of the virtual file system transfers the data of the same label category Clustering is stored in the same sub-data file, and at the same time, the offset of each record in the original tagged data file and its offset in the sub-data file are one-to-one, and stored in the accompanying sub-index file middle. The user's request for reading the data of a specific tag is realized through the function interface provided by the middle layer, and directly reads the corresponding sub-data file in the backend.

The present invention also designs the extensibility scheme of the middle layer of the above-mentioned file system. By unifying the interface standards of the data analysis module to the middle layer and the interface of the middle layer to the back-end module, it is possible to pluggably link the corresponding analysis module and the back-end for different user applications and different back-end storage systems Module, with strong compatibility and scalability;

In addition, for the user's normal reading request and file traversal operation to the tagged data file, the present invention performs translation and merging operations through the saved index table, which can provide the user with an illusion that the file is still saved according to the original structure , so that the original storage structure logic that may be required by the algorithm of the user program is not affected, and the correctness of this storage optimization scheme is guaranteed.

Referring to Fig. 1, the following example illustrates the present invention with the implementation of the user application as ROS Bag API, the back-end storage as Ceph, and the middle layer using the PLFS framework:

1. Mounting of the middle layer virtual file system

1. Edit the .plfsrc configuration file according to the syntax definition of PLFS;

2. Define the backend storage in the configuration file as Ceph parallel storage system;

3. Use the plfs command to mount the middle layer to the specified directory;

2. Write tagged data

1. The user writes a .bag format data file to the mount directory;

2. The data analysis module captures the written data and calls the analyzer;

3. The analyzer parses the data written in this article according to the Bag format;

4. The parsed data message is added to the data file of the corresponding tag;

5. The index mapping of each piece of data is recorded in the index file of the corresponding label;

6. Metadata other than the data itself are uniformly recorded under a special label;

7. Call the write interface of the backend Ceph for actual writing;

8. The write operation is completed;

3. Open the data file

1. The user calls the open interface defined by the middle layer;

2. The intermediate layer skips the original full scan file and index building steps in the ROS API, and directly loads the index file to obtain the index mapping;

3. After opening, return;

4. Read specific tag data

1. The user calls the read interface defined by the middle layer and passes in the tag to be read;

2. The middle layer responds to the read operation and directly locates the data file corresponding to the label on the backend;

3. Call the read interface of the backend Ceph for actual reading, and only read the data file;

4. After the read operation is completed, the data under the label of the user is returned;

5. Traditional POSIX reading support

1. The user calls the traditional POSIX read operation interface;

2. The middle layer captures this read operation, and locates the tag where the read content is located according to the mapping relationship of the index;

3. Obtain the data in the data file corresponding to the label;

4. After the read operation is completed, the data read by the user is returned.

Claims (4)

1. A storage system management system that realizes localized reading and writing optimization of data content. It is located in the middle layer of the file system that clusters and stores labelable data. It is used to transform the data access mode that satisfies content locality into spatial locality. Content locality means that after the user application accesses the data of a certain label content, there is a tendency to continue to access the data of the same or related content. It is characterized in that the management system includes a data analysis module connected with the user program, and the The back-end interface module connected to the parallel storage system and the middle layer of the virtual file system that is transparent to the underlying storage system and semi-transparent to user applications; The data analysis module is used to capture the user's request to write the tagged data file to the file system, then parse the data records in the tagged data file, sequentially extract the tag category Topic of each data record to be written, and obtain the data in the original Offset in the tagged data file; The back-end interface module realizes the interface call to the back-end parallel or distributed storage system, and is used to actually write the data processed by the middle layer of the virtual file system into the storage hardware in a way that conforms to the selected back-end file system; The middle layer of the virtual file system is implemented in the form of a user state library. It is connected to the user program on the upper side, and the default POSIX read and write system call function is packaged, and it is connected to the back-end storage system on the lower side; the user's write request All are captured by the data analysis module, and the extracted data records and their label categories Topic and offset are handed over to the middle layer of the virtual file system for processing, and the middle layer of the virtual file system clusters and stores the data of the same label category Topic into the same sub-data file, and at the same time, the offset of each record in the original tagged data file and its offset in the sub-data file are one-to-one, and stored in the accompanying sub-index file; the user for a specific tag The read request of the data of category Topic is implemented through the function interface provided by the middle layer of the virtual file system, and directly reads the corresponding sub-data file in the backend. 2. A storage system management system that realizes localized reading and writing optimization of data content as claimed in claim 1, wherein the data analysis module regards metadata that is not clearly classified in the tagged data file as A special type of label category Topic. 3. a kind of storage system management system that realizes data content partial reading and writing optimization as claimed in claim 1, is characterized in that, described virtual file system middle layer comprises the extensibility scheme of a set of file system middle layer, By unifying the interface of the data analysis module to the middle layer of the virtual file system and the standard of the interface of the middle layer of the virtual file system to the back-end interface module, it is possible to target different user applications and different back-ends The storage system only needs to provide the corresponding data analysis module and the back-end interface module, and then pluggably link the corresponding data analysis module and the back-end interface module to achieve compatibility and expansion. 4. A storage system management system that realizes localized read-write optimization of data content as claimed in claim 1, wherein, for a user's normal read request and file traversal operation to a tagged data file, the sub-index file Translating and merging the index table saved in the file can provide the user with an illusion that the file is still saved according to the original structure, thus not affecting the original storage structure logic that may be required by the algorithm of the user program.

Images