CN110109884A - A kind of file reading, device, equipment and medium - Google Patents

Abstract

The invention discloses a file reading method, device, equipment and medium. The steps of the method include: reading the corresponding target metadata in the MDS according to the file reading requirement, and obtaining the target file recorded in the target metadata The target data size; when the target data size is within the preset threshold or equal to the preset threshold, the target file is read in the MDS; wherein, the files whose data size is within the preset threshold or equal to the preset threshold are pre-stored in the MDS. This method can relatively reduce the occupancy of OSD communication resources by pre-storing files with a data size within a certain threshold range in the MDS, relatively avoiding the delay and blocking of the OSD in the data communication process, and thus ensuring file reading. take the stability of the process. In addition, the present invention also provides a document reading device, equipment and medium, and the beneficial effects are the same as those described above.

Description

A file reading method, device, equipment and medium

technical field

The invention relates to the field of data communication, in particular to a file reading method, device, equipment and medium.

Background technique

With the popularization of computers and the development of Internet technology, data files are growing massively and rapidly. These data files are usually stored in computer storage systems so that users can retrieve and analyze data files.

In order to achieve the overall reliability of file reading, the current distributed file system is usually used to store data files. The specific implementation is to provide the storage space required by the file system through multiple storage nodes. To read data files in a file system, metadata is usually used to record the relevant information of data files stored in storage nodes. Metadata, also known as intermediary data and relay data, is data describing data (data about data), mainly describes the information of data attributes (property), used to support functions such as indicating storage location, historical data, resource search, file recording, etc. In order to better maintain metadata, in current usage scenarios, metadata is often stored separately in MDS (Metadata Server, metadata server), while data files are stored in OSD (Object Storage Device, object storage device) ), because the total data volume of the current data file is often large, it is usually divided and stored in multiple OSDs in advance, and then when the client of the distributed file system accesses a data file, it needs to access the MDS first and then Obtain the metadata of the data file, and then obtain the corresponding data file fragments in multiple OSDs or multiple paths of the OSD according to the metadata, and finally combine them into a complete data file. In this process, distributed The frequent interaction between the client of the file system and the OSD needs to occupy more communication resources. On this basis, when the client of the distributed file system accesses a small amount of data and can be stored in the data file in the OSD without partitioning At the same time, it is still necessary to follow the above operation process, and finally interact with the OSD through the client of the distributed file system to obtain the data file, which will further increase the occupation of OSD communication resources and may cause delays in the data communication process And abnormal conditions such as blocking, it is difficult to ensure the stability of the file reading process.

It can be seen that providing a file reading method to relatively reduce the occupancy of OSD communication resources, relatively avoid delay and blockage in the data communication process, and then ensure the stability of the file reading process is a matter for those skilled in the art. issues that need resolving.

Contents of the invention

The purpose of the present invention is to provide a file reading method, device, equipment and medium to relatively reduce the occupation of OSD communication resources, relatively avoid delay and blockage in the data communication process, and then ensure the stability of the file reading process sex.

In order to solve the above technical problems, the present invention provides a file reading method, comprising:

Read the corresponding target metadata in the MDS according to the file reading requirements, and obtain the target data size of the target file recorded in the target metadata;

When the target data size is within or equal to the preset threshold, the target file is read in the MDS; wherein, the files whose data size is within the preset threshold or equal to the preset threshold are pre-stored in the MDS.

Preferably, the files whose data size is within the preset threshold are stored in the MDS in advance, including:

Files whose data size is within the preset threshold are pre-stored in MDS as extended attributes of metadata.

Preferably, the method further comprises:

When the target data size of the target file is greater than a preset threshold after the additional data writing operation is performed on the target file, the target file is read from the extended attribute of the target metadata, and the target file is stored in the OSD.

Preferably, the method further comprises:

When the target data size is greater than the preset threshold, the target file is read in the OSD according to the target metadata.

In addition, the present invention also provides a document reading device, including:

The metadata reading module is used to read the corresponding target metadata in the MDS according to the file reading requirements, and obtain the target data size of the target file recorded in the target metadata;

The small file reading module is used to read the target file in the MDS when the target data size is within or equal to the preset threshold; wherein, the files whose data size is within the preset threshold are pre-stored in the MDS.

Preferably, the device further comprises:

The large file reading module is used to read the target file in the OSD according to the target metadata when the target data size is greater than a preset threshold.

In addition, the present invention also provides a file reading device, including:

memory for storing computer programs;

The processor is configured to realize the steps of the above-mentioned file reading method when executing the computer program.

In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned file reading method are realized.

In the file reading method provided by the present invention, the files whose data size is within or equal to the preset threshold are pre-stored in the MDS. The target metadata, and then obtain the target data size of the target file recorded in the target metadata, and when the target data size is within or equal to the preset threshold, directly read the target file in the MDS. It can be seen that this method pre-stores the files whose data size is within a certain threshold range in the MDS, and then when the user obtains the target file whose data size is within the threshold range, there is no need to interact with the OSD, so it can relatively reduce The occupancy of OSD communication resources relatively avoids the delay and blocking of OSD in the data communication process, thereby ensuring the stability of the file reading process. In addition, the present invention also provides a document reading device, equipment and medium, and the beneficial effects are the same as those described above.

Description of drawings

In order to illustrate the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. As far as people are concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

FIG. 1 is a flowchart of a file reading method provided by an embodiment of the present invention;

Fig. 2 is a structural diagram of a document reading device provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The core of the present invention is to provide a file reading method to relatively reduce the occupancy of OSD communication resources, relatively avoid delay and blockage in the data communication process, and ensure the stability of the file reading process. Another core of the present invention is to provide a file reading device, device and medium.

In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Embodiment one

FIG. 1 is a flowchart of a file reading method provided by an embodiment of the present invention. Please refer to Figure 1, the specific steps of the file reading method include:

Step S10: Read the corresponding target metadata in the MDS according to the file reading requirement, and obtain the target data size of the target file recorded in the target metadata.

It should be noted that the execution subject of this method can be a client of a distributed file system other than MDS and OSD in the distributed file system, the client is controlled by the user, and interacts with the MDS and OSD according to the user's needs and Get the corresponding file. This step is to read the target metadata corresponding to the file required by the user in the MDS used to store metadata according to the file reading requirements of the user, and then obtain the relevant attribute information of the file required by the user in the target metadata, namely The target data size of the target file, where the target data size refers to the data space occupied by the target file in the OSD device.

Step S11: When the target data size is within or equal to the preset threshold, read the target file in the MDS.

Wherein, the files whose data size is within or equal to the preset threshold are pre-stored in the MDS.

In this step, when the obtained target data size is within or equal to the preset threshold, it means that the corresponding file with the target data size conforms to the preset target file type, and the target file of this type is stored in advance in the MDS. In this step, when the target data size is within or equal to the preset threshold, the target file is obtained in the MDS, so the target file pre-stored in the MDS is a file with a relatively small data size in the distributed file system. A file with a relatively small data size is stored in the MDS, which can be directly obtained in the MDS when the user accesses the file, without requiring the client of the distributed file system to interact with the OSD device. The preset threshold in this step should be determined according to the actual usage scenario of the distributed file system, such as the data capacity of the MDS device and the amount of metadata in the MDS device, etc., and it should be ensured that the MDS can normally provide metadata to the client.

In the file reading method provided by the present invention, the files whose data size is within or equal to the preset threshold are pre-stored in the MDS. The target metadata, and then obtain the target data size of the target file recorded in the target metadata, and when the target data size is within or equal to the preset threshold, directly read the target file in the MDS. It can be seen that this method pre-stores the files whose data size is within a certain threshold range in the MDS, and then when the user obtains the target file whose data size is within the threshold range, there is no need to interact with the OSD, so it can relatively reduce The occupancy of OSD communication resources relatively avoids the delay and blocking of OSD in the data communication process, thereby ensuring the stability of the file reading process.

Embodiment two

On the basis of the above embodiments, the present invention also provides the following series of preferred implementation manners.

As a preferred implementation, pre-storing files whose data size is within the preset threshold in the MDS includes:

Files whose data size is within the preset threshold are pre-stored in MDS as extended attributes of metadata.

It should be noted that, considering that the data structure of the metadata is pre-set with extended attributes for adding data items, in this embodiment, files are pre-stored in the extended attributes of the metadata to achieve The purpose of storing files in MDS is to relatively ensure the reliability of files stored in MDS because the data structure of metadata has the stability of data storage.

On the basis of the foregoing embodiments, as a preferred embodiment, the method further includes:

When the target data size of the target file is greater than a preset threshold after the additional data writing operation is performed on the target file, the target file is read from the extended attribute of the target metadata, and the target file is stored in the OSD.

It should be noted that this embodiment considers that when additional data is written to the target file, the data size of the target file will continue to increase, and there may be a situation where the target data size of the target file exceeds a preset threshold. In this case, in order to ensure that the storage of the target file does not cause excessive occupation of the MDS space, in this embodiment, when the target data size of the target file is greater than the preset threshold after the additional data writing operation is performed on the target file, the target metadata Read the target file from the extended attribute of the target file, and then store the target file in the OSD, so as to avoid the excessive occupation of the MDS space caused by the continuous increase of the data volume of the target file, and ensure that the metadata can be stored in the MDS normally , thus ensuring that the MDS can normally provide metadata services to the client.

On the basis of the series of implementations above, as a preferred implementation, the method further includes:

When the target data size is greater than the preset threshold, the target file is read in the OSD according to the target metadata.

It should be noted that, in this embodiment, the target file whose target data size is greater than the preset threshold is directly stored in the OSD in advance, and then when the target data size is greater than the preset threshold, the target file is read from the OSD according to the target metadata. file, which relatively avoids excessive occupation of MDS space caused by storing target files with relatively large target data size in MDS, thereby ensuring that metadata can be stored in MDS normally, and ensuring that MDS can normally provide metadata to clients. data service.

Embodiment Three

The embodiment of the file reading method has been described in detail above, and the present invention also provides a file reading device corresponding to the method. Since the embodiment of the device part corresponds to the embodiment of the method part, the device For some embodiments, please refer to the description of the embodiments in the method part, which will not be repeated here.

Fig. 2 is a structural diagram of a document reading device provided by an embodiment of the present invention. The file reading device provided by the embodiment of the present invention includes:

The metadata reading module 10 is configured to read the corresponding target metadata in the MDS according to the file reading requirement, and obtain the target data size of the target file recorded in the target metadata.

The small file reading module 11 is used to read the target file in the MDS when the target data size is within or equal to the preset threshold; wherein, the files whose data size is within the preset threshold are pre-stored in the MDS.

The file reading device provided by the present invention pre-stores files whose data size is within or equal to the preset threshold in the MDS, and first reads the corresponding The target metadata, and then obtain the target data size of the target file recorded in the target metadata, and when the target data size is within or equal to the preset threshold, directly read the target file in the MDS. It can be seen that this device pre-stores the files whose data size is within a certain threshold range in the MDS, and then when the user obtains the target file whose data size is within the threshold range, there is no need to interact with the OSD, so it can relatively reduce The occupancy of OSD communication resources relatively avoids the delay and blocking of OSD in the data communication process, thereby ensuring the stability of the file reading process.

On the basis of embodiment three, the device also includes:

The large file reading module is used to read the target file in the OSD according to the target metadata when the target data size is greater than a preset threshold.

Embodiment Four

In addition, the present invention also provides a file reading device, including:

memory for storing computer programs;

The processor is configured to realize the steps of the above-mentioned file reading method when executing the computer program.

The file reading device provided by the present invention pre-stores files whose data size is within or equal to the preset threshold in the MDS, and first reads the corresponding The target metadata, and then obtain the target data size of the target file recorded in the target metadata, and when the target data size is within or equal to the preset threshold, directly read the target file in the MDS. It can be seen that this device pre-stores the files whose data size is within a certain threshold range in the MDS, and then when the user obtains the target file whose data size is within the threshold range, there is no need to interact with the OSD, so it can relatively reduce The occupancy of OSD communication resources relatively avoids the delay and blocking of OSD in the data communication process, thereby ensuring the stability of the file reading process.

In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned file reading method are implemented.

The computer-readable storage medium provided by the present invention pre-stores files whose data size is within or equal to the preset threshold in the MDS, and first reads the corresponding files in the MDS according to the file reading requirements when executing the file reading operation. The target metadata, and then obtain the target data size of the target file recorded in the target metadata, and when the target data size is within or equal to the preset threshold, the target file is directly read in the MDS. It can be seen that the computer-readable storage medium pre-stores files with a data size within a certain threshold range in the MDS in advance, and then when the user obtains a target file with a data size within the threshold range, there is no need to interact with the OSD, so It can relatively reduce the occupancy of OSD communication resources, relatively avoid the delay and blocking of OSD in the data communication process, and thus ensure the stability of the file reading process.

The file reading method, device, equipment and medium provided by the present invention have been introduced in detail above. Each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

It should also be noted that in this specification, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations There is no such actual relationship or order between the operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Claims (8)

1. a kind of file reading characterized by comprising

Demand is read according to file and reads corresponding target metadata in MDS, and is obtained and be recorded in the target metadata File destination target data size；

When the target data size is in preset threshold or is equal to the preset threshold, the mesh is read in the MDS Mark file；Wherein, size of data is pre-stored within described in the preset threshold or equal to the file of the preset threshold MDS。

2. the method according to claim 1, wherein file of the size of data in the preset threshold is pre- Being first stored in the MDS includes:

File of the size of data in the preset threshold is used as the extended attribute of metadata to be stored in the MDS in advance.

3. according to the method described in claim 2, it is characterized in that, this method further comprises:

The target data size of the file destination is caused to be greater than institute after carrying out additional data writing operation to the file destination When stating preset threshold, the file destination is read in the extended attribute of the target metadata, and the file destination is deposited It stores up to OSD.

4. according to claim 1 to method described in 3, which is characterized in that this method further comprises:

When the target data size is greater than the preset threshold, the mesh is read in OSD according to the target metadata Mark file.

5. a kind of document reading apparatus characterized by comprising

Metadata read module reads corresponding target metadata for reading demand according to file in MDS, and obtains record The target data size of file destination in the target metadata；

Small documents read module is used for when the target data size in preset threshold or when being equal to the preset threshold, The file destination is read in the MDS；Wherein, file of the size of data in the preset threshold is pre-stored within described MDS。

6. device according to claim 5, which is characterized in that the device further comprises:

Big file read module is used for when the target data size is greater than the preset threshold, according to the target element number According to reading the file destination in OSD.

7. a kind of file reads equipment characterized by comprising

Memory, for storing computer program；

Processor realizes such as Claims 1-4 described in any item file readings when for executing the computer program The step of.

8. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium Program is realized when the computer program is executed by processor such as the described in any item file readings of Claims 1-4 Step.