CN106873906A - Method and apparatus for managing metamessage - Google Patents

Abstract

It is an object of the invention to provide a kind of method and apparatus for managing metamessage.The method according to the invention is comprised the following steps：Its corresponding metamessage is stored or updated in each data block of goal systems.Compared with prior art, the present invention has advantages below：It is stored in data block by by metamessage, can avoids causing the loss of mass data due to file corruption, improves the security of data storage；Also, scheme of the invention can create multiple threads and load metamessage in a parallel fashion, so as to improve system toggle speed.

Description

Method and device for managing meta information

technical field

The present invention relates to the field of computer technology, in particular to a method and device for managing meta information.

Background technique

Data storage systems based on the prior art generally store metadata persistently in media such as disks. This method has the following defects: 1) It takes a long time to load and restore the meta information when the system starts. The reading of meta-information files and the deserialization of information into the specified memory data structure will consume quite a long time, thus affecting the startup speed of the system; Disk bad blocks and other problems are damaged, and once the file is damaged, it will cause a large amount of data loss on the entire node.

Contents of the invention

An object of the present invention is to provide a method and apparatus for managing meta information.

According to one aspect of the present invention, a method for managing meta information is provided, wherein the method includes the following steps:

-Store or update its corresponding meta-information in each data block of the target system.

According to one aspect of the present invention, an information management device for managing meta information is provided, wherein the information management device includes:

Means for storing or updating the corresponding meta-information in each data block of the target system.

Compared with the prior art, the present invention has the following advantages: by storing the meta information in the data block, the loss of a large amount of data due to file damage can be avoided, and the security of data storage is improved; and, according to the scheme of the present invention Multiple threads can be created to load meta information in parallel, thus improving the system startup speed.

Description of drawings

Other characteristics, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 schematically shows a flow chart of a method for managing meta-information according to the present invention;

Fig. 2 shows a schematic structural diagram of an information management device for managing meta-information according to the present invention.

The same or similar reference numerals in the drawings represent the same or similar components.

detailed description

The present invention will be described in further detail below in conjunction with the accompanying drawings.

Fig. 1 schematically shows a flowchart of a method for managing meta-information according to the present invention. The method according to the invention comprises a step S1.

Wherein, the method according to the present invention is implemented by an information management device included in the computer equipment. The computer equipment includes an electronic equipment that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but not limited to microprocessors, application-specific integrated circuits (ASICs), programmable gates Arrays (FPGA), digital processors (DSP), embedded devices, etc. The computer equipment includes network equipment and/or user equipment. Wherein, the network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud based on cloud computing (Cloud Computing) composed of a large number of hosts or network servers, wherein cloud computing is distributed computing A super virtual computer consisting of a group of loosely coupled computer sets. The user equipment includes but is not limited to any electronic product that can interact with the user through a keyboard, mouse, remote control, touch pad, or voice-activated device, such as a personal computer, tablet computer, smart phone, PDA , game console, or IPTV, etc. Wherein, the network where the user equipment and the network equipment are located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, and the like.

Preferably, the device where the information management device according to the present invention is located is included in the target system. Wherein, the target system includes various data storage systems.

It should be noted that the user equipment, network equipment, and network described above are only examples, and other existing or future user equipment, network equipment, and networks that are applicable to the present invention should also be included within the protection scope of the present invention , and is incorporated herein by reference.

Referring to FIG. 1 , in step S1 , the information management device stores or updates the corresponding meta information in each data block of the target system.

Preferably, the way the information management device stores the meta information includes but not limited to at least any of the following:

1) storing the meta information in the underlying file system; preferably, such meta information includes the length of the data block, identification information of the device where it is located, etc.;

2) storing the meta information in the data block name; preferably, this type of meta information includes identification information of the data block, compression method information of the data block, etc.;

3) Store the meta information in the data header of the data block; preferably, this type of meta information includes meta information that does not need to be constructed immediately when the system is started, mainly including file compression block index information and the like.

According to a preferred embodiment of the present invention, based on the service type information of the data to be stored, the information management device stores the data block corresponding to the data to be stored and its meta information in a specific storage medium, so as to separately store the data in each storage medium meta information for management.

Wherein, the storage medium includes various types of media media, for example, Serial Advanced Technology Attachment (Serial Advanced Technology Attachment, SATA), Solid State Disk (Solid State Disk, SSD), memory file, archive disk, cloud disk and so on.

Preferably, the information management device manages the meta-information of the data blocks in each storage medium respectively.

More preferably, for the same storage medium, the information management apparatus can respectively manage the metadata of the data blocks stored in different devices.

According to the method of this preferred embodiment, different storage media can be selected for storage based on the service type of the data, and the meta information in different storage media can be managed, thereby reducing the competition between different services and avoiding low-priority services. High-priority services have an impact on performance; and the meta-information of different devices can be managed independently, thereby reducing the competition between devices.

Preferably, the method according to the present invention includes step S2 (not shown in the figure).

In step S2, when the metadata information of the system needs to be loaded, the information management device creates multiple threads to respectively load the metadata information of the data blocks in the multiple disks.

According to a preferred embodiment of the present invention, in the startup phase of the target system, the method includes step S3 (not shown in the figure), and the step S2 includes step S201 (not shown in the figure).

In step S3, the information management device determines the disk information to be filtered out according to the blacklist information of the target system, so as to obtain one or more available disk information.

Preferably, the disk information to be filtered out includes broken disks, or storage media that cannot be used due to other reasons.

In step S201, the information management device creates multiple threads to respectively load metadata information of data blocks in each disk based on the available information of one or more disks.

Preferably, the method according to this preferred embodiment includes step S4 (not shown).

In step S4, the information management device loads the data block information with low compression ratio of each disk, so that each data block with low compression ratio is skipped when the target system performs transparent compression.

Wherein, the data block information of the low compression rate is directly skipped during transparent compression. Among them, transparent compression is a scheduled task that periodically compresses non-compressed data into specified types of data in the background, and periodically compresses non-compressed data into specified types of data in the background. Timed work tasks.

Preferably, the method according to this preferred embodiment includes step S5 (not shown).

In step S5, the information management device deduplicates the data blocks loaded on each disk according to the scanned usage information and status information of each disk.

According to the method of the present invention, by storing the meta information in the data block, the loss of a large amount of data due to file damage can be avoided, and the security of data storage is improved; moreover, according to the scheme of the present invention, multiple threads can be created to parallelize The way to load meta information, thus improving the system startup speed.

Fig. 2 shows a schematic structural diagram of an information management device for managing meta-information according to the present invention. The information management device includes a device for storing or updating corresponding meta information in each data block of the target system (hereinafter referred to as "storage device 1").

The storage device 1 stores or updates corresponding meta information in each data block of the target system.

Preferably, the storage device 1 stores meta information in a manner including but not limited to at least any of the following:

1) storing the meta information in the underlying file system; preferably, such meta information includes the length of the data block, identification information of the device where it is located, etc.;

2) storing the meta information in the data block name; preferably, this type of meta information includes identification information of the data block, compression method information of the data block, etc.;

3) Store the meta information in the data header of the data block; preferably, this type of meta information includes meta information that does not need to be constructed immediately when the system is started, mainly including file compression block index information and the like.

According to a preferred embodiment of the present invention, based on the service type information of the data to be stored, the storage device 1 stores the data block corresponding to the data to be stored and its meta-information in a specific storage medium, so as to separately store data in each storage medium meta information for management.

Wherein, the storage medium includes various types of media media, for example, Serial Advanced Technology Attachment (Serial Advanced Technology Attachment, SATA), Solid State Disk (Solid State Disk, SSD), memory file, archive disk, cloud disk and so on.

Preferably, the information management device manages the meta-information of the data blocks in each storage medium respectively.

More preferably, for the same storage medium, the information management apparatus can respectively manage the metadata of the data blocks stored in different devices.

According to the scheme of this preferred embodiment, different storage media can be selected for storage based on the service type of data, and the meta information in different storage media can be managed, thereby reducing the competition between different services and avoiding low priority services High-priority services have an impact on performance; and the meta-information of different devices can be managed independently, thereby reducing the competition between devices.

Preferably, the information management device according to the present invention includes a plurality of threads for respectively loading metadata information of data blocks in multiple disks when the metadata information of the system needs to be loaded (not shown in the figure, hereinafter referred to as "loading device") ").

When the metadata information of the system needs to be loaded, the loading device creates multiple threads to respectively load the metadata information of the data blocks in the multiple disks.

According to a preferred embodiment of the present invention, at the start-up stage of the target system, the information management device includes a device for determining the disk information that needs to be filtered out according to the blacklist information of the target system, so as to obtain one or more usable disks Information device (not shown in the figure, hereinafter referred to as "determining device").

The determining means determines the disk information to be filtered out according to the blacklist information of the target system, so as to obtain one or more available disk information.

Preferably, the disk information to be filtered out includes broken disks, or storage media that cannot be used due to other reasons.

The loading device creates multiple threads to respectively load the metadata information of the data blocks in each disk based on the available information of one or more disks.

Preferably, the information management device according to this preferred embodiment includes a device for loading data block information with a low compression ratio of each disk, so that when the target system performs transparent compression, each data block with a low compression ratio is skipped (not shown in the figure) hereinafter referred to as "data loading device").

The data loading device loads the data block information with low compression ratio of each disk, so as to skip each data block with low compression ratio when the target system performs transparent compression.

Wherein, the data block information of the low compression rate is directly skipped during transparent compression. Among them, transparent compression is a scheduled task that periodically compresses non-compressed data into specified types of data in the background, and periodically compresses non-compressed data into specified types of data in the background. Timed work tasks.

Preferably, the information management device according to this preferred embodiment includes a device for performing deduplication processing on the data blocks loaded on each disk according to the scanned usage information and status information of each disk (not shown in the figure, hereinafter referred to as "deduplication") reprocessing device").

The deduplication processing device performs deduplication processing on the data blocks loaded on each disk according to the scanned usage information and status information of each disk.

According to the scheme of the present invention, by storing the meta information in the data block, the loss of a large amount of data due to file damage can be avoided, and the security of data storage is improved; moreover, according to the scheme of the present invention, multiple threads can be created to parallelize The way to load meta information, thus improving the system startup speed.

The software program of the present invention can be executed by a processor to realize the steps or functions described above. Likewise, the software program (including associated data structures) of the present invention can be stored in a computer-readable recording medium such as RAM memory, magnetic or optical drive or floppy disk and the like. In addition, some steps or functions of the present invention may be implemented by hardware, for example, as a circuit that cooperates with a processor to execute each function or step.

In addition, a part of the present invention can be applied as a computer program product, such as a computer program instruction. When it is executed by a computer, the method and/or technical solution according to the present invention can be invoked or provided through the operation of the computer. The program instructions for invoking the method of the present invention may be stored in a fixed or removable recording medium, and/or transmitted through broadcasting or data streams in other signal-carrying media, and/or stored in the in the working memory of the computer device on which the program instructions execute. Here, an embodiment according to the present invention comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, a trigger The operation of the device is based on the foregoing methods and/or technical solutions according to multiple embodiments of the present invention.

It will be apparent to those skilled in the art that the invention is not limited to the details of the above-described exemplary embodiments, but that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Accordingly, the embodiments should be regarded in all points of view as exemplary and not restrictive, the scope of the invention being defined by the appended claims rather than the foregoing description, and it is therefore intended that the scope of the invention be defined by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalents of the elements are embraced in the present invention. Any reference sign in a claim should not be construed as limiting the claim concerned. In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or devices stated in the system claims may also be realized by one unit or device through software or hardware. The words first, second, etc. are used to denote names without implying any particular order.

Claims (14)

1. A method for managing meta-information, wherein the method comprises the following steps: -Store or update its corresponding meta-information in each data block of the target system. 2. The method according to claim 1, wherein the step of storing or updating the corresponding meta-information in each data block of the target system stores the meta-information in at least one of the following ways: - Store meta information in the underlying file system; - store meta information in the data block name; - Store meta information in the data header of the data block. 3. The method according to claim 1, wherein said step a comprises the steps of: - Based on the service type information of the data to be stored, storing the data block and its meta information corresponding to the data to be stored in a specific storage medium, so as to manage the meta information in each storage medium respectively. 4. The method according to claim 1, wherein said method comprises the steps of: - When the metadata information of the system needs to be loaded, multiple threads are created to respectively load the metadata information of the data blocks in multiple disks. 5. The method according to claim 4, wherein, during the startup phase of the target system, the method comprises the steps of: -According to the blacklist information of the target system, determine the disk information that needs to be filtered out, so as to obtain one or more disk information that can be used; Wherein, the step of creating multiple threads to respectively load the metadata information of the data blocks in multiple disks includes the following steps: - Based on the available one or more disk information, creating multiple threads to respectively load the metadata information of the data blocks in each disk. 6. The method according to claim 5, wherein said method comprises the steps of: -Load the low compression ratio data block information for each disk, so that each low compression ratio data block is skipped when the target system performs transparent compression. 7. The method according to claim 5, wherein said method comprises the steps of: -According to the usage information and status information of each disk that has been scanned, the data blocks loaded by each disk are deduplicated. 8. An information management device for managing meta-information, wherein the information management device comprises: Means for storing or updating the corresponding meta-information in each data block of the target system. 9. The information management device according to claim 8, wherein the means for storing or updating the corresponding meta information in each data block of the target system stores the meta information in at least one of the following ways: - Store meta information in the underlying file system; - store meta information in the data block name; - Store meta information in the data header of the data block. 10. The information management device according to claim 8, wherein the means for storing or updating the corresponding meta information in each data block of the target system is used for: - Based on the service type information of the data to be stored, the data block corresponding to the data to be stored and its meta information are stored in a specific storage medium, so as to manage the meta information in each storage medium respectively. 11. The information management device according to claim 8, wherein the information management device comprises: It is used to create multiple threads to separately load metadata information of data blocks in multiple disks when the metadata information of the system needs to be loaded. 12. The information management device according to claim 11, wherein, at the startup phase of the target system, the information management device comprises: A device for determining the disk information that needs to be filtered out according to the blacklist information of the target system, so as to obtain one or more disk information that can be used; Wherein, the device for storing or updating the corresponding meta-information in each data block of the target system is used for: - Based on the available one or more disk information, creating multiple threads to respectively load the metadata information of the data blocks in each disk. 13. The information management device according to claim 12, wherein the information management device comprises: A device for loading low-compression data block information of each disk, thereby skipping each low-compression data block when the target system performs transparent compression. 14. The information management device according to claim 12, wherein the information management device comprises: A device for performing deduplication processing on data blocks loaded on each disk according to the scanned usage information and status information of each disk.