[go: up one dir, main page]

CN119759291A - Object multi-version intercommunication access method, system, device and medium - Google Patents

Object multi-version intercommunication access method, system, device and medium Download PDF

Info

Publication number
CN119759291A
CN119759291A CN202411960580.9A CN202411960580A CN119759291A CN 119759291 A CN119759291 A CN 119759291A CN 202411960580 A CN202411960580 A CN 202411960580A CN 119759291 A CN119759291 A CN 119759291A
Authority
CN
China
Prior art keywords
deletion
historical
gateway
version
object set
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202411960580.9A
Other languages
Chinese (zh)
Inventor
王飞平
李小勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Xiao Yun Info Tech Co ltd
Original Assignee
Shanghai Xiao Yun Info Tech Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Xiao Yun Info Tech Co ltd filed Critical Shanghai Xiao Yun Info Tech Co ltd
Priority to CN202411960580.9A priority Critical patent/CN119759291A/en
Publication of CN119759291A publication Critical patent/CN119759291A/en
Pending legal-status Critical Current

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to an object multi-version intercommunication access method, a system, equipment and a medium, which comprises the steps of obtaining an operation request of an s3 gateway, and operating based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least comprises the overwriting operation of the s3 gateway, the deleting operation of the s3 gateway and the historical version operation of the s3 gateway, and the IO request of other gateways is obtained, the identification of a corresponding file is identified, the corresponding file metadata is searched in the current object set, and the reading operation and/or the writing operation of the other gateways are carried out.

Description

Object multi-version intercommunication access method, system, equipment and medium
Technical Field
The invention relates to the technical field of distributed s3 object storage. And more particularly, to a method, system, device, and medium for multi-version interworking access to objects.
Background
S3 object store [ Amazon Simple Storage Service (Amazon S3) ] is an object store service that provides object store through an interface based on the restul API, a technique that stores and manages data in an unstructured format. In practical use scenarios, users usually need to use different protocols to access the same piece of data, for example, data written through s3 needs to access through protocols such as NFS (Network FILE SYSTEM), SAMBA (free software implementing SMB Protocol on Linux and UNIX systems), FTP (FILE TRANSFER Protocol), and the like, and in the method of the present invention, a mode of accessing the same piece of data through multiple different protocols is called interworking access.
The multi-version function of s3 object storage allows objects in a bucket to have multiple versions so that the change history of the objects can be tracked, which is very useful for application scenarios requiring data recovery, auditing, or compliance requirements. After the multi-version function of the object is started, multi-protocol interworking becomes a challenging point, for example, after the multi-version function is started in s3, an object is overwritten, a history version record is required to be checked in the query of the multi-version interface in s3, and history version data can be retrieved, meanwhile, when the object which is just written in s3 is accessed through the protocol such as NFS, SAMBA, FTP, only the latest written data is required to be seen, and old data cannot be accessed, namely, the history data cannot be seen by other protocols such as NFS, SAMBA, FTP which are not s3. In the existing few s3 object storage products supporting multi-protocol intercommunication, the multi-version function of the object and the intercommunication of other protocols cannot be supported.
Therefore, how to invent a multi-version function and other protocol interworking technology of an s3 object storage product is a problem to be solved.
Disclosure of Invention
The invention provides an object multi-version intercommunication access method, system, equipment and medium, which are used for solving the problem that in the prior art, a plurality of protocols such as an s3 object and NFS, SAMBA, FTP are required to be specially converted and can not be directly accessed in an intercommunication way.
To achieve the above object, in a first aspect, the present invention relates to an object multi-version interworking access method, for s3 objects of a distributed storage system, including:
Acquiring an operation request of an s3 gateway, and operating based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least comprises an overwriting operation of the s3 gateway, a deleting operation of the s3 gateway and a historical version operation of the s3 gateway;
Identifying an object written by s3, checking whether the s3 object requested at this time exists in a current object set, if not, writing according to a new object, if so, moving old pair of pixel data from the current object set to a historical deleted object set, recording deleting time, generating a new global unique object ID, positioning the new global unique object ID to a position to be written by the object, and transmitting content to be written at this time to the writing position to finish data writing operation;
The deleting operation of the s3 gateway is that an s3 object to be deleted is moved from the current object set to the historical deleting object set, deleting time is recorded, and a deleting mark is created in a historical deleting mark table;
The historical version enumeration operation of the s3 gateway is that a current version object is listed from the current object set, a historical version object is listed from the historical deletion object set, and a deletion mark is listed from the historical deletion mark table;
And acquiring IO requests of other gateways, identifying the identification of the corresponding file, searching the metadata of the corresponding file in the current object set, and performing reading operation and/or writing operation of the other gateways, wherein the other gateways comprise one or more of an NFS gateway, an SAMBA gateway and an FTP gateway.
In order to achieve the above object, in a second aspect, the present invention relates to an object multi-version interworking access system, which is used for a distributed storage system s3 object, and comprises an operation module of an s3 gateway and other gateway access modules;
The operation module of the s3 gateway is configured to obtain an operation request of the s3 gateway, and perform an operation based on the operation request of the s3 gateway, where the operation of the s3 gateway at least includes an overwriting operation of the s3 gateway, a deleting operation of the s3 gateway, and a history version operation of the s3 gateway;
Identifying an object written by s3, checking whether the s3 object requested at this time exists in a current object set, if not, writing according to a new object, if so, moving old pair of pixel data from the current object set to a historical deleted object set, recording deleting time, generating a new global unique object ID, positioning the new global unique object ID to a position to be written by the object, and transmitting content to be written at this time to the writing position to finish data writing operation;
The deleting operation of the s3 gateway is that an s3 object to be deleted is moved from the current object set to the historical deleting object set, deleting time is recorded, and a deleting mark is created in a historical deleting mark table;
The historical version enumeration operation of the s3 gateway is that a current version object is listed from the current object set, a historical version object is listed from the historical deletion object set, and a deletion mark is listed from the historical deletion mark table;
And the other gateway access module is used for acquiring IO requests of other gateways, identifying the identification of the corresponding file, searching the metadata of the corresponding file in the current object set, and performing read operation and/or write operation of the other gateways, wherein the other gateways comprise one or more of an NFS gateway, an SAMBA gateway and an FTP gateway.
To achieve the above object, in a third aspect, the present invention further relates to an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the above-mentioned one object multi-version interworking access method when executing the computer program.
To achieve the above object, in a fourth aspect, the present invention further relates to a computer readable storage medium having stored therein instructions which, when executed, perform an object multi-version interworking access method as described above.
Compared with the prior art, the method, the system, the equipment and the medium for the multi-version intercommunication access of the object have the following beneficial effects:
The current version of the s3 multi-version is stored in the current object set, metadata of the current version is the same as file metadata created by the NFS, SAMBA, FTP and other protocols, and the NFS, SAMBA, FTP gateway only can access the current object set, so that after the s3 multi-version is overwritten, the NFS, SAMBA, FTP gateway only can access the latest written data, and the historical object metadata is transferred to the historical deleted object set, and the historical object metadata can only be accessed by the s3 gateway. Similarly, the same metadata is written in the file through the NFS, SAMBA, FTP gateway and the like, the file can be immediately reacted into the current version of s3, the latest changed data can be read through the s3 gateway, the bottom metadata is the same, the data access mode is the same, different protocols can be directly accessed through different gateways, but the deleted historical object metadata cannot be accessed by the non-s 3 protocol.
Therefore, by distinguishing and managing the currently used object metadata and the deleted historical object metadata, when other non-s 3 protocols such as NFS, SAMBA, FTP and the like access the s3 object multi-version data, the current latest version of object metadata can be always accessed, and the invention does not need any conversion, so that the intercommunication access of different protocols has no performance loss, normal functions and no performance reduction.
Drawings
FIG. 1 is a flow chart of a method for accessing object multi-version intercommunication in accordance with a first embodiment of the present invention;
FIG. 2 is a flowchart of an example of an overlay operation of an s3 gateway of an object multi-version interworking access method according to a first embodiment of the present invention;
FIG. 3 is a flowchart of an example of the deletion operation of the s3 gateway of the object multi-version interworking access method according to the first embodiment of the present invention;
FIG. 4 is a flowchart of an example of the historical version enumeration operation of the s3 gateway of the method for accessing object multi-version interworking in accordance with the first embodiment of the present invention;
FIG. 5 is a flowchart of an example IO operation of another gateway of the object multi-version interworking access method according to the first embodiment of the present invention;
FIG. 6 is a flowchart of an example of the operation of retrieving historical version data of an object multi-version interworking access method according to the first embodiment of the present invention;
FIG. 7 is a flowchart of an example of the operation of deleting historical version data of an object multi-version interworking access method according to the first embodiment of the present invention;
FIG. 8 is a schematic diagram of a structure of an object multi-version interworking access system according to a second embodiment of the present invention;
Fig. 9 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Example 1
1-6, An object multi-version intercommunication access method is disclosed, which can be implemented by an electronic device with a processor, and can be implemented by a personal computer, an intelligent terminal, a local area network, a server and the like, and is used for S3 objects of a distributed storage system, wherein the distributed storage system comprises a storage network, an S3 gateway, other gateways, a multi-node metadata manager, a container, an object set, a file position generator, a storage node, a hard disk on the node and a local file system after the hard disk is formatted, the other gateways can be one or more of an NFS gateway, an SAMBA gateway and an FTP gateway, the multi-node metadata manager is responsible for generating globally unique IDs for metadata of each object and file, and the file position generator calculates storage positions according to the globally unique IDs to perform corresponding IO operations.
As shown in fig. 1, includes steps S101 to S102.
S101, acquiring an operation request of an S3 gateway, and operating based on the operation request of the S3 gateway, wherein the operation of the S3 gateway at least comprises an overwriting operation of the S3 gateway, a deleting operation of the S3 gateway and a historical version operation of the S3 gateway;
The covering and writing operation of the s3 gateway is as follows:
identifying an object written by s 3;
And if the s3 object is not present, writing the new object, moving the old pair of pixel data from the current object set to the historical deleting object set, recording the deleting time, generating a new global unique object ID, positioning the position to be written by the object through the global unique object ID, and transmitting the content to be written to the writing position to finish the data writing operation.
The overwriting operation of the s3 gateway is specifically illustrated by the following example, as shown in fig. 2.
Step 11, checking whether the object multi-version function of the s3 barrel is started, if not, directly returning, and processing according to a normal overwriting flow;
Step 12, analyzing the s3 request, identifying an s3 object, checking whether the s3 object requested at this time exists in the multi-node metadata manager, if the s3 object does not exist, writing according to a new object, if the s3 object exists, moving old pair of pixel data from a current object set to a history deleted object set, and adding a deleted time field in the history deleted object set;
step 13, generating a new global unique object ID, positioning a position generator to a position to be written by the object through the ID;
Step 14, sending the content to be written to the writing position obtained in the step 3 to finish data writing operation, if the writing fails, the overwriting request processing fails, and performing rollback processing, namely moving the metadata record moved to the history deleted object set in the step 2 to the current object set;
15, storing the attribute such as the time written in this time, the object size, the MD5 value of the data content and the like into the current object set in the multi-node metadata manager by taking the global unique object ID as a Key;
And step 16, ending the writing request processing.
The deleting operation of the s3 gateway is to move the s3 object to be deleted from the current object set to the historical deleting object set and record the deleting time, and create a deleting mark in the historical deleting mark table.
In some embodiments, the deleting operation of the 3S gateway further includes, if the object to be deleted is the first delete marker, checking whether the first delete marker is still present after the first delete marker is deleted, if not, moving the object deleted last in the historical deleted object set to the current object set, and if the first delete marker is still present, ending the operation.
The deletion operation of the s3 gateway is specifically illustrated by the following example, as shown in fig. 3.
Step 21, checking whether the object multi-version function of the s3 barrel is started, if not, directly returning, and processing according to a normal deleting flow;
Step 22, analyzing the s3 request, identifying an s3 object, if the object to be deleted exists, moving the old object to a historical deleted object set, adding deleting time, and finally creating a deleting mark in the historical deleted object set;
Step 23, if the object to be deleted is a delete marker, after the delete marker is finished, checking whether the object has a delete marker, if not, moving the object deleted latest by the historical delete object set to the current object set, if so, directly returning;
Step 24, the deletion request processing ends.
The historical version enumeration operation of the s3 gateway is to list the current version object from the current object set, list the historical version object from the historical deletion object set and list the deletion mark from the historical deletion mark table.
In the embodiment, a history version object is listed from a history deletion object set, a deletion object mark is listed from a history deletion mark table, specifically, the history deletion object set is listed, if the history deletion object set is a deletion object with a deletion mark in the history deletion mark table, the listed file size is set to be 0, and if the history deletion object set is a history object without a deletion mark, the history object metadata is returned according to the s3 protocol requirement.
The historical version enumeration operation of the s3 gateway is specifically illustrated by the following example, as shown in fig. 4.
Step 31, checking whether the object multi-version function of the s3 barrel is started, if not, directly returning, and processing according to a normal deleting flow;
Step 32, analyzing the s3 request, identifying an s3 object, if the object to be deleted exists, moving the old object to a historical deleted object set, adding deleting time, and finally creating a deleting mark in the historical deleted object set;
Step 33, if the object to be deleted is a delete marker, after the delete marker is finished, checking whether the object has a delete marker, if not, moving the object deleted latest by the historical delete object set to the current object set, if so, directly returning;
Step 34 the deletion request processing ends.
S102, acquiring IO requests of other gateways, identifying the identification of the corresponding file, searching the metadata of the corresponding file in the current object set, and performing reading operation and/or writing operation of the other gateways, wherein the other gateways comprise one or more of an NFS gateway, an SAMBA gateway and an FTP gateway.
The IO operation of other gateways is specifically illustrated by the following example, as shown in fig. 5.
Step 41, each gateway analyzes and processes the corresponding protocol request, wherein the NFS gateway processes the NFS protocol, the SAMBA gateway processes the SAMBA protocol and the FTP gateway processes the FTP protocol;
Step 42, after the NFS gateway, the SAMBA gateway and the FTP gateway receive the file access requests of the respective protocols, analyzing and identifying the identification of the corresponding files, searching the corresponding file metadata in the current object set of the multi-node metadata manager, and acquiring information such as file ID, file size and the like;
step 43, the position generator calculates the path of the local file system according to the file ID and performs corresponding data access according to the request type, if the data is written in, the data is written in the local file system, and the metadata such as the file size, the modification time and the like are updated after the data is written in successfully;
and step 44, ending the return.
In some embodiments, the operation of the s3 gateway further comprises a history version data retrieving operation, a history version number ID input by the s3 client is obtained, a file ID and a file size of the history version data are retrieved from the history deletion object set according to the history version number ID, a local file system path is calculated according to the file ID, data is read from the local file system path according to the file size, and the read data is returned to the s3 client.
The retrieval history version data operation is specifically set forth by the following example, as shown in fig. 6.
Step 51, analyzing the s3 request to obtain the retrieved history version number ID;
step 52, obtaining a file ID and a file size from a historical deleted object set in the multi-node metadata manager;
step 53, the position generator calculates the path of the local file system according to the file ID, and reads the data from the local file system according to the file size;
step 54 returns the read data to the client;
Step 55 ends the return.
In some embodiments, the operations of the s3 gateway further comprise a delete history version data operation:
acquiring a retrieved history version number ID;
acquiring a file ID and a file size from a history deleted object set in the multi-node metadata manager;
the position generator calculates the path of the local file system according to the file ID, and deletes the local file system data;
And deleting the object history version metadata recorded by the history deletion object set.
The delete history version data operation is specifically illustrated by the following example, as shown in fig. 7.
Step 61, analyzing s3 request to obtain the retrieved history version number ID;
step 62 obtains a file ID, file size from a set of historical deletion objects in the multi-node metadata manager;
step 63, the position generator calculates the path of the local file system according to the file ID, and deletes the local file system data;
step 64, deleting the object history version metadata recorded by the history deleted object set;
step 65 ends the return.
In summary, according to the method for accessing object multi-version intercommunication, the current versions of the s3 multi-version are stored in the current object set, metadata of the current versions and file metadata created by NFS, SAMBA, FTP and other protocols are the same, and the NFS gateway, the SAMBA gateway, the FTP gateway and other gateways can only access the current object set, so that after the s3 multi-version overwriting, the NFS gateway, the SAMBA gateway, the FTP gateway and other gateways can access the latest written data, and the historical object metadata is transferred to the historical deleted object set and can only be accessed by the s3 gateway. Also, because of the same piece of metadata, files written through NFS gateway, SAMBA gateway, FTP gateway, etc. can also be immediately reacted to the current version of s3, and the latest changed data can be read through s3 gateway. Because the underlying metadata is the same, the data access modes are the same, and different protocols can be directly accessed through different gateways without any conversion, so that the intercommunication access of different protocols has no performance loss. .
Example two
An object multi-version intercommunication access system is used for a distributed storage system s3 object, is realized by electronic equipment hardware with a processor, and can be realized by a personal computer, an intelligent terminal, a local area network, a server and the like. Referring to fig. 8, the system includes an operation module 81 of s3 gateway and other gateway access modules 82;
An operation module 81 of the s3 gateway, configured to obtain an operation request of the s3 gateway, and perform an operation based on the operation request of the s3 gateway, where the operation of the s3 gateway at least includes an overwriting operation of the s3 gateway, a deleting operation of the s3 gateway, and a history version operation of the s3 gateway;
identifying an object written by s3, checking whether the requested s3 object exists in the current object set, and if not, writing according to the new object; if the data exists, moving old pair of pixel data from a current object set to a historical deleted object set, recording deleting time, generating a new global unique object ID, positioning a position to be written by an object through the global unique object ID, and sending the content to be written to a writing position to finish data writing operation;
The deleting operation of the s3 gateway is that the s3 object to be deleted is moved from the current object set to the historical deleting object set, the deleting time is recorded, and a deleting mark is created in the historical deleting mark table;
The historical version enumeration operation of the s3 gateway is that the current version object is listed from the current object set, the historical version object is listed from the historical deletion object set, and the deletion mark is listed from the historical deletion mark table;
And the other gateway access module 82 is configured to obtain IO requests of other gateways, identify an identifier of a corresponding file, search metadata of the corresponding file in the current object set, and perform a read operation and/or a write operation of the other gateways, where the other gateways include one or more of an NFS gateway, an SAMBA gateway, and an FTP gateway.
In some embodiments, the s3 gateway further includes a delete history delete marker table operation, where the delete history delete marker table operation is performed by receiving a request from the history delete marker table operation, removing a corresponding delete object marker from the history delete marker table, and if the s3 object delete marker is completely cleared, selecting an object with the longest delete time from the history delete object set and moving the object to the current object set;
In some embodiments, the operation of the s3 gateway further includes a search history version data operation, where the search history version data operation is specifically performed by obtaining a history version number ID input by the s3 client, searching a file ID and a file size of the history version data from the history deletion object set according to the history version number ID, calculating a local file system path according to the file ID, reading data from the local file system path according to the file size, and returning the read data to the s3 client;
in some embodiments, the s3 gateway further includes a delete history version data operation, where the delete history version data operation is performed by obtaining the retrieved history version number ID, obtaining a file ID and a file size from a set of history delete objects in the multi-node metadata manager, calculating a local file system path according to the file ID, deleting the local file system data, and deleting the object history version metadata recorded by the set of history delete objects.
In some embodiments, the deleting operation of the 3S gateway further includes, if the object to be deleted is the first delete marker, checking whether the first delete marker is still present after the first delete marker is deleted, if not, moving the object deleted last in the historical deleted object set to the current object set, and if the first delete marker is still present, ending the operation.
In some embodiments, the history version object is listed from the history deletion object set, and the deletion object mark is listed from the history deletion mark table, specifically, the history deletion object set is listed, if the history deletion object set is the deletion object with the deletion mark in the history deletion mark table, the listed file size is set to be 0, if the history deletion object set is the history object without the deletion mark, the history object metadata is returned according to the s3 protocol requirement.
The implementation process method and effect of the object multi-version intercommunication access system in this embodiment are the same as those of the object multi-version intercommunication access method described in the first embodiment, and are not described here again.
Example III
As shown in fig. 9, the present embodiment is directed to an electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor. The memory stores a computer program that can be executed by at least one processor, so that the at least one processor can execute an object multi-version interworking access method of embodiment 1, and achieve the corresponding beneficial effects of the object multi-version interworking access method, which are not described herein. The electronic device provided in this embodiment may be a personal computer, for example, a desktop computer, an integrated machine, a notebook computer, a tablet computer, or a terminal device such as a mobile phone, a wearable device, a palm computer, or the like. The electronic device is merely an example, and should not impose any limitations on the functionality and scope of use of embodiments of the present invention.
The components of the electronic device 3 may include, but are not limited to, the at least one processor 4 described above, the at least one memory 5 described above, and a bus 6 connecting the various system components, including the memory 5 and the processor 4.
The bus 6 includes a data bus, an address bus, and a control bus.
The memory 5 may include volatile memory such as Random Access Memory (RAM) 51 and/or cache memory 52, and may further include Read Only Memory (ROM) 53.
The memory 5 may also include a program/utility 55 having a set (at least one) of program modules 54, such program modules 54 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The processor 4 executes various functional applications and data processing, such as an object multi-version interworking access method described above, by running a computer program stored in the memory 5.
The electronic device 3 may also communicate with one or more external devices 7, such as a keyboard, pointing device, etc. Such communication may be through an input/output (I/O) interface 8. And the electronic device 3 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the internet, via the network adapter 9. As shown in fig. 5, the network adapter 9 communicates with other modules of the electronic device 3 via the bus 6. It should be appreciated that although not shown in FIG. 5, other hardware and/or software modules may be used in connection with the electronic device 3, including, but not limited to, microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, among others.
It should be noted that although several units/modules or sub-units/modules of an electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present invention. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
Example IV
The invention relates to a computer readable storage medium, the storage medium stores instructions, the instructions execute an object multi-version intercommunication access method for implementing the first, the implementation process and effect of the instructions are the same as those of the object multi-version intercommunication access method described in the first embodiment, and the description is omitted here.
It should be noted that, in this document, the terms include, comprise, or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by a statement includes, without being limited to, that there is no intention to exclude that there are additional identical elements in the process, method, article or apparatus that includes the element.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims (12)

1.一种对象多版本互通访问方法,其特征在于,用于分布式存储系统的s3对象,包括:1. A method for interoperable access of multiple versions of an object, characterized in that it is used for an S3 object in a distributed storage system, comprising: 获取s3网关的操作请求,则基于所述s3网关的操作请求进行操作,其中,所述s3网关的操作至少包括:s3网关的覆盖写操作、s3网关的删除操作以及s3网关的历史版本操作;Obtaining an operation request of an s3 gateway, and then performing an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: an overwrite operation of the s3 gateway, a delete operation of the s3 gateway, and a historical version operation of the s3 gateway; 其中,所述s3网关的覆盖写操作为:识别出s3写入的对象,查验本次请求的s3对象是否存在于当前对象集合,若不存在,则按照新对象进行写入;若存在,将旧对像元数据从所述当前对象集合移动到历史删除对象集合,并记录删除时间,并生成新全局唯一对象ID,通过所述全局唯一对象ID定位到需要对象要写入的位置,将本次需要写入的内容发送到所述写入位置完成数据写入操作;The overwrite operation of the s3 gateway is as follows: identify the object to be written by s3, check whether the s3 object requested this time exists in the current object set, if not, write it as a new object; if it exists, move the metadata of the old object from the current object set to the historical deletion object set, record the deletion time, and generate a new globally unique object ID, locate the location where the object needs to be written through the globally unique object ID, and send the content to be written this time to the write location to complete the data writing operation; 所述s3网关的删除操作为:将待删除的s3对象从所述当前对象集合移动到所述历史删除对象集合并记录删除时间,并在历史删除标记表中创建删除标记;The deletion operation of the s3 gateway is: moving the s3 object to be deleted from the current object set to the historical deletion object set and recording the deletion time, and creating a deletion mark in the historical deletion mark table; 以及所述s3网关的历史版本列举操作为:从所述当前对象集合中列出当前版本对象,从所述历史删除对象集合中列出历史版本对象,从所述历史删除标记表中列出删除标记;And the historical version listing operation of the s3 gateway is: listing the current version objects from the current object set, listing the historical version objects from the historical deletion object set, and listing the deletion marks from the historical deletion mark table; 获取其他网关的IO请求,则识别出对应文件的标识,在所述当前对象集合中查找对应文件元数据,进行所述其他网关的读取操作和/或写入操作,其中,所述其他网关包括NFS网关、SAMBA网关以及FTP网关中的一种或几种。Obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and/or write operations on the other gateways, wherein the other gateways include one or more of an NFS gateway, a SAMBA gateway, and an FTP gateway. 2.根据权利要求1所述的一种对象多版本互通访问方法,其特征在于,所述s3网关的操作还包括:删除所述历史删除标记表操作,接收所述历史删除标记表操作的请求,则从所述历史删除标记表中移除对应所述删除对象标记,如果该s3对象所述删除标记全部清除完成,则从所述历史删除对象集合中挑选删除时间最长的对象,并将其移动到所述当前对象集合。2. According to the object multi-version intercommunication access method described in claim 1, it is characterized in that the operation of the s3 gateway also includes: deleting the historical deletion mark table operation, receiving a request for the historical deletion mark table operation, then removing the corresponding deletion object mark from the historical deletion mark table, if all the deletion marks of the s3 object are cleared, then selecting the object with the longest deletion time from the historical deletion object set, and moving it to the current object set. 3.根据权利要求2所述的一种对象多版本互通访问方法,其特征在于,所述s3网关的操作还包括:检索历史版本数据操作,获取s3客户端输入的历史版本号ID,根据所述历史版本号ID从所述历史删除对象集合中检索历史版本数据的文件ID和文件大小,根据所述文件ID计算出本地文件系统路径,再根据文件大小从所述本地文件系统路径读取数据,将读取数据返回给所述s3客户端。3. According to the object multi-version intercommunication access method described in claim 2, it is characterized in that the operation of the s3 gateway also includes: retrieving historical version data operation, obtaining the historical version number ID input by the s3 client, retrieving the file ID and file size of the historical version data from the historical deletion object set according to the historical version number ID, calculating the local file system path according to the file ID, and then reading data from the local file system path according to the file size, and returning the read data to the s3 client. 4.根据权利要求3所述的一种对象多版本互通访问方法,其特征在于,所述s3网关的操作还包括删除历史版本数据操作:4. According to the object multi-version intercommunication access method of claim 3, the operation of the s3 gateway also includes deleting historical version data: 获取检索的历史版本号ID;Get the retrieved historical version number ID; 从多节点元数据管理器中的所述历史删除对象集合获取文件ID和文件大小;Obtaining a file ID and a file size from the historical deletion object set in the multi-node metadata manager; 位置生成器根据文件ID计算出其在本地文件系统路径,删除本地文件系统数据;The location generator calculates the path of the file in the local file system based on the file ID and deletes the local file system data; 删除所述历史删除对象集合记录的对象历史版本元数据。Delete the object history version metadata recorded in the history deletion object set. 5.根据权利要求3所述的一种对象多版本互通访问方法,其特征在于,所述3S网关的删除操作,还包括,若所述待删除的对象是第一删除标记,则在删除所述第一删除标记完成后,检查是否还有第一删除标记,如果不存在,则将所述历史删除对象集合中最晚删除的对象移动到所述当前对象集合;如果还存在所述第一删除标记,则结束操作。5. According to the object multi-version intercommunication access method described in claim 3, it is characterized in that the deletion operation of the 3S gateway also includes: if the object to be deleted is a first deletion marker, then after deleting the first deletion marker, checking whether there is still a first deletion marker, if not, moving the object most recently deleted in the historical deletion object set to the current object set; if the first deletion marker still exists, ending the operation. 6.根据权利要求3所述的一种对象多版本互通访问方法,其特征在于,所述从所述历史删除对象集合中列出历史版本对象,从所述历史删除标记表中列出删除对象标记,具体为;列举所述历史删除对象集合,如果是所述所述历史删除标记表有删除标记的删除对象,则设定列举的文件大小为0,如果是不带删除标记的历史对象,将所述历史对象元数据按照s3协议要求返回。6. According to claim 3, a method for interoperable access of multiple versions of objects is characterized in that the historical version objects are listed from the historical deletion object set, and the deletion object marks are listed from the historical deletion mark table, specifically: the historical deletion object set is listed, if it is a deletion object with a deletion mark in the historical deletion mark table, the listed file size is set to 0, if it is a historical object without a deletion mark, the historical object metadata is returned in accordance with the s3 protocol requirements. 7.一种对象多版本互通访问系统,其特征在于,用于分布式存储系统s3对象,包括:s3网关的操作模块以及其他网关访问模块;7. An object multi-version intercommunication access system, characterized in that it is used for distributed storage system s3 objects, including: an s3 gateway operation module and other gateway access modules; 所述s3网关的操作模块,用于获取s3网关的操作请求,则基于所述s3网关的操作请求进行操作,其中,所述s3网关的操作至少包括:s3网关的覆盖写操作、s3网关的删除操作以及s3网关的历史版本操作;The operation module of the s3 gateway is used to obtain an operation request of the s3 gateway, and then perform an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: an overwrite operation of the s3 gateway, a delete operation of the s3 gateway, and a historical version operation of the s3 gateway; 其中,所述s3网关的覆盖写操作为:识别出s3写入的对象,查验本次请求的s3对象是否存在于当前对象集合,若不存在,则按照新对象进行写入;若存在,将旧对像元数据从所述当前对象集合移动到历史删除对象集合,并记录删除时间,并生成新全局唯一对象ID,通过所述全局唯一对象ID定位到需要对象要写入的位置,将本次需要写入的内容发送到所述写入位置完成数据写入操作;The overwrite operation of the s3 gateway is as follows: identify the object to be written by s3, check whether the s3 object requested this time exists in the current object set, if not, write it as a new object; if it exists, move the metadata of the old object from the current object set to the historical deletion object set, record the deletion time, and generate a new globally unique object ID, locate the location where the object needs to be written through the globally unique object ID, and send the content to be written this time to the write location to complete the data writing operation; 所述s3网关的删除操作为:将待删除的s3对象从所述当前对象集合移动到所述历史删除对象集合并记录删除时间,并在历史删除标记表中创建删除标记;The deletion operation of the s3 gateway is: moving the s3 object to be deleted from the current object set to the historical deletion object set and recording the deletion time, and creating a deletion mark in the historical deletion mark table; 以及s3网关的历史版本列举操作为:从所述当前对象集合中列出当前版本对象,从所述历史删除对象集合中列出历史版本对象,从所述历史删除标记表中列出删除标记;And the historical version listing operation of the s3 gateway is: listing the current version objects from the current object set, listing the historical version objects from the historical deletion object set, and listing the deletion marks from the historical deletion mark table; 所述其他网关访问模块,用于获取其他网关的IO请求,则识别出对应文件的标识,在所述当前对象集合中查找对应文件元数据,进行所述其他网关的读取操作和/或写入操作,其中,所述其他网关包括NFS网关、SAMBA网关以及FTP网关中的一种或几种。The other gateway access module is used to obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and/or write operations on the other gateways, wherein the other gateways include one or more of NFS gateways, SAMBA gateways and FTP gateways. 8.根据权利要求7所述的一种对象多版本互通访问系统,其特征在于,所述s3网关的操作还包括删除所述历史删除标记表操作,执行所述删除所述历史删除标记表操作具体为:接收所述历史删除标记表操作的请求,则从所述历史删除标记表中移除对应所述删除对象标记,如果该s3对象所述删除标记全部清除完成,则从所述历史删除对象集合中挑选删除时间最长的对象,并将其移动到所述当前对象集合;8. According to claim 7, a multi-version intercommunication access system for objects is characterized in that the operation of the s3 gateway also includes deleting the historical deletion mark table operation, and executing the deleting the historical deletion mark table operation is specifically: receiving a request for the historical deletion mark table operation, removing the corresponding deletion object mark from the historical deletion mark table, if all the deletion marks of the s3 object are cleared, selecting the object with the longest deletion time from the historical deletion object set, and moving it to the current object set; 所述s3网关的操作还包括检索历史版本数据操作,执行所述检索历史版本数据操作具体为:获取s3客户端输入的历史版本号ID,根据所述历史版本号ID从所述历史删除对象集合中检索历史版本数据的文件ID和文件大小,根据所述文件ID计算出本地文件系统路径,再根据文件大小从所述本地文件系统路径读取数据,将读取数据返回给所述s3客户端;The operation of the s3 gateway also includes an operation of retrieving historical version data. The operation of retrieving historical version data is specifically performed as follows: obtaining the historical version number ID input by the s3 client, retrieving the file ID and file size of the historical version data from the historical deletion object set according to the historical version number ID, calculating the local file system path according to the file ID, and then reading data from the local file system path according to the file size, and returning the read data to the s3 client; 或所述s3网关的操作还包括删除历史版本数据操作,执行所述删除历史版本数据操作具体为:获取检索的历史版本号ID;从多节点元数据管理器中的所述历史删除对象集合获取文件ID和文件大小;根据文件ID计算出其在本地文件系统路径,删除本地文件系统数据;删除所述历史删除对象集合记录的对象历史版本元数据。Or the operation of the s3 gateway also includes an operation of deleting historical version data, and executing the operation of deleting historical version data is specifically as follows: obtaining the retrieved historical version number ID; obtaining the file ID and file size from the historical deletion object set in the multi-node metadata manager; calculating its path in the local file system based on the file ID, and deleting the local file system data; deleting the object historical version metadata recorded in the historical deletion object set. 9.根据权利要求7所述的一种对象多版本互通访问系统,其特征在于:所述3S网关的删除操作,还包括,若所述待删除的对象是第一删除标记,则在删除所述第一删除标记完成后,检查是否还有第一删除标记,如果不存在,则将所述历史删除对象集合中最晚删除的对象移动到所述当前对象集合;如果还存在所述第一删除标记,则结束操作。9. According to claim 7, a multi-version interoperable access system for objects is characterized in that: the deletion operation of the 3S gateway also includes, if the object to be deleted is a first deletion mark, then after deleting the first deletion mark, checking whether there is still a first deletion mark, if not, moving the object most recently deleted in the historical deletion object set to the current object set; if the first deletion mark still exists, ending the operation. 10.根据权利要求7所述的一种对象多版本互通访问系统,其特征在于,所述从所述历史删除对象集合中列出历史版本对象,从所述历史删除标记表中列出删除对象标记,具体为:列举所述历史删除对象集合,如果是所述所述历史删除标记表有删除标记的删除对象,则设定列举的文件大小为0,如果是不带删除标记的历史对象,将所述历史对象元数据按照s3协议要求返回。10. According to claim 7, a multi-version interoperable access system for objects is characterized in that the historical version objects are listed from the historical deletion object set, and the deletion object marks are listed from the historical deletion mark table, specifically: the historical deletion object set is listed, if it is a deletion object with a deletion mark in the historical deletion mark table, the listed file size is set to 0, if it is a historical object without a deletion mark, the historical object metadata is returned according to the s3 protocol requirements. 11.一种电子设备,包括存储器、处理器以及存储在存储器上并可在处理器上运行的计算机程序,其特征在于,所述处理器执行所述计算机程序时实现如权利要求1-7中任一项所述的一种对象多版本互通访问方法。11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements a method for interoperable access to multiple versions of an object as described in any one of claims 1 to 7. 12.一种计算机可读存储介质,其特征在于:存储介质中存储有指令,所述指令运行时执行如权利要求1-7中任一项所述的一种对象多版本互通访问方法。12. A computer-readable storage medium, characterized in that: the storage medium stores instructions, and when the instructions are executed, the object multi-version intercommunication access method according to any one of claims 1 to 7 is executed.
CN202411960580.9A 2024-12-30 2024-12-30 Object multi-version intercommunication access method, system, device and medium Pending CN119759291A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202411960580.9A CN119759291A (en) 2024-12-30 2024-12-30 Object multi-version intercommunication access method, system, device and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202411960580.9A CN119759291A (en) 2024-12-30 2024-12-30 Object multi-version intercommunication access method, system, device and medium

Publications (1)

Publication Number Publication Date
CN119759291A true CN119759291A (en) 2025-04-04

Family

ID=95195644

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202411960580.9A Pending CN119759291A (en) 2024-12-30 2024-12-30 Object multi-version intercommunication access method, system, device and medium

Country Status (1)

Country Link
CN (1) CN119759291A (en)

Similar Documents

Publication Publication Date Title
US8700573B2 (en) File storage service system, file management device, file management method, ID denotative NAS server and file reading method
JP5697754B2 (en) Computer system, file management method, and metadata server
JP4131514B2 (en) Network system, server, data processing method and program
US7974950B2 (en) Applying a policy criteria to files in a backup image
US7730071B2 (en) Data management system and data management method
CN111078464B (en) A method, device and system for backing up data
US11687595B2 (en) System and method for searching backups
US8527556B2 (en) Systems and methods to update a content store associated with a search index
US20080027998A1 (en) Method and apparatus of continuous data protection for NAS
US9367569B1 (en) Recovery of directory information
EP2380090B1 (en) Data integrity in a database environment through background synchronization
US7020744B2 (en) Computer system
US9020994B1 (en) Client-based migrating of data from content-addressed storage to file-based storage
CN102012933A (en) Distributed file system and method for storing data and providing services by utilizing same
US20050240636A1 (en) Storage system storing worm file
JP5241298B2 (en) System and method for supporting file search and file operations by indexing historical file names and locations
US11550760B1 (en) Time-based partitioning to avoid in-place updates for data set copies
JP2006031608A (en) Computer, storage system, file management method performed by computer, and program
JP4952119B2 (en) Content management system, method and program using file server
US7693883B2 (en) Online data volume deletion
CN119759291A (en) Object multi-version intercommunication access method, system, device and medium
CN116049306A (en) Data synchronization method, device, electronic device and readable storage medium
US12386713B2 (en) Managing use of a shared virtual disk for accessing data in file based backups by multiple virtual machines
US12271270B2 (en) Enabling user-based instant access from file based backups
US12306792B2 (en) Managing access to file based backups based on storage units and workload use

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination