[go: up one dir, main page]

WO2019062572A1 - Procédé, dispositif et système de traitement de données - Google Patents

Procédé, dispositif et système de traitement de données Download PDF

Info

Publication number
WO2019062572A1
WO2019062572A1 PCT/CN2018/105965 CN2018105965W WO2019062572A1 WO 2019062572 A1 WO2019062572 A1 WO 2019062572A1 CN 2018105965 W CN2018105965 W CN 2018105965W WO 2019062572 A1 WO2019062572 A1 WO 2019062572A1
Authority
WO
WIPO (PCT)
Prior art keywords
target object
server
location information
container
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/105965
Other languages
English (en)
Chinese (zh)
Inventor
焦辉
董欢庆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of WO2019062572A1 publication Critical patent/WO2019062572A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9537—Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources

Definitions

  • Embodiments of the present invention relate to the field of computer storage, and in particular, to a data processing method, apparatus, and system.
  • the object storage system quickly becomes the mainstream storage system for storing data in the computer storage field with its advantages of high-speed access characteristics, high reliability and scalability.
  • the terminal when a user uses a terminal (such as a browser on a terminal or other application), the terminal can use the terminal to write data to the object storage system (which can be called an object, the object includes metadata and the data itself), and deletes The object stores some data in the system or reads some data from the object storage system.
  • the object storage system may include a terminal 10, a proxy server 11, a container server 12, and an object server 13.
  • the proxy server 11 in the object storage system receives the object read request sent by the terminal 10, and the proxy server 11 sends the query to the container server 12 of the object storage system.
  • the location information message and then the container server 12 can query the location information of the object in the database in the container server 12. If the location information of the object is queried, the proxy server 11 sends the location information of the object to the object server 13, The object server 13 reads the object based on the position information of the object, and returns the read object to the terminal 10.
  • the proxy server 11 When the terminal 10 deletes a certain data (ie, an object) in the object storage system, the proxy server 11 receives the object deletion request sent by the terminal 10. Similarly, the proxy server 11 sends a message to the container server 12 to query the location information of the object. The container server 12 then queries the location information of the object in the database of the container server 12. If the location information of the object is queried, the location information of the object is sent to the proxy server 11, and the proxy server 11 sends the object to the object server 13. Location information, and then the object server 13 deletes the object in the object server 13 according to the location information of the object, and returns a delete response message to the proxy server 11, and the proxy server 11 sends the container server 12 to delete the container. A request message for the location information of the object in the server 12, whereby the container server 12 deletes the location information of the object in the database.
  • the object server when a request for reading an object (for example, the first object) and a request for deleting the object are concurrently present in the object storage system, since the object server needs to be in the process of deleting the first object, The object is marked as deleted. If the object server first receives the location information indicating the deletion of the first object of the first object, and then receives the location information indicating the first object that reads the first object, because the object server is according to the first The location information of the object may have been marked for deletion of the first object. As a result, the object server may not be able to read the first object according to the location information of the first target (ie, the object fails to be read), thus causing There is a conflict between reading the first object and deleting the first object.
  • the present application provides a data processing method, apparatus, and system, which can avoid conflicts between a read object and a deleted object to a certain extent.
  • the first aspect provides a data processing method, where the method includes: the container server receives a delete query message sent by the proxy server, where the delete query message includes the identifier information of the container to which the first target object belongs and the identifier of the first target object.
  • the container server queries the location information of the first target object in the object list of the container server according to the identifier information of the container to which the first target object belongs and the identification information of the first target object; if the first target is queried in the object list
  • the location information of the object the container server deletes the location information of the first target object in the object list, and writes the location information of the first target object into the deletion log of the container server; and the container server sends a deletion response message to the proxy server.
  • the object list includes location information of the first type of object, the first type of object is an object stored in the object server that is not marked and deleted, and the location information of the first type of object is used to indicate that the object is in the object server.
  • the location, the deletion log contains location information of the second type of object, and the second type of object is the location information of the object stored in the object server that is marked and deleted.
  • the container server may delete the location information of the first target object from the object list, and The location of the first target object is newly written into the deletion log of the container server. It can be seen that the container server does not need to send the location information of the first target object to the proxy server, and the proxy server does not need to further locate the location information of the first target object.
  • the object server does not need to delete the first target object in the object server, so when deleting the first target object in the object storage system and the first in the read object storage system
  • the first target object in the object server is not marked as deleted during the process of deleting the first target object, so the first target object can be read in the object server, and Compared with the prior art, it can be avoided that the first target object in the object server is Record for deletion, reading problems caused by the failure of the first target object, that is, avoid reading the first target object and delete the conflict first target object.
  • the proxy server since the container server does not need to send the location information of the first target object to the proxy server, the proxy server does not need to send the location information of the first target object to the object server, so that the object server does not need to be in the object.
  • the first target object in the server is marked for deletion, so that the process of deleting the first target object can be simplified, and the communication resources between the servers can be saved.
  • the location information of the first target object may include a file identifier of the first target object, an offset of the first target object, and a size of the first target object.
  • the file identifier indicates that at least two objects are included in the file; or the location information of the first target object may include an ID of the first target object.
  • the stored method (which can be understood as the method of writing) is different, and the content information of the first target object may be different, and for different storage methods.
  • the above data processing method can implement the operation of deleting the first target object, and can simplify the process of deleting the first target object, and can save communication resources between servers.
  • the data processing method provided by the present application may further include: the container server receives a read query message sent by the proxy server, where the read query message includes the second target object The identifier information of the container and the identifier information of the second target object; the container server then queries the object list for the location information of the second target object according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object; If the location information of the second target object is queried in the object list, the container server sends the location information of the second target object to the proxy server, to instruct the proxy server to read the second target from the object server according to the location information of the second target object.
  • the object, the location information of the second target object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, and the file indicated by the file identifier includes at least two objects.
  • objects in the object storage system can be smoothly read by the above method.
  • the data processing method provided by the application may further include: the container server reads the number in the object list. The location information of a target object.
  • the container server reads the location information of the first target object from the object list after the container server writes the location information of the first target object to the delete log of the container server. Taking the location information of the first target object in the object list, in the process of reading the first target object and deleting the first target object, even if the location information of the first target object in the container server has been deleted, And the location information of the first target object has been written into the deletion log, and the first target object can also be read this time.
  • the location information of the first target object in the container server has been deleted, and the location information of the first target object has been written into the deletion log, and the first target object can also be read this time.
  • the reason is: in the process of deleting the first target object, the location information of the first target object may be written in the deletion log to indicate that the first target object is marked for deletion, and no need to be in the object server. The first target object is marked as deleted. Therefore, the first target object still exists in the object server and has not been reclaimed.
  • the object server may obtain the first target object according to the read query request message according to the container server. Location information, the first target object is read in the object server.
  • the data processing method provided by the embodiment of the present invention can avoid conflicts between the read object and the deleted object to a certain extent (refers to when the read object and the deleted object occur concurrently, the operation of deleting the object firstly causes the read object to be read. The operation could not read the object)
  • the data processing method provided by the application may further include: the container server sending a garbage collection request message to the object server, where the garbage collection request message includes deleting the target object in the log Location information; and the container server receives the garbage collection response message sent by the object server, and the container server deletes the location information of the target object in the deletion log, wherein the garbage collection response message indicates that the target server has deleted the target object.
  • the container server may also delete the target object according to the target object.
  • the location information requests the object server to delete (ie, recycle) the content of the target object in the object server, so that the container server deletes the location information of the target object in the deletion log, thereby completely deleting the target object in the object storage system.
  • the data processing method provided by the application may further include: the container server receives the location update message sent by the proxy server; and the container server updates the third target according to the location update message.
  • Location information of the object is used to update the location information of the third target object, where the location update message includes the identifier information of the container to which the third target object belongs, the identifier information of the third target object, and the third target object.
  • the method for updating the location information of the third target object by the container server according to the location update message may include: the container server according to the identifier information of the container to which the third target object belongs, The identification information of the third target object is used to query the location information of the third target object in the object list of the container server; if the location information of the third target object is not queried in the object list, the container server will be the third in the location update message.
  • the first location information of the target object is written to the object list; if the location information of the third target object is queried in the object list, the location information of the third target object is the second location information, and the second target object is the second location information.
  • the location information is different from the first location information of the third target object in the location update message, the container server replaces the second location information of the third target object with the first location information of the third target object, and the third target object The second location information is written to the delete log of the container server.
  • the container server may update location information of an object (eg, the third target object described above) in the object storage system (eg, update the location of the third target object after writing the third target object) Information), in this way, when the third target object is subsequently read or deleted, the location information of the accurate third target object can be queried from the container server to correctly read the third target object, or delete the third object. target.
  • an object eg, the third target object described above
  • the container server may update location information of an object (eg, the third target object described above) in the object storage system (eg, update the location of the third target object after writing the third target object) Information), in this way, when the third target object is subsequently read or deleted, the location information of the accurate third target object can be queried from the container server to correctly read the third target object, or delete the third object. target.
  • the second aspect provides a data processing method, the method may include: the object server receives a request message of the upload target object sent by the proxy server, where the request message of the upload target object includes a third target object; The third target object is written in the object server; and the object server sends the location information of the third target object to the proxy server, where the location information of the third target object includes the file identifier of the third target object and the offset of the third target object And the size of the third target object, the file identifier indicates that at least two objects are included in the file.
  • the data processing method provided by the present application in the process of uploading an object (for example, may be referred to as a third target object) to the object storage system, because the object server writes the third target object into the file system file of the object server,
  • the file may include at least two objects. It can be known that when multiple small objects are written in the object storage system, multiple small objects can be written into one file, thereby saving the storage resources of the file system of the object server. Improve the performance of the object server.
  • the data processing method provided by the present application may further include: the object server receives location information of the second target object sent by the proxy server; and the object server is configured according to the second target object.
  • the location information reads the second target object; and the object server sends a second target object to the proxy server.
  • the data processing method provided by the application may further include: receiving, by the object server, a garbage collection request message sent by the container server, where the garbage collection request message includes a deletion log of the container server Position information of the first target object; and the object server deletes the first target object according to the location information of the first target object; and the object server sends a garbage collection response message to the container server, the garbage collection response message indicating that the first target has been deleted target.
  • the present application provides a data processing method, the method may include: the proxy server sends a request message for uploading a target object to the object server, where the request message of the upload target object includes a target object; and then the proxy server receives the object server and sends the First location information of the target object, the first location information of the target object includes a file identifier of the target object, an offset of the target object, and a size of the target object, where at least two objects are stored in the file indicated by the file identifier; And the proxy server sends a location update message to the container server, to instruct the container server to update the location information of the target object in the object list of the container server according to the location update message, where the location update message includes the identifier information and the target of the container to which the target object belongs.
  • the identification information of the object and the first location information of the target object is a request message for uploading a target object to the object server, where the request message of the upload target object includes a target object; and then the proxy server receives the object
  • the data processing method provided by the present application may further include: receiving, by the proxy server, a delete response message sent by the container server, where the delete response message is a target of the container service deletion target list The location information of the object, and the location information of the target target object is sent after being written to the delete log of the container server.
  • the identification information of the container to which the first target object belongs and the identification information of the first target object and querying the location information of the first target object in the object list of the container server, where the object list includes location information of the first type of object,
  • the first type of object is an object stored in the object server that is not marked for deletion, and the location information of the first type object is used to indicate the location of the object in the object server; the update module can be used to query the object list in the query module.
  • transmitting module may be configured to send a message deletion response to the proxy server.
  • the location information of the first target object includes a file identifier of the first target object, an offset of the first target object, and a size of the first target object, where The file identifier indicates that at least two objects are included in the file; or the location information of the first target object includes the ID of the first target object.
  • the receiving module may be further configured to receive a read query message sent by the proxy server, where the read query message includes the identifier information of the container to which the second target object belongs and the first
  • the identifier information of the second target object may be used to query the location information of the second target object in the object list according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object received by the receiving module.
  • the location information of the second target object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, where the file indicated by the file identifier includes at least two objects; the sending module
  • the container server may further send the location information of the second target object to the proxy server, to indicate the location information of the second target object according to the location information of the second target object from the object server. Read the second target object in .
  • the container server provided by the present application may further include a reading module.
  • the reading module may be configured to: if the read query message received by the receiving module includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object, the first target object in the object list is deleted in the update module The location information, and the location information of the first target object in the object list is read before the location information of the first target object is written in the deletion log of the container server.
  • the sending module may be further configured to send a garbage collection request message to the object server, where the garbage collection request message includes deleting location information of the target object in the log;
  • the module may be further configured to receive a garbage collection response message sent by the object server, where the garbage collection response message indicates that the target server has deleted the target object; and the update module may further be configured to delete the location information of the target object in the deletion log.
  • the receiving module is further configured to receive a location update message sent by the proxy server, where the location update message is used to update location information of the third target object, where the location update message is The identifier information of the container to which the third target object belongs, the identification information of the third target object, and the first location information of the third target object are included; the update module may be further configured to update the third according to the location update message received by the receiving module. The location information of the target object.
  • the query module may be further configured to query the object list of the container server according to the identifier information of the container to which the third target object belongs and the identifier information of the third target object.
  • Position information of the third target object the update module is specifically configured to: when the query module does not query the location information of the third target object in the object list, write the first location information of the third target object in the location update message to the object a list, if the query module queries the location information of the third target object in the object list, the location information of the third target object is the second location information, and the second location information of the third target object and the third location information in the location update message
  • the first location information of the target object is different, the second location information of the third target object is replaced with the first location information of the third target object, and the second location information of the third target object is written into the deletion log of the container server.
  • an object server may include a receiving module, a write data module, and a sending module.
  • the receiving module may be configured to receive a request message of the upload target object sent by the proxy server, where the request message of the upload target object includes a third target object
  • the write data module may be configured to write the third target object received by the receiving module.
  • the sending module may be configured to send, to the proxy server, location information of the third target object, where the location information of the third target object includes a file identifier of the third target object, an offset of the third target object, and a third The size of the target object, the file identifier indicates that at least two objects are included in the file.
  • the object server provided by the application may further include a reading module.
  • the receiving module may be further configured to receive location information of the second target object sent by the proxy server; the reading module may be configured to read the second target object according to the location information of the second target object received by the receiving module; Used to send a second target object to the proxy server.
  • the object server provided by the application may further include a deletion module.
  • the receiving module may be further configured to receive a garbage collection request message sent by the container server, where the garbage collection request message includes location information of the first target object in the deletion log of the container server; the deletion module may be configured to receive according to the receiving module.
  • the location information of the first target object deletes the first target object;
  • the sending module may be further configured to send a garbage collection response message to the container server, where the garbage collection response message indicates that the first target object has been deleted.
  • a proxy server which can include a sending module and a receiving module.
  • the sending module may be configured to send a request message for uploading the target object to the object server, where the request message of the upload target object includes a target object
  • the receiving module may be configured to receive first location information of the target object sent by the object server, where the target The first location information of the object includes a file identifier of the target object, an offset of the target object, and a size of the target object, where at least two objects are stored in the file indicated by the file identifier; the sending module may be further configured to send to the container server.
  • a location update message to instruct the container server to update the location information of the target object in the object list of the container server according to the location update message, where the location update message includes the identification information of the container to which the target object belongs, the identification information of the target object, and the target object.
  • the receiving module is further configured to receive a deletion response message sent by the container server, where the deletion response message is a location information of the target object in the container service deletion object list, and The location information of the target target object is sent after being written to the delete log of the container server.
  • a container server in a seventh aspect, can include a processor and a memory coupled to the processor. This memory can be used to store computer instructions.
  • the processor executes the computer instructions stored by the memory to cause the container server to perform the data processing method of any of the above first aspects and various alternative implementations thereof.
  • a computer readable storage medium comprising computer instructions.
  • the container server is caused to perform the data processing method of any of the above first aspects and various alternative implementations thereof.
  • a computer program product comprising computer instructions for causing a container server to perform any of the first aspect described above and various alternative implementations thereof when the computer program product is run on a container server The data processing method described.
  • an object server can include a processor and a memory coupled to the processor.
  • This memory can be used to store computer instructions.
  • the processor executes the computer instructions stored by the memory to cause the object server to perform the data processing method of any of the above second aspects and various alternative implementations thereof.
  • a computer readable storage medium comprising computer instructions.
  • the object server is caused to perform the data processing method of any of the above second aspects and various alternative implementations thereof.
  • a computer program product comprising computer instructions, when the computer program product is run on an object server, causes the object server to perform any one of the second aspect described above and various alternative implementations thereof The data processing method described.
  • a proxy server in a thirteenth aspect, can include a processor and a memory coupled to the processor. This memory can be used to store computer instructions.
  • the processor executes the computer instructions stored by the memory to cause the proxy server to perform the data processing method of any of the above third aspects and various alternative implementations thereof.
  • a computer readable storage medium comprising computer instructions.
  • the proxy server is caused to perform the data processing method of any of the above third aspects and various alternative implementations thereof.
  • a computer program product comprising computer instructions, when executed on a proxy server, causes the proxy server to perform any one of the above third aspects and various alternative implementations thereof The data processing method described.
  • an object storage system may include the container server according to any one of the foregoing fourth aspects and various alternative implementations thereof, the fifth aspect and various The object server of any one of the selected implementations, and the proxy server of any of the sixth aspect and various alternative implementations thereof.
  • the object storage system system may include the container server in the seventh aspect, the object server in the tenth aspect, and the proxy server in the thirteenth aspect.
  • FIG. 1 is a schematic structural diagram of an object storage system according to an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of hardware of various servers according to an embodiment of the present invention.
  • FIG. 3 is a schematic diagram 1 of a data processing method according to an embodiment of the present invention.
  • FIG. 4 is a second schematic diagram of a data processing method according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram 3 of a data processing method according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic diagram 4 of a data processing method according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic structural diagram 2 of a container server according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram 1 of an object server according to an embodiment of the present disclosure.
  • FIG. 10 is a schematic structural diagram 2 of an object server according to an embodiment of the present disclosure.
  • FIG. 11 is a schematic structural diagram 1 of a proxy server according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram 2 of a proxy server according to an embodiment of the present invention.
  • first and second in the specification and claims of the embodiments of the present invention are used to distinguish different objects, and are not intended to describe a specific order of the objects.
  • first target object and the second target object are used to distinguish different target objects, rather than to describe a specific order of the target objects.
  • the words “exemplary” or “such as” are used to mean an example, illustration, or illustration. Any embodiment or design described as “exemplary” or “for example” in the embodiments of the invention should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the words “exemplary” or “such as” is intended to present the concepts in a particular manner.
  • a plurality means two or more unless otherwise stated.
  • a plurality of processing units refers to two or more processing units;
  • a plurality of systems refers to two or more systems.
  • Container In the field of object storage, it can also be called a bucket, in which information about an object is stored (for example, it can be called a list of objects, including location information of multiple objects, such as the ID of the object, and the offset of the object in a file) And the size of the object, etc.), the container is located in the container server.
  • information about an object for example, it can be called a list of objects, including location information of multiple objects, such as the ID of the object, and the offset of the object in a file) And the size of the object, etc.
  • the object In the object storage domain, the object is the storage unit, the object is stored in the object server in the object storage system, the object includes the object metadata and the object data, and the object metadata is used to describe the information of the object's attributes, such as the size of the object. Or the type of the object (such as pictures, audio, video, etc.), etc., the object data is the object itself.
  • the data processing method provided by the embodiment of the present invention can be applied to an object storage system.
  • a schematic diagram of the architecture of the object storage system refer to the schematic diagram of the object storage system shown in FIG. 1 , where the proxy server 11 is responsible for the object storage system. Inter-communication between the remaining components, for example, the proxy server 11 can receive hypertext transfer requests (e.g., read requests, delete requests, upload requests, etc.) of the client, and can forward requests from clients (e.g., terminal 10 in Figure 1). The message is to the container server 12 or the object server 13 in the object storage system.
  • hypertext transfer requests e.g., read requests, delete requests, upload requests, etc.
  • the information of the container and the information of the object are maintained (considered to be stored) in the database of the container server 12, and the metadata and data of the plurality of objects are stored in the file system of the object server 13.
  • an operation of uploading (ie, writing) an object, reading an object, and deleting an object may be implemented through interaction between the proxy server 11, the container server 12, and the object server 13.
  • the terminal 10 can be a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (personal).
  • Digital assistant (PDA) smart car, sensing device, internet of thing (IOT) equipment, customer premise equipment (CPE), etc.
  • the proxy server 11, the container server 12, and the object server 13 shown in FIG. 1 may be a hardware server or a software server.
  • the proxy server 11, the container server 12, and the object server 13 are hardware. Server, introduces the hardware structure and software function modules of each server.
  • the proxy server 11 , the container server 12 , and the object server 13 may be servers of the same structure or servers of different configurations.
  • the components of the proxy server, the container server, and the object server are respectively described below in conjunction with the object storage system shown in FIG.
  • FIG. 2 is a schematic diagram of hardware of each server according to an embodiment of the present invention.
  • the proxy server 100 shown in FIG. 2 may include: a processor 101, a memory 102, and a communication interface 103.
  • the processor 101 is a core component of the proxy server and is used to run the server's operating system and applications on the server (including system applications and third-party applications).
  • the processor 101 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), or an application-specific integrated circuit (ASIC). a field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof, which may implement or perform various aspects described in the disclosure of the embodiments of the present invention.
  • exemplary logical blocks, modules and circuits; the processor may also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like.
  • Memory 102 program code and data for storing a proxy server.
  • the memory 102 may specifically include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory.
  • a volatile memory such as a random-access memory (RAM)
  • the memory may also include a non-volatile memory.
  • ROM read-only memory
  • flash memory flash memory
  • HDD hard disk
  • SSD solid-state disk
  • the memory may further include the above-described type of memory. The combination.
  • the communication interface 103 is an interface circuit for the proxy server to communicate with other devices.
  • the communication interface may be a transceiver, a transceiver circuit, or the like having a transceiving function, and the communication interface includes a serial communication interface and a parallel communication interface.
  • the proxy server 100 can communicate with a client (e.g., terminal 10 in FIG. 2) via the communication interface 103, with the container server 200, or with the object server 300.
  • the container server 200 shown in FIG. 2 may include a processor 201, a memory 202, and a communication interface 203.
  • the processor 201 refer to the description of the processor 101 of the proxy server 100, and details are not described herein again.
  • the processor 101 of the container server may process the read query message or delete the query message.
  • the memory 202 of the container server 200 may include at least one container (for example, M containers shown). M ⁇ 1)), each container may include a list of objects and a delete log, where the object list stores location information of a plurality of objects (which may be referred to as first type objects), and the first type of objects are unmarked and deleted.
  • the object stored in the object server deletes the location information of a plurality of objects (which may be referred to as a second type of object), and the second type of object is the location information of the object stored in the object server that is marked and deleted.
  • the communication interface 203 For a description of the communication interface 203, reference may be made to the description of the communication structure 103 of the proxy server 100, in which the container server may communicate with the proxy server 100 via the communication interface 203 or with the object server 300.
  • the file 302 of the object server 300 may include a file system, and the file system includes at least one file (for example, The file system shown in FIG. 2 includes N files (N ⁇ 1), and each of the at least one file can store a plurality of objects (for example, each of the files shown in FIG. 2 can include P objects). (P ⁇ 2)).
  • the communication interface 303 For a description of the communication interface 303, reference may be made to the description of the communication structure 103 of the proxy server 100, in which the object server 300 may communicate with the proxy server 100 via the communication interface 303 or with the container server 200.
  • the object server needs to delete the object according to the location information of the object, and when the object storage system has both read an object (for example, the first object) Request, there is also a request to delete the object, because the object in the object server may have been marked as deleted when the object is read, so the failure to read the object may occur, which leads to the first reading
  • the object conflicts with deleting the first object.
  • the embodiment of the invention provides a data processing method, device and system, which can avoid conflicts between a read object and a deleted object to a certain extent.
  • the user equipment may delete an object stored in the object storage system, or may read an object stored in the object storage system, and may also upload the object to the object storage system, and firstly through the object storage.
  • the interaction process between the various servers in the system describes three different application scenarios of deleting objects, reading objects, and uploading objects.
  • the scene of the deleted object is recorded as the scene 1
  • the scene of the read object is recorded as the scene 2
  • the scene of the uploaded object is recorded as the scene 3.
  • the data processing method provided by the embodiment of the present invention may include S101-S105:
  • the proxy server sends a delete query message to the container server, where the delete query message includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object.
  • the terminal when the terminal needs to delete an object in the object storage system (hereinafter may be referred to as a first target object), the terminal may send a request message for deleting the first target object to the proxy server, and the proxy server receives the terminal.
  • the proxy server After the sent request message for deleting the first target object is sent, and the request message for deleting the first target object is parsed, and the identifier information of the container to which the first target object belongs and the identification information of the first target object are obtained, the proxy server may The identifier information of the container to which the target object belongs and the identifier information of the first target object are carried in the delete query message and sent to the container server, where the location information of the object is stored in the container, and the location information is used to indicate the target server.
  • the location of the object, the identifier information of the container to which the first target object belongs may identify in which container the location of the first target object to be deleted is stored, and the identifier information of the first target object may indicate that the target object to be deleted is in the container Which target object.
  • the identifier information of the first target object may be a name of the first target object or other information that may be specified by the user, which may be a predetermined target, which is not specifically limited.
  • the identifier information of the container to which the first target object belongs may also be the name of the container or other information defined by the user, which may be instructed by the user.
  • the container server receives a delete query message sent by the proxy server.
  • the container server queries the location information of the first target object in the object list of the container server according to the identifier information of the container to which the first target object belongs and the identifier information of the first target object in the delete query message.
  • the object list of the container server includes location information of the first type of object, and the first type of object is an object stored in the object server that is not marked and deleted, and the location information of each first type of object is used to indicate each The location of the object in the object server.
  • the location information of the object is also different.
  • the object is The location information is the ID of the object; if at least two objects are stored in one file, it can be understood that at least two objects are stored in each file, and the location information of the object is the file identifier of the object, the offset of the object, and The size of the object, the offset of the object is the position of the object in its stored file, and the size of the object identifies the size of the object, such as the number of bits occupied by the object.
  • the location information of the first target object may include an ID of the first target object, or the location information of the first target object may include a file identifier of the first target object, an offset of the first target object, and the first The size of the target object, which includes at least two objects in the file indicated by the file identifier.
  • the location information of the object may be jointly represented by the ID of the container and the ID of the object, and the object server
  • the range of objects that can be represented by the ID of the object may not satisfy the number of objects, that is, the actual number of objects exceeds the number of objects that can be represented by the ID of the object. Therefore, the ID of the container and the ID of the object can be used. Two fields identify the location information of the object to indicate the object.
  • the object information of the container server may store location information of multiple objects, that is, multiple pieces of location information may be stored, and each piece of location information corresponds to location information of one object, and each piece of location information may correspond to a location index, and the information tuple composed of the identification information of the container to which the object belongs and the identification information of the object also corresponds to a location index, and the container server according to the identification information of the container to which the first target object belongs and the identifier of the first target object
  • the method for querying the location information of the first target object in the object list of the container server is specifically: the container server determines the location index of the first target according to the identifier information of the container to which the first target object belongs and the identification information of the first target object, The location information of the first target object can then be determined based on the index of the location of the first target object.
  • the information binary consisting of the identification information of the container to which the object in the container server belongs and the identification information of the object (Table 1) An example of the correspondence between the positional index of the object and the positional index of the object and the positional index of the object.
  • object1 represents the identifier of the object 1
  • container1 represents the identifier of the container to which the object 1 belongs.
  • the container server deletes the location information of the first target object in the object list, and writes the location information of the first target object to the deletion of the container server. In the log.
  • the deletion log includes location information of the second type of object, and the second type of object is location information of the object stored in the object server that is marked and deleted.
  • the container server after the container server deletes the location information of the first target object from the object list, and after the location information of the first target object is written into the deletion log, the container server subsequently receives other information about the first target object.
  • the container server When querying a message (for example, reading a query message), the container server will not find the location information of the first target object in the object list, and thus cannot perform corresponding operations on the first target object.
  • the first target object is considered to be deleted successfully.
  • the content of the first target object that is, the metadata of the first target object and the data itself have not been deleted (ie, not recovered), and are also stored in the object server.
  • the location information of the object in the deletion log of the container server indicates the object to be reclaimed in the object server, and the location information of the first target object is written into the deletion log, which can be understood as the first target to be deleted.
  • the object is first marked, and at the appropriate time, the content of the first target object is recovered from the object server based on the location information of the first target object in the deletion log.
  • the container server sends a delete response message to the proxy server.
  • the deletion response message may include indication information indicating that the deletion of the first target object is successful, to indicate that the container server successfully deletes the first target object.
  • the container server may send, to the proxy server, indication information including location information indicating that the first target object is not queried, to indicate The container server does not query the location information of the first target object, that is, the deletion operation ends.
  • the container server may query the object list of the container server.
  • the location information of the first target object and in the case where the location information of the first target object is queried in the object list, the location information of the first target object is deleted from the object list, and the location of the first target object is Information is written to the delete log of the container server.
  • the container server queries the location information of the first target object in the object list
  • the location information of the first target object does not need to be sent to the proxy server
  • the proxy server does not need to be Sending the location information of the first target object to the object server, so that the object server does not need to delete the first target object in the object server, so when deleting the first target object and reading in the object storage system
  • the object can be read in the object server.
  • the proxy server since the container server does not need to send the location information of the first target object to the proxy server, the proxy server does not need to send the location information of the first target object to the object server, so that the object server does not need to be in the object.
  • the first target object in the server is marked for deletion, so that the process of deleting the first target object can be simplified, and the communication resources between the servers can be saved.
  • the data processing method provided by the embodiment of the present invention may include S201-S209:
  • the proxy server sends a read query message to the container server, where the read query message includes the identifier information of the container to which the second target object belongs and the identifier information of the second target object.
  • the terminal when the terminal needs to read an object in the object storage system (which may be referred to as a second target object), the terminal may send a request message for reading the second target object to the proxy server, and the proxy server receives the request message.
  • the proxy server may Carrying the identification information of the container to which the second target object belongs and the identification information of the second target object in the read query message, and sending the information to the container server to obtain the location information of the second target object, thereby reading the second target object.
  • the container service receives a read query message sent by the proxy server.
  • the container server queries the location information of the second target object in the object list of the container server according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object.
  • the location information of the second target object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, where the file identifier includes at least two files. Object.
  • the container server sends the location information of the second target object to the proxy server.
  • the container server sends the location information of the second target object to the proxy server, and may instruct the proxy server to read the second target object from the object server according to the location information of the second target object.
  • the proxy server receives location information of the second target object sent by the container server.
  • the proxy server sends the location information of the second target object to the object server.
  • the object server receives location information of the second target object sent by the container server.
  • the object server reads the second target object according to the location information of the second target object.
  • the object server sends a second target object to the proxy server.
  • the location information of the second target object that can be received by the proxy server is sent to the object server, so that the object server can read the second target object in the corresponding location in the object server according to the location information of the second target object. And sending the second target object to the proxy server, and then the proxy server sends the second target object to the terminal, so that the terminal can smoothly read the second target object.
  • the data processing method provided by the embodiment of the present invention may further include S301-S308:
  • the proxy server sends a request message for uploading a third target object to the object server, where the request message for uploading the third target object includes a third target object.
  • the terminal when the terminal needs to upload an object to the object storage system (for example, it may be referred to as a third target object), the terminal may send an upload request message to the proxy server, where the upload request message includes the third target object.
  • the identification information of the container and the identification information of the third target object, the identification information of the container to which the third target object belongs is used to indicate in which container the third target object is updated after the third target object is written to the object server Location information; the proxy server may send the third target to the object server in the request message for uploading the third target object.
  • the object server receives a request message that is sent by the container server and uploads the third target object.
  • the object server writes the third target object into a file of the object server.
  • the file written to the third target object can support writing multiple objects, and the multiple objects are small objects, and the small object can be understood as the space occupied by the object is relatively small, that is, the size of the object is relatively small ( For example, an image, a small text, etc. can be called a small object.
  • each file contains only one object, and multiple objects (here, will refer to multiple small objects). Writing to a large file can save the storage resources of the file system of the object server, thereby improving the performance of the object server.
  • each file contains only one object.
  • the object server reclaims multiple small objects, multiple space fragments are generated in the server (ie, small files after each small object are reclaimed).
  • multiple small objects are written into one large file, so that when small objects in the large file are recovered, space fragmentation in the server can be reduced, thereby writing a large object (occupied)
  • the storage space is relatively large, it is not necessary to divide the large object into multiple small files, and the entire contents of the large object can be written into a large file, thereby improving the performance of the object storage system.
  • the object server sends the first location information of the third target object to the proxy server.
  • the first location information of the third target object includes a file identifier of the third target object, an offset of the third target object, and a size of the third target object, where the file identifier of the third target object is in the file indicated by the file identifier Save at least two objects.
  • the object server writes the third target object to a certain location of the object server, and if the location is called the first location, the object server may generate the location information of the third target object, because the object server is in the current time.
  • the third target object may already be stored in the object server (possibly the third target object to be written this time is a modified version of the stored third target object), so in order to distinguish the present
  • the positional position information of the third target object that has been written in the second time and the positional information of the third target object that has been previously stored are recorded as the first positional information of the third target object.
  • the proxy server receives first location information of the third target object sent by the object server.
  • the proxy server sends the location update message to the container server.
  • the location update message is used to update location information of the target object in the object list of the container server.
  • the proxy server may carry the identifier information of the container to which the target object belongs, the identifier information of the target object, and the first location information of the target object in the location update message. Is sent to the container server to instruct the container server to update the location information of the target object in the object list of the container server according to the location update message.
  • the container server receives a location update message sent by the proxy server.
  • the container server updates the location information of the third target object in the object list of the container server according to the location update message.
  • S308a-S308b may be specifically implemented by using S308a-S308b:
  • the container server queries the location information of the third target object in the object list of the container server according to the identifier information of the container to which the third target object belongs and the identifier information of the third target object.
  • S308b The container server updates the location information of the third target object according to the query result.
  • the query result of the container server querying the location information of the third target object in the object list may include: the container server does not query the location information of the third target object in the object list, or the container server is in the object list.
  • Query the location information of the third target object may include the following 308b1 or S208b2:
  • the third target object is written in the object server in the object storage system, if the location information of the third target object does not exist in the container server, the first location information of the third target object is obtained.
  • the object list is written, so that in the subsequent process of reading or deleting the third target object, the third target object can be read or deleted by querying the location information of the third target object in the object list.
  • the container server replaces the second location information of the third target object with the first location information of the third target object, and writes the second location information of the third target object to the deletion log of the container server in.
  • the third target object is already stored in the object server when the third target object is written to the object server.
  • the third target object to be written is a modified version of the stored third target object. Therefore, the second location information of the third target object may be replaced with the first location information of the third target object, specifically,
  • the container server deletes the second location information of the third target object, and writes the first location information of the third target object into the object list, such that in subsequent operations (eg, reading the third target object or deleting the third During the target object, the information of the accurate third target object can be queried in the container server to correctly read the third target object or delete the third target object.
  • the second location information of the third target object may be written into the deletion log of the container server, and the object server recovers the second location information of the third target in the object server according to the second location information of the third target object in the deletion log.
  • the proxy server may send a request message for uploading the third target object to the object server, and the object Receiving, by the server, the request message for uploading the third target object, and writing the third target object in the request message to the object server, and sending the first location information of the third target object to the proxy server, the third target
  • the first location information of the object includes a file identifier of the third target object, an offset of the third target object, and a size of the third target object, where the file identifier indicated by the third target object stores at least two files.
  • the file may include at least two objects, and it is known that when the object storage system writes When multiple small objects are entered, multiple small objects can be written to one file. This saves the storage resources of the file system of the object server, thereby improving the performance of the object server.
  • the first target object in the scenario 1 , the second target object in the scenario 2, and the third target object in the scenario 3 may be the same target object, or may be different targets.
  • Object If the first target object, the second target object, and the third target object are the same target object, for example, each is referred to as the first target object, the first target object may be written by the description of the three scenarios in the above embodiment.
  • the object storage system is such that the first target object can be read from the object storage system or the first target object can be deleted.
  • the container server when the container server deletes the location information of an object from the object list, and writes the location information of the object to the deletion log, the container server may The interaction of the object server, which recycles the object in the object server.
  • the following takes the first target object as an example, and exemplifies the method of recycling the object.
  • the data processing method may also include S401-S406:
  • the container server sends a garbage collection request message to the object server.
  • the garbage collection request message includes location information of the first target object in the deletion log.
  • the spam request message is used to request the object server to reclaim the content of the first target object in the object server (ie, including the metadata and data of the first target object).
  • the container server after the container server deletes the location information of the first target object from the object list, and the location information of the first target object is written into the deletion log, the container server is in a preset time interval.
  • the object server sends a garbage collection request message, or the container server can send a garbage collection request to the object server to reclaim the object in the object server (also known as garbage) when the remaining storage space of the object server is relatively small.
  • the object server receives a garbage collection request message sent by the container server.
  • the object server deletes the first target object according to the location information of the first target object in the garbage collection request message.
  • the object server may reclaim the garbage object in the object server, where the garbage object is an object indicated by the location information of the object in the deletion log of the container server, and the method for the object server to collect the first target object is: The location information of the first target object in the garbage collection request message deletes the first target object, thereby releasing the storage space of the object server.
  • the object server sends a garbage collection response message to the container server, where the garbage collection response message indicates that the first target object has been deleted.
  • the container server receives the garbage collection response message sent by the object server.
  • the container server deletes the location information of the first target object in the deletion log of the container server.
  • the container server may delete the location information of the first target object in the deletion log, thereby completing the recovery of the first target object, and the object storage The first target object no longer exists in the system.
  • the object storage system when an object is read and the object is deleted concurrently (that is, it can be understood that the object storage system processes the event of reading an object, and also needs to be processed.
  • the location information of the first target object is received from the object in the container server according to the method of reading the object and deleting the object in the foregoing embodiment. After the list is deleted, and the location information of the first target object is written in the delete log of the container server, the container server has read the location information of the first target object in the object list, and then the first target object is read this time. In the process of deleting and deleting the first target object, even if the location information of the first target object in the container server has been deleted, and the location information of the first target object has been written into the deletion log, this time can be read. Go to the first target object.
  • the location information of the first target object in the container server has been deleted, and the location information of the first target object has been written into the deletion log, and the first target object can also be read this time.
  • the reason is: in the process of deleting the first target object, the location information of the first target object may be written in the deletion log to indicate that the first target object is marked for deletion, and no need to be in the object server. The first target object is marked as deleted. Therefore, the first target object still exists in the object server and has not been reclaimed.
  • the object server may obtain the first target object according to the read query request message according to the container server. Location information, the first target object is read in the object server.
  • the data processing method provided by the embodiment of the present invention can avoid conflicts between the read object and the deleted object to a certain extent (refer to when the read object and the deleted object occur concurrently, the operation of deleting the object firstly causes the read object to be read. The operation could not read the object).
  • the process of deleting the object, reading the object, uploading the object, and reclaiming the object may be completed in the user state, which can reduce the difficulty in developing and maintaining the object storage system.
  • the solution provided by the embodiment of the present invention is mainly introduced from the perspective of interaction between the network elements.
  • various network elements such as a container server, an object server, and a proxy server, etc.
  • a container server such as a container server, an object server, and a proxy server, etc.
  • the embodiments of the present invention can be implemented in a combination of hardware or hardware and computer software in combination with the elements and algorithm steps of the various examples described in the embodiments disclosed herein. Whether a function is implemented in hardware or computer software to drive hardware depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present application.
  • the embodiments of the present invention may divide the function modules of the container server, the object server, the proxy server, and the like according to the foregoing method examples.
  • each function module may be divided according to each function, or two or more functions may be integrated into one.
  • Processing module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of the module in the embodiment of the present invention is schematic, and is only a logical function division, and the actual implementation may have another division manner.
  • FIG. 7 is a schematic diagram of a possible structure of the container server involved in the foregoing embodiment.
  • the container server may include: a receiving module 30, The query module 31, the update module 32, and the sending module 33.
  • the receiving module 30 can be used to support the container server to execute S102, S202, S307, and S405 in the foregoing method embodiment;
  • the query module 31 can be used to support the container server to execute S103, S203 in the foregoing method embodiment;
  • the update module 32 can S104, S308 (including S308a-S308b) and S406 are used to support the container server in the foregoing method embodiment;
  • the sending module 33 can be used to support the container server to execute S105, S204 and S401 in the above method embodiment.
  • the container server may further include a reading module 34.
  • the reading module 34 can be used to support the container server to perform the above-mentioned embodiment: "If the read query message received by the receiving module 30 includes the identification information of the container to which the first target object belongs and the identification information of the first target object, the update is performed. The module deletes the location information of the first target object in the object list, and reads the location information of the first target object in the object list before writing the location information of the first target object to the deletion log of the container server. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.
  • FIG. 8 shows a possible structural diagram of the container server involved in the above embodiment.
  • the container server may include a processing module 40 and a communication module 41.
  • the processing module 40 can be used to control and manage the actions of the container server.
  • the processing module 40 can be used to support the container server to execute S103, S104, S203, S308 (including S308a-S308b) and S406 in the foregoing method embodiment, and / or other processes for the techniques described herein.
  • the communication module 41 can be used to support communication between the container server and other network entities.
  • the communication module 41 can be used to support the container server to execute S102, S105, S202, S204, S307, S401, and S405 in the foregoing method embodiments.
  • the container server may further include a storage module 42 for storing program code and data of the container server.
  • the processing module 40 may be a processor or a controller (for example, the processor 201 shown in FIG. 2 above), and may be, for example, a central processing unit (CPU), a general-purpose processor, and a digital signal processor. (digital signal processor, DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware component or any combination. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the embodiments of the invention.
  • the above processors may also be a combination of computing functions, such as one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like.
  • the communication module 41 may be a transceiver, a transceiver circuit, or a communication interface or the like (for example, may be the communication interface 203 shown in FIG. 2 described above).
  • the storage module 42 can be a memory (which can be, for example, the memory 202 shown above in FIG. 2).
  • the processing module 40 is a processor
  • the communication module 41 is a transceiver
  • the storage module 42 is a memory
  • the processor, the transceiver, and the memory can be connected by a bus.
  • the bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the bus can be divided into an address bus, a data bus, a control bus, and the like.
  • FIG. 9 is a schematic diagram of a possible structure of the object server involved in the foregoing embodiment.
  • the object server may include: a receiving module 50, The data module 51 and the transmitting module 52 are written.
  • the receiving module 50 can be used to support the object server to execute S207, S302, and S402 in the foregoing method embodiment;
  • the write data module 51 can be used to support the object server to execute S303 in the foregoing method embodiment;
  • the sending module 52 can be used to support the object server.
  • S209, S304, and S404 in the above method embodiment are executed.
  • the object server may further include a reading module 53 and a deleting module 54.
  • the reading module 53 can be used to support the object server to execute S208 in the above method embodiment; the deleting module 54 can be used to support the object server to execute S403 in the above method embodiment. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.
  • the processing module 60 may be a processor or a controller (for example, the processor 301 shown in FIG. 2 above), and may be, for example, a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor. Logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the embodiments of the invention.
  • the above processors may also be a combination of computing functions, such as one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like.
  • the communication module 61 may be a transceiver, a transceiver circuit or a communication interface or the like (for example, may be the communication interface 303 shown in FIG. 2 described above).
  • the storage module 62 can be a memory (which can be, for example, the memory 302 shown above in FIG. 2).
  • the processing module 60 is a processor
  • the communication module 61 is a transceiver
  • the storage module 62 is a memory
  • the processor, the transceiver, and the memory can be connected by a bus.
  • the bus can be a PCI bus or an EISA bus.
  • the bus can be divided into an address bus, a data bus, a control bus, and the like.
  • FIG. 11 is a schematic diagram showing a possible structure of the proxy server involved in the foregoing embodiment.
  • the proxy server may include: a sending module 70 and Receive module 71.
  • the sending module 70 can be used to support the proxy server to execute S101, S201, S206, S301, and S306 in the foregoing method embodiment;
  • the receiving module 50 can be used to support the proxy server to execute S205 and S305 in the foregoing method embodiment. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.
  • FIG. 12 shows a possible structural diagram of the proxy server involved in the above embodiment.
  • the proxy server may include a processing module 80 and a communication module 81.
  • the processing module 80 can be used to control and manage the actions of the proxy server.
  • the communication module 81 can be used to support communication between the proxy server and other network entities.
  • the communication module 81 can be used to support the proxy server to execute S101, S201, S206, S205, S301, S305, and S306 in the foregoing method embodiments.
  • the proxy server may further include a storage module 82 for storing program code and data of the proxy server.
  • the processing module 80 may be a processor or a controller (for example, the processor 101 shown in FIG. 2 above), and may be, for example, a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor. Logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the embodiments of the invention.
  • the above processors may also be a combination of computing functions, such as one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like.
  • the communication module 81 may be a transceiver, a transceiver circuit or a communication interface or the like (for example, may be the communication interface 103 as shown in FIG. 2 described above).
  • the storage module 82 can be a memory (which can be, for example, the memory 102 shown above in FIG. 2).
  • the processing module 80 is a processor
  • the communication module 81 is a transceiver
  • the storage module 82 is a memory
  • the processor, the transceiver, and the memory can be connected by a bus.
  • the bus can be a PCI bus or an EISA bus.
  • the bus can be divided into an address bus, a data bus, a control bus, and the like.
  • the above embodiments it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
  • a software program it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present invention are generated in whole or in part.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be wired from a website site, computer, server or data center (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) to another website, computer, server or data center.
  • the computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (such as a solid state drives (SSD)).
  • a magnetic medium for example, a floppy disk, a magnetic disk, a magnetic tape
  • an optical medium for example, a digital video disc (DVD)
  • a semiconductor medium such as a solid state drives (SSD)
  • the disclosed system, apparatus, and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the modules or units is only a logical function division.
  • there may be another division manner for example, multiple units or components may be used. Combinations can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • a computer readable storage medium A number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to perform all or part of the steps of the methods described in various embodiments of the present application.
  • the foregoing storage medium includes: a flash memory, a mobile hard disk, a read only memory, a random access memory, a magnetic disk, or an optical disk, and the like, which can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

Selon les modes de réalisation, la présente invention concerne le domaine du stockage informatique et porte ainsi sur un procédé de traitement de données qui peut éviter, dans une certaine mesure, la survenue d'un conflit entre un objet de lecture et un objet de suppression. Le procédé comprend les étapes suivantes : dans un système de stockage d'objets, un serveur de conteneur reçoit un message de demande de suppression envoyé par un serveur mandataire, le message de demande de suppression comprenant des informations d'identification d'un conteneur auquel appartient un premier objet cible et des informations d'identification du premier objet cible ; le serveur de conteneur demande des informations de localisation du premier objet cible dans une liste d'objets du serveur de conteneur selon les informations d'identification du conteneur auquel appartient le premier objet cible et les informations d'identification du premier objet cible ; si les informations de localisation du premier objet cible sont trouvées dans la liste d'objets, le serveur de conteneur supprime les informations de localisation du premier objet cible dans la liste d'objets et écrit les informations de localisation du premier objet cible dans un journal de suppression du serveur de conteneur ; et le serveur de conteneur envoie un message de réponse de suppression au serveur mandataire.
PCT/CN2018/105965 2017-09-27 2018-09-17 Procédé, dispositif et système de traitement de données Ceased WO2019062572A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710890580.X 2017-09-27
CN201710890580.XA CN107786638B (zh) 2017-09-27 2017-09-27 一种数据处理方法、装置及系统

Publications (1)

Publication Number Publication Date
WO2019062572A1 true WO2019062572A1 (fr) 2019-04-04

Family

ID=61434093

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/105965 Ceased WO2019062572A1 (fr) 2017-09-27 2018-09-17 Procédé, dispositif et système de traitement de données

Country Status (2)

Country Link
CN (1) CN107786638B (fr)
WO (1) WO2019062572A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111966867A (zh) * 2020-08-18 2020-11-20 北京金山云网络技术有限公司 对象删除方法、数据处理方法及装置
CN112948694A (zh) * 2021-03-31 2021-06-11 北京奇艺世纪科技有限公司 数据处理方法、装置、电子设备及可读存储介质
CN113342274A (zh) * 2021-06-10 2021-09-03 北京字节跳动网络技术有限公司 一种数据处理方法及装置
CN114911574A (zh) * 2022-05-16 2022-08-16 联想(北京)有限公司 一种数据处理方法及装置

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107786638B (zh) * 2017-09-27 2020-04-14 华为技术有限公司 一种数据处理方法、装置及系统
CN108628664A (zh) * 2018-05-11 2018-10-09 北京辰森世纪科技股份有限公司 一种容器处理的方法及装置
CN109088913B (zh) * 2018-06-29 2021-05-11 华为技术有限公司 请求数据的方法和负载均衡服务器
CN110896408B (zh) * 2018-09-13 2021-08-13 华为技术有限公司 一种数据处理方法及服务器集群
CN110543325A (zh) * 2019-08-15 2019-12-06 中国银行股份有限公司 数据处理方法及装置
CN113950145B (zh) * 2021-12-20 2022-03-08 支付宝(杭州)信息技术有限公司 数据处理方法及装置
CN117194298B (zh) * 2023-09-18 2024-05-31 上海鸿翼软件技术股份有限公司 一种控制方法、装置、设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130013780A1 (en) * 2007-09-03 2013-01-10 Sony Corporation Content receiving device and method
CN105100149A (zh) * 2014-05-13 2015-11-25 中国电信股份有限公司 用于管理文件的方法和系统
CN105338113A (zh) * 2015-11-25 2016-02-17 北京航空航天大学 一种针对城市数据资源共享的多平台数据互联系统
CN105677250A (zh) * 2016-01-04 2016-06-15 北京百度网讯科技有限公司 对象存储系统中的对象数据的更新方法和更新装置
CN107105050A (zh) * 2017-05-11 2017-08-29 北京奇艺世纪科技有限公司 一种业务对象的存储、下载方法及系统
CN107786638A (zh) * 2017-09-27 2018-03-09 华为技术有限公司 一种数据处理方法、装置及系统

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9326134B2 (en) * 2012-10-19 2016-04-26 Mcafee Inc. Data loss prevention for mobile computing devices
US9419953B2 (en) * 2012-12-23 2016-08-16 Mcafee, Inc. Trusted container
CN107667517B (zh) * 2015-06-03 2021-03-19 瑞典爱立信有限公司 用于能够实现第二容器上的反向代理的在第一服务容器内的被植入代理器

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130013780A1 (en) * 2007-09-03 2013-01-10 Sony Corporation Content receiving device and method
CN105100149A (zh) * 2014-05-13 2015-11-25 中国电信股份有限公司 用于管理文件的方法和系统
CN105338113A (zh) * 2015-11-25 2016-02-17 北京航空航天大学 一种针对城市数据资源共享的多平台数据互联系统
CN105677250A (zh) * 2016-01-04 2016-06-15 北京百度网讯科技有限公司 对象存储系统中的对象数据的更新方法和更新装置
CN107105050A (zh) * 2017-05-11 2017-08-29 北京奇艺世纪科技有限公司 一种业务对象的存储、下载方法及系统
CN107786638A (zh) * 2017-09-27 2018-03-09 华为技术有限公司 一种数据处理方法、装置及系统

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111966867A (zh) * 2020-08-18 2020-11-20 北京金山云网络技术有限公司 对象删除方法、数据处理方法及装置
CN112948694A (zh) * 2021-03-31 2021-06-11 北京奇艺世纪科技有限公司 数据处理方法、装置、电子设备及可读存储介质
CN112948694B (zh) * 2021-03-31 2023-06-27 北京奇艺世纪科技有限公司 数据处理方法、装置、电子设备及可读存储介质
CN113342274A (zh) * 2021-06-10 2021-09-03 北京字节跳动网络技术有限公司 一种数据处理方法及装置
CN113342274B (zh) * 2021-06-10 2024-01-30 北京火山引擎科技有限公司 一种数据处理方法及装置
CN114911574A (zh) * 2022-05-16 2022-08-16 联想(北京)有限公司 一种数据处理方法及装置

Also Published As

Publication number Publication date
CN107786638A (zh) 2018-03-09
CN107786638B (zh) 2020-04-14

Similar Documents

Publication Publication Date Title
WO2019062572A1 (fr) Procédé, dispositif et système de traitement de données
US9792306B1 (en) Data transfer between dissimilar deduplication systems
US8650159B1 (en) Systems and methods for managing data in cloud storage using deduplication techniques
CN108829344A (zh) 数据存储方法、装置及存储介质
CN106302609B (zh) 一种访问方法及装置
US10901643B2 (en) Using log objects in object storage for durability of file objects in volatile memory
CN105205082A (zh) 用于处理hdfs中的文件存储的方法和系统
CN113900972A (zh) 一种数据传输的方法、芯片和设备
US9952933B1 (en) Fingerprint change during data operations
CN107704202A (zh) 一种数据快速读写的方法和装置
US11392545B1 (en) Tracking access pattern of inodes and pre-fetching inodes
US10846301B1 (en) Container reclamation using probabilistic data structures
US11455117B2 (en) Data reading method, apparatus, and system, avoiding version rollback issues in distributed system
CN113568566A (zh) 利用索引物件来进行简易存储服务无缝迁移的方法、主装置以及存储服务器
US11550913B2 (en) System and method for performing an antivirus scan using file level deduplication
US8392437B2 (en) Method and system for providing deduplication information to applications
EP4579423A1 (fr) Système de stockage et procédé de traitement de données
CN107911461A (zh) 云存储系统中的对象处理方法、存储服务器及云存储系统
CN110147203A (zh) 一种文件管理方法、装置、电子设备及存储介质
JP2018526740A (ja) モバイル端末のためのデータ記憶方法及び装置
WO2024119924A1 (fr) Procédé de migration de processus, appareil et système
US10235293B2 (en) Tracking access pattern of inodes and pre-fetching inodes
CN106528876B (zh) 分布式系统的信息处理方法及分布式信息处理系统
CN109542860B (zh) 基于hdfs的业务数据管理方法、终端设备
EP3519993A1 (fr) Suivi de schémas d'accès d'inodes et préextraction d'inodes

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18861027

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18861027

Country of ref document: EP

Kind code of ref document: A1