CN111338834B - Data storage method and device - Google Patents

Abstract

The embodiment of the application discloses a data storage method and device. One embodiment of the method comprises the following steps: acquiring data sent by target terminal equipment and generating metadata of the data; storing the metadata to a key-value dataset and storing the metadata to a relational dataset; if the metadata is determined to have a fault in the storage of any one of the key value data set and the relational data set, the metadata stored in the key value data set and the relational data set is stored in a preset message queue. According to the method and the device for processing the cloud service, a temporary storage function can be achieved by introducing the message queue, and the data set with storage faults is replaced to provide service for the terminal device, so that normal interaction between the terminal device and the server is ensured, and the safety of the cloud service is improved. In addition, the backup device is not needed, so that the reading flow of the data is not needed to be modified, and cost consumption caused by the backup device can be avoided while the stability of cloud service is improved.

Description

Data storage method and device

technical field

The embodiments of the present application relate to the field of computer technology, specifically to the field of Internet technology, and especially to a data storage method and device.

Background technique

With the development of Internet technology, massive amounts of network data are generated every day. Different devices in the network device cluster can implement different functions. However, there is a possibility of failures such as downtime in network equipment that is running around the clock. Once a failure occurs, there may be a situation where one device or even multiple devices cannot provide services.

In order to reduce the impact of equipment failures, related technologies pre-configure equipment clusters for data backup, but the clusters often have a low utilization rate and high maintenance costs.

Contents of the invention

The embodiments of the present application provide a data storage method and device.

In the first aspect, the embodiment of the present application provides a data storage method, including: acquiring the data sent by the target terminal device, generating metadata of the data; storing the metadata in the key-value dataset, and storing the metadata in the relational dataset ; If it is determined that the storage of metadata in any one of the key-value data set and the relational data set fails, the metadata stored in the person will be stored in the preset message queue, wherein the metadata in the message queue is used for interacting with terminal devices.

In some embodiments, key-value datasets, relational datasets, and message queues exist in a distributed device cluster.

In some embodiments, if it is determined that the storage of metadata in any one of the key-value data set and the relational data set fails, the metadata stored in that one will be stored in a preset message queue, including: if it is determined If the storage of metadata fails in the key-value dataset, the metadata stored in the key-value dataset will be converted into binary metadata synchronously, the first binary metadata will be obtained, and the first binary metadata will be stored in the message queue data; or if it is determined that the storage of metadata in the relational dataset fails, synchronously convert the metadata stored in the relational dataset into binary metadata, obtain the second binary metadata, and store the second binary metadata in the message queue Two binary metadata.

In some embodiments, storing metadata to the key-value data set and storing metadata to the relational data set includes: storing metadata to the key-value data set; Data sets store metadata; among them, key-value data sets, relational data sets, and message queues all have at least one data backup.

In some embodiments, the method further includes: in response to determining that the metadata stored in the message queue is successful, writing the metadata stored in the message queue into the distributed cache; in response to receiving a data request from the terminal device, storing the metadata in the distributed cache Look up the metadata corresponding to the data request in the area, and feed back to the terminal device based on the found metadata; in response to receiving the data request from the terminal device, look up the metadata corresponding to the data request in the distributed cache, if not found After that, the metadata corresponding to the data request is searched in the message queue, and feedback is sent to the terminal device based on the metadata found in the message queue.

In some embodiments, the method further includes: in response to detecting that new metadata has been added to the message queue, using a backoff algorithm to asynchronously store new metadata to the failed dataset, wherein, when the asynchronous storage is successful, the failure occurs The data set of has restored storage function.

In some embodiments, after asynchronously storing the metadata in the message queue to the faulty dataset, the method further includes: performing a metadata consistency check on the key-value dataset and the relational dataset; If the result is inconsistent, update the metadata in the relational dataset to the metadata in the key-value dataset.

In the second aspect, the embodiment of the present application provides a data storage device, including: an acquisition unit configured to acquire data sent by the target terminal device, and generate metadata of the data; a storage unit configured to store the data in the key-value data set Metadata, and store metadata in the relational dataset; the queue join unit is configured to store the metadata in the key-value dataset and the relational dataset if it is determined that the storage of the metadata fails. The data is stored in a preset message queue, wherein the metadata in the message queue is used to interact with the terminal device.

In some embodiments, the queue adding unit is further configured to: if it is determined that the storage of metadata in the key-value data set fails, synchronously convert the metadata stored in the key-value data set into binary metadata, and obtain the first Binary metadata, and store the first binary metadata to the message queue; or if it is determined that the storage of metadata in the relational dataset fails, synchronously convert the metadata stored in the relational dataset into binary metadata data, obtain second binary metadata, and store the second binary metadata in the message queue.

In some embodiments, the storage unit is further configured to: store metadata in the key-value dataset; store metadata in the relational dataset in response to success in storing metadata in the key-value dataset; wherein, the key-value dataset , relational data sets, and message queues all have at least one data backup.

In some embodiments, the device further includes: a writing unit configured to write the metadata stored in the message queue into the distributed cache in response to determining that the metadata stored in the message queue is successful; the first search unit is configured In response to receiving a data request from a terminal device, search for metadata corresponding to the data request in the distributed cache, and feed back to the terminal device based on the found metadata; the second search unit is configured to respond to receiving For the data request of the terminal device, search for the metadata corresponding to the data request in the distributed buffer area, if not found, search for the metadata corresponding to the data request in the message queue, and send the data to the terminal based on the metadata found in the message queue The device gives feedback.

In some embodiments, the device further includes: a backoff unit configured to store new metadata asynchronously to the failed dataset by using a backoff algorithm in response to detecting that new metadata has been added to the message queue, wherein the asynchronous When storage succeeds, the failed dataset has restored storage functionality.

In some embodiments, the device further includes: a verification unit configured to, after asynchronously storing the metadata in the message queue to the failed dataset, perform metadata consistency on the key-value dataset and the relational dataset consistency check; the update unit is configured to update the metadata in the relational dataset to the metadata in the key-value dataset if the check result is inconsistent.

In a third aspect, the embodiment of the present application provides an electronic device, including: one or more processors; a storage device for storing one or more programs, when one or more programs are executed by one or more processors , so that one or more processors implement the method in any embodiment of the data storage method.

In a fourth aspect, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method in any embodiment of the data storage method is implemented.

In the data storage solution provided by the embodiment of the present application, firstly, the data sent by the target terminal device is acquired, and metadata of the data is generated. After that, store metadata to key-value datasets and store metadata to relational datasets. Finally, if it is determined that the storage of metadata in any one of the key-value data set and the relational data set fails, the metadata stored in the data set will be stored in the preset message queue, wherein the metadata in the message queue Used to interact with end devices. The embodiment of the present application can introduce a message queue to implement a temporary storage function, and provide services to the terminal device instead of a data set that has a storage failure, thereby ensuring normal interaction between the terminal device and the server, and improving the security of cloud services. Moreover, the embodiment of the present application does not need to use a backup device, so that there is no need to modify the data reading process. In this way, while improving the stability of the cloud service, the cost consumption caused by the backup device can be avoided.

Description of drawings

Other characteristics, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 is an exemplary system architecture diagram to which some embodiments of the present application can be applied;

Fig. 2 is a flow chart according to an embodiment of the data storage method of the present application;

Fig. 3 is a schematic diagram of an application scenario according to the data storage method of the present application;

Fig. 4 is the flowchart according to another embodiment of the data storage method of the present application;

FIG. 5 is a schematic structural diagram of an embodiment of a data storage device according to the present application;

FIG. 6 is a schematic structural diagram of a computer system suitable for implementing electronic devices according to some embodiments of the present application.

Detailed ways

The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, rather than to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

FIG. 1 shows an exemplary system architecture 100 to which embodiments of the data storage method or data storage device of the present application can be applied.

As shown in FIG. 1 , a system architecture 100 may include terminal devices 101 , 102 , 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the terminal devices 101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages and the like. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as video applications, live broadcast applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 101, 102, and 103 here may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices with display screens, including but not limited to smartphones, tablet computers, e-book readers, laptop computers, desktop computers, and the like. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It may be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made here.

The server 105 may be a server providing various services, for example, a background server providing support for the terminal devices 101 , 102 , 103 . The background server can analyze and process the received data such as the data request sent by the target terminal device, and feed back the processing result (for example, the data corresponding to the data request) to the terminal device.

It should be noted that the data storage method provided in the embodiment of the present application can be executed by the server 105 or the terminal equipment 101, 102, 103, and correspondingly, the data storage device can be set in the server 105 or the terminal equipment 101, 102, 103.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Continuing to refer to FIG. 2 , a flow 200 of an embodiment of the data storage method according to the present application is shown. The data storage method comprises the following steps:

Step 201, acquire data sent by a target terminal device, and generate metadata of the data.

In this embodiment, the execution subject of the data storage method (such as the server or the terminal device shown in FIG. 1 ) can obtain the data sent by the target terminal device, and generate metadata of the data. Metadata may be referred to as "data about data", ie, data used to describe other data. Metadata is structured data that is used in storage systems to support functions such as indicating storage locations, historical data, resource lookup, and file records. For example, a user's file is called "data". In the storage system, the metadata of this file can be used to indicate the name, size, creation time, modification time, user, storage location, etc. of the file. Metadata may be basic information about the user, data upload time, user IP (Internet Protocol, Internet Protocol), data index, and the like. For the server, the server can provide data access services for multiple different applications, and terminal devices of different applications can send metadata access requests to the server through different types of service interfaces to obtain metadata.

Step 202, storing metadata in key-value datasets, and storing metadata in relational datasets.

In this embodiment, the execution subject may store the metadata in the key-value dataset, and store the metadata in the relational dataset. Key-value datasets and relational datasets can be set on the same server, or on different servers in the server cluster. The data in the key-value (key-value, K-V) dataset is arranged in the form of key-value pairs. Relational datasets can implement metadata enumeration, for example, relational datasets can be mysql databases.

In practice, the above execution subjects can store metadata in key-value datasets and relational datasets in various orders. For example, the above execution subject may store metadata in the two data sets at the same time.

The data structure of the metadata stored in the key-value dataset is different from the metadata stored in the relational dataset. In addition, the presentation methods of the stored metadata may also be different. For example, the key-value data set has excellent storage performance and high throughput, but there is no enumeration list (list) in it and the enumeration function cannot be realized.

Step 203, if it is determined that the storage of metadata in any one of the key-value data set and the relational data set fails, store the metadata stored in the one in the preset message queue, wherein the metadata in the message queue The data is used to interact with end devices.

In this embodiment, if it is determined that the process of storing metadata to the key-value dataset fails during the process of storing metadata, the execution subject may store the metadata stored in the key-value dataset to the preset in the message queue. If it is determined that the process of storing metadata to the relational dataset fails during the process of storing metadata, the execution subject may store the metadata stored in the relational dataset into a preset message queue.

The metadata in the message queue can be used to exchange metadata with the terminal device. After the execution subject stores the metadata in the message queue, the terminal device can obtain the metadata from the message queue. For example, the above-mentioned terminal device may upload a data request for requesting metadata, and the above-mentioned execution subject may search for the metadata in a message queue.

In this embodiment, a message queue can be introduced to implement a temporary storage function, and provide services to the terminal device instead of a data set that has a storage failure, thereby ensuring normal interaction between the terminal device and the server, and improving the security of cloud services. Moreover, this embodiment does not need to use a backup device, so that there is no need to modify the data reading process. In this way, while improving the stability of the cloud service, the cost consumption caused by the backup device can be avoided.

Continue referring to FIG. 3 , which is a schematic diagram of an application scenario of the data storage method according to this embodiment. In the application scenario of FIG. 3 , the execution subject 301 can obtain the data 302 sent by Zhang San's terminal device, and generate metadata 303 of the data. The execution subject 301 stores the metadata 303 in the key-value dataset 304 and stores the metadata 303 in the relational dataset 305 . If the execution subject 301 determines that the storage of metadata in any one of the key-value data set and the relational data set fails, it will store the metadata stored in the person in the preset message queue 306, wherein the metadata in the message queue The data is used to interact with end devices.

Further referring to FIG. 4 , it shows a flow 400 of another embodiment of the data storage method. The process 400 of the data storage method includes the following steps:

Step 401, acquire data sent by a target terminal device, and generate metadata of the data.

In this embodiment, the execution body of the data storage method (such as the server or the terminal device shown in FIG. 1 ) can obtain the data sent by the target terminal device, and generate metadata of the data. Metadata may be referred to as "data about data", ie, data used to describe other data. Metadata is structured data that is used in storage systems to support functions such as indicating storage locations, historical data, resource lookup, and file records. For example, a user's file is called "data". In the storage system, the metadata of this file is generally used to indicate the name, size, creation time, modification time, user, storage location, etc. of the file.

Step 402, storing metadata in key-value datasets, and storing metadata in relational datasets.

In this embodiment, the execution subject may store the metadata in the key-value dataset, and store the metadata in the relational dataset. Key-value datasets and relational datasets can be set on the same server, or on different servers in the server cluster. Data in a key-value dataset is arranged in key-value pairs. Relational datasets can implement metadata enumeration, for example, relational datasets can be mysql databases.

Step 403, if it is determined that the storage of metadata in any one of the key-value data set and the relational data set fails, store the metadata stored in the person in the preset message queue, wherein the metadata in the message queue Used to interact with end devices.

In this embodiment, if it is determined that in the above process of storing metadata, the process of storing metadata to the key-value data set fails, the metadata stored in the key-value data set can be stored in the preset message queue middle. If it is determined that the process of storing metadata to the relational dataset fails during the above process of storing metadata, the metadata stored in the relational dataset may be stored in a preset message queue.

Step 404, in response to hearing that new metadata has been added to the message queue, use the backoff algorithm to asynchronously store new metadata to the failed dataset, wherein, when the asynchronous storage is successful, the failed dataset has resumed its storage function .

In this embodiment, if the above-mentioned executive body detects that metadata has been added to the message queue, it can use the backoff algorithm to asynchronously transfer the metadata in the message queue to the failed data set (that is, the key-value data set) or relational data set) storage. Specifically, the above-mentioned executive body can convert the binary metadata in the message queue into non-binary metadata for storage in key-value datasets or relational datasets, and store the converted metadata in the data set. The result of this stored procedure can be successful or unsuccessful.

In practice, the above execution subject can store the metadata in the message queue to the failed dataset. At this time, the storage function of the data set may not be restored, which may cause the failure of writing. The above execution subject will (timingly) try to store metadata in the data set according to the back-off algorithm until the metadata storage is successful.

In this embodiment, the back-off algorithm is used to restore the storage of metadata in the data set in time when the data set restores the storage function, so that the service device can resume normal operation. Moreover, asynchronous storage can avoid excessive occupation of operating resources and save operating resources.

In some optional implementations of this embodiment, after step 404, the above method may further include: performing metadata consistency check on the key-value dataset and the relational dataset; if the check result is inconsistent , to update the metadata in the relational dataset to the metadata in the key-value dataset.

In these optional implementation manners, the above execution subject may perform consistency check on the metadata in the key-value dataset and the metadata in the relational dataset. If the result of the verification is inconsistent, the execution subject may update the metadata in the relational dataset according to the metadata in the key-value dataset. Specifically, the execution subject may update the metadata in the relational dataset, so that the updated metadata in the relational dataset is consistent with the metadata in the key-value dataset.

These implementations can avoid service errors caused by the inconsistency between the metadata in the key-value dataset and the metadata in the relational dataset, and ensure the consistency between the metadata of the two datasets.

In some optional implementations of any of the above embodiments, storing metadata to the key-value dataset and storing metadata to the relational dataset may include: storing metadata to the key-value dataset; responding to The key-value dataset stores metadata successfully, and stores metadata to the relational dataset; wherein, there is at least one data backup in the key-value dataset, relational dataset, and message queue.

In these optional implementation manners, the above execution subject may sequentially store metadata in the key-value dataset and the relational dataset. Specifically, the execution subject may store related metadata in the relational dataset after the metadata is successfully stored in the key-value dataset, that is, the metadata is stored.

The key-value data set may be a data set for interacting with the terminal device, and the above execution subject or other execution subjects may search the key-value data set for the metadata requested by the data request of the terminal device. The relational data set may not be used for interaction with terminal devices, but only for detailed storage of metadata.

These implementations can first store metadata to the key-value dataset, so as to ensure that services related to the metadata are provided to users in a timely manner. However, if the metadata is first stored in the relational dataset, the stored metadata cannot be used to interact with the terminal device, which may cause a delay in returning data to the user. In addition, metadata security can be ensured by storing data backups of metadata in distributed device clusters.

In some optional implementation manners of any of the foregoing embodiments, the key-value data set, the relational data set, and the message queue exist in a distributed device cluster.

In these optional implementations, key-value datasets, relational datasets, and message queues can be in the same distributed device cluster. In this way, temporary storage of metadata can be realized only by using devices in the distributed device cluster without using other backup data, and the service stability of the distributed device cluster can be improved.

In some optional implementations of any of the above embodiments, if it is determined that the storage of metadata in any one of the key-value data set and the relational data set fails, the metadata stored in that one will be sent to the preset The set message queue storage may include: if it is determined that the storage of the metadata in the key-value data set fails, synchronously converting the metadata stored in the key-value data set into binary metadata to obtain the first binary metadata, And store the first binary metadata to the message queue; or if it is determined that the storage of the metadata in the relational dataset fails, synchronously convert the metadata stored in the relational dataset into binary metadata to obtain the second binary metadata binary metadata, and store the second binary metadata in the message queue.

In these optional implementation manners, the execution subject may store metadata in the message queue synchronously. The stored metadata may be binary. Specifically, if the process of storing metadata to the above-mentioned key-value data set fails, the above-mentioned executive body can convert the metadata stored in the value data set into binary to obtain binary metadata, and use the binary metadata as First binary metadata. The aforementioned execution subject may store the first binary metadata in a message queue. In addition, if a failure is sent to the process of storing metadata in the relational dataset, the execution subject can convert the metadata stored in the relational dataset into binary to obtain binary metadata, and use the binary metadata as Second binary metadata. The above execution subject may store the second binary metadata in a message queue.

In these implementation manners, the process of storing metadata in the message queue may be synchronous storage, so as to ensure that the metadata is stored in the message queue in a timely manner, and then used for real-time interaction with the terminal device. By converting to binary for storage, the relative unity of the form of metadata in the message queue can be achieved.

In some optional implementations of any of the above embodiments, the above method may further include: in response to determining that the metadata stored in the message queue is successful, writing the metadata stored in the message queue into the distributed cache; in response to receiving For the data request to the terminal device, look up the metadata corresponding to the data request in the distributed cache, and feed back to the terminal device based on the found metadata; in response to receiving the data request from the terminal device, in the distributed cache Search for the metadata corresponding to the data request, if not found, search for the metadata corresponding to the data request in the message queue, and feed back to the terminal device based on the metadata found in the message queue.

In these optional implementation manners, the execution subject may write the metadata written into the message queue into a distributed cache (cache). The distributed cache area may be a cache area on the device where the message queue is located, or a distributed cache area in a distributed device cluster where the message queue is located. Specifically, the execution subject may store the metadata in the distributed cache in response to the success of storing the metadata in the message queue. Here, the metadata stored in the distributed cache may be non-binary metadata, for example, the metadata stored in the distributed cache may be in the form of metadata stored in the faulty dataset.

In practice, after receiving the data request sent by the terminal device or forwarded by other devices, the execution subject may preferentially look up the metadata requested by the data request in the distributed cache. If found, the execution subject may return the found metadata to the terminal device. If not found, the above execution subject may search the message queue for the metadata requested by the data request. Afterwards, the execution subject may feed back the found metadata to the terminal device that sent the data request, or perform preset processing (such as compression, packaging, etc.) on the found metadata, and feed back the processing result to the terminal device.

These implementation manners can preferentially search the metadata required by the terminal device from the cache area, so that the cache area can be used to improve the efficiency of feeding back metadata to the terminal device. Moreover, these implementations utilize the distributed cache area, which can systematically store various and rich data in the cache.

With further reference to Figure 5, as an implementation of the methods shown in the above figures, the present application provides an embodiment of a data storage device, which corresponds to the method embodiment shown in Figure 2, except for the following In addition to the features, the device embodiment may also include the same or corresponding features or effects as those of the method embodiment shown in FIG. 2 . The device can be specifically applied to various electronic devices.

As shown in FIG. 5 , the data storage device 500 of this embodiment includes: an obtaining unit 501 , a storage unit 502 and a queue adding unit 503 . Among them, the acquisition unit 501 is configured to acquire the data sent by the target terminal device, and generate metadata of the data; the storage unit 502 is configured to store metadata to the key-value dataset, and store metadata to the relational dataset; queue The adding unit 503 is configured to store the metadata stored in the key-value data set and the relational data set in a preset message queue if it is determined that the storage of the metadata fails in the key-value data set or the relational data set, wherein the message The metadata in the queue is used to interact with end devices.

In some embodiments, the acquiring unit 501 of the data storage device 500 acquires data sent by a target terminal device, and generates metadata of the data. Metadata may be referred to as "data about data", ie, data used to describe other data. Metadata is structured data that is used in storage systems to support functions such as indicating storage locations, historical data, resource lookup, and file records.

In some embodiments, the storage unit 502 may store the above-mentioned metadata in key-value datasets, and store the above-mentioned metadata in relational datasets. Key-value datasets and relational datasets can be set on the same server, or on different servers in the server cluster. Data in a key-value dataset is arranged in key-value pairs.

In some embodiments, the queue adding unit 503 may store the metadata stored in the key-value data set into a preset message queue. If it is determined that the process of storing metadata to the relational dataset fails during the process of storing metadata, the execution subject may store the metadata stored in the relational dataset into a preset message queue.

In some optional implementations of this embodiment, the key-value data set, the relational data set, and the message queue exist in a distributed device cluster.

In some optional implementations of this embodiment, the queue joining unit is further configured to: if it is determined that the storage of metadata in the key-value dataset fails, synchronously convert the metadata stored in the key-value dataset into binary metadata, get the first binary metadata, and store the first binary metadata in the message queue; or if it is determined that the storage of the metadata in the relational dataset fails, the synchronization will store the metadata in the relational dataset The metadata is converted into binary metadata to obtain second binary metadata, and the second binary metadata is stored in the message queue.

In some optional implementations of this embodiment, the storage unit is further configured to: store metadata in the key-value dataset; store metadata in the relational dataset in response to success in storing metadata in the key-value dataset ; Among them, there is at least one data backup in the key-value data set, the relational data set, and the message queue.

In some optional implementations of this embodiment, the device further includes: a writing unit configured to, in response to determining that the metadata is successfully stored in the message queue, write the metadata stored in the message queue into the distributed cache; The first search unit is configured to, in response to receiving a data request from a terminal device, search for metadata corresponding to the data request in the distributed cache, and feed back to the terminal device based on the found metadata; the second search unit, It is configured to search the metadata corresponding to the data request in the distributed cache in response to receiving the data request from the terminal device, and if not found, search the message queue for the metadata corresponding to the data request, based on the search in the message queue The received metadata is fed back to the terminal device.

In some optional implementations of this embodiment, the device further includes: a backoff unit configured to, in response to detecting that new metadata has been added to the message queue, use a backoff algorithm to asynchronously store new metadata to the failed dataset. Metadata where, when asynchronous storage succeeds, the failed dataset has restored storage functionality.

In some optional implementations of this embodiment, the device further includes: a checking unit configured to check the key-value dataset and the relational data The set is used to verify the consistency of the metadata; the update unit is configured to update the metadata in the relational dataset to the metadata in the key-value dataset if the verification result is inconsistent.

In this embodiment, a message queue can be introduced to implement a temporary storage function, and provide services to the terminal device instead of a data set that has a storage failure, thereby ensuring normal interaction between the terminal device and the server, and improving the security of cloud services. Moreover, this embodiment does not need to use a backup device, so that there is no need to modify the data reading process. In this way, while improving the stability of the cloud service, the cost consumption caused by the backup device can be avoided.

As shown in FIG. 6, an electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may be randomly accessed according to a program stored in a read-only memory (ROM) 602 or loaded from a storage device 608. Various appropriate actions and processes are executed by programs in the memory (RAM) 603 . In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are also stored. The processing device 601 , ROM 602 , and RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to the bus 604 .

Typically, the following devices can be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 607 such as a computer; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While FIG. 6 shows electronic device 600 having various means, it should be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided. Each block shown in FIG. 6 may represent one device, or may represent multiple devices as required.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 609 , or from storage means 608 , or from ROM 602 . When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed. It should be noted that the computer-readable medium in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present application may be implemented by means of software or by means of hardware. The described units may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, a storage unit, and a queue adding unit. Wherein, the names of these units do not constitute a limitation on the unit itself under certain circumstances, for example, the acquisition unit may also be described as "a unit that acquires data sent by the target terminal device and generates metadata of the data".

As another aspect, the present application also provides a computer-readable medium. The computer-readable medium may be included in the device described in the above embodiments, or it may exist independently without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the device, the device: acquires the data sent by the target terminal device, generates metadata of the data; stores the data in the key-value data set Metadata, and store metadata in relational datasets; if it is determined that the storage of metadata in any one of the key-value datasets and relational datasets fails, the metadata stored in that one will be sent to the preset message Queue storage, where metadata in message queues is used to interact with end devices.

The above description is only a preferred embodiment of the present application and an illustration of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover the technical solution formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features with similar functions disclosed in (but not limited to) this application.

Claims (15)

1. A method of data storage, the method comprising:

acquiring data sent by target terminal equipment and generating metadata of the data;

storing the metadata to a key-value dataset and storing the metadata to a relational dataset;

if the metadata is determined to have a fault in the storage of any one of the key value data set and the relational data set, storing the metadata stored to the corresponding key value data set and the relational data set into a preset message queue, wherein the metadata in the message queue are used for interaction with terminal equipment;

the storing the metadata to a key-value dataset and storing the metadata to a relational dataset includes:

storing the metadata to the key-value dataset;

in response to successful storage of the metadata to the key-value dataset, the metadata is stored to the relational dataset.

2. The method of claim 1, wherein,

the key-value dataset, the relational dataset, and the message queue are present in a distributed device cluster.

3. The method of claim 1, wherein if it is determined that the metadata fails to store in any one of the key-value data set and the relational data set, storing the metadata stored to the one in a preset message queue, including:

If the metadata are determined to have faults in the storage of the key value data set, the metadata stored in the key value data set are synchronously converted into binary metadata, first binary metadata are obtained, and the first binary metadata are stored in the message queue;

or (b)

If the metadata are determined to have faults in the storage of the relational data set, the metadata stored in the relational data set are synchronously converted into binary metadata, second binary metadata are obtained, and the second binary metadata are stored in the message queue.

4. The method of claim 1, wherein the key-value dataset, the relational dataset, and the message queue each have at least one data backup.

5. The method of claim 1, wherein the method further comprises:

writing the metadata stored to the message queue to a distributed cache in response to determining that the metadata was successfully stored to the message queue;

responding to a received data request of the terminal equipment, searching metadata corresponding to the data request in the distributed cache area, and feeding back to the terminal equipment based on the searched metadata;

And searching metadata corresponding to the data request in the distributed cache area in response to receiving the data request of the terminal equipment, and if the metadata corresponding to the data request is not searched, searching the metadata corresponding to the data request in the message queue, and feeding back the metadata searched in the message queue to the terminal equipment.

6. The method of claim 1, wherein the method further comprises:

and in response to the fact that new metadata is added into the message queue, asynchronously storing the new metadata into the data set with faults by utilizing a back-off algorithm, wherein the data set with faults has restored the storage function when asynchronous storage is successful.

7. The method of claim 6, wherein after asynchronously storing metadata in the message queue to the failed dataset, the method further comprises:

performing consistency verification on metadata on the key value data set and the relational data set;

and if the verification result is inconsistent, updating the metadata in the relational data set into the metadata in the key value data set.

8. A data storage device, the device comprising:

An acquisition unit configured to acquire data transmitted by a target terminal device, and generate metadata of the data;

a storage unit configured to store the metadata to a key-value dataset and store the metadata to a relational dataset;

a queue adding unit configured to store metadata stored to any one of the key value data set and the relational data set into a preset message queue if it is determined that the metadata is stored in the one of the key value data set and the relational data set fails, wherein the metadata in the message queue is used for interaction with a terminal device;

the memory unit is further configured to:

storing the metadata to the key-value dataset;

in response to successful storage of the metadata to the key-value dataset, the metadata is stored to the relational dataset.

9. The apparatus of claim 8, wherein the queue joining unit is further configured to:

if the metadata are determined to have faults in the storage of the key value data set, the metadata stored in the key value data set are synchronously converted into binary metadata, first binary metadata are obtained, and the first binary metadata are stored in the message queue;

Or (b)

If the metadata are determined to have faults in the storage of the relational data set, the metadata stored in the relational data set are synchronously converted into binary metadata, second binary metadata are obtained, and the second binary metadata are stored in the message queue.

10. The apparatus of claim 8, wherein the key-value dataset, the relational dataset, and the message queue each have at least one data backup.

11. The apparatus of claim 8, wherein the apparatus further comprises:

a writing unit configured to write metadata stored to the message queue to a distributed cache area in response to determining that the metadata is successfully stored to the message queue;

the first searching unit is configured to respond to the received data request of the terminal equipment, search metadata corresponding to the data request in the distributed cache area, and feed back the metadata to the terminal equipment based on the searched metadata;

and the second searching unit is configured to respond to the received data request of the terminal equipment, search the metadata corresponding to the data request in the distributed cache area, and if not, search the metadata corresponding to the data request in the message queue, and feed back the metadata searched in the message queue to the terminal equipment.

12. The apparatus of claim 8, wherein the apparatus further comprises:

and the back-off unit is configured to asynchronously store the new metadata into the data set with faults by utilizing a back-off algorithm in response to the fact that the new metadata is added into the message queue is monitored, wherein the data set with faults is restored to the storage function when the asynchronous storage is successful.

13. The apparatus of claim 12, wherein the apparatus further comprises:

a verification unit configured to perform consistency verification of metadata on the key-value dataset and the relational dataset after asynchronously storing the metadata in the message queue to the failed dataset;

and the updating unit is configured to update the metadata in the relational data set into the metadata in the key value data set if the verification result is inconsistent.

14. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-7.

15. A computer readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1-7.

Images