CN120050295A - Video storage method and related device - Google Patents
Video storage method and related device Download PDFInfo
- Publication number
- CN120050295A CN120050295A CN202510288844.9A CN202510288844A CN120050295A CN 120050295 A CN120050295 A CN 120050295A CN 202510288844 A CN202510288844 A CN 202510288844A CN 120050295 A CN120050295 A CN 120050295A
- Authority
- CN
- China
- Prior art keywords
- storage
- video
- file
- metadata
- disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/561—Adding application-functional data or data for application control, e.g. adding metadata
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a video storage method and a related device, and relates to the technical field of storage, wherein a video storage management system comprises a global metadata management server and a plurality of distributed storage servers, the global metadata management server operates global storage services, each distributed storage server operates node storage services, after the node storage services monitor the subfiles of video files, metadata corresponding to the subfiles are generated, the metadata comprises summaries and details, the summaries comprise corresponding relations between the video files and the distributed storage servers, the summaries and the details are stored in disks for storing the subfiles, the summaries are sent to the global storage services, the metadata of the distributed centralized storage video files are realized, the distributed storage servers corresponding to the video files are determined according to hash values corresponding to video identifications of the video files, and single-point faults and performance bottlenecks caused by centralized metadata management storage are solved.
Description
Technical Field
The present application relates to the field of storage technologies, and in particular, to a video storage method and a related device.
Background
In the field of audio and video, due to the improvement of content production efficiency and the continuous iteration of coding technology, the more the existing content is, the more the video storage requirements of different formats and qualities are increased.
Current primary storage system approaches include decentralised storage and centralised storage. The storage content in the decentralised storage mode is positioned by utilizing hash, metadata is completely abandoned, space cannot be effectively managed, and space unbalance exists. The centralized storage mode is used for centrally managing metadata, and the specific data are stored in a distributed mode, so that the metadata storage method has the advantages of being simple in management and efficient in access, and the defects of single-point failure risk and possible performance bottlenecks of central nodes.
Disclosure of Invention
In view of the above problems, the present application provides a video storage method and related apparatus, which can solve single point failure and performance bottleneck caused by centralized metadata management storage, and can realize load balancing by using global storage service. The specific scheme is as follows:
The first aspect of the present application provides a video storage method applied to a node storage service running on each distributed storage server in a video storage management system, where the video storage management system further includes a global metadata management server, and the global metadata management server runs a global storage service thereon, and the video storage method includes:
After a sub-file of a video file is monitored and a disc is dropped, metadata corresponding to the sub-file is generated, wherein the metadata comprises a summary and details, the summary comprises a corresponding relation between the video file and a distributed storage server, the distributed storage server corresponding to the video file is determined according to a hash value corresponding to a video identification of the video file, and a disc for storing the sub-file is determined according to the hash value corresponding to the file identification of the sub-file;
storing the summary and the detail in the metadata to a disk storing the subfiles;
The summary in the metadata is sent to the global storage service.
In one possible implementation, after the sub-file of the video file is monitored to be dropped, the video storage method further includes:
if the disk space for storing the subfiles is unbalanced, the subfiles are moved to other disks in the distributed storage server, and the position of the subfile dump is recorded in the metadata.
In one possible implementation, the video storage method further includes:
Verifying the corresponding relation between the video file and the distributed storage server according to the summary in the metadata;
checking whether the video file lacks fragments;
if the fragmentation is absent, an alarm or patch is initiated.
In one possible implementation, the video storage method further includes:
Under the condition that a disk fault is monitored, determining a target disk with the disk residual space larger than a threshold value, and establishing a temporary directory in the target disk;
using soft links to direct the mounted directory of the fault disk to the temporary directory;
Reconstructing a file structure of the temporary directory according to storage directories in other disks except the fault disk in the distributed storage server.
In one possible implementation, after reconstructing the file structure of the temporary directory from the storage directories in the other disks of the distributed storage server except for the failed disk, the video storage method further includes:
monitoring the pressure parameter of the target magnetic disk;
if the pressure parameter of the target disk is in a first preset range, initiating a patch and updating metadata corresponding to the temporary directory;
if the pressure parameter of the target magnetic disk is in a second preset range, maintaining unchanged;
if the pressure parameter of the target disk is in a third preset range, migrating the new file under the temporary directory to other disks in the distributed storage server and updating metadata corresponding to the temporary directory;
The pressure parameter in the first preset range is smaller than the pressure parameter in the second preset range, and the pressure parameter in the second preset range is smaller than the pressure parameter in the third preset range.
In one possible implementation, after reconstructing the file structure of the temporary directory from the storage directories in the other disks of the distributed storage server except for the failed disk, the video storage method further includes:
under the condition that the fault disk is monitored to be changed from a soft link to a physical disk or a recovery instruction of the fault disk is received, all contents in the temporary directory are moved to the disk after fault recovery;
And if the new file under the temporary directory is migrated to other disks in the distributed storage server, migrating the new file under the temporary directory back to the disk after fault recovery.
In one possible implementation, the video storage method further includes:
Under the condition that the HTTP service does not locate the target video file in the file system according to a preset hash locating rule, determining a storage directory of the target video file according to local metadata;
if the storage catalog of the target video file cannot be determined according to the local metadata, a query request is sent to the global storage service, so that the global storage service determines the storage catalog of the target video file.
In one possible implementation, the video storage method further includes:
before a new distributed storage server is online, sending a query request to the global storage service;
if the new distributed storage server is a replacement server of the original distributed storage server, receiving metadata of the original distributed storage server fed back by the global storage service;
if the new distributed storage server is a capacity expansion server, receiving metadata to be expanded fed back by the global storage service;
initiating a registration request to the global storage service;
And receiving registration response information fed back by the global storage service, and sending the summary of the stored metadata to the global storage service.
A second aspect of the present application provides a distributed storage server comprising at least one processor and a memory coupled to the processor, wherein:
the memory is used for storing a computer program;
The processor is configured to execute the computer program to enable the distributed storage server to implement the video storage method of the first aspect or any implementation manner of the first aspect.
A third aspect of the present application provides a video storage management system comprising a global metadata management server and a plurality of distributed storage servers of the second aspect;
operating a global storage service on the global metadata management server;
and each distributed storage server runs node storage service.
A fourth aspect of the application provides a computer program product comprising computer readable instructions which, when run on a distributed storage server, cause the distributed storage server to implement the video storage method of the first aspect or any implementation of the first aspect.
By means of the technical scheme, the video storage management system comprises a global metadata management server and a plurality of distributed storage servers, the global metadata management server operates global storage services, each distributed storage server operates node storage services, after the node storage services monitor the subfiles of the video files, metadata corresponding to the subfiles are generated, the metadata comprise summaries and details, the summaries comprise corresponding relations between the video files and the distributed storage servers, the summaries and the details are stored in magnetic discs of the storage subfiles, the summaries are sent to the global storage services, metadata of the distributed centralized storage video files are achieved, the distributed storage servers corresponding to the video files are determined according to hash values corresponding to video identifications of the video files, and single-point faults and performance bottlenecks caused by centralized metadata management storage can be solved.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a schematic diagram of a system architecture according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a video storage management system according to an embodiment of the present application;
Fig. 3 is a schematic flow chart of a video storage method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of content addressing according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a sub-file storage process according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a flow chart of a failure handling procedure for a disc failure according to an embodiment of the present application;
Fig. 7 is a schematic diagram of a load balancing process flow after a disc is dropped according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a recovery flow of a disk failure according to an embodiment of the present application;
fig. 9 is a schematic diagram of a video file access flow provided in an embodiment of the present application;
fig. 10 is a schematic diagram of online capacity expansion flow provided in an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terminology used in the description of the embodiments of the application herein is for the purpose of describing particular embodiments of the application only and is not intended to be limiting of the application.
Embodiments of the present application are described below with reference to the accompanying drawings. As one of ordinary skill in the art can know, with the development of technology and the appearance of new scenes, the technical scheme provided by the embodiment of the application is also applicable to similar technical problems.
The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely illustrative of the manner in which embodiments of the application have been described in connection with the description of the objects having the same attributes. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to the system architecture schematic shown in fig. 1, the monitoring and management layer in the video storage management system includes a global metadata management server, where a global storage service is running on the global metadata management server, and may also run a log management service and a graphics monitoring service, where the global metadata management server may include one or more servers, that is, the global storage service may also be a cluster service.
The storage layer in the video storage management system comprises a plurality of distributed storage servers, each distributed storage server runs node storage service, the distributed storage servers can be distributed in different machine rooms, and the plurality of distributed storage servers are distributed in a machine room A and a machine room B as shown in fig. 1. Each distributed storage server corresponds to a unique node identification (NodeID), and each distributed storage server comprises a plurality of disks, and each disk corresponds to a storage catalog. Each video file corresponds to a video identifier (FileID), and each video file comprises a plurality of subfiles, and each subfile corresponds to a file identifier.
After the video file is uploaded by the client, passes through the CDN (Content Delivery Network ), the access layer carries out hash operation on the video identification of the video file to determine a unique matched node identification (NodeID), and the video file is stored until the sub-file of the video file is stored, and then carries out hash operation on the file identification of the sub-file to determine a unique storage catalog, wherein the sub-file is stored in the NodeID/storage catalog/.
Referring to fig. 2, the video storage management system includes a global metadata management server 201 and a plurality of distributed storage servers 202 (3 distributed storage servers are illustrated in fig. 2 as an example), wherein the global metadata management server 201 runs a global storage service and each distributed storage server 202 runs a node storage service.
The embodiment of the application provides a video storage method which is applied to node storage services running on each distributed storage server in a video storage management system, and the video storage method of the embodiment of the application is described in detail below with reference to the accompanying drawings.
Referring to fig. 3, fig. 3 is a schematic flow chart of a video storage method according to an embodiment of the present application, and as shown in fig. 3, the video storage method according to an embodiment of the present application may include steps 301 to 303, which are respectively described in detail below.
301, Generating metadata corresponding to a sub-file after monitoring the sub-file of the video file to be dropped, wherein the metadata comprises a summary and a detail;
The distributed storage server corresponding to the video file is determined according to the hash value corresponding to the video identification of the video file, and the disk storing the subfiles is determined according to the hash value corresponding to the file identification of the subfiles. Illustratively, as shown in the content addressing diagram of fig. 4, the node storing the video file is located as node a according to the hash value of the video ID, and then the disks storing the subfiles are located as disk 1, disk 2, and disk 12 according to the hash value of the subfile ID.
The summary is used to describe basic properties of the video file, e.g. the summary includes correspondence between the video file and the distributed storage server.
In one possible implementation, the details include a subfile portion and a redundant backup portion. The subfile section is used to describe subfile information, such as which files are in the directory and which states are in each case. The redundant backup section is used for storing the sub-file information of other storage directories.
The summary and detail are combined into a metadata file which is stored in the lowest layer of the video directory and is in the same directory as the subfiles.
Exemplary, a meta data file of meta.json has the following file format:
meta.json's elements, file format is as follows:
{
"Summay" {// summary data
Data for/(patch)
“FileID”: "FSFDDSFS",“Main”:"/c1/xxx/xvd/xxx.m3u8", "NodeID": 1231, "Total": 12, "Size": 123333},
The data only holds the files that are stored in the directory as needed by the hash, with value 1 indicating local, otherwise hold its location, and point out the specific location
“Local”:{"12.ts":1,"24.ts":"/data1","36.ts":"nodeid/data1/xxxx/36.ts"},
Redundant backup
“Recovery”:{“/data11”:{"11.ts":1}, “/data1”:{"1.ts":1}}
}’
Where "Summay" denotes a summary section, "FileID" denotes a video identification, "Main" denotes a Main file description, i.e., a subfile description, "Total" denotes the number of subfiles of a video file, and "Size" denotes a storage Size.
"Local" means a subfile portion.
"Recovery" means a redundant backup section.
302, Storing the summary and the detail in the metadata to a disk storing the subfiles;
the node storage service maintains consistency of metadata and video data, when a related storage catalog on a distributed storage server changes, corresponding metadata is updated in time, for example, a file of which one is newly added in the storage catalog/data 1/AAA/BBB/CCC.m3u8 is monitored, the node storage service establishes corresponding metadata meta.json under the video/data 1/AAA/BBB/and determines whether the metadata is backed up to the same path of other storage catalogs according to a preset redundancy strategy.
It should be noted that, different storage directories on the same distributed storage server have the same directory structure, for example, if a disk/data 1 has an AAA/BBB directory, other disks/data 2,/data 3, etc. necessarily have the same AAA/BBB directory, whether the directory has real data or not, but the related disk metadata needs to be saved, so that after a certain storage directory fails, recovery can be performed according to the redundant metadata of other directories.
And 303, transmitting the summary in the metadata to the global storage service.
By synchronizing the summary part in the metadata to the global metadata management service, a correspondence between the video file and the distributed storage server, that is, a correspondence table of video IDs and nodeids, can be established in the global metadata management service according to the summary information. When the file access service cannot locate the file by the hash value, the global storage service can locate the real position of the video file according to the summary in the metadata.
The video ID and the NodeID may be in a many-to-one or many-to-many relationship, i.e., a video may be stored on only one distributed storage server (i.e., the NodeID is unique), or may be stored on multiple distributed storage servers, which needs to be determined according to different backup levels. If the video file is stored on only one distributed storage server, then the NodeID and video ID must be hash algorithm locationally determined.
According to the video storage method disclosed by the embodiment, metadata of the video file is stored in a distributed centralized manner, a distributed storage server corresponding to the video file is determined according to the hash value corresponding to the video identifier of the video file, and single-point faults and performance bottlenecks caused by centralized metadata management and storage are solved.
In one possible implementation, the node storage service further has a load balancing function, as shown in a schematic drawing of a sub-file storage flow shown in fig. 5, the distributed storage server further operates a node distribution service, the node distribution service downloads a sub-file and writes the sub-file into a disk, and then the node storage service listens to a file system change on the distributed storage server, when the sub-file is dropped, the node storage service generates metadata according to the sub-file and a video file and stores the metadata in a video directory, and sends a summary (meta summary) in the metadata, that is, a correspondence between the video file and the distributed storage server, to the global storage service, evaluates the load condition of a disk when the metadata (meta) is generated, and if the disk space is found to be unbalanced, can move the sub-file to a storage directory of other disks in the distributed storage server, and records the position of the sub-file dump in the metadata. After the dumping is completed, the summary information of the video is reported to the global storage service, and the consistency of metadata and video files is ensured while the load balancing is realized.
In one possible implementation, in order to further ensure that the metadata is consistent with the video file, the node storage service periodically verifies the metadata stored in the distributed storage server, for example, the metadata stored in the distributed storage server is verified every day, specifically, according to the summary verification in the metadata, the correspondence between the video file and the distributed storage server is verified, and if the node identifier of the distributed storage server corresponding to the hash value calculated according to the video identifier in the summary is inconsistent with the local identifier, an alarm is initiated.
If the stored video file is a video file such as HLS (HTTP LIVE STREAMING, HTTP real-time streaming media), MPEG-DASH (DYNAMIC ADAPTIVE STREAMING over HTTP, dynamic adaptive streaming media), it may also be checked whether a slice is missing, for example, whether a sub-file corresponding to the sub-file portion is missing in the distributed storage server is checked according to the sub-file portion in the metadata, if the slice is missing, an alarm or patch will be initiated, in the process of patching, the video file corresponding to the sub-file of the slice is determined according to the summary portion in the metadata, then whether redundant data is stored is queried from the global storage service, if the redundant data is stored, the patch is directly provided according to the redundant data, if the redundant data is not stored, the patch is provided according to an external protocol, generally pulled from the CDN.
When the disk in the distributed storage server is in fault, the node storage service recovers data from metadata redundancy of other storage directories and complements the related files, so that the video files can be used normally.
In one possible implementation, in the case that a disk failure is monitored, a node storage service running on the distributed storage server determines a target disk with a disk remaining space greater than a threshold, establishes a temporary directory on the target disk, then directs a mount directory of the failed disk to the temporary directory by using a flexible link, and then rebuilds a file structure of the temporary directory according to storage directories in other disks except the failed disk in the distributed storage server.
For example, referring to the schematic diagram of the disk-dropping failure processing flow shown in fig. 6, the disk-dropping failure processing flow specifically includes the following steps 601-607:
601, monitoring a disk drop, such as a/data 12 disk drop;
Constructing a temporary directory (such as/data 12 t) on other disks, and generating a soft link of/data 12 to point to the temporary directory/data 12t;
In order to ensure load balancing, a disk with more disk remaining space is selected as a target disk, and a temporary directory is established in the target disk, for example, a disk with the disk remaining space larger than a threshold value is determined as the target disk, and the threshold value is set according to an actual application scene.
603 Traversing the adjacent disc (/ data1 or/data 2) video directory, obtaining/data 12 content from meta.json;
Taking the distributed storage server shown in fig. 4 as an example, if the disc of the distributed storage server/data 12 is dropped, the distributed storage server still leaves/data 1 and/or data2, traversing/data 1 or/data 2 can obtain a metadata file meta.json stored under the same directory as the metadata file meta.json, and determining the content of the metadata file/data 12 from the subfile part in the meta.json.
Restoring the video content on the data12 to be written into the temporary catalogue;
605: file type in meta. Json/data 12;
If the file type in meta.json/data 12 is the file stored in/data 12, that is, the file in meta.json/data 12 is stored in the failed disk/data 12 and cannot be directly written, executing 606, modifying the corresponding file of meta.json as to-be-patched;
if the file type in meta/data 12 is a file saved in another location, then a direct write is performed 607.
If the file type in meta/json/data 12 is the file stored in other positions, the files in other positions are directly written into the temporary directory.
In one possible implementation, in order to ensure load balancing after a disk failure, after the file structure of the temporary directory is rebuilt, the pressure parameter of the target disk may be monitored, if the pressure parameter of the target disk is within a first preset range, a patch is initiated and metadata corresponding to the temporary directory is updated, if the pressure parameter of the target disk is within a second preset range, the patch is maintained unchanged, if the pressure parameter of the target disk is within a third preset range, a new file under the temporary directory is migrated to other disks in the distributed storage server and metadata corresponding to the temporary directory is updated, wherein the pressure parameter in the first preset range is smaller than the pressure parameter in the second preset range, and the pressure parameter in the second preset range is smaller than the pressure parameter in the third preset range, namely, the pressure parameter of the target disk is smaller in the first preset range, the pressure parameter of the target disk is moderate in the second preset range, and the pressure parameter of the target disk is larger in the third preset range.
For example, referring to the schematic diagram of the load balancing process flow after the disc is dropped shown in fig. 7, the load balancing process flow after the disc is dropped specifically includes the following steps 701-705:
701, monitoring a temporary storage disk where a newly built temporary directory/data 12t is located after a disk is dropped;
I.e., the target disk on which the temporary directory resides.
702, Comprehensively measuring the temporary storage disk pressure;
for example, the pressure parameter of the target disk is measured, and the pressure parameter may be the disk residual space, the disk space utilization rate, etc.
If the pressure is low, execute 703 patch and update meta. Json;
corresponding to the above step 606, since the pressure parameter of the target disk changes along with the access of the file, when the pressure of the target disk is smaller, the sub-file to be patched in the above step 606 may be patched, that is, the video file corresponding to the sub-file of the missing piece is determined according to the summary part in the metadata, then whether the redundant data is stored is queried from the global storage service, if the redundant data is stored, the patch is directly provided according to the redundant data, if the redundant data is not stored, the patch is provided according to the external protocol, and is generally pulled from the CDN.
If the pressure is moderate, go to 704, maintain unchanged;
If the pressure is high, 705, migrating the new file under the/data 12t directory to other disks and updating meta.
Similarly, because the pressure parameter of the target disk changes along with the access of the file, when the pressure of the target disk is larger, the new file under the/data 12t catalog is migrated to other disks and meta is updated, so that load balance is ensured.
And after maintaining the fault disk, recovering the mount, and recovering the data of the fault disk, wherein in one possible implementation, under the condition that the fault disk is monitored to be changed from a soft link to a physical disk or a recovery instruction of the fault disk is received, namely, under the condition that the fault disk is recovered and mount again, all contents in the temporary directory are moved to the disk after fault recovery, if a new file under the temporary directory is moved to other disks in the distributed storage server, the new file under the temporary directory is moved back to the disk after fault recovery, and if a patch to be patched exists under the temporary directory, the patch is carried out, so that the consistency of the video file and the metadata in the recovered disk is ensured.
For example, referring to the schematic diagram of the disk failure recovery process shown in fig. 8, the disk failure recovery process specifically includes the following steps 801-804:
801, monitoring a file system, finding/data 12 to change from a soft link to a physical disk, or receiving/data 12 restoration instructions;
As with the example above,/data 12 is a failed disk.
802, Moving all contents in the original temporary directory/data 12t to the new/data 12;
803, migrating the files belonging to/data 12 temporary storage and other places in the original video meta.json back to update the corresponding meta.json;
corresponding to step 705, the files migrated to other disks due to the excessive target disk pressure are migrated back to/data 12, and meta.json is updated to ensure the consistency of the video files in/data 12 with meta.json.
And 804, carrying out patch on the data needing patch in meta.json.
Corresponding to step 606, if there is a patch to be patched in meta. Json, then the patch is performed, and the patch flow is the same as that in the above embodiment, and will not be described again.
The embodiment of the application also provides a video access method, referring to the video file access flow diagram shown in fig. 9, after receiving a request from a node to which the consistent hash is applied, the HTTP service locates in a file system according to a preset hash locating rule, that is, according to the hash value of the target video file, and if the target video file is not found, queries the node storage service running on the distributed storage server corresponding to the hash value of the target video file. The node storage service local metadata determines a storage catalog of the target video file, namely, the video identification of the target video file is queried in the local metadata, if the video identification of the target video file is not queried, the storage catalog of the target video file cannot be determined according to the local metadata, and a query request is sent to the global storage service, so that the global storage service gives a conclusion according to the corresponding relation between the video identification in the metadata and the distributed storage server, and the target video file is not available or the specific position of the target video file is not given.
The video storage management system also supports online capacity expansion, a new distributed storage server (hereinafter referred to as a new node) is added in the video storage management system, the new distributed storage server can be a replacement server (hereinafter referred to as a replacement node) of an original distributed storage server (hereinafter referred to as an original node), and the new distributed storage server can also be a capacity expansion server (hereinafter referred to as a capacity expansion node). In one possible implementation, the online capacity expansion process includes the following steps 901-905:
901, before a new node is online, sending a query request to a global storage service;
the new node is a distributed storage server added by the video storage management system.
902, If the new node is a replacement node of the original node, receiving metadata of the original node fed back by the global storage service;
the original node is the original distributed storage server of the video storage management system.
903, If the new node is a capacity expansion node, receiving metadata to be expanded fed back by the global storage service;
i.e. the original node is not taken off line, and the new node is the newly added node.
904, Initiating a registration request to a global storage service;
905 receiving registration response information fed back by the global storage service and transmitting a summary of the stored metadata to the global storage service.
For example, as shown in the online capacity expansion flow chart shown in fig. 10, before a new node is online, the new node storage service queries the global storage service for its own video content, and if the new node is a replacement of an original node, the metadata of the original node fed back by the global storage service is received, and storage is constructed according to the metadata, where a storage directory in the new node is consistent with a storage directory of the original node. If the new node is a capacity expansion node, the whole capacity expansion node of the batch is required to be submitted to a global storage service, after the global storage service recalculates hash according to the node identification of the existing distributed storage server, metadata of video files belonging to the capacity expansion node are fed back to the capacity expansion node, the video files are currently stored in other distributed storage servers, namely metadata to be expanded are fed back to the capacity expansion node, the capacity expansion node constructs storage according to the metadata to be expanded, a storage catalog of the capacity expansion node and metadata meta are established, and note that dump addresses are stored in the initial stage of the meta information, namely the metadata are redirected to the node for storing the original video data. After the new node establishes the metadata of the new node, the new node initiates a registration request to the global storage service. And after the global storage service is verified, the new node is brought into management, and then the new node reports the existing metadata global storage service to the global storage service according to the reported updated data address.
In summary, the video storage method provided by the embodiment of the application maintains the consistency of metadata and video data, supports fault recovery, file addressing, global retrieval, load balancing and online capacity expansion, avoids single-point fault risks and improves system performance.
The embodiment of the application also provides a distributed storage server, which comprises at least one processor and a memory connected with the processor, wherein:
the memory is used for storing a computer program;
The processor is configured to execute the computer program to enable the distributed storage server to implement any one of the video storage methods provided by the embodiments of the present application.
The embodiment of the application also provides a computer program product, which comprises computer readable instructions, wherein the computer readable instructions enable a distributed storage server to realize any video storage method provided by the embodiment of the application when the computer readable instructions run on the distributed storage server.
The embodiment of the application also provides a computer readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by the distributed storage server, the distributed storage server can realize any video storage method provided by the embodiment of the application.
It should be further noted that the above-described apparatus embodiments are merely illustrative, and that the units described as separate units may or may not be physically separate, and that units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the embodiment of the device provided by the application, the connection relation between the modules represents that the modules have communication connection, and can be specifically implemented as one or more communication buses or signal lines.
From the above description of the embodiments, it will be apparent to those skilled in the art that the present application may be implemented by means of software plus necessary general purpose hardware, or of course by means of special purpose hardware including application specific integrated circuits, special purpose CPUs, special purpose memories, special purpose components, etc. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions can be varied, such as analog circuits, digital circuits, or dedicated circuits. But a software program implementation is a preferred embodiment for many more of the cases of the present application. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk or an optical disk of a computer, etc., comprising several instructions for causing a computer device (which may be a personal computer, a training device, a network device, etc.) to perform the method according to the embodiments of the present application.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via a wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, or the like that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (Solid STATE DISK, SSD)), etc.
Claims (11)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510288844.9A CN120050295A (en) | 2025-03-11 | 2025-03-11 | Video storage method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510288844.9A CN120050295A (en) | 2025-03-11 | 2025-03-11 | Video storage method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN120050295A true CN120050295A (en) | 2025-05-27 |
Family
ID=95749683
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202510288844.9A Pending CN120050295A (en) | 2025-03-11 | 2025-03-11 | Video storage method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN120050295A (en) |
-
2025
- 2025-03-11 CN CN202510288844.9A patent/CN120050295A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10873629B2 (en) | System and method of implementing an object storage infrastructure for cloud-based services | |
| CN111182067B (en) | Data writing method and device based on interplanetary file system IPFS | |
| JP5004975B2 (en) | Data storage system | |
| US9672372B2 (en) | Method for improving mean time to data loss (MTDL) in a fixed content distributed data storage | |
| JP5006395B2 (en) | Transcoding for distributed file systems | |
| CN103763383B (en) | Integrated cloud storage system and its storage method | |
| US7941455B2 (en) | Notification for a distributed file system | |
| US20100235409A1 (en) | System and method for managing data stored in a data network | |
| US10853210B2 (en) | Storage device health status synchronization | |
| US12346227B2 (en) | Cross-platform replication | |
| JP2009529190A (en) | Method for dynamic partitioning of redundant data fabrics | |
| CN111708552B (en) | Server program upgrade method, device, terminal equipment and storage medium | |
| CN114328033B (en) | Method and device for maintaining service configuration consistency of high-availability equipment group | |
| CN112965859A (en) | Data disaster recovery method and equipment based on IPFS cluster | |
| CN106452836A (en) | Method and apparatus for setting host node | |
| CN120050295A (en) | Video storage method and related device | |
| JP2011180658A (en) | Redundancy method in distributed file system | |
| CN107181631A (en) | A kind of Samba clusters interior joint failure switching method and system | |
| CN118069617A (en) | File processing method, device, electronic equipment and computer storage medium | |
| CN116009769A (en) | Fragmentation system and its traffic data distribution method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |