CN118410204B - Video image cross-node storage method and system - Google Patents
Video image cross-node storage method and system Download PDFInfo
- Publication number
- CN118410204B CN118410204B CN202410867722.0A CN202410867722A CN118410204B CN 118410204 B CN118410204 B CN 118410204B CN 202410867722 A CN202410867722 A CN 202410867722A CN 118410204 B CN118410204 B CN 118410204B
- Authority
- CN
- China
- Prior art keywords
- node
- hash
- video image
- storage
- linked list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/71—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a video image cross-node storage method and a system, comprising the following steps: the named nodes of the HDFS system respectively generate different hash identifications according to the categories and the attributes of the video images to be stored, and the named nodes construct a circular linked list node in the storage nodes according to the hash identifications; the naming node sends the corresponding hash identification to the corresponding storage node, and sends the corresponding hash identification to all the circulation list nodes according to the circulation list nodes where the storage node is located; after a video image is acquired, constructing a channel, attribute or logic relation of the video image, distributing corresponding hash identifications by a naming node of an HDFS system according to the channel, attribute or logic relation of the video image, and transmitting the video image and the corresponding hash identifications to a circular linked list node; after the corresponding circular linked list node acquires the video image, verifying according to the hash mark stored by the corresponding circular linked list node, and if the verification is passed, storing the video image into the circular linked list node through the node pointer.
Description
Technical Field
The invention relates to the technical field of video image storage, in particular to a method and a system for storing video images across nodes.
Background
At present, a large data amount storage video storage mode is mainly performed in a cloud storage mode, cloud storage video images have low-cost expandability, and the cloud storage video images allow users to dynamically increase or decrease storage space according to requirements without purchasing additional hardware; and the cloud storage video image can access the data stored in the cloud at any time and any place through the Internet, so that the accessibility and convenience of the data are greatly improved. However, the cloud storage video image is easy to encounter security risks and difficult to manage, and the video image at the cloud end is easy to have the problems of data leakage, data tampering, data deletion and the like. In addition, the video images stored in the cloud can increase the flow cost, and are affected by the network transmission bandwidth, so that the video images stored in the cloud have larger network delay under certain conditions.
Disclosure of Invention
One of the purposes of the invention is to provide a video image cross-node storage method and a system, which utilize an HDFS system (Hadoop Distributed FILE SYSTEM, sea Du Pu distributed file system) to construct a local Hadoop (sea Du Pu) node cluster for video image storage, and construct a circular linked list node in the Hadoop node cluster for storing video images with the same type or the same attribute, wherein the circular linked list node can be used for efficiently storing and managing the video images in a local storage space, and the circular linked list node can realize quick storage and inquiry of the video images through the setting of pointers.
The invention further aims to provide a video image cross-node storage method and a system, which utilize named nodes of the HDFS system to construct the circular linked list node for storage nodes, wherein the circular linked list node adopts a hash verification mode to store and inquire the video image, and two copy storage nodes in the HDFS system also construct corresponding circular linked list nodes, so that the circular linked list node has restorability, the problem of data loss caused by node faults can be avoided, and the robustness of the circular linked list node for video image storage is improved.
The invention further aims to provide a video image cross-node storage method and a video image cross-node storage system, wherein the method and the system can construct images or video data referenced by circular logic through the circular linked list nodes to store the images or the video data, so that the image data storage is more intelligent, and when the reference logic is inserted or deleted in the circular linked list nodes, the pointer pointing of the corresponding circular linked list nodes can be directly adjusted, thereby realizing efficient, safe and manageable storage of the video images.
To achieve at least one of the above objects, the present invention further provides a video image cross-node storage method, the method comprising:
The method comprises the steps that different hash identifications are respectively generated by named nodes of an HDFS system according to relation types of video images to be stored, and the named nodes construct circularly linked list nodes in storage nodes according to the hash identifications;
The named node sends the corresponding hash identification to the corresponding storage node, and sends the corresponding hash identification to all the circulation list nodes according to the circulation list nodes where the storage node is located;
After a video image is acquired, constructing a channel, attribute or logic relation of the video image, distributing corresponding hash identifications by a naming node of the HDFS system according to the channel, attribute or logic relation of the video image, and transmitting the video image and the corresponding hash identifications to a circular linked list node;
and after the corresponding circular linked list node acquires the video image, verifying according to the hash mark stored by the corresponding circular linked list node, and if the verification is passed, storing the video image into the circular linked list node through a node pointer.
According to one preferred embodiment of the present invention, the method for constructing the circularly linked list node includes: defining a class by the named node, including a defined plurality of storage nodes in the defined class, and configuring pointers in the plurality of storage nodes, wherein the defined storage nodes include head node tail nodes, each storage node pointing to a next storage node by the configured pointers, wherein the tail node pointers are configured to point to the head node to construct a circularly linked list node among the defined plurality of storage nodes.
According to another preferred embodiment of the present invention, the method for constructing the hash identifier includes: obtaining a mac address of a video image input interface, calculating the mac address of the input interface by adopting a hash algorithm to obtain a hash value, taking the hash value as a hash identifier of a channel relation of the video image, sending the hash identifier of the channel relation to one of the circular linked list nodes, traversing all nodes in a corresponding circular linked list according to pointers of the one circular linked list node, and sending the hash identifier of the channel relation to all the same circular linked list nodes for storage, wherein the hash identifier of the channel relation is used for channel storage verification of the corresponding video image.
According to another preferred embodiment of the present invention, the method for constructing the hash identifier includes: configuring a random number generator at the named node, generating at least one random number by utilizing the random number generator, judging that the video image comprises attributes of video image formats and video image types after the video image is acquired, distributing a random number according to the attributes of each video image comprising the formats and the types, calculating a hash value of the random number by adopting a hash algorithm, taking the hash value of the random number as a hash identifier of a corresponding video image attribute relationship, transmitting the hash identifier of the video image attribute relationship to one of the circular linked list nodes, traversing all the corresponding circular linked list nodes according to the pointer of one of the circular linked list nodes, and transmitting the hash identifier of the video image attribute relationship to all the same circular linked list nodes for storage, wherein the hash identifier of the video image attribute relationship is used for attribute verification of the corresponding video image.
According to another preferred embodiment of the present invention, the method for constructing the hash identifier includes: configuring a random number generator at the named node, generating at least one random number by utilizing the random number generator, identifying people and events in the video image by utilizing an AI model after the video image is acquired, judging the logic relationship between the people and the events, distributing corresponding random numbers according to the logic relationship between the people and the events, calculating hash values of the random numbers by adopting a hash algorithm, taking the hash values of the random numbers as hash identifications of the logic relationship of the corresponding video image, transmitting the hash identifications of the logic relationship to one of the circular linked list nodes, transmitting the hash identifications of the logic relationship of the video image to all the same circular linked list nodes for storage according to the pointing directions of pointers of the one of the circular linked list nodes, and using the hash identifications of the logic relationship of the video image for logic verification of the corresponding video image.
According to another preferred embodiment of the present invention, the hash verification method includes: defining a hash mark prestored by the node of the circular linked list as a first hash mark, after distributing a corresponding second hash mark to each video image, binding the second hash mark with the corresponding video image by a named node of the HDFS system, and transmitting the second hash mark to one of storage nodes, wherein the storage node compares the first hash mark stored by the storage node with the second hash mark, if the first hash mark is the same as the second hash mark, the storage node verifies that the video image bound by the second hash mark passes, and the named node transmits the video image bound by the second hash mark to the storage node which passes the verification for storage; if the storage node fails to verify, returning to verify failure, and not executing storage of the corresponding video image in the storage node.
According to another preferred embodiment of the present invention, the method for storing nodes of the circular linked list includes: and when the storage node passes the verification, calculating the storage space of the current storage node, if the current storage node does not have the storage space, calculating the storage space of the next storage node according to the pointer of the circular linked list node in the current storage node, and if the next storage node has the storage space, transmitting the video image bound with the second hash mark to the next storage node until the storage space of all the circular linked list nodes is fully traversed.
According to another preferred embodiment of the present invention, the cross-node storage method includes: after the HDFS system acquires video images to be stored and constructs a circular linked list node, copy data of the video images are automatically generated, a first hash mark of the corresponding circular linked list node is acquired, a copy circular linked list node of the corresponding copy data is constructed after a pointer is configured, the first hash mark is stored in each copy circular linked list node, and when one storage node in the circular linked list nodes is down and has data loss, the naming node searches the corresponding data of the corresponding copy circular linked list node through the first hash mark and is used for recovering the corresponding lost data.
In order to achieve at least one of the above objects, the present invention further provides a video image cross-node storage system that performs the above-described video image cross-node storage method.
The present invention further provides a computer readable storage medium storing a computer program for execution by a processor to implement a video image cross-node storage method as described above.
Drawings
Fig. 1 shows a schematic flow chart of a video image cross-node storage method of the invention.
FIG. 2 is a schematic diagram of the construction of nodes of a circular linked list according to the present invention.
Fig. 3 is a schematic diagram of a video map cross-node storage system according to the present invention.
Detailed Description
The following description is presented to enable one of ordinary skill in the art to make and use the invention. The preferred embodiments in the following description are by way of example only and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description may be applied to other embodiments, variations, modifications, equivalents, and other technical solutions without departing from the spirit and scope of the invention.
It will be understood that the terms "a" and "an" should be interpreted as referring to "at least one" or "one or more," i.e., in one embodiment, the number of elements may be one, while in another embodiment, the number of elements may be plural, and the term "a" should not be interpreted as limiting the number.
Referring to fig. 1-3, the invention discloses a method and a system for storing video images across nodes, wherein the method mainly comprises the following steps: at least one named node (name node) is built in the HDFS system, the named node can manage data storage of a plurality of storage nodes (date nodes) in the HDFS system, storage node fault judgment, storage node data recovery and the like, the named node also receives request data from different clients, and relevant operations of corresponding storage nodes are carried out according to the request data of the different clients, and the named node and the storage nodes are built locally by using the HDFS system, so that the video image cannot have traffic and bandwidth pressure. Furthermore, the invention constructs the circular linked list node based on the named node and the storage node, and uses the circular linked list node to store the video images of the client in an efficient and standardized way. According to the method and the device, the corresponding video images are stored and inquired by using the circularly linked list nodes according to channels, attributes or logical relations of the video images, so that the cross-node relational storage of the video images is realized, the cross-node storage between different storage nodes is increased, and the storage scheduling efficiency between the different storage nodes is improved in a pointer mode. In addition, the invention also configures different hash identifications based on channels, attributes or logic relations of the video images respectively, wherein the hash identifications are sent to corresponding circular linked list nodes, the corresponding circular linked list nodes store the hash identifications and are used for verifying whether the video image data to be sent have the corresponding hash identifications, and if so, the verification passes through allowing the video image data to be stored in the corresponding circular linked list nodes. In the process of carrying out related data query of corresponding storage nodes by using the named nodes of the HDFS system, the specified circularly linked list nodes can be searched according to the hash identification, and the circularly fuzzy query is carried out according to the preset pointer direction, so that the query effect of video images can be improved.
Specifically, when a named node of the HDFS system obtains a storage request of a client, the named node obtains a mac address of a corresponding client channel according to the storage request message, where the client channel includes different gateways, so that the mac address may be a gateway mac address, and after obtaining the mac address, the named node further performs a hash operation on the mac address by using a hash algorithm to obtain a hash identifier based on the mac address, and constructs a circular linked list node corresponding to a channel relationship by using the hash identifier.
In one preferred embodiment of the present invention, a circular linked list is constructed by obtaining a plurality of storage node information through a named node of the HDFS system, where the storage node information includes storage node names, and each storage node name in the circular linked list is used as a corresponding circular linked list node. Wherein a class (class) of a circular linked list is configured by the named node, the storage node name acquired in advance is added in the class of the circular linked list as the circular linked list node, pointers are further configured for each storage node in the class of the circular linked list, wherein the storage node comprises a head node and a tail node, there is one pointer and only one pointer for the storage node, each pointer points to one storage node in the circular linked list, and the pointer of the tail node of the storage node points to the head node. For example, defining node names in the circular linked list includes: node1, node2, node3, node4, in the C language or the c++ language, may be constructed by declaring an int pointer in each storage node and pointing the pointer to the corresponding storage node address, for example, defining the node1 storage node as a head node, the node4 node as a tail node, the pointer declared in the node1 pointing to the address adr2 of the node2, and similarly, the pointer declared in the node2 pointing to the address adr3 of the node3, and pointing the pointer of the tail node4 to the head node 1. Of course, in another preferred embodiment of the present invention, a string object may be added in the JAVA language in a class (class) of the created circular link list, where the string object has a corresponding storage node address, and a circular link list node is created by pointing to an instance of the string object by referring to a variable of a storage node name, where the above process is equivalent to a pointer function.
After the construction of the circular linked list node of each storage node is completed, the hash identification is further sent to one of the circular linked list nodes through a naming node of the HDFS system, and after the hash identification is acquired by the one circular linked list node, the hash identification is sent to all nodes of the same circular linked list according to pointers of the circular linked list. And the nodes of the same circular linked list all store the hash identification. It should be noted that there is only one hash identifier in one circular linked list, so that the circular linked list node only stores and queries video images with specific relationships.
Further, in one preferred embodiment of the present invention, when the HDFS system acquires a video image of a client, when the video image is larger than a storage space of a circular linked list constructed in the HDFS system, the video image needs to be split to obtain smaller data blocks bolck, where the size of the data blocks bolck may be set by a dfs.block size parameter, and in general, in the HDFS system, each cut data block is smaller than a complete storage space of a storage node. For example, when one of the videos M is 1000MB in size, the video M may be divided into 10 data blocks bolck-bolck by a named node in the HDFS system, where each data block bolck is smaller than 128MB. In one preferred embodiment of the present invention, the data blocks bolck are assigned corresponding hash identifiers to each of the split data blocks bolck according to the relationship type to which the video entity belongs. When the data block bolck is scheduled to be stored, the allocated hash identifier needs to be compared with the hash identifier stored in advance in the corresponding circular linked list node, if the hash identifiers are the same, the verification is passed, otherwise, the verification fails. At this time, the divided data blocks are all allocated to linked list nodes of the same relationship.
Specifically, after obtaining the mac address of each input video image and performing hash operation to obtain the hash identifier of the corresponding channel relationship, the naming node of the HDFS system divides the video image into data blocks bolck smaller than the storage space of the corresponding circular linked list node according to the size of the video image, and binds the hash identifier of the channel relationship and the divided data blocks bolck, so that the corresponding data blocks bolck are stored in the corresponding circular linked list.
In another preferred embodiment of the present invention, the named node is further configured with a random number generator, wherein the random number generator of the named node can be a pseudo-random number generator or a true random number generator, the random number generator is a software random number generator, the random number generator can be obtained based on a related random number algorithm of software, and the true random number generator can be obtained by acquiring uncertainty related to computer hardware through the software. After the named node generates the corresponding random number, the random number is stored, and a hash algorithm is adopted to carry out hash operation on the random number, so that a hash mark is obtained. The hash identification generated based on the random number is used for configuration and verification of video image attributes or logic relations, wherein after the hash identification of the attribute relations is generated, a circular linked list node based on the attribute relations can be constructed through the named node, namely the hash identification of the attribute relations is sent to a corresponding circular linked list node, and the hash identification of the attribute relations is sent to all nodes of the circular linked list through the pointer pointing of the circular linked list node. So that all nodes of the circular linked list are used to store video images that verify and query for specific attributes. It should be noted that, in the present invention, the attribute relationship is that the named node determines according to the attributes such as the format and the type of the video image sent from the client, for example, the video format may be, but not limited to, MP4, MOV, WMV, AVI, etc., and in the present invention, the hash identifier generated by binding the video with the same random number to the video with the same format is used to bind the hash identifiers with different random numbers to the video with different formats. And storing the video in different formats into different circular linked lists. The format of the picture can be, but not limited to BMP, JPEG, PNG, GIF, etc., a hash identifier generated by binding the same random number for the picture with the same format, and a hash identifier of different random numbers for the picture with different formats.
In the invention, the video images also comprise a logic relationship, wherein the logic relationship can be considered as a logic relationship of a cause and effect or an event between video image contents, for example, when a certain event happens, different video images exist between the occurrence and the effect, for example, a fire event happens in a certain region, the generated video in the fire event comprises a fire video, a corresponding fire alarm video, a follow-up fire result report video and the like, and certain cause and effect relationship exists between the different videos, so that characteristic images or characteristic words in the corresponding video images can be identified through an AI model to serve as a judging basis for the same logic relationship of the corresponding image contents, for example, the fire condition of the same place can be identified through an existing AI image identification model, and the report about the fire occurrence of the same place can be identified from the corresponding video through an AI voice identification model. It should be noted that the above-mentioned AI image recognition model and AI speech recognition model are existing mature models, and the present invention does not improve the AI model, so that the present invention does not describe how to train the AI model. When the characteristics of the related content in the video image are identified through the AI model, binding the hash identification of the logic relation corresponding to the random number to the video image, and sending the hash identification of the logic relation to one of the corresponding circular linked list nodes through the named node, wherein one of the circular linked list nodes sends the hash identification of the logic relation to all circular linked list nodes through the pointer of the node.
In another preferred embodiment of the present invention, the causal or event logic relationship between the video image contents can be identified according to character characteristics, that is, the existing face recognition model and voice recognition model can be used to record the corresponding character as the main characteristic of the video image logic relationship. For example, when the input video is Ji Chuan video of a person, and the naming node acquires the video, the relevant face recognition model or the character recognition model is called to recognize the main person feature in the video, the main person feature is used as the logical relation basis of the video image, and a hash identifier generated by random number calculation is bound to the video image. When the same main character features exist in different videos input subsequently and the same main character features exist in the different videos input subsequently, the same random numbers of the previous videos are bound in the different videos input subsequently to calculate generated hash identifications. And carrying out storage verification and inquiry according to the hash identification.
The storage verification method based on the hash mark comprises channel verification, attribute verification and logic verification, wherein the specific verification method comprises the following steps: defining a hash mark pre-stored by the node of the circular linked list as a first hash mark, binding the second hash mark with a corresponding video image when the hash mark distributed to each video image is a second hash mark by a named node of the HDFS system, transmitting the second hash mark to one of storage nodes through the named node, comparing the first hash mark stored by the storage node with the second hash mark, and if the first hash mark is the same as the second hash mark, transmitting the video image bound by the second hash mark to the storage node for storage after verification by the named node; if the storage node fails to verify, returning to verify failure, and not executing storage of the corresponding video image in the storage node. The hash mark comprises a channel relation hash mark, an attribute relation hash mark and a logic relation hash mark. Hash identifications of different types of relationships can be stored separately by named nodes of the HDFS system. In other words, when the named node of the HDFS system can select at least one of the channel relationship hash identifier, the attribute relationship hash identifier and the logic relationship hash identifier for storing the corresponding circularly linked list node according to the requirement of the storage mode of the named node.
In another preferred embodiment of the present invention, since the HDFS system copies and creates copy data when storing corresponding video data, and stores the copy data in a copy storage node, the present invention further performs the following processing for the copy data recovered by downtime of the storage node: after the HDFS system acquires video images to be stored and constructs a circular linked list node, copy data of the video images are automatically generated, a first hash mark of the corresponding circular linked list node is acquired, a copy circular linked list node of the corresponding copy data is constructed after a pointer is configured, the first hash mark is stored in each copy circular linked list node, and when one storage node in the circular linked list nodes is down and has data loss, the naming node searches the corresponding data of the corresponding copy circular linked list node through the first hash mark and is used for recovering the corresponding lost data. The construction of the duplicate circular linked list node is the same as that of the original circular linked list node, and the invention is not repeated in detail.
Furthermore, the invention also performs the following scheduling aiming at the storage mode of each circular linked list node: and when the storage node passes the verification, calculating the storage space of the current storage node, if the current storage node does not have the storage space, calculating the storage space of the next storage node according to the pointer of the circular linked list node in the current storage node, and if the next storage node has the storage space, transmitting the video image bound with the second hash mark to the next storage node until the storage space of all the circular linked list nodes is fully traversed. When the corresponding circular linked list nodes are all fully stored, the named nodes can obtain the circular linked list node inserted with the new node by inserting the new storage node name into the corresponding circular linked list and adjusting the pointer direction of the storage node name and the pointer direction of the last node name.
The processes described above with reference to flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the application. Embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via a communication portion, and/or installed from a removable medium. The above-described functions defined in the method of the present application are performed when the computer program is executed by a Central Processing Unit (CPU). The computer readable medium of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wire segments, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
It will be understood by those skilled in the art that the embodiments of the present invention described above and shown in the drawings are merely illustrative and not restrictive of the current invention, and that this invention has been shown and described with respect to the functional and structural principles thereof, without departing from such principles, and that any modifications or adaptations of the embodiments of the invention may be possible and practical.
Claims (9)
1.A method for storing video images across nodes, the method comprising:
The method comprises the steps that different hash identifications are respectively generated by named nodes of an HDFS system according to relation types of video images to be stored, and the named nodes construct circularly linked list nodes in storage nodes according to the hash identifications;
The named node sends the corresponding hash identification to the corresponding storage node, and sends the corresponding hash identification to all the circulation list nodes according to the circulation list nodes where the storage node is located;
After a video image is acquired, constructing a channel, attribute or logic relation of the video image, distributing corresponding hash identifications by a naming node of the HDFS system according to the channel, attribute or logic relation of the video image, and transmitting the video image and the corresponding hash identifications to a circular linked list node;
After the corresponding circular linked list node acquires the video image, verifying according to the hash mark stored by the corresponding circular linked list node, and if the verification is passed, storing the video image into the circular linked list node through a node pointer;
The construction method of the circularly linked list node comprises the following steps: defining a class by the named node, including a defined plurality of storage nodes in the defined class, and configuring pointers in the plurality of storage nodes, wherein the defined storage nodes include head node tail nodes, each storage node pointing to a next storage node by the configured pointers, wherein the tail node pointers are configured to point to the head node to construct a circularly linked list node among the defined plurality of storage nodes.
2. The video image cross-node storage method according to claim 1, wherein the hash identification construction method comprises: obtaining a mac address of a video image input interface, calculating the mac address of the input interface by adopting a hash algorithm to obtain a hash value, taking the hash value as a hash identifier of a channel relation of the video image, sending the hash identifier of the channel relation to one of the circular linked list nodes, traversing all nodes in a corresponding circular linked list according to pointers of the one circular linked list node, and sending the hash identifier of the channel relation to all the same circular linked list nodes for storage, wherein the hash identifier of the channel relation is used for channel storage verification of the corresponding video image.
3. The video image cross-node storage method according to claim 1, wherein the hash identification construction method comprises: configuring a random number generator at the named node, generating at least one random number by utilizing the random number generator, judging that the video image comprises attributes of video image formats and video image types after the video image is acquired, distributing a random number according to the attributes of each video image comprising the formats and the types, calculating a hash value of the random number by adopting a hash algorithm, taking the hash value of the random number as a hash identifier of a corresponding video image attribute relationship, transmitting the hash identifier of the video image attribute relationship to one of the circular linked list nodes, traversing all the corresponding circular linked list nodes according to the pointer of one of the circular linked list nodes, and transmitting the hash identifier of the video image attribute relationship to all the same circular linked list nodes for storage, wherein the hash identifier of the video image attribute relationship is used for attribute verification of the corresponding video image.
4. The video image cross-node storage method according to claim 1, wherein the hash identification construction method comprises: configuring a random number generator at the named node, generating at least one random number by utilizing the random number generator, identifying people and events in the video image by utilizing an AI model after the video image is acquired, judging the logic relationship between the people and the events, distributing corresponding random numbers according to the logic relationship between the people and the events, calculating hash values of the random numbers by adopting a hash algorithm, taking the hash values of the random numbers as hash identifications of the logic relationship of the corresponding video image, transmitting the hash identifications of the logic relationship to one of the circular linked list nodes, transmitting the hash identifications of the logic relationship of the video image to all the same circular linked list nodes for storage according to the pointing directions of pointers of the one of the circular linked list nodes, and using the hash identifications of the logic relationship of the video image for logic verification of the corresponding video image.
5. The video image cross-node storage method according to claim 1, wherein the hash identification verification method comprises: defining a hash mark prestored by the node of the circular linked list as a first hash mark, after distributing a corresponding second hash mark to each video image, binding the second hash mark with the corresponding video image by a named node of the HDFS system, and transmitting the second hash mark to one of storage nodes, wherein the storage node compares the first hash mark stored by the storage node with the second hash mark, if the first hash mark is the same as the second hash mark, the storage node verifies that the video image bound by the second hash mark passes, and the named node transmits the video image bound by the second hash mark to the storage node which passes the verification for storage; if the storage node fails to verify, returning to verify failure, and not executing storage of the corresponding video image in the storage node.
6. The method for storing video images across nodes according to claim 5, wherein the method for storing video images across nodes of the circular linked list comprises: and when the storage node passes the verification, calculating the storage space of the current storage node, if the current storage node does not have the storage space, calculating the storage space of the next storage node according to the pointer of the circular linked list node in the current storage node, and if the next storage node has the storage space, transmitting the video image of the second hash mark which is bound to the next storage node until the storage space of all the circular linked list nodes is traversed to be full.
7. The method of cross-node storage of video images according to claim 5, wherein the cross-node storage method comprises: after the HDFS system acquires video images to be stored and constructs a circular linked list node, copy data of the video images are automatically generated, a first hash mark of the corresponding circular linked list node is acquired, a copy circular linked list node of the corresponding copy data is constructed after a pointer is configured, the first hash mark is stored in each copy circular linked list node, and when one storage node in the circular linked list nodes is down and has data loss, the naming node searches the corresponding data of the corresponding copy circular linked list node through the first hash mark and is used for recovering the corresponding lost data.
8. A video image trans-node storage system, characterized in that the system performs a video image trans-node storage method according to any of the preceding claims 1-7.
9. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, which is executed by a processor to implement a video image cross-node storage method according to any of the preceding claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410867722.0A CN118410204B (en) | 2024-07-01 | 2024-07-01 | Video image cross-node storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410867722.0A CN118410204B (en) | 2024-07-01 | 2024-07-01 | Video image cross-node storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118410204A CN118410204A (en) | 2024-07-30 |
| CN118410204B true CN118410204B (en) | 2024-09-03 |
Family
ID=92032949
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410867722.0A Active CN118410204B (en) | 2024-07-01 | 2024-07-01 | Video image cross-node storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118410204B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636286A (en) * | 2015-02-06 | 2015-05-20 | 华为技术有限公司 | Data access method and equipment |
| CN110971872A (en) * | 2019-11-14 | 2020-04-07 | 北京中盾安全技术开发公司 | Video image information acquisition method based on distributed cluster |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8533231B2 (en) * | 2011-08-12 | 2013-09-10 | Nexenta Systems, Inc. | Cloud storage system with distributed metadata |
| CN111708615B (en) * | 2020-05-20 | 2021-10-29 | 腾讯科技（深圳）有限公司 | Transaction processing method, apparatus, computer equipment and storage medium |
| CN112667592B (en) * | 2020-12-24 | 2024-05-07 | 平安科技（深圳）有限公司 | Data storage method, device, computer equipment and storage medium |
| CN114816225B (en) * | 2021-01-28 | 2025-08-19 | 北京金山云网络技术有限公司 | Storage cluster management method and device, electronic equipment and storage medium |
| CN114253979B (en) * | 2021-12-23 | 2023-10-03 | 北京百度网讯科技有限公司 | Message processing method and device and electronic equipment |
| CN117271525A (en) * | 2023-10-27 | 2023-12-22 | 衡阳凯新特种材料科技有限公司 | Special ceramic data long-term storage system and method |
-
2024
- 2024-07-01 CN CN202410867722.0A patent/CN118410204B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636286A (en) * | 2015-02-06 | 2015-05-20 | 华为技术有限公司 | Data access method and equipment |
| CN110971872A (en) * | 2019-11-14 | 2020-04-07 | 北京中盾安全技术开发公司 | Video image information acquisition method based on distributed cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118410204A (en) | 2024-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12038976B2 (en) | Model pushing method and device, model requesting method and device, storage medium and electronic device | |
| US10341509B2 (en) | Client device state collection and network-based processing solution | |
| US12210485B2 (en) | Management of content | |
| CN113726873B (en) | Block chain-based file processing method, system, device and storage medium | |
| JP4205323B2 (en) | Distribution system, distribution server and distribution method, distribution program | |
| CN110046062B (en) | Distributed data processing method and system | |
| CN113900810A (en) | Distributed graph processing method, system and storage medium | |
| CN105302920A (en) | Optimal management method and system for cloud storage data | |
| CN114201362A (en) | Enterprise-level high-availability monitoring system and implementation method based on Prometheus | |
| CN118312626B (en) | Data management method and system based on machine learning | |
| CN114091610A (en) | Intelligent decision method and device | |
| CN113935285A (en) | Business code generation method and system | |
| CN113590711A (en) | High-elasticity extensible multi-chain data hierarchical shared storage system and method | |
| CN116955355A (en) | Block data processing method and device and electronic equipment | |
| KR20190037095A (en) | Method and architecture of Network Infrastructure for Optimal Application Service Processing and Data Sharing among Application domains | |
| CN118410204B (en) | Video image cross-node storage method and system | |
| CN112738153B (en) | Gateway selection method, system, device, server and medium in service system | |
| CN111460034B (en) | Bidirectional database synchronization method, synchronization unit and system | |
| CN113194330A (en) | Fragmented multi-cloud video resource management method and system | |
| CN112131014A (en) | Decision engine system and business processing method thereof | |
| US12072913B1 (en) | Unhandled data protection for programmatic input/output routing to datasets with user-defined partitions | |
| CN109389271B (en) | Application performance management method and system | |
| KR20180004794A (en) | Method and system for handling relationship chain, and storage medium | |
| CN117793095A (en) | Data forwarding method, device, system, equipment and storage medium | |
| CN118158027A (en) | GraphQL-based front-end gateway, data access method and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |