CN1258921C - Distributed video-on-demand system and its method for realizing data storage and access - Google Patents

Abstract

The invention discloses a distributed video-on-demand system and a method for realizing data storage and access thereof. The storage method includes the following steps: the data storage method includes the following steps: divide the data into blocks according to a set method ; Calculate the set number of data blocks to form a verification data block; store the data block and the verification data block in a set manner in turn in the set multiple buffer servers; in each of the buffer servers Generate index information stored in the corresponding data block. By adopting the technical scheme of the present invention, the effect of improving storage efficiency is achieved, and a large amount of storage space is saved; at the same time, because the data adopts distributed storage, the data files are distributed among multiple buffer servers, which improves the security of the data; on the other hand The reliability of data distribution is improved due to the adoption of an optimal data redundancy check method.

Description

Distributed video-on-demand system and its method for realizing data storage and access

technical field

The invention relates to distributed video-on-demand systems, and more particularly to storage and access of data.

Background technique

Data storage is a key technology involved in video on demand system. In the distributed video-on-demand system, there are very special requirements for data storage. Since the demand for data storage space in the video-on-demand system is very large, how to improve the utilization rate of the storage space in the distributed video-on-demand system becomes a very difficult problem.

In the conventional distributed video-on-demand system, such as adopting the method described in U.S. Patent 56491% (Di is recorded by field time. tora, mana sand ment sys is twisted. having a cache server a.d me than the false e hanging fer), so The above distributed method is based on a geographical division. In order for all video servers to share one or some data streams that need to be buffered, the data must be copied to all buffer servers, which means that data is distributed throughout the There are multiple copies in the distributed video-on-demand system, so the data storage space is very large, and the data storage space utilization rate of the entire distributed video-on-demand system is very low. In addition, since complete copies of the data exist on multiple servers, the security of the data is greatly reduced, and the possibility of data being stolen by intruders such as hackers is increased.

Contents of the invention

The technical problem to be solved by the present invention is to overcome the disadvantage of low utilization rate of storage space caused by multiple copies of data streams in distributed buffer servers in the existing distributed video-on-demand system. The technical scheme that realizes the technical problem to be solved by the present invention and takes is summarized as follows: On the one hand, propose the method for realizing data storage and access in the distributed video-on-demand system, it is characterized in that the method for data storage comprises the following steps:

A. Divide the data file into blocks according to the set method;

B. Calculate the set number of data blocks to form a check data block;

C. storing the data block and the verification data block in a set manner in turn in multiple set buffer servers;

D. Generate index information stored in corresponding data blocks in each buffer server. The method of data access includes the following steps:

a. The client establishes a connection with the server;

b. The client reads the index information corresponding to the data file to be accessed from the index server: c. reads the data blocks in corresponding multiple buffer servers according to the index information;

d. De-verify the data block group that has been read to reach the group density;

e. The client reorganizes the read data blocks and restores the data files.

On the other hand, a distributed video-on-demand system is proposed, including several video clients, several buffer servers and index servers storing data files, and the clients and servers are connected and communicated through a communication network, characterized in that :

The client sends a video file request to the index server, and reassembles the data blocks read from the buffer server;

The index server controls the data file to be divided into data blocks according to the set method, calculates the set number of data blocks to form a verification data block, and controls the data block and the verification data block to be stored in turns according to the set method. block storage servers;

The buffer server records the index information stored in the data block and stores the data block.

By adopting the technical solution of the invention, compared with the prior art, the system data storage and access methods are improved, the effect of improving storage efficiency is achieved, and a large amount of storage space is saved. At the same time, because the data is stored in a distributed manner, and the data files are distributed among multiple buffer servers, illegal intruders (such as hackers) cannot obtain complete file data even if they break through one or several servers. The technical scheme of the invention improves the security of the data; on the other hand, due to the adoption of the preferred data redundancy check method, even if a certain buffer server is abnormal, the file data blocks stored in other buffer servers can also be saved. Damaged data blocks can be recovered, so as to realize the reorganization and recovery of data files and improve the reliability of data distribution.

Description of drawings

Fig. 1 is an embodiment of the distributed video-on-demand system provided by the present invention; Fig. 2 is the data structure of the main index information table of the system shown in Fig. 1: Fig. 3 is the secondary index information of the system shown in Fig. 1 Data structure; Fig. 4 is the storage flowchart of the system shown in Fig. 1 to the data file; Fig. 5 is the generation method flow diagram of verification data after block in Fig. 4; Fig. 6 is the flow process of distributed storage processing in Fig. 4 picture;

FIG. 7 is a flow chart of accessing data in the system shown in FIG. 1 .

Detailed ways

An embodiment of the present invention will be illustrated below in conjunction with the accompanying drawings.

Fig. 1 is an embodiment of the distributed video-on-demand system provided by the present invention. In the following description, assume that the video-on-demand system client 10 sends a data request from the index server 50 through the communication network 20, and the data files are evenly stored on N (N is a natural number) secondary servers 60 (the present embodiment is based on a two-level structure , so the cache server is called a secondary server). The user's request is first processed at the index server 50, and the distribution of the requested data in N secondary servers 60 is searched, and then the requested data is sent to the client terminal 10 through the communication network by multiple secondary servers 60, The client terminal 10 completes the data reassembly process.

The index server 50 holds a plurality of catalog tables. One record is generated for each data file. Whenever a data file is stored in a distributed manner, a corresponding main index information 30 is generated. The main index information is a table formed by multiple rows of data, called the main index data table 200. The main index data table is shown in FIG. 2 Show. For the data distributed on each server, there is a secondary index data table 40 on the secondary server. The secondary index data table is shown in Figure 3.

The data records contained in the main index data table 200 include: the data file name, which is used to indicate the distribution degree of the data distribution; the first secondary server identifier of the distributed storage-the initial server; used to identify the set data block in a block group Number of group densities; sequence of secondary server names where data is stored. If the client 10 needs to request a certain data file, it will obtain the main index 30 by connecting to the index server 50, read the main index data table 200 belonging to the requested file, and obtain the distribution information, the initial secondary server, the group density and the distribution information of the file therefrom. Sequence of server names.

After obtaining the primary index information, the client 10 will establish connections one by one with the secondary servers in the server name sequence in the primary index data table 200, utilize the retrieval information request data provided by the secondary index data table 210, and connect from the primary Two secondary servers——the initial server, start until the last data block is read, and the whole process ends.

The secondary index data table 210 contains the first record is the data file name, the data file name is associated with the data file name field in the main index data table, and the second record is the serial number of the data block, which is used to mark the data The position of the block in the source file, the third record is the previous server, the fourth record is the next server, these two records indicate the storage location of the adjacent data block, and the last record is the storage address of the data block , the storage address includes the storage location of the data block in the external memory 70 .

FIG. 4 is a flow chart of storing data files in the system shown in FIG. 1 . When storing data files, first execute the step 300 of reading and analyzing the file by the index server, and then execute the step 310 of dividing the file according to the parameters of file segmentation. In this step, the size of the data block can be specified. It is set that the size of each data block can be equal or unequal. In this embodiment, the size of each data block is equal, that is, the method of uniform block is adopted; after the block is completed, the set number of The data block forms a data block group, and the step 320 of forming a verification data block is performed for each block group calculation, and then

Put the verification data into the data block group as one of the block group data to form a new data block group, and finally perform the process of distributing and storing all the data blocks in the external memory 70 of the designated N secondary servers in turn. Step 330.

Fig. 5 is the flow chart of the generation method of verification data after block in Fig. 4, after having set the block number (i.e. group density) in data block size and block group, main steps are as follows:

1. Step 400 of reading the specified data block data;

2, judge whether to reach the step 410 of last data block, if reach last block, go to step 426 processing, otherwise, execute step 420, promptly judge whether the block number (group density) that data block group designation reaches, if not reach, Then continue to read the next data block, and execute step 400, otherwise it means that a data block group has been read, and execute step 425;

3. Step 425, calculate and generate a verification data block for the read data block group, and then continue to execute step 400;

4. Step 426, judge whether the block number (group density) specified by the data block group reaches, if reached, execute step 440, otherwise execute step 430;

5. Step 430, partially fill the data blocks with insufficient quantity to supplement the insufficient data blocks, and after supplementing the data blocks to form a block group, generate the last data block group; 6. Step 440, calculate and generate the last data block group Verifying the data block; 7. Step 450, performing distributed storage processing on all the data blocks together with the verifying data block, see FIG. 6 for details.

In the present embodiment, there are many methods for calculating and generating check data blocks, here are a few specific examples, assuming group density 200d=M (M is a natural number), correspondingly each block group is D1, Dz, .. .DM, the verification data to be generated is P, and the following verification methods can be adopted:

(1) parity check method: also known as the RAID method, the method is to adopt the method of binary addition bit by bit, according to this method, P=D1 field DZ.....DM. or P=one (D10DZ, .....Gong), where "." represents "exclusive OR" operation, and "one" represents "not" operation. Assuming that an error occurs in the block group, then correspondingly, the data recovery method corresponding to the verification is: a few=D1. DI.DZ....Dk-loP Tian Dk+1....DM).

(2) modulus operation method: method is, block group is regarded as binary number, suppose the number binary digit of each block is K, all block group numbers are added, and ZAK is taken modulus, namely: P==( D1+DZ+...+nM) mod ZAK. Assuming that an error occurs in Dk in the block group, the data recovery method corresponding to the verification is: Dk=P+2-(DI+DZ+...+Dk-1+D-1...+DM)modZAK.

(3) Mirroring method: This method is to generate an identical data copy block for each block, which is equivalent to generating two identical distributed copies in the system.

There are many methods, and we will not give examples one by one here. All the methods must meet the following conditions: Generally, the generation method of the verification data P is: for the data D1, DZ, ..., DM, the corresponding verification generation function f (·), satisfying P=f(D,, DZ,..., DM), correspondingly, for any data recognition, corresponding to a verification data recovery function g(·), satisfying Dk=g(D,, DZ ,—Dk-1, P, D Bubu—DM), the functions f(·) and g(·) can be the same or different.

Fig. 6 is a flowchart of distributed storage processing in Fig. 4, including the following steps:

1. Step 500, setting distribution parameters, that is, setting the number of servers participating in distributed storage and all server names, and generating the main index data table 200;

2. Step 510, setting the starting server, that is, determining the first server to start storing;

3. Step 520, read the data block group;

4. Step 530, distributed storage of data block groups and generation of secondary indexes, that is, starting from the starting server, storing the first data block, and generating the secondary index data table information 210 of the block on the server; and then sequentially Connect to the next server, store the second data block, generate the secondary index data table information 210 of the block on the server, until all the data blocks of the block group are stored on the server step by step (including the corresponding verification data piece);

5. Step 540, judge whether the block group is the last block group, if not, reset the starting server, and execute step 510, if yes, it means that all data block groups have been distributed and stored, and the distributed storage is over ;

6. Step 550 ends. In order to further improve the reliability and flexibility of the data, the verification data can be specified in the block group according to certain rules, such as in a fixed position or a dynamic position, where the dynamic position can be stored in a dynamic loop method, such as the verification data Put the first block in the first data block group, put the second block in the second data block group, ... When the position of the check data block reaches the group density, start from the first block again Loop, selectable during implementation.

Fig. 7 is a flow chart of accessing data in the system shown in Fig. 1, which mainly includes the following steps:

1. Step 610, the client 10 connects with the index server 50 through the communication network 20, and requests a certain data file;

2. Step 620, after the connection is established, the index server searches for the main index information through the request, obtains the main index 30, obtains the index data table 200 identifying the requested data file, and extracts the distribution information of the file from 200, including the file name and distribution degree , initial server, group density, server name sequence and other information;

3. Step 630, connect to the initial server;

4. Step 640, call the process of reading the secondary index information, obtain the secondary index 40, obtain the secondary index data table 210 belonging to the request file, and obtain the corresponding file name and data block of the data block in the server from 210 number, previous server, next server, storage location and other information;

5. Step 650, read the data block in the external memory 70;

6. Step 660, judging whether the number of data blocks read has reached the specified group density value for the file: (1) if it reaches the group density value, then perform step 680, that is, go to the reassembly process of the checksum and data blocks, and go to the calibration The verification and reorganization process removes the redundant verification data that was originally used to ensure data reliability. Proceed to step 690;

(2) If the group density value is not reached, execute step 670, that is, connect to the next server, and then continue to execute step 640;

7. Step 690, judge the data block number, and check whether the last data block is reached: (1) if not, execute step 670, that is, connect to the next server, and continue to execute step 640;

(2) If the last data block has been reached, the entire data retrieval access process ends.

Although the disclosed distributed video-on-demand system and the method for implementing data storage and access thereof have been specifically described with reference to the embodiments, those skilled in the art will understand that, without departing from the scope and spirit of the present invention, the Various obvious modifications are made to it in terms of form and details, such as optional data verification method and data distribution density, and at the same time, the two-level mode of server distribution in the embodiment can be changed to a three-level or multi-level mode, and can also be changed to Single-level, index information tables can also be designed by themselves, adding new fields, etc. Therefore, the embodiments described above are illustrative rather than restrictive, and all changes and modifications are within the scope of the invention without departing from the spirit and scope of the invention.

Claims (10)

1, a method for realizing data storage in a distributed video-on-demand system, is characterized in that the method for data storage comprises the following steps: A. Divide the data file into blocks according to the set method; B. Calculate the set number of data blocks to form a check data block; C. storing the data block and the verification data block in a set manner in turn in multiple set buffer servers; D. Generate index information stored in corresponding data blocks in each buffer server. 2. A method for realizing data access in a distributed video-on-demand system, characterized in that the method for data access comprises the following steps: a. The client establishes a connection with the server; b. The client reads the index information corresponding to the data file to be accessed from the index server; c. Read data blocks in corresponding multiple buffer servers according to the index information; d. De-verify the data block group that has been read to reach the group density; e. The client reorganizes the read data blocks and restores the data files. 3. The storage method according to claim 1, characterized in that uniform data blocks are performed on the data files. 4. The storage method according to claim 1, wherein in each data block group, any data block can be restored by verifying the data block and other data blocks. 5. The storage method according to claim 1, wherein the step of calculating and forming a verification data block for a set number of data blocks further includes the following steps: B.1. Determine whether the number of data groups containing the last data block of the file has reached the set number, and if so, calculate and form a check data block; B.2. If it is not reached, fill the insufficient data blocks, and then calculate to form a check data block. 6. The storage method according to claim 1, wherein the step of storing the data blocks in turn in a plurality of servers in a set manner further includes the following steps: C.1. Set the number and name of distributed storage buffer servers; C.2. Determine the initial storage cache server. 7. The storage method according to claim 1, characterized in that the storage location of the verification data block in the data block group is dynamic. 8. The storage method according to claim 1, wherein in the step of generating index information corresponding to data storage in each buffer server, the index information includes the following content: the file corresponding to the stored data block name; The serial number of the stored data block; The name of the server where the adjacent data block of the stored data block is stored; The storage address of the stored data block in the server. 9. The data access method according to claim 2, wherein the step of reading data blocks in corresponding multiple servers according to the index information further comprises the following steps: c.1. Connect to the origin server; c.2. Read the data block corresponding to the file according to the index information; c.3. Judging whether the number of data blocks read is equal to the group density, if not, execute the next step, if so, perform the steps of the client reorganizing the read data blocks and restoring the data file; c.4. Connect to the next server, continue to read data blocks, and execute step c.2. 10, distributed video-on-demand system, comprises several video clients, buffer server and index server of several storage video data files, described client and server are connected and communicated by communication network, it is characterized in that: The client sends a file request to the index server to reorganize the data blocks read from the buffer server; the index server controls the data file to be divided into data blocks according to the set method, and calculates the set number of data blocks to form a check data block, Controlling the data block and the verification data block to be stored in a plurality of data block storage servers in turn in a set manner; The buffer server records the index information stored in the data block and stores the data block.

Images