CN102819406A - Front-end data storage method and device - Google Patents

Abstract

The present invention provides a front-end data storage device, which is applied to a front-end device, and the front-end device is connected to a network storage device through a network. The device includes: a write judgment unit, which is used to judge Whether the currently written index data can cover the entire index area, if so, write it, otherwise go to the data filling unit for processing; the data filling unit is used to fill the index data with invalid data so that the filled index data can be covered corresponding index area, and submit the filled index data to the data writing unit. Compared with the prior art, since the present invention fills the index data at the front end, it avoids the operation of additionally reading disk data caused by reconfiguration writing when writing index data, and improves the performance and efficiency of writing .

Description

A front-end data storage method and device

technical field

The invention relates to data processing technology of front-end terminal equipment, in particular to a front-end data storage method and device.

Background technique

The Redundant Array of Independent Disks (RAID) technology is widely used in the storage field, especially popular in network storage. The RAID technology mainly guarantees the availability of data stored on the disk through the redundancy of the disk. With the continuous increase of the capacity of a single disk, the size of the current mainstream disk has reached the TB level, and the probability of errors or track failures on the same disk has also increased.

When network storage is used in the monitoring system, the front-end encoding device first writes the index area of the video stream, and then writes the video stream data. In order to increase the reliability of the stored data, the storage device is generally configured with a RAID5 array for data protection. Since all video data (even high-definition video data) is stored in surveillance applications, the amount of data is extremely large, which makes users more sensitive to storage operating costs. Therefore, disk fault tolerance technology and prolonging the service life of disks have become customer concerns. Focus.

At present, the network storage has the following problems in monitoring applications: the front-end encoding device usually writes some monitoring index data in the continuous video data stream that needs to be stored, and the monitoring index data is generally smaller than the pre-allocated space, resulting in storage devices that support raid5 Data writing cannot be performed in units of entire stripes. At this time, the storage device may adopt a reconstruction write mode or a read-modify write mode. The process of Reconstruct Write includes: reading the original data from the stripe block (Segment) that does not need to be modified in the current stripe, and calculating the parity with the new data on all the segments that need to be modified in this stripe check value, and write the new Segment data, the Segment data that has not been changed, and the new parity value to the current stripe. Obviously, reconstructing writes involves more I/O operations, so the efficiency is much lower than writing the entire stripe together.

The process of read-modify-write mode is as follows: first read the old data from the Segment that needs to be modified, and then read the old parity value from the stripe; The data calculates the new parity value on this stripe; finally writes the new data and the new parity value.

Please refer to Figure 1, assuming that the level of the RAID array is RAID5. The first thing to note is that the second row of the table indicates the nature of the data stored in the disk space. For example, Index indicates the index area, and data indicates the data area, which is a graphical representation, not disk data. When the front-end encoding device performs data writing, if the data currently to be written includes index data, it may occur that the space in the pre-allocated index area cannot be exhausted. Assume that the data is stored in a RAID5 array with 64KB stripe blocks and 9 disks. Assume that the size of the allocated index area is 128KB. At present, only 64KB of actual index data needs to be written, and the stripe block of Disk2 shown in Figure 1 is empty, and no data needs to be written. At this time, the storage device will find that using reconstructed write is a better performance method. That is, after reading the data of Disk2 (invalid for business), calculate the verification data Disk9, and then write the data of Disk1-9 at one time. Assuming that a read error occurs when reading the data on Disk2 at this time, it is necessary to read Disk1 first, and calculate the data on Disk2 from Disk3 to Disk9. Big discount. Further, assuming that at this time, if Disk8 has a read error due to various reasons, the data on Disk2 cannot be calculated at this time because RAID5 only supports it, and the reconstruction write cannot be completed.

Contents of the invention

In view of this, the present invention provides a front-end data storage device, which is applied to a front-end device, and the front-end device is connected to a network storage device through a network, and the device includes:

The writing judging unit is used to judge whether the currently written index data can cover the entire index area when the index data is written into the corresponding index area, if it is processed by the data writing unit, otherwise it is processed by the data filling unit; The index area is a storage space pre-allocated by the network storage device, which includes one or more stripe blocks allocated by the network storage device;

The data filling unit is used to fill the index data with invalid data so that the filled index data can cover the corresponding index area, and submit the filled index data to the data writing unit;

The data writing unit is configured to write index data into a corresponding index area.

The present invention also provides a front-end data storage method, which is applied to a front-end device, and the front-end device is connected to a network storage device through a network, and is characterized in that the method includes:

A. When writing index data into the corresponding index area, judge whether the currently written index data can cover the entire index area, if it is processed by the data writing unit, otherwise it is processed by the data filling unit; wherein the index area is the network Storage space pre-allocated by the storage device, which includes one or more stripe blocks allocated by the network storage device;

B. Fill the index data with invalid data so that the filled index data can cover the corresponding index area, and submit the filled index data to the data writing unit;

C. Write the index data into the corresponding index area.

Compared with the prior art, since the present invention fills the index data at the front end, it avoids the operation of additionally reading out the disk data caused by reconfiguration writing when writing the index data, and improves the performance and efficiency of writing ; At the same time, it also avoids the problem of inability to write caused by the read error of two stripe blocks in the reconstruction write.

Description of drawings

Fig. 1 is a schematic diagram of organization of index data and video data on a storage device in the prior art.

Fig. 2 is a logical structure diagram of a front-end data storage device in an embodiment of the present invention.

Fig. 3 is a flowchart of a front-end data storage method in an embodiment of the present invention.

Fig. 4 is a schematic diagram of organization of index and video data on a network storage device in an embodiment of the present invention.

Detailed ways

Please refer to FIG. 2 , the present invention provides a front-end data storage device, which is applied to a front-end device, and the front-end device is connected to a network storage device through a network. Taking computer program implementation as an example, the front-end data storage device includes a writing judging unit, a data filling unit and a data writing unit. Take the front-end device in the video surveillance system as a front-end encoding device (such as an encoder EC) as an example for illustration. The present invention is not limited to the front-end encoding equipment of the video surveillance system. When applied in other systems or networks, the difference is only the business itself. In the monitoring system, the business data is video data, while in other systems or networks, it may be other types of business data. The front-end data storage device performs the following steps during operation:

Step 101, the writing judging unit judges whether the currently written index data can cover the entire index area when the index data is written into the corresponding index area, if it is processed by the data writing unit, otherwise it is processed by the data filling unit; The index area is a storage space pre-allocated by the network storage device, which includes one or more stripe blocks allocated by the network storage device;

In addition to sending the live video stream to the decoding end, the EC usually also needs to send a stored video stream to the network storage device in the video surveillance system, so that the user can play back the stored video later. Taking IP storage as an example, EC will establish an iSCIS connection with the network storage device, and then send the data to be written to the network storage device through this iSCIS. The operation of EC to store the video stream can be decomposed into two data storage operations according to the data type, one is to write index data to the index area, and the other is to write video data to the data area. The prior art does not consider the respective characteristics of writing index data and writing video data. In fact, video data is usually continuous data, but index data is usually not continuous data. The present invention requires the writing judging unit to jump out of the normal data writing process when the EC needs to write the index data to the storage device, and then go to step 102 for processing by the data filling unit.

Step 102, the data filling unit fills the index data with invalid data so that the filled index data can cover the corresponding index area, and submits the filled index data to the data writing unit;

As mentioned above, since video data is usually continuous data, generally there is no situation that a stripe on a storage device cannot be filled with video data. Please refer to FIG. 4 , assuming that the network storage device has a stripe block size of 64KB, which includes a RAID5 array of 9 disks. Assume that the index area allocated by the network storage device to the EC is 128KB (the size of two stripe blocks). If the current actual index data of the EC is only 64KB, the data filling unit can fill another 64KB of invalid data at this time. Then submit the filled index data to the data writing unit.

Step 103, the data writing unit writes the index data into the corresponding index area.

Step 104, the network storage device receives the index data sent by the EC and stores the index data in the cache, and waits for the arrival of the video data pointed to by the index data;

Step 105, the data writing unit continues to write the video data corresponding to the index data into the data area;

In step 106, the network storage device writes the received video data sent by the EC and the index data in the cache to form a whole strip of data into the RAID array.

In a preferred embodiment, the EC can establish a two-level index on the network storage device. The present invention has a better effect on writing index data in the secondary index area, because the secondary index area will continuously write index data, and the secondary index A zone's index data points to a corresponding data zone. There are relatively few changes in the first-level index area, and relatively more changes in the second-level index area. The index data in it can be updated synchronously with business data (such as video data). For example, in the storage of video data, user-planned The storage space may be limited. For example, it can only store a month's worth of space. When one month arrives, the previous video data will be gradually covered by new video data, so the corresponding index data will also be updated synchronously.

As mentioned above, the index data in the secondary index area points to a data area, where the size of the index data may change due to the change in the number of data blocks in the data area. If the user divides the current data area into more data blocks, the index data will undoubtedly increase, and if it is divided into fewer data blocks, then the index data will undoubtedly become smaller, so the size of the index data depends on the user's use of the data area . So in fact, when planning the size of the secondary index area, the user's use of the data area is usually taken into consideration, so the index area size is relatively large to ensure that it can accommodate the index data to be written.

Please refer to FIG. 4 , the index area to be written by the EC includes two stripe blocks on one stripe, one stripe is the user's index data, and the other is invalid data randomly filled. After the subsequent video data arrives, the network storage device can obtain the data of a complete strip, and then perform the write operation of the entire strip data. There is no need to perform reconstruction and writing operations as in the prior art.

Compared with the prior art, since the EC of the present invention fills the index data at the front end, the network storage device always writes data in the form of the whole stripe. First of all, it avoids the need to read additional disk data operations caused by refactoring when writing index data, and improves the performance and efficiency of writing. Secondly, since the writing of the entire stripe does not involve a read operation, the problem of inability to write caused by the read error of two stripe blocks in the reconstruction writing is avoided, because if there is a problem with the current stripe block, for example, a track fault causes the There is a problem that the data cannot be read with the block. At this time, the problem can only be solved by writing. When the network storage device writes data to the network, if a track failure occurs in the physical space where the current stripe block is located, the disk of the network storage device usually supports replacing the stripe block to another physical space at the bottom layer. This feature cannot be applied when reading data, because the required data does not exist in another space. The present invention just utilizes the data writing characteristic of the bottom layer of the network storage device to solve the storage problem in the prior art. In addition, considering that there are usually a large number of ECs in a video surveillance system, a small modification of the EC at the software level can greatly improve the performance and efficiency of the network storage device.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (8)

1. a front end data memory storage is applied on the front-end equipment, and this front-end equipment is connected to the network storage equipment through network, it is characterized in that, this device comprises:

Write judging unit, be used for when writing index data and write the manipulative indexing district, judging whether the current index data that writes can cover whole index area, if revolution handles according to writing unit, otherwise revolution is handled according to filler cells; Wherein said index area is the pre-assigned storage space of the network storage equipment, and it comprises the band piece that one or more network storage equipment distributes;

The data filler cells be used to use invalid data index data to be filled so that the index data after filling can cover corresponding index area, and the index data after will filling is submitted to data write unit;

Data write unit is used for index data is write corresponding index area.

2. device as claimed in claim 1 is characterized in that, said index data is the secondary index data, and said index area is the secondary index district on the network storage equipment.

3. device as claimed in claim 2 is characterized in that, said data write unit is further used for business datum is write corresponding data field, and said index data points to corresponding data field.

4. device as claimed in claim 3 is characterized in that, said front-end equipment is the front end encoding device, and said business datum is a video data, and wherein said index data is along with video data upgrades synchronously.

5. a front end data storage means is applied on the front-end equipment, and this front-end equipment is connected to the network storage equipment through network, it is characterized in that, this method comprises:

A, when writing index data and write the manipulative indexing district, judge whether the current index data that writes can cover whole index area, if revolution handles according to writing unit, otherwise revolution is handled according to filler cells; Wherein said index area is the pre-assigned storage space of the network storage equipment, and it comprises the band piece that one or more network storage equipment distributes;

B, use invalid data are filled index data so that the index data after filling can cover corresponding index area, and the index data after will filling is submitted to data write unit;

C, index data is write corresponding index area.

6. method as claimed in claim 5 is characterized in that, said index data is the secondary index data, and said index area is the secondary index district on the network storage equipment.

7. method as claimed in claim 5 is characterized in that, said step C further comprises: business datum is write corresponding data field, and said index data points to corresponding data field.

8. method as claimed in claim 7 is characterized in that, said front-end equipment is the front end encoding device, and said business datum is a video data, and wherein said index data is along with video data upgrades synchronously.

Images