WO2024074042A1 - Data storage method and apparatus, data reading method and apparatus, and device - Google Patents

Abstract

The present application discloses a data storage method. The method comprises the following steps: receiving a target data set to be stored; obtaining the data size of each piece of data in said target data set, wherein the size of each piece of data in said target data set is the same; and storing each piece of data in said target data set into target blocks in a hard disk which are continuous and have the same size, wherein the block size of each target block is determined according to the data size. The present application further discloses a data storage apparatus, a data reading method and apparatus, a device, and a storage medium.

Description

Data storage method and device, data reading method and device, and equipment

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to the Chinese patent application filed with the China Patent Office on October 8, 2022, with application number 202211219584.2, and application name “A data storage method and device, data reading method and device, and equipment”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of data processing technology, and in particular to a data storage method and device, a data reading method and device, a device and a non-volatile storage medium.

Background technique

Artificial intelligence has developed rapidly in recent years. Machine learning in artificial intelligence requires the collection, labeling and preprocessing of data sets, etc. Only then can it be read and used in the training and inference of machine learning and deep learning.

However, the reading and writing of data sets may have a great negative impact on the overall performance of artificial intelligence training and inference. The main reasons include: (1) Depending on the needs of different algorithms, the number of data sets may be tens of thousands or more (each of which is, for example, an image, text or voice); (2) The data set needs to be preprocessed and written to the hard disk as usable training/test data; (3) After the data set is preprocessed, each data item will usually become smaller and its size is fixed; (4) After the above three steps are completed, the training and inference process is actually "reading" thousands of small data sets for calculation. In other words, to access a data set, it is actually necessary to execute many system programs and spend time searching the hard disk for all data items of the data set to restore it to the original data set. It takes a lot of time to search for most of the discontinuous blocks on the hard disk before it can be combined into the original data set, resulting in low data reading and writing efficiency.

In summary, how to effectively solve the problem of spending a lot of time searching for most of the discontinuous blocks on the hard disk to combine them into the original data set, resulting in low data reading and writing efficiency, is an issue that technicians in this field urgently need to solve.

Summary of the invention

An object of an embodiment of the present application is to provide a data storage method, which saves data reading time and improves data reading and writing efficiency; another object of the present application is to provide a data storage device, a data reading method and device, an equipment and a non-volatile storage medium.

In order to solve the above technical problems, this application provides the following technical solutions:

According to a first aspect of an embodiment of the present application, a data storage method is provided, including:

receiving a target data set to be stored;

Obtain the data size of each item of data in the target data set; wherein the size of each item of data in the target data set is the same;

Each data in the target data set is stored in each continuous target block of the hard disk with the same size; wherein the block size of each target block is determined according to the data size.

In a specific implementation of the present application, after receiving the target data set to be stored and before obtaining the data size of each item of data in the target data set, the method further includes:

A first preprocessing operation is performed on the target data set; wherein the first preprocessing operation is a preprocessing operation that does not increase the data size.

In a specific implementation of the present application, a first preprocessing operation is performed on the target data set, including:

Perform preprocessing operations on the target dataset except normalization preprocessing.

In a specific implementation of the present application, receiving a target data set to be stored includes:

Receive a target data set to be stored for artificial intelligence model training.

In a specific implementation of the present application, obtaining the data size of each item of data in the target data set includes:

Get the data size of each data item in the target data set, which consists of the data itself, data label, and data file name.

In a specific implementation of the present application, a process of determining a block size of a target block according to the data size is also included. The process of determining the block size of a target block according to the data size includes:

Get the preset optional block sizes;

The block size of the target block is obtained by selecting from various optional block sizes that are larger than the data size.

In a specific implementation of the present application, the block size of the target block is selected from various optional block sizes that are larger than the data size, including:

Selecting the optional block size with the smallest difference with the data size from the optional block sizes that are larger than the data size;

The optional block size with the smallest difference from the data size is determined as the block size of the target block.

In a specific implementation of the present application, after obtaining each preset optional block size, it also includes:

Determine whether the data size is less than or equal to the maximum value of each optional block size;

If yes, the step of selecting a block size of the target block from each optional block size larger than the data size is performed;

If not, the maximum value among the optional block sizes is determined as the block size of the target block.

According to a second aspect of an embodiment of the present application, a data reading method is provided, comprising:

Receive data read command;

Reading each item of data of the target data set from each target block of the hard disk that is continuous and has the same size; wherein the block size of each target block is determined according to the data size of each item of data, and the size of each item of data in the target data set is the same;

The read data are returned to the sender of the data read command.

In a specific implementation of the present application, after reading each item of data of the target data set from each consecutive target block of the same size in the hard disk, and before returning each item of data read to the sender of the data read command, the method further includes:

A second preprocessing operation is performed on each of the read data; wherein the second preprocessing operation is a preprocessing operation for increasing the size of the data.

In a specific implementation of the present application, a second preprocessing operation is performed on each piece of data read, including:

Perform normalization preprocessing operations on the read data.

In a specific implementation of the present application, receiving a data read command includes:

Receive a data read command to read a target data set for artificial intelligence model training.

In a specific implementation of the present application, reading each item of data of the target data set from consecutive target blocks of the same size in the hard disk includes:

When the block size of the target block is greater than or equal to the data size, each data item of the target data set is read from each consecutive target block of the same size in the hard disk according to the one-to-one relationship between the target block and each data item.

In a specific implementation of the present application, reading each item of data of the target data set from consecutive target blocks of the same size in the hard disk includes:

When the block size of the target block is smaller than the data size, each data item of the target data set is read from each consecutive target block of the same size in the hard disk according to the many-to-one relationship between the target block and each data item; wherein each data item is pre-stored in adjacent consecutive blocks.

According to a third aspect of an embodiment of the present application, there is provided a data storage device, including:

A data set receiving module, used for receiving a target data set to be stored;

A data size acquisition module is used to acquire the data size of each item of data in the target data set; wherein the size of each item of data in the target data set is the same;

The data storage module is used to store each data in the target data set into each continuous target block of the same size in the hard disk; wherein the block size of each target block is determined according to the data size.

According to a fourth aspect of an embodiment of the present application, there is provided a data reading device, including:

A read command receiving module, used for receiving a data read command;

The data reading module is used to read each item of the target data set from each consecutive target block of the same size in the hard disk. wherein the block size of each target block is determined according to the data size of each item of data, and the size of each item of data in the target data set is the same;

The data return module is used to return the read data to the sender of the data read command.

According to a fifth aspect of an embodiment of the present application, there is provided an electronic device, including:

Memory for storing computer programs;

The processor is used to implement the steps of the above data storage method or data reading method when executing a computer program.

According to a sixth aspect of an embodiment of the present application, a non-volatile readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above data storage method or data reading method are implemented.

The data storage method provided in the embodiment of the present application receives a target data set to be stored; obtains the data size of each data item in the target data set; wherein the size of each data item in the target data set is the same; stores each data item in the target data set into consecutive target blocks of the same size in the hard disk; wherein the block size of each target block is determined according to the data size.

It can be seen from the above technical solution that by setting the block size of the target block of the hard disk according to the fixed size of each data item in the target data set to be stored, it is ensured that each data item in the target data set is stored in a continuous block of the hard disk. This greatly optimizes data storage, and when reading data, it can be directly read from the continuous block of the hard disk, saving data reading time and improving data reading and writing efficiency.

Correspondingly, the embodiments of the present application also provide a data storage device, a data reading method and device, an apparatus and a non-volatile storage medium corresponding to the above-mentioned data storage method, which have the above-mentioned technical effects and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flowchart of an implementation method of data storage in some embodiments of the present application;

FIG2 is another implementation flow chart of the data storage method in some embodiments of the present application;

FIG3 is a flowchart of an implementation of a data reading method in some embodiments of the present application;

FIG4 is another implementation flow chart of the data reading method in some embodiments of the present application;

FIG5 is a schematic diagram of a handwriting recognition data set image file in some embodiments of the present application;

FIG6 is a schematic diagram of a normalized handwriting recognition data set image file in some embodiments of the present application;

FIG7 is a schematic diagram of a data set image classification in some embodiments of the present application;

FIG8 is a structural block diagram of a data storage device in some embodiments of the present application;

FIG9 is a structural block diagram of a data reading device in some embodiments of the present application;

FIG10 is a structural block diagram of an electronic device in some embodiments of the present application;

FIG. 11 is a schematic diagram of a specific structure of an electronic device provided in some embodiments of the present application.

Detailed ways

In the existing data storage method, the operating system and the file system do not guarantee the continuity of each piece of data stored in the hard disk. In other words, each piece of data will be cut into data blocks because of the block size planned by the hard disk and the file system. The continuity cannot be guaranteed and the data is often stored discontinuously in the hard disk. The part of reading data still requires the CPU (Central Processing Unit), main memory and hard disk I/O (Input/Output) system plus related software and operating system to complete data reading and writing. This means that the GPU (Graphics Processing Unit), CPU, main memory and hard disk I/O as well as related software and operating system need to communicate and transmit non-contiguous data with each other frequently. In other words, if a user wants to access a piece of data at the application layer, in fact, Many operating system and file system programs need to be executed, and time needs to be spent on searching all the data blocks of the data in the hard disk to combine and restore the original files. As a result, the reading and writing of the data set may have a great negative impact on the overall performance of artificial intelligence training and inference, thereby reducing the reading and writing performance.

To this end, in the data storage method provided in the present application, it is ensured that all data in the target data set are stored in continuous blocks of the hard disk, so that data storage is greatly optimized. When reading data, it can be read directly from continuous blocks of the hard disk, saving data reading time and improving data reading and writing efficiency.

In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

Referring to FIG. 1 , FIG. 1 is a flowchart of an implementation method of a data storage method in an embodiment of the present application. The method may include the following steps:

S101: Receive a target data set to be stored.

When a pre-acquired target data set (DataSet) needs to be stored, the target data set to be stored is sent to the accelerator. For example, the target data set to be stored may be sent to the accelerator via a CPU or a GPU, and the accelerator receives the target data set to be stored.

The target dataset can be a dataset used for artificial intelligence machine learning training, such as a dataset used for training an image recognition model, or a dataset used for an item recommendation model, etc. The data types in the target dataset can be pictures, text, voice, etc.

The CPU host or GPU host and the accelerator may be connected via a physical connection or via a network, which is not limited in the embodiments of the present application.

S102: Obtain the data size of each item of data in the target data set.

Among them, the sizes of each data item in the target data set are the same.

After receiving the target data set to be stored, the target data set can be preprocessed and transformed into usable training data or test data so that the size of each data item in the target data set to be stored is the same, and the data size of each data item in the target data set is obtained.

Take image preprocessing for deep learning as an example. Given a compressed image, one or more of the following preprocessing steps are usually performed:

(1) Image decoding: Decode the compressed image. Color images are decoded and stored in three pixel channels: R (Red), G (Green), and B (Blue). Some model algorithms will need to be trained on one or more of the R, G, and B channels later;

(2) Grayscale conversion Grayscale conversion simply converts an image from color to black and white. It is often used to reduce computational complexity in artificial intelligence algorithms. Since most images do not require color recognition, it is wise to use grayscale conversion, which reduces the number of pixels in the image and thus reduces the amount of computation required;

(3) Normalization: Normalization is the process of projecting image data pixels (intensities) to a predefined range, usually (0, 1) or (-1, 1), but different algorithms have different definitions. Its purpose is to improve fairness for all images. For example, scaling all images to an equal range of [0, 1] or [-1, 1] allows all images to contribute equally to the total loss, rather than having strong and weak losses when other images have high and low pixel ranges, respectively. The purpose of normalization also includes providing a standard learning rate. Since high-pixel images require a low learning rate, while low-pixel images require a high learning rate, rescaling helps provide a standard learning rate for all images.

(4) Data Augmentation: Data augmentation is the process of making small changes to existing data to increase its diversity without collecting new data. This is a technique used to expand a dataset. Standard data augmentation techniques include horizontal and vertical flipping, rotation, cropping, shearing, etc. Performing data augmentation helps prevent neural networks from learning irrelevant features and improves model performance.

(5) Image standardization: Standardization is a method of scaling and preprocessing images to make them have similar or consistent height and width. When training, testing, and inference of artificial intelligence, if the size of the image is consistent, the processing will be more efficient.

S103: storing each data item in the target data set into consecutive target blocks of the same size in the hard disk; wherein the block size of each target block is determined according to the data size.

The minimum write block size available for selection is pre-set, such as 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes, etc. Some solid state drives can support a larger range of block sizes. After obtaining the data size of each data item in the target data set, the block size of the target block is determined according to the data size. If it is determined that there is a selectable block size greater than or equal to the data size, the block size closest to the data size is selected from the selectable block sizes greater than or equal to the data size.

After determining the block size of the target block according to the data size, the data in the target data set is stored in each target block of the same size in the hard disk. Compared with the existing method of reading data stored discontinuously in the hard disk, the present application can directly read from the continuous blocks of the hard disk when reading data, saving the time of data reading and improving the efficiency of data reading and writing.

It can be seen from the above technical solution that by setting the block size of the target block of the hard disk according to the fixed size of each data item in the target data set to be stored, it is ensured that each data item in the target data set is stored in a continuous block of the hard disk. This greatly optimizes data storage, and when reading data, it can be directly read from the continuous block of the hard disk, saving data reading time and improving data reading and writing efficiency.

It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

Referring to FIG. 2 , FIG. 2 is another implementation flow chart of the data storage method in an embodiment of the present application. The method may include the following steps:

S201: Receive a target data set to be stored.

In a specific implementation of the present application, step S201 may include the following steps:

Receive a target data set to be stored for artificial intelligence model training.

When an artificial intelligence model needs to be trained, a target data set for artificial intelligence model training is collected in advance and sent to an accelerator, which receives the target data set for artificial intelligence model training to be stored.

The target dataset can contain training set, validation set and test set.

First, the model is fitted on a training dataset. For supervised learning, the training set is a collection of examples used to fit parameters (such as the weights of the connections between neurons in an artificial neural network). In practice, the training set is usually a data pair consisting of an input vector and an output vector. The output vector is called the target. During training, the current model makes predictions for each example in the training set and compares the predictions with the target. Based on the results of the comparison, the learning algorithm updates the parameters of the model. The process of model fitting may include both feature selection and parameter estimation.

Next, the fitted model is used to make predictions on a validation dataset. The validation set provides an unbiased evaluation of the model fitted on the training set when tuning the model's hyperparameters (e.g., the number of neurons in the hidden layer of a neural network). The validation set can be used for early stopping in regularization, i.e., stopping training when the validation set error rises (a sign of overfitting on the training set).

Finally, the test dataset can be used to provide an unbiased evaluation of the final model. If the test dataset is never used during training (for example, not used in cross-validation), it is also called a holdout set.

Many artificial intelligence algorithms only need to use the training set as training, and then use the test set or validation set as a test of the training completion, and then deploy inference. That is to say, the test set and validation set can be the same set.

In a specific implementation of the present application, after step S201 and before step S202, the method may further include the following steps:

A first preprocessing operation is performed on the target data set; wherein the first preprocessing operation is a preprocessing operation that does not increase the data size.

After receiving the target data set to be stored, a first preprocessing operation is performed on the target data set. In the process of storing the target data set to the accelerator, the first preprocessing operation includes partial preprocessing, which is generally performed without increasing the size of the data. Some preprocessing operations, such as horizontal flipping, vertical flipping, and rotation of the image to be recognized, can avoid data preprocessing during data storage, leading to data enlargement, save storage space, and reduce costs.

In a specific implementation of the present application, performing a first preprocessing operation on the target data set may include the following steps:

Perform preprocessing operations on the target dataset except normalization preprocessing.

Since the floating-point values obtained by normalization processing need to occupy more storage space, after receiving the target data set to be stored, when preprocessing the target data set, preprocessing operations other than normalization preprocessing are performed on the target data set, thereby saving storage space.

S202: Obtain the data size of each item of data in the target data set.

Among them, the sizes of each data item in the target data set are the same.

In a specific implementation of the present application, step S202 may include the following steps:

Get the data size of each data item in the target data set, which consists of the data itself, data label, and data file name.

In addition to the data itself, each data item in the target data set also includes a data tag and a data file name, so that the data itself, the data tag and the data file name together constitute a complete data item. After receiving the target data set to be stored, the data size of each data item in the target data set consisting of the data itself, the data tag and the data file name is obtained.

The data label is the reference standard corresponding to the data item, and the data file name is the identification information that uniquely identifies the data item.

S203: Obtain preset optional block sizes.

The minimum write block size that can be selected is preset. For example, for a solid state drive (SSD), it can usually be set to 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes, etc. Some solid state drives can support a larger range of block sizes. After obtaining the data size of each data item in the target data set, obtain the preset optional block sizes.

S204: Determine whether the data size is less than or equal to the maximum value of the optional block sizes. If the data size is less than or equal to the maximum value of the optional block sizes, execute step S205; if the data size is greater than the maximum value of the optional block sizes, execute step S206.

After obtaining the preset optional block sizes, determine whether the data size is less than or equal to the maximum value of the optional block sizes. If the data size is less than or equal to the maximum value of the optional block sizes, it means that there is a selectable block size so that each data item is stored in only one complete block, and step S205 is executed. If the data size is greater than the maximum value of the optional block sizes, it means that the size of each data item has exceeded the maximum supported block size of the hard disk, and multiple blocks are required to accommodate each data item, and step S206 is executed.

S205: Selecting a block size of a target block from various optional block sizes that are larger than the data size.

When it is determined that the data size is less than or equal to the maximum value of the optional block sizes, it means that there is an optional block size, so that each data item is stored in only one complete block, and the block size of the target block is selected from the optional block sizes that are larger than the data size. This enables the storage of each data item in the target data set to a continuous separate block on the hard disk, and optimizes the writing of the target data set to the hard disk, so that subsequent data reading can be directly read from the continuous blocks on the hard disk, saving data reading time and improving data reading and writing efficiency.

In a specific implementation of the present application, step S205 may include the following steps:

Step 1: Select the optional block size with the smallest difference with the data size from the optional block sizes that are larger than the data size;

Step 2: Determine the optional block size with the smallest difference with the data size as the block size of the target block.

For the convenience of description, the above two steps can be combined for explanation.

When it is determined that the data size is less than or equal to the maximum value of each optional block size, the optional block size with the smallest difference from the data size is selected from the optional block sizes larger than the data size, and the optional block size with the smallest difference from the data size is determined as the block size of the target block. This enables the storage of each data in the target data set to a continuous separate block on the hard disk, so that subsequent data reading can be directly read from the continuous blocks on the hard disk, saving data reading time and improving data reading and writing efficiency.

For example, when the size of each data item is 3136 bytes, if the available block sizes are 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes, etc., then the target block size is determined to be 4096 bytes. Group.

S206: Determine the maximum value among the optional block sizes as the block size of the target block.

When it is determined that the data size is larger than the maximum value of each optional block size, it means that the size of each data item has exceeded the maximum supported block size of the hard disk, and multiple blocks are required to accommodate each data item. In this case, the maximum value of each optional block size is determined as the block size of the target block. Taking the handwriting recognition data set as an example, each item size is 3200 bytes. When the maximum value of the optional block size is 2048 bytes, the block size of the target block is set to 2048 bytes. Although each data item is stored in multiple blocks, the data is still read in continuous blocks when it is read, and the performance is still very high.

S207: storing each data item in the target data set into consecutive target blocks of the same size in the hard disk; wherein the block size of each target block is determined according to the data size.

Referring to FIG. 3 , FIG. 3 is a flowchart of an implementation of a data reading method in an embodiment of the present application. The method may include the following steps:

S301: Receive a data read command.

After the target data set is stored in the accelerator, when the target data set needs to be read, a data read command is sent to the accelerator, such as the CPU or GPU sending the data read command to the accelerator, and the accelerator receives the data read command.

S302: reading each data item of the target data set from each consecutive target block of the hard disk with the same size; wherein the block size of each target block is determined according to the data size of each data item, and the size of each data item in the target data set is the same.

The target data set pre-stored in the accelerator is pre-processed before storage so that the size of each data item in the target data set is the same, and the block size of the target block is determined according to the data size of each data item in the target data set, so that the target data set is stored in a continuous block. After receiving the data read command, the accelerator reads each data item of the target data set from each continuous and same-sized target block in the hard disk. By reading each data item of the target data set from continuous blocks, the data reading rate is greatly improved.

S303: Return the read data to the sender of the data read command.

After reading each item of data in the target data set from each continuous block of the block size of the target block in the hard disk, the read data is returned to the sender of the data read command, thereby completing the fast reading of each item of data in the target data set.

The sending end is generally the host CPU or host GPU that interacts with the accelerator to read and write data.

Referring to FIG. 4 , FIG. 4 is another implementation flow chart of the data reading method in the embodiment of the present application. The method may include the following steps:

S401: Receive a data read command.

In a specific implementation of the present application, step S401 may include the following steps:

Receive a data read command to read a target data set for artificial intelligence model training.

The target data set pre-stored in the accelerator may be a data set for artificial intelligence model training. When the artificial intelligence model needs to be trained, a data read command for reading the target data set for artificial intelligence model training is sent to the accelerator. The accelerator receives the data read command for reading the target data set for artificial intelligence model training.

S402: Read each data item of the target data set from each consecutive target block of the hard disk with the same size; wherein the block size of each target block is determined according to the data size of each data item, and the size of each data item in the target data set is the same.

In a specific implementation of the present application, step S402 may include the following steps:

When the block size of the target block is greater than or equal to the data size, each data item of the target data set is read from each consecutive target block of the same size in the hard disk according to the one-to-one relationship between the target block and each data item.

When the block size of the target block is greater than or equal to the data size, it means that each data item is stored in a storage block during data storage. According to the one-to-one relationship between the target block and each data item, each data item of the target data set is read from each continuous and same-sized target block in the hard disk, thereby realizing fast reading of each data item in the target data set.

In a specific implementation of the present application, step S402 may include the following steps:

When the block size of the target block is smaller than the data size, each data item of the target data set is read from each consecutive target block of the same size in the hard disk according to the many-to-one relationship between the target block and each data item; wherein each data item is pre-stored in adjacent consecutive blocks.

When the block size of the target block is smaller than the data size, it means that each data item is stored in adjacent continuous blocks during data storage. According to the many-to-one relationship between the target block and each data item, each data item of the target data set is read from each continuous and same-sized target block in the hard disk. This enables continuous reading of each data item in the target data set, improving data reading efficiency.

S403: performing a second preprocessing operation on each item of the read data; wherein the second preprocessing operation is a preprocessing operation for increasing the size of the data.

After reading each data of the target data set, a second preprocessing operation to increase the data size is performed on each data set read. When the CPU host or GPU host system needs to read a data set, the accelerator first reads the previously partially preprocessed data set from the continuous blocks of the hard disk, and then performs all the remaining preprocessing that was not performed when writing, and finally transmits the data set that has completed all preprocessing back to the CPU host or GPU host system, so that the written part of the preprocessing (processing that does not increase the size) is moved from the CPU host or GPU host system to the accelerator, further freeing up the CPU host or GPU host, achieving a more coordinated adjustment of the preprocessing steps and order, and reducing the size of the data set when storing data.

By not performing the first preprocessing operation to increase the data size when storing data, and only performing the second preprocessing operation to increase the data size on each item in the target data set in the accelerator when reading the data, this process embodies the concept of computational storage. By adopting the strategy of computational storage, the read and write performance of the data set is optimized, that is, "computing power" is exchanged for space reduction. The accelerator implements normalization preprocessing, which actually only passes through the fast matrix operation circuit and then returns the data item to the CPU host or GPU host. This process only requires a small amount of additional computing time and circuit cost, which can greatly reduce the storage space of the data set.

In-storage computing/computing storage refers to placing computing components closer to the storage device, while minimizing the use of the CPU's computing power. This speeds up the overall storage performance and offloads the CPU.

In a specific implementation of the present application, step S403 may include the following steps:

Perform normalization preprocessing operations on the read data.

After reading each item of data in the target data set, normalization preprocessing operations may be performed on each item of data read. For example, using the handwriting recognition data set as an example, the value of each point in the 28x28 array that has not yet been normalized and preprocessed is represented by 0 to 255 to represent its color intensity, which means that only 1 byte is needed to represent the value of each point, but after normalization preprocessing, the floating-point value of each point requires 4 bytes. Therefore, during preprocessing, normalization preprocessing is not performed first, and the data set obtained after partial preprocessing is written to the hard disk, but during reading, normalization preprocessing is performed in the accelerator, and then transmitted back to the CPU host or GPU host. In this way, each data item, compared to storing the normalized floating-point value, only 1/4 of the size is used when storing on the hard disk in this application, and the space occupied is smaller.

S404: Return the read data to the sender of the data read command.

In a specific example application, see Figures 5 and 6. Figure 5 is a schematic diagram of a handwriting recognition data set image file in an embodiment of the present application, and Figure 6 is a schematic diagram of a handwriting recognition data set image file after normalization in an embodiment of the present application. Taking the famous entry-level 0-9 handwriting recognition data set (MNIST handwritten digit database) as an example, the left side of Figure 5 is a 28x28 grayscale monochrome image of "9", and the right side shows the value of each image point in the 28x28 array (normalization preprocessing has not yet been performed), with its color intensity represented by 0 to 255. The left side of Figure 6 is a picture, which is finally represented by a 28x28 floating point array after normalization preprocessing (each array element is divided by 255).

In a specific example application, see Figure 7, which is a schematic diagram of a data set image classification in an embodiment of the present application. Even the relatively simple MNIST data set has 60,000 training sets and 10,000 test data. These 70,000 records need to be preprocessed. The image of each number is stored in a 28x28 array, and each one has a label (1abel) to annotate its real number. If the label range is 0 to 9, only one more byte is needed to store the label annotation. Subsequent model training will frequently read 60,000 items (training sets) depending on the training process. After each batch of training is completed, 10,000 items (test sets) may also be read multiple times to confirm whether the test is successful or not.

In a specific example application, the CPU/GPU host system 0/1/2 writes and reads the data set access accelerator 0/1/2 respectively. The writing process of multiple data sets with a more advanced data set hardware core architecture is as follows:

(1) The CPU/GPU host system 0/1/2 directly transmits the data set that is “not pre-processed at all” to the corresponding data set access accelerator 0/1/2;

(2) Data set access accelerator 0/1/2 receives the written data set and first performs partial preprocessing that does not cause the data to become larger. In this way, the written data set requires the least storage space. This step transfers part of the preprocessing operations of the CPU/GPU to the accelerator;

(3) Determine the size of the partially preprocessed data set (including labels or file names);

(4) Setting the minimum read block size for hard disk optimization based on the size of each data set;

(5) Write all data sets sequentially to continuous blocks on the hard disk.

The process of accelerating reading of multi-channel data sets is as follows:

(1) The CPU/GPU host system 0/1/2 is received from the transmission medium to set the settings and parameters for read preprocessing. This step usually only needs to be performed once and is applicable to the reading of all data items;

(2) receiving a command from the CPU/GPU host system to read the preprocessed data set through the transmission medium;

(3) reading a portion of the pre-processed data set previously written from a continuous block of the hard disk according to the command;

(4) Perform all remaining preprocessing that was not done during writing, and complete all preprocessing done by the artificial intelligence model;

(5) The pre-processed data set is transmitted back to the CPU/GPU host system 0/1/2.

Corresponding to the above data storage method embodiment, the present application also provides a data storage device. The data storage device described below and the data storage method described above can be referenced to each other.

Referring to FIG. 8 , FIG. 8 is a structural block diagram of a data storage device in an embodiment of the present application, and the device may include:

The data set receiving module 81 is used to receive the target data set to be stored;

The data size acquisition module 82 is used to acquire the data size of each item of data in the target data set; wherein the size of each item of data in the target data set is the same;

The data storage module 83 is used to store each data in the target data set into each continuous target block of the same size in the hard disk; wherein the block size of each target block is determined according to the data size.

It can be seen from the above technical solution that by setting the block size of the target block of the hard disk according to the fixed size of each data item in the target data set to be stored, it is ensured that each data item in the target data set is stored in a continuous block of the hard disk. This greatly optimizes data storage, and when reading data, it can be directly read from the continuous block of the hard disk, saving data reading time and improving data reading and writing efficiency.

In a specific embodiment of the present application, the device may further include:

The first preprocessing module is used to perform a first preprocessing operation on the target data set after receiving the target data set to be stored and before obtaining the data size of each data item in the target data set; wherein the first preprocessing operation is a preprocessing operation that does not increase the data size.

In a specific implementation of the present application, the first preprocessing module is specifically a module that performs preprocessing operations on the target data set except for normalization preprocessing.

In a specific embodiment of the present application, the data set receiving module 81 is specifically a module for receiving a target data set to be stored for artificial intelligence model training.

In a specific implementation of the present application, the data size acquisition module 82 is specifically a module for acquiring the data size of each data item in the target data set, which is composed of the data itself, the data label, and the data file name.

In a specific implementation of the present application, the device may further include a block size determination module, the block size determination module:

The optional block size acquisition submodule is used to obtain preset optional block sizes;

The block size selection submodule is used to select the block size of the target block from various optional block sizes that are larger than the data size.

In a specific implementation of the present application, the block size selection submodule includes:

The block size selection unit is used to select the block size with the smallest difference with the data size from the optional block sizes larger than the data size. Optional block size of ;

The block size determining unit is used to determine the optional block size with the smallest difference with the data size as the block size of the target block.

In a specific embodiment of the present application, the device may further include:

A judging module, used for judging whether the data size is less than or equal to the maximum value of the optional block sizes after obtaining the preset optional block sizes;

The block size selection submodule is specifically a module for selecting the block size of the target block from the optional block sizes that are larger than the data size when it is determined that the data size is less than or equal to the maximum value of the optional block sizes;

The block size determination module is specifically a module that determines the maximum value among the optional block sizes as the block size of the target block when it is determined that the data size is greater than the maximum value among the optional block sizes.

Corresponding to the above data reading method embodiment, the present application further provides a data reading device. The data reading device described below and the data reading method described above can refer to each other.

Referring to FIG. 9 , FIG. 9 is a structural block diagram of a data reading device in an embodiment of the present application, and the device may include:

A read command receiving module 91, used for receiving a data read command;

The data reading module 92 is used to read each data item of the target data set from each target block of the hard disk that is continuous and has the same size; wherein the block size of each target block is determined according to the data size of each data item, and the size of each data item in the target data set is the same;

The data return module 93 is used to return the read data to the sending end of the data read command.

In a specific embodiment of the present application, the device may further include:

The second preprocessing module is used to perform a second preprocessing operation on each item of data read from each consecutive target block of the same size in the hard disk before returning the read data to the sending end of the data reading command; wherein the second preprocessing operation is a preprocessing operation to increase the data size.

In a specific implementation of the present application, the second preprocessing module is specifically a module that performs normalization preprocessing operations on each piece of data read.

In a specific embodiment of the present application, the read command receiving module 91 is specifically a module that receives a data read command to read a target data set for artificial intelligence model training.

In a specific embodiment of the present application, the data reading module 92 is specifically a module that reads each data item of the target data set from consecutive target blocks of the same size in the hard disk according to a one-to-one relationship between the target block and each data item when the block size of the target block is greater than or equal to the data size.

In a specific embodiment of the present application, the data reading module 92 is specifically a module that reads each data item of the target data set from consecutive target blocks of the same size in the hard disk according to the many-to-one relationship between the target block and each data item when the block size of the target block is smaller than the data size; wherein each data item is pre-stored in adjacent consecutive blocks.

Corresponding to the above method embodiment, referring to FIG. 10 , FIG. 10 is a schematic diagram of an electronic device provided by the present application, and the device may include:

A memory 332, for storing computer programs;

The processor 322 is used to implement the steps of the data storage method or the data reading method of the above method embodiment when executing a computer program.

Specifically, please refer to Figure 11, which is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer applications 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Optionally, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.

The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and/or one or more operating systems 341 .

The steps in the data storage method or data reading method described above can be implemented by the structure of an electronic device.

Corresponding to the above method embodiment, the present application further provides a non-volatile readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

Receive a target data set to be stored; obtain the data size of each data item in the target data set; wherein the sizes of each data item in the target data set are the same; store each data item in the target data set into target blocks of the same size that are continuous in the hard disk; wherein the block size of each target block is determined according to the data size.

The non-volatile readable storage medium may include: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

For an introduction to the non-volatile readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment and non-volatile storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the technical solution and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Industrial Applicability

The solution provided by the embodiment of the present application can be applied to the field of data processing technology. In the embodiment of the present application, a target data set to be stored is received; the data size of each data item in the target data set is obtained, wherein the size of each data item in the target data set is the same; and each data item in the target data set is stored in each continuous target block of the same size in the hard disk, wherein the block size of each target block is determined according to the data size, thereby achieving the technical effect of saving data reading time and improving data reading and writing efficiency.

Claims (22)

A data storage method, comprising: receiving a target data set to be stored; Acquire the data size of each item of data in the target data set, wherein the size of each item of data in the target data set is the same; The data in the target data set are stored in target blocks of the same size and continuous in the hard disk, wherein the block size of each target block is determined according to the data size. The data storage method according to claim 1, characterized in that after receiving the target data set to be stored and before obtaining the data size of each data item in the target data set, it also includes: A first preprocessing operation is performed on the target data set, wherein the first preprocessing operation is a preprocessing operation that does not increase the data size. The data storage method according to claim 2, characterized in that performing a first preprocessing operation on the target data set comprises: The target data set is subjected to a preprocessing operation other than normalization preprocessing. The data storage method according to claim 2, characterized in that when the data type in the target data set is a picture, performing a first preprocessing operation on the target data set further comprises: Perform at least one of the following processing operations on the image: horizontal flipping, vertical flipping, and rotation. The data storage method according to claim 1, wherein receiving a target data set to be stored comprises: Receive a target data set to be stored for artificial intelligence model training. The data storage method according to claim 5, wherein the target data set comprises: a training set, a validation set and a test set, wherein: The training set is configured to fit the artificial intelligence model; The validation set is set to make predictions on the artificial intelligence model that has been fitted; The test set is set to evaluate the final artificial intelligence model. The data storage method according to claim 1, characterized in that obtaining the data size of each item of data in the target data set comprises: The data size of each data item in the target data set, which is composed of the data itself, the data label and the data file name, is obtained. The data storage method according to any one of claims 1 to 7, characterized in that it also includes The process of determining the block size of the target block according to the data size includes: Get the preset optional block sizes; The block size of the target block is obtained by selecting from the optional block sizes that are larger than the data size. The data storage method according to claim 8, characterized in that the block size of the target block is selected from each of the optional block sizes that is larger than the data size, comprising: Selecting a selectable block size having the smallest difference with the data size from among the selectable block sizes that are larger than the data size; The optional block size with the smallest difference from the data size is determined as the block size of the target block. The data storage method according to claim 8, characterized in that after obtaining the preset optional block sizes, it also includes: Determine whether the data size is less than or equal to the maximum value of the optional block sizes; If the data size is less than or equal to the maximum value of the optional block sizes, then executing the step of selecting the block size of the target block from the optional block sizes that are larger than the data size; If the data size is larger than a maximum value among the optional block sizes, the maximum value among the optional block sizes is determined as the block size of the target block. The data storage method according to claim 1, wherein receiving a target data set to be stored comprises: The target data set to be stored is received through an accelerator, wherein the target data set to be stored is sent to the accelerator through a central processing unit or a graphics processing unit. A data reading method, comprising: Receive data read command; Reading each item of data of the target data set from consecutive target blocks of the same size in the hard disk, wherein the block size of each target block is determined according to the data size of each item of data, and the size of each item of data in the target data set is the same; The read data are returned to the sending end of the data reading command. The data reading method according to claim 12 is characterized in that the sending end is a central processing unit or a graphics processing unit that interacts with the accelerator for data reading and writing. The data reading method according to claim 10, characterized in that after reading each item of data of the target data set from each consecutive target block of the same size in the hard disk, and before returning the read data to the sending end of the data reading command, it also includes: A second preprocessing operation is performed on each item of the read data, wherein the second preprocessing operation is a preprocessing operation for increasing the size of the data. The data reading method according to claim 14, characterized in that the second preprocessing operation is performed on each item of the read data, comprising: Perform normalization preprocessing operations on the read data. The data reading method according to claim 12, wherein receiving a data reading command comprises: Receive a data read command to read a target data set for artificial intelligence model training. The data reading method according to claim 12, wherein reading each item of data of the target data set from consecutive target blocks of the same size in the hard disk comprises: When the block size of the target block is greater than or equal to the data size, each data item of the target data set is read from each of the target blocks that are consecutive and have the same size in the hard disk according to a one-to-one relationship between the target block and each data item. The data reading method according to claim 12, wherein reading each item of data of the target data set from consecutive target blocks of the same size in the hard disk comprises: When the block size of the target block is smaller than the data size, each data item of the target data set is read from consecutive target blocks of the same size in the hard disk according to a many-to-one relationship between the target block and each data item; wherein each data item is pre-stored in adjacent consecutive blocks. A data storage device, comprising: A data set receiving module, configured to receive a target data set to be stored; a data size acquisition module, configured to acquire the data size of each item of data in the target data set, wherein the size of each item of data in the target data set is the same; The data storage module is configured to store each data in the target data set into consecutive target blocks of the same size in the hard disk, wherein the block size of each target block is determined according to the data size. A data reading device, comprising: A read command receiving module, configured to receive a data read command; a data reading module, configured to read each item of data of a target data set from consecutive target blocks of the same size in a hard disk, wherein the block size of each target block is determined according to the data size of each item of data, and the size of each item of data in the target data set is the same; The data return module is configured to return each item of read data to the sending end of the data read command. An electronic device, comprising: a memory arranged to store a computer program; A processor, configured to implement the steps of the data storage method according to any one of claims 1 to 11 or the data reading method according to any one of claims 12 to 18 when executing the computer program. A non-volatile readable storage medium, characterized in that a computer program is stored on the non-volatile readable storage medium, and when the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 11 or the data reading method according to any one of claims 12 to 18 are implemented.