CN110109873B - File management method for message queue - Google Patents

Abstract

The invention discloses a file management method for a message queue, which is characterized in that: a new type of non-volatile storage device is adopted, and a message file and a metadata file are set in the storage module, and two key-value storage structures are established. To record the number of messages contained in each message file and the size of the message file, the message file and the metadata file are stored in a one-to-one correspondence and stored separately, wherein the message file is used to store the message entity data, and the metadata file is used to store the information of each message Description information, including message size, offset in the message file, and message generation time. The effect is: using NVM as a persistent storage medium for messages, the message data can be accessed through the virtual address of the process, avoiding complex I/O software stacks, and the message data can be randomly accessed by accessing the message number, and a message production time-based The multi-precision message hierarchical index structure can quickly trace back the message.

Description

A file management method for message queue

technical field

The invention relates to computer data storage technology, more specifically, a file management method for message queues.

Background technique

Message queue, also known as message middleware, is a container for storing messages during message delivery and an important component in a distributed system. It mainly solves problems such as application decoupling and asynchronous consumption. Under normal circumstances, the message queue adopts the publish and subscribe mode, provides a unified interface to help the client and the server decouple, provides a message persistence mechanism, ensures that the message can be received correctly, and will not be lost in an emergency, so that the server can cope with High-throughput message processing scenarios. Message queues are widely used in mobile Internet, e-commerce and other fields.

Most of the existing message queues store messages in memory or in file systems based on block devices such as disks. Stored in memory, real-time operations on messages can be realized, speed block, but it cannot be stored persistently, and the data will be lost after power failure. Saving messages in file systems based on block devices such as disks and SSDs can persist messages. However, accessing a file system based on a block device needs to go through a complex I/O software stack, which greatly reduces the performance of the message queue.

The new type of non-volatile memory (non-volatile memory, NVM) has a reading and writing speed close to that of DRAM (Dynamic Random Access Memory), can be directly mounted on the memory bus, has high density, low energy consumption, will not be lost when power is off, and can be written section addressing. The new type of non-volatile memory mainly includes phase change memory (PCM), resistive memory (RRAM), memristor (Memristor) and so on. NVM is simulated as a disk device and initialized with the existing disk file system, which cannot take advantage of NVM. Because the operating system is connected to the block device through the I/O bus, accessing the file system based on the block device needs to go through a complex I/O software stack. The I/O software stack of the Linux file system shown in Figure 1 is first a virtual file system (Virtual File System, referred to as VFS). Next is the internal operation of the file system, such as data block lookup, etc. Then an I/O request to access the block device will be issued, and then go through device-oriented software layers such as the general block layer, the I/O scheduling layer, and the block device driver layer. Using the traditional I/O software stack to access data will incur a huge overhead. Take reading file data as an example. First, the file data is copied from the block device to the kernel I/O buffer through DMA. Generally, it needs to be copied to the page high-speed Cache, and finally copy to the user mode buffer. In this process, process context switching is also required, which takes up CPU resources.

Contents of the invention

Aiming at the problems existing in the prior art, the present invention proposes a file management method for message queues, based on a new type of non-volatile storage device, accessing file data by accessing the virtual address of the process, avoiding complex I/O software Context switching for stacks and processes.

In order to achieve the above object, the concrete technical scheme adopted in the present invention is as follows:

A file management method for message queues, the key of which is: a new type of non-volatile storage device is adopted, a message file and a metadata file are set in the storage module, and two key-value storage structures are established for recording each The number of messages contained in the message file and the size of the message file, the message file and the metadata file are stored in a one-to-one correspondence and stored separately, wherein the message file is used to store the message entity data, and the metadata file is used to store the description information of each message, including Message size, offset within the message file, message generation time.

Optionally, the system uses shared memory to store messages, and writes and reads messages through virtual addresses of processes.

Optionally, during the message reading process, the metadata file is read first, and the size of the message and the offset of the message in the message file are obtained from the metadata file, and then the corresponding message file.

Optionally, the message writing process includes three steps: writing the message file, writing the metadata file, and calculating and updating the number of messages and the key-value storage structure of the message file size.

Optionally, only one producer process runs on the same topic partition at the same time for writing operations. If there are two producer processes that are generating messages of a certain topic, different partitions are assigned to them.

Optionally, updating the size of the message file in the key-value storage structure and the number of messages in the message file are calculated by the background thread detecting the metadata items in the metadata file by sliding the detection point.

Optionally, in the message file, messages are stored sequentially in order of generation time.

Optionally, a message fast index structure based on generation time is used in message query. The system establishes a three-layer tree index structure of hours, minutes, and seconds. Each index node of each layer of index is a continuous space. An index node consists of several fixed-size index items, and each index item can be accessed randomly, where:

The first layer is based on the full-time index, and has an index node, each index node is composed of 24 index items, and each index item is composed of the file index number, the message number in the message file, and the starting address of the next layer index node;

The second layer is based on the minute index, with 24 index nodes, each index node is composed of 60 index items, and each index item is composed of the file index number, the message number in the message file, and the starting address of the next layer index node;

The third layer is based on the second-level index, with 24*60 index nodes, each index node is composed of 60 index items, and each index item is composed of the file index number and the message number in the message file.

Notable effect of the present invention is:

Using NVM as the persistent storage medium for messages, the message data can be accessed through the virtual address of the process, avoiding complex I/O software stacks, and the message data can be randomly accessed by accessing the message number, establishing multi-accuracy based on message production time The hierarchical index structure of messages can quickly trace back messages.

Description of drawings

The present invention will be further described below in conjunction with accompanying drawing and embodiment, in the accompanying drawing:

Fig. 1 is the I/O software stack schematic diagram in the existing Linux system;

Fig. 2 is the logical structure of the message storage module in the specific embodiment of the present invention;

Fig. 3 is a functional block diagram of shared memory in a specific embodiment of the present invention;

Fig. 4 is a relational diagram of a message file and a metadata file in a specific embodiment of the present invention;

Fig. 5 is a functional block diagram of a message writing operation in a specific embodiment of the present invention;

Fig. 6 is a schematic diagram of sliding detection in a specific embodiment of the present invention;

Fig. 7 is a structural diagram of a message index based on production time in a specific embodiment of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, and are not intended to limit the invention.

As shown in Fig. 2, a kind of file management method for message queue proposed by the present invention is used for the persistent storage of the message of the message queue based on topic Topic, and Topic is an abstract concept, that is, the type of a certain type of message, It can be horizontally expanded into partitions (hereinafter referred to as P), and each Partition contains multiple files for storing messages. The present invention adopts a new type of non-volatile storage device to persistently save messages in the form of files, separately saves message data and message description information (referred to as message metadata hereinafter), and searches for message data by checking message metadata information. There are two types of files in the storage module: message file and metadata file. The message file stores the message entity data, and the metadata file stores the description information of each message, including message size, message offset in the message file, and message production time. wait. The size of each message file and metadata file is fixed and configurable. Usually, the metadata file is smaller than the message file. During specific implementation, the naming method of the message file is Topic_P-date-index.log, where Topic_P is the name of the TopicPartition, date is the current date, index is the file id, and increments from 0, representing the first file of the Partition on the day, the message The files are stored in the log/topic directory. Corresponding to the message file, there is a metadata file with the same name. The suffix of the metadata file is .meta, and it is stored in the meta/topic directory. Two key-value storage structures are established in the storage module to record the number of messages contained in each message file and the size of the message file. The message file name Topic_P-date-index is used as the key, and the counts of messages in the message file are used as the value; Topic_P -date-index is the key, and the message file size is the value.

In this system, the message file needs to be written by the writing process and read by the reading process. These two processes usually work together. The Linux system provides a shared memory (shm) mechanism. The so-called shared memory allows the same piece of memory to be mapped in the address spaces of different processes. As shown in Figure 3, process 1 can instantly see the data in the shared memory of process 2. updates, and vice versa. This system uses NVM physical space as a shared NVM memory pool, and each message file and metadata file is essentially a shared NVM memory.

The specific implementation of shared memory mainly includes the following four system call functions:

①Create shared memory int shmget(key_t key, size_t size, int shmflg);

The first parameter is the name of the shared memory segment;

The second parameter is the memory capacity to be created;

The third is the permission flag;

When the shmget function succeeds, it returns a shared memory identifier shm_id associated with the key.

② Map the shared memory in the address space of the process void*shmat(int shm_id, const void*shm_addr, int shmflg);

The first parameter is the shared memory identifier returned by the shmget function;

The second parameter is to specify the address of the shared memory mapped to the current process, which is usually empty, which means let the system choose the address of the shared memory;

The third parameter, shm_flg is a set of flag bits, usually 0;

Returns the first address of the mapped space after the call is successful.

③Separate int shmdt(const void*shmaddr) from the current process of shared memory;

This function is used to detach shared memory from the current process. Note that detaching shared memory does not delete it, it just makes the shared memory no longer available to the current process.

④Control shared memory int shmctl(int shm_id, int command, struct shmid_ds*buf); for example, change the associated value of shared memory, delete shared memory segment, etc.

The size of the message file and the metadata file is fixed. When a new file needs to be created after the file is full, hash values are taken for the new message file name and metadata name respectively, and the Linux shared memory function shmget is called as a parameter to obtain the corresponding shared memory ID The character shm_id, when the process needs to access the message file and metadata file, the shmat function can be called to map in the virtual address space of the process, and the data access can be completed by accessing the returned address.

In the message reading process, the metadata file is read first, and the size of the message and the offset of the message in the message file are obtained from the metadata file, and then the corresponding message file is read.

Specifically, it can be quickly queried and read according to the message id. The message id is composed of the file name of the file where the message is located and the sequence number of the message in the file. The message id in each message file increases from 0. There is a one-to-one correspondence between the message file and the metadata file. As shown in Figure 4, the metadata file stores the metadata information of each message. The offset in , the length of the message. The size of the metadata item is fixed, and after the metadata file is mapped in the virtual address space of the process, any metadata item can be randomly accessed in the user mode. When reading a message, first read the corresponding metadata item, obtain the offset and message size of the message in the message file, and then you can access the real message data.

In the message writing process, there are three steps including writing the message file, writing the metadata file, and calculating and updating the number of messages and the key-value storage structure of the message file size.

During specific implementation, only one producer process is allowed to write to the same Partition at the same time. If there are two producer processes that are both producing messages of a certain topic, different Partitions are assigned to them. Now take the message 3 of the Topic1_P0-#-# file produced by the producer process as an example to describe the message writing operation, as shown in FIG. 5 .

1) The generation process calls the API to generate a message, and writes the message file in append mode after encapsulation and processing.

2) After the message file is successfully written, the metadata file corresponding to the message file will be updated, and the metadata item of message 3 will be written in the corresponding metadata file in the append mode.

3) The background thread calculates the total number of messages in the message file and the size of the message file by detecting metadata items, and updates the relevant key-value storage structure. Failure in any step in the middle means that the writing of the message failed this time, and it returns to the first step to re-execute.

Updating the size of the message file in the key-value storage structure and the number of messages in the message file are calculated by the background thread through the sliding detection point to detect metadata items. The detection process is shown in Figure 6.

After the metadata file is created, all fields of the metadata item will be initialized to 0. After the metadata file is mapped in the virtual address space of the process, any metadata item can be accessed because the size of each metadata item is fixed. The field size representing the length of the message in the message metadata item is placed at the end, and is represented by an integer. When the CPU performs a write operation, it is an atomic operation, so as long as the size field is not 0, it means that the write operation of the message has been completed. This metadata item identifies a new message, then changes the number of messages in the message file and the size of the message file while moving the detection point to the next metadata item.

The persistent message queue system plays the role of message storage at the same time. It is very necessary to provide a fast message query function for the message queue. Users may need to go back to the messages of a certain period of time in the past, and it is obviously unreasonable to search one by one. . It is not difficult to locate the message file of a given topic in the message storage module, but the difficulty lies in how to quickly locate the messages produced by a given topic in a certain period of time.

In this system, the message file name contains the current date, but if it is a finer-grained time division, such as hours, minutes, and seconds, it is necessary to establish a data structure to assist in quickly locating messages. Create a three-tier index in seconds. The metadata information of each message records the production time of the message, so the index structure can be established by detecting the metadata items during operation, and the establishment of the index structure is implemented according to Partition, as shown in Figure 7.

In the message file, messages are stored in the order of production time, so you only need to record the id of the production message at the entire time point, then the messages after the id are all produced after that time point. For the requirements of different time granularities, this paper establishes a three-layer tree index according to hours, minutes, and seconds. Each index node of each index layer is a continuous space, and each index node is composed of several fixed-size index items, and each index item can be accessed randomly. The first layer index is based on the whole time index, and there is only one index node. Since a day is divided into 24 hours, the index node is composed of 24 index items. Each index item consists of the file index, the message id in the message file, the next The starting address of the layer index node is composed of addr. Because each Topic Partition may generate more than one file per day, this paper uses an incremental index to indicate different files, and id identifies the message id in the current file. The index item 0 indicates that from the first message of a certain message file, all messages are produced after 0 o'clock, and so on. Because an hour is divided into 60 minutes, the addr field of each index entry points to the index node of the next layer representing the 60 minutes. Similarly, in the minute-level index items, the index item 0 identifies the number of messages from a certain message file, and all messages are produced after 0 minutes. One minute includes 60 seconds, so the addr field of each index item in the second-level index points to the third-level second-level index node. The index items of the second-level index node are only index and id, and all messages after the id are identified are produced after this second.

Through the above-mentioned three-tier index structure, the messages produced per second can be quickly located. For example, if you want to query the news after 8:07:56, you can first search for the index item with the subscript 8 in the first layer, find the index node in the second layer pointed to by it, and then search for the index item with the subscript 7 for the node, and find It points to the starting address of the third-level index node, and then queries the index item whose subscript is 56 to find the message id that meets the requirements, then reads the corresponding metadata item, and finally obtains the message data.

To sum up, the present invention uses NVM as a persistent storage medium for messages, can access message data through the virtual address of the process, avoids complex I/O software stacks, and can randomly access message data by accessing message ids. A multi-accuracy message hierarchical index structure based on message production time is established, which can quickly trace back messages.

As a final note, as used herein, the term "comprises," "comprises," or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a set of elements includes not only those elements , but also includes other elements not expressly listed, or also includes elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

The serial numbers of the above-mentioned embodiments of the present invention are only for description, and do not represent the advantages and disadvantages of the embodiments. Through the description of the above implementation modes, those skilled in the art can clearly understand that the methods of the above-mentioned embodiments can be implemented with the help of software plus the necessary general-purpose hardware platform. Of course, it can also be implemented through hardware, but in many cases the former is a better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present invention.

Embodiments of the present invention have been described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementations, and the above-mentioned specific implementations are only illustrative, rather than restrictive, and those of ordinary skill in the art will Under the enlightenment of the present invention, many forms can also be made without departing from the gist of the present invention and the protection scope of the claims, and these all belong to the protection of the present invention.

Claims (6)

1. A file management method for message queue, characterized in that: adopt phase-change memory, resistive change memory or memristor as storage module, and be provided with message file and metadata file in storage module and set up two A key-value storage structure is used to record the number of messages contained in each message file and the size of the message file. The message file and the metadata file are stored in a one-to-one correspondence, wherein the message file is used to store message entity data, and the metadata file is used to store the message entity data. It is used to store the description information of each message, including message size, offset in the message file, and message generation time; In the message file, the messages are stored in order of generation time; When querying messages, a fast message index structure based on generation time is adopted. The system establishes a three-layer tree index structure of hours, minutes, and seconds. Each index node of each layer of index is a continuous space, and each index node is composed of It consists of several fixed-size index items, and each index item can be accessed randomly, among which: The first layer is based on the full-time index, and has an index node, each index node is composed of 24 index items, and each index item is composed of the file index number, the message number in the message file, and the starting address of the next layer index node; The second layer is based on the minute index, with 24 index nodes, each index node is composed of 60 index items, and each index item is composed of the file index number, the message number in the message file, and the starting address of the next layer index node; The third layer is based on the second-level index, with 60*24 index nodes, each index node is composed of 60 index items, and each index item is composed of the file index number and the message number in the message file. 2. The file management method for message queue according to claim 1, characterized in that: the message is stored in a shared memory mode, and the writing and reading of the message is realized by using the virtual address of the process. 3. The file management method for message queue according to claim 1 or 2, characterized in that: in the message reading process, first read the metadata file, and obtain the metadata file from the metadata file The size of the message and the offset of the message in the message file, and then read the corresponding message file. 4. The file management method for message queue according to claim 1 or 2, characterized in that: in the message writing process, including writing message files, writing metadata files and calculating and updating message quantity and message The file size key-value store structure has three steps. 5. The file management method for message queues according to claim 4, characterized in that: the same topic partition runs only one producer process at the same time for writing operations, if there are two producer processes that are producing a certain When the topic's messages are assigned different partitions. 6. the file management method that is used for message queue according to claim 4, it is characterized in that: update the message file size in the key-value storage structure, the number of messages in the message file is that the background thread detects in the metadata file by the sliding detection point Metadata items are computed.

Images