CN108984132A - A kind of I O scheduling method and device based on persistence memory file system - Google Patents

Abstract

The application discloses a kind of I O scheduling method and device based on persistence memory file system, this method comprises: after receiving at least one I/O Request, first the I/O Request is inserted into discrete operations queue, judge whether the I/O Request and the IO subqueue in similar operations queue are related again, wherein, discrete operations queue and similar operations queue such as belong at the queue to be processed, if the I/O Request is related to the IO subqueue in similar operations queue, then as in subqueue insertion similar operations queue, and then it can be according to the similar operations queue of lookup, execute I O scheduling, then, further according to the discrete operations queue of lookup, execute I O scheduling.It can be seen that, the application is after receiving I/O Request, I O scheduling is no longer carried out immediately, but it is inserted into similar queue after being integrated associated I/O Request as subqueue, and then by searching for similar operations queue, then I O scheduling is executed, to realize intelligent load I O scheduling, reduce network communication number, improves performance.

Description

An IO scheduling method and device based on a persistent memory file system

technical field

The present application relates to the technical field of data storage, in particular to an IO scheduling method and device based on a persistent memory file system.

Background technique

With the development of science and technology in today's society, the performance gap between the processor and the memory is getting bigger and bigger, especially the huge gap between the processor and the external memory makes the IO bottleneck problem increasingly prominent.

The current way to alleviate the IO bottleneck is to set up a disk file cache in memory, thereby reducing the number of accesses to external memory. However, with the advent of the era of big data, the demand for memory capacity due to the explosive growth of data is increasing, and the benefits brought by traditional methods of increasing cache capacity will be reduced, and frequent data exchange between internal and external memory will cause relatively large Based on this, new types of non-volatile storage media are gradually appearing in people's field of vision. Its byte-addressable and non-volatile characteristics enable data to be stored persistently at the memory level, and its storage speed is fast , Stored data is not easy to lose, so that new non-volatile storage media can be used as memory to form a persistent memory file system, replacing the traditional way of coexistence of internal and external memory, and using it as the only storage medium to achieve persistent data storage.

Therefore, how to use a more advanced IO scheduling method to replace the traditional scheduling method to realize intelligent IO scheduling based on the persistent memory file system, so as to effectively alleviate the IO bottleneck, has become an urgent problem to be solved.

Contents of the invention

In order to solve the above problems, the application provides an IO scheduling method and device based on a persistent memory file system. The specific technical solutions are as follows:

In a first aspect, the present application provides an IO scheduling method based on a persistent memory file system, the method comprising:

Receive at least one IO request;

After the IO request is inserted into the discrete operation queue, it is judged whether the IO request is related to the IO sub-queue in the similar operation queue, and the discrete operation queue and the similar operation queue belong to the waiting processing queue;

If so, inserting the IO request as a subqueue into the similar operation queue;

Execute IO scheduling according to the searched similar operation queue;

Perform IO scheduling according to the discrete queue operation column found.

In an optional implementation, the method also includes:

Presetting the waiting processing queue, the waiting processing queue includes a similar operation queue and a discrete operation queue;

Wherein, the similar operation queue is composed of IO subqueues, and the discrete operation queue is composed of IO requests.

In an optional implementation manner, the performing IO scheduling according to the searched similar operation queue includes:

Find similar operation queues in the waiting processing queue;

Judging whether the similar operation queue contains at least one sub-column packet, the sub-column packet is composed of at least two related sub-queues;

If yes, perform IO scheduling by package according to the sub-pair packages contained in the similar operation queue.

In an optional implementation manner, if the IO request is not related to the IO subqueue in the similar operation queue, the method further includes:

judging whether the IO request is related to other IO requests in the discrete operation queue;

If so, according to the IO request and other related IO requests in the discrete operation queue, create a new IO sub-queue packet and insert it into the similar operation queue.

In an optional implementation manner, the memory is a new type of non-volatile storage medium.

In a second aspect, the present application provides an IO scheduling device based on a persistent memory file system, the device comprising:

a receiving unit, configured to receive at least one IO request;

A first judging unit, configured to, after inserting the IO request into a discrete operation queue, judge whether the IO request is related to an IO subqueue in a similar operation queue, and the discrete operation queue and the similar operation queue belong to a waiting processing queue ;

an insertion unit, configured to insert the IO request as a subqueue into the similar operation queue if the IO request is related to the IO subqueue in the similar operation queue;

A first scheduling unit, configured to perform IO scheduling according to the searched similar operation queue;

The second scheduling unit is configured to perform IO scheduling according to the searched discrete queue operation sequence.

In an optional implementation, the device further includes:

The setting unit is used to pre-set the waiting processing queue, and the waiting processing queue includes similar operation queues and discrete operation queues;

Wherein, the similar operation queue is composed of IO subqueues, and the discrete operation queue is composed of IO requests.

In an optional implementation manner, the first scheduling unit includes:

A search subunit is used to search for similar operation queues in the queue waiting to be processed;

The first judging subunit is used to judge whether the similar operation queue contains at least one sub-column packet, and the sub-column packet is composed of at least two related sub-queues;

The first scheduling subunit is configured to perform IO scheduling by package according to the sub-pair packets contained in the similar operation queue if the similar operation queue contains at least one sub-pair packet.

In an optional implementation manner, if the IO request is not related to the IO subqueue in the similar operation queue, the device further includes:

A second judging unit, configured to judge whether the IO request is related to other IO requests in the discrete operation queue;

A new unit, configured to create an IO sub-queue according to the IO request and other related IO requests in the discrete operation queue if the IO request is related to other IO requests in the discrete operation queue The packet is inserted into the similar operation queue.

In an optional implementation manner, the memory is a new type of non-volatile storage medium.

In the IO scheduling method based on the persistent memory file system provided by this application, after receiving at least one IO request, first insert the IO request into the discrete operation queue, and then judge whether the IO request is consistent with the IO in the similar operation queue. Whether the subqueues are related. Among them, the discrete operation queue and the similar operation queue belong to the waiting processing queue. If the IO request is related to the IO subqueue in the similar operation queue, it will be inserted into the similar operation queue as a subqueue, and then can be processed according to The searched similar operation queue performs IO scheduling, and then performs IO scheduling according to the searched discrete operation queue. It can be seen that after receiving the IO request, this application does not immediately perform IO scheduling, but integrates the associated IO requests and inserts them into similar queues as subqueues, and then performs IO scheduling by searching for similar operation queues. In this way, the intelligent load IO scheduling is realized, the number of network communication and the number of device erasing and writing are reduced, the delay is reduced, and the performance is also improved.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a flow chart of an IO scheduling method based on a persistent memory file system provided in an embodiment of the present application;

FIG. 2 is a flow chart of performing IO scheduling according to the searched similar operation queue provided by the embodiment of the present application;

FIG. 3 is a schematic structural diagram of an IO scheduling device based on a persistent memory file system provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

In order to facilitate the understanding of the technical solution provided by the present application, a brief description of the research background of the technical solution of the present application is given below.

As we all know, as described in the background technology, with the development of science and technology, the performance gap between the processor and the memory is getting bigger and bigger, especially the huge gap between the processor and the external memory makes the IO bottleneck problem increasingly prominent. If it is still The traditional method of setting disk file cache in memory to alleviate IO bottlenecks will not be able to meet the demand for memory capacity due to the explosive growth of data, and, with the advancement of industrial technology, new types of non-volatile storage media are gradually appearing in In people's view, its byte-addressable and non-volatile characteristics allow data to be stored persistently at the memory level. The high density, read and write speed of the new non-volatile memory device is getting closer to the traditional dynamic random access memory (Dynamic Random Access Memory, referred to as DRAM), and the characteristics that the data will not be lost when the power is turned off make people use it as a substitute for memory Even the first choice for next-generation storage devices. On this basis, how to replace the traditional scheduling method with a more advanced IO scheduling method to realize intelligent IO scheduling based on the persistent memory file system, so as to coordinate the computing overhead of the central processing unit (Central Processing Unit, referred to as CPU) and the new non- Balancing the overhead of volatile storage media as memory, and effectively alleviating IO bottlenecks, has become an urgent problem to be solved.

Based on this, the present application proposes an IO scheduling method and device based on a persistent memory file system for realizing intelligent IO scheduling based on a persistent memory file system.

The IO scheduling method based on the persistent memory file system provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings. Referring to FIG. 1, it shows a flowchart of a persistent memory file system-based IO scheduling method provided by an embodiment of the present application. This embodiment may include the following steps:

S101: Receive at least one IO request.

In this embodiment, as more and more people use new non-volatile storage media as memory, a persistent memory file system is formed. In order to coordinate the computing overhead of the CPU and the Overhead balance can effectively alleviate the IO bottleneck. After receiving at least one IO request, intelligent scheduling of the received IO requests can be realized through subsequent steps.

S102: After inserting the IO request into the discrete operation queue, determine whether the IO request is related to the IO subqueue in the similar operation queue, wherein the discrete operation queue and the similar operation queue belong to the waiting processing queue.

In this embodiment, in order to realize intelligent IO scheduling, an optional implementation method is to pre-set the waiting processing queue, which includes similar operation queues and discrete operation queues, wherein the similar operation queues are composed of IO sub-queues Composition, the discrete operation queue is composed of IO requests.

In this implementation, in order to realize intelligent IO scheduling based on semantic analysis to achieve aggregated processing of operations, first, before processing IO request instructions, it is necessary to pre-set multiple waiting processing queues, wherein the waiting processing queues may include IO A similar operation queue composed of subqueues and a discrete operation queue composed of IO requests. In the initial situation, the similar operation queue is empty. After receiving at least one IO request through step S201, it can be inserted into the discrete operation queue, and then Based on semantic analysis, find out the associated IO requests and perform aggregation processing so that similar IO operation requests can be combined during multi-threaded operations, such as operations on the same file, multiple operations on the same directory etc. Aggregate into a large I/O sub-queue package and insert into a similar operation queue. For example, suppose the three received IO requests are "read the first 50 bytes of A", "read the last 20 bytes of A section" and "modify the 80th byte in A", then from the semantic analysis, it can be concluded that these three IO operation requests are operations on the same file, and can be aggregated, that is, the three can be aggregated into A large I/O subqueue packet, inserted into a similar operation queue.

Then, after the IO request is received in step S201, it can be judged whether the IO request is related to the IO sub-queue in the similar operation queue. Specifically, it can be calculated whether the IO request is related to the IO sub-queue in the similar operation queue. Each sub-queue in each IO sub-queue package in the similar operation queue has an association relationship. If so, proceed to step S103.

S103: If yes, insert the IO request into a similar operation queue as a subqueue.

In this embodiment, if it is determined through step S102 that the received IO request is related to the IO subqueue in the similar operation queue, the IO request can be inserted as a subqueue into the corresponding IO subqueue in the similar operation queue. in the queue package.

For example: Based on the above example, assume that in the similar operation queue, there are three IO requests "read the first 50 bytes of A", "read the last 20 bytes of A" and "modify the The 80th byte of A" constitutes an I/O sub-queue packet. At this time, if an IO request is received through step S102 as "reading the 60th to 70th byte of A", then through semantic analysis, it can be obtained The IO request is related to the IO subqueue in the similar operation queue "read the first 50 bytes of A", "remove the last 20 bytes of A" and "modify the 80th byte in A" Then, the IO request can be inserted as a subqueue into the above IO subqueue package in the similar operation queue, so that the IO subqueue package includes four I/O subqueues (that is, the above four IO requests).

It should be noted that, in some possible implementations of the present application, if in step S102, it is judged that the received IO request has nothing to do with the IO subqueue in the similar operation queue, then further, it can be judged that the IO request is not related to the discrete operation Whether other IO requests in the queue are related, and if they are related, a new IO sub-queue packet can be created and inserted into the similar operation queue according to the IO request and other related IO requests in the discrete operation queue.

In this implementation, if it is judged that the received IO request has nothing to do with the IO sub-queue in the similar operation queue, it cannot be directly inserted into the similar operation queue as a sub-queue, but it needs to be further judged that the IO request is related to the discrete operation Whether other IO requests in the queue are related, for example, can also be judged by semantic analysis. For example, assuming that the IO request is semantically related to the IO request B in the discrete operation queue, the IO request can be compared with the IO Request B to integrate, build an IO sub-queue package, the two are respectively used as sub-queues in the package, and insert the IO sub-queue package into a similar operation queue, so as to use the subsequent steps to realize the scheduling of IO requests.

It should be noted that, if it is determined that the IO request is not related to other IO requests in the discrete operation queue, it will be kept in the discrete operation queue for subsequent scheduling through step S105.

S104: Execute IO scheduling according to the found similar operation queues.

In practical applications, after the received IO request is inserted into the similar operation queue as a subqueue in step S103, the corresponding IO scheduling can be executed by searching all the similar operation queues when performing multi-threaded operations.

In some possible implementations of the present application, as shown in FIG. 2, this step S104 may specifically include steps S201-S203:

Step S201: Search for similar operation queues in the queues waiting to be processed.

In this implementation, after all the IO sub-queues in the similar operation queue are formed through S103, the subsequent step S202 can be performed by searching all the similar operation queues in the waiting processing queue.

Step S202: Determine whether the similar operation queue contains at least one sub-queue packet, wherein the sub-queue packet is composed of at least two related sub-queues.

After finding all the similar operation queues in the waiting processing queue through step S201, it can be further judged whether there is at least one sub-queue packet in the similar operation queue, and if so, step S203 can be continued.

Step S203: If yes, perform IO scheduling by package according to the sub-queue packages contained in the similar operation queue.

In this implementation, if it is determined in step S202 that the similar operation queue contains at least one sub-queue packet, IO scheduling can be performed on these sub-queue packets by packet, for example, IO requests can be sent by packet. For example, after receiving multiple IO requests, the client can form multiple sub-queue packets through the above steps and insert them into similar operation queues, and then send these sub-queue packets to the server at a time, sending one sub-queue packet at a time (where Including multiple IP requests), until all sub-queue packets in similar operation queues are sent to the server, thereby realizing intelligent load IO scheduling, reducing the number of network communications and device erasing times, reducing delays, and improving performance. Similarly, after receiving multiple IO requests sent by the client, the server can also reintegrate the IO requests through the above steps. On the basis of using a new type of non-volatile storage medium as the memory, faster The specific process can be referred to the above-mentioned steps, and the present application will not repeat them here.

S105: Execute IO scheduling according to the found discrete operation queue.

In this embodiment, if there are still IO requests that are not related to other IO requests in the discrete operation queue after the intelligent IO scheduling of similar operation queues is implemented through the above steps, they can be retained in the discrete operation queue, and then Scheduling of these IO requests can be performed by looking up the discrete operation queue.

It should be noted that, in some possible implementation manners of the present application, the system memory is a new type of non-volatile storage medium.

Specifically, the novel non-volatile storage medium may be at least one of a storage class memory (Storage Class Memory, SCM for short) and a non-volatile memory (NonVolatile Memory, NVM for short).

In this way, in the IO scheduling method based on the persistent memory file system provided by the present application, after receiving at least one IO request, the IO request is first inserted into the discrete operation queue, and then it is judged that the IO request is different from the similar operation queue. Whether the IO sub-queues are related. Among them, the discrete operation queue and the similar operation queue belong to the waiting processing queue. If the IO request is related to the IO sub-queue in the similar operation queue, it will be inserted into the similar operation queue as a sub-queue, and then IO scheduling can be performed according to the searched similar operation queues, and then IO scheduling can be performed according to the searched discrete operation queues. It can be seen that after receiving the IO request, this application does not immediately perform IO scheduling, but integrates the associated IO requests and inserts them into similar queues as subqueues, and then performs IO scheduling by searching for similar operation queues. In this way, the intelligent load IO scheduling is realized, the number of network communication and the number of device erasing and writing are reduced, the delay is reduced, and the performance is also improved.

Based on the above IO scheduling method based on the persistent memory file system, the present application also provides an IO scheduling device based on the persistent memory file system, the device comprising:

a receiving unit 301, configured to receive at least one IO request;

The first judging unit 302 is configured to, after inserting the IO request into the discrete operation queue, judge whether the IO request is related to the IO sub-queue in the similar operation queue, and the discrete operation queue and the similar operation queue belong to waiting processing queue;

Inserting unit 303, configured to insert the IO request into the similar operation queue as a subqueue if the IO request is related to the IO subqueue in the similar operation queue;

The first scheduling unit 304 is configured to perform IO scheduling according to the searched similar operation queue;

The second scheduling unit 305 is configured to perform IO scheduling according to the searched discrete queue operation sequence.

Optionally, the device also includes:

The setting unit is used to pre-set the waiting processing queue, and the waiting processing queue includes similar operation queues and discrete operation queues;

Wherein, the similar operation queue is composed of IO subqueues, and the discrete operation queue is composed of IO requests.

Optionally, the first scheduling unit 304 includes:

A search subunit is used to search for similar operation queues in the queue waiting to be processed;

The first judging subunit is used to judge whether the similar operation queue contains at least one sub-column packet, and the sub-column packet is composed of at least two related sub-queues;

The first scheduling subunit is configured to perform IO scheduling by package according to the sub-pair packets contained in the similar operation queue if the similar operation queue contains at least one sub-pair packet.

Optionally, if the IO request is not related to the IO subqueue in the similar operation queue, the device further includes:

A second judging unit, configured to judge whether the IO request is related to other IO requests in the discrete operation queue;

A new unit, configured to create an IO sub-queue according to the IO request and other related IO requests in the discrete operation queue if the IO request is related to other IO requests in the discrete operation queue The packet is inserted into the similar operation queue.

Optionally, the memory is a new type of non-volatile storage medium.

In this way, in the IO scheduling device based on the persistent memory file system provided by the present application, after receiving at least one IO request, the IO request is first inserted into the discrete operation queue, and then it is judged that the IO request is consistent with the similar operation queue. Whether the IO subqueues of the similar operation queues are related. Among them, the discrete operation queue and the similar operation queue belong to the waiting processing queue. If the IO request is related to the IO subqueue in the similar operation queue, it will be inserted into the similar operation queue as a subqueue, and then IO scheduling can be performed according to the searched similar operation queues, and then IO scheduling can be performed according to the searched discrete operation queues. It can be seen that after receiving the IO request, this application does not immediately perform IO scheduling, but integrates the associated IO requests and inserts them into similar queues as subqueues, and then performs IO scheduling by searching for similar operation queues. In this way, the intelligent load IO scheduling is realized, the number of network communication and the number of device erasing and writing are reduced, the delay is reduced, and the performance is also improved.

It should be noted that each embodiment in this specification is described in a progressive manner, each embodiment focuses on the differences from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for relevant details, please refer to the description of the method part.

It should also be noted that in this article, relational terms such as first and second etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations Any such actual relationship or order exists between. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. an IO scheduling method based on persistent memory file system, it is characterized in that, described method comprises: Receive at least one IO request; After the IO request is inserted into the discrete operation queue, it is judged whether the IO request is related to the IO sub-queue in the similar operation queue, and the discrete operation queue and the similar operation queue belong to the waiting processing queue; If so, inserting the IO request as a subqueue into the similar operation queue; Execute IO scheduling according to the searched similar operation queue; Perform IO scheduling according to the discrete queue operation column found. 2. The IO scheduling method based on the persistent memory file system according to claim 1, wherein the method further comprises: Presetting the waiting processing queue, the waiting processing queue includes a similar operation queue and a discrete operation queue; Wherein, the similar operation queue is composed of IO subqueues, and the discrete operation queue is composed of IO requests. 3. The IO scheduling method based on the persistent memory file system according to claim 1 or 2, wherein the IO scheduling is performed according to the similar operation queue searched, comprising: Find similar operation queues in the waiting processing queue; Judging whether the similar operation queue contains at least one sub-column packet, the sub-column packet is composed of at least two related sub-queues; If yes, perform IO scheduling by package according to the sub-pair packages contained in the similar operation queue. 4. The IO scheduling method based on the persistent memory file system according to claim 1, wherein if the IO request has nothing to do with the IO subqueue in the similar operation queue, the method further comprises: judging whether the IO request is related to other IO requests in the discrete operation queue; If so, according to the IO request and other related IO requests in the discrete operation queue, create a new IO sub-queue packet and insert it into the similar operation queue. 5. The IO scheduling method based on a persistent memory file system according to claim 1, wherein the memory is a new type of non-volatile storage medium. 6. An IO scheduling device based on a persistent memory file system, characterized in that the device comprises: a receiving unit, configured to receive at least one IO request; A first judging unit, configured to, after inserting the IO request into a discrete operation queue, judge whether the IO request is related to an IO subqueue in a similar operation queue, and the discrete operation queue and the similar operation queue belong to a waiting processing queue ; an insertion unit, configured to insert the IO request as a subqueue into the similar operation queue if the IO request is related to the IO subqueue in the similar operation queue; A first scheduling unit, configured to perform IO scheduling according to the searched similar operation queue; The second scheduling unit is configured to perform IO scheduling according to the searched discrete queue operation sequence. 7. The device according to claim 6, further comprising: The setting unit is used to pre-set the waiting processing queue, and the waiting processing queue includes similar operation queues and discrete operation queues; Wherein, the similar operation queue is composed of IO subqueues, and the discrete operation queue is composed of IO requests. 8. The device according to claim 6 or 7, wherein the first scheduling unit comprises: A search subunit, configured to search for similar operation queues in the queue waiting to be processed; A first judging subunit, configured to judge whether at least one sub-column packet is included in the similar operation queue, and the sub-column packet is composed of at least two related sub-queues; The first scheduling subunit is configured to perform IO scheduling by package according to the sub-pair packets contained in the similar operation queue if the similar operation queue contains at least one sub-pair packet. 9. The device according to claim 6, wherein if the IO request has nothing to do with the IO sub-queue in the similar operation queue, the device further comprises: a second judging unit, configured to judge whether the IO request is related to other IO requests in the discrete operation queue; A new unit, configured to create a new IO sub-queue according to the IO request and other related IO requests in the discrete operation queue if the IO request is related to other IO requests in the discrete operation queue The packet is inserted into the similar operation queue. 10. The device according to claim 6, wherein the memory is a new type of non-volatile storage medium.