CN117891815B - Database parallel scanning method and device, electronic equipment and storage medium - Google Patents

Abstract

The present application discloses a database parallel scanning method, device, electronic device and storage medium, and relates to the field of database technology. In the method, the data index tree of the database is first obtained and multiple scanning threads are created. The data index tree includes multiple index pages, and the index pages are stored in order according to the page numbers. The first scanning thread is used as the initial thread. The initial thread is used to obtain a synchronization lock, select an initial index page from the index page and determine a preset number, and perform data scanning on the initial index page. When performing data scanning, the initial thread releases the synchronization lock. Then, based on the page number of the initial index page, a preset number of index pages are selected in order as target index pages, and the target page numbers of the target index pages are written into the preset shared parameters in sequence. After each write, the scanning thread is awakened by the synchronization lock, and the scanning thread is used to perform data scanning on the target index page, which can effectively improve the efficiency of database query scanning.

Description

Database parallel scanning method, device, electronic device and storage medium

Technical Field

The present application relates to the field of database technology, and in particular to a database parallel scanning method, device, electronic device and storage medium.

Background Art

In the daily application of databases, for scenarios with large amounts of data, parallel scanning is usually used to scan and read data to speed up the process. In the process of parallel scanning of databases, related technologies usually group the data pages of the database, and have multiple threads perform scanning tasks simultaneously, with each thread responsible for scanning one group of data pages. Although parallel execution can improve the efficiency of database scanning queries, it does not fully utilize the database index, and the efficiency improvement effect is limited.

Summary of the invention

The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application provides a database parallel scanning method, device, electronic device and storage medium, which can effectively improve the efficiency of the database.

In a first aspect, an embodiment of the present application provides a database parallel scanning method, comprising:

Obtain a data index tree of the database and create multiple scanning threads; the data index tree includes multiple index pages, the index pages are stored in order according to page numbers, and the first scanning thread is used as the initial thread;

Using the initial thread to acquire a synchronization lock, selecting an initial index page from the index pages and determining a preset number, and performing data scanning on the initial index page, when performing the data scanning, the initial thread releases the synchronization lock;

Based on the initial index page and the page number, sequentially select the preset number of index pages as target index pages;

The target page numbers of the target index pages are sequentially written into the preset shared parameters. After each writing, the scanning thread is awakened by using the synchronization lock, and the scanning thread is used to perform data scanning on the target index page.

In some embodiments of the present application, the database further includes at least one data page, the data page includes at least one data tuple, the index page includes at least one index tuple, the index tuple includes a data page number for indicating a first position of the data page and a data sequence number for indexing a second position of the data tuple; the data scanning process includes the following steps:

Start scanning from the first index tuple of the index page, and use the data page number to find the data page at the first position as the target data page;

Searching the target data page for the data tuple at the second position using the data sequence number;

All the index tuples of the index page are scanned in sequence to obtain the data tuples corresponding to the index tuples.

In some embodiments of the present application, sequentially selecting the preset number of index pages as target index pages based on the initial index page and the page number includes:

Obtaining the initial page number of the initial index page;

The initial page number is increased sequentially, a preset number of target page numbers are selected, and the index pages of the target page numbers are used as the target index pages.

In some embodiments of the present application, waking up the scanning thread by using the synchronization lock, and performing data scanning on the target index page by using the scanning thread, further includes:

Select the scanning thread as the target thread in order, and use the target thread to acquire the synchronization lock;

Acquire the target page number from the shared parameter, and select the target index page based on the target page number;

The target thread is used to scan the data of the target index page, and the synchronization lock is released at the same time, and the shared parameter is pointed to the next target page number.

In some embodiments of the present application, the selecting an initial index page from the index pages and determining a preset number includes:

Filtering the index page according to the initial filter value to obtain the initial index page;

An end index page is determined according to the filtering end value, and the preset number is obtained according to the end index page and the initial index page.

In some embodiments of the present application, the method further includes:

When the scanning thread accesses the end index page, a scanning end status is generated;

The scanning end status is broadcasted to each scanning thread, and after the scanning thread finishes scanning the corresponding index page, the scanning thread is terminated according to the scanning end status.

In some embodiments of the present application, the data index tree includes a plurality of leaf nodes; the method further includes:

For each of the index pages, obtain the page number of the adjacent index page to obtain the adjacent page number;

The index page and the adjacent page numbers are used as node elements, and the node elements are stored in the leaf nodes in sequence according to the order of the page numbers to obtain the data index tree.

In a second aspect, the embodiment of the present application further provides a database parallel scanning device, which applies the database parallel scanning method described in the embodiment of the first aspect of the present application, including:

An acquisition module is used to acquire a data index tree of a database and create multiple scanning threads; the data index tree includes multiple index pages, the index pages are stored in order according to page numbers, and the first scanning thread is used as an initial thread;

A first scanning module, used to acquire a synchronization lock using the initial thread, select an initial index page from the index pages and determine a preset number, and perform data scanning on the initial index page. When performing the data scanning, the initial thread releases the synchronization lock;

A selection module, configured to sequentially select the preset number of index pages as target index pages based on the initial index page and the page number;

The second scanning module is used to write the target page number of the target index page into the preset shared parameters in sequence, and after each writing, wake up the scanning thread by using the synchronization lock, and use the scanning thread to scan the data of the target index page.

In a third aspect, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the database parallel scanning method as described in the embodiment of the first aspect of the present application is implemented.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the database parallel scanning method as described in the embodiment of the first aspect of the present application.

The embodiments of the present application include at least the following beneficial effects:

The embodiment of the present application provides a database parallel scanning method, device, electronic device and storage medium. In the method, the data index tree of the database is first obtained and multiple scanning threads are created. The data index tree includes multiple index pages. The index pages are stored in order according to the page numbers, and the first scanning thread is used as the initial thread. The initial thread is used to obtain a synchronization lock, select an initial index page from the index page and determine a preset number, and perform data scanning on the initial index page. When performing data scanning, the initial thread releases the synchronization lock. Then, based on the page number of the initial index page, a preset number of index pages are selected in order as target index pages, and the target page numbers of the target index pages are written into the preset shared parameters in sequence. After each write, the scanning thread is awakened by the synchronization lock, and the scanning thread is used to perform data scanning on the target index page. Therefore, by setting the synchronization lock and shared parameters, different scanning threads can read the index page in order and perform data scanning, combining parallel scanning and index scanning, and effectively improving the efficiency of database scanning query.

Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

FIG1 is a schematic diagram of a flow chart of a database parallel scanning method provided by an embodiment of the present application;

FIG2 is a schematic diagram of a flow chart of a database parallel scanning method provided by another embodiment of the present application;

FIG3 is a schematic diagram of the flow chart of step S103 in FIG1 ;

FIG4 is a schematic diagram of the process of step S104 in FIG1 ;

FIG5 is a schematic diagram of the process of step S102 in FIG1 ;

FIG6 is a schematic flow chart of a database parallel scanning method provided by another embodiment of the present application;

FIG7 is a schematic flow chart of a database parallel scanning method provided by yet another embodiment of the present application;

FIG8 is a flow chart of a database parallel scanning method provided by an embodiment of the present application;

FIG9 is a schematic diagram of a database parallel scanning device module provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

Reference numerals: acquisition module 100 , first scanning module 200 , selection module 300 , second scanning module 400 , electronic device 1000 , processor 1001 , memory 1002 .

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

In the description of the present application, it should be understood that descriptions involving orientation, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

In the description of this application, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

In the daily application of databases, for scenarios with large amounts of data, parallel scanning is usually used to scan and read data to speed up the process. Related technologies In the process of parallel scanning of databases, the data pages of the database are usually grouped, and multiple threads perform the scanning task at the same time. Each thread is responsible for scanning one group of data pages, and finally the data is aggregated to form the final result. In the scanning process, some filtering conditions are usually included. To improve the scanning speed, common practices are: 1. By creating a partition table, the partition table is trimmed according to the filtering conditions to reduce the data to be scanned and improve the query performance. 2. Using parallel scanning, each thread is responsible for scanning and filtering a part of the data. 3. Use index scanning to quickly find the page containing the specified data and improve the scanning speed. 4. For the parallel partition tables after partition trimming and filtering, each thread scans a single partition table and uses the index of the partition table to perform index scanning.

However, the scenario coverage in the related art is not comprehensive enough. For parallel index scanning between partition tables, the target table needs to be a partition table, and an index is created in each partition table. Or the database is divided into directories, and each thread creates an index for a directory separately, that is, there are multiple indexes, but different indexes are independent of each other. It is equivalent to sharding the database and creating a separate index for each shard. Then use hashing to rebuild these multiple indexes, and distribute the data to different indexes according to the hash value. In the scanning stage, if the scanning condition meets the hash value, it is hashed to the corresponding index and scanned. Here, a single thread or a single process scans an index, which is a serial process. If the hash condition is not met, parallel index scanning is performed, and the parallelism here means that each thread scans an index separately, and multiple threads are parallel. There is still only one thread scanning a single index. Similar to the partition table index in the related art, an index is created separately for each partition data, and each thread scans different partition indexes to achieve a parallel effect. It is essentially parallel between multiple indexes and serial within a single index. Therefore, parallel scanning and index scanning are not used in combination, the database index is not fully utilized, and the performance improvement is not maximized, so the efficiency improvement of database query scanning is limited.

Based on this, the embodiments of the present application provide a database parallel scanning method, device, electronic device and storage medium, which can allow different scanning threads to read the index pages of a single data index tree in an orderly manner and perform data scanning by setting synchronization locks and shared parameters. It combines parallel scanning and index scanning, and even scanning a single index of a single table can be executed in parallel, effectively improving the efficiency of database scanning queries.

The embodiments of the present application provide a database parallel scanning method, device, electronic device and storage medium, which are specifically described through the following embodiments. First, the database parallel scanning method in the embodiments of the present application is described.

The database parallel scanning method provided in the embodiment of the present application relates to the field of database technology, and in particular to the field of database scanning technology. The database parallel scanning method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, a computer program can be a native program or a software module in an operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, such as a client that supports database parallel scanning, that is, a program that only needs to be downloaded to a browser environment to run. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server via a network. The database parallel scanning method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (Peer To Peer, P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. The server can be installed with a server end of a database parallel scanning system, through which the server end can interact with the terminal, for example, the corresponding software is installed on the server end, and the software can be an application that implements a database parallel scanning method, etc., but is not limited to the above form. The terminal and the server can be connected through a communication connection method such as Bluetooth, Universal Serial Bus (USB) or a network, and this embodiment is not limited here.

The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

The following describes a database parallel scanning method in an embodiment of the present invention.

FIG1 is an optional flow chart of a database parallel scanning method provided by an embodiment of the present invention. The method in FIG1 may include but is not limited to steps S101 to S104. It is also understood that the present embodiment does not specifically limit the order of steps S101 to S104 in FIG1. The order of steps may be adjusted or some steps may be reduced or increased according to actual needs.

Step S101, obtaining a data index tree of a database and creating multiple scanning threads.

In some embodiments, the data index tree of the database is a B-Tree, which is a self-balancing tree that can keep data in order and retrieve data efficiently. Specifically, the data index tree includes multiple index pages, and the index pages are stored in order according to page numbers. For example, the data index tree has three index pages, and the corresponding page numbers are 1, 2, and 3, respectively. The three index pages can be stored in order from small to large numbers, and this embodiment does not limit this.

In some embodiments, multiple scanning threads are created, and the first scanning thread is used as the initial thread. Specifically, if the database is completely scanned, the number of scanning threads created is consistent with the number of index pages; if the database is partially scanned according to the filtering condition, the number of scanning threads created is consistent with the number of index pages that meet the filtering condition, which is not limited in this embodiment.

Step S102, using the initial thread to obtain a synchronization lock, selecting an initial index page from the index pages and determining a preset number, and performing data scanning on the initial index page. When performing the data scanning, the initial thread releases the synchronization lock.

In some embodiments, in order to avoid conflicts and inconsistencies caused by multiple scanning threads accessing the same index page of the database at the same time, the present application uses a synchronization mechanism, such as a synchronization lock. The synchronization lock can ensure that only one thread can perform a preset operation at a certain time. Specifically, the initial thread is used to obtain the synchronization lock, so when the initial thread starts executing, it first obtains a synchronization lock. This means that before this synchronization lock is released, other scanning threads that attempt to obtain the synchronization lock will be blocked, ensuring that the initial thread can safely perform the preset operation.

In some embodiments, the initial thread selects an initial index page from the index pages and determines a preset number. Specifically, if the database is completely scanned, the initial index page is the first index page; if the database is partially scanned according to the filtering condition, the initial index page is the first index page that meets the filtering condition, which is not limited in this embodiment.

The synchronization lock is released when the initial thread starts scanning the data of the initial index page, thereby allowing other scanning threads to obtain the synchronization lock during the waiting period and then perform related operations, thereby ensuring data consistency while maximizing concurrency performance.

Step S103 : selecting a preset number of index pages as target index pages in order based on the initial index pages and the page numbers.

In some embodiments, a preset number of index pages are selected in order as target index pages based on the initial index page and the page number. For example, assuming that the data index tree includes 100 index pages, which are stored in order according to the page code of [1,100], the initial index page obtained according to the filtering condition is the 15th index page and the preset number is 10, then the target index pages are the 10 index pages corresponding to the page numbers [16,25], and this embodiment does not limit this.

Step S104, writing the target page numbers of the target index pages into the preset shared parameters in sequence, and after each writing, waking up the scanning thread by using the synchronization lock, and using the scanning thread to scan the data of the target index page.

In some embodiments, the target page number of the target index page is sequentially written into the preset shared parameters, and after each write, a scanning thread is awakened by the synchronization lock, so that the target index page is scanned by the scanning thread. Specifically, while the initial scanning thread obtains the initial index page, the target page number 16 is written into the shared parameters, and then the synchronization lock is released. The next scanning thread is awakened by the synchronization lock to obtain the target index page according to the target page number, and then the next target page number 17 is written into the shared parameters, and then the synchronization lock is released. And so on, by setting the synchronization lock and shared parameters, different scanning threads can be used to perform data scans on different index pages in parallel and orderly, which effectively improves the efficiency of database query scanning, and this embodiment does not limit this.

2 , in some embodiments of the present application, the data scanning process in the above steps may include but is not limited to the following steps S201 to S202 .

Step S201 , starting scanning from the first index tuple of the index page, and using the data page number to find the first data page as the target data page.

In some embodiments, the database further includes at least one data page, each data page includes at least one data tuple, and the data tuple stores corresponding data. The index page includes at least one index tuple, each index tuple includes a data page number for indicating the first position of the data page.

When the scanning thread performs data scanning on the index page, the scanning starts from the first index tuple of the index page, and the data page number in the index tuple is used to find the data page at the first position and use it as the target data page. For example, the database includes 10,000 data pages. When the data page number in the index tuple is 3,000, the first position corresponds to the 3,000th data page, thereby obtaining the target data page, which is not limited in this embodiment.

Step S202: Search the data tuple at the second position in the target data page using the data sequence number.

In some embodiments, the index tuple also includes a data sequence number for the second position of the index data tuple, and the scanning thread can use the data sequence number to find the data tuple at the second position in the target data page. Exemplarily, the target data page includes 500 data tuples. When the data sequence number in the index tuple is 100, the second position corresponds to the 100th data tuple in the target data page, and this embodiment does not limit this.

Step S203, scan all index tuples of the index page in sequence to obtain data tuples corresponding to the index tuples.

In some embodiments, the index tuple may also include a key value, and the corresponding index tuple may be searched through the key value, thereby searching the data tuple in the corresponding data page according to the data page number and data sequence number in the index tuple. Specifically, each index tuple in the index page is stored in order according to the key value, for example, the key value in index tuple A = (10, (3000, 100)) is 10, and the key value in index tuple B = (11, (1000, 200)) is 11, so index tuple A is stored before index tuple B, and this embodiment does not limit this.

The scanning thread is used to scan all index tuples of the index page in sequence to obtain the data tuple corresponding to the index tuple, thereby realizing the data scanning of the index page. By scanning different index pages in parallel through multiple scanning threads, the simultaneous parallel scanning of indexes and data is realized, and the index is fully utilized in the scanning process, which effectively improves the efficiency and performance of database query scanning.

3 , in some embodiments of the present application, in the above step S103 , a preset number of index pages are sequentially selected as target index pages based on the initial index page and the page number, which may include but is not limited to the following steps S301 to S302 .

Step S301, obtaining the initial page number of the initial index page.

In some embodiments, each index page corresponds to a page number, and the index pages are stored in order according to the page numbers. The initial page number of the initial index page is obtained. Correspondingly, when the initial index page is the first index page, the initial page number is 1; when the initial index page is the 15th index page, the initial page number is 15. The specific determination depends on whether there is a filtering condition, and this embodiment does not limit this.

Step S302 , increasing the initial page number in sequence, selecting a preset number of target page numbers, and using the index pages of the target page numbers as target index pages.

In some embodiments, the initial page number is increased sequentially, a preset number of target page numbers are selected, and the index page of the target page number is used as the target index page. Exemplarily, when the preset number is 10 and the initial index page is 15, the initial page number is increased to 16, and 1 target page number is selected at this time, and the 16th index page is obtained as the target index page; then the initial page number is increased to 17, and 2 target page numbers are selected at this time, and the 17th index page is obtained as the target index page. And so on, until the initial page number is increased to 25, 10 target page numbers are selected correspondingly, and finally 10 target index pages are obtained, which is not limited in this embodiment.

As shown in FIG. 4 , in some embodiments of the present application, in the above step S104 , the scanning thread is awakened by using a synchronization lock, and the scanning thread is used to perform data scanning on the target index page, which may include but is not limited to the following steps S401 to S403 .

Step S401, selecting a scanning thread as a target thread in order, and using the target thread to acquire a synchronization lock.

In some embodiments, scanning threads are selected as target threads in sequence. For example, if there are 10 scanning threads, each scanning thread is selected as a target thread to acquire a synchronization lock in turn. This embodiment does not impose any limitation on this.

Step S402 , obtaining a target page number from the shared parameters, and selecting a target index page based on the target page number.

In some embodiments, the target thread obtains the target page number from the shared parameters and selects the target index page based on the target page number. For example, if the target page number in the shared parameters is 17, the target thread can select the 17th index page as the target index page according to the target page number. This embodiment does not impose any restrictions on this.

Step S403: Use the target thread to scan the target index page for data, and release the synchronization lock at the same time, and point the shared parameter to the next target page number.

In some embodiments, the target thread is used to scan the target index page for data, and the synchronization lock is released at the same time, and the shared parameter is pointed to the next target page number. It can be understood that since the index page is stored in order according to the page number, the next target page number can be obtained when the target thread scans the target index page for data. Before releasing the synchronization lock, the shared parameter is updated to point to the next target page number, and this embodiment does not limit this.

5 , in some embodiments of the present application, selecting an initial index page from the index pages and determining a preset number in the above step S102 may include but is not limited to the following steps S501 to S502 .

Step S501, filtering in the index pages according to the initial filtering value to obtain the initial index page.

In some embodiments, the filtering condition includes a filtering initial value, and the index page is screened according to the filtering initial value to obtain the initial index page. For example, for a database query statement "select * from t1 where c1>10000", where the filter condition is after where, c1>10000 is the filtering initial value, the starting thread can select the index page with c1>10000 from the database table t1 according to the filtering initial value, and this embodiment does not limit this.

Step S502, determining the end index page according to the filtering end value, and obtaining a preset number according to the end index page and the initial index page.

In some embodiments, the filter condition may also include a filter end value. For example, for the database query statement "select * from t1 where c1>10000and c1<20000", the filter condition is after where, c1>10000 is the filter initial value, and c1<20000 is the filter end value. Therefore, the starting thread can find the initial index page containing data 10000 from the database table t1 according to the filter condition. Due to the orderliness of the index page, it only needs to search from the initial index page to the end index page containing data 20000. This embodiment does not limit this.

The initial index page is determined according to the initial filtering value, the end index page is determined according to the end filtering value, and the preset number is obtained according to the end index page and the initial index page. For example, the page number corresponding to the initial index page is 15, and the page number corresponding to the end index page is 25, and the preset number is 25-15=10, which is not limited in this embodiment.

6 , in some embodiments of the present application, the database parallel scanning method may also include but is not limited to the following steps S601 to S602 .

Step S601: When the scanning thread accesses the end index page, a scanning end status is generated.

In some embodiments, when the scanning thread accesses the end index page, a scanning end status is generated. The scanning end status can be stored in a shared parameter, which is not limited in this embodiment.

Step S602, broadcasting the scanning end status to each scanning thread, and terminating the scanning thread according to the scanning end status after the scanning thread finishes scanning the corresponding index page.

In some embodiments, the scan end status is broadcast to each scanning thread. After each scanning thread has scanned the corresponding index page, that is, after each scanning thread has completed the data scan of each index tuple in the corresponding index page, the scanning thread is terminated according to the scan end status, and the entire scan is ended. This embodiment does not impose any restrictions on this.

7 , in some embodiments of the present application, the database parallel scanning method may also include but is not limited to the following steps S701 to S702 .

Step S701: for each index page, obtain the page number of its adjacent index page to obtain the adjacent page number.

In some embodiments, for each index page, the page numbers of its adjacent index pages are obtained to obtain the adjacent page numbers. For example, 100 index pages are stored in order according to the page numbers of [1,100]. For the first index page, its adjacent index page is only the second index page, so the adjacent page number is 2; for the 50th index page, its adjacent index pages are the 49th index page and the 51st index page, so the adjacent page numbers are 49 and 51, and this embodiment does not limit this.

Step S702: Use the index page and adjacent page numbers as node elements, and store the node elements into leaf nodes in sequence according to the order of page numbers to obtain the data index tree.

In some embodiments, the data index tree includes multiple leaf nodes. The index page and the adjacent page numbers are used as node elements, and the node elements are stored in the leaf nodes in the order of the page numbers to obtain the data index tree. Therefore, when the scanning thread reads the index page of the leaf node, the adjacent page number of the index page can also be obtained, so that the adjacent page number is stored as the target page number in the shared parameter, and then the synchronization lock is released or the next one or more scanning threads are awakened by other conditional parameters, so as to read the target page number in the shared parameter to perform data scanning on the corresponding target index page, which is not limited in this embodiment.

In some embodiments, the data scan within a single scanning thread still maintains the orderliness of the index data output, and the upper-level query plan can use this orderliness to directly perform operations such as sorting, aggregation, and merge joining without additional sorting. This embodiment does not limit this.

The present application is described below with a complete embodiment:

This application adds thread synchronization functions to the index scanning in the executor and the B-Tree index access in the storage engine, so that multiple scanning threads can cooperate to access the B-Tree index in parallel, thereby achieving the effect of parallel access to the data page pointed to by the index. Referring to the database parallel scanning flow chart shown in Figure 8, the data index tree B-Tree includes multiple index pages (leaf nodes), and the index pages are stored in order according to the page numbers, and point to each other with adjacent index pages through adjacent page numbers. For each index page that meets the filtering conditions, a scanning thread can be created. The initial scanning thread obtains the synchronization lock, accesses the initial index page according to the filtering conditions, obtains the adjacent page number of the initial index page as the target page number, and stores it in the shared parameters, releases the synchronization lock at the same time, and performs data scanning on the initial index page. When the synchronization lock is released, the next scanning thread is awakened, the scanning thread reads the target page number from the shared parameters, and accesses the corresponding target index page to obtain the next target page number.

Therefore, different scanning threads are awakened to access the corresponding index pages, and according to the information of the index tuple in the index page, the data page is searched to read the corresponding data tuple. And when the previous scanning thread is scanning the index page and data page, because the synchronization lock is released, other awakened scanning threads are also performing the same scanning process, reading the index page and data page at the same time. Therefore, all scanning threads are performing the scanning process of index and data at the same time, and the scanning process greatly improves the scanning efficiency due to the use of index.

The database parallel scanning method in the embodiment of the present application is mainly the parallelization of the B-Tree index type in the index scanning stage, that is, multiple scanning threads perform parallel scanning on a single index tree, which is page-level parallelism, and there is no need to partition the data and create a separate index for each partition. The related technology only partitions the data and creates multiple indexes. Each partition index scan is essentially serial, but multiple partition indexes are scanned simultaneously. The data in the embodiment of the present application does not need to be partitioned, and a single index tree is scanned by multiple threads, including a single partition index. It can also be scanned by multiple threads in parallel, which is a lower-level parallelism. The parallel index scanning performance can be improved by more than 40%, thereby effectively improving the efficiency and performance of database query scanning.

The embodiment of the present invention further provides a database parallel scanning device, which can implement the above database parallel scanning method. As shown in FIG. 9 , in some embodiments of the present application, the device includes:

The acquisition module 100 is used to acquire a data index tree of a database and create multiple scanning threads; the data index tree includes multiple index pages, and the index pages are stored in order according to page numbers, and the first scanning thread is used as the initial thread;

The first scanning module 200 is used to obtain a synchronization lock using an initial thread, select an initial index page from the index pages and determine a preset number, and perform data scanning on the initial index page. When performing data scanning, the initial thread releases the synchronization lock;

A selection module 300, configured to sequentially select a preset number of index pages as target index pages based on the initial index page and the page number;

The second scanning module 400 is used to write the target page number of the target index page into the preset shared parameters in sequence, and after each writing, wake up the scanning thread by using the synchronization lock, and use the scanning thread to scan the data of the target index page.

The specific implementation of the database parallel scanning device of this embodiment is basically the same as the specific implementation of the above-mentioned database parallel scanning method, and will not be described in detail here.

Fig. 10 shows an electronic device 1000 provided in an embodiment of the present application. The electronic device 1000 includes: a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001, and the computer program is used to execute the above-mentioned database parallel scanning method when it is executed.

The processor 1001 and the memory 1002 may be connected via a bus or other means.

The memory 1002 is a non-transient computer-readable storage medium that can be used to store non-transient software programs and non-transient computer executable programs, such as the database parallel scanning method described in the embodiment of the present application. The processor 1001 implements the above-mentioned database parallel scanning method by running the non-transient software program and instructions stored in the memory 1002.

The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store and execute the above-mentioned database parallel scanning method. In addition, the memory 1002 may include a high-speed random access memory 1002, and may also include a non-transient memory 1002, such as at least one storage device storage device, a flash memory device or other non-transient solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory 1002 remotely arranged relative to the processor 1001, and these remote memories 1002 may be connected to the electronic device 1000 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The non-transient software programs and instructions required to implement the above-mentioned database parallel scanning method are stored in the memory 1002. When executed by one or more processors 1001, the above-mentioned database parallel scanning method is executed, for example, method steps S101 to S104 in Figure 1, method steps S201 to S203 in Figure 2, method steps S301 to S302 in Figure 3, method steps S401 to S403 in Figure 4, method steps S501 to S502 in Figure 5, method steps S601 to S602 in Figure 6, and method steps S701 to S702 in Figure 7 are executed.

The present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, which implements the above-mentioned database parallel scanning method when the computer program is executed by the processor. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The database parallel scanning method, device, electronic device and storage medium provided by the embodiment of the present application first obtain the data index tree of the database and create multiple scanning threads. The data index tree includes multiple index pages, and the index pages are stored in order according to the page numbers. The first scanning thread is used as the initial thread. The initial thread is used to obtain the synchronization lock, select the initial index page from the index page and determine the preset number, and perform data scanning on the initial index page. When performing data scanning, the initial thread releases the synchronization lock. Then, based on the page number of the initial index page, a preset number of index pages are selected in order as the target index page, and the target page number of the target index page is written into the preset shared parameters in sequence. After each write, the scanning thread is awakened by the synchronization lock, and the scanning thread is used to perform data scanning on the target index page. Therefore, by setting the synchronization lock and shared parameters, different scanning threads can read the index pages in a single data index tree in order and perform data scanning, which combines parallel scanning and index scanning, and effectively improves the efficiency of database scanning query.

The above described embodiments are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, storage device storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

It should also be understood that the various implementations provided in the embodiments of the present application can be arbitrarily combined to achieve different technical effects. The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementations, and technicians familiar with the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present application.

Claims (7)

1. A database parallel scanning method, characterized by comprising: Obtain a data index tree of the database and create multiple scanning threads; the data index tree includes multiple index pages, the index pages are stored in order according to page numbers, and the first scanning thread is used as the initial thread; Using the initial thread to acquire a synchronization lock, selecting an initial index page from the index pages and determining a preset number, and performing data scanning on the initial index page, when performing the data scanning, the initial thread releases the synchronization lock; Based on the initial index page and the page number, sequentially select the preset number of index pages as target index pages; Writing the target page number of the target index page into the preset shared parameters in sequence, and after each writing, waking up the scanning thread by using the synchronization lock, and performing data scanning on the target index page by using the scanning thread; The database further includes at least one data page, the data page includes at least one data tuple, the index page includes at least one index tuple, the index tuple includes a data page number for indicating a first position of the data page and a data sequence number for indexing a second position of the data tuple; the data scanning process includes the following steps: Start scanning from the first index tuple of the index page, and use the data page number to find the data page at the first position as the target data page; Searching the target data page for the data tuple at the second position using the data sequence number; Scanning all the index tuples of the index page in sequence to obtain the data tuples corresponding to the index tuples; The selecting an initial index page from the index pages and determining a preset number includes: Filtering the index page according to the initial filter value to obtain the initial index page; Determining an end index page according to the filtering end value, and obtaining the preset number according to the end index page and the initial index page; When the scanning thread accesses the end index page, a scanning end status is generated; The scanning end status is broadcasted to each scanning thread, and after the scanning thread finishes scanning the corresponding index page, the scanning thread is terminated according to the scanning end status. 2. The database parallel scanning method according to claim 1, characterized in that the step of sequentially selecting the preset number of index pages as target index pages based on the initial index pages and the page numbers comprises: Obtaining the initial page number of the initial index page; The initial page number is increased sequentially, a preset number of target page numbers are selected, and the index pages of the target page numbers are used as the target index pages. 3. The database parallel scanning method according to claim 2, characterized in that the step of waking up the scanning thread by using the synchronization lock and scanning the target index page by using the scanning thread further comprises: Select the scanning thread as the target thread in order, and use the target thread to acquire the synchronization lock; Acquire the target page number from the shared parameter, and select the target index page based on the target page number; The target thread is used to scan the data of the target index page, and the synchronization lock is released at the same time, and the shared parameter is pointed to the next target page number. 4. The database parallel scanning method according to any one of claims 1 to 3, characterized in that the data index tree includes a plurality of leaf nodes; the method further comprises: For each of the index pages, obtain the page number of the adjacent index page to obtain the adjacent page number; The index page and the adjacent page numbers are used as node elements, and the node elements are stored in the leaf nodes in sequence according to the order of the page numbers to obtain the data index tree. 5. A database parallel scanning device, characterized in that the database parallel scanning method according to any one of claims 1 to 4 is applied, comprising: An acquisition module is used to acquire a data index tree of a database and create multiple scanning threads; the data index tree includes multiple index pages, the index pages are stored in order according to page numbers, and the first scanning thread is used as an initial thread; A first scanning module, used to acquire a synchronization lock using the initial thread, select an initial index page from the index pages and determine a preset number, and perform data scanning on the initial index page. When performing the data scanning, the initial thread releases the synchronization lock; A selection module, configured to sequentially select the preset number of index pages as target index pages based on the initial index page and the page number; A second scanning module is used to write the target page number of the target index page into the preset shared parameter in sequence, and after each writing, wake up the scanning thread by using the synchronization lock, and use the scanning thread to perform data scanning on the target index page; The database further includes at least one data page, the data page includes at least one data tuple, the index page includes at least one index tuple, the index tuple includes a data page number for indicating a first position of the data page and a data sequence number for indexing a second position of the data tuple; the data scanning process includes the following steps: Start scanning from the first index tuple of the index page, and use the data page number to find the data page at the first position as the target data page; Searching the target data page for the data tuple at the second position using the data sequence number; Scanning all the index tuples of the index page in sequence to obtain the data tuples corresponding to the index tuples; The selecting an initial index page from the index pages and determining a preset number includes: Filtering the index page according to the initial filter value to obtain the initial index page; Determining an end index page according to the filtering end value, and obtaining the preset number according to the end index page and the initial index page; When the scanning thread accesses the end index page, a scanning end status is generated; The scanning end status is broadcasted to each scanning thread, and after the scanning thread finishes scanning the corresponding index page, the scanning thread is terminated according to the scanning end status. 6. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the database parallel scanning method according to any one of claims 1 to 4 when executing the computer program. 7. A computer-readable storage medium, characterized in that the storage medium stores a program, and the program is executed by a processor to implement the database parallel scanning method according to any one of claims 1 to 4.