CN111259029A - Data read and write batch processing method, server and computer readable storage medium - Google Patents

Abstract

The invention relates to a data processing technology, and discloses a data read-write batch processing method, which comprises the following steps: when batch data read-write requests aiming at a single database or multiple databases are received from a calling party, respectively allocating a unique request ID for each read-write request; creating a storage structure for each read-write request in a hash memory, wherein the storage structure comprises a keyword and a value structure, writing a request ID into the keyword, and writing a connection mark of a calling party, a read-write sql statement and a read-write sql number into the value structure; sending each read-write task of the read-write request to a corresponding database for parallel processing according to the database identifier in each read-write request; and receiving a processing result returned by the database and updating the processing result into the value structure corresponding to the read-write request. The invention also provides a server and a computer readable storage medium. The invention can perform parallel processing on the batch read-write tasks and reduce the time consumption of access.

Description

Data read and write batch processing method, server and computer readable storage medium

technical field

The present invention relates to the technical field of data processing, and in particular, to a method for batch processing of data reading and writing, a server and a computer-readable storage medium.

Background technique

At present, when it involves a single database or batch data reading and writing of multiple databases, it cannot be perfectly processed. For batch reading and writing of a single database, most of the current practice is that the processing module first executes a SQL task, and then executes the next SQL task after the database processing returns. If the read or write processing fails, it is returned to the caller. For multi-database batch read and write, most of the current practice is that the processing module first determines which database the data table corresponding to the first SQL task is located, and then executes the SQL task from the database, and then determines the second one after the data is returned. In which database the data table corresponding to the sql task is located, and so on. If one of the sql tasks fails, it will be returned to the caller.

The disadvantage of the above method is that there is no sequential execution relationship between multiple SQL tasks, but the current practice is to perform sequential execution for convenient and simple processing, which takes too long. Moreover, each additional type of reading and writing requires additional logic code maintenance and labor costs.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a method for batch processing data read and write, a server and a computer-readable storage medium to solve at least one of the above technical problems.

First, in order to achieve the above purpose, the present invention proposes a method for batch processing of data read and write, the method comprising the steps of:

When a batch data read/write request for a single database or multiple databases is received from the caller, a unique request ID is assigned to each read/write request, and the request parameters in the read/write request include the database identifier and the read/write task, Each read-write task includes a read-write sql number and a read-write sql statement to be executed;

Create a storage structure for each read and write request in the hash memory, including a keyword and a value structure, write the request ID into the keyword, and write the caller corresponding to each read and write request The connection sign, and the read-write sql statement and the read-write sql number corresponding to each read-write task in the read-write request are written into the value structure as an array;

According to the database identifier in each read and write request, each read and write task of the read and write request is sent to the corresponding database for parallel processing; and

The processing result of each read/write task for each read/write request returned by the database is received and updated into the value structure corresponding to the read/write request.

Optionally, the method further includes the steps:

According to the value structure, determine whether the read and write tasks of the read and write request have all returned processing results;

When the read-write tasks of the read-write request have all returned processing results, the processing results of all the read-write tasks of the read-write request are combined and returned to the caller.

Optionally, in the step of sending each read-write task of the read-write request to the corresponding database for parallel processing according to the database identifier in each of the read-write requests, the read-write task sent to each database The data includes request ID, read and write SQL numbers, and read and write SQL statements.

Optionally, the processing result includes a request ID, a read/write SQL number, a task return identifier, and task return data.

Optionally, the value structure also includes a task return identifier and task return data corresponding to each read-write task, and the processing result of each read-write task returned by the receiving database for the read-write request is updated to the corresponding task return identifier. The steps in the value structure corresponding to the read-write request include:

Asynchronously receive the processing results returned by each database, obtain the corresponding value structure from the hash memory according to the request ID in each processing result, and then find out the corresponding read and write tasks in the value structure according to the read and write SQL numbers, and update the The task return identifier and the task return data of the read and write tasks.

Optionally, the task return identifier is used to indicate whether the read-write sql statement is executed successfully or not, and the return value 0 indicates that the execution is successful, and a non-zero value indicates that the execution fails.

Optionally, the task return data is used to indicate the return data corresponding to the read-write sql statement, when the read-write sql statement is a query task and the execution is successful, the specific query result data is returned; when the read-write sql statement is When it is a non-query task or a query task fails to execute, it returns null.

Optionally, the method further includes the steps:

When the read and write tasks of the read and write requests do not all return processing results and need to continue to wait, other read and write requests continue to be processed at the same time.

In addition, in order to achieve the above object, the present invention also provides a server, including a memory and a processor, the memory stores a data read and write batch processing system that can run on the processor, and the data read and write batch processing system When the system is executed by the processor, the steps of the above-mentioned data read and write batch processing method are implemented.

Further, in order to achieve the above object, the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a data read-write batch processing system, and the data read-write batch processing system can be processed by at least one processor. Execute, so that the at least one processor executes the steps of the above-mentioned data read and write batch processing method.

Compared with the prior art, the data read and write batch processing method, server and computer-readable storage medium proposed by the present invention can provide a unified multi-database read and write task interface. The unique request ID of each read and write request is used as the key to save the corresponding information of each read and write task of each read and write request, and the batch of read and write tasks are sent to the corresponding database for parallel processing. The processing results returned by each read and write task are also written into the hash memory, so that batch read and write tasks of multiple databases can be processed concurrently without data confusion, and the access time is greatly reduced.

Description of drawings

1 is a schematic diagram of an optional application environment architecture of each embodiment of the present invention;

Fig. 2 is a schematic diagram of an optional hardware architecture of the server in Fig. 1;

3 is a schematic diagram of a program module of the first embodiment of the data read-write batch processing system of the present invention;

Fig. 4 is the schematic diagram of the storage structure that the present invention creates in hash memory;

5 is a schematic diagram of a program module of the second embodiment of the data read-write batch processing system of the present invention;

6 is a schematic flowchart of the first embodiment of the data read/write batch processing method of the present invention;

7 is a schematic flowchart of the second embodiment of the data read/write batch processing method of the present invention;

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the descriptions involving "first", "second", etc. in the present invention are only for the purpose of description, and should not be construed as indicating or implying their relative importance or implying the number of indicated technical features . Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of such technical solutions does not exist. , is not within the scope of protection required by the present invention.

Referring to FIG. 1 , it is a schematic diagram of an optional application environment architecture of each embodiment of the present invention.

In this embodiment, the present invention can be applied to an application environment including, but not limited to, the caller 1 , the server 2 , and the database 3 . Wherein, the caller 1 can be a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, a vehicle-mounted device, etc. Mobile devices, and stationary terminals such as digital TVs, desktop computers, notebooks, servers, etc. The server 2 may be a computing device such as a rack server, a blade server, a tower server or a cabinet server, and the server 2 may be an independent server or a server cluster composed of multiple servers.

The server 2 is connected in communication with one or more of the callers 1 and the database 3 through a network (not shown in the figure), respectively, for data transmission and interaction. The network can be an intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network Network, Bluetooth (Bluetooth), Wi-Fi, call network and other wireless or wired networks.

Wherein, each of the databases 3 corresponds to a database access module 30, through which the database access module 30 can receive and process read and write requests for the database 3, and then return the processing results.

Referring to FIG. 2 , it is a schematic diagram of an optional hardware architecture of the server 2 of the present invention.

In this embodiment, the server 2 may include, but is not limited to, the memory 11 , the processor 12 , and the network interface 13 that can communicate with each other through a system bus. It should be pointed out that FIG. 2 only shows the server 2 having the components 11-13, but it should be understood that it is not required to implement all the shown components, and more or less components may be implemented instead.

The server 2 may be a computing device such as a rack server, a blade server, a tower server or a cabinet server, and the server 2 may be an independent server or a server cluster composed of multiple servers.

The memory 11 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), random access memory (RAM), static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 2 , such as a hard disk or a memory of the server 2 . In other embodiments, the memory 11 may also be an external storage device of the server 2, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) equipped on the server 2 , SD) card, flash memory card (Flash Card) and so on. Of course, the memory 11 may also include both the internal storage unit of the server 2 and its external storage device. In this embodiment, the memory 11 is generally used to store the operating system and various application software installed on the server 2 , such as program codes of the data read and write batch processing system 200 . In addition, the memory 11 can also be used to temporarily store various types of data that have been output or will be output.

In some embodiments, the processor 12 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. This processor 12 is typically used to control the overall operation of the server 2 . In this embodiment, the processor 12 is configured to run program codes or process data stored in the memory 11 , for example, run the data read and write batch processing system 200 and the like.

The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is generally used to establish a communication connection between the server 2 and other electronic devices.

So far, the application environment architecture of the present invention and the hardware structure and functions of the related devices have been described in detail. In the following, various embodiments of the present invention will be proposed based on the above description.

First, the present invention proposes a data read/write batch processing system 200 .

Referring to FIG. 3 , it is a program module diagram of the first embodiment of the data read/write batch processing system 200 of the present invention.

In this embodiment, the data read and write batch processing system 200 includes a series of computer program instructions stored in the memory 11 , and when the computer program instructions are executed by the processor 12 , the data read and write according to the embodiments of the present invention can be implemented. Write batch operations. In some embodiments, the data read and write batch processing system 200 may be divided into one or more modules based on the specific operations implemented by the various parts of the computer program instructions. For example, in FIG. 3 , the data read/write batch processing system 200 may be divided into an allocation module 201 , a storage module 202 , a transmission module 203 , and an update module 204 . in:

The assigning module 201 is configured to assign a unique request ID to each read and write request received.

Specifically, the caller 1 initiates a batch data read/write request for a single database 3 or multiple databases 3 to the server 2 in json format. The read-write request includes request parameters such as a database identifier (db_id), a read-write SQL number (sql_id, the sql_id cannot be the same in the same request), and a SQL statement to be executed. The database identifier is used to identify the database access module 30 , for example, A indicates that the database A is accessed through the database access module A. Each read and write SQL number and the corresponding SQL statement represent a read and write task. Each read and write request may include one or more database identification and read and write tasks. The allocation module 201 receives the read and write requests, and allocates a unique request ID, ie request_id, to each of the read and write requests.

A batch data read and write request for a single database 3, for example, a database has a student table (stu), a teacher table (teac), and a student score table (stu_score), and now a batch data read and write request for this database is received. At the same time, query the data of "Xiao Wang" in the student table (sql1:select*from stu where stu_name='Xiaowang'), and query the data of "Mr. Zhang" in the teacher table (sql2:select*from teac where teac_name='Mr.Zhang' '), update the data of "Xiao Li" in the student score table (sql3:update stu_score set score=90where stu_name='Xiao Li'). The allocation module 201 can allocate a unique request ID, such as 178652986, to the read and write request.

Batch data read and write requests for multiple databases 3, for example, database A contains student table (stu), teacher table (teac), database B contains student score table (stu_score), and now a batch for database A and B is received For data read and write requests, it is necessary to query the data of "Xiao Zhang" in the student table of database A and the data of "Zhang San" in the teacher table (sql1:select*from stu where stu_name='Xiao Zhang'; sql2:select *from teac whereteac_name='Zhang San'), query the data of "Xiao Li" in the student transcript of database B and modify the data of "Xiao Ming" (sql3:select*from stu_score where stu_name='Xiao Li'; sql4:update stu_score setscore=90where stu_name='Xiao Ming'). Similarly, the allocation module 201 can also allocate a unique request ID, such as 178652987, to the read and write request.

The saving module 202 is configured to write the read and write requests into the hash memory.

Specifically, a storage structure for each of the read and write requests is created in the hash memory, and each of the storage structures includes a key (key) and a value structure. Write the read-write request into the hash memory, wherein the request ID, that is, request_id, is used as the key corresponding to the read-write request in the hash memory, and the caller connection corresponding to the read-write request is marked tcp socket and the read/write request. The read-write sql statement and the read-write sql_id corresponding to each read-write task in the write request are taken as an array and written into the value structure corresponding to the request ID. In addition, the array of the value structure also includes two items, a task return identifier (sql_ret) and a task return data (sql_data), which are used for subsequent writing of the processing result returned for the read and write request. Referring to FIG. 4, it is a schematic diagram of the storage structure created in the hash memory.

For example, using the request IDs of the above two read and write requests as the key, the information of the two read and write requests is stored in the hash memory, and the caller connection of the two read and write requests is used to identify, read and write SQL statements respectively. And read and write sql_id as an array, and write it into the value structure of the hash memory corresponding to the respective request ID. Taking the above read and write requests for databases A and B as an example, the request ID "178652987" of the read and write requests is used as the key in the hash memory, and the caller connection is marked to read and write SQL statements (select*from stu where stu_name= 'Xiao Zhang'; select*from teac whereteac_name='Zhang San'; select*from stu_score where stu_name='Xiao Li'; update stu_score set score=90where stu_name='Xiao Ming') and read and write sql_id (sql1, sql2, sql3 , sql4) as an array, write the corresponding value structure.

The sending module 203 is configured to send each read-write task of the read-write request to the corresponding database 3 for parallel processing according to the database identifier in the read-write request.

Specifically, obtain the request parameters in the read-write request, and send multiple data read-write tasks belonging to different databases 3 (or the same database 3 ) to the corresponding database 3 at the same time according to the database identifier in the request parameters, that is, db_id . The read and write task data sent to each database 3 includes a request ID (request_id), a task ID (task_id, which is consistent with sql_id), read and write SQL statements, and the like. Each database 3 includes a corresponding database access module 30 for receiving and processing read and write requests of the database 3, and then returning the processing result. The returned processing result includes a request ID (request_id), a task ID (task_id, which is consistent with sql_id), a task return identifier (sql_ret), and task return data (sql_data). Among them, the task return identifier is used to indicate whether the SQL statement is executed successfully or not. The return value of 0 means the execution is successful, and non-0 means the execution fails. The task return data is used to indicate the return data corresponding to the sql statement. When the sql statement is a query task and the execution is successful, the specific query result data is returned. When the sql statement is a non-query task or the query execution fails, a null value is returned. null.

For example, for the query task of querying the data of "Xiao Li" in the student's score table (sql3:select*fromstu_score where stu_name='Xiao Li'), the returned processing result in database B is 178652987 for the request ID and 3 for the task ID. The task return identifier is 0, and the task return data is {"stu_name": "Xiao Li", "stu_score": "95"}; for modifying the data of "Xiao Ming" in the student score table (sql4:update stu_score set score=90where stu_name ='Xiao Ming') This non-query task, in the returned processing result, the request ID is 178652987, the task ID is 4, the task return ID is 0, and the task return data is null.

In this embodiment, all read and write tasks in multiple read and write requests can be sent to the corresponding database 3 at the same time, all subtasks are processed in parallel, and the time consumption is the time consumption of one of the largest subtasks.

The updating module 204 is configured to receive the processing result returned by the database 3 and update it into the value structure corresponding to the read-write request.

Specifically, the update module 204 asynchronously receives the processing results of each read/write task returned by each database 3 for each of the read/write requests, and obtains the corresponding value structure from the hash memory according to the request ID in each processing result , and then find the corresponding read/write task in the value structure according to the sql_id (consistent with the task ID), and update the task return identifier and task return data of the read/write task.

For example, update the processing result returned by the above database B to the value structure corresponding to the request ID "178652987" in the hash memory.

The data read/write batch processing system provided by this embodiment can provide a unified multi-database read/write task interface. After receiving a batch data read/write request, the unique request ID of each read/write request is used as a key in the hash memory. Save the corresponding information of each read and write task of each read and write request, and send the batch of read and write tasks to the corresponding database for parallel processing, and also write the processing result returned for each read and write task into the hash memory , so that the batch read and write tasks of multiple databases can be processed concurrently without data confusion, and the access time is greatly reduced.

Referring to FIG. 5, it is a program module diagram of the second embodiment of the data read/write batch processing system 200 of the present invention. In this embodiment, the data read and write batch processing system 200 includes, in addition to the allocation module 201 , the storage module 202 , the transmission module 203 , and the update module 204 in the first embodiment, a judgment module 205 , a return module 206.

The judging module 205 is configured to judge whether the read and write tasks of the read and write request have all returned processing results.

Specifically, for each read/write request, it is determined whether all read/write tasks in the corresponding value structure have returned processing results. If not all are returned, continue to wait for the corresponding database 3 to return the processing result, until all the read-write tasks of the read-write request return the processing result.

It is worth noting that, in this embodiment, when the read and write tasks of the read and write requests have not all returned processing results and need to continue to wait, other read and write requests can be processed at the same time, that is, other read and write requests do not need to wait in line, Does not affect the processing time of other read and write requests.

The returning module 206 is configured to combine the processing results and return them to the caller 1 when all the reading and writing tasks of the reading and writing request have returned processing results.

Specifically, if all the read/write tasks of the read/write request have returned processing results, the caller connection flag in the corresponding value structure is obtained, and the processing results of all read/write tasks of the read/write request are combined. The result is returned to caller 1. The combined processing result returned to the caller 1 includes database ID (db_id), read/write sql_id, task return identifier (sql_ret), task return data (sql_data) and other data.

For example, for the read/write request whose request ID is 178652987, determine the four read/write tasks in the corresponding value structure (sql1:select*from stu where stu_name='Xiao Zhang'; sql2:select*from teac where teac_name='Zhang San' '; sql3:select*from stu_score where stu_name='Xiao Li';sql4:update stu_score set score=90where stu_name='Xiao Ming') have all been returned, that is, the task return identifier and task return data corresponding to these four task IDs Is it updated. If not all are returned, for example, only database B returns but database A has not returned, continue to wait for the processing result returned by database A (other read and write requests can be processed at the same time). If all have been returned, the processing results of all four tasks are combined and returned to the caller 1.

It is worth noting that, each module of the data read/write batch processing system 200 and the database access module 30 of each database 3 can be configured with multiple processes for synchronous processing. This embodiment processes read and write requests for a single database 3 or multiple databases 3 in batches, and can process multiple different read and write requests at the same time, that is, multiple read requests and multiple write requests of a single or multiple databases 3 can be mixed Batch and concurrent processing together.

The data read and write batch processing system provided by this embodiment can provide a unified multi-database read and write task interface, and the batch read and write tasks are simultaneously sent to the corresponding database for parallel processing, with high concurrency capability and greatly reduced access time. The caller only needs to assemble the read and write task list of the database to be accessed according to the unified interface, and then perform logical processing according to the processing result data returned by the unified interface, so as to achieve the effect of abstract isolation between the business layer and the data access layer, and improve production efficiency. In addition, the reading and writing of this embodiment does not depend on the specific table, and the fields are abstractly obtained, which supports dynamic expansion of the table without modifying the code, and improves the production capacity. In addition, this embodiment supports distributed deployment and configuration of multi-process processing, greatly utilizes CPU multi-core hardware resources, and improves processing capability.

In addition, the present invention also provides a batch processing method for data read and write.

Referring to FIG. 6 , it is a schematic flowchart of the first embodiment of the data read/write batch processing method of the present invention. In this embodiment, according to different requirements, the execution order of the steps in the flowchart shown in FIG. 6 can be changed, and some steps can be omitted.

The method includes the following steps:

Step S400, assign a unique request ID to each read and write request received.

Specifically, the caller 1 initiates a batch data read/write request for a single database 3 or multiple databases 3 to the server 2 in json format. The read-write request includes request parameters such as a database identifier (db_id), a read-write SQL number (sql_id, the sql_id cannot be the same in the same request), and a SQL statement to be executed. The database identifier is used to identify the database access module 30 , for example, A indicates that the database A is accessed through the database access module A. Each read and write SQL number and the corresponding SQL statement represent a read and write task. Each read and write request may include one or more database identification and read and write tasks. The server 2 receives the read and write requests, and assigns a unique request ID, ie, request_id, to each of the read and write requests.

A batch data read and write request for a single database 3, for example, a database has a student table (stu), a teacher table (teac), and a student score table (stu_score), and now a batch data read and write request for this database is received. At the same time, query the data of "Xiao Wang" in the student table (sql1:select*from stu where stu_name='Xiaowang'), and query the data of "Mr. Zhang" in the teacher table (sql2:select*from teac where teac_name='Mr.Zhang' '), update the data of "Xiao Li" in the student score table (sql3:update stu_score set score=90where stu_name='Xiao Li'). The server 2 can assign a unique request ID, such as 178652986, to the read and write request.

Batch data read and write requests for multiple databases 3, for example, database A contains student table (stu), teacher table (teac), database B contains student score table (stu_score), and now a batch for database A and B is received For data read and write requests, it is necessary to query the data of "Xiao Zhang" in the student table of database A and the data of "Zhang San" in the teacher table (sql1:select*from stu where stu_name='Xiao Zhang'; sql2:select *from teac whereteac_name='Zhang San'), query the data of "Xiao Li" in the student transcript of database B and modify the data of "Xiao Ming" (sql3:select*from stu_score where stu_name='Xiao Li'; sql4:update stu_score setscore=90where stu_name='Xiao Ming'). Similarly, the server 2 can also assign a unique request ID, such as 178652987, to the read and write request.

Step S402, write the read/write request into the hash memory.

Specifically, a storage structure for each of the read and write requests is created in the hash memory, and each of the storage structures includes a key (key) and a value structure. Write the read-write request into the hash memory, wherein the request ID, that is, request_id, is used as the key corresponding to the read-write request in the hash memory, and the caller connection corresponding to the read-write request is marked tcp socket and the read/write request. The read-write sql statement and the read-write sql_id corresponding to each read-write task in the write request are taken as an array and written into the value structure corresponding to the request ID. In addition, the array of the value structure also includes two items, a task return identifier (sql_ret) and a task return data (sql_data), which are used for subsequent writing of the processing result returned for the read and write request.

For example, using the request IDs of the above two read and write requests as the key, the information of the two read and write requests is stored in the hash memory, and the caller connection of the two read and write requests is used to identify, read and write SQL statements respectively. And read and write sql_id as an array, and write it into the value structure of the hash memory corresponding to the respective request ID. Taking the above read and write requests for databases A and B as an example, the request ID "178652987" of the read and write requests is used as the key in the hash memory, and the caller connection is marked to read and write SQL statements (select*from stu where stu_name= 'Xiao Zhang'; select*from teac whereteac_name='Zhang San'; select*from stu_score where stu_name='Xiao Li'; update stu_score set score=90where stu_name='Xiao Ming') and read and write sql_id (sql1, sql2, sql3 , sql4) as an array, write the corresponding value structure.

Step S404, according to the database identifier in the read/write request, send each read/write task of the read/write request to the corresponding database 3 for parallel processing.

Specifically, obtain the request parameters in the read-write request, and send multiple data read-write tasks belonging to different databases 3 (or the same database 3 ) to the corresponding database 3 at the same time according to the database identifier in the request parameters, that is, db_id . The read and write task data sent to each database 3 includes a request ID (request_id), a task ID (task_id, which is consistent with sql_id), read and write SQL statements, and the like. Each database 3 includes a corresponding database access module 30 for receiving and processing read and write requests of the database 3, and then returning the processing result. The returned processing result includes a request ID (request_id), a task ID (task_id, which is consistent with sql_id), a task return identifier (sql_ret), and task return data (sql_data). Among them, the task return identifier is used to indicate whether the SQL statement is executed successfully or not. The return value of 0 means the execution is successful, and non-0 means the execution fails. The task return data is used to indicate the return data corresponding to the sql statement. When the sql statement is a query task and the execution is successful, the specific query result data is returned. When the sql statement is a non-query task or the query execution fails, a null value is returned. null.

For example, for the query task of querying the data of "Xiao Li" in the student's score table (sql3:select*fromstu_score where stu_name='Xiao Li'), the returned processing result in database B is 178652987 for the request ID and 3 for the task ID. The task return identifier is 0, and the task return data is {"stu_name": "Xiao Li", "stu_score": "95"}; for modifying the data of "Xiao Ming" in the student score table (sql4:update stu_score set score=90where stu_name ='Xiao Ming') This non-query task, in the returned processing result, the request ID is 178652987, the task ID is 4, the task return ID is 0, and the task return data is null.

In this embodiment, all read and write tasks in multiple read and write requests can be sent to the corresponding database 3 at the same time, all subtasks are processed in parallel, and the time consumption is the time consumption of one of the largest subtasks.

Step S406, the processing result returned by the database 3 is received and updated into the value structure corresponding to the read and write request.

Specifically, the server 2 asynchronously receives the processing results of each read and write task returned by each database 3 for each of the read and write requests, and obtains the corresponding value structure from the hash memory according to the request ID in each processing result, Then find out the corresponding read/write task in the value structure according to sql_id (consistent with the task ID), and update the task return identifier and task return data of the read/write task.

For example, update the processing result returned by the above database B to the value structure corresponding to the request ID "178652987" in the hash memory.

The data read and write batch processing method provided by this embodiment can provide a unified multi-database read and write task interface. After receiving a batch data read and write request, the unique request ID of each read and write request is used as a key in the hash memory, Save the corresponding information of each read and write task of each read and write request, and send the batch of read and write tasks to the corresponding database for parallel processing, and also write the processing result returned for each read and write task into the hash memory , so that the batch read and write tasks of multiple databases can be processed concurrently without data confusion, and the access time is greatly reduced.

As shown in FIG. 7 , it is a schematic flowchart of the second embodiment of the data read and write batch processing method of the present invention. In this embodiment, steps S500-S506 of the data read/write batch processing method are similar to steps S400-S406 of the first embodiment, the difference is that the method further includes steps S508-S510.

The method includes the following steps:

Step S500, assigning a unique request ID to each read and write request received.

Specifically, the caller 1 initiates a batch data read/write request for a single database 3 or multiple databases 3 to the server 2 in json format. The read-write request includes request parameters such as a database identifier (db_id), a read-write SQL number (sql_id, the sql_id cannot be the same in the same request), and a SQL statement to be executed. The database identifier is used to identify the database access module 30 , for example, A indicates that the database A is accessed through the database access module A. Each read and write SQL number and the corresponding SQL statement represent a read and write task. Each read and write request may include one or more database identification and read and write tasks. The server 2 receives the read and write requests, and assigns a unique request ID, ie, request_id, to each of the read and write requests.

A batch data read and write request for a single database 3, for example, a database has a student table (stu), a teacher table (teac), and a student score table (stu_score), and now a batch data read and write request for this database is received. At the same time, query the data of "Xiao Wang" in the student table (sql1:select*from stu where stu_name='Xiaowang'), and query the data of "Mr. Zhang" in the teacher table (sql2:select*from teac where teac_name='Mr.Zhang' '), update the data of "Xiao Li" in the student score table (sql3:update stu_score set score=90where stu_name='Xiao Li'). The server 2 can assign a unique request ID, such as 178652986, to the read and write request.

Batch data read and write requests for multiple databases 3, for example, database A contains student table (stu), teacher table (teac), database B contains student score table (stu_score), and now a batch for database A and B is received For data read and write requests, it is necessary to query the data of "Xiao Zhang" in the student table of database A and the data of "Zhang San" in the teacher table (sql1:select*from stu where stu_name='Xiao Zhang'; sql2:select *from teac whereteac_name='Zhang San'), query the data of "Xiao Li" in the student transcript of database B and modify the data of "Xiao Ming" (sql3:select*from stu_score where stu_name='Xiao Li'; sql4:update stu_score setscore=90where stu_name='Xiao Ming'). Similarly, the server 2 can also assign a unique request ID, such as 178652987, to the read and write request.

Step S502, write the read/write request into the hash memory.

Specifically, a storage structure for each of the read and write requests is created in the hash memory, and each of the storage structures includes a key (key) and a value structure. Write the read-write request into the hash memory, wherein the request ID, that is, request_id, is used as the key corresponding to the read-write request in the hash memory, and the caller connection corresponding to the read-write request is marked tcp socket and the read/write request. The read-write sql statement and the read-write sql_id corresponding to each read-write task in the write request are taken as an array, and written into the value structure of the hash memory corresponding to the request ID. In addition, the array of the value structure also includes two items, a task return identifier (sql_ret) and a task return data (sql_data), which are used for subsequent writing of the processing result returned for the read and write request.

For example, using the request IDs of the above two read and write requests as the key, the information of the two read and write requests is stored in the hash memory, and the caller connection of the two read and write requests is used to identify, read and write SQL statements respectively. And read and write sql_id as an array, and write it into the value structure of the hash memory corresponding to the respective request ID. Taking the above read and write requests for databases A and B as an example, the request ID "178652987" of the read and write requests is used as the key in the hash memory, and the caller connection is marked to read and write SQL statements (select*from stu where stu_name= 'Xiao Zhang'; select*from teac whereteac_name='Zhang San'; select*from stu_score where stu_name='Xiao Li'; update stu_score set score=90where stu_name='Xiao Ming') and read and write sql_id (sql1, sql2, sql3 , sql4) as an array, write the corresponding value structure.

Step S504, according to the database identifier in the read/write request, each read/write task of the read/write request is sent to the corresponding database 3 for parallel processing.

Specifically, obtain the request parameters in the read-write request, and send multiple data read-write tasks belonging to different databases 3 (or the same database 3 ) to the corresponding database 3 at the same time according to the database identifier in the request parameters, that is, db_id . The read and write task data sent to each database 3 includes a request ID (request_id), a task ID (task_id, which is consistent with sql_id), read and write SQL statements, and the like. Each database 3 includes a corresponding database access module 30 for receiving and processing read and write requests of the database 3, and then returning the processing result. The returned processing result includes a request ID (request_id), a task ID (task_id, which is consistent with sql_id), a task return identifier (sql_ret), and task return data (sql_data). Among them, the task return identifier is used to indicate whether the SQL statement is executed successfully or not. The return value of 0 means the execution is successful, and non-0 means the execution fails. The task return data is used to indicate the return data corresponding to the sql statement. When the sql statement is a query task and the execution is successful, the specific query result data is returned. When the sql statement is a non-query task or the query execution fails, a null value is returned. null.

For example, for the query task of querying the data of "Xiao Li" in the student's score table (sql3:select*fromstu_score where stu_name='Xiao Li'), the returned processing result in database B is 178652987 for the request ID and 3 for the task ID. The task return identifier is 0, and the task return data is {"stu_name": "Xiao Li", "stu_score": "95"}; for modifying the data of "Xiao Ming" in the student score table (sql4:update stu_score set score=90where stu_name ='Xiao Ming') This non-query task, in the returned processing result, the request ID is 178652987, the task ID is 4, the task return ID is 0, and the task return data is null.

In this embodiment, all read and write tasks in multiple read and write requests can be sent to the corresponding database 3 at the same time, all subtasks are processed in parallel, and the time consumption is the time consumption of one of the largest subtasks.

Step S506, the processing result returned by the database 3 is received and updated into the value structure corresponding to the read and write request.

Specifically, the server 2 asynchronously receives the processing results of each read and write task returned by each database 3 for each of the read and write requests, and obtains the corresponding value structure from the hash memory according to the request ID in each processing result, Then find out the corresponding read/write task in the value structure according to sql_id (consistent with the task ID), and update the task return identifier and task return data of the read/write task.

For example, update the processing result returned by the above database B to the value structure corresponding to the request ID "178652987" in the hash memory.

Step S508, judging whether the read and write tasks of the read and write request have all returned processing results.

Specifically, for each read/write request, it is determined whether all read/write tasks in the corresponding value structure have returned processing results. If not all are returned, continue to wait for the corresponding database 3 to return the processing result, until all the read-write tasks of the read-write request return the processing result.

It is worth noting that, in this embodiment, when the read and write tasks of the read and write requests have not all returned processing results and need to continue to wait, other read and write requests can be processed at the same time, that is, other read and write requests do not need to wait in line, Does not affect the processing time of other read and write requests.

Step S510 , when all the read/write tasks of the read/write request have returned processing results, combine the processing results and return them to the caller 1 .

Specifically, if all the read/write tasks of the read/write request have returned processing results, the caller connection flag in the corresponding value structure is obtained, and the processing results of all read/write tasks of the read/write request are combined. The result is returned to caller 1. The combined processing result returned to the caller 1 includes database ID (db_id), read/write sql_id, task return identifier (sql_ret), task return data (sql_data) and other data.

For example, for the read/write request whose request ID is 178652987, determine the four read/write tasks in the corresponding value structure (sql1:select*from stu where stu_name='Xiao Zhang'; sql2:select*from teac where teac_name='Zhang San' '; sql3:select*from stu_score where stu_name='Xiao Li';sql4:update stu_score set score=90where stu_name='Xiao Ming') have all been returned, that is, the task return identifier and task return data corresponding to these four task IDs Is it updated. If not all are returned, for example, only database B returns but database A has not returned, continue to wait for the processing result returned by database A (other read and write requests can be processed at the same time). If all have been returned, the processing results of all four tasks are combined and returned to the caller 1.

It is worth noting that, each step of the data read/write batch processing method and the database access module 30 of each database 3 can be configured with multiple processes for synchronous processing. This embodiment processes read and write requests for a single database 3 or multiple databases 3 in batches, and can process multiple different read and write requests at the same time, that is, multiple read requests and multiple write requests of a single or multiple databases 3 can be mixed Batch and concurrent processing together.

The data read and write batch processing method provided by this embodiment can provide a unified multi-database read and write task interface, and the batch read and write tasks are simultaneously sent to the corresponding database for parallel processing, with high concurrency capability and greatly reduced access time. The caller only needs to assemble the read and write task list of the database to be accessed according to the unified interface, and then perform logical processing according to the processing result data returned by the unified interface, so as to achieve the effect of abstract isolation between the business layer and the data access layer, and improve production efficiency. In addition, the reading and writing of this embodiment does not depend on the specific table, and the fields are abstractly obtained, which supports dynamic expansion of the table without modifying the code, and improves the production capacity. In addition, this embodiment supports distributed deployment and configuration of multi-process processing, greatly utilizes CPU multi-core hardware resources, and improves processing capability.

Another embodiment of the present invention is to provide a computer-readable storage medium, where the computer-readable storage medium stores a data read and write batch processing program, and the data read and write batch processing program can be processed by at least one The processor executes, so that the at least one processor executes the steps of the above-mentioned data read and write batch processing method.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields , are similarly included in the scope of patent protection of the present invention.

Claims (10)

1. a data read-write batch processing method, is characterized in that, described method comprises the steps: When one or more batch data read/write requests for a single database or multiple databases are received from the caller, each read/write request is assigned a unique request ID, and the request parameters in the read/write request include the database identifier and Read and write tasks, where each read and write task includes a read and write SQL number and the read and write SQL statements to be executed; Create a storage structure for each read and write request in the hash memory, including a keyword and a value structure, write the request ID into the keyword, and write the caller corresponding to each read and write request The connection sign, and the read-write sql statement and the read-write sql number corresponding to each read-write task in the read-write request are written into the value structure as an array; According to the database identifier in each read and write request, each read and write task of the read and write request is sent to the corresponding database for parallel processing; and The processing result of each read and write task for each of the read and write requests returned by the database is received and updated into the value structure corresponding to each of the read and write requests. 2. data read-write batch processing method as claimed in claim 1, is characterized in that, this method also comprises the step: According to the value structure, determine whether the read and write tasks of the read and write request have all returned processing results; When the read-write tasks of the read-write request have all returned processing results, the processing results of all the read-write tasks of the read-write request are combined and returned to the caller. 3. data read-write batch processing method as claimed in claim 1 or 2 is characterized in that, in described according to the database identification in each described read-write request, each read-write task of described read-write request is sent to In the corresponding database parallel processing step, the read/write task data sent to each database includes a request ID, a read/write SQL number, and a read/write SQL statement. 4. The method for batch processing of data reading and writing according to claim 1 or 2, wherein the processing result comprises a request ID, a reading and writing sql number, a task return identifier and task return data. 5. data read-write batch processing method as claimed in claim 4, is characterized in that, in described value structure, also comprises the task return identification and the task return data corresponding to each read-write task, the described receiving database returns for all. The steps of updating the processing results of each read-write task of the read-write request to the value structure corresponding to the read-write request include: Asynchronously receive the processing results returned by each database, obtain the corresponding value structure from the hash memory according to the request ID in each processing result, and then find out the corresponding read and write tasks in the value structure according to the read and write SQL numbers, and update the The task return identifier and the task return data of the read and write tasks. 6. The data read-write batch processing method according to claim 5, wherein the task return identifier is used to indicate whether the read-write sql statement is executed successfully or not, and a return value of 0 indicates that the execution is successful, and non-0 indicates that the execution fails. 7. data read-write batch processing method as claimed in claim 5 is characterized in that, described task return data is used to mark the return data corresponding to read-write sql statement, when described read-write sql statement is query task and executes successfully When the query result data is returned; when the read-write sql statement is a non-query task or the query task fails to execute, the null value null is returned. 8. data read-write batch processing method as claimed in claim 2, is characterized in that, this method also comprises the step: When the read and write tasks of the read and write requests do not all return processing results and need to continue to wait, other read and write requests continue to be processed at the same time. 9. A server, characterized in that the server comprises a memory and a processor, and the memory stores a data read/write batch processing system that can run on the processor, and the data read/write batch processing system is The processor implements the steps of the data read and write batch processing method according to any one of claims 1-8 when executed. 10. A computer-readable storage medium, wherein the computer-readable storage medium stores a data read-write batch processing system, and the data read-write batch processing system can be executed by at least one processor, so that the At least one processor executes the steps of the data read and write batch processing method according to any one of claims 1-8.

Images