CN117633111A - ClickHouse-based request processing method, device, equipment and medium - Google Patents

Abstract

The disclosure provides a request processing method, device, equipment and medium based on ClickHouse, which can be applied to the technical field of big data. The method comprises the following steps: obtaining M copies of a ClickHouse, obtaining a first copy group and a second copy group based on the M copies, wherein N copies in the first copy group are used for processing read requests, P copies in the second copy group are used for processing write requests, M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and M=N+P; acquiring a request and determining the type of the request; acquiring read-write separation parameters; and responding to a first instruction generated based on the read-write separation parameter, and correspondingly forwarding the request to one copy in the first copy group or one copy in the second copy group for processing based on the type of the request, wherein the first instruction is used for indicating that the separation processing of the read request and the write request is performed.

Description

ClickHouse-based request processing method, device, equipment and medium

Technical Field

The disclosure relates to the field of big data, in particular to a request processing method, device, equipment and medium based on ClickHouse.

Background

At present, when a technician processes a user's request, the user's request is generally randomly forwarded to a replica cluster for processing, so that when a large number of concurrent requests occur, some replica processing requests are excessive, and some replicas are still in an idle state, so that the performance of the replica cluster cannot be fully utilized. Meanwhile, because of the diversity of the requests, the same copy can process both read requests and write requests, overload of the copy can occur in the process of processing the requests, thereby increasing the management and maintenance cost of the copy cluster, and causing the performance of the whole system to be reduced, so that the experience of a user is affected.

Disclosure of Invention

In view of the foregoing, the present disclosure provides a clickHouse-based method, apparatus, device, and medium.

According to a first aspect of the present disclosure, there is provided a clickHouse-based request processing method, including: obtaining M copies of the ClickHouse, wherein each copy is used for processing a read request or a write request; obtaining a first copy group and a second copy group based on M copies, wherein the first copy group comprises N copies, N copies in the first copy group are used for processing read requests, the second copy group comprises P copies, P copies in the second copy group are used for processing write requests, M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and M=N+P; acquiring a request, and determining the type of the request, wherein the type of the request comprises a read request or a write request; acquiring a read-write separation parameter, wherein the read-write separation parameter is used for indicating whether to perform separation processing of a read request and a write request; and responding to a first instruction generated based on the read-write separation parameter, and correspondingly forwarding the request to one copy in the first copy group or one copy in the second copy group for processing based on the type of the request, wherein the first instruction is used for indicating that the separation processing of the read request and the write request is performed.

According to an embodiment of the present disclosure, the clickHouse-based request processing method further includes: and responding to a second instruction generated based on the read-write separation parameter, and forwarding the request to any one copy of the M copies for processing, wherein the second instruction is used for indicating that the separation processing of the read request and the write request is not performed.

According to an embodiment of the present disclosure, obtaining a first replica group and a second replica group based on M replicas includes: setting a mapping relation for each copy in the M copies, wherein the mapping relation comprises a first mapping relation and a second mapping relation, the copy with the first mapping relation is used for processing a read request, and the copy with the second mapping relation is used for processing a write request; obtaining N copies with a first mapping relation and P copies with a second mapping relation based on the mapping relation of each copy in the M copies; and taking N copies with the first mapping relation as a first copy group, and taking P copies with the second mapping relation as a second copy group.

According to an embodiment of the present disclosure, setting a mapping relationship for each of the M copies includes: acquiring a sequence number, wherein the sequence number comprises a first sequence number and a second sequence number; mapping each copy in the M copies with a sequence number to obtain M mapping relations, wherein N copies in the M copies are mapped with first sequence numbers to obtain N first mapping relations, and the first mapping relations indicate that the copy with the first sequence numbers is used for processing a read request; and mapping P copies in the M copies to second sequence numbers to obtain N second mapping relations, wherein the second mapping relations indicate that the copies mapped with the second sequence numbers are used for processing write requests.

According to an embodiment of the present disclosure, determining the type of request specifically includes: acquiring keywords of a read request; and determining the type of the request based on the keywords of the read request. According to an embodiment of the present disclosure, determining the type of request based on the keywords of the read request includes: acquiring a request head and a request body of a request, and determining whether keywords of a read request exist in the request head and/or the request body; and determining the request as a read request in the case that the keyword of the read request exists in the request header and/or the request body, and otherwise determining the request as a write request.

According to an embodiment of the present disclosure, a read-write separation parameter is stored in a memory, and the read-write separation parameter is used to indicate whether to perform separation processing of a read request and a write request, including: determining a parameter value of a read-write separation parameter, wherein the parameter value of the read-write separation parameter comprises a first parameter value or a second parameter value; obtaining a first instruction based on the first parameter value, wherein the first instruction is used for indicating separation processing of a read request and a write request; or deriving a second instruction based on the second parameter value, the second instruction indicating that no separate processing of the read request and the write request is performed.

According to an embodiment of the present disclosure, forwarding a request to one of the first group of replicas or one of the second group of replicas, respectively, based on a type of the request, includes: obtaining M mapping relations of M copies; acquiring N copies for processing the read request and P copies for processing the write request based on M mapping relations of the M copies; selecting one copy of the N copies for processing the read request or one copy of the P copies for processing the write request based on a copy selection algorithm according to the type of the request; the request is forwarded to one of the copies selected based on the copy selection algorithm for processing.

According to an embodiment of the disclosure, forwarding the request to any one of the M copies for processing in response to a second instruction generated based on the read-write separation parameter includes: selecting any one of the M copies based on a copy selection algorithm; the request is forwarded to one of the copies selected based on the copy selection algorithm for processing.

A second aspect of the present disclosure provides a ClickHouse-based request processing apparatus, comprising: the system comprises a loading copy module, a copy processing module and a copy processing module, wherein the loading copy module is used for acquiring M copies of a ClickHouse, and each copy is used for processing a read request or a write request; the copy distinguishing module is used for obtaining a first copy group and a second copy group based on M copies, the first copy group comprises N copies, the N copies in the first copy group are used for processing read requests, the second copy group comprises P copies, the P copies in the second copy group are used for processing write requests, M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and M=N+P; the acquisition request module is used for acquiring a request and determining the type of the request, wherein the type of the request comprises a read request or a write request; the separation parameter module is used for acquiring a read-write separation parameter, wherein the read-write separation parameter is used for indicating whether to perform separation processing of a read request and a write request; and a request processing module, configured to respond to a first instruction generated based on the read-write separation parameter, and forward the request to one copy in the first copy group or one copy in the second copy group for processing based on the type of the request, where the first instruction is used to instruct separation processing of the read request and the write request.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method described above.

A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described method.

According to the embodiment of the disclosure, read-write separation processing of the request is realized by distinguishing the purpose of the copy in the ClickHouse and forwarding the request to the corresponding copy for processing by determining the type of the request. And the inquiry load can be evenly distributed to different copies through the read-write separation processing of the requests, so that the read-write requests are not interfered with each other, the efficiency of processing the requests by the copies is improved, and the overall performance and throughput of the system are further improved.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a ClickHouse-based request processing method, apparatus, device and storage medium according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a ClickHouse-based request processing method in accordance with an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a method of partitioning a first replica group and a second replica group in accordance with an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a method of determining a request type according to an embodiment of the disclosure;

FIG. 5 schematically illustrates a flow chart of a method of forwarding a request to a corresponding copy according to an embodiment of the disclosure;

FIG. 6 schematically illustrates a block diagram of a ClickHouse-based request processing apparatus in accordance with an embodiment of the present disclosure; and

fig. 7 schematically illustrates a block diagram of an electronic device adapted to implement a ClickHouse-based request processing method in accordance with an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the technical scheme of the disclosure, the related data (such as including but not limited to personal information of a user) are collected, stored, used, processed, transmitted, provided, disclosed, applied and the like, all conform to the regulations of related laws and regulations, necessary security measures are adopted, and the public welcome is not violated.

The embodiment of the disclosure provides a request processing method based on ClickHouse, which comprises the following steps: obtaining M copies of the ClickHouse, wherein each copy is used for processing a read request or a write request; obtaining a first copy group and a second copy group based on M copies, wherein the first copy group comprises N copies, N copies in the first copy group are used for processing read requests, the second copy group comprises P copies, P copies in the second copy group are used for processing write requests, M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and M=N+P; acquiring a request, and determining the type of the request, wherein the type of the request comprises a read request or a write request; acquiring a read-write separation parameter, wherein the read-write separation parameter is used for indicating whether to perform separation processing of a read request and a write request; and responding to a first instruction generated based on the read-write separation parameter, and correspondingly forwarding the request to one copy in the first copy group or one copy in the second copy group for processing based on the type of the request, wherein the first instruction is used for indicating that the separation processing of the read request and the write request is performed.

As shown in fig. 1, an application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium used to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 through the network 104 using at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, to receive or send messages, etc. Various communication client applications, such as a shopping class application, a web browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only) may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

The first terminal device 101, the second terminal device 102, the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the clickHouse-based request processing method provided by the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the clickHouse-based request processing apparatus provided by embodiments of the present disclosure may be generally disposed in the server 105. The clickHouse-based request processing method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105. Accordingly, the clickHouse-based request processing apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The clickHouse-based request processing method of the disclosed embodiment will be described in detail below with reference to the scenario described in FIG. 1, by way of FIGS. 2-5.

Fig. 2 schematically illustrates a flow chart of a ClickHouse-based request processing method in accordance with an embodiment of the present disclosure.

As shown in fig. 2, the ClickHouse-based request processing method of this embodiment includes operations S210 to S250.

In operation S210, M copies of the ClickHouse are obtained, where each copy is used to process a read request or a write request. According to an embodiment of the present disclosure, the clickHouse includes M copies, where M is greater than or equal to 2, each copy for processing a read request or a write request, where a read request may be understood as reading data, such as a query request including data, and a write request may be understood as modifying data, such as including insertion, update, and deletion requests of data.

According to embodiments of the present disclosure, a ClickHouse is an open-sourced, columnar, distributed database management system (DBMS) that supports real-time data processing and streaming computing, including for receiving and processing data streams, and performing aggregation, computation, and querying in real-time to meet the needs of real-time data analysis and decision making, where a ClickHouse stores data on a sliced basis, each slice having multiple copies (replica).

According to the embodiment of the disclosure, the copies in the ClickHouse are distinguished and divided into the first copy group and the second copy group, the first copy group and the second copy group are respectively used for processing the read request and the write request, and the types of the copies are determined, when an instruction for starting the read-write separation is acquired, the request is forwarded to the corresponding copy in the first copy group or the second copy group for processing according to the types of the copies, so that the read-write separation is realized.

According to an embodiment of the present disclosure, read/Write Splitting (Read/Write Splitting) is a policy applied in a database system to improve performance and scalability of the system by processing Read requests and Write requests separately.

In operation S220, a first copy group and a second copy group are obtained based on the M copies.

According to an embodiment of the disclosure, the first copy group includes N copies, the N copies in the first copy group are used for processing read requests, the second copy group includes P copies, the P copies in the second copy group are used for processing write requests, where N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and m=n+p.

For example, 10 copies are included in the clickHouse, any 5 of which are used to process the read request, and the remaining 5 copies are used to process the write request, i.e., the first copy group is the 5 copies used to process the read request and the second copy group is the 5 copies used to process the write request.

Fig. 3 schematically illustrates a flow chart of a method of partitioning a first replica group and a second replica group according to an embodiment of the present disclosure.

As shown in fig. 3, the method of dividing the first copy group and the second copy group according to the embodiment includes operations S310 to S330.

In operation S310, a mapping relationship is set for each of the M copies.

According to an embodiment of the present disclosure, setting a mapping relationship for each of the M copies includes: acquiring a sequence number, wherein the sequence number comprises a first sequence number and a second sequence number; and mapping each copy in the M copies with a sequence number to obtain M mapping relations.

For example, the first sequence number is 1, the second sequence number is 2, and the M copies are mapped with 1 or 2 in sequence, so as to obtain M mapping relations, that is, each of the M copies corresponds to one sequence number, and the sequence number is 1 or 2.

According to an embodiment of the present disclosure, the mapping relationship includes a first mapping relationship and a second mapping relationship, a copy having the first mapping relationship is used to process the read request, and a copy having the second mapping relationship is used to process the write request.

According to the embodiment of the disclosure, mapping N copies in M copies to first sequence numbers to obtain N first mapping relations, wherein the first mapping relations indicate that the copy mapped to the first sequence numbers is used for processing a read request; and mapping P copies in the M copies to second sequence numbers to obtain N second mapping relations, wherein the second mapping relations indicate that the copies mapped with the second sequence numbers are used for processing write requests.

For example, if 10 copies are included in the clickHouse, any 5 copies of the 10 copies are mapped to the first sequence number 1, then the copy mapped with the sequence number 1 is indicated to be used for processing the read request, and the remaining 5 copies are mapped to the second sequence number 2, and the copy mapped with the sequence number 2 is indicated to be used for processing the write request.

According to the embodiment of the disclosure, the copies are mapped through the sequence numbers and the purpose of the copies is determined through the mapping relationship, so that the copies can be distinguished simply, conveniently and effectively, and meanwhile, in the process of forwarding the request to the corresponding copies, the corresponding copies can be searched directly through the mapping relationship, so that the efficiency of managing the copies can be improved.

In operation S320, N copies having the first mapping relationship and P copies having the second mapping relationship are obtained based on the mapping relationship of each of the M copies.

According to an embodiment of the present disclosure, the mapping relationship and usefulness each copy has is determined by determining the sequence number mapped by the copy, e.g., one copy has a first mapping relationship and is used to process read requests if the sequence number mapped by the copy is 1.

In operation S330, N copies having the first mapping relationship are used as the first copy group, and P copies having the second mapping relationship are used as the second copy group.

According to the embodiment of the disclosure, the numbers mapped by the N copies in the first copy group are all 1, and the numbers mapped by the P copies in the second copy group are all 2.

According to the embodiment of the disclosure, the copy is divided into the first copy group for performing the read request and the second copy group for performing the write request, so that the copy group can be better managed by the system, and the copy nodes in the first copy group or the second copy group can be flexibly added or deleted according to the load conditions of the read request and the write request, so that the maintenance of the system is more flexible and controllable, and the utilization and management of system resources are optimized.

According to the embodiment of the disclosure, the mapping relation of the M copies is encapsulated into one object, and the object is used for inquiring the copy corresponding to the request in the subsequent request forwarding process, so that convenience is brought to the processing of the request.

Referring back to fig. 1, in operation S230, a request is acquired and a type of the request is determined, the type of the request including a read request or a write request.

Fig. 4 schematically illustrates a flow chart of a method of determining a request type according to an embodiment of the disclosure.

As shown in fig. 4, the method of determining a request type of this embodiment includes operations S410 to S420.

In operation S410, a keyword of a read request is acquired.

According to an embodiment of the present disclosure, the key word of the read request is a necessary word contained in the read request code, such as select.

In operation S420, the type of request is determined based on the keywords of the read request.

According to the embodiment of the disclosure, the type of the request is determined based on the keyword of the request, so that the request can be quickly and accurately classified and forwarded according to the type of the request.

According to an embodiment of the present disclosure, determining the type of the request based on the keyword of the read request specifically includes: acquiring a request head and a request body of a request, and determining whether keywords of a read request exist in the request head and/or the request body; and determining the request as a read request in the case that the keyword of the read request exists in the request header and/or the request body, and otherwise determining the request as a write request.

For example, a request includes a request header (header) and a request body (body), and neither the request header (header) nor the request body (body) contains a keyword select, the request is determined to be a write request.

According to the embodiment of the disclosure, the determination of the type of the request is a precondition for the subsequent dynamic read-write separation processing of the request, so that the performance and efficiency of the system for processing the request can be correspondingly improved by effectively distinguishing the requests.

Referring back to fig. 1, in operation S240, a read-write separation parameter indicating whether to perform separation processing of a read request and a write request is acquired

According to an embodiment of the disclosure, a read-write separation parameter is stored in a memory, and a parameter value of the read-write separation parameter includes a first parameter value or a second parameter value, wherein a first instruction is obtained based on the first parameter value, and the first instruction is used for indicating separation processing of a read request and a write request; or deriving a second instruction based on the second parameter value, the second instruction indicating that no separate processing of the read request and the write request is performed.

For example, when the separation process of the read-write request needs to be started, the parameter value of the read-write separation parameter in the memory is set as the hole, the read-write separation parameter in the memory is read by the system to obtain a first instruction, and when the separation process of the read-write request needs to be closed, the parameter value of the read-write separation parameter in the memory is set as the false, and the read-write separation parameter in the memory is read by the system to obtain a second instruction.

According to the embodiment of the disclosure, the read-write separation parameters are set and stored in the memory, so that the mode of processing the request by the system can be dynamically adjusted in real time according to the actual situation, the resource of the system can be utilized to the maximum, the request processing efficiency is improved, and the user experience is improved.

In operation S250, in response to the first instruction generated based on the read-write separation parameter, the request is forwarded to one of the first copy group or one of the second copy group for processing based on the type of the request, respectively.

Fig. 5 schematically illustrates a flow chart of a method of forwarding a request to a corresponding copy according to an embodiment of the disclosure.

As shown in fig. 5, the method for forwarding the request to the corresponding copy of this embodiment includes operations S510 to S530.

In operation S510, obtaining M mapping relations of M copies; and acquiring N copies for processing the read request and P copies for processing the write request based on M mapping relations of the M copies.

According to the embodiment of the disclosure, M mapping relations of M copies are packaged into an object, the object comprises M copies and M mapping relations of the copies, the sequence number mapped by each copy is determined according to the mapping relation of each copy, and the copy is determined to be used for a read request or a write request according to the sequence number corresponding to the copy.

In operation S520, one of N copies for processing the read request or one of P copies for processing the write request is selected based on the copy selection algorithm according to the type of request.

According to the embodiment of the disclosure, a corresponding first copy cluster or second copy cluster in the object is selected according to the type of the request, the first copy cluster or the second copy cluster is traversed based on a copy selection algorithm, and one copy with smaller current load is selected.

For example, if the type of a request is determined to be a read request, all copies mapped with the first sequence number in the object are screened out to obtain a first copy group, then a low-load copy is selected based on a list of the first copy group traversed by a copy selection algorithm.

In operation S530, the request is forwarded to one copy selected based on the copy selection algorithm for processing.

According to the embodiment of the disclosure, the requests are distributed to the special copies according to the types of the requests, so that conflicts and synchronous delays among a plurality of copies can be avoided, the response speed and processing capacity of request processing can be effectively improved, meanwhile, the copy selection algorithm can select the copy with lower copy load to process the requests, so that the load of a copy cluster layer is balanced, when a certain copy fails, the problem copy can be avoided, and the stability of the system is greatly improved.

According to the embodiment of the disclosure, when the separation processing of the read-write request is closed, the request is forwarded to any one copy of the M copies for processing in response to a second instruction generated based on the read-write separation parameter, wherein the second instruction is used for indicating that the separation processing of the read request and the write request is not performed.

According to the embodiment of the disclosure, the read-write separation switch can be dynamically controlled in real time according to actual scene requirements, so that the performance of the system can be flexibly managed and utilized.

According to an embodiment of the present disclosure, when the separation processing of the read-write request is closed, the processing of the request includes: selecting any one of the M copies based on a copy selection algorithm;

the request is forwarded to one of the copies selected based on the copy selection algorithm for processing.

For example, when the separation processing of the read-write request is required to be closed currently, the parameter value of the read-write separation parameter is set to be a second parameter value, at this time, when the request is processed, the object encapsulated by the M copies is traversed directly through the copy selection algorithm, and a copy with smaller current load is selected to process the request.

According to the embodiment of the disclosure, the parameter values of the read-write separation parameters can be intelligently adjusted according to the real-time copy load condition, so that the processing mode of the request can be changed, and further, a proper copy is selected for processing the request based on a copy selection algorithm, so that the resource utilization rate of the system can be maximized.

Based on the above-mentioned request processing method based on the ClickHouse, the present disclosure also provides a request processing device based on the ClickHouse. The device will be described in detail below in connection with fig. 6.

Fig. 6 schematically illustrates a block diagram of a ClickHouse-based request processing apparatus according to an embodiment of the present disclosure.

As shown in FIG. 6, the ClickHouse-based request processing apparatus 600 of this embodiment includes a load copy module 610, a differentiate copy module 620, an acquire request module 630, a separate parameters module 640, and a request processing module 650.

The load copy module 610 is configured to obtain M copies of a ClickHouse, where each copy is configured to process a read request or a write request. In an embodiment, the loading copy module 610 may be configured to perform the operation S210 described above, which is not described herein.

The distinguishing copy module 620 is configured to obtain a first copy group and a second copy group based on M copies, where the first copy group includes N copies, the N copies in the first copy group are used for processing a read request, the second copy group includes P copies, the P copies in the second copy group are used for processing a write request, where M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and m=n+p. In an embodiment, the copy distinguishing module 620 may be used to perform the operation S220 described above, which is not described herein.

The get request module 630 is configured to get a request and determine a type of the request, where the type of the request includes a read request or a write request. In an embodiment, the acquisition request module 630 may be used to perform the operation S230 described above, which is not described herein.

In accordance with an embodiment of the present disclosure, the separation parameter module 640 is configured to obtain a read-write separation parameter, where the read-write separation parameter is used to indicate whether to perform separation processing of a read request and a write request, and in an embodiment, the separation parameter module 640 may be configured to perform the operation S240 described above, which is not described herein.

According to an embodiment of the present disclosure, the request processing module 650 is configured to, in response to a first instruction generated based on the read-write separation parameter, forward the request to one of the first copy group or one of the second copy group for processing based on a type of the request, where the first instruction is configured to instruct separation processing of the read request and the write request. In an embodiment, the request processing module 650 may be configured to perform the operation S350 described above, which is not described herein.

Any of the multiple modules of the loading copy module 610, the distinguishing copy module 620, the obtaining request module 630, the separating parameter module 640, and the request processing module 650 may be combined in one module or any of the multiple modules may be split into multiple modules according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the loading copy module 610, the distinguishing copy module 620, the obtaining request module 630, the separation parameter module 640, and the request processing module 650 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware. Alternatively, at least one of the loading copy module 610, the distinguishing copy module 620, the obtaining request module 630, the separating parameter module 640, and the request processing module 650 may be at least partially implemented as a computer program module, which when executed, may perform the corresponding functions.

Fig. 7 schematically illustrates a block diagram of an electronic device adapted to implement a ClickHouse-based request processing method in accordance with an embodiment of the present disclosure.

As shown in fig. 7, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. The processor 701 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 701 may also include on-board memory for caching purposes. The processor 701 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.

In the RAM 703, various programs and data necessary for the operation of the electronic apparatus 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The processor 701 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 702 and/or the RAM 703. Note that the program may be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

According to an embodiment of the present disclosure, the electronic device 700 may further include an input/output (I/O) interface 705, the input/output (I/O) interface 705 also being connected to the bus 704. The electronic device 700 may also include one or more of the following components connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 702 and/or RAM 703 and/or one or more memories other than ROM 702 and RAM 703 described above.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 701. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed over a network medium in the form of signals, downloaded and installed via the communication section 709, and/or installed from the removable medium 711. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 701. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims (12)

1. A ClickHouse-based request processing method, comprising:

obtaining M copies of the ClickHouse, wherein each copy is used for processing a read request or a write request;

obtaining a first copy group and a second copy group based on the M copies, wherein the first copy group includes N copies, the N copies in the first copy group are used for processing a read request, the second copy group includes P copies, and the P copies in the second copy group are used for processing a write request, where M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and m=n+p;

Acquiring a request and determining the type of the request, wherein the type of the request comprises a read request or a write request;

acquiring a read-write separation parameter, wherein the read-write separation parameter is used for indicating whether to perform separation processing of a read request and a write request; and

in response to a first instruction generated based on the read-write separation parameter, forwarding the request to one of the copies in the first copy group or one of the copies in the second copy group for processing based on the type of the request, wherein the first instruction is used for indicating that separation processing of the read request and the write request is performed.

2. The method as recited in claim 1, further comprising:

and responding to a second instruction generated based on the read-write separation parameter, and forwarding the request to any copy in M copies for processing, wherein the second instruction is used for indicating that the separation processing of the read request and the write request is not performed.

3. The method of claim 1, wherein the deriving a first copy group and a second copy group based on the M copies comprises:

setting a mapping relation for each copy in the M copies, wherein the mapping relation comprises a first mapping relation and a second mapping relation, the copy with the first mapping relation is used for processing a read request, and the copy with the second mapping relation is used for processing a write request;

Obtaining N copies with the first mapping relation and P copies with the second mapping relation based on the mapping relation of each copy in the M copies;

and taking the N copies with the first mapping relation as a first copy group, and taking the P copies with the second mapping relation as a second copy group.

4. A method according to claim 1 or 3, wherein said setting a mapping relationship for each of said M copies comprises:

acquiring a sequence number, wherein the sequence number comprises a first sequence number and a second sequence number;

mapping each of the M copies to one of the sequence numbers to obtain M mapping relations, wherein,

mapping N copies in the M copies to the first sequence numbers to obtain N first mapping relations, wherein the first mapping relations indicate that the copies mapped to the first sequence numbers are used for processing read requests;

and mapping the P copies in the M copies to the second sequence numbers to obtain N second mapping relations, wherein the second mapping relations indicate that the copies mapped with the second sequence numbers are used for processing write requests.

5. The method according to claim 1, wherein said determining the type of the request comprises in particular:

acquiring keywords of the read request; and

the type of the request is determined based on the keywords of the read request.

6. The method of claim 5, wherein the determining the type of the request based on the read request's keywords comprises:

acquire a request header and a request body of the request,

determining whether keywords of the read request exist in the request header and/or the request body; and

and determining the request as a read request under the condition that the keyword of the read request exists in the request head and/or the request body, and otherwise, determining the request as a write request.

7. The method according to claim 1 or 2, wherein the read-write separation parameter is stored in a memory, and the read-write separation parameter is used for indicating whether to perform separation processing of a read request and a write request, and includes:

determining a parameter value of the read-write separation parameter, wherein the parameter value of the read-write separation parameter comprises a first parameter value or a second parameter value;

obtaining the first instruction based on the first parameter value, wherein the first instruction is used for indicating to perform separation processing of the read request and the write request; or alternatively

And obtaining the second instruction based on the second parameter value, wherein the second instruction is used for indicating that the separation processing of the read request and the write request is not performed.

8. A method according to claim 1 or 3, the processing of the request to one of the copies in the first copy group or one of the copies in the second copy group based on the type of the request, respectively, comprising:

obtaining M mapping relations of the M copies; acquiring N copies for processing the read request and P copies for processing the write request based on M mapping relations of the M copies;

selecting one copy of the N copies for processing read requests or one copy of the P copies for processing write requests based on a copy selection algorithm according to the type of the request;

the request is forwarded to a copy selected based on the copy selection algorithm for processing.

9. The method of claim 2 or 8, wherein forwarding the request to any one of the M copies for processing in response to a second instruction generated based on the read-write separation parameter comprises:

Selecting any one of the M copies based on the copy selection algorithm;

forwarding the request to one copy selected based on the copy selection algorithm for processing.

10. A ClickHouse-based request processing apparatus comprising:

the copy loading module is used for acquiring M copies of the ClickHouse, wherein each copy is used for processing a read request or a write request;

the copy distinguishing module is used for obtaining a first copy group and a second copy group based on the M copies, wherein the first copy group comprises N copies, the N copies in the first copy group are used for processing read requests, the second copy group comprises P copies, the P copies in the second copy group are used for processing write requests, M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, P is an integer greater than or equal to 1, and M=N+P;

the system comprises an acquisition request module, a storage module and a storage module, wherein the acquisition request module is used for acquiring a request and determining the type of the request, and the type of the request comprises a read request or a write request;

the separation parameter module is used for acquiring a read-write separation parameter, and the read-write separation parameter is used for indicating whether separation processing of a read request and a write request is performed or not; and

And the request processing module is used for responding to a first instruction generated based on the read-write separation parameter, and correspondingly forwarding the request to one copy in the first copy group or one copy in the second copy group for processing based on the type of the request, wherein the first instruction is used for indicating that the read request and the write request are separated.

11. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-9.

12. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1 to 9.