CN116610748A - Data writing method, terminal equipment and computer readable storage medium - Google Patents

Abstract

The application discloses a data writing method, terminal equipment and a computer readable storage medium, wherein the data writing method comprises the following steps: acquiring a plurality of data to be written from a data source, wherein the data to be written comprises attribute information with unique identification; distributing a plurality of data to be written into corresponding data buffer queues according to the attribute information; and responding to the writing request, and sending the data to be written of the data buffer queue to the corresponding data node. The data writing method of the application distributes a plurality of data to be written into corresponding data buffer queues according to attribute information, so that the data to be written in the data buffer queues can be sent to corresponding data nodes, the writing of the data to be written is only related to the corresponding data nodes, so that the distributed long tail effect is avoided, the cluster resources are fully utilized, and the data writing efficiency is improved.

Description

A data writing method, terminal device and computer-readable storage medium

technical field

The present application relates to the technical field of data processing, and in particular to a data writing method, a terminal device, and a computer-readable storage medium.

Background technique

With the advent of the big data era, the amount of business data is increasing, and the amount of data imported or exported between databases is also increasing. In the context of massive data and high concurrency, how to achieve high-speed and secure data interaction has become a top priority.

When the existing data writing methods are written to the distributed nodes of the cluster, because they cannot perceive the index status of the cluster, there is a distributed long-tail effect. When processing batch write requests, the write performance is limited by the load The highest node is not conducive to the improvement of cluster write performance, so that cluster resources cannot be fully utilized.

Contents of the invention

In order to solve the above technical problems, the present application provides a data writing method, a terminal device, and a computer-readable storage medium.

In order to solve the above problems, the present application provides a first technical solution: providing a data writing method, including: obtaining a number of data to be written from a data source, the data to be written includes attribute information with a unique identifier; according to the Allocate the data to be written to the corresponding data buffer queue according to the attribute information; and send the data to be written in the data buffer queue to the corresponding data node in response to the write request.

Wherein, after the step of obtaining a plurality of data to be written from the data source, and the data to be written includes attribute information with a unique identifier, the data writing method further includes: transmitting the plurality of data to be written to thread pool; obtaining the load status of several data nodes; adjusting the number of concurrent threads of the thread pool according to the load status, and the concurrent threads are used to initiate writing to the data node based on the data to be written ask.

Wherein, the adjusting the number of concurrent threads in the thread pool according to the load state, the concurrent threads being used to initiate a write request to the data node based on the data to be written, includes: according to the data node control the concurrent thread to initiate a write request to the data node, or control the concurrent thread to suspend the initiation of the write request.

Wherein, the controlling the concurrent thread to initiate a write request to the data node according to the load state of the data node, or controlling the concurrent thread to suspend the initiation of the write request includes: responding to the data node being In a low load state, assign the data to be written to the concurrent thread, so that the concurrent thread initiates a write request to the data node based on the data to be written; in response to the high load of the data node state, controlling the concurrent thread to suspend initiating write requests, and continue to obtain the load state of the data node after a preset time.

Wherein, in response to the data node being in a low load state, allocating the data to be written to the concurrent thread, so that the concurrent thread initiates writing to the data node based on the data to be written After the step of requesting, the data writing method further includes: increasing the number of concurrent threads in response to the data node being in an ultra-low load state.

Wherein, after the step of controlling the concurrent threads to suspend and initiate write requests in response to the data node being in a high load state, and continuing to obtain the load state of the data node after a preset time, the data write The method further includes: reducing the number of concurrent threads in response to the data node being in an extremely high load state.

Wherein, said obtaining the load states of several said data nodes includes: obtaining the load states of several said data nodes through said thread pool, and storing said load states in the cache of said thread pool, so that said The concurrent thread acquires the load status corresponding to the data to be written from the cache.

Wherein, the allocating the data to be written to the corresponding data buffer queue according to the attribute information includes: calculating the data node corresponding to the data to be written according to the attribute information; The corresponding relationship between the data nodes and the data buffer queues is determined, and the data to be written is allocated to the corresponding data buffer queues.

In order to solve the above problems, this application provides a second technical solution: provide a terminal device, including a processor and a memory connected to the processor, wherein program data is stored in the memory, and the processor calls The program data stored in the memory is used to execute the above-mentioned data writing method.

In order to solve the above problems, the present application provides a third technical solution: provide a computer-readable storage medium storing program instructions, the program instructions are executed to implement the above-mentioned data writing method.

The present application provides a data writing method, a terminal device, and a computer-readable storage medium. The data writing method includes: obtaining a number of data to be written from a data source, and the data to be written includes attribute information with a unique identifier; according to The attribute information assigns several data to be written to the corresponding data buffer queue; in response to the write request, the data to be written in the data buffer queue is sent to the corresponding data node. The data writing method of the present application allocates a number of data to be written to the corresponding data buffer queue according to the attribute information, so that the data to be written in the data buffer queue can be sent to the corresponding data node, and the writing of the data to be written only It is related to the corresponding data nodes, avoiding the distributed long tail effect, making full use of cluster resources and improving the efficiency of data writing.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort. in:

Fig. 1 is a schematic flow chart of the first embodiment of the data writing method provided by the present application;

FIG. 2 is a schematic flow diagram of a second embodiment of the data writing method provided by the present application;

FIG. 3 is a schematic flowchart of a third embodiment of the data writing method provided by the present application;

FIG. 4 is a schematic flowchart of a fourth embodiment of the data writing method provided by the present application;

FIG. 5 is a schematic structural diagram of an embodiment of a terminal device provided by the present application;

Fig. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

It should be noted that if there are directional indications (such as up, down, left, right, front, back...) in the embodiment of the present application, the directional indications are only used to explain the position in a certain posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes, and cannot be interpreted as indications or hints Its relative importance or implicitly indicates the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In addition, the technical solutions of the various embodiments can be combined with each other, but it must be based on the realization of those skilled in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist , nor within the scope of protection required by the present application.

Existing data writing methods can use the open source tool elasticsearch-hadoop to write ecological data into the cluster when massive data is loaded into the database with high concurrency, for example, when data is written into the Elasticsearch cluster.

However, when this data writing method writes data to the data nodes of the cluster, because it cannot perceive the status of the cluster index, such as the cluster load is too high or the index status is not good, it will continue to send write requests to the cluster, causing the data node to crash , causing cluster avalanche; and, write requests are distributed to multiple data nodes, and the load between each data node is unbalanced, so that the load status of the entire cluster needs to be considered when writing data to data nodes, and there is a distributed long tail effect. When batching write requests, the write performance is limited by the data node with the highest load.

When this data writing method is used for concurrency control by other development tools, such as when using Spark/MapReduce integration, it can only pre-apply for and then determine computing resources. Users often can only determine the size of computing resources to apply for based on the size of the data source. When too many resources are applied for, it is easy to cause too much pressure on the cluster and the cluster load is too high; when the application resources are too small, the resources of the cluster are idle and the load of the cluster is too low, and the existing data writing method cannot control the load status of the cluster and data nodes. Perform elastic adjustment so that cluster resources cannot be fully utilized.

The existing data writing method can also process batch write requests through the application gateway. Specifically, the data writing method uses the gateway to receive batch write requests, and then the gateway forwards the write requests to the corresponding However, this method needs to deploy the application gateway independently, and the cost of data writing is high.

In view of this, this application proposes a data writing method, the data writing method is applied to a terminal device, and the terminal device is used to write batches of data to be written into the data nodes of the cluster, so that the data nodes Process the data to be written. Understandably, the cluster includes several distributed data nodes, and each data node may be a service instance corresponding to a different processing function. The terminal device is used to allocate the data to be written to the corresponding data buffer queue according to the attribute information of the data to be written, so as to send the data to be written in the data buffer queue to the corresponding data node when the load status of the data node is low . The terminal device can be applied in the technical field of data processing. Among them, the cluster applied by the data writing method can be used for financial technology, for example, it can be used for high-speed retrieval in business scenarios such as e-commerce, electronic payment, securities, electronic banking, tax transactions, credit cards, online shopping, insurance, etc. No specific limitation is made here.

Wherein, the terminal device in the present application may be a server, or may be a system in which the server and the local terminal cooperate with each other. Correspondingly, various parts included in the terminal device, such as various units, subunits, modules, and submodules, may all be set in the server, or may be set in the server and the local terminal respectively.

Further, the above server may be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide a distributed server, or as a single software or software module, which is not specifically limited here. In some possible implementation manners, the data writing method in the embodiment of the present application may be implemented in a manner in which a processor invokes a computer-readable instruction stored in a memory.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of a first embodiment of a data writing method provided by the present application. As shown in Figure 1, in this embodiment, the data writing method includes the following steps:

Step S11: Obtain a number of data to be written from the data source, the data to be written includes attribute information with a unique identifier.

Specifically, the data source can be a data acquisition system, a business system, a data processing system, etc. The data source is used to acquire massive data in real time, so that the terminal device obtains several data to be written from the data source, and each data to be written It includes uniquely corresponding attribute information, and the attribute information is used to identify the data to be written. Exemplarily, when the data to be written is a data file, the attribute information is used to index the data to be written, so that the data to be written can be stored and searched. For example, when the data processing method is used to write data into the Elasticsearch cluster, the attribute information can be generated through the version number of the Elasticsearch document, the user_account field of the document, etc. In other application scenarios, it can also be automatically generated by using tools such as plug-ins It is used to identify the attribute information of the data to be written, and the attribute information is not specifically limited here.

Step S12: Allocate a number of data to be written into corresponding data buffer queues according to the attribute information.

After the data to be written is acquired, the data to be written is allocated to the corresponding data buffer queue according to the attribute information of each data to be written. It can be understood that the terminal device is used to write the data to be written into several data nodes of the cluster, and the corresponding relationship between the data buffer queue and the data node is stored in the terminal device, and each data node has a unique corresponding data buffer queue, so that the data to be written in the same data buffer queue is written by the corresponding data node.

Step S13: In response to the write request, send the data to be written in the data buffer queue to the corresponding data node.

In an optional implementation, the terminal device obtains the load status of the data node, and generates a write request for data to be written when the load status is low load, and in response to the write request, the terminal device buffers the data in the queue to be written The incoming data is sent to the corresponding data node.

In the embodiment of this application, the terminal device assigns the data to be written to the corresponding data buffer queue after receiving the data to be written by setting the data buffer queue corresponding to the data node, so as to sort the data to be written in advance , which avoids the data flow of the data to be written inside the data node, improves the security of the data to be written, and improves the efficiency of data writing; due to the establishment of the corresponding relationship between the data node and the data buffer queue, the write to be written The input data is only related to the corresponding data node, which avoids the distributed long tail effect, makes full use of cluster resources, and improves the efficiency of data writing.

Please refer to FIG. 2 . FIG. 2 is a schematic flowchart of a second embodiment of the data writing method provided by the present application. As shown in Figure 2, in this embodiment, after step S11, the data writing method may also include:

Step S21: Transfer some data to be written to the thread pool.

Specifically, when using the client to write data, a thread pool is set in the client, and the thread pool is used to judge whether the condition for initiating a write request is met according to the load status of the data node, and to initiate the write request when the load status is satisfied. Initiate a write request to the corresponding data node when the condition of the input request is met.

Step S22: Obtain the load status of several data nodes.

Specifically, the terminal device can obtain the load status of the data node in the previous preset time period within the current preset time period, or the terminal device can obtain the current load status of the data node within the preset time period, so that The thread pool can judge whether the conditions for initiating a write request to the data node are met according to the load status of the data node. In an optional implementation manner, the terminal device may acquire the load status of the data node through the thread pool.

Step S23: Adjust the number of concurrent threads in the thread pool according to the load status, and the concurrent threads are used to initiate a write request to the data node based on the data to be written.

After obtaining the load status of several data nodes, adjust the number of concurrent threads in the thread pool according to the load status. The concurrent threads are used to initiate write requests to the data nodes based on the data to be written. Each concurrent thread can process a data to be written. write request.

Therefore, adjusting the number of concurrent threads in the thread pool according to the load status can enable the terminal device to control the writing speed of the data to be written, so as to make full use of the resources of the cluster and improve the efficiency of data writing.

Optionally, step S23 includes: controlling the concurrent thread to initiate a write request to the data node according to the load status of the data node, or controlling the concurrent thread to suspend the initiation of the write request.

Further, according to the load state of the data node, controlling the concurrent thread to initiate a write request to the data node, or the step of controlling the concurrent thread to suspend and initiate the write request includes: in response to the data node being in a low load state, allocating the data to be written To the concurrent thread, so that the concurrent thread initiates a write request to the data node based on the data to be written; in response to the high load state of the data node, suspend the writing of the data to be written, and continue to obtain the data node after the preset time load status.

Specifically, in this embodiment, the number of concurrent threads in the thread pool is dynamically adjusted according to the load state of the data node, so that when the data node is in a low-load state, the terminal device can allocate the data to be written to the concurrent threads to Make concurrent threads initiate a write request to the data node based on the data to be written; when the data node is in a high-load state, the terminal device can suspend the writing process of the current data to be written, and continue to obtain the load of the data node after a preset time state until the data node is in a low load state.

It can be understood that the above-mentioned low-load state and high-load state are relative states, and the low-load state and high-load state can be determined by numerical judgment; for example, when the load of a data node is greater than a first preset value, it is determined that the data node In the high load state, when the load of the data node is less than a second preset value, it is determined that the data node is in a low load state, and the second preset value is smaller than the first preset value. This embodiment dynamically adjusts the thread pool according to the load state, so that the load of the cluster is in a relatively high state when data is written and does not exceed the load capacity of the cluster, so that the cluster resources are fully utilized, and the current limiting fuse of the cluster is realized. Improve the data writing performance of the cluster.

Further, according to the load state of the data node, controlling the concurrent thread to initiate a write request to the data node, or the step of controlling the concurrent thread to suspend the initiation of the write request further includes: in response to the data node being in an ultra-low load state, increasing the number of concurrent threads quantity.

Specifically, the increase in the number of concurrent threads increases the number of threads that can process data to be written in the thread pool, increases the number of data to be processed by the cluster, and then increases the cluster load, making full use of cluster resources and reducing server costs. , to reduce costs and increase efficiency.

Wherein, the ultra-low load state is a relative state of the low-load state, and the ultra-low-load state can be determined on the basis of the low-load state. For example, when the load of the data node is less than the second preset value, it is determined that the data node is in a low load state, and continues to determine whether the load is less than the third preset value (the third preset value is less than the second preset value), so as to When the load is less than the third preset value, it is determined that the data node is in an ultra-low load state.

Further, according to the load state of the data node, controlling the concurrent thread to initiate a write request to the data node, or the step of controlling the concurrent thread to suspend the initiation of the write request further includes: in response to the data node being in an ultra-high load state, reducing the number of concurrent threads quantity.

Specifically, the reduction in the number of concurrent threads reduces the processing capacity of the thread pool for the data to be written, which can effectively reduce the cluster load and make the data nodes in the normal load range, which is conducive to the full utilization of cluster resources. Wherein, the ultra-high load state is a relative state of the high load state, and the ultra-high load state can be determined on the basis of the high load state. For example, when the load of the data node is greater than the first preset value, it is determined that the data node is in a high load state, and continue to determine whether the load is greater than the fourth preset value (the fourth preset value is greater than the first preset value), so as to When the load is greater than the fourth preset value, it is determined that the data node is in an ultra-high load state.

Optionally, step S22 includes: acquiring the load status of several data nodes through the thread pool, and storing the load status in the cache of the thread pool, so that the concurrent threads can acquire the load status corresponding to the data to be written from the cache.

Specifically, in order to avoid excessive cluster pressure caused by concurrent threads frequently initiating requests to obtain load status, in an optional implementation, a dedicated thread for obtaining load status is set in the thread pool. Set a time period to initiate a request to the cluster to obtain the load status of the data nodes of the cluster, and store the acquired load status in the cache of the thread pool; after the concurrent thread receives the data to be written, the concurrent thread according to the data to be written and the data The mapping relationship of the nodes, obtain the load status of the corresponding data node from the cache of the thread pool, and judge whether to initiate a write request according to the load status.

Further, the cache of the thread pool includes a local cache and/or a distributed cache.

Specifically, the local cache and the distributed cache (Redis) are the caches of the client in the stand-alone environment and the distributed environment, and the cache of the thread pool can select the local cache and/or the distributed cache according to the client's environment.

Please refer to FIG. 3 . FIG. 3 is a schematic flowchart of a third embodiment of the data writing method provided by the present application. As shown in Figure 3, in this embodiment, step S12 may also include:

Step S31: Calculate the data node corresponding to the data to be written according to the attribute information.

Specifically, the process of obtaining the data to be written from the data source to the data node to complete the writing of the data to be written is defined as the data routing of the data to be written, that is, the data routing is the data to be written from the data source to the destination data node dynamic process. In this embodiment, the data node corresponding to the data to be written can be calculated according to the routing algorithm and the attribute information of the data to be written. Among them, the routing algorithm is an algorithm for finding the least-cost path of the data to be written from the data source to the destination data node.

Step S32: According to the preset correspondence between data nodes and data buffer queues, allocate the data to be written to the corresponding data buffer queues.

The corresponding relationship between the data node and the data buffer queue is stored in the terminal device. After obtaining the data node corresponding to the data to be written, the data buffer queue corresponding to the data to be written can be obtained according to the corresponding relationship between the data node and the data buffer queue. And allocate the data to be written to the corresponding data buffer queue.

Therefore, this embodiment can pre-sort the data to be written into the data buffer queue according to the corresponding relationship between the data node and the data buffer queue, avoiding the routing of the data to be written between the data nodes inside the cluster, and improving the data efficiency. The locality improves the data writing performance; since the data to be written is only related to the corresponding data node, the distributed long tail effect is avoided, the cluster resources are fully utilized, and the data writing efficiency is improved.

Optionally, please refer to FIG. 4 , which is a schematic flowchart of a fourth embodiment of the data writing method provided by the present application. As shown in Figure 4, the terminal device obtains some data to be written from the data source, and obtains the load status of the data node corresponding to the data to be written through the thread pool, so as to adjust the number of concurrent threads in the thread pool according to the load status. The terminal device judges whether to control the concurrent thread to initiate a write request to the data node according to the load status of the data node; when the load status of the data node is low, the terminal device increases the number of concurrent threads to increase the load of the data node, and controls the concurrent The thread sends the data to be written and the write request to the data buffer queue, so that the data buffer queue sends the data to be written to the corresponding data node according to the write request, so that the data node completes the writing of the data to be written; When the load status of the data node is high load, the control concurrent thread suspends to initiate the write request, and the data buffer queue suspends writing the data to be written into the data node, so as to continue to initiate the write request when the load status of the data node is low; When the load state of the data node is extremely high, the terminal device reduces the number of concurrent threads to reduce the load of the data node.

Further, as shown in Figure 4, the data buffer queue corresponds to the data nodes of the cluster, for example, when the cluster includes the first data node, the second data node, ..., the Nth data node, the data buffer queue includes the corresponding first Buffer queue, the second buffer queue, ..., the Nth buffer queue; when the first data node is in a low-load state, the control concurrent thread sends the data to be written and the write request to the first buffer queue, so that the first buffer The queue writes the data to be written to the first data node; when the second data node is in a high-load state, the control concurrent thread sends the data to be written to the second buffer queue and suspends initiating write requests, and the second buffer queue is in the When the write request is not obtained, the sending of the write data to the second data node is suspended, so that the load of the second data node will not continue to increase, and the current limit fuse of the cluster load is realized.

Different from the prior art, the data writing method in the embodiment of the present application improves data writing performance through data pre-routing and a dynamic thread pool. In the process of data pre-routing, the data to be written is pre-sorted to the data buffer queue, which avoids the routing of data to be written between data nodes, improves the locality of data, and improves the performance of data writing; Whether the written data is written is only related to the load of the corresponding data node, which avoids the distributed long-tail effect, makes full use of cluster resources, and improves the efficiency of data writing; in the dynamic thread pool, by obtaining the data node of the cluster The load status is used to dynamically adjust the number of concurrent threads that initiate write requests, so that the cluster load can be fully increased. At the same time, when the cluster load is too large, it also plays the role of current limiting and fuse, so that the cluster resources can be fully utilized. The data writing method of the embodiment of the present application improves the timeliness of data by improving the writing performance, and also reduces the purchase cost of the server, thereby achieving the effect of cost reduction and efficiency increase.

Please refer to FIG. 5 , which is a schematic structural diagram of an embodiment of a terminal device provided in this application. As shown in FIG. 5 , the terminal device 100 includes a processor 101 and a memory 102 connected to the processor 101, wherein program data is stored in the memory 102, and the processor 101 calls the program data stored in the memory 102 to execute the above-mentioned all methods.

Optionally, in an embodiment, the processor 101 is configured to execute the program data to implement the following method: obtain the general text of several general tasks, input the general text into the language model to be trained, so as to obtain the corresponding first prompt text; Clustering the first prompt texts of several common tasks to obtain cluster cluster information; obtaining training text and task parameters of tasks to be trained, and obtaining second prompt texts based on the cluster cluster information, the first prompt text and task parameters; The language model to be trained is trained by using the training text and the second prompt text.

Optionally, in another embodiment, the processor 101 is configured to execute program data to implement the following method: receive the example text, and input the example text into the language model to obtain the second prompt text of the example text; The prompt text is input to the language model to obtain the prediction result of the language model based on the second prompt text to the example text output; the answer text corresponding to the example text is output based on the prediction result to complete the question and answer; the language model is obtained by using the above model training method of.

Wherein, the processor 101 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 101 may be an electronic chip with signal processing capabilities. The processor 101 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components . A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The memory 102 can be a memory stick, a TF card, etc., and can store all information in the terminal device 100, including input raw data, computer programs, intermediate running results and final running results are all stored in the memory 102. It stores and retrieves information according to locations specified by the processor 101. With the memory 102, the terminal device 100 has a memory function to ensure normal operation. The memory 102 of the terminal device 100 can be divided into main memory (internal memory) and auxiliary memory (external memory) according to the purpose, and there is also a classification method of external memory and internal memory. External storage is usually magnetic media or optical discs, which can store information for a long time. Memory refers to the storage unit on the motherboard, which is used to store data and programs currently being executed, but it is only used to store programs and data temporarily, and the data will be lost when the power is turned off or cut off.

Please refer to FIG. 6 , which is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by the present application. As shown in FIG. 6 , the computer-readable storage medium 110 stores program instructions 111 capable of implementing all the above-mentioned methods.

If the integrated units of the functional units in various embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they may be stored in the computer-readable storage medium 110 . Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of software products, and the computer-readable storage medium 110 is in a program Instruction 111 includes several instructions to make a computer device (which can be a personal computer, a system server, or a network device, etc.), an electronic device (such as MP3, MP4, etc., or a mobile phone, a tablet computer, a wearable device, etc.) The mobile terminal may also be a desktop computer, etc.) or a processor (processor) to execute all or part of the steps of the methods in various implementation manners of the present application.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media 110 (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each process and/or block in the flowchart and/or block diagram, and a combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by the computer-readable storage medium 110 . These computer-readable storage media 110 can be provided to a general purpose computer, a special purpose computer, an embedded processor, or a processor of other programmable data processing devices to produce a machine such that the The program instructions 111 generate means for implementing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer-readable storage media 110 can also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the program instructions 111 stored in the computer-readable storage medium 110 generate An article of manufacture of a device that instructs a device to implement the function specified in a flow or steps of a flowchart and/or a block or blocks of a block diagram.

These computer-readable storage media 110 can also be loaded onto a computer or other programmable data processing device, so that a series of operational steps are performed on the computer or other programmable device to produce computer-implemented processing, thereby The program instructions 111 executed on it provide steps for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For the use of instruction execution systems, devices or equipment (which may be personal computers, servers, network equipment or other systems that can fetch instructions from instruction execution systems, devices or devices and execute instructions), or in combination with these instruction execution systems, devices or devices And use. The above is only the implementation of the application, and does not limit the patent scope of the application. Any equivalent structure or equivalent process conversion made by using the specification and drawings of the application, or directly or indirectly used in other related technologies fields, are all included in the scope of patent protection of this application in the same way.

Claims (10)

1. A data writing method, characterized in that, comprising: Obtain a number of data to be written from a data source, the data to be written includes attribute information with a unique identifier; Allocating the data to be written to a corresponding data buffer queue according to the attribute information; In response to the write request, the data to be written in the data buffer queue is sent to the corresponding data node. 2. The data writing method according to claim 1, characterized in that, after the step of acquiring a number of data to be written from the data source, the data to be written includes attribute information with a unique identifier, the data Write methods also include: Transfer some of the data to be written to the thread pool; Obtain the load status of several of the data nodes; The number of concurrent threads in the thread pool is adjusted according to the load status, and the concurrent threads are used to initiate a write request to the data node based on the data to be written. 3. The data writing method according to claim 2, wherein the number of concurrent threads of the thread pool is adjusted according to the load state, and the concurrent threads are used to write data to the database based on the data to be written. The data node initiates a write request, including: According to the load state of the data node, control the concurrent thread to initiate a write request to the data node, or control the concurrent thread to suspend the initiation of the write request. 4. The data writing method according to claim 3, wherein, according to the load state of the data node, the concurrent thread is controlled to initiate a write request to the data node, or the concurrent thread is controlled to The thread suspends to initiate a write request, including: In response to the data node being in a low-load state, allocating the data to be written to the concurrent thread, so that the concurrent thread initiates a write request to the data node based on the data to be written; In response to the data node being in a high load state, the concurrent threads are controlled to suspend initiating write requests, and continue to obtain the load state of the data node after a preset time. 5. The data writing method according to claim 4, wherein the data to be written is allocated to the concurrent thread in response to the low load state of the data node, so that the concurrent After the thread initiates a write request to the data node based on the data to be written, the data writing method further includes: In response to the data node being in an ultra-low load state, increasing the number of concurrent threads. 6. The data writing method according to claim 4, characterized in that, in response to the high load state of the data node, the concurrent thread is controlled to suspend the initiation of the write request, and continue to obtain the data after a preset time After the step of loading state of the data node, the data writing method further includes: Responding to the data node being in an ultra-high load state, reducing the number of concurrent threads. 7. The data writing method according to claim 2, wherein said obtaining load statuses of several said data nodes comprises: Obtain the load status of several data nodes through the thread pool, and store the load status in the cache of the thread pool, so that the concurrent threads can obtain from the cache the data corresponding to the data to be written load status. 8. The data writing method according to claim 1 or 2, wherein the distributing the plurality of data to be written to the corresponding data buffer queue according to the attribute information comprises: calculating a data node corresponding to the data to be written according to the attribute information; According to the preset correspondence between the data nodes and the data buffer queues, the data to be written is allocated to the corresponding data buffer queues. 9. A terminal device, characterized by comprising a processor and a memory connected to the processor, wherein program data is stored in the memory, and the processor calls the program data stored in the memory , to execute the data writing method according to any one of claims 1-8. 10. A computer-readable storage medium, characterized by storing program instructions, the program instructions are executed to implement the data writing method according to any one of claims 1-8.