CN114442907B - Data migration method and device, server and network system - Google Patents

Abstract

The embodiment of the application provides a data migration method and device, a server and a network system. The data migration method comprises the following steps: responding to a migration instruction for migrating data of an original data queue in a first server, predicting the total number of times that a consumer reads first data of the original data queue, wherein the first data is written in the original data queue when the migration instruction is received; when the total number of times is less than or equal to the set number of times, selecting a first migration strategy to determine a migration starting position, wherein the migration starting position of the first migration strategy is positioned at the downstream of the earliest data position of the original data queue; copying the data in the original data queue to the second server according to a set sequence to form a migration data queue, wherein the set sequence is from a migration starting position to a position of the data which is newly written into the original data queue by a producer in the migration process. The migration strategy is selected according to the consumption condition of the consumer, so that the migration speed is improved, and the io resource consumption of the disk is reduced.

Description

Data migration method and device, server, and network system

Technical Field

The present application relates to the technical field of data storage, and in particular to a data migration method and device, a server, and a network system.

Background technique

Data channel products are mostly used for upstream systems to publish messages and downstream systems to subscribe to and use messages. Messages are stored in the data channel in the form of a first-in-first-out queue, and downstream tasks also read and process data sequentially.

Currently, the high reliability of data in most message channel products is achieved through data redundancy, that is, business data is stored in multiple copies in the cluster. When some nodes fail or nodes are expanded or reduced, necessary data copies need to be migrated to ensure the high reliability of the original data. The existing data copy migration technology has a large amount of data, a slow migration speed, and consumes a lot of disk IO, network bandwidth and other resources, which is bound to have an impact on the original normal business.

Summary of the invention

The embodiments of the present application provide a data migration method and device, a server, and a network system. Under the premise of ensuring that upstream and downstream data can be effectively transmitted and that data loss does not occur, different migration strategies can be selected according to different consumption situations of consumers, which helps to improve the speed of completing replica migration and reduce the IO resource consumption of the disk storing data, thereby reducing the impact of replica migration on normal business.

In a first aspect, an embodiment of the present application provides a data migration method, the data migration method comprising: responding to a migration instruction for migrating data in an original data queue in a first server, predicting the total number of times a consumer reads the first data in the original data queue, wherein the first data is the data that has been written in the original data queue when the migration instruction is received; when the total number of times is less than or equal to a set number of times, selecting a first migration strategy to determine a start migration position, wherein the start migration position of the first migration strategy is located downstream of the earliest data position of the original data queue; copying the data in the original data queue to a second server in a set order to form a migration data queue, wherein the set order is an order from the start migration position of the selected migration strategy to the position of the data most recently written to the original data queue by the producer during the migration process.

In the above scheme, when a migration instruction for migrating data in the original data queue in the first server is received, the total number of times the consumer reads the first data in the original data queue again after receiving the migration instruction can be predicted. If the total number is less than or equal to the set number, that is, the number of times the consumer reads the first data in the original data queue is small, the first migration strategy is selected to determine the start migration position, and the data in the original data queue is copied to the second server in the order from the start migration position of the selected migration strategy to the position of the data most recently written to the original data queue by the producer during the migration process to form a migration data queue. Since the start migration position of the first migration strategy is located downstream of the earliest data position of the original data queue, the number of copies of the first data is reduced, and since the data is first stored in the page cache, the data in the page cache is stored in the disk after a certain period of time, that is, the data that is not copied is generally the data located in the disk, thereby increasing the proportion of data read from the page cache, reducing the probability of a real disk read, thereby reducing the io resource consumption of the disk storing the data, reducing the impact of the copy migration on normal business, and there is no need to migrate all the data in the original data queue, thereby improving the speed of completing the copy migration.

In a possible implementation, the selecting of the first migration strategy to determine the starting migration position specifically includes: obtaining the reading position and the latest data position of the earliest active consumer of the original data queue at the current moment, wherein the current moment is the moment when it is determined that the total number of times is less than the set number of times, and the latest data position is the position of the data written by the producer to the original data queue at the current moment; when the distance between the reading position of the earliest active consumer and the latest data position is less than or equal to the set distance, determining that the starting migration position of the first migration strategy is the latest data position.

That is to say, in this implementation, it is determined that the distance between the reading position of the earliest active consumer in the original data queue at the current moment when the total number of times is less than the set number and the latest data position is less than or equal to the set distance. The earliest active consumer is the real-time consumer of the original data queue. Since the earliest active consumer is farthest from the latest data position, when it is a real-time consumer, other active consumers are also real-time consumers. Therefore, the starting migration position of the first migration strategy can be determined to be the latest data position of the original data queue. In this way, there is no need to copy the first data between the earliest data position of the original data queue and the latest data position at the moment when the total number of times is determined to be less than the set number, which can improve the speed of completing the copy migration. In this way, data is mainly read from the page cache, which reduces the probability of real disk reads and reduces the io resource consumption of the disk storing data, so that the disk io resources can be mainly used for normal business, reducing the impact of copy migration on normal business.

In a possible implementation, when the starting migration position is the latest data position, the data migration method further includes: obtaining the reading position of the earliest active consumer of the original data queue during the data migration process; when the migration data queue is copied to the data most recently written to the original data queue by the producer, determining that the reading position of the earliest active consumer of the original data queue during the data migration process enters the range of the second data of the original data queue, and completing the data migration, wherein the second data is the data that has been copied by the migration data queue.

That is to say, in this implementation, since the starting migration position of the first migration strategy is determined to be the latest data position of the original data queue, all consumers in the original data queue are real-time consumers of the original data queue. When the migrated data queue is copied to the data most recently written to the original data queue by the producer, as long as the reading position of the earliest active consumer in the migration process enters the range of the second data of the original data queue that has been copied by the migrated data queue, it can be ensured that the reading positions of all active consumers of the original data queue are within the range of the second data, and the data migration can be completed.

In a possible implementation, the data migration method further includes: when the distance between the reading position of the earliest active consumer and the latest data position is greater than a set distance, determining the starting migration position of the first migration strategy to be the reading position of the earliest active consumer.

That is to say, in this implementation, when the distance between the reading position of the earliest active consumer in the original data queue and the latest data position is greater than the set distance, that is, at least the earliest active consumer among all active consumers is a non-real-time consumer, and it stays in the first data position of the original data queue for a long time, at this time, the starting migration position of the first migration strategy can be determined as the reading position of the earliest active consumer, so that the earliest active consumer can work normally throughout the entire process of data migration, and there is no need to copy the first data between the earliest data position of the original data queue and the reading position of the earliest active consumer, which can improve the speed of completing the copy migration and reduce the probability of real disk reading to a certain extent, thereby reducing the io resource consumption of the disk storing data and reducing the impact of copy migration on normal business.

In a possible implementation, when the migration start position is the earliest active consumer's reading position, the data migration method further includes: determining whether the migration data queue is copied to the data most recently written by the producer to the original data queue to complete the data migration.

That is to say, in this implementation, since the starting migration position of the first migration strategy is the reading position of the earliest active consumer, when the migration data queue is copied to the data most recently written by the producer to the original data queue during the data migration process, it can be ensured that all active consumers are within the range of the copied second data of the original data queue, thereby completing the data migration.

In a possible implementation, obtaining the reading position of the earliest active consumer of the original data queue specifically includes: obtaining the reading position of each active consumer in all the data queues that are read, wherein the all the data queues include the original data queue; transferring the mapping structure of each active consumer and the reading position of each active consumer in all the data queues that are read into a mapping structure of each data queue and the reading position of all active consumers in each data queue, thereby obtaining the reading positions of all active consumers of the original data queue; and comparing the reading positions of all the active consumers in the original data queues to obtain the reading position of the earliest active consumer of the original data queue.

That is to say, in this implementation, it is impossible to directly obtain all active consumers existing in the original data queue and the reading positions of these active consumers, but it is possible to obtain the data queue read by each active consumer and the reading position in the data queue. In this way, the reading position of each active consumer in all the data queues read can be obtained first, and then the mapping structure/correspondence between each active consumer and the reading position of each active consumer in all the data queues read can be transferred to the mapping structure/correspondence between each data queue and the reading positions of all active consumers in each data queue, thereby obtaining the active consumers existing in the original data queue and their reading positions, and then comparing the reading positions of all active consumers in the original data queue to obtain the reading position of the earliest active consumer in the original data queue.

In a possible implementation, before obtaining the reading position of each active consumer in all the data queues being read, obtaining the reading position of the earliest active consumer in the original data queue also includes: obtaining the reading positions of all consumers in the data queue being read, wherein all consumers include inactive consumers and active consumers; querying the reading status of all consumers to classify each of the all consumers as the active consumer or the inactive consumer, wherein the reading status of the active consumer is working and the reading status of the inactive consumer is paused; removing the information of the inactive consumer and the reading position of the inactive consumer in the data queue being read.

That is to say, in this implementation, it is not convenient to directly obtain the reading positions of all active consumers in all data queues being read. You can first obtain the reading positions of all consumers in the data queue being read, and all consumers include inactive consumers and active consumers. Then, query the reading status of all consumers to divide consumers into active consumers and inactive consumers. The reading status of active consumers is working, and the reading status of inactive consumers is paused. Then, remove the information of the reading positions of inactive consumers and inactive consumers in the data queue being read, and you can obtain the reading position of each active consumer in all data queues being read.

In a possible implementation, after completing the data migration, the data migration method further includes: determining that the consumer has read the first data that has not been migrated in the original data queue, and deleting the original data queue according to a set strategy; or determining that the consumer has not read the first data that has not been migrated in the original data queue, and deleting the original data queue.

That is to say, in this implementation, when the first migration strategy is selected to determine the starting migration position, since the starting migration position of the first migration strategy is located downstream of the earliest data position of the original data queue, the data between the earliest data position in the original data queue and the starting migration position are not copied to the migration data queue. After the data migration is completed, it is necessary to determine whether the first data that has not been migrated in the original data queue has been read by the consumer. If it has not been read, the original data queue can be deleted; if it has been read, the original data queue can be deleted according to the set strategy. The set strategy can, for example, be to delete the original data queue after the consumer reads the first data that has not been migrated, to ensure that it will not affect normal work.

In a possible implementation, the data migration method includes: when the total number of times is greater than a set number of times, selecting a second migration strategy to determine a start migration position, wherein the start migration position of the second migration strategy is the earliest data position of the original data queue.

That is to say, in this implementation, when the total number of times the consumer reads the first data in the original data queue is greater than the set number of times, after the data migration is completed by selecting the first migration strategy, although the original data queue can be deleted according to the set strategy to ensure that the consumer can read the unmigrated first data in the original data queue again after the migration is completed, the consumption of switching the consumer from the migrated data queue to the original data queue is large. Therefore, when the consumer reads the first data in the original data queue a large number of times, in order to reduce consumption, the second migration strategy can be selected to start migration from the earliest data position in the original data queue to ensure that the consumer can complete all reading operations in the migrated data queue.

In a possible implementation, the data migration method further includes: determining that the migration data queue is copied to the data most recently written by the producer into the original data queue, thereby completing the data migration.

That is to say, in this implementation, when the second migration strategy is selected to start migration from the earliest data position of the original data queue, when the migration data queue copies the data most recently written by the producer to the original data queue, it can be guaranteed that the migration data queue copies all the data to the original data queue, and the data migration is completed.

In a possible implementation manner, the data migration method further includes: deleting the original data queue after completing the data migration.

That is to say, in this implementation, since the starting migration position of the second migration strategy is the earliest data position of the original data queue, after the migration data queue copies the data most recently written by the producer to the original data queue and completes the data migration, the migration data queue can copy all the data of the original data queue. Therefore, after the data migration is completed, the original data queue can be deleted immediately to free up memory.

In a second aspect, an embodiment of the present application provides a data migration method, which includes: responding to a migration instruction for migrating data in an original data queue in a first server, obtaining the read position and the latest data position of the earliest active consumer of the original data queue at the current moment, wherein the current moment is the moment when the migration instruction is received, and the latest data position is the position of the data written by the producer to the original data queue at the current moment; when the distance between the read position of the earliest active consumer and the latest data position is less than or equal to a set distance, copying the data in the original data queue to the second server in the order from the latest data position to the position of the data most recently written to the original data queue by the producer during the migration process to form a migration data queue; and/or, when the distance between the read position of the earliest active consumer and the latest data position is greater than the set distance, copying the data in the original data queue to the second server in the order from the read position of the earliest active consumer to the position of the data most recently written to the original data queue by the producer during the migration process to form a migration data queue.

In the above scheme, when the distance between the reading position of the earliest active consumer in the original data queue at the current moment when the migration instruction is received and the latest data position is less than or equal to the set distance, all active consumers in the original data queue are real-time consumers, and therefore data migration starts from the latest data position of the original data queue; when the distance between the reading position of the earliest active consumer in the original data queue at the current moment when the migration instruction is received and the latest data position is greater than the set distance, at least the earliest active consumer in the original data queue is not a real-time consumer, and therefore data migration starts from the reading position of the earliest active consumer in the original data queue, and the data of the original data queue in the first server is migrated. The data is copied to the second server in the order from the starting migration position to the position of the data most recently written by the producer to the original data queue during the migration process to form a migration data queue. Compared with the scheme of migrating data from the earliest data position, since the data written by the producer to the original data queue is first stored in the page cache for a certain period of time and then stored in the disk from the page cache, the data migration method of the embodiment of the present application mainly copies data from the page cache for migration, which reduces the probability of copying the first data from the disk, that is, reduces the probability of a real disk read, reduces the impact of the copy migration on the normal business of the disk, and does not need to migrate all the data in the original data queue, which can improve the speed of completing the copy migration.

In one possible implementation, when migrating data starting from the latest data position, the data migration method includes: obtaining the reading position of the earliest active consumer of the original data queue during the data migration process; when the migrated data queue is copied to the data most recently written to the original data queue by the producer, determining that the reading position of the earliest active consumer of the original data queue during the data migration process enters the second data range of the original data queue, and completing the data migration, wherein the second data is the data that has been copied by the migrated data queue.

That is to say, in this implementation, since the starting migration position is the latest data position of the original data queue, all consumers in the original data queue are real-time consumers, when the migration data queue is copied to the data most recently written by the producer to the original data queue, as long as the reading position of the earliest active consumer farthest from the latest data position during the migration process enters the range of the migration data queue, it can be ensured that the reading positions of all active consumers of the original data queue are within the range of the migration data queue, and the data migration can be completed.

In a possible implementation, when migrating data starting from the read position of the earliest active consumer, the data migration method further includes: determining that the migration data queue is copied to the data most recently written by the producer to the original data queue, to complete the data migration.

That is to say, in this implementation, since the starting migration position is the reading position of the earliest active consumer, when the migration data queue is copied to the data most recently written by the producer to the original data queue during the data migration process, it can be ensured that all active consumers are within the range of the migration data queue, thereby completing the data migration.

In a possible implementation, obtaining the reading position of the earliest active consumer in the original data queue specifically includes: obtaining the reading position of each active consumer in all the data queues that are read, wherein the all the data queues include the original data queue; transferring the mapping structure of each active consumer and the reading position of each active consumer in all the data queues that are read into a mapping structure of each data queue and the reading position of all active consumers in each data queue, thereby obtaining the reading positions of all active consumers in the original data queue; and comparing the reading positions of all the active consumers in the original data queue to obtain the reading position of the earliest active consumer of the original data queue.

That is to say, in this implementation, it is impossible to directly obtain all active consumers in the original data queue and the reading positions of these active consumers, but it is possible to obtain the data queue read by each active consumer and the reading position in the data queue. In this way, the mapping structure/correspondence between each active consumer and the reading position of each active consumer in all data queues read by each active consumer can be transferred to the mapping structure/correspondence between each data queue and the reading positions of all active consumers in each data queue, thereby obtaining the active consumers existing in the original data queue and their reading positions, and then comparing the reading positions of all active consumers in the original data queue to obtain the reading position of the earliest active consumer in the original data queue.

In a possible implementation, before obtaining the reading position of each active consumer in all the data queues being read, obtaining the reading position of the earliest active consumer in the original data queue also includes: obtaining the reading positions of all consumers in the data queue being read, wherein all consumers include inactive consumers and active consumers; querying the reading status of all consumers to divide each of the all consumers into the active consumer and the inactive consumer, wherein the reading status of the active consumer is working and the reading status of the inactive consumer is paused; removing the information of the inactive consumer and the reading position of the inactive consumer in the data queue being read.

That is to say, in this implementation, it is not convenient to directly obtain the reading positions of all active consumers in all data queues being read. You can first obtain the reading positions of all consumers in the data queue being read, and all consumers include inactive consumers and active consumers. Then, query the reading status of all consumers to divide consumers into active consumers and inactive consumers. The reading status of active consumers is working, and the reading status of inactive consumers is paused. Then, remove the information of the reading positions of inactive consumers and inactive consumers in the data queue being read, and you can obtain the reading position of each active consumer in all data queues being read.

In a possible implementation, after completing the data migration, the data migration method further includes: determining that the consumer has read the unmigrated first data of the original data queue, and deleting the original data queue according to a set strategy, wherein the first data is the data written in the original data queue when the migration instruction is received; or determining that the consumer has not read the unmigrated first data of the original data queue, and deleting the original data queue.

That is to say, in this implementation, after completing the data migration, it is necessary to determine whether the first data that has not been migrated in the original data queue has been read again by the consumer. If it has not been read again, the original data queue can be deleted immediately; if it has been read again, the original data queue can be deleted according to the set strategy, for example, the original data queue can be deleted after the consumer reads the first data that has not been migrated again, so as to ensure that it will not affect normal work.

In a third aspect, an embodiment of the present application provides a data migration device, comprising: a prediction unit, for responding to a migration instruction for migrating data in an original data queue in a first server, and predicting the total number of times the consumer reads the first data in the original data queue, wherein the first data is the data that has been written in the original data queue when the migration instruction is received; a selection unit, for selecting a first migration strategy to determine a start migration position when the total number of times is less than or equal to a set number of times, wherein the start migration position of the first migration strategy is downstream of the earliest data position of the original data queue; a migration unit, for copying the data in the original data queue to the second server in a set order to form a migration data queue, wherein the set order is an order from the start migration position of the selected migration strategy to the position of the data most recently written to the original data queue by the producer during the migration process.

In a possible implementation, the selection unit includes: an acquisition module, used to acquire the read position and the latest data position of the earliest active consumer of the original data queue at the current moment, wherein the current moment is the moment when it is determined that the total number of times is less than the set number of times, and the latest data position is the position of the data written by the producer to the original data queue at the current moment; a determination module, used to determine that the starting migration position of the first migration strategy is the latest data position when the distance between the read position of the earliest active consumer and the latest data position is less than or equal to the set distance.

In a possible implementation, when the starting migration position is the latest data position, the acquisition module is also used to obtain the reading position of the earliest active consumer of the original data queue during the data migration process; the data migration device also includes: a first determination unit, used to determine that the reading position of the earliest active consumer of the original data queue during the data migration process enters the range of the second data of the original data queue when the migration data queue is copied to the data most recently written to the original data queue by the producer, thereby completing the data migration, wherein the second data is the data that has been copied by the migration data queue.

In a possible implementation, the determination module is further configured to determine that the starting migration position of the first migration strategy is the reading position of the earliest active consumer when the distance between the reading position of the earliest active consumer and the latest data position is greater than a set distance.

In a possible implementation, when the starting migration position is the reading position of the earliest active consumer, the data migration device also includes: a second determination unit, used to determine the data copied from the migration data queue to the data most recently written by the producer to the original data queue to complete the data migration.

In a possible implementation, the acquisition module includes: a first acquisition submodule, used to acquire the reading position of each active consumer in all data queues read, wherein the all data queues include the original data queue; a transfer submodule, used to transfer the mapping structure of each active consumer and the reading position of each active consumer in all data queues read into a mapping structure of each data queue and the reading position of all active consumers in each data queue, thereby obtaining the reading positions of all active consumers of the original data queue; and a comparison acquisition submodule, used to compare the reading positions of all active consumers in the original data queue to obtain the reading position of the earliest active consumer of the original data queue.

In a possible implementation, the acquisition module also includes: a second acquisition submodule, used to acquire the reading position of all consumers in the data queue being read, wherein all consumers include inactive consumers and active consumers; a query submodule, used to query the reading status of all consumers to divide each of all consumers into the active consumer or the inactive consumer, wherein the reading status of the active consumer is working, and the reading status of the inactive consumer is paused; a removal submodule, used to remove the inactive consumer and the information of the reading position of the inactive consumer in the data queue being read.

In a possible implementation, after completing the data migration, the data migration device also includes: a first deletion unit, used to determine whether the consumer has read the first data that has not been migrated in the original data queue, and delete the original data queue according to a set strategy; or, used to determine whether the consumer has not read the first data that has not been migrated in the original data queue, and delete the original data queue.

In a possible implementation, the selection unit is further configured to select a second migration strategy to determine a start migration position when the total number of times is greater than a set number of times, wherein the start migration position of the second migration strategy is the earliest data position of the original data queue.

In a possible implementation, the data migration device further includes: a third determination unit, configured to determine that the migration data queue is copied to the data most recently written by the producer into the original data queue, to complete the data migration.

In a possible implementation manner, the data migration device further includes: a second deleting unit, configured to delete the original data queue after completing the data migration.

In a fourth aspect, an embodiment of the present application provides a data migration device, which includes: an acquisition module, which is used to respond to a migration instruction for migrating data of an original data queue in a first server, and acquire the read position and the latest data position of the earliest active consumer of the original data queue at the current moment, wherein the current moment is the moment when the migration instruction is received, and the latest data position is the position of the data written by the producer to the original data queue at the current moment; a migration module, which is used to copy the data in the original data queue in the order from the latest data position to the position of the data most recently written to the original data queue by the producer during the migration process to the second server to form a migration data queue when the distance between the read position of the earliest active consumer and the latest data position is less than or equal to a set distance; and/or, when the distance between the read position of the earliest active consumer and the latest data position is greater than a set distance, copy the data in the original data queue in the order from the read position of the earliest active consumer to the position of the data most recently written to the original data queue by the producer during the migration process to the second server to form a migration data queue.

In one possible implementation, when migrating data starting from the latest data position, the acquisition module is also used to obtain the reading position of the earliest active consumer of the original data queue during the data migration process; the data migration device also includes: a first determination module, which is used to determine that the reading position of the earliest active consumer of the original data queue during the data migration process enters the second data range of the original data queue when the migration data queue is copied to the data most recently written to the original data queue by the producer, thereby completing the data migration, wherein the second data is the data that has been copied by the migration data queue.

In a possible implementation, when migrating data starting from the reading position of the earliest active consumer, the data migration device also includes: a second determination module, used to determine the data copied from the migration data queue to the data most recently written by the producer to the original data queue to complete the data migration.

In a possible implementation, the acquisition module includes: a first acquisition submodule, used to acquire the reading position of each active consumer in all data queues read, wherein the all data queues include the original data queue; a transfer submodule, used to transfer the mapping structure of each active consumer and the reading position of each active consumer in all data queues read into a mapping structure of each data queue and the reading position of all active consumers in each data queue, thereby obtaining the reading positions of all active consumers of the original data queue; and a comparison acquisition submodule, used to compare the reading positions of all active consumers in the original data queue to obtain the reading position of the earliest active consumer of the original data queue.

In a possible implementation, the acquisition module also includes: a second acquisition submodule, used to acquire the reading position of all consumers in the data queue being read, wherein all consumers include inactive consumers and active consumers; a query submodule, used to query the reading status of all consumers to divide each of all consumers into the active consumer or the inactive consumer, wherein the reading status of the active consumer is working, and the reading status of the inactive consumer is paused; a removal submodule, used to remove the inactive consumer and the information of the reading position of the inactive consumer in the data queue being read.

In a possible implementation, after completing the data migration, the data migration device also includes: a deletion module, used to determine whether the consumer has read the unmigrated first data of the original data queue, and delete the original data queue according to a set strategy, wherein the first data is the data written in the original data queue when the migration instruction is received; or, used to determine whether the consumer has not read the unmigrated first data of the original data queue, and delete the original data queue.

In a fifth aspect, an embodiment of the present application provides a server, comprising: a transceiver for receiving and sending data; a memory storing a computer program; and a processor for executing the computer program stored in the memory so that the server implements the above-mentioned data migration method, wherein the server is the above-mentioned first server or second server.

In a sixth aspect, an embodiment of the present application provides a network system, comprising: a first server and a second server, wherein the first server or the second server is capable of executing the above-mentioned data migration method.

In a seventh aspect, an embodiment of the present application provides a computer storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data migration method is implemented.

In an eighth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned data migration method.

The scheme of the embodiment of the present application can select different migration strategies for different consumption situations of consumers, under the premise of ensuring that upstream and downstream data can be effectively transmitted and no data loss problem occurs. Specifically, when the total number of times the consumer reads the first data of the original data queue is small, that is, the total number is less than or equal to the set number, the first migration strategy can be selected to determine the starting migration position, wherein the first data is the data written in the original data queue when the migration instruction is received, and the starting migration position of the first migration strategy is located downstream of the earliest data position of the original data queue, thereby reducing the number of copies of the first data. Since the data is first stored in the page cache, and the data in the page cache is stored in the disk after a certain period of time, that is, the data that is not copied is generally the data located on the disk, thereby increasing the proportion of data read from the page cache, reducing the probability of real disk reading, thereby reducing the io resource consumption of the disk storing data, reducing the impact of copy migration on normal business, and there is no need to copy all the data in the original data queue, thereby improving the speed of completing the copy migration.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is an architectural diagram of a Kafka cluster;

FIG2 is a schematic diagram of a scenario when the Kafka cluster shown in FIG1 performs data migration;

FIG3 is a process diagram of a data migration method applicable to the Kafka cluster shown in FIG1 ;

FIG4 is a diagram showing changes in disk utilization when migrating data according to the data migration method shown in FIG3 ;

FIG5 is a flow chart of a data migration method provided in an embodiment of the present application;

FIG6 is a specific flow chart of step 531 in FIG5 ;

FIG7 is a process diagram of a first solution of a first migration strategy;

FIG8 is a process diagram of a second solution of the first migration strategy;

FIG9 is a comparison chart of disk utilization when the optimized data migration method according to the embodiment of the present application is used and the original data migration method is used;

FIG10 is a flow chart of another data migration method provided in an embodiment of the present application;

FIG11 is a schematic diagram of the structure of a data migration device provided in an embodiment of the present application;

FIG12 is a schematic diagram of the structure of the selection unit in FIG11;

FIG13 is a schematic diagram of the structure of the acquisition module in FIG12;

FIG14 is a schematic diagram of the structure of another data migration device provided in an embodiment of the present application;

FIG15 is a schematic diagram of the structure of a server provided in an embodiment of the present application;

FIG. 16 is a schematic diagram of the structure of a network system provided in an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

In the description of the embodiments of the present application, the term "and/or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and/or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple screen terminals refers to two or more screen terminals.

In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

Data channel products are mostly used for upstream systems to publish messages and downstream systems to subscribe to and use messages. Messages are stored in the data channel in the form of a first-in-first-out queue, and downstream tasks also sequentially obtain and process data. Message queues generally play the role of decoupling, peak shaving, and asynchronous processing in architectural design. Producers write messages to the queue, and consumers read messages from the queue to perform business logic. Kafka is a high-throughput distributed publish-subscribe messaging system with high performance, persistence, multi-copy backup, and horizontal expansion capabilities. It can handle all action stream data of consumers on the website. Such actions, such as web browsing, searching, and other user actions, are a key factor in many social functions on modern networks. These data are usually solved by processing logs and log aggregation due to throughput requirements.

Figure 1 is an architectural diagram of a Kafka cluster. As shown in Figure 1, a Kafka cluster contains one or more servers, which are called brokers. Each message published to the Kafka cluster has a topic, and the types of messages/data under the same topic are the same. Each topic contains one or more partitions. Partitions are physical concepts, and messages in each partition must be kept in order. Producers are clients that write messages/data to the specified topic of the broker, and consumers are clients that read messages/data from the specified topic from the broker for business processing. In order to achieve horizontal expansion, horizontal expansion can be achieved by increasing the number of partitions. For each new message/data written, Kafka appends a copy of the data to the corresponding file to ensure the high reliability of the original data.

In Figure 1, topic 0 has two partitions, namely partition 0 and partition 1. Each partition has three replica backups. The solid arrow points to the topic to which the producer writes messages/data, which is the leader replica, providing data production and consumption services; the topic pointed to by the dotted arrow is the follower replica, which copies messages/data from the leader replica. When a problem occurs in the leader replica, the follower replica can be selected as the leader replica to provide services. The topics connected by the double-dash arrows are the replicas read by the consumers. When the proxy instance is expanded or reduced in capacity, in order to make the replicas as balanced as possible on each proxy, data migration is required, which is the main scenario used in the solution of the embodiment of the present application.

FIG2 is a schematic diagram of a scenario when the Kafka cluster shown in FIG1 performs data migration. Specifically, the scenario of FIG2 is a schematic diagram when capacity expansion is performed. As shown in FIG2, when capacity expansion is performed, agent 3 can be added, and the data of "Topic 0 Partition 0" in agent 0 and "Topic 0 Partition 1" in agent 1 can be copied to agent 3, as shown by the single-point dashed arrow in FIG2. In addition, when capacity reduction is performed, assuming that the Kafka cluster includes agents 1-5 at this time, each agent includes 4 partitions, and the four partitions in agent 5 can be migrated to agents 1-4 respectively.

FIG3 is a process diagram of a data migration method applicable to the Kafka cluster shown in FIG1. As shown in FIG3, in the data migration method, copying starts from the earliest data position of the original data queue. During the data migration process, the original data queue is still providing services, consumers 1 and 2 are in the working state of reading data, and the producer is still writing new data to the original data queue. The data represented by the black rectangle in the original data queue of the lower figure "Migration Completed" in FIG3 is the data newly written by the producer after the data copy starts. After a long data copy, the data migration is completed when all the data in the original data queue is copied to the new data queue. The migrated data queue starts to provide services, and thereafter the newly written data will be written to the migrated data queue, and the data in the original data queue will be deleted.

FIG4 is a diagram showing changes in disk utilization when data is migrated according to the data migration method shown in FIG3. As shown in FIG3 and FIG4, the data migration method has the following disadvantages:

1. Before the data migration is completed, there is still a large amount of data (such as the data represented by the black rectangle in the original data queue of "Migration Completed" in the lower figure of Figure 3) written into the original data queue. If the migration speed is not much faster than the data writing speed, the data migration process will take a lot of time.

2. During the data copy process, a large amount of historical data reading and writing work occurs, which occupies a large amount of IO resources of the disk storing the data, affecting normal business. In addition, Linux uses page cache to cache recent read and write data, and it is difficult for historical data copied too early to hit the page cache, resulting in real disk reading. As shown in Figure 4, at the beginning of the migration, the utilization rate of the disk rises sharply and remains at a high level during the migration process, which will affect other businesses on the disk.

Since the main function of the message channel product is to provide upstream business data publishing and downstream business data subscription, it is necessary to ensure that the upstream and downstream data can be effectively transmitted and no data will be lost during the data migration process. The embodiment of the present application provides a data migration method, a data migration device, a server and a network system, which can be applied to message channel products such as kafka, and can also be applied to other similar storage systems, such as a distributed file system (Hadoop Distributed FileSystem, HDFS). Under the premise of ensuring that the upstream and downstream data can be effectively transmitted and no data will be lost, the migration strategy can be selected according to the different consumption situations of consumers, which helps to improve the efficiency of data migration and reduce the IO resource consumption of the disk storing data, thereby reducing the impact of replica migration on normal business.

FIG5 is a flow chart of a data migration method provided by an embodiment of the present application. The data migration method is used to migrate data in a data queue, wherein a producer is used to write data to the data queue in sequence, and at least one consumer is used to read data from the data queue in sequence. As shown in FIG5 , the data migration method includes:

Step 51, in response to the migration instruction for migrating the data in the original data queue in the first server, predict the total number of times the consumer reads the first data in the original data queue, wherein the first data is the data written in the original data queue when the migration instruction is received, such as the data represented by the white rectangle in FIG3. In addition, it should be noted that the "total number of times" here refers to the number of times the consumer reads the first data again after receiving the migration instruction, and the total number of times can be predicted based on the historical consumption records of the consumer before receiving the migration instruction and the business type to which the consumer belongs. Specifically, the historical consumption records may include the re-consumption frequency of each consumer and its changing trend. In addition, if the business type of the consumer allows missed reading of data, that is, when the consumer needs to reread the first data, the consumer can be allowed not to read these data, and the number of times the consumer reads can be calculated as 0. Correspondingly, if the business type of the consumer does not allow missed reading of data, the number of times the consumer needs to read the unmigrated first data each time. In addition, the total number of times can be the total number of times different consumers read the first data in the original data queue, and when the same consumer reads the first data multiple times, it is calculated as multiple times; if there is only one consumer reading the original data queue, the total number of times is the number of times the consumer reads the first data in the original data queue.

Step 52, determining whether the total number of times is less than or equal to the set number of times.

The number of times can be set according to work needs, for example, 3 times or 5 times.

Step 53: If the judgment result is yes, then select the first migration strategy to determine the start migration position, wherein the start migration position of the first migration strategy is located downstream of the earliest data position of the original data queue.

Continuing to refer to Figure 3, the "earliest data position of the original data queue" here is the position of the earliest message (i.e., the earliest data) in the original data queue. In Figure 3, the earliest data position is the data position on the far left of the original data queue. "Downstream" refers to the position behind the "earliest message", such as the "latest data position" to be introduced below, which is the position of the data written by the producer to the original data queue. In Figure 3, the "latest data position" is the data position on the far right of the original data queue. Another example is the "earliest active consumer position", which is the position of the active consumer at the front of the original data queue. In Figure 3, the "earliest active consumer position" is the position of consumer 1.

Step 54, copying the data in the original data queue to the second server in a set order to form a migration data queue, wherein the set order is the order from the start migration position of the selected migration strategy to the position of the data most recently written into the original data queue by the producer during the migration process.

In the above scheme, when receiving the migration instruction of the original data queue in the first server, the total number of times the consumer reads the first data of the original data queue again after receiving the migration instruction can be predicted. If the total number is less than or equal to the set number, that is, the number of times the consumer reads the first data of the original data queue is small, the first migration strategy can be selected to determine the start migration position. Since the start migration position of the first migration strategy is located downstream of the earliest data position of the original data queue, there is no need to copy the first data between the earliest data position of the original data queue and the start migration position of the first migration strategy, which can improve the speed of copy migration completion. In addition, the data written by the producer to the original data queue is first stored in the page cache. After a period of time, the data stored in the page cache will be stored in the disk. When migrating the data queue, first check whether there is data in the page cache that needs to be migrated. If there is, migrate the data from the page cache, so that no real disk read occurs. If there is no data in the page cache that needs to be migrated, migrate the data from the disk, and a real disk read will occur. When the start migration position is downstream of the earliest data position in the original data queue, the number of migrations of the first data can be reduced, thereby increasing the proportion of data migrated from the page cache, reducing the probability of real disk reads, and reducing the io resource consumption of the disk storing data. The disk io resources can be mainly used for normal business, reducing the impact of replica migration on normal business.

Specifically, the first migration strategy may include two schemes. The starting migration position of the first scheme is the latest data position of the original data queue; the starting migration position of the second scheme is the reading position of the earliest active consumer in the original data queue. In addition, the second migration strategy to be introduced below includes a scheme, namely the third scheme, whose starting migration position is the earliest data position of the original data queue. Among them, the starting migration positions of the first scheme and the second scheme are located downstream of the earliest data position of the original data queue (the starting migration position of the third scheme).

The first scheme of the data migration method of the embodiment of the present application is first introduced in detail below. In mainstream business scenarios, the location of the consumer's consumption task (i.e., the reading location) is generally close to the latest message/data location written by the producer to the original data queue. At this time, as shown in Figure 5, step 53 selects the first migration strategy to determine the starting migration location, which may specifically include:

Step 531, obtain the reading position and latest data position of the earliest active consumer of the original data queue at the current moment, where the "current moment" here is the moment when the total number is determined to be less than the set number, and the latest data position is the position of the data written by the producer to the original data queue at the current moment.

Step 532, comparing whether the distance between the earliest active consumer's reading position and the latest data position is less than or equal to a set distance;

Step 533, if the comparison result is yes, then determine that the starting migration position of the first migration strategy is the latest data position, that is, the first scheme of the data migration method of the embodiment of the present application. That is to say, determine that the distance between the reading position of the earliest active consumer in the original data queue at the current moment whose total number of times is less than the set number and the latest data position is less than or equal to the set distance, the earliest active consumer is a real-time consumer, and the earliest active consumer will closely follow the latest data. Since the earliest active consumer is the consumer farthest from the latest data position, when it is a real-time consumer, it can be determined that other active consumers are all real-time consumers, and data can be migrated from the latest data position of the original data queue.

Next, step 54 may be executed to copy the data in the original data queue to the second server according to the set sequence to form a migration data queue.

Since the migration starts at the location of the latest data, all or most of the migrated data is migrated from the page cache at this time, which can avoid or greatly reduce the occurrence of real disk reads and reduce the IO resource consumption of the disk storing data. That is, the disk IO resources can be mainly used for normal business, reducing the impact of replica migration on normal business. In addition, the amount of migrated data is small, which can increase the speed of replica migration completion.

Fig. 6 is a specific flow chart of step 531 in Fig. 5. As shown in Fig. 6, obtaining the reading position of the earliest active consumer of the original data queue in step 531 may specifically include:

Step 5314, obtaining the reading position of each active consumer in all the data queues being read, where all the data queues include the original data queue.

Step 5315, transfer the mapping structure between each active consumer and the reading position of each active consumer in all data queues read into a mapping structure between each data queue and the reading position of all active consumers in each data queue, thereby obtaining the reading positions of all active consumers of the original data queue.

Step 5316, compare the read positions of all active consumers in the original data queue to obtain the read position of the earliest active consumer in the original data queue.

Kafka only has an Application Program Interface (API) for a specified consumer group to query the read status of all topic data queues, that is, it can only obtain the data queue read by each consumer in the consumer group (as shown in Figure 1). However, the solution of the embodiment of the present application needs to know the read status of all downstream active consumers in each topic data queue in order to determine which oil mark/starting position (offset) to migrate from and when the migration work is completed. To solve the above problems, the embodiment of the present application adopts the following solution;

First, obtain the information of the subject data queue read by all/all active consumers, which includes the data queue and the reading position. At this point, a mapping structure is formed with the Key value being the active consumer and the Value value being the subject data queue, as shown in Table 1 below. Next, the information is transferred to a mapping structure with the subject data queue as the Key value and all active consumers and their reading positions as the Value value, as shown in Table 2 below, so that the API adds a new service to view the reading positions of all active consumers in the subject data queue. Among them, the API excludes the consumption of downstream inactive consumers. This is because many consumption activities are only temporary consumption and exit after completion, so this situation needs to be excluded to ensure that the reading position of active consumers (ie, valid consumers) is always close to new data.

Key Value Active Consumers 1 Data queue 1 position a, data queue 3 position b Active Consumer 2 Data queue 1 position c, data queue 2 position d Active Consumer 3 Data queue 2 position e, data queue 3 position f

Table 1

Key Value Data queue 1 Active consumer 1 position a, active consumer 2 position c Data Queue 2 Active consumer 2 position d, active consumer 3 position e Data Queue 3 Active consumer 1 position b, active consumer 3 position f

Table 2

That is to say, in this implementation, it is impossible to directly obtain all active consumers existing in the original data queue and the reading positions of these active consumers, but it is possible to obtain the data queue read by each active consumer and the reading position in the data queue. In this way, the mapping structure/correspondence between each active consumer and the reading position of each active consumer in all data queues read can be transferred to the mapping structure/correspondence between each data queue and the reading positions of all active consumers in each data queue, that is, the active consumers existing in each data queue and their reading positions are obtained, and then the reading positions of all active consumers in the original data queue are compared, so that the reading position of the earliest active consumer in the original data queue can be obtained.

Continuing to refer to FIG. 6 , when obtaining the reading position of the earliest active consumer of the original data queue, before performing step 5314 , the following steps may be performed first:

Step 5311, obtaining the reading positions of all consumers in the read data queue, wherein all consumers include inactive consumers and active consumers.

Step 5312, query the reading status of all consumers to classify each consumer as an active consumer or an inactive consumer, wherein the reading status of an active consumer is working, and the reading status of an inactive consumer is paused.

Step 5313, remove the inactive consumers and the information of the reading positions of the inactive consumers in the read data queue.

That is to say, in this implementation, it is not convenient to directly obtain the reading positions of all active consumers in all data queues being read. You can first obtain the reading positions of all consumers in the data queues being read, and all consumers include inactive consumers and active consumers. Then, query the reading status of all consumers to classify each consumer as an active consumer or an inactive consumer. The reading status of an active consumer is working, and the reading status of an inactive consumer is paused. Then, remove the information of the reading positions of inactive consumers and inactive consumers in the data queues being read, and you can obtain the reading position of each active consumer in all data queues being read.

5, after determining in step 533 that the starting migration position of the first migration strategy is the latest data position of the original data queue and migrating the original data queue according to the set order in step 54, the data migration method further includes:

Step 55, obtaining the reading position of the earliest active consumer of the original data queue during the data migration process;

Step 56, determine whether the migration data queue is copied to the data most recently written into the original data queue by the producer;

Step 57, if the judgment result is yes, then judging whether the reading position of the earliest active consumer of the original data queue during the data migration process enters the range of the second data of the original data queue, wherein the second data is the data that has been copied by the migration data queue;

Step 58: If the result of the judgment is yes, the data migration is completed. After the data migration is completed, no new data is written to the original data queue. When the migration data queue is used as the leader copy, the producer starts to write data to the migration data queue; when the migration data queue is used as the follower copy, the migration data queue continues to copy the data in the new leader copy.

Since the starting position of migration is determined to be the latest data position of the original data queue, all consumers in the original data queue are real-time consumers and will move closely following the latest data. In this way, after the migration data queue is copied to the data that the producer has recently written to the original data queue, as long as it is confirmed that the reading position of the earliest active consumer farthest from the latest data position is within the range of the migration data queue, it can be ensured that the reading positions of all active consumers in the original data queue are within the range of the migration data queue, thereby completing the data migration.

In addition, it should be noted that the “earliest active consumer” used in determining whether the migration is complete in step 57 may be the same as or different from the “earliest active consumer” used in determining the start migration position of the first migration strategy in step 532 .

Figure 7 is a process diagram of the first solution of the first migration strategy. Since the consumption progress of downstream consumption tasks in the main use scenario of the data channel generally follows the latest data to ensure the real-time acquisition of data, and a large amount of first data in the original copy has been processed by downstream tasks and will not be consumed again unless there is a special need, therefore, in the first solution, the starting migration position is the latest data position of the original data queue, that is, the position of the data currently written by the producer to the original data queue, as shown in the upper part of Figure 7 "Pre-migration State", when migrating data, the first data before the latest data position (that is, the data represented by the white rectangle in Figure 7) is not copied, and only the latest data written to the original data queue (that is, the data represented by the black rectangle in the original data queue in Figure 7) is copied, which ensures that the migrated copy can immediately follow the latest position of the original copy. At this time, the sign of completion of the replica migration is that the migration data queue copies the latest data to the original data queue, and the reading positions of all active downstream consumers all enter the range of the migration data queue. As shown in the lower part of Figure 7 "Migration Completed", the migration data queue copies the latest data to the original data queue, and the reading positions of consumer 1 and consumer 2 enter the range of the second data that has been copied in the original data queue, thereby ensuring that the migrated replica can effectively provide services after the original data queue (original replica) exits the service.

The migration process of the first solution of the first migration strategy is introduced above with reference to FIG5 to FIG7 , and the migration process of the second solution of the first migration strategy is introduced below. Continuing to refer to FIG5 , at this time, the data migration method includes:

Step 534, if the comparison result of step 532 is no, then determine that the starting migration position of the first migration strategy is the reading position of the earliest active consumer in the original data queue, that is, the second scheme of the data migration method of the embodiment of the present application.

That is to say, after receiving the migration instruction and selecting the first migration strategy, the distance between the reading position of the earliest active consumer of the original data queue and the latest data position is greater than the set distance. Among all active consumers, at least the earliest active consumer is a non-real-time consumer of the original data queue, and it stays in the first data position of the original data queue for a long time. If the data is migrated according to the first solution, the earliest active consumer may not enter the range of the migration data queue quickly. Therefore, the starting migration position of the first migration strategy can be determined as the reading position of the earliest active consumer. This can ensure that the earliest active consumer can always work normally during the data migration process, and there is no need to copy the first data between the earliest data position of the original data queue and the reading position of the earliest active consumer. The speed of completing the copy migration can be improved, and the probability of real disk reading can be reduced, thereby reducing disk io resource consumption and reducing the impact of copy migration on normal business.

Next, step 54 may be executed to copy the data in the original data queue to the second server in a set order to form a migration data queue. Furthermore, when the start migration position of the first migration strategy is determined to be the reading position of the earliest active consumer, the data migration method may further include:

Step 55', determine whether the migration data queue has copied the data most recently written into the original data queue by the producer;

Step 56': if the judgment result is yes, the data migration is completed.

That is to say, for some consumption tasks of non-real-time data channels, their consumption progress may stagnate for a long time. If the first solution is adopted, it may happen that consumers cannot enter the area of the copied second data in the original data queue for a long time. Therefore, a second solution is provided in which the migration position is the reading position of the earliest active consumer. In this way, when the migration data queue is copied to the data most recently written to the original data queue by the producer during the data migration process, it can be ensured that all active consumers are within the range of the migration data queue, and the data migration can be completed.

FIG8 is a process diagram of the second scheme of the first migration strategy. In the second scheme, the starting migration position is the reading position of the earliest active consumer in the original data queue, such as the reading position of consumer 1 shown in the upper part of FIG8 "Pre-migration State". The sign of the completion of the copy migration is that the migrated copy follows the latest written data in the original copy, as shown in the lower part of FIG8 "Migration Completed", and the migrated data queue has been copied to the latest data on the far right of the original data queue. As can be seen from FIG8, the data actually migrated by this scheme only includes the data in the original data that has not been read by the earliest active consumer such as consumer 1 (including the newly written data during the migration process, that is, the data represented by the black rectangle in the original data queue). Compared with the existing migration scheme, the amount of copied data can be greatly reduced while ensuring the normal reading of consumers 1 and 2.

FIG9 is a comparison chart of disk utilization when the optimized data migration method of the embodiment of the present application is used and the original data migration method is used. As shown in FIG9, in the first and second schemes optimized in the embodiment of the present application, only a small part of the data is actually migrated, and because the data of the original copy is the latest written, the data is likely to still exist in the page cache of the operating system. The page cache is a transparent cache of pages from auxiliary storage devices (such as hard disk drives or solid-state drives), so reading the original copy generally does not trigger a real disk read operation, which not only improves the read performance, but also reduces the impact of the disk io impact of the stored data on normal business. While using the original scheme, that is, starting the migration from the earliest data position, the first data copied too early is difficult to hit the page cache, resulting in a real disk read, which will affect the normal business of the disk processing.

In addition, in the first and second solutions above, after the data migration is completed, the migrated copy can take over the work of the original copy, and the timing of deleting the original data queue can be selected according to whether the data has a situation of reading the unmigrated first data of the original data queue and the disk load of the original node. Specifically, referring to FIG5 , after the data migration is completed, the data migration method further includes:

Step 59: determine whether there is a situation of reading the unmigrated first data in the original data queue.

Step 510: If the result of the judgment is yes, the original data queue is deleted according to the set policy. That is, the original data queue does not provide data writing service, and the data will be gradually reduced according to the set policy until it is completely deleted. If the business has a scenario of reading the first data, it can still be used.

Step 511: If the result of the judgment is no, the original data queue is deleted, so that space can be released immediately and the cluster load can be reduced.

That is to say, since the starting migration position of the first migration strategy is located downstream of the earliest data position of the original data queue, when the first migration strategy is selected to determine the starting migration position, the data between the earliest data position in the original data queue and the starting migration position are not copied to the migration data queue. After completing the data migration, it is necessary to determine whether the unmigrated first data in the original data queue has been read by the consumer. If it has not been read, that is, the number of times the unmigrated first data of the original data queue is read after the migration is completed is 0, the original data queue can be deleted immediately; if the unmigrated first data of the original data queue is read after the migration is completed, that is, the number of times the unmigrated first data of the original data queue is read after the migration is completed is not 0, but less than or equal to the set number, for example, it can be 1 or 2 times, then the original data queue can be deleted according to the set strategy, for example, the original data queue is deleted after the consumer reads the unmigrated first data, to ensure that it will not affect normal work.

The above two schemes of the first migration strategy of the data migration method of the embodiment of the present application are introduced with reference to Figures 5 to 9. The scheme of the second migration strategy of the data migration method of the embodiment of the present application is introduced below. Specifically, continuing to refer to Figure 5, the data migration method also includes:

Step 53', if the judgment result of step 52 is no, that is, the total number of times the consumer reads the first data of the original data queue is greater than the set number, then the second migration strategy is selected to determine the starting migration position, wherein the starting migration position of the second migration strategy is the earliest data position of the original data queue, that is, the third scheme of the data migration method of the embodiment of the present application.

That is to say, although the first and second schemes of the first migration strategy can delete the original data queue according to the set strategy after completing the data migration to ensure that the consumer can read the unmigrated first data in the original data queue, the consumption of switching the consumer from the migrated data queue in the second server to the original data queue in the first server is relatively large. Therefore, when the consumer reads the first data in the original data queue many times, in order to reduce consumption, the second migration strategy can be selected so as to start migration from the earliest data position in the original data queue to ensure that the consumer's reading business can proceed normally.

In addition, at this time, the data migration method may further include:

Step 55”, determine whether the migration data queue has copied the data most recently written into the original data queue by the producer;

In step 56, if the judgment result is yes, the data migration is completed.

That is to say, when the second migration strategy is selected to start migration from the earliest data position of the original data queue, when the migration data queue copies the data most recently written by the producer to the original data queue, it can be guaranteed that the migration data queue copies all the data to the original data queue, and the data migration is completed.

In step 57, the original data queue is deleted after the data migration is completed.

Since the starting migration position of the second migration strategy is the earliest data position of the original data queue, the migrated data queue copies the data most recently written by the producer to the original data queue. After the data migration is completed, the migrated data queue can copy all the data of the original data queue. Therefore, after the data migration is completed, the original data queue can be deleted immediately to free up memory.

The data migration method provided in the embodiment of the present application can select different migration plans according to different consumption situations of consumers, while ensuring that upstream and downstream data can be effectively transmitted and no data loss problem occurs. This helps to improve the speed of completing the copy migration, reduce the IO resource consumption of the disk storing data, and thus reduce the impact of the copy migration on normal business.

In summary, the technical solution claimed for protection in the embodiments of this application is introduced by taking Kafka, the most widely used message channel product, as an example. The key technical points are as follows:

1. Determine the position of the original leader replica from which to start migration based on the situation of consumers reading data in the original data queue, that is, select the starting migration position. Specifically, the starting migration position can be one of the latest data position of the original data queue (the first solution), the reading position of the earliest active consumer in the original data queue (the second solution), and the earliest data position of the original data queue (the third solution).

2. Obtain the reading position of the earliest active consumer in the original data queue, see steps 5314 to 5316 or steps 5311 to 5316.

3. Criteria for judging the completion of data migration

For the first solution, first, it is necessary to determine whether the migrated data queue has copied the data most recently written by the producer to the original data queue (i.e., the leader copy). When the judgment result is yes, then, it is necessary to call the application interface API introduced above to confirm that the reading position of the earliest active consumer in the migration process enters the range of the second data of the original data queue that has been copied by the migrated data queue, thereby completing the data migration.

For the second and third solutions, the data migration is completed when the migration data queue is copied to the data that the producer has most recently written into the original data queue during the migration process.

4. Whether to delete the original data queue

For the two schemes of the first migration strategy, determine whether to read the unmigrated first data in the original data queue. If so, delete the original data queue according to the set strategy, that is, after the migration is completed, the deletion strategy of the original data queue follows Kafka's own deletion strategy, such as only saving the latest data at a fixed time or data files of a fixed size; if there is no reading situation, the original data queue can be deleted immediately after the data migration is completed.

For the third solution, since the migrated data queue has copied all the data of the original data queue, the original data queue can be deleted immediately after the data migration is completed, so as to release the disk space as soon as possible.

FIG10 is a flow chart of another data migration method provided by an embodiment of the present application. The data migration method is used to migrate data in a data queue, wherein a producer is used to write data to the data queue in sequence, and at least one consumer is used to read data from the data queue in sequence. As shown in FIG10 , the data migration method includes:

Step 1001, in response to a migration instruction for migrating data in an original data queue in a first server, obtain a read position and a latest data position of the earliest active consumer of the original data queue at the current moment, wherein the "current moment" here is the moment when the migration instruction is received, and the latest data position is the position of the data written by the producer into the original data queue at the current moment;

Step 1002, comparing whether the distance between the earliest active consumer's reading position and the latest data position is less than or equal to a set distance;

Step 1003, if the comparison result is yes, the data in the original data queue is copied to the second server in the order from the latest data position to the position of the data most recently written to the original data queue by the producer during the migration process to form a migration data queue, that is, the starting migration position is the latest data position of the original data queue; and/or,

Step 1004, if the comparison result is no, the data in the original data queue is copied to the second server in the order from the reading position of the earliest active consumer to the position of the data most recently written into the original data queue by the producer during the migration process to form a migration data queue, that is, the starting migration position is the reading position of the earliest active consumer of the original data queue.

In the above scheme, at the current moment when the migration instruction is received, if the distance between the reading position of the earliest active consumer in the original data queue and the latest data position is less than or equal to the set distance, the active consumer is a real-time consumer that follows the latest data, and the data can be migrated from the latest data position of the original data queue; if the distance between the reading position of the earliest active consumer in the original data queue and the latest data position is greater than the set distance, the active consumer stays at the first data of the original data queue for a long time and is a non-real-time consumer of the original data queue, and the data can be migrated from the reading position of the earliest consumer in the original data queue, and the data of the original data queue in the first server is migrated from the beginning to the end. The data queue is copied to the second server in sequence from the starting migration position to the position of the data most recently written by the producer to the original data queue during the migration process to form a migration data queue. Compared with the scheme of migrating data from the earliest data position, since the data newly written by the producer to the original data queue is first stored in the page cache and then stored in the disk from the page cache after a certain period of time, the data migration method of the embodiment of the present application can increase the ratio of copying data from the page cache for migration, reduce the number of first data copied from the disk, reduce the probability of real disk reading, reduce the impact of copy migration on normal business, and since it is not necessary to migrate all the data in the original data queue, it can increase the speed of completing the copy migration.

In addition, when migrating data from the latest data position of the original data queue, the data migration method further includes:

Step 1005, obtaining the reading position of the earliest active consumer of the original data queue during the data migration process;

Step 1006, determining whether the migration data queue has copied the data most recently written into the original data queue by the producer;

Step 1007, if the result of the determination is yes, then determining whether the reading position of the earliest active consumer of the original data queue during the data migration process enters the second data range of the original data queue, where the second data is the data that has been copied by the migration data queue;

Step 1008: If the judgment result is yes, the data migration is completed.

Since the starting position of migration is the latest data position of the original data queue, all active consumers in the original data queue are real-time consumers. When the migrated data queue is copied to the data that the producer has recently written into the original data queue during the migration process, as long as the reading position of the earliest active consumer farthest from the latest data position during the migration process enters the range of the second data of the original data queue that has been copied by the migrated data queue, it can be ensured that the reading positions of all active consumers of the original data queue are within the range of the migrated data queue, and the data migration can be completed. The specific migration process can be seen in Figure 7.

When migrating data from the read position of the earliest active consumer in the original data queue, the data migration method further includes:

Step 1005', determine whether the migration data queue has copied the data most recently written into the original data queue by the producer;

Step 1006', if the judgment result is yes, the data migration is completed.

Since the starting position of migration is the reading position of the earliest active consumer, when the migration data queue is copied to the data most recently written by the producer to the original data queue during the data migration process, all active consumers must be within the range of the migration data queue, and the data migration can be completed. The specific migration process can be seen in Figure 8.

It should be noted that the process of "obtaining the earliest active consumer's reading position of the original data queue" at step 1001 and step 1005 can be performed according to the steps 5314 to 5316 or steps 5311 to 5316 described above. In addition, as shown in FIG10 , after completing the data migration at step 1008 or step 1006', the data migration method further includes:

Step 1009 , determining whether there is a situation of reading the unmigrated first data in the original data queue, wherein the first data is the data written in the original data queue when the migration instruction is received.

Step 1010: If the judgment result is yes, the original data queue is deleted according to the set strategy.

Step 1011, if the judgment result is no, the original data queue is deleted.

That is to say, after completing the data migration, it is necessary to determine whether the first data that has not been migrated in the original data queue has been read by the consumer. If it has not been read, the original data queue can be deleted immediately; if it has been read, the original data queue can be deleted according to the original set strategy, for example, the original data queue can be deleted after the consumer reads the first data that has not been migrated, to ensure that it will not affect normal work.

The data migration method of the embodiment of the present application can perform limited data copying according to user decision when copying the replica data, thereby improving the migration efficiency and reducing the impact of migration under the premise of satisfying data reliability. In addition, after the data migration is completed, two solutions for deleting the original data are also provided: 1. Delete the original data queue according to the original set strategy, so that the business can be replayed in data; 2. Immediately delete the original data queue, so that the disk capacity can be released.

FIG11 is a schematic diagram of the structure of a data migration device provided by an embodiment of the present application. The data migration device 1100 is used to migrate data in a data queue, wherein the producer is used to write data to the data queue in sequence, and at least one consumer is used to read data from the data queue in sequence. As shown in FIG11, the data migration device 1100 includes a prediction unit 1101, a selection unit 1102, and a migration unit 1103. The prediction unit 1101 is used to respond to a migration instruction for migrating data in the original data queue in the first server, and predict the total number of times the consumer reads the first data in the original data queue, wherein the first data is the data written in the original data queue when the migration instruction is received. The selection unit 1102 is used to select a first migration strategy to determine the start migration position when the total number of times is less than or equal to the set number of times, wherein the start migration position of the first migration strategy is located downstream of the earliest data position of the original data queue. The migration unit 1103 is used to copy the data in the original data queue to the second server in a set order to form a migration data queue, wherein the set order is the order from the start migration position of the selected migration strategy to the position of the data written to the original data queue by the producer most recently during the migration process.

FIG12 is a schematic diagram of the structure of the selection unit in FIG11. As shown in FIG12, the selection unit 1102 includes an acquisition module 21 and a determination module 22. The acquisition module 21 is used to obtain the read position and the latest data position of the earliest active consumer of the original data queue at the current moment, wherein the "current moment" here is the moment when the total number of times is determined to be less than the set number of times, and the latest data position is the position of the data written by the producer to the original data queue at the current moment. The determination module 22 is used to determine that the starting migration position of the first migration strategy is the latest data position of the original data queue when the distance between the read position of the earliest active consumer in the original data queue and the latest data position is less than or equal to the set distance.

When the determination module 22 determines that the starting migration position of the first migration strategy is the latest data position of the original data queue, the acquisition module 21 is also used to obtain the reading position of the earliest active consumer of the original data queue during the data migration process. As shown in Figure 11, the data migration device 1100 also includes a first determination unit 1104, which is used to determine that the reading position of the earliest active consumer of the original data queue during the data migration process enters the second data range of the original data queue when the migration data queue is copied to the data most recently written to the original data queue by the producer, thereby completing the data migration, wherein the second data is the data that has been copied to the migration data queue.

Furthermore, the determination module 22 can also be used to determine that the start migration position of the first migration strategy is the read position of the earliest active consumer when the distance between the read position of the earliest active consumer and the latest data position is greater than a set distance. When the determination module 22 determines that the start migration position of the first migration strategy is the read position of the earliest active consumer, as shown in FIG11, the data migration device 1100 can also include a second determination unit 1105, which is used to determine the data copied from the migration data queue to the latest data written to the original data queue by the producer to complete the data migration.

FIG13 is a schematic diagram of the structure of the acquisition module in FIG12. As shown in FIG13, the acquisition module 21 may include a first acquisition submodule 211, a transfer submodule 212, and a comparison acquisition submodule 213. The first acquisition submodule 211 is used to obtain the reading position of each active consumer in all data queues read, and all data queues include the original data queue. The transfer submodule 212 is used to transfer the mapping structure of each active consumer and the reading position of each active consumer in all data queues read into a mapping structure of each data queue and the reading position of all active consumers in each data queue, thereby obtaining the reading positions of all active consumers in the original data queue. The comparison acquisition submodule 213 is used to compare the reading positions of all active consumers in the original data queue to obtain the reading position of the earliest active consumer of the original data queue.

Furthermore, the acquisition module 21 may also include a second acquisition submodule 214, a query submodule 215, and a removal submodule 216. The second acquisition submodule 214 is used to obtain the reading position of all consumers in the read data queue, wherein all consumers include inactive consumers and active consumers. The query submodule 215 is used to query the reading status of all consumers to classify each consumer as an active consumer or an inactive consumer, wherein the reading status of an active consumer is working, and the reading status of an inactive consumer is paused. The removal submodule 216 is used to remove the information of the reading position of inactive consumers and inactive consumers in the read data queue, thereby obtaining the reading position of the earliest active consumer in the original data queue.

Continuing to refer to Figure 11, when the selection unit 1102 selects the first migration strategy to determine the starting migration position, the data migration device 1100 also includes a first deletion unit 1106, which is used to determine, after the data migration is completed, whether the consumer has read the unmigrated first data in the original data queue, and delete the original data queue according to the set strategy; and is used to determine whether the consumer has not read the unmigrated first data in the original data queue, and delete the original data queue.

In addition, the selection unit 1102 is also used to select a second migration strategy to determine the starting migration position when the total number of times is greater than the set number of times, wherein the starting migration position of the second migration strategy is the earliest data position of the original data queue. When the selection unit 1102 selects the second migration strategy to determine the starting migration position, as shown in FIG11, the data migration device 1100 may also include a third determination unit 1107, which is used to determine the data that the migration data queue copies to the latest data written to the original data queue by the producer to complete the data migration. Furthermore, the data migration device 1100 may also include a second deletion unit 1108, which is used to delete the original data queue after completing the data migration.

According to the data migration device of the embodiment of the present application, under the premise of ensuring that the upstream and downstream data can be effectively transmitted and no data loss problem occurs, the migration scheme can be selected according to the different consumption situations of consumers. Specifically, when the total number of times the consumer reads the first data of the original data queue is small, that is, the total number is less than or equal to the set number, the first migration strategy can be selected to determine the start migration position, and the start migration position of the first migration strategy is located downstream of the earliest data position of the original data queue. When the distance between the reading position of the earliest active consumer in the original data queue and the latest data position is less than or equal to the set distance, the start migration position of the first migration strategy is determined to be the latest data position of the original data queue, that is, the first scheme; when the distance between the reading position of the earliest active consumer in the original data queue and the latest data position is greater than the set distance, the start migration position of the first migration strategy is determined to be the reading position of the earliest active consumer in the original data queue, that is, the second scheme. Since it is not necessary to copy all the data of the original data queue in the first scheme and the second scheme, that is, the number of copies of the first data is reduced, the speed of completing the copy migration is improved, the io resource consumption of the disk storing the data is reduced, and the impact of the copy migration on normal business is reduced. In addition, after the data migration is successful, two methods of deleting the original data are also provided, namely, deleting the original data queue according to the set policy and deleting the original data queue immediately. If you choose to delete the original data queue according to the set policy, the data can be replayed, that is, the consumer can read the first data again. If you choose to delete the original data immediately, the disk capacity can be released as soon as possible, so as to facilitate the business to make a choice between data replayability and disk capacity. When the total number of times the consumer reads the first data in the original data queue after the data migration is completed is large, that is, when the total number is greater than the set number, the second migration strategy can be selected to determine the starting migration position. The starting migration position of the second migration strategy is the earliest data position of the original data queue, that is, the third solution. In this way, the migration data queue copies all the data in the original data queue. After the migration is completed, the original data queue can be deleted immediately to release space, and it can also meet the consumer's need to read the first data again.

Figure 14 is a schematic diagram of the structure of another data migration device provided in an embodiment of the present application. The data migration device is used to migrate data in a data queue, wherein a producer is used to write data to the data queue in sequence, and at least one consumer is used to read data from the data queue in sequence. As shown in Figure 14, the data migration device 1400 includes an acquisition module 1401 and a migration module 1402. The acquisition module 1401 is used to respond to the migration instruction of the original data queue in the first server, and obtain the reading position and the latest data position of the earliest active consumer of the original data queue at the current moment, wherein the "current moment" here is the moment when the migration instruction is received, and the latest data position is the position of the data written by the producer to the original data queue at the current moment. Migration module 1402 is used to copy the data in the original data queue to the second server in the order from the latest data position to the position of the data most recently written to the original data queue by the producer during the migration process to form a migration data queue when the distance between the reading position of the earliest active consumer in the original data queue and the latest data position is less than or equal to the set distance, that is, the migration position starts at the latest data position at this time; and/or, when the distance between the reading position of the earliest active consumer in the original data queue and the latest data position is greater than the set distance, copy the data in the original data queue to the second server in the order from the reading position of the earliest consumer to the position of the data most recently written to the original data queue by the producer during the migration process to form a migration data queue, that is, the migration position starts at the reading position of the earliest consumer. Among them, the specific structure of the acquisition module 1401 can be the same as the specific structure of the acquisition module 21 mentioned above.

When migrating data from the latest data position of the original data queue, the acquisition module 1401 is also used to obtain the reading position of the earliest active consumer of the original data queue during the data migration process. The data migration device 1400 also includes a first determination module 1403, which is used to determine that the reading position of the earliest active consumer of the original data queue during the data migration process enters the second data range of the original data queue when the migration data queue is copied to the data that the producer has recently written to the original data queue, and complete the data migration, wherein the second data is the data that has been copied by the migration data queue.

When migrating data from the earliest consumer's reading position in the original data queue, the data migration device 1400 further includes a second determination module 1404 for determining the data copied from the migrated data queue to the latest data written into the original data queue by the producer to complete data migration.

Furthermore, after completing the data migration, the data migration device 1400 may also include a deletion module 1405, which is used to determine whether the consumer has read the unmigrated first data of the original data queue, and delete the original data queue according to the set strategy, wherein the first data is the data written in the original data queue when the migration instruction is received; and is used to determine whether the consumer has not read the unmigrated first data of the original data queue, and delete the original data queue immediately.

The solution of the embodiment of the present application is based on the downstream data reading status, determines the migration starting position and migration end conditions, and ensures that the downstream task data consumption is not lost, and can quickly complete the data copying work of the message channel. For example, for a typical 200G copy, the original migration solution takes 2-3 hours. Using the solution of the embodiment of the present application, the migration can be completed within ten minutes, greatly improving the migration efficiency. At the same time, the data copy process reduces a large amount of disk reading and writing, reducing the impact on normal business. In addition, two deletion schemes for the original data copy are determined according to business needs. The scheme of deleting the original copy according to the set strategy makes the data replayable and consumable; and the scheme of immediately deleting the original copy can achieve space optimization.

In addition, since in a system such as a Kafka cluster, information related to producers, consumers, and data queues is shared among multiple servers, the above-mentioned data migration method can be executed by the first server or by the second server. Figure 15 is a schematic diagram of the structure of a server provided in an embodiment of the present application. As shown in Figure 15, the server 1500 includes a transceiver 1501, a memory 1502, and a processor 1503. Among them, the transceiver 1501 is used to receive and send data. The memory 1502 stores a computer program. The processor 1503 is used to execute the computer program stored in the memory 1502, so that the server 1500 implements the data migration method of an embodiment of the present application. Among them, the server is the first server or the second server mentioned in the above-mentioned data migration method.

Figure 16 is a schematic diagram of the structure of a network system provided in an embodiment of the present application. As shown in Figure 16, the network system includes at least two servers, namely server A and server B, the original data queue is located in server A, and the migration data queue is located in server B. Among them, server A or server B can execute the data migration method of the embodiment of the present application.

It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or software-based instructions in a processor. The processor can be a CPU, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims (22)

1. A data migration method, characterized in that the data migration method comprises:

Responding to a migration instruction for migrating data of an original data queue in a first server, and predicting the total number of times that a consumer reads first data of the original data queue, wherein the first data is written in the original data queue when the migration instruction is received;

when the total number of times is less than or equal to a set number of times, selecting a first migration strategy to determine a migration starting position, wherein the migration starting position of the first migration strategy is positioned at the downstream of the earliest data position of the original data queue;

Copying the data in the original data queue to a second server according to a set sequence to form a migration data queue, wherein the set sequence is from a migration starting position of the selected migration strategy to a position of the data which is newly written into the original data queue by a producer in the migration process.

2. The data migration method according to claim 1, wherein the selecting the first migration policy to determine the migration start location specifically comprises:

Acquiring a reading position and a latest data position of the earliest active consumer of the original data queue at the current moment, wherein the current moment is a moment when the total times are smaller than the set times, and the latest data position is a position of data written into the original data queue by the producer at the current moment;

and when the distance between the reading position of the earliest active consumer and the latest data position is smaller than or equal to a set distance, determining the starting migration position of the first migration strategy as the latest data position.

3. The data migration method according to claim 2, wherein when the start migration position is the latest data position, the data migration method further comprises:

Acquiring the reading position of the earliest active consumer of the original data queue in the data migration process;

When the migrated data queue is copied to the data which is newly written into the original data queue by the producer, determining that the reading position of the earliest active consumer of the original data queue enters the range of second data of the original data queue in the data migration process, and completing data migration, wherein the second data is the copied data of the migrated data queue.

4. The data migration method of claim 2, further comprising:

And when the distance between the reading position of the earliest active consumer and the latest data position is larger than a set distance, determining the starting migration position of the first migration strategy as the reading position of the earliest active consumer.

5. The data migration method of claim 4, wherein when the start migration location is a read location of the earliest active consumer, the data migration method further comprises:

And determining that the data which is copied to the original data queue by the migration data queue is latest written into the original data queue by the producer, and finishing data migration.

6. The data migration method according to any one of claims 2 to 5, wherein said obtaining a read location of an earliest active consumer of said original data queue at a current time comprises:

Acquiring reading positions of each active consumer in all read data queues, wherein all the data queues comprise the original data queue;

The mapping structure of the reading positions of each active consumer and each active consumer in the read all data queues is transferred into the mapping structure of the reading positions of each data queue and all active consumers in each data queue, so that the reading positions of all active consumers in the original data queues are obtained;

and comparing the read positions of all active consumers in the original data queue to obtain the read position of the earliest active consumer in the original data queue.

7. The data migration method of claim 6, wherein prior to said obtaining the read locations of each active consumer in all of the read data queues, said obtaining the read location of the earliest active consumer of the original data queue at the current time further comprises:

Obtaining read positions of all consumers in the read data queue, wherein all consumers comprise inactive consumers and active consumers;

Querying the read status of the all consumers to divide each of the all consumers into the active consumer or the inactive consumer, wherein the read status of the active consumer is in operation and the read status of the inactive consumer is suspended;

and removing information of the inactive consumers and the reading positions of the inactive consumers in the read data queue.

8. The data migration method according to any one of claims 1 to 5, further comprising, after completion of data migration:

determining that the consumer has the condition of reading the first data which is not migrated in the original data queue, and deleting the original data queue according to a set strategy; or alternatively, the first and second heat exchangers may be,

Determining that the consumer does not read the first data of the original data queue which is not migrated, and deleting the original data queue.

9. The data migration method of claim 1, wherein the data migration method comprises:

And when the total times are greater than the set times, selecting a second migration strategy to determine a starting migration position, wherein the starting migration position of the second migration strategy is the earliest data position of the original data queue.

10. The data migration method of claim 9, further comprising:

And determining that the data which is copied to the original data queue by the migration data queue is latest written into the original data queue by the producer, and finishing data migration.

11. The data migration method according to claim 9 or 10, further comprising:

and deleting the original data queue after finishing data migration.

12. A data migration method, characterized in that the data migration method comprises:

Responding to a migration instruction for migrating data of an original data queue in a first server, and acquiring a reading position and a latest data position of an earliest active consumer of the original data queue at the current moment, wherein the current moment is the moment when the migration instruction is received, and the latest data position is the position of the data written into the original data queue by a producer at the current moment;

copying the data in the original data queue to a second server according to the sequence from the latest data position to the position of the data which is newly written into the original data queue by a producer in the migration process when the distance between the reading position of the earliest active consumer and the latest data position is smaller than or equal to a set distance so as to form a migration data queue; and/or the number of the groups of groups,

And when the distance between the reading position of the earliest active consumer and the latest data position is larger than a set distance, copying the data in the original data queue to a second server according to the sequence from the reading position of the earliest active consumer to the position of the data which is newly written into the original data queue by a producer in the migration process so as to form a migration data queue.

13. The data migration method of claim 12, wherein when migrating data from the latest data location, the data migration method comprises:

Acquiring the reading position of the earliest active consumer of the original data queue in the data migration process;

When the migrated data queue is copied to the data which is newly written into the original data queue by the producer, determining that the reading position of the earliest active consumer of the original data queue enters a second data range of the original data queue in the data migration process, and completing data migration, wherein the second data is the copied data of the migrated data queue.

14. The data migration method of claim 12, wherein when migrating data from a read location of the earliest active consumer, the data migration method further comprises:

And determining that the data which is copied to the original data queue by the migration data queue is latest written into the original data queue by the producer, and finishing data migration.

15. The data migration method according to any one of claims 12-14, wherein said obtaining a read location of an earliest active consumer in said raw data queue at a current time comprises:

Acquiring reading positions of each active consumer in all read data queues, wherein all the data queues comprise the original data queue;

The mapping structure of the reading positions of each active consumer and each active consumer in the read all data queues is transferred into the mapping structure of the reading positions of each data queue and all active consumers in each data queue, so that the reading positions of all active consumers in the original data queues are obtained;

and comparing the read positions of all active consumers in the original data queue to obtain the read position of the earliest active consumer in the original data queue.

16. The data migration method of claim 15, wherein prior to said obtaining the read locations of each active consumer in all of the read data queues, said obtaining the read location of the earliest active consumer of the original data queue at the current time further comprises:

obtaining reading positions of all consumers in the read data queue, wherein all consumers comprise inactive consumers and active consumers;

Querying the read status of the all consumers to divide each of the all consumers into the active consumer or the inactive consumer, wherein the read status of the active consumer is in operation and the read status of the inactive consumer is suspended;

and removing information of the inactive consumers and the reading positions of the inactive consumers in the read data queue.

17. The data migration method according to any one of claims 12 to 14, wherein after completion of data migration, the data migration method further comprises:

Determining that the consumer has the condition of reading first data which is not migrated in the original data queue, and deleting the original data queue according to a set strategy, wherein the first data is written in the original data queue when the migration instruction is received; or alternatively, the first and second heat exchangers may be,

Determining that the consumer does not read the first data of the original data queue which is not migrated, and deleting the original data queue.

18. A data migration apparatus, the data migration apparatus comprising:

the system comprises a prediction unit, a first server and a second server, wherein the prediction unit is used for responding to a migration instruction of data of an original data queue in the first server and predicting the total number of times of reading first data of the original data queue by a consumer, wherein the first data is written in the original data queue when the migration instruction is received;

A selecting unit, configured to select a first migration policy to determine a migration start position when the total number of times is less than or equal to a set number of times, where the migration start position of the first migration policy is located downstream of an earliest data position of the original data queue;

And the migration unit is used for copying the data in the original data queue to the second server according to a set sequence to form a migrated data queue, wherein the set sequence is from the starting migration position of the selected migration strategy to the position of the data which is newly written into the original data queue by a producer in the migration process.

19. A data migration apparatus, the data migration apparatus comprising:

The system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for responding to a migration instruction for migrating data of an original data queue in a first server, and acquiring a reading position and a latest data position of an earliest active consumer of the original data queue at the current moment, wherein the current moment is the moment when the migration instruction is received, and the latest data position is the position of the data written into the original data queue by a producer at the current moment;

The migration module is used for copying the data in the original data queue to a second server according to the sequence from the latest data position to the position of the data which is newly written into the original data queue by a producer in the migration process when the distance between the reading position of the earliest active consumer and the latest data position is smaller than or equal to a set distance so as to form a migration data queue; and/or copying the data in the original data queue to a second server in order from the reading position of the earliest active consumer to the position of the data which is newly written into the original data queue by a producer in the migration process when the distance between the reading position of the earliest active consumer and the latest data position is larger than a set distance, so as to form a migration data queue.

20. A server, comprising:

A transceiver for receiving and transmitting data;

A memory storing a computer program;

A processor configured to execute the computer program stored in the memory, so as to cause the server to implement the data migration method according to any one of claims 1 to 17, wherein the server is the first server or the second server.

21. A network system comprising a first server and a second server, wherein the first server or the second server is capable of performing the data migration method of any one of claims 1-17.

22. A computer storage medium, characterized in that the computer storage medium has stored therein a computer program which, when executed by a processor, implements the data migration method according to any one of claims 1-17.