CN116701426A - Data processing method, electronic device and storage medium - Google Patents

Abstract

The embodiment of the application relates to the technical field of information, in particular to a data processing method, electronic equipment and a storage medium. In the method, the electronic device can determine the log file with missing content in the acquired log files. Next, the electronic device may obtain a log flag bit based on the log file of the content deletion. And obtaining the log file based on the log flag bit in the next data processing. Thus, when the log file is acquired next time, the electronic device can acquire the log file with the determined content missing again. The method can relieve the phenomenon of leakage in the data processing process, and further can improve the consistency of the data of the source end and the target end, thereby reducing the problem of data distortion of the target end.

Description

Data processing method, electronic device and storage medium

technical field

The embodiments of the present application relate to the field of information technology, and specifically relate to a data processing method, electronic equipment, and a storage medium.

Background technique

Data integration refers to centralizing data logically or physically; for example, integrating the data in the source database into the target database through the log files of the source database.

When obtaining log files from the source database, there are cases where log files with missing content are obtained, which will cause the log data in these log files with missing content to not be integrated into the target database, that is, the phenomenon of missing numbers will occur. The database at the source end is inconsistent with the database at the target end, resulting in data distortion at the target end.

Contents of the invention

The embodiment of the present application provides a data processing method, electronic equipment and storage medium, which can alleviate the phenomenon of missing numbers in the data integration process, and further improve the data consistency between the source end and the target end, thereby reducing the problem of data distortion at the target end .

In order to achieve the above object, the embodiments of the present application adopt the following technical solutions:

In a first aspect, the present application provides a data processing method, which can be applied to electronic devices in a data storage system. Wherein, the electronic device may be, for example, a personal computer, a notebook computer, etc., which have data transmission and processing capabilities; the data storage system includes a source-end database and a target-end database. The method includes: the electronic device obtains the log file from the source database according to the first identification range, the first identification range is used to indicate the log data identification range of the log file to be obtained by the electronic device; the obtained log file is used for the target End database for data manipulation. Next, if the electronic device determines that there is at least one missing log file from the obtained log files, the electronic device obtains the log flag bit based on the starting log data identifier and the first identification range of the at least one missing log file. The aforementioned missing log file is a log file in which included log data is missing. Then, the electronic device determines a second identification range according to the log flag bit, and obtains the log file from the source database according to the second identification range. Wherein, the starting log data identifier of the missing log file is the log data identifier of the first piece of log data included in the missing log file.

In the above method, the electronic device determines the missing log file from the log file obtained for the first time (that is, the log file obtained using the first identification range), and obtains the log flag bit according to the missing log file. And obtain the second identification range according to the flag bit, and obtain the log file from the source database according to the second identification range (that is, obtain the log file from the source database for the second time). In this way, the missing log files in the log files obtained for the first time can be obtained from the source database for the second time through the log flag bit and the second identification range; to alleviate the phenomenon of missing numbers in the data integration process, It can improve the data consistency between the source end and the target end, thereby reducing the data distortion problem at the target end.

In a possible design of the first aspect, the above-mentioned electronic device obtains the log flag bit based on the starting log data identifier of at least one missing log file and the first identifier range, which may include: If it is determined that there is a missing log file in the log file, the electronic device will use the log data identifier with the largest (numerical value) among the start log data identifier of the above-mentioned one missing log file and the range start point of the first identifier range as the log flag bit. Or, when the electronic device determines that there are at least two missing log files from the log files obtained according to the first identification range, the electronic device determines the target missing log file in the at least two missing log files; Among the starting point log data identifier and the range starting point of the first identifier range, the (numerical) largest log data identifier is used as the log flag bit; wherein, the target missing log file is the missing log with the smallest starting point log data identifier among at least two missing log files file, the starting log data identifier of the target missing log file is the log data identifier of the first piece of log data included in the target missing log file.

It can be understood that the log flag bit is positively related to the start point of the second identification range; and the log data identification is related to the generation time of the log data. That is to say, the smaller the log identification bit, the smaller the starting point of the second identification range; the more log files the electronic device obtains from the source database by using the second identification range, and the more log files the electronic device obtains The earlier the generation time of the log data in the file is, the greater the difference from the current time of the electronic device will be. It can be seen that the smaller the log flag, the lower the real-time performance of data integration when the electronic device obtains the log file through the second identification range. In this design, by setting the log data identification bit as large as possible, the electronic device can obtain the log file from the source end for the second time, and the electronic device can obtain log files from the source end database for the first time. When the log file is determined, the missing log file can make the number of log files obtained by the electronic device relatively small, and can also control the timeliness of obtaining log files from the source database to a certain extent. In this way, the problem of missing numbers during data integration can be alleviated, and the real-time performance of data integration by electronic equipment can be improved.

In another possible design of the first aspect, the above-mentioned electronic device determines that there is at least one missing log file from the obtained log file, which may include: reading log data from the obtained log file, and if there is read If there are no log files with no log data, it is determined that there is at least one missing log file.

In another possible design of the first aspect, the electronic device determines from the obtained log files that there is at least one missing log file, and may further include: reading the log from the log files obtained based on the first identification range data, the log files with no log data can be read as missing log files.

In yet another possible design of the first aspect, the above-mentioned electronic device obtaining log files from the source database according to the first identification range may include: if the electronic device does not obtain each log file corresponding to the first identification range, Then the electronic device acquires log files again according to the first identification range until each log file corresponding to the first identification range is acquired. Afterwards, the electronic device performs data operations on the target database based on each acquired log file corresponding to the first identification range.

It can be understood that, considering that the electronic device may not be able to obtain every log file corresponding to the first identification range during the process of the electronic device acquiring log files from the source database according to the first identification range. Based on this, in this design, the electronic device can determine whether each log file corresponding to the first identification range has been obtained, and if not, obtain the log file from the source database again. Until the electronic device obtains each log file corresponding to the first identification range. This can alleviate the missing data phenomenon caused by the subsequent data processing of the electronic device because the electronic device cannot obtain each log file corresponding to the first identification range. The data consistency between the source end and the target end can be further improved, thereby alleviating the data distortion problem at the target end.

In yet another possible design of the first aspect, the log files not obtained by the above-mentioned electronic device and corresponding to the first identification range include: switching log files; switching log files can understand that the source database is being archived The log file for the operation.

In another possible design of the first aspect, the above-mentioned electronic device obtains log files from the source database according to the first identification range, including: if each log file corresponding to the first identification range is not obtained, re-according to Obtaining log files in the first identification range; until the number of acquisitions of log files according to the first identification range is greater than or equal to a preset number of times threshold (eg, 5 times, 10 times). Next, the electronic device performs data operations on the target database according to the last obtained log file.

It is understandable, considering that if the source database frequently generates switching log files, the electronic device will repeatedly obtain log files from the source, and the electronic device does not perform subsequent data integration steps, and the data in the source database to the target database The delay will be relatively long, which will affect the real-time performance of data integration. Based on this, in this design, when the electronic device obtains log files from the source database too many times, the electronic device will not obtain log files from the source database again, and directly updates the target terminal based on the last obtained log file. The database performs data operations, so that the log files generated by the source database can be integrated into the target database in a timely manner, which can reduce the time delay of integrating the log files of the source database into the target database, and can improve the real-time data integration of electronic equipment. sex.

In yet another possible design of the first aspect, the above-mentioned log file may also include a log file number, and the above-mentioned method further includes: the electronic device determines the log file number of the missing log file, and the missing log file is not obtained. A log file corresponding to an identifier range. Next, the electronic device obtains the start log data identifier or the end log data identifier of the missing log file according to the log file number of the missing log file. Then, the electronic device obtains the missing log file from the source database according to the starting log data identifier or the ending log data identifier of the missing log file; then, the electronic device can perform data operations on the target database based on the missing log file. Wherein, the start log data identifier of the missing log file is the log data identifier of the first log data included in the missing log file; the end log data identifier of the missing log file is the log data identifier of the last log data included in the missing log file .

In this implementation manner, the electronic device can obtain the aforementioned missing log file through the starting log data identifier and the ending log data identifier of the missing log file. And in the follow-up, the log data in the missing log file will be integrated into the target database. In this way, the problem of missing data during data integration can be further alleviated, the data consistency between the source database and the target database can be improved, and the data at the target can be alleviated. Distortion problem.

In another possible design of the first aspect, the aforementioned log files include: online log files and archived log files. And, the above-mentioned electronic device obtaining the log file from the source database according to the first identification range includes: in the case that the log file is an online log file, the electronic device obtains the starting log data of the online log file from a range that is less than or equal to the first identification range Online log files for endpoints. In the case that the log file is an archived log file, the electronic device acquires the archived log file whose starting point log data identifier is less than or equal to the end point of the range of the first identification range and greater than or equal to the starting point of the range of the first identification range; or, obtains the end point log The data identifies archived log files that are less than or equal to the range end of the first identified range and greater than or equal to the range start of the first identified range. Wherein, the starting log data identifier of the online log file is the log data identifier of the first log data included in the online log file, and the starting log data identifier of the archived log file is the log data of the first log data included in the archived log file ID, the terminal log data ID of the archived log file is the log data ID of the last piece of log data included in the archived log file.

In this design, when the electronic device obtains the online log file, it can obtain the online log file more comprehensively and completely by obtaining the online log file whose start point log data identifier is less than or equal to the end point of the first identification range, thereby reducing the number of electronic devices. Occurrence of missing numbers on the device. And, when the electronic device acquires the archived log files, the online log files can be obtained more comprehensively and completely by obtaining the archived log files whose log data identification falls within the above-mentioned first identification range, thereby reducing the occurrence of leaks on the electronic equipment. number phenomenon.

In yet another possible design of the first aspect, the electronic device performing data operations on the target database based on the obtained log file may include: the electronic device parses the obtained log file, and based on the parsing The last log file performs data operations on the target database. Wherein, the data operation includes adding data operation, deleting data operation or modifying data operation.

In another possible design of the first aspect, the above method further includes: if the electronic device determines from the obtained log files that there is no missing log file, the electronic device obtains the log flag based on the end point of the first identification range .

In yet another possible design of the first aspect, the electronic device determining the second identification range according to the log flag may include: the electronic device takes the log flag as the starting point of the second identification range, and calculates the second identification range through the dynamic step Gets the range endpoint of the second identified range. Wherein, the dynamic step size is related to the maximum log data identifier in the source database.

In this design, by setting a dynamic step size, the electronic device can dynamically adjust the speed at which the electronic device acquires log files from the source according to the speed at which the source generates log data. When the log data generated by the source is fast, the electronic device uses more resources to load more log data, so that the electronic device loads log files from the source at a faster speed; when the log data generated by the source is slow When , the electronic device uses relatively few resources to load relatively small log data, so that the electronic device loads log files from the source at a relatively slow speed. In this way, electronic equipment and source resources (such as processor resources, input and output interface resources, etc.) can be used more reasonably, and the real-time performance of data integration can be improved.

In a second aspect, the present application provides an electronic device, which includes: a memory, one or more processors, and a Bluetooth module; the memory is coupled to the processor; wherein, a computer program code is stored in the memory, and the computer program code includes a computer Instructions; when the computer instructions are executed by the processor, the electronic device is made to execute the method provided by the above-mentioned first aspect and any possible design of the first aspect.

In a third aspect, the present application provides a data storage system, and the data storage system includes: a source-end database and a target-end database. Wherein, the data storage system further includes the electronic device provided in the second aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer instructions are run on the electronic device, the electronic device executes any one of the above-mentioned first aspect and the first aspect. A possible design approach is provided.

In the fifth aspect, the present application provides a computer program product containing instructions. When the computer program product is run on the electronic device, the electronic device can execute the above-mentioned first aspect and any possible design of the first aspect. method.

Wherein, the technical effect brought by any one of the design methods in the second aspect to the fifth aspect can refer to the technical effect brought by different design methods in the first aspect, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic structural diagram of a data system provided by an embodiment of the present application;

FIG. 2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

FIG. 3 is a schematic flow chart of an example of a data processing method provided in an embodiment of the present application;

FIG. 4 is a schematic structural diagram of a log file stored by a data storage node provided in an embodiment of the present application;

FIG. 5 is a schematic flowchart of another example of the data processing method provided by the embodiment of the present application;

FIG. 6 is a schematic flowchart of another example of the data processing method provided by the embodiment of the present application;

FIG. 7 is a schematic flow diagram of obtaining a log file by an electronic device provided in an embodiment of the present application;

FIG. 8 is a schematic diagram of an example of the data processing method provided by the embodiment of the present application;

FIG. 9 is a schematic diagram of another example of the data processing method provided by the embodiment of the present application;

FIG. 10 is a schematic diagram of another example of the data processing method provided by the embodiment of the present application;

FIG. 11 is a schematic diagram of another example of the data processing method provided by the embodiment of the present application;

FIG. 12 is a schematic diagram of another example of the data processing method provided by the embodiment of the present application;

FIG. 13 is a schematic flowchart of another example of the data processing method provided by the embodiment of the present application;

FIG. 14 is a schematic diagram of another example of the data processing method provided by the embodiment of the present application;

FIG. 15 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

FIG. 16 is a schematic structural diagram of another electronic device provided by an embodiment of the present application.

Detailed ways

In the following, before introducing the embodiments of the present application, relevant terms involved in the embodiments of the present application will be introduced first.

A database is a warehouse that organizes, stores and manages data according to its data structure. A database is a collection of organized, shareable, and large amounts of data stored in electronic devices for a long time. The files in the database can be divided into: data files, log files and control files according to the contents of the files. Wherein, the data file means that the content of the file is the data that needs to be stored in the database; for example, the order of the customer, the quantity of the commodity, the inventory of the raw material, the page views of the commodity, and the like. The control file means that the content of the file is the data operation on the data in the database; for example, query the data in the database, add the data in the database, delete the data in the database, etc. The log file means that the content of the file records the changes of the database; for example, the AA data in the database is added at XX time, and the BB data in the database is deleted at YY time, etc. For ease of understanding, the content in the log file is referred to as log data hereinafter.

When the data storage node of the database generates log data, it will generate the identifier of the log data (hereinafter referred to as the log data identifier for short) according to the generation time sequence of the log data; for example, log point (system changenumber, scn) or log label (binlog). Afterwards, the database will add the log data with the log data identifier (such as scn) to the log file. That is, log data identifiers can be used to characterize the sequence in which data operations on the database occur. It is understandable that the scn of the log data is positively correlated with the generation time of the log data; the larger the scn of the log data, the later the log data is generated; the smaller the scn of the log data, the earlier the log data is generated . Similarly, the binlog of the log data is positively correlated with the generation time of the log data; the larger the binlog of the log data, the later the generation time of the log data; the smaller the binlog of the log data, the earlier the generation time of the log data.

And, the data storage node of the database will also generate the log file identification of the log file based on the number of log files in the data storage node; for example, the log file number (redolog).

In some databases, in order to avoid a single log file being too large; the database usually divides a single log file into multiple ones, that is, when the size of the log file (online log file) being written reaches the preset size, the online log file become an archived log file, and then scroll the unwritten log content to the next log file (ready log file). And in some databases with multiple data storage nodes, if a single log data is relatively large, the database can disperse a single log data into multiple log data; and then disperse it to multiple data storage nodes for storage. In this way, a stand-alone resource bottleneck can be avoided.

It should be pointed out that, for the online log file, since the file is still being written and has not been written, the file will have a log file number and a starting point log data identifier. Wherein, the log data identifier of the starting point is the log data identifier of the first log data written in the log file. For an archived log file, since the file has been written, the file will have a log file number, a starting point log data identifier, and an end point log data identifier. Wherein, the terminal log data identifier is the log data identifier of the last piece of log data written in the log file.

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Among them, in the description of this application, unless otherwise specified, "/" indicates that the objects associated with each other are an "or" relationship, for example, A/B can indicate A or B; in this application, "and/or "It is just an association relationship describing associated objects, which means that there can be three kinds of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone. , B can be singular or plural. Moreover, in the description of the embodiments of the present application, unless otherwise specified, "plurality" refers to two or more than two. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple . In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. To be precise, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner for easy understanding.

Data integration is to centralize data logically or physically, which can be understood as bringing together data from multiple data storage nodes in the database. For example, the electronic device integrates data from a source-side database device or a source-side database device cluster (hereinafter referred to as a source for short) into a target-side database device or a target-side database device cluster (hereinafter referred to as a target for short). In the process of data integration, there will be many factors that lead to the omission of data at the source end, and the situation where the data at the source end is not integrated to the target end, that is, the phenomenon of missing data will occur. This will cause inconsistency between the data at the source end and the data at the target end, resulting in data distortion at the target end.

For example, in the real-time data analysis business, the problem of data distortion will lead to the inability to capture accurate data changes in time, resulting in the delay of data changes in the data analysis business, and the inability to deal with data changes in a timely manner; For businesses that require high accuracy, such as financial data analysis business, traffic flow analysis business, order processing business, etc., data distortion will cause great business risks. For another example, in the offline data analysis business, the problem of data distortion will lead to relatively large errors in subsequent data calculations, which will waste computing resources.

In some solutions, the electronic device will load the log file from the source; after that, the electronic device will perform data operations on the data of the target end based on the log data in the above log file (for example, adding data, deleting data, modifying data). Next, the electronic device determines the range of data to be loaded by the electronic device next time based on the data operation result at the target end (for example, the log data identifier corresponding to the last data operation). In this way, the electronic device loads log files from the source end cyclically, and then performs data operations on the target end based on the log data in the log file, so that the data at the source end can be integrated into the target end. However, in this scheme, the situation of obtaining log files with missing content is not taken into account. Data integration based on this scheme will cause missing numbers, which will cause data inconsistencies between the source end and the target end, resulting in target end data. Distortion problem.

In view of this, an embodiment of the present application provides a data processing method. In the method, the electronic device can determine a log file with missing content among the obtained log files. Next, the electronic device can obtain the log flag bit based on the log file with missing content. And in the next data integration, the log file is obtained based on the log flag. In this way, when the log file is acquired next time, the electronic device can acquire the log file whose content determined this time is missing again. It can alleviate the missing data phenomenon in the data integration process, and then can improve the data consistency between the source end and the target end, thereby reducing the problem of data distortion at the target end.

The data processing method provided in the embodiment of the present application can be applied to a data storage system. For example, refer to FIG. 1 , which shows a schematic structural diagram of a data storage system. Wherein, according to the flow direction of data, the data storage nodes in the data storage system can be divided into a source database device cluster (abbreviated as source 200 ) and a target database device cluster (referred to as target 300 ). Between the source end 200 and the target end 300, one or more electronic devices 100 (only one is shown in the figure) can control the data integration between the source end 200 and the target end 300, or control the source end 200 or the target end Operate data in terminal 300 (eg, add data, delete data, modify data, etc.). And, the source end 200 may establish a communication connection with one or more data operation devices (only one is shown in the figure). The data operation device is used for performing data operations on the data in the source end 200 (eg, adding data, deleting data, modifying data, etc.). Similarly, the target end 300 may also establish a communication connection with one or more data operation devices, and the data operation devices are used to perform data operations on the data in the target end 300 . Wherein, the source end 200 may include multiple data storage node devices (hereinafter referred to as nodes for short); and the target end 300 may include multiple data storage nodes. Log files and data files are stored in the nodes of the source end 200 or the target end 300 . That is, nodes store both log files and data files.

In some embodiments, the architecture of the data storage system shown in FIG. 1 may be a multi-node cluster architecture (for example, an Oracle database multi-node cluster architecture). In a multi-node cluster architecture, each node will scroll and write the log content of the node to the log file. Wherein, the log content is used to indicate that the data in the database has changed. Write the log content to the log file to record the changes of the table data in the database. In order to prevent a single log file from being too large, a single log file is usually divided into multiples, that is, when the size of the current log file reaches a preset size (such as 1GB, 3GB, or 5GB, etc.), the unwritten log content Scrolling writes to the next log file. At the same time, the multi-node architecture can scatter oversized data and log content to multiple nodes for storage, which can avoid single-node resource bottlenecks.

Exemplarily, in FIG. 1 , the data operation device adds data to the source end 200 . Afterwards, the source end 200 distributes the data added by the data operation device to multiple data storage nodes (eg, node 1 or other nodes) at the source end for storage. And, the source end 200 can generate a log file for the data operation device adding data this time, and the log file can be stored on any node of the source end 200 . Next, the electronic device 100 may integrate data in multiple data storage nodes of the source end 200 into the target end 300 . If the target end 300 has multiple data storage nodes, the target end 300 will also perform distributed storage.

It can be understood that the structure of the database storage system shown in FIG. 1 does not constitute any limitation on the database system. In actual use, the database system may include more or less components than those shown in the illustration, or combine certain some components, or a different arrangement of components. Relevant technical personnel can design the structure of the database system according to the actual usage.

The data storage node device provided in this embodiment of the application may be some devices with data storage and processing capabilities; for example, servers (such as heterogeneous servers, cloud servers, etc.), personal computers, notebook computers, servers, network-attached storage ( network attached storage, NAS), mobile phone, on-board computer, etc. The electronic equipment provided in the embodiment of the present application may be some electronic equipment with data transmission and processing capabilities; for example, personal computer (personal computer, PC), notebook computer, tablet computer, personal digital assistant (personal digital assistant, PDA), super Mobile personal computer (ultra mobile personal computer, UMPC), server, car computer, etc. The embodiments of the present application do not impose any restrictions on specific product forms of the data storage node device and the electronic device. In some embodiments, the aforementioned electronic device may be an electronic device running an operating system and installing application programs. For example, the operating system run by the electronic device may be Android™ system, Windows™ system, Linux™ and so on.

FIG. 2 shows a schematic diagram of the hardware structure of the electronic device 100 .

As shown in FIG. 2, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a power management module 141, an antenna, and a wireless communication module 150, display screen 140 and the like.

It can be understood that, the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

The wireless communication module 150 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system, etc. (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.

The display screen 140 is used for displaying images, videos and the like. The display screen 140 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light emitting diode). AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emitting diodes, QLED), etc.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 may be used to store computer-executable program codes including instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 . For example, in the embodiment of the present application, the processor 110 may execute instructions stored in the internal memory 121, and the internal memory 121 may include a program storage area and a data storage area. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to provide electric power for the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

Exemplarily, the electronic device may obtain log files from the source database through the antenna and the wireless communication module 150; afterward, the electronic device may store the obtained log files through the internal memory 121. Alternatively, the electronic device can also acquire log files from the source database through the USB interface 130 . Next, the electronic device may parse the acquired log file through the processor 100 . Then, the electronic device can send the parsed log file to the target database through the antenna and the wireless communication module 150 . Alternatively, the electronic device can also acquire log files from the source database through the USB interface 130 . In this way, the data of the source database can be integrated into the target database.

As another example, the processor 100 of the electronic device may determine missing log files from the obtained log files through instructions stored in the internal memory 121 . Next, the electronic device can determine the log file that the electronic device needs to obtain when obtaining the log file next time based on the log data identifier of the missing log file; in this way, the electronic device can obtain the log file obtained this time when obtaining the log file next time log files that were missed. It can alleviate the missing data phenomenon in the data integration process, and then can improve the data consistency between the source end and the target end, thereby reducing the problem of data distortion at the target end.

In the following, in the data system shown in FIG. 1 , the electronic device 100 is a PC, and the source 200 includes at least two data storage nodes (such as node 1 and node 2) as an example, the data processing method provided by the embodiment of the present application will be described Make an introduction.

Referring to FIG. 3 , the data processing method provided in this application may include steps S101-S103.

In some embodiments, before step S101, the data processing method provided in this application may further include step S100.

S100. The source end generates a log file.

In some embodiments, the source end may receive data operations (such as adding data, deleting data, modifying data, etc.) performed on the source end by a data operation device that has established a communication connection with the source end. Afterwards, since the data operation of the source end by the data operation device will cause changes in the data files of the source end, the source end will generate log data and write the log data to the online log file of any data storage node at the source end.

Exemplarily, it is assumed that the data operation device deletes data Z in node 1, and the data in node 1 changes. At this time, the source end will generate log data, and the log data is recorded, and the data Z is deleted. Afterwards, the source end will randomly write the log data into the online log file of any node (for example, node 1, or other nodes except node 1) at the source end.

In some embodiments, the origin can also generate groups of log files. For example, when the source end generates a log file, the data storage node at the source end will back up the log file to generate a backup file of the log file; and package the log file and the backup file of the log file into a file group (log file group) . In this way, the reliability of the log files can be improved.

In some embodiments, the data storage node at the source end may also use a rolling file mechanism to initialize multiple log files (log file groups), which can improve the real-time performance of storing log files in the data storage node at the source end.

Exemplarily, referring to FIG. 4 , node 1 adopts a rolling file mechanism and initializes three log file groups (log file group 1, log file group 2, and log file group 3). And each log file group includes a log file and a backup file of the log file. Assuming that node 1 generates log data, node 1 will write the log data into the log file and the backup log file of log file group 1. If the log file group or the log files in the log file group reaches a preset file size (for example, 1 gigabyte (GB), 3GB), node 1 will perform an archiving operation on the log file. Archived log files are generated afterwards. Afterwards, node 1 will write the newly generated log data into log file group 2, and initialize a new log file group. It is understandable that since log file group 2 is in the state of initialization, node 2 can immediately write log data to log file group 2 after log file group 1 reaches the preset file size, which can improve the Real-time performance of log files stored in data storage nodes. And, considering that the log data generation speed of node 1 may be relatively fast, it may be faster than the speed of initializing the log file group of node 1. Therefore, node 1 can set log file group 3; and write log data into log file group 3 after log file group 2 reaches a preset file size. In this way, even if the log data on node 1 generates data relatively quickly, it is faster than the speed at which node 1 initializes the log file group. The node 1 may also have an initialized log file group for the node 1 to write log data in time. In this way, the real-time performance of storing log files by the data storage node can be further improved.

Understandably, since the log file in log file group 1 is the log file currently being written by node 1, the log file is an online log file, and the largest log file number in the archived log file is 139. The log file of this file File number is 140. Afterwards, after the log file reaches the preset file size, node 1 will archive the log file, and put the log file into the relevant location where the archive log file is stored in node 1 (for example, the archive log file queue ).

Exemplarily, it is assumed that the data operation device modifies the source data A; after that, the source can record the operation in the log file A of node 1 . That is to say, the log content (log data) of the log file A will record that the data operation device modifies the source data A.

Exemplarily, it is assumed that the data operation device performs a delete operation on the data B of the source end; after that, the source end may record the operation in the log file B of the node 2 . That is to say, the log content (log data) of the log file B will record that the data operation device performs a deletion operation on the source data B.

Exemplarily, it is assumed that the data operation device modifies the source data D; after that, the source can record the operation in the log file A of node 1 . That is to say, the log content (log data) of the log file A will record that the data operation device modifies the source data A.

Exemplarily, it is assumed that the data operation device adds source data E to the source; after that, the source can record the operation of adding source data E in node 2 . At this time, because the size of the log file B of node 2 has reached a preset size (for example, 2Gb). Node 2 will archive log file B, and node 2 will create a new log file C, and record the operation of adding source data E in log file C of node 2. That is to say, the log content (log data) of the log file C of node 2 will record the addition operation of the data operation device on the source data E.

S101. The electronic device obtains the log file from the source.

In some embodiments, the electronic device may obtain log files whose log data identifiers are within the log data identifier range from the source through the log data identifier range. It can be understood that the log data identification range can be scn range or binlog range.

In some embodiments, obtaining the log file from the source by the electronic device may include that the electronic device loads (loads) the log file from the source.

Exemplarily, the electronic device may obtain log files whose scn is in the above-mentioned scn range from each data storage node at the source through the scn range.

As another example, the electronic device may obtain, through the binlog range, from each data storage node at the source, the log files whose binlog is in the above-mentioned binlog range.

It can be understood that there are certain rules for generating log data identifiers, for example, log data identifiers are generated sequentially according to the generation time of log data. That is to say, the log data identifier will increase in sequence, and the larger the log data identifier, the later the log data generation time will be. It can be seen that the log data identifier is related to time.

It can be understood that the log data identifies a range, which may be composed of a range start point and a range end point. In this way, when the electronic device acquires log files from the source, it can acquire log files whose log data identifiers are greater than or equal to the start point of the range, or less than or equal to the end point of the range.

Exemplarily, for an online log file in the data storage node at the source end, if the log data identifier of the starting point of the online log file is less than or equal to the end point of the range, the electronic device may acquire the online log file.

It can be understood that for online log files, the log data in the online log files will continuously increase. Therefore, when obtaining online log files, by obtaining the online log files whose starting point log data identifier is less than or equal to the end point of the range, the online log files can be obtained more comprehensively and completely, thereby reducing the phenomenon of missing data on electronic devices.

Exemplarily, for an archived log file in the source data storage node, if the start log data identifier of the archived log file is greater than or equal to the start point of the range and less than or equal to the end point of the range; or, the end log data identifier of the archived log file greater than or equal to the start of the range and less than or equal to the end of the range; the electronic device can obtain the archived log file. That is to say, the electronic device will acquire the archived log files whose log data identifiers fall within the aforementioned range of log data identifiers.

It can be understood that when an electronic device obtains a log file from a source, most of the electronic device only obtains a complete log file. Based on this, when obtaining archived log files, by obtaining the archived log files whose log data identification falls within the range of the above-mentioned log data identification, online log files can be obtained more comprehensively and completely, thereby reducing the number of omissions that occur on electronic devices Phenomenon.

It can be understood that the above step S101 will be slightly different in the first data integration scenario and the non-first data integration scenario. The first data integration scenario refers to that the electronic device has not performed data integration before, or the electronic device has not performed data integration for a period of time.

In the first data integration scenario, the above log data identification range may be specified by the user of the electronic device.

In the non-initial data integration scenario, the above-mentioned log data identification range is obtained based on historical log flag bits, or is specified by the user of the electronic device. Wherein, the historical log flag can be understood as the log flag obtained by the electronic device when performing data integration last time.

In some embodiments, the electronic device may determine whether it is the first data integration scenario through the history log flag. If the electronic device can obtain the history log flag, the electronic device is determined to be in a non-first data integration scenario. If the electronic device cannot obtain the history log flag from the electronic device's memory, the electronic device is determined to be in the first data integration scenario.

In the following, step S101 will be introduced by using an electronic device in the first data integration scenario.

In the first data integration scenario, because the electronic device has not performed data integration before, or in a period of time before; that is to say, the electronic device cannot obtain the historical log flag. Based on this, the electronic device can obtain the starting point of the range specified by the user of the electronic device. Afterwards, the electronic device obtains the end point of the range through a dynamic step size or a preset step size (eg, 500, 5000). Next, the electronic device fetches log files from the source based on the range start and range end. Referring to Fig. 5 for example, the above step S101 may include: steps S101a1-S101a4.

S101a1. The electronic device determines that it is in the first data integration scenario.

Exemplarily, if the electronic device does not acquire the history log flag, the electronic device determines that it is in the first data integration scenario.

S101a2. The electronic device acquires the starting point of the specified range.

Exemplarily, the electronic device may acquire the starting point of the range input by the user of the electronic device on the electronic device.

S101a3. Based on the starting point of the specified range, the electronic device calculates the end point of the range through the dynamic step size.

Among them, the dynamic step size is a step size that will change according to the maximum log data identification of the current source. By setting the dynamic step size, the speed at which the electronic device loads (obtains) log files from the source can be dynamically adjusted according to the speed at which the source generates log data. When the log data generated by the source is fast, the electronic device uses more resources to load more log data, so that the electronic device loads log files from the source at a faster speed; when the log data generated by the source is slow When , the electronic device uses relatively few resources to load relatively small log data, so that the electronic device loads log files from the source at a relatively slow speed. In this way, electronic equipment and source resources (such as processor resources, input and output interface resources, etc.) can be used more reasonably, and the real-time performance of data integration can be improved.

Exemplarily, referring to FIG. 6, the above step S101a3 may include: S101a31-S101a36.

Wherein, the starting point of the range is recorded as x ₀ , the initial step size is recorded as y ₀ , and the current source maximum log data identifier is recorded as z ₀ .

S101a31. The electronic device calculates the log data identifier x ₁ ; x ₁ = x ₀ + y ₀ . Wherein, y ₀ may be a preset value (eg, 500, 1000, 2000, etc.).

S101a32. The electronic device acquires the maximum log data identifier z ₀ of the current source.

Exemplarily, the electronic device may acquire the largest log data identifier in the online log files of multiple nodes at the current source end. And, the largest log data identifier among the largest log data identifiers in the online log files of the above multiple nodes is used as the largest log data identifier z ₀ at the source.

S101a33. The electronic device calculates the difference z ₀ - x ₁ between the log data identifier x ₁ and the above-mentioned maximum log data identifier z ₀ .

S101a34. The electronic device judges whether the difference z ₀ - x ₁ is within a preset range k ₀ (eg, [-100, 100], [-500, 500]). If it is within the preset range, then output x ₁ as the end point of the range; if it is not within the preset range, then execute step S101a35.

S101a35. The electronic device determines whether the step size y ₀ is within a preset interval k ₁ (eg, [100, 10000], [300, 5000], etc.). If it exceeds the preset interval, output x1 as the end point of the range; if it does not exceed the preset interval, _execute step S101a36.

S101a36. The electronic device adjusts the step size y ₀ based on the above z ₀ - x ₁ . After that, S101a31 is executed again. until the end of the output range.

For example, the electronic device adds the above z ₀ - x ₁ for the step size y ₀ , or, adds 0.8 ( z ₀ - x ₁ ), 1.2 ( z ₀ - x ₁ ), etc. for the step size. Understandably, for the source end, the source end will continuously generate log data identifiers. That is to say, in the process of calculating the end point of the range by the electronic device, the source end will generate a new log data identifier. Therefore, by dynamically adjusting the step size according to the above difference z ₀ - x ₁ , the speed at which the electronic device obtains log data can be adjusted according to the log data generated by the source, and the resources of the electronic device and the source can be reasonably used.

S101a4. The electronic device obtains the log file from the source based on the range start point and the range end point.

In some other embodiments, the electronic device may acquire the specified starting point of the range. Afterwards, the electronic device obtains the maximum log data identifier of the source end, uses the maximum log data identifier as the end point of the range, and obtains the log file from the source end. Wherein, the maximum log data identifier at the source end refers to the log data identifier with the largest value among the log data included in the online log files of multiple data storage nodes at the source end.

In some other embodiments, the electronic device can also compare the size of the above difference z ₀ - x ₁ and y ₀ ; if z ₀ - x ₁ > y ₀ , then increase y ₀ (for example, increase by 10, 100, etc. ); After that, calculate the end point of the range based on the increased y ₀ , and then obtain the log file from the source based on the end point of the range. If z ₀ - x ₁ < y ₀ , log files are obtained from the source based on x ₁ . And/or, the electronic device can also compare the size of x ₁ - z ₀ and y ₀ ; if x ₁ - z ₀ > y ₀ , reduce y ₀ (for example, decrease by 10, 100, etc.); The smaller y ₀ calculates the end point of the range, and then obtains the log file from the source based on the end point of the range. If x ₁ - z ₀ < y ₀ , log files are obtained from the source based on x ₁ .

In some embodiments, the above step S101 may further include: step S101a5.

S101a5. If the electronic device determines that the source end generates the first type of missing log file, the electronic device re-executes step S101a4; if the electronic device does not determine that the source end generates the first type of missing log file, the electronic device performs subsequent steps.

In some embodiments, if the electronic device determines that the source end generates the first type of missing log file, the electronic device may execute step S101a4; or, the electronic device may also execute step S101a4 and any steps before step S101a4 (such as S101a1-S101a4 , S101a2-S101a4, S101a3-S101a4).

Among them, the first type of missing log files can be understood as log files that cannot be obtained by electronic devices. For example, a log file that is being archived; for the data storage node at the source, the data storage node will archive the online log file when the online log file reaches the preset size, and the online log file becomes an archived log file; that is, the data storage node at the source will switch the log file. If the electronic device obtains log files from the source at this time, the electronic device cannot obtain log files that are being archived, but can only obtain current online log files and archived log files. Afterwards, if the electronic device directly performs subsequent data integration steps based on the obtained log files, then this data integration will generate missing numbers, and log files that are being archived will be missed.

In some embodiments, the electronic device may determine whether the source end generates the first type of missing log file by judging whether the source end has a file switch during the process of obtaining the log file by the electronic device (eg, the above step S101a4). If file switching occurs at the source, the electronic device determines that the source has generated the first type of missing log file; if no file switching occurs at the source, the electronic device determines that the source does not generate the first type of missing log file.

Based on this, the electronic device can determine whether a file switch occurs at the source, and if a file switch occurs, the log file is acquired from the source again. In this way, the log file of the file switch at the source can be obtained by the electronic device in the form of an archive log file after the file switch occurs (that is, it becomes an archive file). In this way, when the electronic device performs subsequent data integration steps, the log file in which the file switching has occurred will not be missed. It can alleviate the problem of missing numbers in the process of data integration, and then can improve the consistency of data between the source and the target, thereby reducing the data distortion problem at the target.

Exemplarily, the electronic device may determine whether log file switching occurs at the source end by determining whether the log file numbers of the log files of the same data storage node are continuous. If the file numbers of the log files from the same data storage node are continuous, log file switching does not occur; if the file numbers of the log files from the same data storage node are discontinuous, log file switching occurs.

For example, referring to FIG. 7 , the log file number of the online log file of node 1 is 140, and the log file numbers of the archived log files are 139-134. If the log file with log file number 140 reaches the preset size, node 1 will archive the log file with log file number 140; after that, log data will be written into the log file in log file group 2 and sent to the log Log file set number 141 for filegroup 2. At this time, if the electronic device obtains the log file from node 1, the electronic device will not be able to obtain the log file with the log file number 140, and the log numbers of the log files that the electronic device can obtain are: 141, 139, 138, 137, 136, 135, 134. Next, the electronic device detects that the log numbers of the obtained log files are discontinuous, and the electronic device determines that a log file switch has occurred on node 1, and the electronic device obtains the log file from the source again.

Exemplarily, the electronic device may determine whether the source end has generated the first type of missing log file according to the file switching instruction sent by the source end. For example, the data storage node at the source can generate and send a file switching instruction to the electronic device before archiving the log (that is, before the file switching occurs). If the electronic device receives the file switching instruction, it is determined that the source end generates the first type of missing log file; if the file switching instruction is not received, it is determined that the source end does not generate the first type of missing log file.

As a possible implementation manner, in the above step S101, the electronic device may also perform a cyclic acquisition process to obtain all the source end log files corresponding to the above log data identification range.

Wherein, the cyclic acquisition process includes: if all the source-end log files corresponding to the above-mentioned log data identification range cannot be obtained, the electronic device reacquires the source-end log files corresponding to the above-mentioned log data identification range until all the source-end log files corresponding to the above-mentioned log data identification range are obtained. The source end log file corresponding to the above log data identification range.

As another possible implementation, if the number of times the electronic device performs the above cyclic acquisition process is equal to the preset number of times threshold (for example, 5 times, 10 times), then according to the source log obtained by the last cyclic acquisition process of the electronic device The log file number of the file, which determines the log file number of the source log file that was not obtained in the last cyclic acquisition process. Next, the electronic device obtains the start log data identifier or the end log data identifier of the source log file not obtained in the last cyclic acquisition process according to the log file number of the source log file not obtained in the last cyclic acquisition process. Then, the electronic device obtains the source log files not obtained in the last cyclic acquisition process from the source database according to the start log data identifier or the end log data identifier of the source log files not obtained in the last cyclic acquisition process, and the source The terminal log data identifier of the end log file is the log data identifier of the last log data included in the source end log file.

It is understandable that if log file switching occurs frequently at the source end, the source end will frequently generate the first type of missing log files; this will cause the electronic device to execute step S101a5 and step S101a4 cyclically, without performing subsequent data integration steps ; That is to say, the log file integration delay at the source end to the target end will be relatively long, which will affect the real-time performance of data integration.

Based on this, in some embodiments, the electronic device may record the number of times the electronic device reloads the log file due to log file switching at the source (eg, the number of times step S101a4 is performed after step S101a5 is performed). If the number of times exceeds the preset number of times threshold (for example, 5 times, 10 times), the electronic device may not reload the log file (for example, step S101a4 is not performed), and directly execute subsequent data integration steps. (For example, perform data operations on the target database directly based on the log file loaded to.)

In this way, when the electronic device repeatedly executes step S101a5 and step S101a4, reaching a certain number of times; even if the first type of missing log files still exist, the electronic device also performs subsequent data integration steps (for example, according to the source end log file obtained last time. The target database performs data operations). The log files generated by the source end can be integrated to the target end in a timely manner, the time delay for integrating the log files at the source end to the target end can be reduced, and the real-time performance of data integration by electronic devices can be improved.

And, considering that the source end will continue to generate log files; if the source end frequently switches log files, that is, the electronic device executes step S101a4 after step S101a5 for many times; and, when the electronic device executes step S101a4 after step S101a5, the electronic device The device obtains the newly generated log files at the source; this will cause the number of log files obtained by the electronic device to continue to increase when the electronic device executes step S101a4, which will result in a large amount of log file data that the electronic device needs to process, which may exceed the electronic device. Data processing power can even cause electronic equipment to crash.

Based on this, by setting a preset number of times threshold, when the number of times the electronic device reloads the log file exceeds the number of times threshold, the electronic device directly performs subsequent data integration steps. The number of log files acquired by the electronic device when executing step S101a4 can be controlled, the data volume of the log files to be processed by the electronic device can be controlled, and the crash of the electronic device can be reduced.

In some embodiments, if the log file switching at the source recorded by the electronic device causes the electronic device to reload the log file for a number of times that exceeds a preset threshold (5 times, 10 times), the electronic device may determine to switch the log file ( That is, the file number of the first type of missing log file). Afterwards, the electronic device obtains the start log data identifier or the end log data identifier of the switch log file based on the file number of the switch log file. Next, the electronic device directly executes subsequent data integration steps without reloading the log file, and loads the switching log file through the starting point log data identifier or the end point log data identifier when subsequently loading the log file. Wherein, the switching log file can be understood as a log file that has been switched during the process of obtaining the log file from the source by the electronic device; that is, a log file that is being archived and switched to an archived log file. In some embodiments, the first type of missing log files (such as switching log files) may also be referred to as missing log files.

In this way, the electronic device can obtain the aforementioned switch log file through the start log data identifier and the end log data identifier of the switch log file. And in the follow-up, the log data in the switching log file will be integrated into the target database. In this way, the problem of missing numbers during data integration can be further alleviated, the data consistency between the source database and the target database can be improved, and the data at the target can be alleviated. Distortion problem.

Assume that the log numbers of the log files that can be obtained by the electronic device are: 141, 139, 138, 137, 136, 135, and 134. Next, the electronic device detects that the log numbers of the acquired log files are discontinuous, and determines that the discontinuity point that causes the log numbers of the log files to be discontinuous, that is, the log file with the log number 140, is the switching log file. Afterwards, the electronic device sends a query request to the source to query the start log data identifier and the end log data identifier of the log file whose log number is 140. Next, when the log file is acquired from the source next time, based on the start log data identifier and the end log data identifier of the log file with the log number 140, the log file with the log number 140 is loaded from the source.

In some embodiments, after the electronic device obtains the log file from the source, the electronic device may also analyze (for example, code conversion, decryption, etc.) the obtained log file. Understandably, log files on some databases exist in some special encoding formats, or these log files are encrypted by data storage nodes. Based on this, the electronic device can parse these log files, and perform subsequent data integration steps after parsing.

S102. The electronic device determines the second type of missing log files from the acquired log files.

Wherein, the second type of missing log file can be understood as a log file in which log data is missing among the log files obtained by the above-mentioned electronic device. It can also be understood as a log file with missing log file content. That is to say, the electronic device has obtained information such as the log file number and the starting point log data identifier of the second type of missing log file, but has not obtained the log data of the second type of missing log file. The file content of the second type of missing log file, that is, the log data of the second type of missing log file is missing.

In some embodiments, the electronic device can read the log data in the log file. If the log data in the log file is not read, it is determined that the log data in the log file is missing, and the log file is a second type of missing log file.

For example, the electronic device has obtained the log file number of the log file. According to the log file number, the electronic device only reads the name, attribute, starting point log data identifier, and end point log data identifier of a certain log file, but the electronic device does not read the log file number. The log data included in the log file; the electronic device can determine that the log file is a second type of missing log file.

For another example, the electronic device has read the name, attribute, start log data identifier, and end point log data identifier of a certain log file; however, the electronic device has read abnormal log data from the log file, and the electronic device cannot The log data is used for data manipulation; the electronic device can determine that the log file is the second type of missing log file.

Wherein, the log data may correspond to any one of a data modification operation, a data addition operation, and a data deletion operation.

For example, if the data operation recorded in the log data is to modify data A, the electronic device will modify data A at the target end. For another example, if the data operation recorded in the log data is to delete data B, the electronic device will delete data B at the target end. For another example, the data operation recorded in the log data is an addition operation on data C, and data C is added. Then the electronic device will perform an increase operation of data C on the target end, and add data C to the target end.

In some embodiments, the probability of missing data when using archived log files for data integration is smaller than the probability of missing data when using online log files for data integration. Based on this, the electronic device may extract online log files from the obtained log files, and then determine the second type of missing log files from the online log files. In this way, the number of log files that the electronic device needs to determine whether it is the second type of missing log files can be reduced. The processing capacity of electronic equipment can be reduced, and the efficiency of data integration by electronic equipment can be improved.

In some embodiments, the electronic device can also perform data operations on the target end based on the obtained log files, so that data from multiple data storage nodes at the source end can be aggregated to the target end.

S103. The electronic device obtains the log flag bit based on the second type of missing log file and the current log data identification range.

Wherein, the above log data identification range of this time can be understood as the log data identification range (range starting point, range end point) used by the electronic device when the log file is obtained in the above step S101 during data integration this time. And, the log flag can also be called the snapshot flag.

It can be understood that, in the embodiment of the present application, the log flag bit is obtained based on the second type of missing log files. Therefore, the log flag will be related to the second type of missing log files determined during this data integration. That is to say, the log flag bit will be related to the log file that the electronic device determines that content omission occurs during this data integration. Afterwards, the electronic device uses the log flag bit when obtaining the log file next time, and when obtaining the log file from the source, it can generate a log identification range based on the log flag bit (that is, when the electronic device obtains the log file next time, the electronic device The log identification range used); because this log flag bit is related to the second type of missing log files determined during this data integration. Therefore, when the electronic device obtains the log file next time, it can obtain the second type of missing log file determined during this data integration from the source through the log identification range generated based on the log flag bit. Based on this, the electronic device can mark the second type of missing log files found during this data integration through the log flag, and obtain the second type of missing logs found during this data integration when the log files are obtained next time document. In this way, the electronic device can alleviate the missing data phenomenon in the data integration process, and can improve the data consistency between the source end and the target end, thereby reducing the data distortion problem at the target end.

In some embodiments, if the electronic device does not determine that the log file is missing in step S102. Then in step S103, the electronic device may obtain the log flag bit based on the log data identification range.

Exemplarily, if the electronic device does not determine the second type of missing log file in step S102, the electronic device may directly use the end point of the log data identification range as the log flag in the step.

For example, referring to FIG. 8 , the electronic device has acquired archived log file 1 , archived log file 2 , online log file 1 and online log file 2 . Electronics did not identify a second category of missing log files. Afterwards, the electronic device uses the end of the range as a log flag.

In some embodiments, the electronic device may obtain the log flag bit based on the start log data identifier of the second type of missing log file and the range start point of the current log data identifier range.

For example, the electronic device may refer to the log data identifier, whichever has the largest value among the log data identifier of the second type of missing log file and the start point of the range of the log data identifier range. Next, the electronic device may obtain the log flag bit by using the value identified by the reference data, and the value of the log flag bit may be less than or equal to the value identified by the reference log data. It can be understood that when the value of the log flag bit is equal to the value of the reference log data identifier, that is to say, the log flag bit may be the reference log data identifier.

It can be understood that, from the above description, when the electronic device obtains log files from the source, it obtains log files whose log data identifier is greater than or equal to the start point of the range, or smaller than or equal to the end point of the range. That is to say, if the starting point of the scope is smaller, the number of log files obtained by the electronic device from the source will be larger. The log data identifier of the obtained log file will also be smaller. And, the smaller the log data identifier, the earlier the generation time of the log file at the source end, and the greater the difference from the current time of the electronic device. In this way, in the next data integration, if the electronic device is using a relatively small log data identifier, the log data identifier range used in the next data integration is obtained. Electronic devices may obtain more log files that were generated earlier; that is to say, electronic devices will obtain some log files with poor timeliness, which will affect the real-time performance of data integration by electronic devices.

And, the second type of missing log file is a log file that has been obtained but the content of the log file is missing when the log file is obtained this time. That is to say, the log identifier of the second type of missing log file is a log file that is greater than or equal to the starting point of the current log data identification range (hereinafter referred to as the current starting point for short) (that is, the electronic device can obtain the The second category of missing log files). And, the starting point log data identifier is the smallest log data identifier in the log file. That is to say, the electronic device can also obtain the second type of missing log file by using the starting log data identifier of the second type of missing log file.

Therefore, the electronic device uses the log data identification of the starting point of the second type of missing log file, or the electronic device using this starting point can obtain the second type determined during this data integration when the log file is obtained from the source next time. Missing log files.

Based on this, the electronic device may determine a threshold (that is, refer to the log data identifier) from the log data identifier of the second type of missing log file and the range starting point of the log data identifier range. If the electronic device obtains log files from the source next time, it uses a range starting point whose value is larger than the reference log data identifier to obtain log files from the source; Class II missing date files.

To sum up, if the log flag bit is equal to the reference log data identifier, the electronic device can not only obtain the second type of missing log files determined this time when it obtains log files from the source next time, but also control to a certain extent from The number of log files acquired by the source can reduce the number of log files acquired by the electronic device; it can also control the timeliness of acquiring log files from the source database to a certain extent. It can not only alleviate the missing data problem during data integration, but also improve the efficiency of electronic equipment when acquiring log files, and improve the real-time performance of electronic equipment for data integration.

In some embodiments, the electronic device may use the smallest numerical value among the log data identifier of the missing log file and the range starting point of the log data identifier range as the log flag bit.

Exemplarily, referring to FIG. 9 , the electronic device has obtained the archived log file 1, the archived log file 2, the online log file 1 and the online log file 2. The electronic device determines that online log file 2 is a missing log file. Afterwards, the electronic device compares the log data identifier of the missing log file with the starting point of the range, and uses the larger value of the two, that is, the log data identifier of the online log file 2, as the log flag bit.

Exemplarily, referring to FIG. 10 , the electronic device has acquired online log file 1 and online log file 2 . Next, the electronic device determines that the online log file 1 is a missing log file. Afterwards, the electronic device compares the log data identifier of the missing log file with the starting point of the range, and takes the larger value of the two, that is, the starting point of the range, as the log flag bit.

In some embodiments, when the electronic device determines a plurality of second-type missing log files, the electronic device determines a target missing log file from the plurality of second-type missing log files, and the target missing log file is the above-mentioned plurality of second-type missing log files. Among the second type of missing log files, the log file with the smallest starting point log data identifier. Afterwards, the electronic device uses the log data identifier of the target omission log file and the starting point of the range with a larger value as the reference log data identifier. Next, the electronic device obtains the log flag bit based on the reference log data identification.

If the electronic device has determined multiple second-type missing log files, in order to enable the electronic device to obtain the multiple second-type missing log files determined during this data integration when it obtains the log files next time, it can first Among the second type of missing log files, determine the target missing log file. From the manner in which the electronic device obtains the log file, it can be known that when the electronic device obtains the log file from the source, it obtains the log file whose log data identifier is greater than or equal to the start point of the range. Therefore, if the electronic device uses the target missing log file to obtain log files from the source, it can obtain the above-mentioned multiple second-type missing log files. Based on this, the target missing log file can be determined from multiple second-type missing log files; this can make the electronic device subsequently obtain the log flag bit based on the target missing log file; when obtaining the log file from the source next time, it can be based on The log flag bit, the multiple second-type missing log files determined this time are obtained from the source database.

Exemplarily, referring to FIG. 11 , the electronic device has acquired online log file 1 , online log file 2 and online log file 3 . Next, the electronic device determines that the online log file 1 and the online log file 3 are missing log files. Afterwards, among the missing log files, the electronic device identifies a relatively small log data as the target missing log file, that is, the electronic device takes the online log file 1 as the target missing log file. Next, the electronic device compares the log data identifier of the line log file 1 with the starting point of the range, and takes the larger value of the two, that is, the starting point of the range, as the log flag bit.

Exemplarily, referring to FIG. 12 , the electronic device has acquired online log file 1 , online log file 2 and online log file 3 . Next, the electronic device determines that the online log file 1, the online log file 2, and the online log file 3 are all missing log files. After that, the electronic device identifies the smaller log data as the target missing log file among the missing log files, that is, the electronic device uses the online log file 3 as the target missing log file. Next, the electronic device compares the log data identifier of the online log file 3 with the starting point of the range, and takes the larger value of the two, that is, the starting point of the range, as the log flag bit.

It can be understood that after the electronic device obtains the log flag, the electronic device will be in a non-first data integration scenario when the electronic device performs data integration next time.

Next, step S101 will be supplemented with an electronic device in a non-first data integration scenario.

In the non-first data integration scenario, because the electronic device has performed data integration before, or in a period of time before; that is to say, the electronic device can obtain the historical log flag. Wherein, the historical log flag can be understood as the log flag obtained by the electronic device when performing data integration last time. Based on this, the electronic device can obtain the history log flag. Afterwards, the electronic device obtains the end of the range by means of a dynamic step size or a preset step size. Next, the electronic device obtains the log file from the source based on the log flag and the end point of the range.

Exemplarily, referring to FIG. 13 , the above step S101 may include steps S101b1-S101b4.

S101b1. The electronic device determines that it is not in the first data integration scenario.

It can be understood that, for the implementation manner of step S101b1, reference may be made to the relevant description of step S101a1 above, which will not be repeated here.

S101b2. The electronic device uses the historical log flag as the starting point of the data integration range.

Understandably, since the history log flag is obtained during the last data integration of the electronic device, it can reflect the second type of missing log files determined by the electronic device during the last data integration of the electronic device, that is, the electronic device is in the The starting log file identifier of the second type of missing log file determined during the last data integration, which will be greater than or equal to the historical log flag. And, when the electronic device obtains the log file, what is obtained is the log file whose log data identifier is greater than or equal to the start point of the range. Therefore, by setting the historical log mark as the starting point of this range, the electronic device can obtain the second type of missing log files determined by the electronic device during the last data integration. In this way, the electronic device can obtain the second type of missing log files determined by the electronic device during the last data integration during this data integration. And subsequently integrate the log data in the second type of missing log files determined during the last data integration to the target end. In this way, the phenomenon of missing numbers during data integration can be alleviated, and the data consistency between the source and target ends can be improved, thereby reducing the problem of data distortion at the target end.

S101b3. The electronic device calculates and obtains the end point of the range through the dynamic step size based on the start point of the range during this data integration.

Wherein, the starting point of the range during data integration this time may be the starting point of the range obtained in the above step S101b2.

It can be understood that, for the implementation manner of step S101b3, reference may be made to the relevant description of step S101a3 above, which will not be repeated here.

S101b4. The electronic device obtains the log file from the source based on the range start point and the range end point.

It can be understood that, for the implementation manner of step S101b4, reference may be made to the relevant description of step S101a4 above, which will not be repeated here.

In some embodiments, the above step S101 may also include step S101b5.

S101b5. If the electronic device determines that the source end generates the first type of missing log file, the electronic device executes step S101b4; if the electronic device does not determine that the source end generates the first type of missing log file, the electronic device executes subsequent steps.

It can be understood that, for the implementation manner of step S101b5, reference may be made to the relevant description of step S101a5 above, which will not be repeated here.

In the following, the data processing method provided by the embodiment of the present application will be introduced in combination with multiple data integration processes of the electronic device. The method may include: the electronic device acquires the source-end log file from the source-end database according to the first identification range, where the first identification range is used to indicate the log data identification range of the source-end log file to be acquired by the electronic device. For the implementation of this step, reference may be made to the introduction of the above step S101, which will not be repeated here.

Afterwards, the electronic device performs data operations on the target database based on the acquired source log file.

For example, if the data operation recorded in the log file at the source is to modify data A, the electronic device will modify data A at the target. For another example, if the data operation recorded in the log file at the source is to delete data B, the electronic device will delete data B at the target. For another example, the data operation recorded in the source log file is an addition operation on data C, which adds data C. Then the electronic device will perform an increase operation of data C on the target end, and add data C to the target end.

Next, if the electronic device determines that there is at least one missing log file from the obtained source log files, the electronic device obtains the log flag bit based on the starting log data identifier and the first identification range of the at least one missing log file. The aforementioned missing log file is a log file in which included log data is missing. The starting log data identifier of the missing log file is the log data identifier of the first piece of log data included in the missing log file. For the implementation of this step, reference may be made to the introduction of the above step S102 and step S103, which will not be repeated here.

Then, the electronic device determines a second identification range according to the log flag bit, and obtains the source-end log file from the source-end database according to the second identification range.

For the implementation of this step, reference may be made to the introduction of the above step S101, which will not be repeated here.

In the following, the data processing method provided in the embodiment of the present application will be illustrated by taking data integration of source data in an order processing business scenario as an example.

Exemplarily, in the order processing business scenario, the source end in the data storage system may include three data storage nodes (node 1, node 2, and node 3), and the target end may include two data storage nodes (node a and node b ). In the order processing business scenario, the source end can be used to receive order information and perform data operations on the source end based on the order information. For example, assuming that the merchant puts 100 items of product A on the shelves, the source end will receive the data operation of adding data, and add the data "product A, quantity 100" to the data file at the source end. For another example, if the user places an order for 10 items of product A, the source end will receive the data modification operation and change the "product A, quantity 100" in the data file to "product A, quantity 90". For another example, when a merchant removes product A from the shelves, the source end will receive a data operation to delete the data, and delete the data "product A, quantity 90" in the data file at the source end. And, the target end can be used to synchronize the data of the source end, and based on the data of the target end, carry out order delivery, product quantity display, and so on.

Assume, as shown in Figure 14, that the source end receives data file 1, data file 2, and data file 3; among them, data file 1 is "commodity A, quantity 100", and data file 2 is "commodity B, quantity 200", Data file 3 is "commodity C, quantity 150". Afterwards, the data storage node at the source will generate log data for the above-mentioned data file 1, data file 2, and data file 3. And randomly store the log data into the above three data storage nodes (node 1, node 2 and node 3). For example, the source can store log data 1 for data file 1 in node 1, log data 2 for data file 2 in node 2, and store log data 3 for data file 3 in node 3. After that, the source end receives the order placing operation, the source end will add data file 4 "customer A, purchase 50 items of product A", and the source end will modify data file 1, and modify data file 1 to "product A, quantity 50" . Next, the source end will generate log data 4 for data file 4, and store the log data in node 2; and, the source end will generate log data 5 for the modification operation of data file 1, and store the log data 5 Deposit in node 3. Afterwards, the source end receives the removal operation of product C, and the source end deletes data file 3. After that, the source end generates log data 6 for the operation of deleting data file 3, and stores the log data in node 1. At this time, the online log file a1 of node 1 has reached the preset size, and the node will archive the log file a1 and store the log data 6 in the log file a2.

Based on this, log file a1 of node 1 will record log data 1, log file a2 of node 1 will record log data 6; log file b1 of node 2 will record log data 2 and log data 4, node The log file c1 of 3 will record log data 3 and log data 5. And, when the source end generates log data, it will generate the log identifier (for example, scn) of the log data at the same time; for example, the scn of log data 1 is 1001, the scn of log data 2 is 1002, and the scn of log data 3 is 1003, the scn of log data 4 is 1005, the scn of log data 5 is 1008, and the scn of log data 6 is 1010. That is to say, the start log data identifier of log file a1 is 975, the end log data identifier of log file a1 is 1001, the start log data identifier of log file b1 is 980, and the start log data identifier of log file c1 is 1000. The starting log data identifier of the log file a2 is 1010.

Exemplarily, the log data identification of the log file is shown in Table 1 below.

Table 1

Among them, since the log file a2, the log file b1, and the log file c1 are online log files, and will be written into log data at any time, they do not have an end-point log data identifier based on this.

It is understandable that during the process of generating the above log data 1-6 at the source end, the source end will also generate some log data (not shown in the figure) for the source end database system, and these data will also occupy scn, so the above log The scn of data 1-6 may not be consecutive.

Next, the electronic device performs data integration, integrating the data of the three nodes (node 1, node 2 and node 3) at the source end to the target end. Because the electronic device has not acquired the history log flag; therefore, the electronic device is determined to be in the first integration scene. The electronic device loads the log file from the source for the first time. Next, the electronic device obtains the starting point of the specified range, such as 988. Afterwards, the electronic device will calculate through the dynamic step size, and the terminal identified by the log data is 1011, that is, the range of the log data identification is [988,1011]. Then, the electronic device obtains the log file from the source based on the log data identification range.

Exemplarily, referring to FIG. 14 again, the log data identifiers of the log file a1, log file a2, log file b1, and log file c1 all overlap with the range of the above-mentioned log data identifiers. Therefore, the electronic device needs to obtain the log file a1, log file a2, log file b1 and log file c1. However, because the log file a1 is being archived, the electronic device does not obtain the log file a1. That is to say, the electronic device has obtained the log file a2, the log file b1 and the log file c1. Afterwards, the electronic device obtains the archived log file number of node 1 (for example, the log file number of log file a0) and compares the log file number of log file a2 and finds that the two are not continuous, and a log file switch has occurred. Next, the electronic device obtains the log files from the source again; for example, the electronic device obtains the log file a1, the log file a2, the log file b1, and the log file c1. Then, the electronic device determines that no file switching has occurred in the acquired log file. In this way, the missing data caused by the switching of online log files at the source can be alleviated.

Next, since the log file a1, the log file a2, the log file b1 and the log file c1 are all encrypted, the electronic device decrypts the log file a1, the log file a2, the log file b1 and the log file c1; and after decryption , based on the log data in log file a1, log file a2, log file b1, and log file c1, perform data operations on the target end.

Afterwards, the electronic device can read the log data in the analyzed log file a1, log file a2, log file b1 and log file c1, if the log file b1 does not read the log data, then determine that the log file b1 is a missing log file . Alternatively, the electronic device may also perform data operations on the target end through the parsed log file a1, log file a2, log file b1, and log file c1. For example, when the data operation is performed based on the log file c1, the target end does not obtain the data operation corresponding to the log data in the log file c1, and the data at the target end does not change; the electronic device determines that the log file c1 is a missing log file.

Next, the electronic device determines the target missing file in the log file b1 and the log file c1. It can be seen from FIG. 14 that the start log data identifier (for example, scn=980) of the log file b1 is smaller than the start log data identifier (for example, scn=1000) of the log file c1. Therefore, the electronic device determines that the log file b1 is the target missing file.

Then, the electronic device takes the log data identifier of the starting point of the log file b1 and the starting point of the range, whichever has the largest value, as the log flag bit. That is, the starting point of the range is used as the log flag, which is 988.

Exemplarily, the numerical comparison between the missing log file and the starting point of the range can be found in Table 2 below.

Table 2

Next, the electronic device obtains the log data identification range based on the log flag obtained during the first data integration, that is, 988. And based on the log data identification range, data integration is performed for the second time to obtain log files from the source. For example, log file a1, log file a2, log file b1, and log file c1 are obtained. Afterwards, the electronic device parses the above-mentioned log file a1, log file a2, log file b1, and log file c1, and then performs data operations on the target end. It can be understood that for some targets, there are some mechanisms for avoiding repeated data operations. This mechanism filters out the same data operations on the target side. Therefore, even if the electronic device performs data operations on the target end again based on the parsed log file a1 and log file a2, the data at the target end will not be confused.

In this way, during the second data integration, the electronic device can obtain the log file b1 and log file c1 that were missed in the first data integration again. In this way, the situation that the log file b1 and the log file c1 are not integrated into the target end can be reduced. It can alleviate the situation that the data at the target end is distorted because the target end does not obtain log data 2, log data 3, log data 4, and log data 5. It can improve the accuracy of order delivery and product quantity display based on the target data.

In this way, the electronic device performs data integration this time by using the log flag bits obtained from the last data integration. The electronic device can obtain the data from the source that was missed in the previous data integration during this data integration. In this way, the phenomenon of missing numbers in the data integration process can be alleviated, and the consistency of the source and target data can be improved, thereby Reduce the problem of data distortion on the target side.

Then, when data integration is performed for the second time, since the electronic device does not read the content of the log file c1, the electronic device determines that the log file c1 is a missing file.

Next, the electronic device takes the log data identifier (for example, scn=1000) and the range start point (for example, scn=988) of the log file c1 as the log flag, whichever has the largest value. That is, the starting log data identifier of the log file c1 is used as the log flag bit, that is, scn=1000.

Exemplarily, the numerical comparison between the missing log file and the starting point of the range can be found in Table 3 below.

table 3

Afterwards, when the electronic device performs the next data integration, it will use the log flag bit obtained by this data integration, that is, scn=1000, to obtain the log data identification range. And based on the log data identification range, the log file is obtained from the source for the third time.

In some embodiments, if the log file a1 has been archived for a long time, the electronic device fails to obtain the log file a1 for many times (3 times, 5 times, etc.). The electronic device may use the log file a1 as a switching log file. Afterwards, the electronic device may send a query request to the source to query the start log data identifier and the end log data identifier of the log file a1. In response to the log query request, the source end may send the start log data identifier or the end log data identifier of the log file a1 to the electronic device. In this way, the electronic device can obtain the log file a1 from the source through the start log data identifier (for example, 975) or the end log data identifier (for example, 980) of the log file a1. For example, the log file a1 is acquired from the source in the form of specifying the log data identifier as 975 or specifying the log data identifier as 980. And in the follow-up, data operations are performed on the target database based on the source log file a1.

It can be understood that, in order to realize the above functions, the electronic device includes hardware and/or software modules corresponding to each function. Combining the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In this embodiment, the functional modules of the electronic device may be divided according to the above method example. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division methods in actual implementation.

Exemplarily, referring to FIG. 15 , the embodiment of the present application also provides a data integration device, which includes: a log acquisition module, a log switch detection module, a log analysis module, a log data operation module, and a log identification snapshot module.

Wherein, the log obtaining module is configured to realize the functions of the above steps S101a1-S101a4, or realize the functions of the above steps S101b1-S101b4. For details, refer to the introduction of the above-mentioned related steps, which will not be repeated here.

The log switching detection module is configured to realize the function of the above step S101a5 or S101b5. For details, refer to the introduction of the above-mentioned related steps, which will not be repeated here.

The log parsing module is configured to realize the above-mentioned function of parsing log files. For details, refer to the introduction of the above-mentioned related steps, which will not be repeated here.

The log data operation module is configured to realize the above-mentioned function of performing data operations on the target end based on the log files. For details, refer to the introduction of the above-mentioned related steps, which will not be repeated here.

The snapshot module is configured to realize the function of the above step S102 or S103. For details, refer to the introduction of the above-mentioned related steps, which will not be repeated here.

The embodiment of the present application also provides an electronic device. As shown in FIG. 16 , the electronic device may include one or more processors 1001 , a memory 1002 and a communication interface 1003 .

Wherein, the memory 1002 and the communication interface 1003 are coupled with the processor 1001 . For example, the memory 1002 , the communication interface 1003 and the processor 1001 may be coupled together through the bus 1004 .

Wherein, the communication interface 1003 is used for data transmission with other devices. Computer program code is stored in the memory 1002 . The computer program code includes computer instructions, and when the computer instructions are executed by the processor 1001, the electronic device performs the device authentication in the embodiment of the present application.

Wherein, the processor 1001 may be a processor or a controller, such as a central processing unit (central processing unit, CPU), a general processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various illustrative logical blocks, modules and circuits described in connection with the present disclosure. The processor may also be a combination of computing functions, for example, a combination of one or more microprocessors, a combination of DSP and a microprocessor, and so on.

Wherein, the bus 1004 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus or the like. The above bus 1004 can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 16 , but it does not mean that there is only one bus or one type of bus.

The embodiment of the present application also provides a computer-readable storage medium, the computer storage medium stores computer program code, and when the above-mentioned processor executes the computer program code, the electronic device executes the relevant method steps in the above-mentioned method embodiment.

The embodiment of the present application also provides a computer program product, which, when running on a computer, causes the computer to execute the relevant method steps in the above method embodiments.

Wherein, the electronic equipment, computer-readable storage medium or computer program product provided in this application are all used to execute the corresponding method provided above, therefore, the beneficial effects that it can achieve can refer to the corresponding method provided above The beneficial effects in the above will not be repeated here.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned by Completion of different functional modules means that the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places . Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the essence of the technical solution of the embodiment of the present application or the part that contributes or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several The instructions are used to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (random access memory, RAM), magnetic disk or optical disk, and other media capable of storing program codes.

The above content is only the specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered within the protection scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (11)

1. A data processing method, characterized in that the method comprises: Obtain the log file from the source database according to the first identification range, where the first identification range is used to indicate the range of the log data identification of the log file to be acquired; If it is determined from the acquired log files that there is at least one missing log file, then the log flag bit is obtained based on the starting point log data identifier of the at least one missing log file and the first identification range; the missing log file is obtained from the Describe the log files obtained by the source database and missing log data; Determine a second identification range according to the log flag, and obtain a log file from the source database according to the second identification range; Wherein, the log files obtained from the source database are used to perform data operations on the target database. 2. The method according to claim 1, wherein said obtaining log flags based on the starting point log data identification of said at least one missing log file and said first identification range comprises: When it is determined from the log files obtained according to the first identification range that there is a missing log file, the maximum value of the starting point log data identification of the one missing log file and the range starting point of the first identification range is determined. The log data identifier is used as the log flag bit; Or, when it is determined from the log files obtained according to the first identification range that there are at least two missing log files, determine the target missing log file in the at least two missing log files; Among the log data identifier of the starting point of the log file and the range starting point of the first identifier range, the largest log data identifier is used as the log flag bit; Wherein, the target missing log file is the missing log file with the smallest starting log data identifier among the at least two missing log files. 3. The method according to claim 1, wherein the determining that there is at least one missing log file from the obtained log file comprises: The log data is read from the log files obtained based on the first identification range, and if there is a log file whose log data cannot be read, it is determined that there is at least one missing log file. 4. The method according to claim 1, wherein said obtaining the log file from the source database according to the first identification range comprises: If each log file corresponding to the first identification range is not obtained, then reacquire the log file according to the first identification range until each log file corresponding to the first identification range is obtained; The method also includes: Perform data operations on the target database based on each acquired log file corresponding to the first identification range. 5. The method according to claim 1, wherein the obtaining the log file from the source database according to the first identification range comprises: If each log file corresponding to the first identification range is not obtained, then re-acquire the log file according to the first identification range until the number of acquisitions of the log file according to the first identification range is greater than or equal to the preset The number of thresholds; The method also includes: Perform data operations on the target database according to the last acquired log file. 6. The method according to claim 5, wherein the log file includes a log file number, and the method also includes: Determine the log file number of the missing log file, where the missing log file is a log file corresponding to the first identification range that has not been obtained; According to the log file number of the missing log file, the starting log data identifier or the ending log data identifier of the missing log file is obtained; Acquiring the missing log file from the source database according to the starting log data identifier or the ending log data identifier of the missing log file; Perform data operations on the target database based on the missing log file. 7. The method according to any one of claims 4-6, wherein the unacquired log file corresponding to the first identification range includes a switching log file; the switching log file is the source end A log file of ongoing archiving operations by the database. 8. The method according to any one of claims 1-3, wherein the log files include: online log files or archived log files, The obtaining log files from the source database according to the first identification range includes: In the case that the log file is the online log file, acquiring the online log file whose starting point log data identifier of the online log file is less than or equal to the end point of the range of the first identification range; In the case that the log file is the archived log file, the log data identifier of the starting point of the archived log file is less than or equal to the range end point of the first identification range, and greater than or equal to the range start point of the first identification range or, acquire the archived log files whose endpoint log data identifiers of the archived log files are less than or equal to the end point of the first identified range and greater than or equal to the start point of the first identified range. 9. The method according to any one of claims 1-6, wherein the method further comprises: If it is determined from the obtained log files that the missing log file does not exist, the log flag bit is obtained based on the end point of the first identification range. 10. An electronic device, characterized in that, the electronic device includes a memory, one or more processors, and the memory is coupled to the processor; wherein, computer program codes are stored in the memory, and the computer The program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device is made to execute the method according to any one of claims 1-9. 11. A computer-readable storage medium, characterized by comprising computer instructions, and when the computer instructions are run on an electronic device, the electronic device is made to execute the method according to any one of claims 1-9.