CN112527903A - Data synchronization method based on canal platform - Google Patents
Data synchronization method based on canal platform Download PDFInfo
- Publication number
- CN112527903A CN112527903A CN202011445008.0A CN202011445008A CN112527903A CN 112527903 A CN112527903 A CN 112527903A CN 202011445008 A CN202011445008 A CN 202011445008A CN 112527903 A CN112527903 A CN 112527903A
- Authority
- CN
- China
- Prior art keywords
- data synchronization
- database
- server
- data
- computer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
- H04L41/0816—Configuration setting characterised by the conditions triggering a change of settings the condition being an adaptation, e.g. in response to network events
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Environmental & Geological Engineering (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Transfer Between Computers (AREA)
- Synchronisation In Digital Transmission Systems (AREA)
Abstract
The invention discloses a data synchronization method based on a canal platform, which relates to the technical field of data synchronization.
Description
Technical Field
The invention relates to the technical field of data synchronization, in particular to a data synchronization method based on a canal platform.
Background
With the increasing of domestic internet flow, the service functions of each large internet company are also regionally complicated, services among service modules are also decoupled, so that databases and services among the modules and among the modules are independently deployed, and when the A service module is modified and the B service module wants to know the modified content and data immediately, the database data of the A service module is required to be synchronized to the B service module in real time.
The current scheme for data synchronization mainly comprises the following schemes:
(1) timing pull synchronization scheme
The C server modifies the database data, and the D server regularly goes to the C server to pull the data. Due to the fact that the timing pulling time has intervals, pulling is too frequent, the database of the server C cannot be used, pulling is too slow, and the data of the server D are not real-time. Meanwhile, in the service scene, a database is added for synchronization, and a set of timing pull function is required to be written, so that the development cost is increased.
(2) Sending message queue asynchronous consumption scheme
After the E-server successfully modifies the data in the database, it sends a message queue to the F-server telling the F-server that those contents have been modified. The F server consumes the message queue again, and the data can be obtained. Every time a service scene is added, the service of the MQ sending party needs to write a function of sending messages, development cost is increased, and dynamic expansion is not supported.
Disclosure of Invention
In order to solve the defects of the prior art, an embodiment of the present invention provides a data synchronization method based on a canal platform, which includes the following steps:
acquiring a binary log generated by an operation database in real time by using a canal platform;
judging whether data modification operation exists in the database or not according to the binary log;
and if so, acquiring the modified data in the database according to a preset frequency.
Preferably, after obtaining the modified data in the database, the method further comprises:
and judging whether the difference value between the current time and the self-increment time field in the data is greater than a set threshold value, if so, sending corresponding alarm information.
Preferably, the method further comprises:
and judging whether the connection state of data synchronization based on the canal platform is normal or not in real time, and if not, sending corresponding alarm information.
Preferably, the method further comprises:
according to the pressure of data synchronization, multiple servers are deployed.
Preferably, the method further comprises:
aiming at a plurality of servers synchronizing the same database, a master server and a slave server are arranged for each server in the plurality of servers according to the time for the server to occupy synchronous connection.
The data synchronization method based on the canal platform provided by the embodiment of the invention has the following beneficial effects:
(1) based on the canal platform, the modified content of the database can be obtained in real time by obtaining the binary log generated by operating the database. As long as the database is modified, the modified content can be obtained immediately and is suitable for different service scenes, so that the real-time performance is improved, and the development cost is reduced;
(2) aiming at the magnitude of the later pressure of the synchronization service, the dynamic capacity expansion of the server is realized, and the pressure of data synchronization is reduced.
Detailed Description
The present invention will be described in detail with reference to the following embodiments.
The data synchronization method based on the canal platform provided by the embodiment of the invention comprises the following steps:
s101, a binary log generated by the operation database is obtained in real time by using a canal platform.
Wherein, the canal platform has a dynamic configuration for each database.
S102, judging whether the database has data modification operation according to the binary log.
And S103, if the frequency exists, acquiring the modified data in the database according to the set frequency.
As a specific example, the set frequency is 0.01 second/time.
Optionally, after obtaining the modified data in the database, the method further comprises:
and judging whether the difference value between the current time and the self-increment time field in the data is greater than a set threshold value, if so, sending corresponding alarm information.
As a specific embodiment, when the difference is greater than 5 seconds, synchronization is not real-time, and a warning message of synchronization not being real-time is sent.
Optionally, the method further comprises:
and judging whether the connection state of data synchronization based on the canal platform is normal or not in real time, and if not, sending corresponding alarm information.
Optionally, the method further comprises:
according to the pressure of data synchronization, multiple servers are deployed.
As a specific embodiment, for the IP address of each server, based on the gray scale distribution configuration function of the Apollo configuration center, the control server a connects three instances a, B, and c of the canal platform, the control server B connects three instances d, e, and f of the canal platform, and if it is desired to continue capacity expansion, the instances connected to the canal platform may be dynamically allocated.
The server a and the server B are servers in which 2 different service modules in the same system are located.
Optionally, the method further comprises:
aiming at a plurality of servers synchronizing the same database, a master server and a slave server are arranged for each server in the plurality of servers according to the time for the servers to occupy synchronous connection.
As a specific example, only one of the plurality of servers can be active, and the other servers can only wait for synchronization. For example, for the server a and the server B, if the server a first occupies the synchronization connection, the server B is in a waiting state, and when the server a is disconnected, the server B automatically starts the synchronization work.
The data synchronization method based on the canal platform provided by the embodiment of the invention utilizes the canal platform to obtain the binary log generated by operating the database in real time, judges whether the operation for modifying the data exists in the database according to the binary log, and obtains the modified data in the database according to the set frequency if the operation for modifying the data exists, thereby improving the real-time property, reducing the development cost and lightening the pressure of data synchronization.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (7)
1. A data synchronization method based on a canal platform is characterized by comprising the following steps:
acquiring a binary log generated by an operation database in real time by using a canal platform;
judging whether data modification operation exists in the database or not according to the binary log;
and if so, acquiring the modified data in the database according to the set frequency.
2. The canal platform-based data synchronization method of claim 1, further comprising, after obtaining the modified data in the database:
and judging whether the difference value between the current time and the self-increment time field in the data is greater than a set threshold value, if so, sending corresponding alarm information.
3. The canal platform-based data synchronization method of claim 1, further comprising:
and judging whether the connection state of data synchronization based on the canal platform is normal or not in real time, and if not, sending corresponding alarm information.
4. The canal platform-based data synchronization method of claim 1, further comprising:
according to the pressure of data synchronization, multiple servers are deployed.
5. The canal platform-based data synchronization method of claim 4, further comprising:
aiming at a plurality of servers synchronizing the same database, a master server and a slave server are arranged for each server in the plurality of servers according to the time for the server to occupy synchronous connection.
6. A computer program product, characterized in that the computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the method of claims 1-5.
7. A non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the method of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011445008.0A CN112527903A (en) | 2020-12-11 | 2020-12-11 | Data synchronization method based on canal platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011445008.0A CN112527903A (en) | 2020-12-11 | 2020-12-11 | Data synchronization method based on canal platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112527903A true CN112527903A (en) | 2021-03-19 |
Family
ID=75000285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011445008.0A Pending CN112527903A (en) | 2020-12-11 | 2020-12-11 | Data synchronization method based on canal platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527903A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070242765A1 (en) * | 2006-04-14 | 2007-10-18 | Vladimir Parizhisky | Methods and apparatus related to composite beacon and wideband synchronization signaling |
| CN108804237A (en) * | 2017-05-05 | 2018-11-13 | 北京京东尚科信息技术有限公司 | Data real-time statistical method, device, storage medium and electronic equipment |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
| CN110275893A (en) * | 2019-06-11 | 2019-09-24 | 上海上湖信息技术有限公司 | A kind of data manipulation method and device |
| CN110750594A (en) * | 2019-09-30 | 2020-02-04 | 上海视云网络科技有限公司 | Mysql-based real-time cross-network database synchronization method for incremental logs |
| CN110781197A (en) * | 2019-09-29 | 2020-02-11 | 北京淇瑀信息科技有限公司 | A kind of hive offline synchronization verification method, device and electronic equipment |
| CN111177173A (en) * | 2020-01-02 | 2020-05-19 | 普元信息技术股份有限公司 | System and method for realizing data synchronization optimization processing under big data environment |
| CN112000737A (en) * | 2020-08-14 | 2020-11-27 | 苏州浪潮智能科技有限公司 | Data synchronization method, system, terminal and storage medium based on multi-cloud management |
-
2020
- 2020-12-11 CN CN202011445008.0A patent/CN112527903A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070242765A1 (en) * | 2006-04-14 | 2007-10-18 | Vladimir Parizhisky | Methods and apparatus related to composite beacon and wideband synchronization signaling |
| CN108804237A (en) * | 2017-05-05 | 2018-11-13 | 北京京东尚科信息技术有限公司 | Data real-time statistical method, device, storage medium and electronic equipment |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
| CN110275893A (en) * | 2019-06-11 | 2019-09-24 | 上海上湖信息技术有限公司 | A kind of data manipulation method and device |
| CN110781197A (en) * | 2019-09-29 | 2020-02-11 | 北京淇瑀信息科技有限公司 | A kind of hive offline synchronization verification method, device and electronic equipment |
| CN110750594A (en) * | 2019-09-30 | 2020-02-04 | 上海视云网络科技有限公司 | Mysql-based real-time cross-network database synchronization method for incremental logs |
| CN111177173A (en) * | 2020-01-02 | 2020-05-19 | 普元信息技术股份有限公司 | System and method for realizing data synchronization optimization processing under big data environment |
| CN112000737A (en) * | 2020-08-14 | 2020-11-27 | 苏州浪潮智能科技有限公司 | Data synchronization method, system, terminal and storage medium based on multi-cloud management |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3490224B1 (en) | Data synchronization method and system | |
| CN111641700B (en) | Ceph object-based management and retrieval implementation method for storage metadata | |
| CN112804332B (en) | Message processing system, method, device, equipment and computer readable storage medium | |
| CN111078426A (en) | High concurrency solution under back-end micro-service architecture | |
| CN101667181A (en) | Method, device and system for data disaster tolerance | |
| CN107656937B (en) | Method and apparatus for realizing read and write data consistency | |
| CA3131954A1 (en) | Information synchronization method, system, apparatus, computer device and storage medium | |
| CN108055322B (en) | Request message processing method and device | |
| CN110442564B (en) | Incremental data verification method and storage medium | |
| CN111460038A (en) | Quasi-real-time data synchronization method and device | |
| CN108153794B (en) | Page cache data refreshing method, device and system | |
| CN111931105B (en) | A method for processing data with specified push time for Kafka consumption | |
| CN115567477B (en) | Method, equipment and storage medium for processing message accumulation | |
| CN112258266A (en) | Distributed order processing method, device, equipment and storage medium | |
| CN112527903A (en) | Data synchronization method based on canal platform | |
| CN114594907A (en) | Data read-write processing method, device, equipment and storage medium | |
| CN107404511B (en) | Method and device for replacing servers in cluster | |
| CN111831221A (en) | Distributed storage method and system based on cloud storage | |
| CN109471595A (en) | A kind of data read-write method and system | |
| CN112615944B (en) | Method and system for master-slave synchronization of distributed DNS (Domain name System) | |
| CN109005059A (en) | A kind of system and method for realizing Redis automated back-up | |
| CN116149814A (en) | KAFKA-based data persistence task distributed scheduling method and system | |
| CN113961538A (en) | Method and system for supporting automatic data synchronization of multi-source heterogeneous database | |
| CN117992257B (en) | Parallel data acquisition and processing method for distributed database | |
| CN112217712A (en) | Message multi-terminal synchronization method and client |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210319 |