[go: up one dir, main page]

CN109800239B - Redis-based distributed architecture data sharing method - Google Patents

Redis-based distributed architecture data sharing method Download PDF

Info

Publication number
CN109800239B
CN109800239B CN201910036828.5A CN201910036828A CN109800239B CN 109800239 B CN109800239 B CN 109800239B CN 201910036828 A CN201910036828 A CN 201910036828A CN 109800239 B CN109800239 B CN 109800239B
Authority
CN
China
Prior art keywords
data
redis
node
memory
subscription
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910036828.5A
Other languages
Chinese (zh)
Other versions
CN109800239A (en
Inventor
冯强中
范文斌
王亚平
吴彤彤
刘成彪
王涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ustc Sinovate Software Co ltd
Original Assignee
Ustc Sinovate Software Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ustc Sinovate Software Co ltd filed Critical Ustc Sinovate Software Co ltd
Priority to CN201910036828.5A priority Critical patent/CN109800239B/en
Publication of CN109800239A publication Critical patent/CN109800239A/en
Application granted granted Critical
Publication of CN109800239B publication Critical patent/CN109800239B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a distributed architecture data sharing method based on Redis, and relates to the field of data processing. The invention comprises the following steps: step S1, initializing cache data: after each node of the distributed system is started, loading data in redis into a memory of each node; step S2, deploying subscription listening: defining a subscription callback method, subscribing a redis channel by each node, and deploying monitoring; step S3, data synchronization: and after the data change is monitored, updating the memory data of each node. The invention uses the redis as a cache data synchronization manager in the distributed architecture system, and utilizes the releasing and subscribing mechanism of the redis to synchronize the cache data of the distributed multi-nodes in real time, thereby realizing the data consistency among the multi-nodes, simultaneously, the cache data in the redis can be self-defined according to the actual requirement of the service system, and the configurable releasing and subscribing functions of the cache data are realized, and the invention is not limited by the specific service system and has strong flexibility.

Description

Redis-based distributed architecture data sharing method
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a distributed architecture data sharing method based on Redis.
Background
Redis is a fully open source free high performance key-value in-memory database. Various data storage structures such as character strings (string), lists (list), sets (set), hash tables (hash) and the like are supported. Redis has a publish-subscribe (pub/sub) messaging mode and is therefore useful in scenarios such as caching, data publishing and subscribing, high-speed message queuing, etc.
CAP theory in distributed domain: any distributed system cannot simultaneously meet the three basic requirements of Consistency, Availability and Partition tolerance. In a distributed architecture, high availability is realized by adopting a multi-node deployment mode, partition fault tolerance is one of the most basic requirements, and how to balance consistency and availability is a main difficult problem facing the distributed architecture. Therefore, the BASE theory arises from the end that the BASE theory is based on the CAP theory, and the result of balancing consistency and availability in the CAP theory is summarized through the distributed practice of a large-scale internet system.
In the specific solution process of realizing the data consistency, a data consistency model is derived, wherein the data consistency model is strong consistency, weak consistency and final consistency. The strong consistency means that after the updating operation is completed, the access of any plurality of subsequent operations returns the latest updated value, and the data acquired each time is forced to be the latest data, so that the accuracy of the data is ensured. The weak consistency means that after the updating operation is completed, the system does not ensure that the access of the subsequent operation returns the latest updated value, and does not ensure that the latest value can be accessed after a long time, and a compensation measure of data consistency is not taken in the implementation; the final consistency is a special form of weak consistency, which means that a system ensures that a value of the last updating operation is finally returned on the premise of no subsequent updating operation, and two implementation schemes of message queue real-time triggering data compensation and timing task triggering data compensation are mainly provided. The existing scheme mainly has the following defects:
(1) the two-stage submission protocol is a synchronous blocking type, the performance of a distributed system is greatly limited, a single-point fault problem exists, a perfect fault-tolerant mechanism is not designed, and the failure of any node can cause the failure of the whole transaction;
(2) although the three-stage submission protocol solves the problem of blocking and single-point failure of the two-stage submission protocol, when partition inconsistency is introduced, the nodes cannot communicate, and data inconsistency is affected;
(3) although strong consistency implementation schemes such as a TCC (transmission control center) protocol and a Paxos algorithm can ensure strong consistency of data, the performance is greatly influenced during concurrency, and the strong consistency is not the best solution scheme in a distributed system;
(4) the data compensation scheme triggered by the message queue is influenced by the performance of the message queue, and once the performance of the message queue is reduced, the data cannot be synchronized in real time, so that the data consistency is influenced;
(5) the timing task data compensation scheme can frequently carry out data synchronization operation when the timing time interval is too small, and thus the system performance is influenced; when the timing time interval is too large, data synchronization delay is caused, and data consistency is affected.
Disclosure of Invention
The invention aims to provide a Redis-based distributed architecture data sharing method, which is characterized in that through using Redis as a cache data synchronization manager in a distributed architecture system, the cache data of distributed multiple nodes are synchronized in real time by using a releasing and subscribing mechanism of the Redis, so that the data consistency among the multiple nodes is realized, and the problems that the existing node data is inconsistent and the system performance is influenced by data compensation are solved.
In order to solve the technical problems, the invention is realized by the following technical scheme:
the invention relates to a data sharing method of a distributed architecture based on Redis, which comprises the following steps:
step S1, initializing cache data: after each node of the distributed system is started, loading data in redis into a memory of each node;
step S2, deploy subscription listening: defining a subscription callback method, subscribing a redis channel by each node, and deploying monitoring;
step S3, data synchronization: after monitoring the data change, updating the memory data of each node;
in step S2, the specific steps of deploying subscription listening are as follows:
step S21: defining a callback method, and automatically calling the callback method to process subscribed data when data is published;
step S22: when subscription data are published, adding the data into a built-in message queue, polling the message queue, and taking out the data in the message queue for processing when the data exist;
step S23: subscribing a configured redis channel by using a data subscription method of the redis, and monitoring on a defined callback method;
step S24: after the subscription monitoring is deployed, each node deploys the subscription of the redis channel and sets monitoring;
in step S3, the specific data synchronization method includes the following steps:
step S31: when data of an application system related to cache data in a node memory is added or deleted, the data is updated into redis;
step S32: the data are issued to the configured redis channel by using a data issuing method of the redis;
step S33: each node in step S23 will monitor the data issue, and call a defined callback method, and add the data to the built-in message queue in the callback method;
step S34: the built-in message queue monitors that data exist in the queue, takes out the data in the queue, acquires the latest data from the redis, and updates the corresponding memory data in the node;
step S35: executing a monitoring method for the change of the node memory data;
step S36: and finishing data synchronization.
Preferably, in step S1, the specific steps of initializing the cache data are as follows:
step S11: determining service data to be loaded into a memory according to system service requirements;
step S12: after the distributed nodes are started, firstly loading configuration parameters, and then initializing a connection pool of redis according to the configured redis addresses;
step S13: acquiring a connection instance of the redis from the redis connection pool, and performing regular matching according to the configured key prefix of the redis to query all matched key lists in the redis;
step S14: traversing the key list, acquiring data from the redis, and storing the acquired entity object data into a memory of the node;
step S15: executing a monitoring method for the change of the node memory data;
step S16: and after the initialization of the cache data is completed, each node stores a piece of data mirror image of redis in a memory after being started.
Preferably, in step S12, the loaded configuration parameters include: the prefix of key in the redis, the name of the redis channel, the connecting address of the redis, and the entity types subscribed and published.
The invention has the following beneficial effects:
(1) the invention achieves the final consistency of the data of the distributed system by combining the service characteristics of the distributed system under the guidance of the CAP theory and the BASE theory and adopting a redis-based mode, thereby having theoretical and practical basis.
(2) According to the invention, each node in the distributed architecture system subscribes data in the same channel by using a redis publishing and subscribing mechanism, when the data changes, the redis publishes the data change, each node subscribing the channel can acquire the data change and update the data in real time, the data consistency among each node is maintained, the performance of the distributed architecture system and the data consistency are ensured, and the data access speed is improved.
(3) The cache data in the redis can be customized according to the actual requirements of the service system, and the configurable publishing and subscribing functions of the cache data are realized, so that the method is not limited by the specific service system and has strong flexibility.
Of course, it is not necessary for any product in which the invention is practiced to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a Redis-based distributed architecture data sharing method according to the present invention;
FIG. 2 is a detailed flow chart of cache data initialization;
FIG. 3 is a detailed flow diagram of deployment subscription monitoring;
fig. 4 is a detailed flow chart of data synchronization.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, the present invention is a method for data sharing based on a Redis distributed architecture, including the following steps:
step S1, initializing cache data: after each node of the distributed system is started, loading data in the redis into a memory of each node;
step S2, deploy subscription listening: defining a subscription callback method, subscribing a redis channel by each node, and deploying monitoring;
step S3, data synchronization: when the data change is monitored, updating the memory data of each node, and keeping the data consistency;
referring to fig. 3, in step S2, the specific steps of deploying subscription listening are as follows:
step S21: defining a callback method, and automatically calling the callback method to process subscribed data when data is published;
step S22: in the callback method, in order to decouple the processing logic of different subscription data, a built-in message queue mode is adopted, when the subscription data is published, the data is added into the built-in message queue, the message queue is polled at the same time, and the data in the message queue is taken out for processing when the data exists;
step S23: subscribing a configured redis channel by using a data subscription method of the redis, and monitoring on a defined callback method;
step S24: after the subscription monitoring is deployed, each node deploys the subscription of the redis channel and sets monitoring;
referring to fig. 4, in step S3, the specific data synchronization method includes the following steps:
step S31: when data of an application system associated with cache data in a node memory is subjected to add, delete and modify operations, updating and synchronizing the memory data of each node, and updating the data into redis;
step S32: after data updating in the redis is finished, releasing data to a configured redis channel by using a data release method of the redis;
step S33: each node in step S23 will monitor the data issue, and call a defined callback method, and add the data to the built-in message queue in the callback method;
step S34: the built-in message queue monitors that data exist in the queue, takes out the data in the queue, acquires the latest data from the redis, and updates the corresponding memory data in the node;
step S35: executing a monitoring method for the change of the node memory data;
step S36: and finishing data synchronization, and when the data changes, updating the memory data in each node to keep data consistency.
Referring to fig. 2, in step S1, the specific steps of initializing the cache data are as follows:
step S11: determining which common service data of the system need to be cached according to the service requirements of the system, and determining the service data which needs to be loaded into a memory;
step S12: after the distributed node is started, firstly loading configuration parameters, and then initializing a connection pool of the redis according to the configured redis address;
step S13: acquiring a connection instance of the redis from the redis connection pool, and performing regular matching according to the configured key prefix of the redis to query all matched key lists in the redis;
step S14: traversing the key list, acquiring data from the redis, and storing the acquired entity object data into a memory of the node;
step S15: the monitoring method for executing the change of the memory data of the nodes comprises the following steps that entity beans are reloaded when the attribute of a real object changes, database connection is created again when the attribute of database connection changes, and the like;
step S16: and after the initialization of the cache data is completed, each node stores a piece of data mirror image of redis in a memory after being started.
In step S12, the loaded configuration parameters include: a prefix of a key in a redis, a redis channel name, a redis connection address, a type of entity subscribed and published, and the like.
It should be noted that, in the foregoing system embodiment, each unit included is only divided according to functional logic, but is not limited to the above division as long as the corresponding function can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
In addition, it is understood by those skilled in the art that all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing associated hardware, and the corresponding program may be stored in a computer-readable storage medium.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims (2)

1. A data sharing method of a Redis-based distributed architecture is characterized by comprising the following steps:
step S1, initializing cache data: after each node of the distributed system is started, loading data in the redis into a memory of each node;
step S2, deploy subscription listening: defining a subscription callback method, subscribing a redis channel by each node, and deploying monitoring;
step S3, data synchronization: after the data change is monitored, updating the memory data of each node;
in step S2, the specific steps of deploying subscription listening are as follows:
step S21: defining a callback method, and automatically calling the callback method to process subscribed data when data is published;
step S22: when subscription data are published, adding the data into a built-in message queue, polling the message queue, and taking out the data in the message queue for processing when the data exist;
step S23: subscribing the configured redis channel by using a data subscription method of the redis, and monitoring on a defined callback method;
step S24: after the subscription monitoring is deployed, each node deploys the subscription of the redis channel and sets monitoring;
in step S3, the specific data synchronization method includes the following steps:
step S31: when data of an application system related to cache data in a node memory is added or deleted, the data is updated into redis;
step S32: the data are issued to the configured redis channel by using a data issuing method of the redis;
step S33: each node in step S23 will monitor the data issue, and call a defined callback method, and add the data to the built-in message queue in the callback method;
step S34: the built-in message queue monitors that data exist in the queue, takes out the data in the queue, acquires the latest data from the redis, and updates the corresponding memory data in the node;
step S35: executing a monitoring method for the change of the node memory data;
step S36: finishing data synchronization;
in step S1, the specific steps of initializing the cache data are as follows:
step S11: determining service data to be loaded into a memory according to system service requirements;
step S12: after the distributed nodes are started, firstly loading configuration parameters, and then initializing a connection pool of redis according to the configured redis addresses;
step S13: acquiring a connection instance of the redis from the redis connection pool, and performing regular matching according to the configured key prefix of the redis to query all matched key lists in the redis;
step S14: traversing the key list, acquiring data from the redis, and storing the acquired entity object data into a memory of the node;
step S15: executing a monitoring method for the change of the node memory data;
step S16: and after the initialization of the cache data is completed, each node stores a piece of data mirror image of redis in a memory after being started.
2. The Redis-based distributed architecture data sharing method according to claim 1, wherein in the step S12, the loaded configuration parameters include: the prefix of key in the redis, the name of the redis channel, the connecting address of the redis, and the entity types subscribed and published.
CN201910036828.5A 2019-01-15 2019-01-15 Redis-based distributed architecture data sharing method Active CN109800239B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910036828.5A CN109800239B (en) 2019-01-15 2019-01-15 Redis-based distributed architecture data sharing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910036828.5A CN109800239B (en) 2019-01-15 2019-01-15 Redis-based distributed architecture data sharing method

Publications (2)

Publication Number Publication Date
CN109800239A CN109800239A (en) 2019-05-24
CN109800239B true CN109800239B (en) 2022-09-06

Family

ID=66559372

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910036828.5A Active CN109800239B (en) 2019-01-15 2019-01-15 Redis-based distributed architecture data sharing method

Country Status (1)

Country Link
CN (1) CN109800239B (en)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111080500A (en) * 2019-11-26 2020-04-28 武汉烽火众智数字技术有限责任公司 Spark streaming based real-time rule deployment and control early warning method and device
CN110968439B (en) * 2019-11-28 2020-12-15 蜂助手股份有限公司 Inter-system message notification method, device, server, system and storage medium
CN111194012A (en) * 2019-12-30 2020-05-22 广东飞企互联科技股份有限公司 A development process and processing method of a redis-based timing timeout service
CN111787055B (en) * 2020-05-22 2021-07-27 中国科学院信息工程研究所 A Redis-based, transaction-oriented and multi-data center data distribution method and system
CN111866170B (en) * 2020-07-30 2023-08-08 小视科技(江苏)股份有限公司 Method for sending synchronization messages in IOT cluster
CN111984663B (en) * 2020-08-21 2024-03-12 西安寰宇卫星测控与数据应用有限公司 REDIS database update method, device, computer equipment, storage medium
CN111966756B (en) * 2020-08-21 2024-03-12 西安寰宇卫星测控与数据应用有限公司 Table data automatic synchronization method, device, computer equipment, storage medium
CN111988427A (en) * 2020-08-31 2020-11-24 湖南快乐阳光互动娱乐传媒有限公司 Distributed local cache processing method and system
CN112187896A (en) * 2020-09-18 2021-01-05 成都精灵云科技有限公司 Cache system and method supporting real-time notification of changes and real-time synchronization of data
CN112243030B (en) * 2020-10-14 2025-02-07 深圳前海微众银行股份有限公司 Data synchronization method, device, equipment and medium of distributed storage system
CN112235405A (en) * 2020-10-14 2021-01-15 深圳前海微众银行股份有限公司 Distributed storage system and data delivery method
CN112416955B (en) * 2020-11-18 2022-02-22 四川长虹电器股份有限公司 Cache consistency problem solving method based on high concurrency environment
CN112286950A (en) * 2020-11-20 2021-01-29 四川长虹电器股份有限公司 Method for keeping consistency of valid data in distributed cache
CN112597197B (en) * 2020-12-17 2022-07-26 杭州当虹科技股份有限公司 Distributed cache system based on memory
CN114116200B (en) 2021-10-27 2024-04-05 北京百度网讯科技有限公司 Resource sharing method, service creation method, device and electronic equipment
CN114500655B (en) * 2022-04-15 2022-07-26 武汉中科通达高新技术股份有限公司 Interface request processing method and device, micro-service framework cluster system and medium
CN115242671A (en) * 2022-07-05 2022-10-25 中国电子科技集团公司第十四研究所 Message monitoring method based on redis
CN116467384A (en) * 2023-03-31 2023-07-21 河南合众伟奇云智科技有限公司 Real-time data synchronization method and system based on redis database

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101373509A (en) * 2008-10-10 2009-02-25 华南理工大学 A Kind of RFID Middleware Based on Distributed ALE
CN102880475A (en) * 2012-10-23 2013-01-16 上海普元信息技术股份有限公司 Real-time event handling system and method based on cloud computing in computer software system
CN105677916A (en) * 2016-03-02 2016-06-15 浪潮软件集团有限公司 A method used by Redis
CN108418894A (en) * 2018-03-26 2018-08-17 威创软件南京有限公司 A kind of Distributed Data Synchronization method based on push technologies

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10417254B2 (en) * 2016-02-01 2019-09-17 Vmware, Inc. Intelligent content synchronization between content libraries
US10082983B2 (en) * 2016-04-05 2018-09-25 International Business Machines Corporation Monitoring storage systems

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101373509A (en) * 2008-10-10 2009-02-25 华南理工大学 A Kind of RFID Middleware Based on Distributed ALE
CN102880475A (en) * 2012-10-23 2013-01-16 上海普元信息技术股份有限公司 Real-time event handling system and method based on cloud computing in computer software system
CN105677916A (en) * 2016-03-02 2016-06-15 浪潮软件集团有限公司 A method used by Redis
CN108418894A (en) * 2018-03-26 2018-08-17 威创软件南京有限公司 A kind of Distributed Data Synchronization method based on push technologies

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
A Distributed Redis Framework for Use in the UCWW;Zhanlin Ji etc.;《2014 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery》;20141215;第241-244页 *
Redis可扩展与高效复制方案的设计与实现;李法平 等;《西南师范大学学报(自然科学版)》;20180331;第83-89页 *
Towards Scalable and Reliable In-Memory Storage System: A Case Study with Redis;Shanshan Chen etc.;《 2016 IEEE Trustcom/BigDataSE/ISPA》;20170209;第1660-1666页 *
基于MySQL Binlog的数据增量同步系统的设计与实现;苏子权;《中国优秀硕士学位论文全文数据库(信息科技辑)》;20181015;第I138-203页 *

Also Published As

Publication number Publication date
CN109800239A (en) 2019-05-24

Similar Documents

Publication Publication Date Title
CN109800239B (en) Redis-based distributed architecture data sharing method
CN109815248B (en) A Zookeeper-based Distributed Architecture Data Consistency Method
CN106878376B (en) Configuration management method and system
US9239767B2 (en) Selective database replication
US10979286B2 (en) Method, device and computer program product for managing distributed system
US8676914B2 (en) Synchronizing services across network nodes
US9367261B2 (en) Computer system, data management method and data management program
CN108255592B (en) Quartz cluster timing task processing system and method
CN110391940B (en) Service address response method, device, system, equipment and storage medium
US11228664B2 (en) Method and apparatus for acquiring RPC member information, electronic device and storage medium
CN105162879B (en) Method, device and system for realizing data consistency in multiple computer rooms
CN107483250A (en) Distributed configuration management method, device and system for realizing distributed configuration management
CN107404509B (en) Distributed service configuration system and information management method
CN101227265B (en) Method for data synchronization among client ends in equipment monitoring system
CN113064950B (en) Data synchronization method, device, equipment and storage medium
CN103973725A (en) Distributed collaboration method and collaboration device
CN101369923B (en) A Method of Using Distributed Hash Table to Improve the Performance of Clustered Web Services
WO2022134797A1 (en) Data fragmentation storage method and apparatus, a computer device, and a storage medium
CN112153133A (en) Data sharing method, device and medium
CN108566291A (en) A kind of method of event handling, server and system
CN113900728B (en) Synchronous configuration method, system, electronic equipment and storage medium
US9602615B2 (en) Masterless cache replication
CN115189931A (en) Distributed key management method, device, equipment and storage medium
CN117834421B (en) RocketMQ-based cluster configuration management method and RocketMQ-based cluster configuration management system
CN119946074A (en) A distributed cluster and state synchronization method based on Raft consensus algorithm

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
EE01 Entry into force of recordation of patent licensing contract

Application publication date: 20190524

Assignee: Kedaduochuang cloud Technology Co.,Ltd.

Assignor: USTC SINOVATE SOFTWARE CO.,LTD.

Contract record no.: X2023980034512

Denomination of invention: Redis based distributed architecture data sharing method

Granted publication date: 20220906

License type: Common License

Record date: 20230407

EE01 Entry into force of recordation of patent licensing contract