CN110083579A - Incremental data synchronous method, apparatus, computer equipment and computer storage medium - Google Patents
Incremental data synchronous method, apparatus, computer equipment and computer storage medium Download PDFInfo
- Publication number
- CN110083579A CN110083579A CN201910218794.1A CN201910218794A CN110083579A CN 110083579 A CN110083579 A CN 110083579A CN 201910218794 A CN201910218794 A CN 201910218794A CN 110083579 A CN110083579 A CN 110083579A
- Authority
- CN
- China
- Prior art keywords
- incremental data
- synchronization
- synchronized
- incremental
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请公开了一种增量数据同步的方法、装置、计算机设备及计算机存储介质，涉及云存储技术领域，可以将合适的增量数据同步方式推送给需要同步的数据表，提高增量数据的同步效率。所述方法包括：获取适用于增量数据同步方式的模式字符；根据所述模式字符确定适用于增量数据同步方式的预设匹配规则，构造多个增量数据同步方式的正则表达式；当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串；若存在，则根据所述与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对所述待同步增量数据表进行增量数据同步。
This application discloses a method, device, computer equipment, and computer storage medium for incremental data synchronization, which relate to the technical field of cloud storage, and can push an appropriate incremental data synchronization method to a data table that needs to be synchronized to improve the accuracy of incremental data. synchronization efficiency. The method includes: obtaining a pattern character suitable for an incremental data synchronization method; determining a preset matching rule suitable for an incremental data synchronization method according to the pattern character, and constructing a plurality of regular expressions for an incremental data synchronization method; When an incremental data synchronization request is received, match the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode, and find out whether there is a data corresponding to the data in the data table to be synchronized matching character string; if it exists, perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method with the largest number of strings matching the incremental data in the data table to be synchronized.
Description
技术领域technical field
本发明涉及云存储技术领域，尤其是涉及增量数据同步的方法、装置、计算机设备及计算机存储介质。The present invention relates to the technical field of cloud storage, in particular to a method, a device, a computer device and a computer storage medium for synchronizing incremental data.
背景技术Background technique
计算机网络的飞速发展和信息化的推进使得数据成为很多企业的宝贵资源，使得数据采集、存储和处理任务与日俱增，基于网络的数据共享已经成为人们获取数据资源不可或缺的方式。The rapid development of computer networks and the advancement of informatization have made data a valuable resource for many companies, and the tasks of data collection, storage and processing are increasing day by day. Network-based data sharing has become an indispensable way for people to obtain data resources.
为了更好地实现数据共享需要对共享数据进行同步，数据同步对企业来说至关重要，比如企业的服务器端用于管理分属于企业不同部门的业务端，并且在服务器端存储有管理数据，那么为了方便服务器端对各个业务端进行管理，同时提高业务端的数据处理效率，需要保证服务器端与各个业务端之间的数据同步。In order to better realize data sharing, it is necessary to synchronize the shared data. Data synchronization is very important for enterprises. For example, the server side of the enterprise is used to manage the business ends belonging to different departments of the enterprise, and the management data is stored on the server side. Then, in order to facilitate the management of each business end on the server side and improve the data processing efficiency of the business end, it is necessary to ensure the data synchronization between the server end and each business end.
通常情况下，根据数据写入后是否会发生变化将需要同步的数据分为两类数据，一类是不会发生变化的数据，一般是日志数据等；另一类是会变化的数据，一般是人员表等。针对会变化的数据，随着时间维度的改版，业务端会不断新增记录的数据，该新增记录的数据称为增量数据。用户可以根据实际需求选择不同的增量数据同步方式，例如，全表删除的增量数据同步方式、时间戳动态更新的增量数据同步方式以及逐条对比的增量数据同步方式。然而，不同的增量数据同步方式需要对数据表的结构、大小进行手动判断，需要耗费大量人工操作时间来选择合适的增量数据同步方式，并且在人工操作过程中容易产生错误，将不合适的增量数据同步方式推送给需要同步的数据表，导致增量数据同步的效率降低。Under normal circumstances, the data that needs to be synchronized is divided into two types of data according to whether the data will change after writing. One type is data that will not change, usually log data, etc.; the other type is data that will change, generally Is the personnel table and so on. For data that will change, with the revision of the time dimension, the business end will continuously add recorded data, and the newly recorded data is called incremental data. Users can choose different incremental data synchronization methods according to actual needs, for example, the incremental data synchronization method of full table deletion, the incremental data synchronization method of timestamp dynamic update, and the incremental data synchronization method of item-by-item comparison. However, different incremental data synchronization methods require manual judgment on the structure and size of the data table, and it takes a lot of manual operation time to select the appropriate incremental data synchronization method, and errors are prone to occur during manual operation, so it will not be suitable The incremental data synchronization method is pushed to the data tables that need to be synchronized, resulting in a decrease in the efficiency of incremental data synchronization.
发明内容SUMMARY OF THE INVENTION
有鉴于此，本发明提供了一种增量数据同步的方法、装置、计算机设备及计算机存储介质，主要目的在于解决目前相关技术中增量数据同步效率低下的问题。In view of this, the present invention provides a method, device, computer equipment and computer storage medium for incremental data synchronization, the main purpose of which is to solve the problem of low efficiency of incremental data synchronization in the related art.
依据本发明一个方面，提供了一种增量数据同步的方法，该方法包括：According to one aspect of the present invention, a method for incremental data synchronization is provided, the method comprising:
获取适用于增量数据同步方式的模式字符；Obtain the mode character applicable to the incremental data synchronization method;
根据所述模式字符确定适用于增量数据同步方式的预设匹配规则，构造多个增量数据同步方式的正则表达式；Determining preset matching rules applicable to incremental data synchronization methods according to the pattern characters, and constructing regular expressions for multiple incremental data synchronization methods;
当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串；When an incremental data synchronization request is received, the data in the incremental data table to be synchronized is matched with the regular expressions of each incremental synchronization mode, and whether there is data in the data table to be synchronized the matching string;
若存在，则根据所述与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对所述待同步增量数据表进行增量数据同步。If it exists, perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method with the largest number of character strings matching the incremental data in the data table to be synchronized.
进一步地，所述获取适用于增量数据同步方式的模式字符包括：Further, the acquisition of mode characters suitable for incremental data synchronization includes:
通过解析适用于增量数据同步方式的数据表定义的字段，获取适用于增量数据同步方式的字段；Obtain the fields applicable to the incremental data synchronization method by parsing the fields defined in the data table applicable to the incremental data synchronization method;
从所述适用于增量数据同步方式的字段中选取预设字段作为适用于增量数据同步方式的模式字符。A preset field is selected from the fields applicable to the incremental data synchronization method as the pattern character applicable to the incremental data synchronization method.
进一步地，所述模式字符中包含适用于增量数据同步方式的特征信息，所述根据所述模式字符确定适用于增量数据同步方式的预设匹配规则，构造多个增量数据同步方式的正则表达式包括：Further, the pattern character contains feature information suitable for incremental data synchronization, and the preset matching rules suitable for incremental data synchronization are determined according to the pattern character, and multiple incremental data synchronization methods are constructed. Regular expressions include:
根据所述模式字符中适用于增量数据同步方式的特征信息，确定所述模式字符在增量数据表中的匹配规则；According to the feature information applicable to the incremental data synchronization mode in the pattern character, determine the matching rule of the pattern character in the incremental data table;
根据所述模式字符在增量数据表中的匹配规则，构造多个增量数据同步方式的正则表达式。According to the matching rules of the pattern characters in the incremental data table, multiple regular expressions of incremental data synchronization modes are constructed.
进一步地，所述根据所述模式字符中适用于增量数据同步方式的特征信息，确定所述模式字符在增量数据表中的匹配规则包括：Further, according to the feature information in the pattern character applicable to the incremental data synchronization mode, determining the matching rule of the pattern character in the incremental data table includes:
如果所述模式字符中适用于增量数据同步方式的特征信息为位置信息，以所述模式字符在增量数据表中的字段位置作为匹配规则；If the feature information applicable to the incremental data synchronization method in the pattern character is position information, the field position of the pattern character in the incremental data table is used as the matching rule;
如果所述模式字符中适用于增量数据同步方式的特征信息为时间信息，以所述模式字符在增量数据表中的时间戳作为匹配规则；If the feature information applicable to the incremental data synchronization method in the pattern character is time information, the time stamp of the pattern character in the incremental data table is used as the matching rule;
如果所述模式字符中适用于增量数据同步方式的特征信息为频率信息，以所述模式字符在增量数据表中的出现次数作为匹配规则。If the feature information applicable to the incremental data synchronization method in the pattern character is frequency information, the number of occurrences of the pattern character in the incremental data table is used as the matching rule.
进一步地，所述根据所述模式字符在增量数据表中的匹配规则，构造多个增量数据同步方式的正则表达式包括：Further, according to the matching rules of the pattern characters in the incremental data table, the regular expressions for constructing multiple incremental data synchronization methods include:
根据所述模式字符在增量数据表中的匹配规则，从字符表中筛选出符合增量数据同步方式的普通字符串以及特殊字符串，所述字符表包括用于组合各种匹配规则的普通字符串以及特殊字符串；According to the matching rules of the pattern characters in the incremental data table, the ordinary character strings and special character strings that meet the incremental data synchronization method are screened out from the character table, and the character table includes ordinary character strings used to combine various matching rules strings and special strings;
将所述符合增量数据同步方式的普通字符串以及特殊字符串导入至表达式模板中，构造多个增量数据同步方式的正则表达式。Import the normal character strings and special character strings conforming to the incremental data synchronization method into the expression template to construct multiple regular expressions of the incremental data synchronization method.
进一步地，在所述当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串之后，所述方法还包括：Further, when the incremental data synchronization request is received, the data in the incremental data table to be synchronized is respectively matched with the regular expressions of each incremental synchronization mode, and it is searched whether there is an After synchronizing the data in the data table to match the string, the method further includes:
若不存在，则根据待同步增量数据表对应的数据量大小确定适用于待同步增量数据表的增量数据同步方式；If it does not exist, then determine the incremental data synchronization method suitable for the incremental data table to be synchronized according to the data volume corresponding to the incremental data table to be synchronized;
根据所述适用于待同步增量数据表的增量数据同步方式对所述待同步增量数据表进行增量数据同步。Perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method applicable to the incremental data table to be synchronized.
进一步地，所述根据待同步增量数据表对应的数据量大小确定增量数据同步方式包括：Further, the determining the incremental data synchronization method according to the data volume corresponding to the incremental data table to be synchronized includes:
统计所述待同步增量数据表中的数据条数；Count the number of data items in the incremental data table to be synchronized;
如果所述待同步增量数据表中的数据条数小于或等于预设数值，则采用全表删除的增量数据同步方式；If the number of data items in the incremental data table to be synchronized is less than or equal to a preset value, the incremental data synchronization method of deleting the entire table is adopted;
如果所述待同步增量数据表中的数据条数大于预设数值，则采用逐条比对的增量数据同步方式。If the number of data items in the incremental data table to be synchronized is greater than a preset value, an incremental data synchronization method of comparing item by item is adopted.
依据本发明另一个方面，提供了一种增量数据同步的装置，所述装置包括：According to another aspect of the present invention, an incremental data synchronization device is provided, the device comprising:
获取单元，用于获取适用于增量数据同步方式的模式字符；An acquisition unit, configured to acquire a pattern character suitable for incremental data synchronization;
构造单元，用于根据所述模式字符确定适用于增量数据同步方式的预设匹配规则，构造多个增量数据同步方式的正则表达式；A construction unit, configured to determine preset matching rules applicable to incremental data synchronization methods according to the pattern characters, and construct multiple regular expressions for incremental data synchronization methods;
查找单元，用于当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串；A search unit, configured to match the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode when receiving the incremental data synchronization request, and find out whether there is a The string to match the data in the data table;
第一同步单元，用于若存在，则根据所述与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对所述待同步增量数据表进行增量数据同步。The first synchronization unit is configured to, if it exists, perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method with the largest number of strings matching the incremental data in the data table to be synchronized .
进一步地，所述获取单元包括：Further, the acquisition unit includes:
获取模块，用于通过解析适用于增量数据同步方式的数据表定义的字段，获取适用于增量数据同步方式的字段；An acquisition module, configured to acquire fields applicable to incremental data synchronization by parsing fields defined in data tables applicable to incremental data synchronization;
选取模块，用于从所述适用于增量数据同步方式的字段中选取预设字段作为适用于增量数据同步方式的模式字符。A selection module, configured to select a preset field from the fields applicable to the incremental data synchronization method as a pattern character applicable to the incremental data synchronization method.
进一步地，所述模式字符中包含适用于增量数据同步方式的特征信息，所述构造单元包括：Further, the pattern character contains characteristic information suitable for incremental data synchronization, and the construction unit includes:
第一确定模块，用于根据所述模式字符中适用于增量数据同步方式的特征信息，确定所述模式字符在增量数据表中的匹配规则；The first determination module is used to determine the matching rules of the pattern characters in the incremental data table according to the feature information in the pattern characters applicable to the incremental data synchronization mode;
构造模块，用于根据所述模式字符在增量数据表中的匹配规则，构造多个增量数据同步方式的正则表达式。A construction module, configured to construct multiple regular expressions of incremental data synchronization methods according to the matching rules of the pattern characters in the incremental data table.
进一步地，所述第一确定模块，具体用于如果所述模式字符中适用于增量数据同步方式的特征信息为位置信息，以所述模式字符在增量数据表中的字段位置作为匹配规则；Further, the first determination module is specifically configured to use the field position of the pattern character in the incremental data table as the matching rule if the feature information applicable to the incremental data synchronization method in the pattern character is position information ;
所述第一确定模块，具体还用于如果所述模式字符中适用于增量数据同步方式的特征信息为时间信息，以所述模式字符在增量数据表中的时间戳作为匹配规则；The first determination module is specifically further configured to use the time stamp of the pattern character in the incremental data table as the matching rule if the feature information applicable to the incremental data synchronization method in the pattern character is time information;
所述第一确定模块，具体还用于如果所述模式字符中适用于增量数据同步方式的特征信息为频率信息，以所述模式字符在增量数据表中的出现次数作为匹配规则。The first determination module is specifically further configured to use the number of occurrences of the pattern character in the incremental data table as the matching rule if the feature information applicable to the incremental data synchronization method in the pattern character is frequency information.
进一步地，所述构造模块，具体用于根据所述模式字符在增量数据表中的匹配规则，从字符表中筛选出符合增量数据同步方式的普通字符串以及特殊字符串，所述字符表包括用于组合各种匹配规则的普通字符串以及特殊字符串；Further, the construction module is specifically used to filter out ordinary character strings and special character strings conforming to the incremental data synchronization method from the character table according to the matching rules of the pattern characters in the incremental data table, and the characters The table includes ordinary strings and special strings for combining various matching rules;
所述构造模块，具体还用于将所述符合增量数据同步方式的普通字符串以及特殊字符串导入至表达式模板中，构造多个增量数据同步方式的正则表达式。The construction module is specifically further configured to import the normal character strings and special character strings conforming to the incremental data synchronization method into the expression template, and construct multiple regular expressions of the incremental data synchronization method.
进一步地，所述装置还包括：Further, the device also includes:
确定单元，用于在所述当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串之后，若不存在，则根据待同步增量数据表对应的数据量大小确定适用于待同步增量数据表的增量数据同步方式；The determination unit is configured to match the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode when the incremental data synchronization request is received, and find out whether there is a After describing the character string that matches the data in the data table to be synchronized, if it does not exist, then determine the incremental data synchronization method applicable to the incremental data table to be synchronized according to the corresponding data size of the incremental data table to be synchronized;
第二同步单元，用于根据所述适用于待同步增量数据表的增量数据同步方式对所述待同步增量数据表进行增量数据同步。The second synchronization unit is configured to perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method applicable to the incremental data table to be synchronized.
进一步地，所述确定单元包括：Further, the determining unit includes:
统计模块，用于统计所述待同步增量数据表中的数据条数；A statistical module, configured to count the number of data items in the incremental data table to be synchronized;
第二确定模块，用于如果所述待同步增量数据表中的数据条数小于或等于预设数值，则确定采用全表删除的增量数据同步方式；The second determination module is used to determine that the incremental data synchronization method of deleting the entire table is adopted if the number of data items in the incremental data table to be synchronized is less than or equal to a preset value;
所述第二确定模块，还用于如果所述待同步增量数据表中的数据条数大于预设数值，则确定采用逐条比对的增量数据同步方式。The second determination module is further configured to determine to adopt an incremental data synchronization method of comparing items one by one if the number of data items in the incremental data table to be synchronized is greater than a preset value.
依据本发明又一个方面，提供了一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现增量数据同步的方法的步骤。According to yet another aspect of the present invention, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the incremental data synchronization method when executing the computer program.
依据本发明再一个方面，提供了一种计算机存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现增量数据同步的方法的步骤。According to still another aspect of the present invention, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for synchronizing incremental data are realized.
借由上述技术方案，本发明提供一种增量数据同步的方法及装置，当接收到增量数据同步请求时，通过多个增量数据同步方式的正则表达式来匹配待同步增量数据，查找是否含存在与待同步数据表中增量数据相匹配的字符串，若存在，则说明待同步数据表中的数据适用于多个增量数据同步方式中的一种，进一步根据与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对待同步增量数据表进行增量数据同步，以提高增量数据同步的效率。与现有技术中通过人工操作来选择合适的增量数据同步方式相比，本发明实施例通过构造适用于不同增量数据同步方式的正则表达式来匹配待同步增量数据表中的字段，可以针对增量数据表的特性选择适合增量数据同步方式完成增量数据同步任务，由于合适的增量数据同步方式可以优化增量数据同步效果，以及增量数据同步的准确性，通过合适的增量数据同步方式进行增量数据同步，节约系统资源，减少人为操作，以及提高增量数据同步的资源利用率。By means of the above technical solution, the present invention provides a method and device for incremental data synchronization. When an incremental data synchronization request is received, the incremental data to be synchronized is matched through regular expressions of multiple incremental data synchronization modes. Find whether there is a character string that matches the incremental data in the data table to be synchronized. If it exists, it means that the data in the data table to be synchronized is suitable for one of the multiple incremental data synchronization methods. Further according to the The incremental data synchronization method with the largest number of incremental data matching character strings in the data table to be synchronized performs incremental data synchronization on the incremental data table to be synchronized, so as to improve the efficiency of incremental data synchronization. Compared with selecting an appropriate incremental data synchronization method through manual operation in the prior art, the embodiment of the present invention matches the fields in the incremental data table to be synchronized by constructing a regular expression suitable for different incremental data synchronization methods, According to the characteristics of the incremental data table, the incremental data synchronization method can be selected to complete the incremental data synchronization task. Because the appropriate incremental data synchronization method can optimize the incremental data synchronization effect and the accuracy of the incremental data synchronization, through the appropriate The incremental data synchronization method performs incremental data synchronization, saves system resources, reduces manual operations, and improves resource utilization of incremental data synchronization.
附图说明Description of drawings
通过阅读下文优选实施方式的详细描述，各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的，而并不认为是对本发明的限制。而且在整个附图中，用相同的参考符号表示相同的部件。在附图中：Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached image:
图1示出了本发明实施例提供的一种增量数据同步的方法流程示意图；FIG. 1 shows a schematic flowchart of a method for incremental data synchronization provided by an embodiment of the present invention;
图2示出了本发明实施例提供的另一种增量数据同步的方法流程示意图；FIG. 2 shows a schematic flowchart of another incremental data synchronization method provided by an embodiment of the present invention;
图3示出了本发明实施例提供的一种增量数据同步的装置的结构示意图；FIG. 3 shows a schematic structural diagram of an incremental data synchronization device provided by an embodiment of the present invention;
图4示出了本发明实施例提供的另一种增量数据同步的装置的结构示意图。FIG. 4 shows a schematic structural diagram of another incremental data synchronization device provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例，然而应当理解，可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反，提供这些实施例是为了能够更透彻地理解本公开，并且能够将本公开的范围完整的传达给本领域的技术人员。Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
本发明实施例提供了一种增量数据同步的方法，可以实现提高增量数据同步效率的目的，如图1所示，该方法包括：The embodiment of the present invention provides a method for incremental data synchronization, which can achieve the purpose of improving the efficiency of incremental data synchronization. As shown in Figure 1, the method includes:
101、获取适用于增量数据同步方式的模式字符。101. Acquire a mode character applicable to an incremental data synchronization mode.
其中，增量数据同步方式可以为时间戳动态更新的增量数据同步方式、数据递增的增量数据同步方式，在增量数据同步的过程中，不同增量数据同步方式有固有的同步特点，针对某一类型的数据表具有较好的同步效果，而对于其他类型的数据表，可能同步效果并不是很好，这里的模式字符相当于针对增量数据同步方式配置的匹配字符串，该匹配字符串可以为表名或者表字段等，这里不进行限定。Among them, the incremental data synchronization method can be the incremental data synchronization method of dynamically updating the timestamp, or the incremental data synchronization method of incremental data. In the process of incremental data synchronization, different incremental data synchronization methods have inherent synchronization characteristics. A certain type of data table has a good synchronization effect, but for other types of data tables, the synchronization effect may not be very good. The pattern character here is equivalent to the matching string configured for the incremental data synchronization method. The matching The string can be a table name or a table field, etc., which is not limited here.
上述的时间戳动态更新的增量数据同步方式为通过查找待同步数据表中的特定时间戳，并对特定时间戳之后的增量数据进行同步，并同步至数据中心数据表相应的时间戳位置，通常针对记录时间戳的数据表，所以适用于时间戳动态更新的增量数据同步方式的模式字符可以为数据表的时间字段。上述的数据递增的增量数据同步方式为通过查找待同步数据中增量数据差值，并对增量数据差值进行同步，并同步至数据中心数据表的末尾，通常针对逐条写入的日志数据表，所以适用于数据递增的增量数据同步方式的模式字符可以为数据表的表名字段。The incremental data synchronization method of the above-mentioned timestamp dynamic update is to find the specific timestamp in the data table to be synchronized, and synchronize the incremental data after the specific timestamp, and synchronize to the corresponding timestamp position of the data center data table , usually for the data table that records the timestamp, so the mode character applicable to the incremental data synchronization method for the dynamic update of the timestamp can be the time field of the data table. The incremental data synchronization method of the above-mentioned data increment is to find the incremental data difference in the data to be synchronized, and synchronize the incremental data difference, and synchronize to the end of the data center data table, usually for the log written one by one Data table, so the pattern character applicable to the incremental data synchronization method of data increment can be the table name field of the data table.
由于不同增量数据表的结构不同，不同结构使得增量数据表适用于不同增量数据同步方式，例如，对于待同步增量数据表在写入数据时表字段中记录有时间戳，说明可以根据时间戳来判断待同步增量数据表中的增量数据，该待同步增量数据表更适用于时间戳动态更新的增量数据同步方式，而对于待同步增量数据表在写入数据时是逐条写入到表中，说明增量数据会写入到表末尾，可以通过比对数据表中数据量差值来判断待同步增量数据表中的增量数据，该待同步增量数据表更适用于数据递增的增量数据同步方式。Due to the different structures of different incremental data tables, different structures make the incremental data tables suitable for different incremental data synchronization methods. For example, for the incremental data table to be synchronized, when the data is written, the time stamp is recorded in the table field, indicating that it can be Judging the incremental data in the incremental data table to be synchronized according to the timestamp, the incremental data table to be synchronized is more suitable for the incremental data synchronization method of dynamic update of the timestamp, and the incremental data table to be synchronized is writing data It is written into the table one by one, indicating that the incremental data will be written to the end of the table. You can judge the incremental data in the incremental data table to be synchronized by comparing the data amount difference in the data table. The incremental data to be synchronized The data table is more suitable for the incremental data synchronization method of incremental data.
102、根据所述模式字符确定适用于增量数据同步方式的预设匹配规则，构造多个增量数据同步方式的正则表达式。102. Determine a preset matching rule applicable to the incremental data synchronization method according to the pattern characters, and construct multiple regular expressions for the incremental data synchronization method.
对于本发明实施例，模式字符中包含有适用于增量数据同步方式的数据表的表格特征，而该表格特征可能为匹配字段在表格中的时间信息，可能为匹配字段在表格中出现的频率，可能为匹配字段在表格中的位置等，不同的表格特征需要赋予不同的匹配规则，进一步根据模式字符确定每一种增量数据同步方式的匹配规则，例如，如果模式字符包含的表格特征为匹配字段在表格中的时间信息，则确定增量数据同步方式的匹配规则为涉及数据写入时间戳的匹配规则，如果模式字符包含的表格特征为匹配字段在表格中的位置信息，则确定增量数据同步方式的匹配规则为涉及数据写入位置的匹配规则，这里不进行限定。For the embodiment of the present invention, the pattern character contains table features suitable for incremental data synchronization, and the table feature may be the time information of the matching field in the table, and may be the frequency of the matching field appearing in the table , may be the position of the matching field in the table, etc. Different table features need to be given different matching rules, and further determine the matching rules for each incremental data synchronization method according to the pattern characters. For example, if the table features contained in the pattern characters are The time information of the matching field in the table, the matching rule for determining the incremental data synchronization method is the matching rule involving the data writing timestamp, if the table feature contained in the pattern character is the position information of the matching field in the table, then determine the incremental data synchronization method. The matching rule of the volume data synchronization method is a matching rule related to the location where the data is written, which is not limited here.
对于本发明实施例，正则表达式描述了一种字符串匹配的模式，用来定义适用于增量数据同步方式的预设匹配规则，通过正则表达式能够检查待同步数据表中是否存在符合适用于增量数据同步方式的预设匹配规则的字符串，从而为待同步数据表确定合适的增量数据同步方式。For the embodiment of the present invention, the regular expression describes a pattern of character string matching, which is used to define the preset matching rules applicable to the incremental data synchronization method. Through the regular expression, it can be checked whether there is a matching rule in the data table to be synchronized. The character string of the preset matching rule based on the incremental data synchronization method, so as to determine the appropriate incremental data synchronization method for the data table to be synchronized.
103、当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串。103. When an incremental data synchronization request is received, match the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode, and find out whether the data in the incremental data table to be synchronized exists The string that matches the data.
对于本发明实施例，对于随着时间变化的数据表，业务方在将数据表同步至数据中心后，会间隔一段时间启动增量数据同步任务，并发送增量数据同步请求，请求将数据表中变化的数据同步至数据中心，这里数据表中变化的数据就是增量数据，对应存储在待同步增量数据表中。For the embodiment of the present invention, for a data table that changes over time, after the business party synchronizes the data table to the data center, it will start an incremental data synchronization task at intervals, and send an incremental data synchronization request, requesting that the data table The changed data is synchronized to the data center, where the changed data in the data table is incremental data, which is correspondingly stored in the incremental data table to be synchronized.
其中，待同步增量数据表中的增量数据可以为服务器自动生成的增量数据，客户端输入的用户信息增量数据等，还可以为平台中记录用户行为的日志增数据，例如，用户浏览网页的数据，用户购买商品的数据等，这里不进行限定。Among them, the incremental data in the incremental data table to be synchronized can be the incremental data automatically generated by the server, the incremental data of user information input by the client, etc., and can also add data to the log that records user behavior in the platform, for example, user The data of browsing webpages, the data of products purchased by users, etc., are not limited here.
对于本发明实施例，当接收到增量数据同步请求时，由于待同步增量数据表中的增量数据并不适用于所有增量数据同步方式，如果选择不合适的增量数据同步方式可能会降低增量数据的同步效率，这里通过将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与待同步数据表中增量数据相匹配的字符串，从而针对待同步增量数据表中增量数据的特点为增量数据选择合适的增量数据同步方式，从而提高增量数据的同步效率。For the embodiment of the present invention, when the incremental data synchronization request is received, since the incremental data in the incremental data table to be synchronized is not applicable to all incremental data synchronization methods, if an inappropriate incremental data synchronization method is selected, it may It will reduce the synchronization efficiency of incremental data. Here, by matching the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode, find out whether there is incremental data in the data table to be synchronized Matching character strings, so as to select an appropriate incremental data synchronization method for the incremental data according to the characteristics of the incremental data in the incremental data table to be synchronized, thereby improving the synchronization efficiency of the incremental data.
104、若存在，则根据所述与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对所述待同步增量数据表进行增量数据同步。104. If yes, perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method with the largest number of character strings matching the incremental data in the data table to be synchronized.
对于本发明实施例，若存在与待同步增量数据表中增量数据相匹配的字符串，则说明多个增量数据同步方式中存在适合待同步数据表中增量数据的同步方式，并且与待同步数据表中的数据匹配字符串数量越多，说明待同步数据表中的数据越适用于该增量数据同步方式，使用该增量数据同步方式能够节省同步时间，进一步选择与待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式进行增量数据同步。For the embodiment of the present invention, if there is a string matching the incremental data in the incremental data table to be synchronized, it means that there is a synchronization method suitable for the incremental data in the incremental data table to be synchronized among the multiple incremental data synchronization methods, and The more strings that match the data in the data table to be synchronized, the more suitable the data in the data table to be synchronized is for this incremental data synchronization method. Using this incremental data synchronization method can save synchronization time and further select the Incremental data synchronization is performed in the incremental data synchronization method with the largest number of incremental data matching character strings in the data table.
示例性的，如果待同步增量数据表中的表名字段以及列字段中都存在与数据递增的增量数据同步方式相匹配的字符串，那么说明待同步数据表适合数据递增的增量数据同步方式，进一步判断待同步数据表与已同步至数据中心的数据表进行比对，将比对差异的数据量直接同步至数据中心的数据表的末尾，完成增量数据同步任务。Exemplarily, if the table name field and the column field in the incremental data table to be synchronized have strings that match the incremental data synchronization method of data increment, then it means that the data table to be synchronized is suitable for the incremental data of data increment The synchronization method further judges the comparison between the data table to be synchronized and the data table that has been synchronized to the data center, and directly synchronizes the data amount of the comparison difference to the end of the data table in the data center to complete the incremental data synchronization task.
本发明提供一种增量数据同步的方法，当接收到增量数据同步请求时，通过多个增量数据同步方式的正则表达式来匹配待同步增量数据，查找是否含存在与待同步数据表中增量数据相匹配的字符串，若存在，则说明待同步数据表中的数据适用于多个增量数据同步方式中的一种，进一步根据与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对待同步增量数据表进行增量数据同步，以提高增量数据同步的效率。与现有技术中通过人工操作来选择合适的增量数据同步方式相比，本发明实施例通过构造适用于不同增量数据同步方式的正则表达式来匹配待同步增量数据表中的字段，可以针对增量数据表的特性选择适合增量数据同步方式完成增量数据同步任务，由于合适的增量数据同步方式可以优化增量数据同步效果，以及增量数据同步的准确性，通过合适的增量数据同步方式进行增量数据同步，节约系统资源，减少人为操作，以及提高增量数据同步的资源利用率。The present invention provides a method for synchronizing incremental data. When an incremental data synchronization request is received, the incremental data to be synchronized is matched through regular expressions of multiple incremental data synchronization modes, and whether the data to be synchronized exists or not is searched. If there is a character string matching the incremental data in the table, it means that the data in the data table to be synchronized is suitable for one of multiple incremental data synchronization methods, and further according to the incremental data in the data table to be synchronized The incremental data synchronization method with the largest number of matching character strings performs incremental data synchronization on the incremental data table to be synchronized to improve the efficiency of incremental data synchronization. Compared with selecting an appropriate incremental data synchronization method through manual operation in the prior art, the embodiment of the present invention matches the fields in the incremental data table to be synchronized by constructing a regular expression suitable for different incremental data synchronization methods, According to the characteristics of the incremental data table, the incremental data synchronization method can be selected to complete the incremental data synchronization task. Because the appropriate incremental data synchronization method can optimize the incremental data synchronization effect and the accuracy of the incremental data synchronization, through the appropriate The incremental data synchronization method performs incremental data synchronization, saves system resources, reduces manual operations, and improves resource utilization of incremental data synchronization.
本发明实施例提供了另一种增量数据同步的方法，可以实现提高增量数据同步效率的目的，如图2所示，所述方法包括：The embodiment of the present invention provides another method for incremental data synchronization, which can achieve the purpose of improving the efficiency of incremental data synchronization. As shown in FIG. 2, the method includes:
201、通过解析适用于增量数据同步方式的数据表定义的字段，获取适用于增量数据同步方式的字段。201. Obtain the fields applicable to the incremental data synchronization manner by parsing the fields defined in the data table applicable to the incremental data synchronization manner.
由于不同类型数据表具有不同的结构，可能导致增量数据具有不同的特点，这里适用于增量数据同步方式的数据表定义的字段可以为待同步数据表中的表名或者行字段等，例如，有的数据表的表名称中包含日志，说明该数据表中的数据是逐条写入的，所以增量数据也是逐条写入的，适用于增量数据同步方式的字段为表名字段中至少包含日志，有的数据表是的表列字段中记录有时间戳，说明该数据表中每写入一条数据记录有时间戳，所以增量数据也是具有时间戳记录的，适用于增量数据同步方式的字段为表列字段中至少包含时间戳。Since different types of data tables have different structures, incremental data may have different characteristics. Here, the fields defined in the data table applicable to the incremental data synchronization method can be table names or row fields in the data table to be synchronized, for example , the table name of some data tables contains logs, indicating that the data in the data table is written one by one, so incremental data is also written one by one, and the field suitable for incremental data synchronization is at least Including logs, some data tables have timestamps recorded in the column fields, indicating that each data record written in the data table has a timestamp, so incremental data also has timestamp records, which is suitable for incremental data synchronization The field of the method is a table column field that contains at least a timestamp.
对于本发明实施例，具体可以通过扫描业务方所有数据表的表结构，解析适用于增量数据同步方式的数据表定义的字段，如表名、表编号等，根据数据表定义的字段获取适用于增量数据同步方式的字段，例如，适用于增量数据表的字段为表名，适用于增量数据同步方式的字段为表名中包含的具体内容，适用于增量数据表的字段为表列字段，适用于增量数据同步方式的字段为表列字段中包含的具体内容。For the embodiment of the present invention, it is possible to scan the table structure of all data tables of the business party, analyze the fields defined in the data tables applicable to the incremental data synchronization method, such as table name, table number, etc., and obtain the applicable fields according to the fields defined in the data table. The fields applicable to the incremental data synchronization method, for example, the field applicable to the incremental data table is the table name, the field applicable to the incremental data synchronization method is the specific content contained in the table name, and the field applicable to the incremental data table is Tabular fields, the fields applicable to the incremental data synchronization method are the specific contents contained in the tabular fields.
202、从所述适用于增量数据同步方式的字段中选取预设字段作为适用于增量数据同步方式的模式字符。202. Select a preset field from the fields applicable to the incremental data synchronization manner as a pattern character applicable to the incremental data synchronization manner.
对于本发明实施例，由于适用于增量数据同步方式的字段可能包含多个，例如表名字段名称中包含日志、报表等，为了提高后续字符串匹配的准确性，可以从适用于增量数据同步方式的字段中选择匹配度较高的字段作为增量数据同步方式的模式字符，这里匹配度较高的字段通常为在适用于增量数据同步方式对应的数据表中出现频率较高的字段，这里不进行限定。For the embodiment of the present invention, since the fields applicable to incremental data synchronization may contain multiple fields, for example, the field name of the table name contains logs, reports, etc., in order to improve the accuracy of subsequent character string matching, it can be applied to incremental data from In the fields of the synchronization method, select the field with a high matching degree as the pattern character of the incremental data synchronization method. Here, the field with a high matching degree is usually a field with a high frequency in the data table corresponding to the incremental data synchronization method , is not limited here.
203、根据所述模式字符中适用于增量数据同步方式的特征信息，确定所述模式字符在增量数据表中的匹配规则。203. Determine a matching rule of the pattern character in the incremental data table according to the feature information in the pattern character applicable to the incremental data synchronization manner.
其中，模式字符中包含适用于增量数据同步方式的特征信息，这里的适用于增量数据同步方式的特征信息可以为位置信息、时间信息或者频率信息等，这里不进行限定。Wherein, the pattern character contains characteristic information applicable to the incremental data synchronization method, where the characteristic information applicable to the incremental data synchronization method may be position information, time information, or frequency information, etc., which are not limited here.
具体地，而不同的特征信息对应有不同的匹配规则，如果模式字符中适用于增量数据同步方式的特征信息为位置信息，以模式字符在增量数据表中的字段位置作为匹配规则；如果模式字符中适用于增量数据同步方式的特征信息为时间信息，以模式字符在增量数据表中的时间戳作为匹配规则；如果模式字符中适用于增量数据同步方式的特征信息为频率信息，以模式字符在增量数据表中的出现次数作为匹配规则。Specifically, different feature information corresponds to different matching rules. If the feature information applicable to the incremental data synchronization method in the pattern character is position information, the field position of the pattern character in the incremental data table is used as the matching rule; if The characteristic information applicable to the incremental data synchronization method in the pattern character is time information, and the time stamp of the pattern character in the incremental data table is used as the matching rule; if the characteristic information applicable to the incremental data synchronization method in the pattern character is frequency information , the number of occurrences of the pattern characters in the incremental data table is used as the matching rule.
204、根据所述模式字符在增量数据表中的匹配规则，构造多个增量数据同步方式的正则表达式。204. Construct multiple regular expressions of incremental data synchronization methods according to the matching rules of the pattern characters in the incremental data table.
其中，构造的增量数据同步方式的正则表达式是由普通字符串(例如a到z)以及特殊字符串(称为“元字符”)组成的文字模式，可以用来检查一个串中是否含有某种子串，将匹配的子串做替换或者从某个串中取出符合某个条件的子串等。Among them, the regular expression of the constructed incremental data synchronization method is a text pattern composed of ordinary character strings (such as a to z) and special character strings (called "metacharacters"), which can be used to check whether a string contains Some kind of substring, replace the matching substring or take out a substring that meets a certain condition from a certain string, etc.
对于本发明实施例，具体可以根据模式字符在增量数据表中的匹配规则，从字符表中筛选出符合增量数据同步方式的普通字符串，该字符表包括用于组合各种匹配规则的普通字符串以及特殊字符串，如数字或者字母等，以及特殊字符串，如下划线、斜杠等，并且这里的正则表达式具有特定的表达式模板，将符合增量数据同步方式的普通字符串以及特殊字符串导入至表达式模板中，构造多个增量数据同步方式的正则表达式，能够将某个字符模式与所搜索的字符进行匹配。For the embodiment of the present invention, according to the matching rules of the pattern characters in the incremental data table, the ordinary character strings that meet the incremental data synchronization method can be screened out from the character table. The character table includes a combination of various matching rules Ordinary strings and special strings, such as numbers or letters, and special strings, such as underscores, slashes, etc., and the regular expressions here have specific expression templates, which will conform to the ordinary strings of the incremental data synchronization method And special strings are imported into the expression template to construct multiple regular expressions for incremental data synchronization, which can match a certain character pattern with the searched character.
205、当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串。205. When an incremental data synchronization request is received, match the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode, and find out whether the data in the incremental data table to be synchronized exists The string that matches the data.
对于本发明实施例，随着时间的更新，业务中心会将新增的数据放入待同步增量数据表中，当待同步增量数据表中累积到一定数据量或者业务方发送增量数据同步请求时，将增量数据同步至数据中心，这里主要针对的增量数据进行同步，该数据可以为数据库中用户的行为数据，可以为平台中记录的日志数据等。For the embodiment of the present invention, as time is updated, the business center will put the newly added data into the incremental data table to be synchronized, when a certain amount of data is accumulated in the incremental data table to be synchronized or the business party sends incremental data When a synchronization request is made, the incremental data is synchronized to the data center. Here, the incremental data is mainly synchronized. The data can be user behavior data in the database or log data recorded in the platform.
具体可以根据增量数据同步方式的正则表达式去匹配待同步增量数据表中的字段内容，查找待同步增量数据表中的字段内容是否存在符合适用于增量数据同步方式的匹配规则的字符串。Specifically, you can match the field content in the incremental data table to be synchronized according to the regular expression of the incremental data synchronization method, and find out whether the field content in the incremental data table to be synchronized meets the matching rules applicable to the incremental data synchronization method. string.
206a、若存在，则根据所述与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对所述待同步增量数据表进行增量数据同步。206a. If yes, perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method with the largest number of character strings matching the incremental data in the data table to be synchronized.
对于本发明实施例，如果待同步增量数据表中存在适用于增量数据同步方式的匹配规则的字符串，说明待同步增量数据表对应的结构满足该种增量数据同步方式的条件，从而将匹配字符串数量最多的增量数据同步方式作为适用待同步增量数据表的增量数据同步方式，从而使用该增量数据同步方式对待同步数据表进行增量数据同步，以提高增量数据的同步速度。For the embodiment of the present invention, if there is a character string applicable to the matching rules of the incremental data synchronization mode in the incremental data table to be synchronized, it means that the structure corresponding to the incremental data table to be synchronized satisfies the conditions of this incremental data synchronization method, Therefore, the incremental data synchronization method with the largest number of matching character strings is used as the incremental data synchronization method applicable to the incremental data table to be synchronized, so that the incremental data synchronization method is used to perform incremental data synchronization on the data table to be synchronized to improve the incremental data. Data synchronization speed.
与步骤206a相对应的有步骤206b、若不存在，则根据待同步增量数据表对应的数据量大小确定适用于待同步增量数据表的增量数据同步方式。Corresponding to step 206a, there is step 206b. If it does not exist, then determine the incremental data synchronization method suitable for the incremental data table to be synchronized according to the data volume corresponding to the incremental data table to be synchronized.
需要说明的是，这里的正则表达式并不适用于所有的增量数据同步方式，有的待同步增量数据表中的字段内容无法通过正则表达式匹配的方式来分析适合该待同步增量数据表的增量数据同步方式，那么通过正则表达式是无法查找到待同步增量数据表中存在匹配的字符串，所以需要采用其他的增量数据同步方式，例如，全表删除的增量数据同步方式等。It should be noted that the regular expressions here are not applicable to all incremental data synchronization methods, and some field contents in the incremental data tables to be synchronized cannot be analyzed by regular expression matching to be suitable for the incremental data to be synchronized. The incremental data synchronization method of the data table, then the regular expression cannot be used to find the matching string in the incremental data table to be synchronized, so other incremental data synchronization methods need to be used, for example, the incremental data deleted from the entire table Data synchronization method, etc.
对于本发明实施例，如果不存在适用于增量数据同步方式的匹配规则的字符串，则说明构造多个增量数据同步方式的正则表达式都不适用于待同步数据表，需要选择其他的增量数据同步方式，进一步根据待同步增量数据表对应的数据量大小确定适用于待同步增量数据表的增量数据同步方式。For the embodiment of the present invention, if there is no string applicable to the matching rules of the incremental data synchronization method, it means that none of the regular expressions for constructing multiple incremental data synchronization methods is applicable to the data table to be synchronized, and other options need to be selected The incremental data synchronization method further determines the incremental data synchronization method applicable to the incremental data table to be synchronized according to the data volume corresponding to the incremental data table to be synchronized.
207b、根据所述适用于待同步增量数据表的增量数据同步方式对所述待同步增量数据表进行增量数据同步。207b. Perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method applicable to the incremental data table to be synchronized.
对于本发明实施例，具体可以通过统计待同步增量数据表中的数据条数，判断待同步增量数据表中的数据量大小，如果待同步增量数据表中的数据条数小于或等于预设数值，说明待同步增量数据表的数据量较小，则确定采用全表删除的增量数据同步方式，如果待同步增量数据表中的数据条数大于预设数值，说明待同步增量数据表的数量较大，则确定采用逐条比对的增量数据同步方式。For the embodiment of the present invention, it is possible to judge the amount of data in the incremental data table to be synchronized by counting the number of data items in the incremental data table to be synchronized specifically. If the number of data items in the incremental data table to be synchronized is less than or equal to The preset value indicates that the amount of data in the incremental data table to be synchronized is small, and the incremental data synchronization method of deleting the entire table is determined. If the number of data entries in the incremental data table to be synchronized is greater than the preset value, it indicates that the incremental data table to be synchronized is to be synchronized If the number of incremental data tables is large, it is determined to use the incremental data synchronization method of one-by-one comparison.
进一步地，作为图1所述方法的具体实现，本发明实施例提供了一种增量数据同步的装置，如图3所示，所述装置包括：获取单元31、构造单元32、查找单元33、第一同步单元34。Further, as a specific implementation of the method described in FIG. 1 , an embodiment of the present invention provides an incremental data synchronization device. As shown in FIG. 3 , the device includes: an acquisition unit 31 , a construction unit 32 , and a search unit 33 , the first synchronization unit 34 .
获取单元31，可以用于获取适用于增量数据同步方式的模式字符；The obtaining unit 31 can be used to obtain the mode character applicable to the incremental data synchronization method;
构造单元32，可以用于根据所述模式字符确定适用于增量数据同步方式的预设匹配规则，构造多个增量数据同步方式的正则表达式；The construction unit 32 can be used to determine the preset matching rules applicable to the incremental data synchronization method according to the pattern characters, and construct a plurality of regular expressions for the incremental data synchronization method;
查找单元33，可以用于当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串；The search unit 33 can be used to match the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode when receiving the incremental data synchronization request, and find out whether there is an The string to match the data in the data table to be synchronized;
第一同步单元34，可以用于若存在，则根据所述与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对所述待同步增量数据表进行增量数据同步。The first synchronization unit 34 may be configured to, if it exists, increment the incremental data table to be synchronized according to the incremental data synchronization method with the largest number of strings matching the incremental data in the data table to be synchronized data synchronization.
本发明提供一种增量数据同步的装置，当接收到增量数据同步请求时，通过多个增量数据同步方式的正则表达式来匹配待同步增量数据，查找是否含存在与待同步数据表中增量数据相匹配的字符串，若存在，则说明待同步数据表中的数据适用于多个增量数据同步方式中的一种，进一步根据与所述待同步数据表中增量数据匹配字符串数量最多的增量数据同步方式对待同步增量数据表进行增量数据同步，以提高增量数据同步的效率。与现有技术中通过人工操作来选择合适的增量数据同步方式相比，本发明实施例通过构造适用于不同增量数据同步方式的正则表达式来匹配待同步增量数据表中的字段，可以针对增量数据表的特性选择适合增量数据同步方式完成增量数据同步任务，由于合适的增量数据同步方式可以优化增量数据同步效果，以及增量数据同步的准确性，通过合适的增量数据同步方式进行增量数据同步，节约系统资源，减少人为操作，以及提高增量数据同步的资源利用率。The present invention provides a device for synchronizing incremental data. When an incremental data synchronization request is received, the incremental data to be synchronized is matched through regular expressions of multiple incremental data synchronization modes, and the incremental data to be synchronized is searched for whether it contains the data to be synchronized. If there is a character string matching the incremental data in the table, it means that the data in the data table to be synchronized is suitable for one of multiple incremental data synchronization methods, and further according to the incremental data in the data table to be synchronized The incremental data synchronization method with the largest number of matching character strings performs incremental data synchronization on the incremental data table to be synchronized to improve the efficiency of incremental data synchronization. Compared with selecting an appropriate incremental data synchronization method through manual operation in the prior art, the embodiment of the present invention matches the fields in the incremental data table to be synchronized by constructing a regular expression suitable for different incremental data synchronization methods, According to the characteristics of the incremental data table, the incremental data synchronization method can be selected to complete the incremental data synchronization task. Because the appropriate incremental data synchronization method can optimize the incremental data synchronization effect and the accuracy of the incremental data synchronization, through the appropriate The incremental data synchronization method performs incremental data synchronization, saves system resources, reduces manual operations, and improves resource utilization of incremental data synchronization.
作为图3中所示增量数据同步的装置的进一步说明，图4是根据本发明实施例另一种增量数据同步的装置的结构示意图，如图4所示，所述装置还包括：As a further description of the device for incremental data synchronization shown in FIG. 3 , FIG. 4 is a schematic structural diagram of another device for incremental data synchronization according to an embodiment of the present invention. As shown in FIG. 4 , the device further includes:
确定单元35，可以用于在所述当接收到增量数据同步请求时，将所述待同步增量数据表中的数据分别与每个增量同步方式的正则表达式进行匹配，查找是否存在与所述待同步数据表中的数据相匹配的字符串之后，若不存在，则根据待同步增量数据表对应的数据量大小确定适用于待同步增量数据表的增量数据同步方式；The determining unit 35 may be configured to match the data in the incremental data table to be synchronized with the regular expressions of each incremental synchronization mode when the incremental data synchronization request is received, and find whether there is After the character string that matches the data in the data table to be synchronized, if it does not exist, then determine the incremental data synchronization method suitable for the incremental data table to be synchronized according to the corresponding data size of the incremental data table to be synchronized;
第二同步单元36，可以用于根据所述适用于待同步增量数据表的增量数据同步方式对所述待同步增量数据表进行增量数据同步。The second synchronization unit 36 may be configured to perform incremental data synchronization on the incremental data table to be synchronized according to the incremental data synchronization method applicable to the incremental data table to be synchronized.
进一步地，所述确定单元35包括：Further, the determining unit 35 includes:
统计模块351，可以用于统计所述待同步增量数据表中的数据条数；Statistical module 351, which can be used to count the number of data items in the incremental data table to be synchronized;
第二确定模块352，可以用于如果所述待同步增量数据表中的数据条数小于或等于预设数值，则确定采用全表删除的增量数据同步方式；The second determination module 352 can be used to determine to adopt the incremental data synchronization method of deleting the entire table if the number of data items in the incremental data table to be synchronized is less than or equal to the preset value;
所述第二确定模块352，还可以用于如果所述待同步增量数据表中的数据条数大于预设数值，则确定采用逐条比对的增量数据同步方式。The second determining module 352 may also be configured to determine to adopt an incremental data synchronization method of item-by-item comparison if the number of data items in the incremental data table to be synchronized is greater than a preset value.
进一步地，所述获取单元31包括：Further, the acquisition unit 31 includes:
获取模块311，可以用于通过解析适用于增量数据同步方式的数据表定义的字段，获取适用于增量数据同步方式的字段；The obtaining module 311 can be used to obtain fields suitable for incremental data synchronization by analyzing fields defined in data tables suitable for incremental data synchronization;
选取模块312，可以用于从所述适用于增量数据同步方式的字段中选取预设字段作为适用于增量数据同步方式的模式字符。The selection module 312 may be configured to select a preset field from the fields applicable to the incremental data synchronization method as a pattern character applicable to the incremental data synchronization method.
进一步地，所述模式字符中包含适用于增量数据同步方式的特征信息，所述构造单元32包括：Further, the pattern character contains feature information suitable for incremental data synchronization, and the construction unit 32 includes:
第一确定模块321，可以用于根据所述模式字符中适用于增量数据同步方式的特征信息，确定所述模式字符在增量数据表中的匹配规则；The first determination module 321 can be used to determine the matching rules of the pattern characters in the incremental data table according to the feature information in the pattern characters applicable to the incremental data synchronization mode;
构造模块322，可以用于根据所述模式字符在增量数据表中的匹配规则，构造多个增量数据同步方式的正则表达式。The construction module 322 may be configured to construct multiple regular expressions of incremental data synchronization methods according to the matching rules of the pattern characters in the incremental data table.
进一步地，所述第一确定模块321，具体可以用于如果所述模式字符中适用于增量数据同步方式的特征信息为位置信息，以所述模式字符在增量数据表中的字段位置作为匹配规则；Further, the first determination module 321 can be specifically configured to use the field position of the pattern character in the incremental data table as the feature information applicable to the incremental data synchronization method in the pattern character if the feature information is position information. matching rules;
所述第一确定模块321，具体还可以用于如果所述模式字符中适用于增量数据同步方式的特征信息为时间信息，以所述模式字符在增量数据表中的时间戳作为匹配规则；The first determining module 321 can also be specifically configured to use the time stamp of the pattern character in the incremental data table as the matching rule if the feature information applicable to the incremental data synchronization method in the pattern character is time information ;
所述第一确定模块321，具体还可以用于如果所述模式字符中适用于增量数据同步方式的特征信息为频率信息，以所述模式字符在增量数据表中的出现次数作为匹配规则。The first determining module 321 can also be specifically configured to use the number of occurrences of the pattern character in the incremental data table as the matching rule if the feature information applicable to the incremental data synchronization method in the pattern character is frequency information .
进一步地，所述构造模块322，具体可以用于根据所述模式字符在增量数据表中的匹配规则，从字符表中筛选出符合增量数据同步方式的普通字符串以及特殊字符串，所述字符表包括用于组合各种匹配规则的普通字符串以及特殊字符串；Further, the construction module 322 can specifically be used to filter out ordinary character strings and special character strings that meet the incremental data synchronization method from the character table according to the matching rules of the pattern characters in the incremental data table, so The above character table includes common character strings and special character strings used to combine various matching rules;
所述构造模块322，具体还可以用于将所述符合增量数据同步方式的普通字符串以及特殊字符串导入至表达式模板中，构造多个增量数据同步方式的正则表达式。The construction module 322 can also be specifically configured to import the normal character strings and special character strings conforming to the incremental data synchronization method into the expression template, and construct multiple regular expressions of the incremental data synchronization method.
需要说明的是，本实施例提供的一种增量数据同步的装置所涉及各功能单元的其他相应描述，可以参考图1和图2中的对应描述，在此不再赘述。It should be noted that, for other corresponding descriptions of the functional units involved in an incremental data synchronization device provided in this embodiment, reference may be made to the corresponding descriptions in FIG. 1 and FIG. 2 , which will not be repeated here.
基于上述如图1和图2所示方法，相应的，本实施例还提供了一种存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述如图1和图2所示的增量数据同步的方法。Based on the method shown in Figure 1 and Figure 2 above, correspondingly, this embodiment also provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned process as shown in Figure 1 and Figure 2 is realized. method for incremental data synchronization.
基于这样的理解，本申请的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施场景所述的方法。Based on this understanding, the technical solution of the present application can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.), including several The instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in various implementation scenarios of the present application.
基于上述如图1、图2所示的方法，以及图3、图4所示的虚拟装置实施例，为了实现上述目的，本申请实施例还提供了一种计算机设备，具体可以为个人计算机、服务器、网络设备等，该实体设备包括存储介质和处理器；存储介质，用于存储计算机程序；处理器，用于执行计算机程序以实现上述如图1和图2所示的增量数据同步的方法。Based on the method shown in Figure 1 and Figure 2 above, and the virtual device embodiment shown in Figure 3 and Figure 4, in order to achieve the above purpose, the embodiment of this application also provides a computer device, which can be a personal computer, A server, a network device, etc., the physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to realize the above incremental data synchronization as shown in Figure 1 and Figure 2 method.
可选地，该计算机设备还可以包括用户接口、网络接口、摄像头、射频(RadioFrequency，RF)电路，传感器、音频电路、WI-FI模块等等。用户接口可以包括显示屏(Display)、输入单元比如键盘(Keyboard)等，可选用户接口还可以包括USB接口、读卡器接口等。网络接口可选的可以包括标准的有线接口、无线接口(如蓝牙接口、WI-FI接口)等。Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (Radio Frequency, RF) circuit, a sensor, an audio circuit, a WI-FI module, and the like. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the like, and optional user interfaces may also include a USB interface, a card reader interface, and the like. Optionally, the network interface may include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface) and the like.
本领域技术人员可以理解，本实施例提供的增量数据同步的的实体设备结构并不构成对该实体设备的限定，可以包括更多或更少的部件，或者组合某些部件，或者不同的部件布置。Those skilled in the art can understand that the entity device structure for incremental data synchronization provided in this embodiment does not constitute a limitation on the entity device, and may include more or less components, or combine certain components, or have different Part placement.
存储介质中还可以包括操作系统、网络通信模块。操作系统是管理上述计算机设备硬件和软件资源的程序，支持信息处理程序以及其它软件和/或程序的运行。网络通信模块用于实现存储介质内部各组件之间的通信，以及与该实体设备中其它硬件和软件之间通信。The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned computer equipment, and supports the operation of information processing programs and other software and/or programs. The network communication module is used to realize the communication between various components inside the storage medium, and communicate with other hardware and software in the physical device.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到本申请可以借助软件加必要的通用硬件平台的方式来实现，也可以通过硬件实现。通过应用本申请的技术方案，与目前现有技术相比，本发明实施例通过构造适用于不同增量数据同步方式的正则表达式来匹配待同步增量数据表中的字段，可以针对增量数据表的特性选择适合增量数据同步方式完成增量数据同步任务，由于合适的增量数据同步方式可以优化增量数据同步效果，以及增量数据同步的准确性，通过合适的增量数据同步方式进行增量数据同步，节约系统资源，减少人为操作，以及提高增量数据同步的资源利用率。Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software plus a necessary general-purpose hardware platform, or by hardware. By applying the technical solution of the present application, compared with the current prior art, the embodiment of the present invention matches the fields in the incremental data table to be synchronized by constructing regular expressions suitable for different incremental data synchronization methods, which can target incremental The feature selection of the data table is suitable for the incremental data synchronization method to complete the incremental data synchronization task. Because the appropriate incremental data synchronization method can optimize the incremental data synchronization effect and the accuracy of the incremental data synchronization, through the appropriate incremental data synchronization Incremental data synchronization is performed in this way, which saves system resources, reduces manual operations, and improves the resource utilization rate of incremental data synchronization.
本领域技术人员可以理解附图只是一个优选实施场景的示意图，附图中的模块或流程并不一定是实施本申请所必须的。本领域技术人员可以理解实施场景中的装置中的模块可以按照实施场景描述进行分布于实施场景的装置中，也可以进行相应变化位于不同于本实施场景的一个或多个装置中。上述实施场景的模块可以合并为一个模块，也可以进一步拆分成多个子模块。Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art can understand that the modules in the devices in the implementation scenario can be distributed among the devices in the implementation scenario according to the description of the implementation scenario, or can be located in one or more devices different from the implementation scenario according to corresponding changes. The modules of the above implementation scenarios can be combined into one module, or can be further split into multiple sub-modules.
上述本申请序号仅仅为了描述，不代表实施场景的优劣。以上公开的仅为本申请的几个具体实施场景，但是，本申请并非局限于此，任何本领域的技术人员能思之的变化都应落入本申请的保护范围。The serial numbers of the above application are for description only, and do not represent the pros and cons of the implementation scenarios. The above disclosures are only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes conceivable by those skilled in the art shall fall within the protection scope of the present application.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910218794.1A CN110083579A (en) | 2019-03-21 | 2019-03-21 | Incremental data synchronous method, apparatus, computer equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910218794.1A CN110083579A (en) | 2019-03-21 | 2019-03-21 | Incremental data synchronous method, apparatus, computer equipment and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110083579A true CN110083579A (en) | 2019-08-02 |
Family
ID=67413370
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910218794.1A Pending CN110083579A (en) | 2019-03-21 | 2019-03-21 | Incremental data synchronous method, apparatus, computer equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110083579A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633321A (en) * | 2019-08-30 | 2019-12-31 | 中国人民财产保险股份有限公司 | Data synchronization method, device and equipment |
| CN113206891A (en) * | 2021-05-31 | 2021-08-03 | 浙江大华技术股份有限公司 | Scheduling method, storage system, scheduling device and storage medium for metadata reporting |
| CN113961572A (en) * | 2021-12-23 | 2022-01-21 | 中电云数智科技有限公司 | Database synchronization method and synchronization device based on increment field |
| US20220083510A1 (en) * | 2020-09-15 | 2022-03-17 | Open Text Holdings, Inc. | Connector for content repositories |
| CN114911862A (en) * | 2022-07-18 | 2022-08-16 | 国网江苏省电力有限公司营销服务中心 | System and method for transmitting big data of network operation link of network |
| CN115510166A (en) * | 2022-11-23 | 2022-12-23 | 云账户技术(天津)有限公司 | Data synchronization service method and device, remote platform and electronic equipment |
| CN115883582A (en) * | 2021-08-02 | 2023-03-31 | 中国移动通信有限公司研究院 | Data transmission method, electronic equipment and digital twin platform |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289460A (en) * | 2011-07-13 | 2011-12-21 | 中国工商银行股份有限公司 | Method and system for performing report metadata synchronization on testing environments |
| CN103164477A (en) * | 2011-12-19 | 2013-06-19 | 江南大学 | Interactive display device based on Action Script 3.0 platform network remote data capture |
| CN103440317A (en) * | 2013-08-28 | 2013-12-11 | 电子科技大学 | Incremental data inquiry method based on database time labels |
| CN105373448A (en) * | 2015-10-27 | 2016-03-02 | 北京百度网讯科技有限公司 | Method and system for recovering failure data in database |
| US20170132245A1 (en) * | 2015-11-05 | 2017-05-11 | International Business Machines Corporation | Storage controller and method for managing a solid-state memory |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
| CN107273471A (en) * | 2017-06-07 | 2017-10-20 | 国网上海市电力公司 | A kind of binary electric power time series data index structuring method based on Geohash |
| CN107330003A (en) * | 2017-06-12 | 2017-11-07 | 上海藤榕网络科技有限公司 | Method of data synchronization, system, memory and data syn-chronization equipment |
| CN108595522A (en) * | 2018-03-27 | 2018-09-28 | 北京明朝万达科技股份有限公司 | A kind of increment synchronization method, apparatus and system |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN108804487A (en) * | 2017-12-28 | 2018-11-13 | 中国移动通信集团公司 | A kind of method and device of extraction target character |
| CN109388676A (en) * | 2018-08-21 | 2019-02-26 | 中国平安人寿保险股份有限公司 | Data synchronize generation method, device, computer equipment and storage medium |
| CN115658382A (en) * | 2022-10-11 | 2023-01-31 | 四川大学 | Incremental object backup and recovery method, system and storage medium for object storage |
-
2019
- 2019-03-21 CN CN201910218794.1A patent/CN110083579A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289460A (en) * | 2011-07-13 | 2011-12-21 | 中国工商银行股份有限公司 | Method and system for performing report metadata synchronization on testing environments |
| CN103164477A (en) * | 2011-12-19 | 2013-06-19 | 江南大学 | Interactive display device based on Action Script 3.0 platform network remote data capture |
| CN103440317A (en) * | 2013-08-28 | 2013-12-11 | 电子科技大学 | Incremental data inquiry method based on database time labels |
| CN105373448A (en) * | 2015-10-27 | 2016-03-02 | 北京百度网讯科技有限公司 | Method and system for recovering failure data in database |
| US20170132245A1 (en) * | 2015-11-05 | 2017-05-11 | International Business Machines Corporation | Storage controller and method for managing a solid-state memory |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN107273471A (en) * | 2017-06-07 | 2017-10-20 | 国网上海市电力公司 | A kind of binary electric power time series data index structuring method based on Geohash |
| CN107330003A (en) * | 2017-06-12 | 2017-11-07 | 上海藤榕网络科技有限公司 | Method of data synchronization, system, memory and data syn-chronization equipment |
| CN108804487A (en) * | 2017-12-28 | 2018-11-13 | 中国移动通信集团公司 | A kind of method and device of extraction target character |
| CN108595522A (en) * | 2018-03-27 | 2018-09-28 | 北京明朝万达科技股份有限公司 | A kind of increment synchronization method, apparatus and system |
| CN109388676A (en) * | 2018-08-21 | 2019-02-26 | 中国平安人寿保险股份有限公司 | Data synchronize generation method, device, computer equipment and storage medium |
| CN115658382A (en) * | 2022-10-11 | 2023-01-31 | 四川大学 | Incremental object backup and recovery method, system and storage medium for object storage |
Non-Patent Citations (2)
| Title |
|---|
| 庞秋奔;李银;: "基于Web Service多源异构系统增量同步的实现", 计算机应用与软件, no. 05, 12 May 2018 (2018-05-12), pages 172 - 176 * |
| 赵文芳;刘旭林;: "Spark Streaming框架下的气象自动站数据实时处理系统", 计算机应用, no. 01, 10 January 2018 (2018-01-10), pages 38 - 43 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633321A (en) * | 2019-08-30 | 2019-12-31 | 中国人民财产保险股份有限公司 | Data synchronization method, device and equipment |
| CN110633321B (en) * | 2019-08-30 | 2022-06-03 | 中国人民财产保险股份有限公司 | Data synchronization method, device and equipment |
| US20220083510A1 (en) * | 2020-09-15 | 2022-03-17 | Open Text Holdings, Inc. | Connector for content repositories |
| US12242426B2 (en) * | 2020-09-15 | 2025-03-04 | Open Ext Holdings, Inc. | Bi-directional synchronization of content and metadata between repositories |
| CN113206891A (en) * | 2021-05-31 | 2021-08-03 | 浙江大华技术股份有限公司 | Scheduling method, storage system, scheduling device and storage medium for metadata reporting |
| CN115883582A (en) * | 2021-08-02 | 2023-03-31 | 中国移动通信有限公司研究院 | Data transmission method, electronic equipment and digital twin platform |
| CN113961572A (en) * | 2021-12-23 | 2022-01-21 | 中电云数智科技有限公司 | Database synchronization method and synchronization device based on increment field |
| CN114911862A (en) * | 2022-07-18 | 2022-08-16 | 国网江苏省电力有限公司营销服务中心 | System and method for transmitting big data of network operation link of network |
| CN114911862B (en) * | 2022-07-18 | 2022-12-06 | 国网江苏省电力有限公司营销服务中心 | System and method for transmitting big data of network national network operation link |
| CN115510166A (en) * | 2022-11-23 | 2022-12-23 | 云账户技术(天津)有限公司 | Data synchronization service method and device, remote platform and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110083579A (en) | Incremental data synchronous method, apparatus, computer equipment and computer storage medium | |
| US10180992B2 (en) | Atomic updating of graph database index structures | |
| US8924426B2 (en) | Joining tables in a mapreduce procedure | |
| US8924373B2 (en) | Query plans with parameter markers in place of object identifiers | |
| US9679021B2 (en) | Parallel transactional-statistics collection for improving operation of a DBMS optimizer module | |
| US20170255708A1 (en) | Index structures for graph databases | |
| CN111339171B (en) | Data query method, device and device | |
| CN114328632A (en) | User data analysis method and device based on bitmap and computer equipment | |
| WO2018097846A1 (en) | Edge store designs for graph databases | |
| US10445370B2 (en) | Compound indexes for graph databases | |
| CN112559913B (en) | Data processing method, device, computing equipment and readable storage medium | |
| US10824803B2 (en) | System and method for logical identification of differences between spreadsheets | |
| CN106503186A (en) | A kind of data managing method, client and system | |
| CN110489483B (en) | Data synchronization method, device, computer equipment and storage medium | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| CN110928900B (en) | Multi-table data query method, device, terminal and computer storage medium | |
| US10216747B2 (en) | Customized synthetic data creation | |
| CN107609038B (en) | Data cleaning method and device | |
| CN113591447A (en) | Data table creating method, device, equipment and storage medium | |
| CN111737316A (en) | A project list query method, device, computer equipment and storage medium | |
| US20180101622A1 (en) | Perform graph traversal with graph query language | |
| CN116383192A (en) | Data query method, device, equipment and storage medium | |
| CN114254166A (en) | Federated Graph Database Architecture | |
| CN114281921A (en) | Data processing method, device, storage medium and device | |
| JP2013171495A (en) | Data management device, data management method and data management program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190802 |