CN102508834B - Message-based event application device and message-based event application method - Google Patents
Message-based event application device and message-based event application method Download PDFInfo
- Publication number
- CN102508834B CN102508834B CN201110284525.9A CN201110284525A CN102508834B CN 102508834 B CN102508834 B CN 102508834B CN 201110284525 A CN201110284525 A CN 201110284525A CN 102508834 B CN102508834 B CN 102508834B
- Authority
- CN
- China
- Prior art keywords
- message
- data source
- affairs
- scale
- action statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 230000009471 action Effects 0.000 claims description 52
- 230000008569 process Effects 0.000 claims description 26
- 230000001360 synchronised effect Effects 0.000 claims description 11
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 13
- 238000006243 chemical reaction Methods 0.000 description 9
- 238000013075 data extraction Methods 0.000 description 8
- 238000012856 packing Methods 0.000 description 7
- 239000000284 extract Substances 0.000 description 5
- 230000009897 systematic effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005194 fractionation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000002203 pretreatment Methods 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a message-based event application device, which is used for enabling incremental data manipulation statements applied to a first data source to be synchronously applied to a second data source, and comprises a dividing unit and an application unit, wherein the dividing unit is used for obtaining the incremental data manipulation statements from the first data source and packeting the incremental data manipulation statements according to transactions, judging whether the transactions are events, dividing the events into multiple parts and sending a message formed by the incremental data manipulation statements in each part into the application unit; and the application unit is used for creating local transactions of the second data source and corresponding application tasks when receiving the message, so that the application tasks apply the incremental data manipulation statements in the message to the second data source based on the local transactions. The invention further provides a message-based event application method. According to the technical scheme, synchronism of real-time incremental data can be improved.
Description
Technical field
The present invention relates to data synchronization technology field, in particular to message based large-scale transactional applications apparatus and method.
Background technology
In the synchronous schematic diagram of the data shown in Fig. 1, first, incremental data extraction module (SEM) 102 extracts increment operation from OLTP data source, to these data, to do basic analysis simultaneously and classify by affairs, then pack and be transferred to data conversion module (SPM) 104, data conversion module (SPM) 104 carries out conversion process to the SQL operation of affairs after receiving packed affairs as requested, and after finishing dealing with, notification data application module (SAM) 106 carrys out applied transaction.In order to guarantee synchronous definitely reliable of whole data increment, in distributed environment lower floor, can dispose a transmission platform (such as message-oriented middleware etc.), by it, guarantee synchronous reliability.This is no problem in the ordinary course of things, but for example, when to run into the affairs that are packaged into are large-scale affairs (a batch updating operation has 100,000,000 records), in this case, there will be following problem:
Once generate the large-scale affairs that comprise so many SQL operation, system is probably because low memory directly collapses;
Even if internal memory can pack processing containing the message of large-scale like this affairs, but be all after receiving whole affairs by current realization mechanism, SPM and SAM just can start follow-up processing, and the synchronous flow process of whole like this data is equivalent to become serial processing, to data net synchronization capability, also can have a great impact.
Therefore, need a kind of can, for the synchrodata treatment technology of large-scale affairs, making can improve data net synchronization capability when processing large-scale affairs.
Summary of the invention
In view of this, technical matters to be solved by this invention is, a kind of message based large-scale transactional applications technology is provided, and utilizes this platform of message-oriented middleware to process the remote transmission of large-scale affairs, to improve data net synchronization capability.
According to an aspect of the present invention, a kind of message based large-scale transactional applications device is provided, be used for the incremental data action statement synchronous applications that is applied to the first data source in the second data source, comprise: resolving cell, from described the first data source, obtaining incremental data action statement, described incremental data action statement is packed by affairs, when judging described affairs and being large-scale affairs, described large-scale affairs are decomposed into many parts, every part of described incremental data action statement comprising is configured to a message, described message is sent to applying unit; Described applying unit, when receiving described message, create the local matter of described the second data source and create corresponding application task, described application task is applied to described the second data source based on described local matter by the incremental data action statement comprising in described message.
In technique scheme, preferably, described resolving cell can also comprise: message constructing module, the described large-scale affairs after many parts of decomposition are formed to a plurality of described message, and a plurality of described message forms a message groups; Module is set, the attribute of message described in each is set, the identity identification information of message groups, the message number in affiliated message groups, the sequence number of described message under described attribute comprises, the identity identification information of described message groups comprises the title of described the first data source and the local matter of described the second data source sign.
In technique scheme, preferably, described applying unit can also comprise: the second judge module, when the incremental data action statement comprising in current described message has been applied, judge whether described message belongs to message groups, if judge, belong to message groups, according to the attribute of current described message, continue to judge whether the message in described message groups all finishes dealing with, when judgment result is that while all finishing dealing with, submit the local matter of described the second data source to, when judgment result is that while all not finishing dealing with, continue to process the untreated message of residue.
In technique scheme, preferably, described resolving cell can also comprise: the first judge module, when the number of the incremental data action statement in described affairs surpasses the threshold value of the incremental data action statement that default described message can comprise, judging described affairs is large-scale affairs.
In technique scheme, preferably, described applying unit creates the local matter of described the second data source based on XA interface specification, and described the first data source is OLTP data source, and described the second data source is OLAP data source.
Technique scheme splits by the ultra-large type affairs to excavating from OLTP data source, can obviously improve the situation of incremental data application: by split can control message size, that avoids that system may occur can not send the situation that data are even collapsed because of low memory; Setting by systematic parameter (the maximum operand that message can comprise), can adapt to hardware environment intelligently; By splitting super large affairs, original serial application is changed into Parallel application, obviously improve the performance of system.
According to another aspect of the invention, a kind of message based large-scale transactional applications method is also provided, be used for the incremental data action statement synchronous applications that is applied to the first data source in the second data source, comprise: step 402, from described the first data source, obtain incremental data action statement, described incremental data action statement is packed by affairs, when judging described affairs and be large-scale affairs, described large-scale affairs are decomposed into many parts, every part of described incremental data action statement comprising is configured to a message; Step 404, when receiving described message, create the local matter of described the second data source and create corresponding application task, described application task is applied to described the second data source based on described local matter by the incremental data action statement comprising in described message.
In technique scheme, preferably, described step 402 can also comprise: the described large-scale affairs after many parts of decomposition are formed to a plurality of described message, a plurality of described message forms a message groups, the attribute of message described in each is set, the identity identification information of message groups, the message number in affiliated message groups, the sequence number of described message under described attribute comprises, the identity identification information of described message groups comprises the title of described the first data source and the local matter of described the second data source sign.
In technique scheme, preferably, described step 404 can also comprise: when the incremental data action statement comprising in current described message has been applied, judge whether described message belongs to message groups, if judge, belong to message groups, according to the attribute of current described message, continue to judge whether the message in described message groups all finishes dealing with, when judgment result is that while all finishing dealing with, submit the local matter of described the second data source to, when judgment result is that while all not finishing dealing with, continue to process the untreated message of residue.
In technique scheme, preferably, described step 402 can also comprise: when the number of the incremental data action statement in described affairs surpasses the threshold value of the incremental data action statement that default described message can comprise, judging described affairs is large-scale affairs.
In technique scheme, preferably, the local matter of described the second data source creates and forms based on XA interface specification, and described the first data source is OLTP data source, and described the second data source is OLAP data source.
Technique scheme is by splitting the large-scale affairs of excavating from OLTP data source, can obviously improve the situation of incremental data application: by split can control message size, that avoids that system may occur can not send the situation that data are even collapsed because of low memory; Setting by systematic parameter (the maximum operand that message can comprise), can adapt to hardware environment intelligently; By splitting super large affairs, original serial application is changed into Parallel application, obviously improve the performance of system.
Accompanying drawing explanation
Fig. 1 shows the synchronous schematic diagram of data of correlation technique;
Fig. 2 shows the block diagram of message based according to an embodiment of the invention large-scale transactional applications system;
Fig. 3 shows the block diagram of message based according to an embodiment of the invention large-scale transactional applications device;
Fig. 4 shows the process flow diagram of message based according to an embodiment of the invention large-scale transactional applications method;
Fig. 5 shows the process flow diagram that extracts according to an embodiment of the invention incremental data action statement;
Fig. 6 shows the process flow diagram of the affairs that send according to an embodiment of the invention single packing; And
Fig. 7 shows the process flow diagram of applying according to an embodiment of the invention large-scale affairs.
Embodiment
In order more clearly to understand above-mentioned purpose of the present invention, feature and advantage, below in conjunction with the drawings and specific embodiments, the present invention is further described in detail.
Set forth in the following description a lot of details so that fully understand the present invention, still, the present invention can also adopt other to be different from other modes described here and implement, and therefore, the present invention is not limited to the restriction of following public specific embodiment.
The implication of english abbreviation and being explained as follows of related terms that hereinafter occur:
DI:Data Integration, the abbreviation of data integration.
OLTP data source: the data base management system (DBMS) that is used for processing trade type business in enterprise-oriented system.
OLAP data source: the data base management system (DBMS) that is used for processing inquiry/analytic type business in enterprise-oriented system.Consideration for performance, large enterprise is when the distributed service operation system of framework oneself, generally all this two classes business separately can be processed, correspondingly data source also can separately be used, and across the data of data source, synchronously can or regularly complete by hand with instrument.
Affairs: refer to the set of a series of operation.Affairs have atomicity, consistance, four characteristics of isolation and persistence (ACID).
SQL operation: mainly refer to here in business and use maximum DML to operate, as update (INSERT), deletion action (DELETE) and renewal operation (UPDATE), in fact this is the form of expression of incremental data in OLTP data source, by these operations, the legacy data in OLAP data source can be updated to up-to-date state.
The master data action statement extraction module of SQL Redo Info Extractor Module:OLTP data source, is called for short SEM.
SQL Redo Info Processing Module: the processing module that the basic statement excavating is out processed, is called for short SPM.
SQL Statement Applying Module: the application module that the SQL statement of handling well is applied, is called for short SAM.
Message-oriented middleware: be that a kind of framework is in the basic software of system bottom (between application software and operating system).
The concrete introduction about message-oriented middleware below:
Message-oriented middleware (Message Oriented Middleware is called for short MOM), its major function is the complicacy of shielding bottom-layer network data transmission, realizes high performance transmitting; The data consistency when function by transaction management guarantees many message operation cooperations; For distributed environment, MOM can also carry out the linear expansion of performance flexibly and form cluster.These advantages are also to select in the present invention MOM as the reason of bottom transmission platform.In MOM, there is following topmost object:
Message a: encapsulation of data, consists of attribute head (HEAD) and data volume (BODY).Can analogize to the record in database table.
Queue: the container of message, is equivalent to the table of lane database.
The producer/message person: be responsible for respectively the application of sending and receiving message in the point-to-point application model in MOM system.
Below in conjunction with Fig. 2, message based large-scale transactional applications system according to an embodiment of the invention is described.
At infosystem layer, comprise OLTP data source (the first data source) 110, OLAP data source (the second data source) 112 and message-oriented middleware (MOM) 108, select MOM as bottom transmission platform.System also comprises incremental data extraction module 102, data conversion module 104 and market demand module 106.Incremental data extraction module 102 is responsible for extracting incremental data operation in OLTP data source, then after fundamental analysis, carry out classifying packing and become affairs, incremental data extraction module 102 also checks whether the affairs that are packaged into belong to large-scale affairs, if large-scale affairs, also large-scale affairs are split into many parts, SQL operative configuration in described every part is become to a message, these message are sent to data conversion module 104, data conversion module 104 is responsible for the operation in the message receiving to filter and change, then send the message after conversion to market demand module 106, market demand module 106 creates the local matter of OLAP, and create corresponding task and carry out the operation comprising in application message.Therefore, in the present embodiment, the data extraction module in Fig. 1 102 is improved, enabled that the affairs of super large are carried out to intelligence and split; Also data application module 106 has been carried out improving and having increased other middleware, for example message-oriented middleware, enables to improve the performance of remote transmission and processes the affairs across a plurality of message.
When MOM is introduced above, describe its high efficiency of transmission and the reliable characteristic of data, therefore in the present embodiment, will select MOM as bottom transmission platform.Select MOM to also have another one reason as transmission platform: after the SQL operation of super large affairs is split, every part all will be packaged into a message and process (because message only has and really enter after MOM, could real trigger network transmission or the treatment scheme of message recipient) by MOM affairs independently.How the message at the SQL operation place in these transmission can become a problem associated, here the group message mechanism of using MOM is carried out to association, the group message of MOM can be across affairs, each message groups can have a group sign (group id), each MOM message can make them interrelated by group sign is set, to facilitate reception and the processing of message.
In incremental data extraction module, the fractionation rule of formulating large-scale affairs by the parameter of system definition is:
When data are extracted, in order to split large-scale affairs by expection, what need in system, pre-define is large-scale affairs.So-called large-scale affairs refer to that comprised SQL operation amount has surpassed the affairs that maximum SQL that the single MOM message of system definition can comprise operates number.From definition, can find out if defined the maximum SQL operation number that single MOM message can comprise and also just define super large affairs, so the parameter defining in system has:
The maximum SQL operation number of MOM message: had this rule, incremental data extraction module 102 just can go to check that the affairs that will send are super large affairs according to this parameter when the affairs packing to excavating sends, if, according to the SQL operation maximum number of this parameter-definition, affairs are cut apart, then using each message as MOM affairs, (like this could be in whole affairs also in when transmission, the message of first coming in may be transferred to market demand module 106 and start to have applied) be submitted in MOM and goes.Better associated to facilitate SAM to process for the message in group, following attribute can be set at the head of MOM message:
Group ID: use the data source Transaction Identifier of DSN+OLTP of OLTP to carry out unique one group of message of determining;
Message number in group; And the sequence number of this message.
In market demand module 106, need do respective handling for many message transactions: during service data in receiving message processing messages, need to first check and whether organize message (checking that the affairs from OLTP data source are large-scale affairs), if find by be processed be large-scale affairs, after this Message Processing, can not submit at once, need to be according to the message number in group ID and group and the sequence number of current message, the next message reaching in group is then processed, until after all message in this group all finish dealing with, submit again the local matter in OLAP data source to.
Fig. 3 shows the block diagram of message based according to an embodiment of the invention large-scale transactional applications device.
As shown in Figure 3, message based according to an embodiment of the invention large-scale transactional applications device 300, be used for the incremental data action statement synchronous applications that is applied to the first data source in the second data source, comprise: resolving cell 302 (for example data extraction module in Fig. 2 102), from described the first data source, obtaining incremental data action statement, described incremental data action statement is packed by affairs, when judging the described affairs that are packaged into and being large-scale affairs, described large-scale affairs are decomposed into many parts, every part of described incremental data action statement comprising is configured to a message, described message is sent to applying unit, described applying unit 304 (for example market demand module 106 in Fig. 2), when receiving described message, create the local matter of described the second data source and create corresponding application task, described application task is applied to described the second data source based on described local matter by the incremental data action statement comprising in described message.
In technique scheme, preferably, described resolving cell 302 can also comprise: message constructing module 3022, the described large-scale affairs after many parts of decomposition are formed to a plurality of described message, and a plurality of described message forms a message groups; Module 3024 is set, the attribute of message described in each is set, the identity identification information of message groups, the message number in affiliated message groups, the sequence number of described message under described attribute comprises, the identity identification information of described message groups comprises the title of described the first data source and the local matter of described the second data source sign.
In technique scheme, preferably, described applying unit 304 can also comprise: the second judge module 3042, when the incremental data action statement comprising in current described message has been applied, judge whether described message belongs to message groups, if judge, belong to message groups, according to the attribute of current described message, continue to judge whether the message in described message groups all finishes dealing with, when judgment result is that while all finishing dealing with, submit the local matter of described the second data source to, when judgment result is that while all not finishing dealing with, continue to process the untreated message of residue.
In technique scheme, preferably, described resolving cell 302 can also comprise: the first judge module 3026, when the number of the incremental data action statement in described affairs surpasses the threshold value of the incremental data action statement that default described message can comprise, judging described affairs is large-scale affairs.
In technique scheme, preferably, described applying unit creates the local matter of described the second data source based on XA interface specification, and described the first data source is OLTP data source, and described the second data source is OLAP data source.
Technique scheme is by splitting the large-scale affairs of excavating from OLTP data source, can obviously improve the situation of incremental data application: by split can control message size, that avoids that system may occur can not send the situation that data are even collapsed because of low memory; Setting by systematic parameter (the maximum operand that message can comprise), can adapt to hardware environment intelligently; By splitting super large affairs, original serial application is changed into Parallel application, obviously improve the performance of system.
Fig. 4 shows the process flow diagram of message based according to an embodiment of the invention large-scale transactional applications method.
As shown in Figure 4, message based according to an embodiment of the invention large-scale transactional applications method, be used for the incremental data action statement synchronous applications that is applied to the first data source in the second data source, comprise: step 402, from described the first data source, obtain incremental data action statement, described incremental data action statement is packed by affairs, when judging the described affairs that are packaged into and be large-scale affairs, described large-scale affairs are decomposed into many parts, every part of described incremental data action statement comprising is configured to a message; Step 404, when receiving described message, create the local matter of described the second data source and create corresponding application task, described application task is applied to described the second data source based on described local matter by the incremental data action statement comprising in described message.
In technique scheme, preferably, described step 402 can also comprise: the described large-scale affairs after many parts of decomposition are formed to a plurality of described message, a plurality of described message forms a message groups, the attribute of message described in each is set, the identity identification information of message groups, the message number in affiliated message groups, the sequence number of described message under described attribute comprises, the identity identification information of described message groups comprises the title of described the first data source and the local matter of described the second data source sign.
In technique scheme, preferably, described step 404 can also comprise: when the incremental data action statement comprising in current described message has been applied, judge whether described message belongs to message groups, if judge, belong to message groups, according to the attribute of current described message, continue to judge whether the message in described message groups all finishes dealing with, when judgment result is that while all finishing dealing with, submit the local matter of described the second data source to, when judgment result is that while all not finishing dealing with, continue to process the untreated message of residue.
In technique scheme, preferably, described step 402 can also comprise: when the number of the incremental data action statement in described affairs surpasses the threshold value of the incremental data action statement that default described message can comprise, judging described affairs is large-scale affairs.
In technique scheme, preferably, the local matter of described the first data source creates and forms based on XA interface specification, and described the first data source is OLTP data source, and described the second data source is OLAP data source.
Technique scheme is by splitting the affairs of excavating from OLTP data source, can obviously improve incremental data applicable cases: by split can control message size, that avoids that system may occur can not send the situation that data are even collapsed because of low memory; Setting by systematic parameter (the maximum operand that message can comprise), can adapt to hardware environment intelligently; By splitting super large affairs, original serial application is changed into Parallel application, obviously improve the performance of system.
Fig. 5 shows the process flow diagram that extracts according to an embodiment of the invention incremental data action statement.
As shown in Figure 5, step 502, first carries out the initialization of related resource, as the connection of MOM, and the connection of data source etc.After completing, initialization will carry out a circular treatment flow process.
In above-mentioned steps 512, " affairs that circulation is sent completely " are movable, and (uncompleted affairs can be buffered in this locality, until excavate, after all operations of affairs, just can trigger transmission flow) concrete process of transmitting flow process as shown in Figure 6, Fig. 6 shows the process flow diagram of the affairs that send according to an embodiment of the invention single packing.
As shown in Figure 6, step 602, whether for the affairs of each packing, before transmission, can do is the inspection of complete transaction.
Step 608 is taken out a SQL operation from the set splitting.
Fig. 7 shows the process flow diagram of applying according to an embodiment of the invention large-scale affairs.
As shown in Figure 7, step 702, after related resource initialization completes, monitors and obtains system state.
Step 704, judges that system state, whether for stopping, if the determination result is YES, enters step 722, if the determination result is NO, enters step 708.
Step 722, once find that system requirements stops, discharging immediately relevant resource and finishes.
Step 708, waits for by blocking the message that comprises SQL operation.
Step 710, after receiving a message that comprises SQL operation, will create XA (based on the XA standard) local matter of an OLAP.
Step 712, then creates corresponding task and goes to apply the SQL operation (these SQL operations may be parts for OLTP ultra-large type affairs) comprising in this message.
Whether step 714, after the SQL operational applications comprising in this message completes, is a group message by this message of inspection.
Step 718, if not group message, will submit the XA local matter of OLAP data source to, and epicycle has operated.
Step 716, if find that the message of just having applied belongs to group message (this shows to apply ultra-large type affairs), according to the attribute of message, check all message in group whether all finish dealing with (number of group message, when the sequence number of the message of pre-treatment etc.).
Step 718, if processed the completing all of all message in discovery group will be submitted to or the XA local matter of rollback OLAP according to application result, epicycle has operated.
Step 720, if it is processed to also have message not have in discovery group, needs to go to receive other message in untreated group according to the attribute of group message, then after receiving, repeats application in front and check logic, until processed the completing all of all message in group.
More than be described with reference to the accompanying drawings according to technical scheme of the present invention, by the affairs of excavating from OLTP data source are split, can obviously improve the situation of incremental data application: by split can control message size, that avoids that system may occur can not send the situation of even collapsing because of low memory; By the setting of systematic parameter, can adapt to intelligently hardware environment.By splitting large-scale affairs, original serial application is changed into Parallel application, obviously improved the performance of system.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (6)
1. a message based large-scale transactional applications device, for by the incremental data action statement synchronous applications that is applied to the first data source in the second data source, it is characterized in that, comprising:
Resolving cell, from described the first data source, obtaining incremental data action statement, described incremental data action statement is packed by affairs, when judging described affairs and being large-scale affairs, described large-scale affairs are decomposed into many parts, every part of described incremental data action statement comprising is configured to a message, described message is sent to applying unit, described resolving cell comprises: the first judge module, when the number of the incremental data action statement in described affairs surpasses the threshold value of the incremental data action statement that default described message can comprise, judging described affairs is large-scale affairs,
Described applying unit, when receiving described message, based on XA interface specification, create the local matter of described the second data source and create corresponding application task, described application task is applied to described the second data source based on described local matter by the incremental data action statement comprising in described message;
Wherein, described the first data source is OLTP data source, and described the second data source is OLAP data source.
2. message based large-scale transactional applications device according to claim 1, is characterized in that, described resolving cell also comprises:
Message constructing module, forms a plurality of described message by the described large-scale affairs after many parts of decomposition, and a plurality of described message forms a message groups;
Module is set, the attribute of message described in each is set, the identity identification information of message groups, the message number in affiliated message groups, the sequence number of described message under described attribute comprises, the identity identification information of described message groups comprises the title of described the first data source and the local matter of described the second data source sign.
3. message based large-scale transactional applications device according to claim 1, is characterized in that, described applying unit also comprises:
The second judge module, when the incremental data action statement comprising in current described message has been applied, judge whether described message belongs to message groups, if judge, belong to message groups, according to the attribute of current described message, continue to judge whether the message in described message groups all finishes dealing with, when judgment result is that while all finishing dealing with, submit the local matter of described the second data source to, when judgment result is that while all not finishing dealing with, continue to process the untreated message of residue.
4. a message based large-scale transactional applications method, for by the incremental data action statement synchronous applications that is applied to the first data source in the second data source, it is characterized in that, comprising:
Step 202, from described the first data source, obtain incremental data action statement, described incremental data action statement is packed by affairs, when judging described affairs and be large-scale affairs, described large-scale affairs are decomposed into many parts, every part of described incremental data action statement comprising is configured to a message, when the number of the incremental data action statement in described affairs surpasses the threshold value of the incremental data action statement that default described message can comprise, judging described affairs is large-scale affairs;
Step 204, when receiving described message, based on XA interface specification, create the local matter of described the second data source and create corresponding application task, described application task is applied to described the second data source based on described local matter by the incremental data action statement comprising in described message;
Wherein, described the first data source is OLTP data source, and described the second data source is OLAP data source.
5. message based large-scale transactional applications method according to claim 4, it is characterized in that, described step 202 also comprises: the described large-scale affairs after many parts of decomposition are formed to a plurality of described message, a plurality of described message forms a message groups, the attribute of message described in each is set, the identity identification information of message groups, the message number in affiliated message groups, the sequence number of described message under described attribute comprises, the identity identification information of described message groups comprises the title of described the first data source and the local matter of described the second data source sign.
6. message based large-scale transactional applications method according to claim 5, it is characterized in that, described step 204 also comprises: when the incremental data action statement comprising in current described message has been applied, judge whether described message belongs to message groups, if judge, belong to message groups, according to the attribute of current described message, continue to judge whether the message in described message groups all finishes dealing with, when judgment result is that while all finishing dealing with, submit the local matter of described the second data source to, when judgment result is that while all not finishing dealing with, continue to process the untreated message of residue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110284525.9A CN102508834B (en) | 2011-09-22 | 2011-09-22 | Message-based event application device and message-based event application method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110284525.9A CN102508834B (en) | 2011-09-22 | 2011-09-22 | Message-based event application device and message-based event application method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102508834A CN102508834A (en) | 2012-06-20 |
| CN102508834B true CN102508834B (en) | 2014-05-07 |
Family
ID=46220924
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110284525.9A Expired - Fee Related CN102508834B (en) | 2011-09-22 | 2011-09-22 | Message-based event application device and message-based event application method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102508834B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104184602B (en) * | 2013-05-23 | 2019-01-04 | 中兴通讯股份有限公司 | Incremental data synchronization method, Element management system and Network Management System |
| CN103354560B (en) * | 2013-06-28 | 2017-05-17 | 广州市动景计算机科技有限公司 | Method and system for data decomposition and synchronization |
| CN105354087B (en) * | 2015-12-14 | 2018-10-23 | 交控科技股份有限公司 | The virtual processing method and processing device of real-time task |
| CN106897123B (en) * | 2015-12-21 | 2021-07-16 | 阿里巴巴集团控股有限公司 | Database operation method and device |
| CN109388481B (en) * | 2018-09-21 | 2021-08-17 | 网易（杭州）网络有限公司 | Transaction information transmission method, system, device, computing equipment and medium |
| CN110263105B (en) * | 2019-05-21 | 2021-09-10 | 北京百度网讯科技有限公司 | Query processing method, query processing system, server, and computer-readable medium |
| CN112039753B (en) * | 2020-08-31 | 2024-08-20 | 北京百度网讯科技有限公司 | Message packaging and receiving method, device, electronic equipment and medium |
| CN112883045B (en) * | 2021-03-31 | 2024-05-17 | 中国工商银行股份有限公司 | Database transaction splitting execution method and device |
| CN113918650A (en) * | 2021-09-30 | 2022-01-11 | 浪潮云信息技术股份公司 | Data synchronization method from OLTP to OLAP |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101784988A (en) * | 2007-08-30 | 2010-07-21 | 国际商业机器公司 | Transaction aggregation to increase transaction processing throughput |
| CN102024052A (en) * | 2010-12-17 | 2011-04-20 | 北京握奇数据系统有限公司 | Method and device for realizing transaction of data storage system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7490083B2 (en) * | 2004-02-27 | 2009-02-10 | International Business Machines Corporation | Parallel apply processing in data replication with preservation of transaction integrity and source ordering of dependent updates |
-
2011
- 2011-09-22 CN CN201110284525.9A patent/CN102508834B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101784988A (en) * | 2007-08-30 | 2010-07-21 | 国际商业机器公司 | Transaction aggregation to increase transaction processing throughput |
| CN102024052A (en) * | 2010-12-17 | 2011-04-20 | 北京握奇数据系统有限公司 | Method and device for realizing transaction of data storage system |
Non-Patent Citations (2)
| Title |
|---|
| JMS在解决企业信息系统长事务中的应用;闻敬谦等;《计算机与现代化》;20031031;第60-62,65页 * |
| 闻敬谦等.JMS在解决企业信息系统长事务中的应用.《计算机与现代化》.2003,第60-62,65页. |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102508834A (en) | 2012-06-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102508834B (en) | Message-based event application device and message-based event application method | |
| CN102306200B (en) | Device and method for concurrently applying incremental data manipulation statements | |
| EP3120261B1 (en) | Dependency-aware transaction batching for data replication | |
| AU648263B2 (en) | Portable and dynamic distributed applications architecture | |
| CN104317843B (en) | A kind of data syn-chronization ETL system | |
| CN107220142B (en) | Method and device for executing data recovery operation | |
| CN101807210A (en) | Database data synchronic method, system and device | |
| CN102542007A (en) | Method and system for synchronization of relational databases | |
| CN102495853B (en) | Aspect-oriented cloud storage engine construction method | |
| JP2010539616A (en) | Non-overlapping ETL-less system and method for reporting OLTP data | |
| CN111061788A (en) | Multi-source heterogeneous data conversion integration system based on cloud architecture and implementation method thereof | |
| CN111176627A (en) | Device and method for separating front end from back end based on micro-service | |
| CN102722368B (en) | Plug-in software designing method based on document tree and message pump | |
| US7720884B1 (en) | Automatic generation of routines and/or schemas for database management | |
| CN105786950A (en) | Method and device for processing data | |
| CN104317957A (en) | Open platform and system for processing reports and report processing method | |
| CN113934792A (en) | Processing method and device of distributed database, network equipment and storage medium | |
| CN111177244A (en) | Data association analysis method for multiple heterogeneous databases | |
| CN116414801A (en) | Data migration method, device, computer equipment and storage medium | |
| CN111651758B (en) | Method for auditing result set of relational database of operation and maintenance auditing system | |
| US20180121532A1 (en) | Data table partitioning management method and apparatus | |
| US7475088B2 (en) | Systems and methods of providing data from a data source to a data sink | |
| CN112685485B (en) | Data acquisition method, device, electronic equipment, storage medium and program product | |
| CN103186413A (en) | Realization architecture and realization method for mailbox service | |
| CN112486646A (en) | Job task processing method and device of computing system, storage medium and processor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C56 | Change in the name or address of the patentee |
Owner name: YONYOU NETWORK TECHNOLOGY CO., LTD. Free format text: FORMER NAME: UFIDA SOFTWARE CO., LTD. |
|
| CP03 | Change of name, title or address |
Address after: 100094 Haidian District North Road, Beijing, No. 68 Patentee after: Yonyou Network Technology Co., Ltd. Address before: 100094 Beijing city Haidian District North Road No. 68, UFIDA Software Park Patentee before: UFIDA Software Co., Ltd. |
|
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20140507 Termination date: 20200922 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |