JP2006031631A - Data base processing system and method, and program for data base processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a data base retrieval system capable of speeding up processing on the whole, for all data base processing systems. <P>SOLUTION: The data base processing system is provided with a grouping means 11 for grouping transaction data constituted of a set of a series of processing for a master data base by a predetermined rule, a master extraction means 13 for extracting a sub data base including data which can process the transaction data from the master data base on the basis of the transaction data in a grouped specific group and a transaction processing means 14 for processing the transaction data grouped for the sub data base. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a database processing system, and more particularly to a database processing system such as a batch system or an online system that obtains an output result by performing a search process such as database collation on a large amount of transaction data. The present invention also relates to a database processing method and a database processing program.

  Conventionally, in database processing, technological development for speeding up the processing has been advanced. For example, the following two technologies existed.

  First, the technology called “cache technology” stores the data on the database once accessed in the memory for a certain period of time, and then accesses and processes the data on the memory without accessing the disk when the same data is accessed. It is a technology that speeds up. However, in such a cache technology, generally, the access target to be copied is a physical block, the key distribution of transaction data is distributed over a wide range, and if the distribution is large compared to the cache capacity, the cache hit rate is The problem of being reduced arises.

  In addition, the technology called “pre-sorting technology” is equipped with business logic so that DB data is accessed only at the timing when the key value of the transaction data breaks in advance in the order of the key for database access and the key value of the transaction data breaks. It is a technology that speeds up. However, such a pre-sort technique has a problem that the processing order of transaction data is meaningful and cannot be applied to processing that is not suitable for sorting, and the key distribution in transaction data is wide and the continuity rate of key values is high. If it is low, sorting causes a problem that processing time increases as a result of the entire system, which may be counterproductive. Furthermore, there are problems that only batch processing that can be sorted in advance is applied and lacks versatility, and that break processing is required in business logic.

  Patent Document 1 below discloses a system that creates a specific database having a business purpose required by the user based on the instruction information of the user and performs processing on the database. Yes. However, with such a configuration, if the user changes, the target database also changes, so that there is a problem that effective processing cannot be expected for a newly created database.

JP 2002-366401 A

  As described above, in the above-described conventional database search system, it takes time to process each case when the number of base data of the search target data is large, and the entire processing time is required for a large number of transactions. However, since this method works universally on the system, there are conditions and limitations in the effective application area. there were. That is, it is impossible to increase the processing speed for all database processing.

  Considering business characteristics, it may be possible to improve the efficiency significantly in a specific environment. However, such characteristics generally require a business-specific approach, and depending on the characteristics. In other words, there are many individual correspondences such as judging the degree of effect after performing the analysis, and it is impossible to speed up all the database processing.

  Therefore, the present invention has an object to provide a database processing system that can improve the disadvantages of the above-described conventional example, and in particular, can increase the overall processing speed for any database processing system. To do.

Therefore, as one form of the database processing system according to the present invention,
Grouping means for grouping transaction data consisting of a set of processes for the master database according to a predetermined rule;
A master extraction means for extracting a sub-database including data capable of processing the transaction data from the master database based on the transaction data in the grouped specific group;
Transaction processing means for processing transaction data grouped in the sub-database;
It is characterized by having.

  The grouping means may be configured to perform grouping according to processing characteristics for a master database that is a processing target of transaction data. Specifically, the grouping unit may perform grouping based on data related to a search key for a master database existing in transaction data.

In addition to the above configuration,
The grouping means sets a plurality of predetermined rules to be used for grouping, and uses all or a part of the plurality of rules so that the number of groups falls within a predetermined numerical range It may be configured to perform.

  With the above configuration, rules for grouping according to transaction data feature values, such as master assigned key values and processing time zones, are set in advance. Grouped. At this time, grouping is performed using a plurality of rules so as to be divided into a predetermined number of groups. Subsequently, based on characteristics common to transaction data in each group, a temporary sub-database is extracted and generated from the master database. In particular, the sub-database is extracted to include data that can process transaction data in a predetermined group. For example, a sub-database including the search key used for grouping is generated. Thereafter, the transaction data of the group is processed for the sub-database corresponding to each group.

  Therefore, it is highly possible that the sub-database contains data that can execute transaction data processing, so the processing accuracy is maintained and the number of hosts is small, so the processing time is reduced. And the load on the computer that performs transaction processing can be reduced.

Further, as another embodiment of the present invention, a database processing program is provided, and the program is
A computer that processes transaction data stored in a predetermined storage device for a master database, which is a data group to be processed, stored in a predetermined storage device,
Grouping means for grouping transaction data consisting of a set of processes for the master database according to a predetermined rule;
A master extraction means for extracting a sub-database including data capable of processing the transaction data from the master database based on the transaction data in the grouped specific group;
And a transaction processing means for processing the transaction data grouped with respect to the sub-database.

Furthermore, as another aspect of the present invention, a database processing method is provided,
A database processing method for processing transaction data stored in a predetermined storage device with respect to a master database that is a processing target data group stored in a predetermined storage device using a computer,
A grouping step of grouping transaction data consisting of a set of processes for the master database according to a predetermined rule;
A master extraction step for extracting a sub-database including data capable of processing the transaction data from the master database based on the transaction data in the grouped specific group;
A transaction processing step for processing transaction data grouped in the sub-database.

  Even the database processing program and the database processing method configured as described above operate in the same manner as the above-described database processing system, and thus the above-described object can be achieved.

  Since the present invention is configured and functions as described above, according to this, the accuracy of database processing is maintained for a system that performs any transaction processing, and the number of bases to be processed is also small. The present invention has an unprecedented excellent effect that the processing time can be shortened and the addition of a computer for performing transaction processing can be reduced.

  The database processing system according to the present invention is, for example, a batch system or an online system that obtains an output result by performing a search process such as database collation. Then, the transaction data is grouped by a certain index value, and from the regular database (master database) to another single or multiple temporary databases (sub-databases) according to the distribution characteristics of the transaction data for each group It is characterized by being automatically generated and obtaining the same or similar result as the original process using the generated temporary database group. As a result, processing results can be obtained in a short processing time for the entire system, and the speed of database processing can be increased while suppressing the burden on the system.

  And by doing as mentioned above, the effect of speeding up the process beyond the limit by the conventional general-purpose method can be acquired, without putting a hand in the business program used as a core. In particular, the present invention can be applied to both batch processing and online inquiry processing, and throughput can be improved. This leads to an improvement in average response value in online processing and an improvement in Elax in batch processing under an environment where a certain load is applied. Since batch processing is easier to improve the analysis accuracy of business characteristics than online processing, the expectation of the effect is large. Hereinafter, the configuration of the system will be described in detail with reference to examples.

  A first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a functional block diagram showing a configuration of a computer that becomes a database processing system. FIG. 2 is an explanatory diagram showing a state of database processing. FIG. 3 is an explanatory diagram showing the structure of the database. FIG. 4 is an explanatory diagram showing the operation of the system. FIG. 5 is an explanatory diagram showing a part of the operation. 6 to 7 are explanatory diagrams for illustrating the effects of the present invention.

  As described above, the database processing system described in the present embodiment is, for example, a batch system or an online system that obtains an output result by performing a search process such as database collation. However, the present invention is not limited to such a system, and any system may be used as long as it is a system that processes a plurality of predetermined transaction data for a database.

  Here, the search processing based on transaction data is, for example, indicated by “Processing A” as shown in FIG. 2, the database has a simple configuration (product master) like a master table, and an input code value ( Assume a process of assigning a name (product name) from a master using a product code) as a key. Then, the process assumed for the “process A” is executed on the sub-database as described later in this embodiment. In addition to the “product code”, other related information exists in the transaction data item.

<Configuration>
The database processing system according to the present invention is constructed by a computer 1 as shown in FIG. The computer 1 is composed of a general computer, and includes a CPU 10 that is an arithmetic processing unit, a hard disk 20 that is a storage device, and a memory 30 such as a RAM. The CPU 10 incorporates a database processing program for realizing the database processing that is a feature of the present invention, whereby the processing units 11 to 15 described below are constructed. The hard disk 20 includes storage units 21 to 23 that store preset data. The memory 30 stores storage units that store data generated by the operations of the processing units 11 to 15. Portions 31 to 33 are formed. The above configuration will be described in detail below.

  First, a master database storage unit 23 that stores a master database that is a database to be processed is formed in the hard disk 20. In addition, a transaction data storage unit 21 for storing transaction data including a series of processes for the master database is formed. Further, a grouping rule data storage unit 22 is formed for storing grouping rule data representing a classification criterion for grouping the transaction data stored in the transaction data storage unit 21. Since the grouping rule data is data used when grouping by the dispatch processing unit 11 described later, it will be described in detail when the dispatch processing unit 11 is described. The data stored in the hard disk 20 may be stored in a storage device of another computer connected to the computer 1.

  Next, as described above, the CPU 10 incorporates a predetermined program so that transaction data consisting of a series of processes for the master database is grouped according to a predetermined rule. Grouping means), a transaction analysis processing unit 12 for analyzing the grouped transaction data, and data capable of processing the transaction data from the master database based on the transaction data in the specific grouped group. A master extraction processing unit 13 (master extraction unit) that extracts a sub-database including the transaction processing unit 14 (transaction processing unit) that processes transaction data grouped in the sub-database. , An output processing unit 15 for outputting the processing result, is constructed.

  The dispatch processing unit 11 groups transaction data with a certain index value (hereinafter referred to as a grouping function). At this time, data corresponding to the processing characteristics of the transaction data with respect to the master database is selected as one of the arguments of the grouping function. In particular, if transaction data has a correlation with the distribution characteristics of the database search key, the data having the correlation is selected as one of the arguments of the function. That is, an instruction for selecting such a grouping function is stored in the grouping rule data storage unit 22 or incorporated in the dispatch processing unit 11 in advance, and dispatching is performed based on the grouping function. Grouping processing is performed in the processing unit 11.

  And, specifically, as the property of the grouping function, it is desirable to reflect the distribution characteristics of the key value assigned by the master, so for example, it is moderately aggregated from the transaction data, such as “branch code”, In addition, a plurality of variables correlated with the distribution of “product code” are selected as arguments. In batch processing, it is desirable that these variables have a property of appearing as moderately as possible in the order of data processing.

  Even if there is no variable correlated with the search key as described above, for example, the data corresponding to the processing characteristics of the transaction data shown below for the master database is the argument of the grouping function. Selected as one. For example, in the online system, by processing time zone in the transaction data, classification by the first two digits of the access terminal number, 10,000 items in order of the reception number, etc. In the batch processing system, input the connection source, record number, etc. Indicators such as 10,000 records for each data type and data order are candidates for variables (arguments). In addition, adopting a daily variable type condition as a variable, such as using “the first two digits of the product code” as a variable or using statistical information of the processing result of the previous day, can be considered as an application example. This is because selecting these variables is generally expected to correlate with the key value distribution as compared to selecting a random function.

  The grouping function to be used is set by the system user specifying it according to the characteristics of the database and the characteristics of transaction processing and inputting them to the computer 1.

  In batch processing, when the variable P = (P1, P2,...) Is moderately continuous in the data processing order, the value of Go (P) changes instead of the original grouping function Go. A function G that counts up each time (P is called a break) is configured as a new grouping function. This is because batch processing tends to be scheduled on a group-by-group basis, so that data ordering is necessary and adverse effects caused by high load on grouping processing are used. .

  In the following, T is a set of all transaction data, M is a set of the entire master database, Φ is a master assigning function (function that assigns data in M from T), G is a grouping function, and {1 is a range of G , 2,..., N}.

  The dispatch processing unit 11 stores the grouped transaction data in the grouped transaction data storage unit 31 formed in the memory 31. At this time, the dispatch processing unit 11 causes the argument or return value to be appropriately aggregated, the function not adopted, or the number of divisions is too small when the number of divisions is too large so that the number of divisions for grouping becomes an appropriate distribution. At times, adjust the number of divisions by adding arguments or adding another function. That is, it has a function of automatically performing grouping so that the number of groups falls within a predetermined numerical range by increasing or decreasing the rules used when grouping. This is because the sub-database is reduced to a level that can effectively use the system cache, as will be described later.

  Next, the transaction analysis processing unit 12 will be described. The transaction analysis processing unit 12 performs a scan analysis on the grouped transaction data (data in reference numeral 31), and identifies what is used as a key value at the time of database allocation. The data used as the key value is stored in the group analysis data storage unit 32 in the memory 30 as group analysis data for each group.

  Next, the master extraction processing unit 13 will be described. The master extraction processing unit 13 creates a temporary database (hereinafter referred to as a sub-database) corresponding to the distribution characteristics of transaction data in the group for each group of transaction data grouped in the original search target database. Generate automatically from a master database. At this time, based on the key value stored in the group analysis data storage unit 32 analyzed by the transaction analysis processing unit 12, the corresponding data can be extracted from the master database. The sub-database is a subset of the master database, and includes data that can process transaction data in the group.

  Specifically, for example, a set of transactions belonging to group n is Tn, and a subset of the master database (referred to as a sub-database) referenced by the key of Tn is Mn. In batch processing, Mn is determined by conducting a total survey before starting processing of a group as a schedule unit. Further, the online processing is composed of two stages of initial prediction and addition each time. For this reason, a configuration is adopted in which it is ensured that search target data based on transaction data to be processed exists in the sub-database. As a simple method, there is a method of executing a sub database inquiry process for each transaction data. However, since the sub database search process occurs twice as a result, the effect of the present invention is weakened. To suppress this, usually select another index value that is guaranteed to already exist under different conditions in the transaction data, and only the data that deviates from that index is queried in the sub-database search process. Configure.

  The set of Mn configured as described above is configured as a minimum necessary so as not to be mis-hit in the search process. Therefore, it is possible that a small amount of extra data is included in addition to this. In particular, in online processing, it is difficult to generate Φ (Tn) as a mapping of Φ of Tn in advance. Therefore, M′n as a certain degree of prediction is configured in advance, and Mn is expressed as Φ ( The configuration is performed as a union of Tn) and M′n. Then, generally, as shown in FIG. 3, each set of sub-databases has an overlapping element, and even if all sub-databases are combined, the original master database is not always obtained.

  It should be noted that at the generation timing of the sub-database, it is most efficient in batch processing to perform batch processing by checking all transaction data in advance for each group. In online processing, as described above, it is created in advance by forecasting under certain assumptions for grouping (this pre-creation processing does not necessarily have to be performed), and a method for adding missing ones each time is used. Do. In this case, a separate process for always determining whether it is sufficient or not is necessary.

  The sub-database created in this way is stored in the sub-master database storage unit 33 in the memory 31, and the transaction processing unit 14 applies the database to each grouped transaction instead of the master database. A processing result equivalent to the original purpose can be obtained without changing the allocation processing.

<Operation>
Next, the operation of the system configured as described above will be described with reference to FIGS. FIG. 4 is an explanatory diagram showing a state of processing in the CPU 10. FIG. 5 is an explanatory diagram for explaining processing when there are a plurality of groups.

  First, the dispatch processing unit 11 groups transaction data using a grouping function. That is, the transaction data T (21) is processed to be decomposed into T1 to TN. Then, a group n (31) is generated. As shown in FIG. 5, a plurality of groups (group 1 (31a), group 2 (31b)) may be generated by this dispatch process, and the number of generated groups is not limited.

  Next, analysis of the grouped transaction data is performed by the transaction analysis processing unit 12, and the analysis data 32 is stored and held. That is, processing for analyzing information for creating Mn from Tn is performed. Then, using this analysis data 32, the master extraction processing unit 13 performs master extraction processing from the master database 23 to generate a sub-database 33. That is, processing for creating M1 to MN from M is performed. At this time, when a plurality of groups are generated, a plurality of sub-databases (sub-DB1 (33a), sub-DB2 (33b)) are generated correspondingly (see FIG. 5).

  Thereafter, the transaction data of each group is processed by the transaction processing unit 14 on the sub-database 33 (33a, 33b) corresponding to the group. Thereby, output data 40 representing the output result for the transaction is generated.

  By doing so, it is highly possible that the sub-database contains data that can execute transaction data processing, so the accuracy of the processing is maintained and the number of hosts is also small. The processing time can be shortened and the addition of a computer that performs transaction processing can be reduced.

The above effect will be further described with reference to FIGS. FIG. 6 shows the relationship between transaction data and key value distribution. As the time for processing one transaction data, the processing time of the search processing has a correlation with the size of the base of the search target master. If the scale order of this correlation is expressed on the vertical axis, the processing time corresponding to the processing A of the entire transaction is proportional to the area of the graph of FIG. When these are compared, the area is as follows.
When grouping is not performed: Area = T × M
When grouped and divided: Area = Σ _{n = 1 to N} (Tn × Mn)

  As is clear from the above, the grouping / dividing is obviously smaller. In the present invention, since "transaction analysis" processing and "master extraction" processing are required, some overhead occurs, but usually these overhead processing is proportional to the base size of transaction data. However, since the correlation with the size of the master matrix tends to be low, the advantage of application of the present invention increases as the master matrix increases. This is shown in FIG. (A) of this figure shows a graph showing the processing time of transaction processing when the present invention is not applied, and (B) shows the total processing time including transaction processing when the present invention is applied. The graph is shown. Note that (B) is a processing time obtained by combining the processing time (B1) of only the transaction processing when the present invention is applied and the overhead processing time (B2) when the present invention is applied ((B ) = (B1) + (B2)). Referring to this figure, the processing speed can be increased when the number of data in the search target database exceeds the value a.

  Here, in addition to the above-described configuration, the processing speed can be further increased by using the above-described conventional cache technology together. In addition, although random access of the original database occurs during fetching into the cache, in the present invention, in some cases, this portion can be suppressed with a smaller load than in the case of the cache technology in the application of batch processing. This is because although it depends on the implementation of the database processing system, sequential access such as matching processing is possible in the batch processing.

  The present invention has industrial applicability because it can be processed in a database processing system for processing a large amount of transaction data to increase the processing speed of the transaction data.

It is a functional block diagram which shows the structure of the computer which operate | moves as a database processing system which is this invention. It is explanatory drawing which shows an example of general transaction processing. It is explanatory drawing which shows the relationship between a master database and a sub database. It is explanatory drawing which shows operation | movement of a database processing system. It is explanatory drawing which shows a part of operation | movement of a database processing system. It is a figure which shows the relationship between transaction distribution and key value distribution. It is the figure which compared the processing time of a prior art example and this invention.

Explanation of symbols

1 Computer (database processing system)
10 CPU
11 Dispatch processing part (grouping means)
12 Transaction analysis processing unit 13 Master extraction processing unit (master extraction means)
14 Transaction processing unit (transaction processing means)
15 Output processing unit 20 Hard disk 21 Transaction data storage unit (transaction data)
22 Grouping rule data storage unit 23 Master database storage unit (master database)
30 Memory 31 Grouped Transaction Data Storage Unit 32 Group Analysis Data Storage Unit 33 Sub Database Storage Unit (Sub Database)

Claims (6)

Grouping means for grouping transaction data consisting of a set of processes for the master database according to a predetermined rule;
Master extraction means for extracting a sub-database including data capable of processing the transaction data from the master database based on transaction data in a specific grouped group;
Transaction processing means for processing transaction data grouped with respect to the sub-database;
A database processing system comprising:   2. The database processing system according to claim 1, wherein the grouping means performs grouping according to processing characteristics for the master database to be processed of the transaction data.   The database processing system according to claim 1, wherein the grouping means performs grouping based on data related to a search key for the master database existing in the transaction data.   The grouping means sets a plurality of predetermined rules to be used for grouping, and uses all or a part of the plurality of rules so that the number of groups falls within a predetermined numerical range. 4. The database processing system according to claim 1, wherein the database processing system is configured. A computer that processes transaction data stored in a predetermined storage device for a master database, which is a data group to be processed, stored in a predetermined storage device,
Grouping means for grouping transaction data consisting of a set of processes for the master database according to a predetermined rule;
Master extraction means for extracting a sub-database including data capable of processing the transaction data from the master database based on transaction data in a specific grouped group;
Transaction processing means for processing transaction data grouped with respect to the sub-database;
A database processing program for realizing A database processing method for processing transaction data stored in a predetermined storage device with respect to a master database that is a processing target data group stored in a predetermined storage device using a computer,
A grouping step of grouping transaction data consisting of a set of processes for the master database according to a predetermined rule;
A master extraction step of extracting a sub-database including data capable of processing the transaction data from the master database based on transaction data in a specific grouped group;
A transaction processing step for processing transaction data grouped with respect to the sub-database;
A database processing method characterized by comprising:

Images