WO2013118287A1 - Data processing method, data processing program and computer system - Google Patents

Description

Data processing method, data processing program, and computer system

The present invention relates to a data processing method, and more particularly, to a data processing method for processing data input to a job.

When a general-purpose computer such as a mainframe processes a large number of files using a batch job, the conventional general-purpose computer divides the records contained in the file to be processed into multiple files, and multiple files are processed in parallel by the batch job Processing efficiency has been improved by processing.
As a file division method when batch jobs are executed in parallel, a process of distributing each record identified by a key value into a plurality of files according to a key range determined in advance by the number of file divisions is used.

Conventionally, since a batch job is distributed and executed in parallel, a file key range dividing method for distributing records in which the key values of records are in the same range (key range) to the same file has been disclosed (for example, cited reference 1). reference).

Also, as a method of distributing records in the database, the number of records is counted when the records are saved, the average value of the key range is obtained, and the saved records in the key range according to the obtained average value Is disclosed (for example, see cited document 2).

JP 2007-86951 A JP 11-345157 A

As described above, a plurality of records included in a file (input data) input to a job are divided into a plurality of files (divided data) according to a plurality of key ranges, and a plurality of computers process the divided data. In many cases, batch processing for distributed processing of jobs is conventionally used. The problems in batch processing are shown below.

When the number of records in each key range varies greatly, the job end time in each computer becomes uneven, and as a result, the entire batch processing time becomes longer. Further, when a failure occurs in any of the computers in a state where the size of any one of the divided data (the number of records included in the divided data) is large, the time required to re-execute the job becomes long.

Furthermore, when the size of each divided data is small, the overhead for starting and ending the job increases, throughput decreases, and batch processing time increases. As described above, in order to shorten the batch processing time, it is necessary to keep the scale of the divided data within a certain range.

However, even if the optimal key range is determined so that the scale is within a certain range at a certain point in time, if the job is executed periodically over a long period of time, the size of the divided data or the key of each record in the divided data It is necessary to change the key range so that the distribution of values fluctuates and becomes optimal again.

Also, since the batch processing time includes processing for dividing input data and determining the optimal key range, it is necessary to shorten these processing times.

In the method disclosed in the cited document 1, the key range is determined with reference to the key value of the record. For this reason, the program disclosed in the cited document 1 needs to refer to the record of the entire input data to divide the record in order to determine the optimum key range so that the scale of each divided data falls within a certain range. .

In addition, the program disclosed in the cited document 1 needs to read all records included in the input data after the key range is determined, and distribute the records. As a result, the program disclosed in the cited document 1 reads out all records in a total of two processes, ie, a process for determining a key range and a process for distributing input data. Takes time.

Since the apparatus disclosed in the cited document 2 determines the key range as data is saved, the record is not read again in the process of determining the key range. However, the process in the cited document 2 includes a premise that there is a data saving process. For this reason, when the apparatus disclosed in the cited document 2 performs batch processing for processing input data transferred from another system, for example, in order to accurately determine the optimum key range, all the records are set to 2 It is necessary to read it once. As a result, it takes time to divide the input data.

Also, the method of the cited document 2 determines whether or not the key range needs to be divided for all the key ranges, and obtains the number of records and the dividing points, which causes a processing overhead for checking.

In the present invention, it is an object to keep each divided data input to a job at an appropriate scale, and further to reduce a key range determination processing time and an input data division processing time in order to keep the appropriate scale.

According to a representative aspect of the present invention, there is provided a data processing method in a computer system that divides input data including a plurality of records, and processes each of the divided input data by each of a plurality of jobs. The system includes a processor and a memory, and each of the plurality of records is assigned a key value, and the key value is classified into one of a key range defined by a minimum value and a maximum value, and the method includes: When the processor processes the records divided according to the key ranges by the plurality of jobs, the number of processed records and the key value assigned to the processed records are classified. A history indicating a key range is acquired, and the processor is configured to input the input data based on the acquired plurality of histories. Determining a pattern of change in the number of records included in the data, the processor, according to the pattern of the determined change, determines the key range of changing at least one of the minimum value and the maximum value.

According to a typical embodiment of the present invention, it is possible to keep the divided data input to the job at an appropriate scale.

It is a block diagram which shows the hardware constitutions of the computer system of the 1st Embodiment of this invention. It is a block diagram which shows the process which produces | generates the divided data of the 1st Embodiment of this invention. It is explanatory drawing which shows the division | segmentation log | history of the 1st Embodiment of this invention. It is explanatory drawing which shows the key range management table of the 1st Embodiment of this invention. It is explanatory drawing which shows the increase / decrease type value of the 1st Embodiment of this invention. It is explanatory drawing which shows the parameter management table of the 1st Embodiment of this invention. It is explanatory drawing which shows the 1st change pattern of distribution of the number of records in the input data of the 1st Embodiment of this invention. It is explanatory drawing which shows the 2nd change pattern of distribution of the number of records in the input data of the 1st Embodiment of this invention. It is explanatory drawing which shows the 3rd change pattern of distribution of the number of records in the input data of the 1st Embodiment of this invention. It is explanatory drawing which shows the 4th change pattern of distribution of the number of records in the input data of the 1st Embodiment of this invention. It is a flowchart which shows the process by the key range selection part of the 1st Embodiment of this invention. It is a flowchart which shows the history selection process by the key range selection part of the 1st Embodiment of this invention. It is a flowchart which shows the key range record number prediction process by the key range selection part of the 1st Embodiment of this invention. It is a flowchart which shows the key range selection process by the key range selection part of the 1st Embodiment of this invention. It is a flowchart which shows the key range reconstruction process by the key range selection part of the 1st Embodiment of this invention. It is a flowchart which shows the division | segmentation process of the input data by the data division part of the 1st Embodiment of this invention. It is a flowchart which shows the division | segmentation process of the key range by the data division part of the 2nd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each embodiment does not limit the scope of the claims, and all the features described in the embodiment are not necessarily essential to the solution means of the invention.

The computer system according to the present embodiment determines the record type variation type from the history of a plurality of divided data input to the job, and determines the divided data for changing the number of records according to the determination result. Then, according to the determination, the divided data is output in the execution of the job.

(First embodiment)

Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of a computer system 1 according to the first embodiment of this invention.

The computer system 1 includes a computer 10 and a storage device 20. The computer system 1 is a system for executing a plurality of jobs.

The storage device 20 is a storage device such as a hard disk driver. The storage device 20 includes input data 21, a plurality of divided data 22, and a plurality of divided histories 100.

The input data 21 includes a plurality of records. The divided data 22 is a plurality of files in which the divided records are stored after the records included in the input data 21 are divided.

The division history 100 includes information indicating a key range, the number of records, and the like included in the divided data 22 when the records are divided into the divided data 22 in the past.

Here, the record indicates one piece of data that is input to the program and processed by the program. The key range is a range of key values. The key value of this embodiment is a numerical value indicating a position for specifying a record.

The computer 10 uses a key to identify a record when processing the record. The record included in the input data 21 is output to each divided data 22 according to the key range including the key value of the record. In the present embodiment, one divided data 22 corresponds to one key range.

Job is an execution unit of a program that processes records. Jobs are often executed periodically, such as at a fixed time every day. In this case, the program executed in any job execution is the same, but the input data input to the job differs depending on the execution timing. Further, it is generally required that a job is always completed within a specified time.

When a job for processing each divided data 22 is executed in parallel and the number of records included in the divided data 22 is too large, other jobs following the job are processed until the divided data 22 having a large number of records is processed. , Wait without being executed. Also, if a failure occurs when the job is executed, the time for re-executing the job becomes longer, and the risk that the job cannot be completed within a specified time increases.

On the other hand, when the job for processing each divided data 22 is executed in parallel and the number of records included in the divided data 22 is too small, the ratio of the overhead time for starting the job to the time for processing the divided data 22 Increases and the overall job throughput decreases. For this reason, the number of records included in the divided data 22 is required to be within a certain range.

The computer 10 includes a main storage device 11, a processor 12, and an input / output I / F 13.

The processor 12 is an arithmetic device such as a CPU (Central Processing Unit). The processor 12 is a device that loads and executes an instruction code included in the main storage device 11.

The input / output I / F 13 is an interface for connecting to the storage device 20, and is a device for transferring the input data 21 and the divided data 22 from the storage device 20 to the computer 10.

The main storage device 11 includes instruction codes for the key range selection unit 1000 and the data division unit 1100. The main storage device 11 includes a record buffer 14, a key range management table 300, and a parameter management table 200.

The key range management table 300 holds the boundary value (minimum value and maximum value) of the key range of the past divided data 22 and the number of records. When a plurality of different jobs are executed in the computer system 1, the main storage device 11 holds the key range management table 300 for each job.

The parameter management table 200 holds parameters necessary for determination in the processing of the key range selection unit 1000 and the data division unit 1100. For example, the parameter management table 200 holds an upper limit value and a lower limit value of the number of records included in the divided data 22.

The record buffer 14 holds a record included in the input data 21 or a key value assigned to the record.

The key range selection unit 1000 has a function of predicting the number of records in the key range corresponding to each divided data 22 and the increase / decrease pattern of the number of records. In addition, the key range selection unit 1000 has a function of determining that the divided data 22 in which the number of records included does not fall within a certain range as a result of prediction corresponds to a key range that needs to be divided or integrated.

The data dividing unit 1100 has a function of outputting the divided data 22 when the input data 21 is inputted.

The computer 10 of this embodiment reads the divided data 22 and executes a job. However, the computer system 1 may have a plurality of computers, and a computer other than the computer 10 may read the divided data 22 and execute the job. . The computer (including the computer 10) included in the computer system 1 may include a plurality of processors, and may process the divided data 22 by executing a plurality of jobs in parallel.

Further, the computer system 1 may transmit the divided data 22 generated by the processing described later in the present embodiment to another computer system connected to the computer system 1 via a network. Then, another computer system may process the transmitted divided data 22 in parallel by a job.

The key range selection unit 1000 and the data division unit 1100 shown in FIG. 1 are implemented by a program. However, the computer 10 of the present embodiment includes a physical device such as an integrated circuit having the function of the key range selection unit 1000 or the data division unit 1100, so that the key range selection unit 1000 and the data division unit 1100 Functions may be implemented.

In addition, the key range selection unit 1000 and the data division unit 1100 may be implemented by a single program. In addition, a program for implementing each of the key range selection unit 1000 and the data division unit 1100 may be implemented by a plurality of programs such as a main program and a sub program.

Further, the key range management table 300 and the parameter management table 200 shown in FIG. 1 hold information by the table, but the key range management table 300 and the parameter management table 200 of the present embodiment can store information by any method such as CSV. It may be held. The division history 100 may also hold information by any method.

FIG. 2 is a block diagram illustrating processing for generating the divided data 22 according to the first embodiment of this invention.

The key range selection unit 1000 reads information such as the job execution date, the boundary value of each key range, and the number of records in each key range from the division history 100, and stores the information in the key range management table 300. Then, the key range selection unit 1000 predicts the increase / decrease pattern of the key range and the number of records when a future job is executed, and compares the predicted result with the parameter of the parameter management table 200, thereby dividing or Select the key range that needs to be merged.

In addition, when the division history 100 is not stored in the storage device 20 such as when the job has not been executed, it is determined that it is advantageous in terms of performance that the key range selection unit 1000 regenerates all the key ranges. In this case, the key range selection unit 1000 reads the input data 21 and generates a new key range. Then, information indicating the generated new key range is stored in the key range management table 300.

The data dividing unit 1100 refers to the key range boundary value from the key range management table 300 and reads the record from the input data 21. Then, a record in which the key value is included in the referenced boundary value is output to the divided data 22 corresponding to each key range.

The data dividing unit 1100 of the first embodiment temporarily stores a part of the input data 21 in the record buffer 14 in order to determine a new boundary value. The data dividing unit 1100 counts the number of records included in each divided data 22 in order to generate the divided history 100 each time a job is executed, and the key range corresponding to each divided data 22 and each key range. Are stored in the division history 100.

FIG. 3 is an explanatory diagram illustrating the division history 100 according to the first embodiment of this invention.

The division history 100 is a file output by the data dividing unit 1100 every time the data dividing unit 1100 is executed. Each file of the division history 100 includes areas 101 to 104 for storing values.

The area 101 stores an execution date and time when the data dividing unit 1100 is executed. That is, the execution date and time when the data dividing unit 1100 is executed by executing the job is stored. That is, the execution date and time indicated by the area 101 is the job execution date and time.

In the area 102, a job name indicating an identifier of a job for inputting the divided data 22 is stored. That is, when the data dividing unit 1100 is executed, a job name indicating a job designated by a user or an administrator is stored.

The area 103 stores the total amount of input data 21 input to the job. Information indicating the boundary value of the key range corresponding to each divided data 22 and the number of records of each divided data 22 is stored in the area 104.

FIG. 4 is an explanatory diagram illustrating the key range management table 300 according to the first embodiment of this invention.

The key range management table 300 is stored in the main storage device 11 corresponding to each job when a plurality of jobs are executed in the computer system 1.

The key range management table 300 holds an entry corresponding to each key range. Each entry includes a minimum value 301, a maximum value 302, a history 303, an increase / decrease type 304, a predicted number 305, a record real number 306, and a division point survey flag 307.

The minimum value 301 indicates the minimum value of the key range. The maximum value 302 indicates the maximum value of the key range.

The history 303 indicates the number of records in the key range included in the past division history 100. The history 303 includes an area indicating the number of records for each past division history 100.

The history 303 shown in FIG. 4 indicates the number of records in the key range included in the past three divided history 100 in the history 1 (303a), history 2 (303b), and history 3 (303c) areas.

Increase / decrease type 304 indicates an increase / decrease pattern of the number of records in the key range. The predicted number 305 indicates the predicted number of records in each key range when the job is executed in the future, which is obtained from the record number increase / decrease pattern and the history 303.

The record real number 306 indicates the number of records actually output to the divided data 22. The division point investigation flag 307 is an area that holds a value indicating “ON” for the key range that the key range selection unit 1000 has determined to be divided because the predicted number 305 exceeds a predetermined upper limit value. .

In the key range management table 300 shown in FIG. 4, the history 303 includes three histories of history 1 (303a) to history 3 (303c), but any number of histories may be included as long as two or more histories are included. . In addition, the increase / decrease type 304 of the present embodiment shown below indicates any one of the increase / decrease type 0, the increase / decrease type + 1, the increase / decrease type + 2, the increase / decrease type-1, and the increase / decrease type-2.

FIG. 5 is an explanatory diagram illustrating values of the increase / decrease type 304 according to the first embodiment of this invention.

The line graph 501 to line graph 505 shown in FIG. 5 show examples of the number of records in one key range. The horizontal axis shown in FIG. 5 indicates the execution date and time (indicated by history 1 to history 3 in FIG. 5) indicated by the past division history 100. The vertical axis indicates the number of records indicated by each division history 100.

5 corresponds to the history 1 (303a), the history 2 (303b), and the history 3 (303c) shown in FIG. The execution date and time of history 1 is the latest, and the execution date and time of history 3 is the oldest.

The number of records indicated by the line graph 501 to the line graph 505 is a relative value based on the number of records in the history 3.

The line graph 502 shows an increase / decrease pattern of increase / decrease type + 1. The increase / decrease type +1 includes a difference d23 (= the number of records in history 2−the number of records in history 3) and a difference d12 (= the number of records in history 1−the number of records in history 2) which are differences in the number of records between the respective histories. Both are positive values, and the increase / decrease pattern when the difference between the difference d23 and the difference d12 is within a predetermined range (for example, a range calculated from a later-described record number threshold 203 included in the parameter management table 200).

The line graph 504 shows an increase / decrease pattern of increase / decrease type-1. The increase / decrease type-1 is an increase / decrease pattern when both the difference d23 and the difference d12 are negative values and the difference between the difference d23 and the difference d12 is within a predetermined range.

The line graph 501 shows an increase / decrease pattern of increase / decrease type +2. The increase / decrease type +2 is an increase / decrease pattern when the difference d23 and the difference d12 are both positive values and the difference d12 is larger than the difference d23 beyond a predetermined range.

A line graph 505 shows an increase / decrease pattern of increase / decrease type-2. The increase / decrease type-2 is an increase / decrease pattern in which both the difference d23 and the difference d12 are negative values and the difference d12 is larger than the difference d23 beyond a predetermined range.

The line graph 503 shows an increase / decrease pattern of the increase / decrease type 0. The increase / decrease type 0 is an increase / decrease pattern when the difference d23 and the difference d12 are numbers within a predetermined range, or an increase / decrease pattern when the signs of the difference d23 and the difference d12 are different.

In this embodiment, the increase / decrease type +1 and the increase / decrease type −1 are increase / decrease patterns that increase or decrease substantially linearly. In addition, the increase / decrease type +2 and the increase / decrease type −1 are increase / decrease patterns that increase or decrease nonlinearly, such as an exponential function.

The key range selection unit 1000 according to the present embodiment determines the increase / decrease pattern of the number of records between the histories 303 as described above, and the job is executed in the future based on the determined increase / decrease pattern and a mathematical expression or rule stored in advance. It is possible to calculate the estimated number of records when the record is made.

When there are only two division histories 100, the key range selection unit 1000 of this embodiment may determine increase / decrease type + 1, increase / decrease type 0, or increase / decrease type-1 as the increase / decrease pattern.

FIG. 6 is an explanatory diagram illustrating the parameter management table 200 according to the first embodiment of this invention.

The parameter management table 200 holds parameters used in processing by the key range selection unit 1000 and the data division unit 1100. The parameters stored in the parameter management table 200 are values input to the computer system 1 in advance by a user or an administrator.

The parameter management table 200 includes a distribution variation type 201, a job name 202, a record number threshold 203, a data amount threshold 204, an upper limit record number 205, a lower limit record number 206, and a reconstruction lower limit rate 207.

The distribution variation type 201 indicates a distribution change pattern in the input data 21 for the number of records included in each key range.

The job name 202 indicates the identifier of the job that inputs the input data 21.

The record number threshold 203 is a threshold for the key range selection unit 1000 to determine the distribution variation type. The data amount threshold value 204 is a threshold value for the key range selection unit 1000 to select the division history 100 suitable for determining whether the distribution of the number of records included in the key range is appropriate.

The upper limit record number 205 is a threshold value for determining whether each key range should be divided by comparing with the predicted number of records in each key range. The lower limit record number 206 is a threshold value for determining whether each key range should be integrated with any key range by comparing with the predicted number of records in each key range.

The reconstruction lower limit rate 207 is a threshold value for determining whether or not it is necessary to regenerate all the key ranges included in the input data 21.

For example, the key range selection unit 1000 compares the difference between the total number of records in the history 1 and the total number of records in the history 2 by the total number of records in the history 1 with the record number threshold 203. Accordingly, the key range selection unit 1000 can determine the distribution variation type based on the increase rate (or decrease rate) of the number of records regardless of the total number of records included in the input data 21.

For example, the key range selection unit 1000 divides the difference between the total data amount of the input data 21 in the history 1 and the total data amount of the input data 21 in the history 2 by the total data amount of the input data 21 in the history 1. The number is compared with the data amount threshold value 204. Thereby, the key range selection unit 1000 can select an appropriate division history 100 in the history selection processing 1010 described later. The key range selection unit 1000 can reconfigure the key range of the input data 21 when the total data amount of the input data 21 is extremely increased or decreased.

In this embodiment, the values of the record number threshold 203 and the data amount threshold 204 are values that do not correspond to the distribution variation type 201. However, the parameter management table 200 may hold a plurality of upper limit record numbers 205, a plurality of lower limit record numbers 206, and a plurality of reconstruction lower limit rates 207, and the key range selection unit 1000 stores them in the distribution variation type 201. The upper limit record number 205, lower limit record number 206, or reconstruction lower limit rate 207 value may be selected according to the value to be set.

Further, the parameter management table 200 of the present embodiment may hold a predetermined value used in processing to be described later, or a numerical value indicating a predetermined range. A predetermined value or a numerical value indicating a predetermined range used in each process is stored in the parameter management table 200 by a user or an administrator.

FIG. 7A is an explanatory diagram illustrating a first change pattern of the distribution of the number of records in the input data 21 according to the first embodiment of this invention.

6 stores a value indicating one of the distribution variation type A, the distribution variation type B, the distribution variation type C, or the distribution variation type D by processing described later. FIG. 7A shows a variation pattern of the distribution variation type A.

7A and the horizontal axis shown in FIGS. 7B to 7D described later are the key ranges of records included in the input data 21. In FIG. Also, the vertical axis shown in FIG. 7A and FIGS. 7B to 7D described later is the number of records in each key range. Each line graph indicates the number of records in each key range indicated by the division history 100, and indicates the distribution of the number of records when the job indicated by each history is executed.

The job input data 21 differs every time the job is executed, and the data amount or the number of records often changes. On the other hand, the number of records of the input data 21 generally changes regularly.

Specifically, each input data 21 includes a record that is generated as a result of continuous business execution and is divided at specific times. Then, a job that uses the input data 21 as a processing target is periodically executed. For this reason, the distribution of the data amount of the records included in the input data 21 and the distribution of the number of records in each key range vary depending on the time, day, or month when the business is executed, and have a certain degree of regularity. It depends on.

For this reason, the key range selection unit 1000 refers to the number of records in the history 303 indicated by the key range management table 300, so that the change pattern of the distribution of the number of records in each key range of the input data 21 is changed to the distribution variation type A, The distribution variation type B, the distribution variation type C, or the distribution variation type D can be classified.

In the distribution variation type A, the total amount of data is included in a predetermined range and the total number of records is included in a predetermined range at a periodic timing such as a month, a day, or a day of the week when the job is executed. It is a change pattern of the number of records in the case. For example, when the input data 21 includes records relating to transactions, and the job is executed once every half month in a business in which transactions are concentrated on the deadline at the end of the month, the number of records in the input data 21 is as follows: Change.

For example, the total number of records in the history 1 and the total number of records in the history 3 shown in FIG. 7A are substantially the same, and are included in a predetermined range given in advance by the user or the administrator. However, the total number of records in history 2 shown in FIG. 7A is significantly different from the total number of records in history 1 and is not included in the predetermined range.

Thus, when the total number of records is periodically included in the predetermined range, the key range selection unit 1000 may classify the change pattern of the distribution of the number of records included in the input data 21 into the distribution variation type A. it can.

When the input data 21 is the distribution variation type A and the key range in which the number of records exceeds a predetermined value is present in the input data 21 at a predetermined ratio, the key range selection unit 1000 re-keys the key range in the input data 21. Must be configured.

FIG. 7B is an explanatory diagram illustrating a second change pattern of the record number distribution in the input data 21 according to the first embodiment of this invention.

FIG. 7B shows a distribution variation type B change pattern. In the distribution variation type B, the total amount of data included in the input data 21 gradually increases or decreases every time the job is executed, or the ratio of the number of records in each key range is increased every time the job is executed. It is a pattern that gradually increases or decreases.

The number of records in each key range shown in FIG. 7B increases gradually as the date of execution becomes new.

When the input data 21 is the distribution variation type B and the key range in which the number of records exceeds a predetermined value is present in the input data 21 at a predetermined ratio, the key range selection unit 1000 re-keys the key range in the input data 21. Must be configured.

FIG. 7C is an explanatory diagram illustrating a third change pattern of the distribution of the number of records in the input data 21 according to the first embodiment of this invention.

FIG. 7C shows a change pattern of the distribution variation type C. The distribution variation type C is a pattern in which the rate of increase in the number of records in the final key range is higher than the rate of increase in the number of records in other key ranges. The final key range in this embodiment is a key range in which the minimum value of the key range is larger than the key value of any other key range.

For example, when a record indicating a new customer or a new product is added to the end of the key of the input data 21 every time the business is executed, the number of records included in the input data 21 is as in the distribution variation type C. Change.

Specifically, the key range (201-300) shown in FIG. 7C is the final key range. The number of records in the key range (201-300) increases more each time a job is executed than the number of records in the other key ranges (001-100, 101-200).

When the input data 21 is the distribution variation type C and the number of records in the final key range exceeds a predetermined value, the data dividing unit 1100 needs to divide the final key range.

FIG. 7D is an explanatory diagram showing a fourth change pattern of the distribution of the number of records in the input data 21 according to the first embodiment of this invention.

FIG. 7D shows a variation pattern of the distribution variation type D. The distribution variation type D is a pattern in which only the number of records in one or a plurality of key ranges rapidly increases or decreases. For example, when transactions concentrate on a specific type of product, the number of records in the key range including a record indicating the specific product increases rapidly.

Specifically, the number of records in the key range (001-100) and key range (201-300) shown in FIG. 7D increases or decreases each time the job is executed.

When the input data 21 is the distribution variation type D and the key range in which the number of records exceeds a predetermined value is included in the input data 21 at a predetermined ratio, the key range selection unit 1000 determines that the number of records is out of the predetermined range. It is necessary to divide or integrate the key range.

FIG. 8 is a flowchart showing processing by the key range selection unit 1000 according to the first embodiment of this invention.

The key range selection unit 1000 can determine whether the key range of the input data 21 is reconfigured or the key range that needs to be divided or integrated into the input data 21 is included by the process shown in FIG.

The key range selection unit 1000 may execute the process shown in FIG. 8 at any timing as long as the input data 21 is input. That is, the processing shown in FIG. 8 may be executed using the latest input data 21 at the timing designated by the user or the administrator, or after the job is executed, the latest input data 21 is used. You may perform the process shown in FIG.

When the key range selection unit 1000 performs the process shown in FIG. 8, the job name of the input data 21 for performing the process shown in FIG. 8 is designated in advance by a user or an administrator. As a result, the key range selection unit 1000 reads the parameter management table 200 indicating the job name for which the job name 202 is designated at the start of the processing shown in FIG.

Also, the parameter management table 200 read at the start of the process shown in FIG.

When the process shown in FIG. 8 is started, the key range selection unit 1000 executes a history selection process 1010 described later. The key range selection unit 1000 selects, from the storage device 20, a plurality of division histories 100 to be used for the later-described process for predicting the number of records by the history selection process 1010.

Specifically, the key range selection unit 1000 uses the history selection process 1010 to determine that the total data amount indicated by the area 102 is the division history 100 within a predetermined value, and the job name specified in advance in the key range selection unit 1000. A plurality of division histories 100 indicated by the area 102 can be selected.

Also, the key range selection unit 1000 stores information included in the selected division history 100 in the key range management table 300 in the history selection processing 1010. Specifically, the key range selection unit 1000 stores the number of records in each key range indicated by the area 103 of the selected division history 100 in the history 303 of the key range management table 300.

After the history selection process 1010, the key range selection unit 1000 determines whether or not the key range selection unit 1000 has selected the division history 100 in the history selection process 1010 (1001). For example, if the key range selection unit 1000 cannot select the division history 100 because the job has not yet been executed and the division history 100 is not stored in the storage device 20, the key range selection unit 1000 uses the key described later. A range reconstruction process 1040 is executed.

When the total data amount of each division history 100 stored in the storage device 20 is significantly different from the total data amount of the input data 21, the key range selection unit 1000 performs the division history 100 in the history selection processing 1010. Cannot be selected. In this case, since the total data amount of the input data 21 has suddenly increased or decreased, it is necessary to regenerate the key range of the input data 21. For this reason, the key range selection unit 1000 executes a key range reconstruction process 1040 described later.

When it is determined in step 1001 that the division history 100 has been selected, the key range selection unit 1000 displays the job execution date indicated by each selected division history 100 and the total number of records indicated by each selected division history 100. , It is determined whether or not the total number of records of each division history 100 is periodically included in a predetermined range.

When the key range selection unit 1000 determines that the total number of records of each division history 100 is periodically included in the predetermined range, the distribution range type of the plurality of selected division histories 100 is changed to the distribution variation type A. To decide. Then, a value indicating the distribution variation type A is stored in the distribution variation type 201 of the parameter management table 200 (1002).

In step 1002, the key range selection unit 1000 specifies the total number of records in each division history 100 selected by the history selection processing 1010, and calculates the difference in the total number of records between the division histories 100. Then, when the value obtained by dividing the calculated difference by the total number of records of the division history 100 in which the area 101 indicates a new execution date and time is within a predetermined change rate, the key range selection unit 1000 determines each division history It is determined that the total number of records in 100 is included in the predetermined range.

Here, the predetermined change rate may be the record number threshold 203 of the parameter management table 200, or may be a value designated in advance in the key range selection unit 1000 by a user or an administrator.

When it is determined that the total number of records in each division history 100 is included in the predetermined range, and the execution date and time indicated by the area 101 of each division history 100 is periodic, the key range selection unit It is determined that the total number of 100 records is periodically included in the predetermined range.

Note that the key range selection unit 1000 holds in advance the conditions for determination in step 1002, that is, the execution date and time cycle. For example, the key range selection unit 1000 holds the calendar date and day of the week in advance, and in step 1002, the execution date and time indicated by the area 101 of the division history 100 is the end of every month or every Monday. Determine whether. Thus, the key range selection unit 1000 can determine whether or not the total number of records is periodically included in the predetermined range.

Since the division history 100 processed in step 1002 is the division history 100 in which the total data amount is included in the predetermined range, when it is determined in step 1002 that the total number of records is periodically included in the predetermined range, The key range selection unit 1000 can determine the distribution variation type of each division history 100 as the distribution variation type A shown in FIG. 7A.

After step 1002, the key range selection unit 1000 executes a key range record number prediction process 1020 described later based on the key range management table 300. By the key range record number prediction process 1020, the key range selection unit 1000 stores a value in the prediction number 305 of the key range management table 300.

After the key range record number prediction process 1020, the key range selection unit 1000 has the distribution variation type B or the distribution variation type C of the plurality of division histories 100 selected by the history selection process 1010. Or distribution variation type D (1003). In step 1003, the key range selection unit 1000 stores a value indicating the determined distribution variation type in the distribution variation type 201 of the parameter management table 200.

In step 1003, the key range selection unit 1000 calculates the difference in the number of records in the final key range between the division histories 100. If the difference in the number of records in the final key range is greater than the difference in the number of records in the other key ranges, the key range selection unit 1000 is selected by the history selection processing 1010. The distribution variation type of the plurality of division histories 100 is determined as the distribution variation type C.

For example, the difference d12 in the final key range shown in FIG. 7C is larger than the difference d12 in the key ranges (001-100, 101-200) shown in FIG. 7C, and the difference d23 in the final key range shown in FIG. When the difference d23 is larger than the difference d23 in the key ranges (001-100, 101-200) shown in FIG. 7C, the key range selection unit 1000 determines the distribution variation type of the plurality of division histories 100 selected by the history selection processing 1010 as distribution. It is determined as variable type C.

In step 1003, the key range selection unit 1000 further determines that the values obtained by dividing the difference in the number of records in each key range between the division histories 100 by the total number of records in the division history 100 with the new execution date and time indicated by the area 101 are all When the value is within the value indicated by the record number threshold 203 of the parameter management table 200, the distribution variation type of the plurality of division histories 100 selected by the history selection processing 1010 is determined as the distribution variation type B.

In step 1003, the key range selection unit 1000 determines that the distribution variation type A is not determined in step 1002, and if the distribution variation type B or the distribution variation type C is not determined by the above-described determination, the history selection processing 1010. The distribution variation type of the plurality of division histories 100 selected by is determined as the distribution variation type D.

After step 1003, the key range selection unit 1000 determines whether or not the distribution variation type 201 of the parameter management table 200 indicates the distribution variation type A or the distribution variation type B (1004). When the distribution variation type 201 indicates the distribution variation type A or the distribution variation type B, the key range selection unit 1000 executes Step 1005.

In step 1005, the key range selection unit 1000 acquires the upper limit record number 205, the lower limit record number 206, and the reconstruction lower limit rate 207 of the parameter management table 200.

In step 1005, the key range selection unit 1000 determines that the number of entries including the predicted number 305 exceeding the acquired upper limit record number 205 or lower than the acquired lower limit record number 206 is the key range management table 300. Calculate the percentage of all entries. Then, the key range selection unit 1000 determines whether or not the calculated ratio exceeds the reconstruction lower limit ratio 207.

If it is determined in step 1005 that the calculated ratio exceeds the reconstruction lower limit rate 207, it is necessary to change the key values included in all the key ranges. Processing 1040 is executed.

When it is determined in step 1004 that the distribution variation type 201 does not indicate the distribution variation type A or the distribution variation type B, or when the ratio calculated in step 1005 is determined to be the reconstruction lower limit rate 207 or less, The key range selection unit 1000 determines whether or not the distribution variation type 201 indicates the distribution variation type C (1006).

When it is determined in step 1006 that the distribution variation type 201 indicates the distribution variation type C, the key range selection unit 1000 sets “ON” in the division point investigation flag 307 of the entry corresponding to the final key range of the key range management table 300. "Is stored (1007). Accordingly, when the job specified at the start of the process shown in FIG. 8 is executed after the end of the process shown in FIG. 8, the data dividing unit 1100 described later divides the final key range, and a new division history 100 is obtained. Is output.

When it is determined in step 1006 that the distribution variation type 201 does not indicate the distribution variation type C, or after step 1007, the key range selection unit 1000 determines whether or not the distribution variation type 201 indicates the distribution variation type D. Determine (1008).

When it is determined in step 1008 that the distribution variation type 201 indicates the distribution variation type D, the key range selection unit 1000 executes a key range selection process 1030 described later. By the key range selection processing 1030, the key range selection unit 1000 can determine which key range is to be divided.

After the key range selection processing 1030, the key range selection unit 1000 divides the number of entries in which the division point investigation flag 307 of the key range management table 300 indicates “ON” by the total number of entries in the key range management table 300. Is calculated. Then, it is determined whether or not the calculated value exceeds the value indicated by the reconstruction lower limit rate 207 (1009).

When the value calculated in Step 1009 exceeds the value indicated by the reconstruction lower limit rate 207, the key range selection unit executes a key range reconstruction process 1040.

If it is determined in step 1008 that the distribution variation type 201 does not indicate the distribution variation type D, the value calculated in step 1009 is less than or equal to the value indicated by the reconstruction lower limit rate 207, or the key range reconstruction processing After 1040, the key range selection unit 1000 ends the process shown in FIG. Here, the key range selection unit 1000 stores, in the storage device 20, the parameter management table 200 in which values are stored in the distribution variation type 201 by the processing shown in FIG.

The parameter management table 200 holds the reconstruction lower limit rate 207 in step 1005 and the reconstruction lower limit rate 207 in step 1009, and the key range selection unit 1000 determines that the reconstruction lower limit rate with different values in step 1009 and step 1005. 207 may be used.

When the parameter management table 200 holds the reconstruction lower limit rate 207 having a plurality of different values and a specific condition is input from the user, the key range selection unit 1000 performs steps 1005 and 1009 according to the input condition. The value of the reconstruction lower limit rate 207 used in the above may be selected from the parameter management table 200.

For example, to reduce the processing time of the key range reconfiguration process 1040 when the computer system 1 is input in advance by a user such as a condition that the time from the start of the process shown in FIG. In addition, the key range selection unit 1000 may select the reconstruction lower limit rate 207 having a large value.

FIG. 9 is a flowchart showing the history selection processing 1010 by the key range selection unit 1000 according to the first embodiment of this invention.

In the history selection process 1010 after the start of the process shown in FIG. 8, the key range selection unit 1000 acquires the current date and time and the total data amount of the input data 21 (1011).

After step 1011, the key range selection unit 1000 extracts from the storage device 20 the division history 100 indicating the job name specified in the key range selection unit 1000 when the area 102 starts the processing shown in FIG. 8 or in advance. To do. Then, the execution date and total data amount of each division history 100 is acquired with reference to the extracted area 101 and area 103 of the division history 100 (1012).

After step 1012, the key range selection unit 1000 determines that the value obtained by dividing the difference between the total data amount of the input data 21 and the total data amount of the division history 100 by the total data amount of the input data 21 is the parameter management table 200. The division history 100 that is within the data amount threshold 204 is specified. Then, the key range selection unit 1000 selects a predetermined number of division histories 100 from the division histories with the latest job execution date / time among the identified division histories 100 (division histories with the smallest difference between the execution date / time and the current date / time). Is selected (1013).

In step 1013, the key range selection unit 1000 uses the division history 100 in which the current date / time and the execution date / time are too far apart, or the change in the number of records such as the division history 100 in which the total data amount is too different. Selecting an appropriate division history 100 can be avoided.

Note that the number of division histories 100 selected in step 1013 (the predetermined number described above) may be designated in advance by a user or an administrator. As the number of division histories 100 selected in step 1013 increases, the key range selection unit 1000 can obtain the predicted number 305 with higher accuracy in the key range record number prediction process 1020 shown in FIG.

After step 1013, the key range selection unit 1000 calculates the boundary value (minimum value and maximum value of the key value) of each key range and the number of records in each key range from all the division histories 100 selected in step 1013. The extracted information is stored in the minimum value 301, the maximum value 302, and the history 303 of the entry of the key range management table 300 corresponding to each division history 100 (1014).

In step 1014, the key range selection unit 1000 stores the number of records of each division history 100 in the history 303 of the key range management table 300 in the order of the execution date and time of each division history 100 or in the new order.

After step 1014, the key range selection unit 1000 ends the history selection process 1010.

As described above, the key range selection unit 1000 uses the process shown in FIG. 9 to obtain the division history 100 in which the total data amount is too different from the total data amount of the input data 21 and the division history for acquiring the change state of the number of records. Selecting as 100 can be avoided.

However, if the key range selection unit 1000 has previously acquired from the results of jobs executed a plurality of times in the past that the total data amount of the input data 21 does not vary greatly, the key range selection unit 1000 determines in step 1013. The total data amount of the input data 21 and the total data amount of the division history 100 need not be compared. And the key range selection part 1000 may select the division | segmentation log | history 100 only by the execution date in step 1013.

FIG. 10 is a flowchart showing the key range record number prediction process 1020 by the key range selection unit 1000 according to the first embodiment of this invention.

After step 1002, the key range selection unit 1000 starts the key range record number prediction process 1020 in order to store a value in the prediction number 305 of the key range management table 300. After the key range record number prediction process 1020 starts, the key range selection unit 1000 calculates the difference in the number of records between the histories 303 for each entry of the key range management table 300 (1021).

For example, in step 1021, the key range selection unit 1000 calculates the difference d12 = r1−r2 between the record number r1 of the history 1 (303a) and the record number r2 of the history 2 (303b) shown in FIG. Further, the key range selection unit 1000 calculates a difference d23 = r2−r3 between the record number r2 of the history 2 (303b) and the record number r3 of the history 3 (303c).

After step 1021, the key range selection unit 1000 compares the difference in the number of records between the histories calculated in step 1021 in each key range, and determines the increase / decrease type 304 of each entry (1022).

Specifically, in step 1022, the key range selection unit 1000 compares the record number difference d12 and the difference d23 in each key range. Then, the key range selection unit 1000 determines the increase / decrease type 304 according to the comparison result.

The method of determining the increase / decrease type 304 includes an increase / decrease pattern in which the number of records increases or decreases approximately linearly, an increase / decrease pattern in which the number of records increases or decreases abruptly, such as an exponential function, and an increase / decrease pattern in which the number of records hardly changes. Any method may be used as long as it can be determined. Below, the example of the method of determining the increase / decrease type | mold 304 in this embodiment is shown.

In step 1022, when the sign of the difference d12 and the sign of the difference d23 in a certain key range are positive and the difference d12 and the difference d23 are equal, the key range selection unit 1000 increases or decreases the entry of the key range. 304 is determined to be the increment / decrement type +1.

In addition, when the sign of the difference d12 and the sign of the difference d23 in a certain key range are positive and the difference between the difference d12 and the difference d23 is within a first predetermined value, the key range selection unit 1000 The increase / decrease type 304 of the key range entry is determined to be the increase / decrease type + 1.

In step 1022, if the sign of the difference d12 and the sign of the difference d23 of a certain key range are negative and the difference d12 and the difference d23 are equal, the key range selection unit 1000 determines the increase / decrease type 304 of the entry. Is determined to be increase / decrease type-1.

In addition, when the sign of the difference d12 and the sign of the difference d23 in a certain key range are negative and the difference between the difference d12 and the difference d23 is within a first predetermined value, the key range selection unit 1000 The increase / decrease type 304 of the entry is determined to be increase / decrease type-1.

In step 1022, if the value of the difference d12 and the value of the difference d23 of a certain entry is 0, or if the difference d12 and the difference d23 are within the second predetermined value, the key range selection unit 1000 The increase / decrease type 304 of the entry is determined to be the increase / decrease type 0.

In step 1022, when the signs of the difference d12 and the difference d23 are different, the key range selection unit 1000 determines the increase / decrease type 304 of the entry as the increase / decrease type 0.

In step 1022, if the sign of the difference d12 and the sign of the difference d23 of a certain entry are positive and the difference between the difference d12 and the difference d23 exceeds the first predetermined value, the key range selection unit 1000 Determines the increase / decrease type 304 of the entry to increase / decrease type +2.

In step 1022, if the sign of the difference d12 and the sign of the difference d23 of a certain entry are negative and the difference between the difference d12 and the difference d23 exceeds the first predetermined value, the key range selection unit 1000 Determines the increase / decrease type 304 of the entry to increase / decrease type-2.

Note that the key range selection unit 1000 may determine the increase / decrease pattern in step 1022 using a method other than the method described above. For example, a curve that approximates the distribution of the number of records in each key range indicated by the plurality of division histories 100 may be obtained, and the increase / decrease pattern may be determined from the shape of the curve.

Also, different values may be used for the first threshold and the second threshold for determining the increase / decrease pattern in step 1022 depending on the determined increase / decrease pattern. Furthermore, the first threshold value and the second threshold value may be the same value, and may be obtained by multiplying the number of records included in the input data 21 by the record number threshold 203 of the parameter management table 200.

After step 1022, the key range selection unit 1000 calculates the predicted number of records according to the increase / decrease pattern determined in step 1022, and stores the calculated predicted value in the predicted number 305 of the key range management table 300 (1023). ).

The key range selection unit 1000 holds in advance formulas or rules for calculating the predicted value of the number of records according to the increase / decrease pattern. The mathematical formula or rule for calculating the predicted value may be any mathematical formula or rule as long as it corresponds to the increase / decrease pattern.

The following shows an example of processing for calculating a predicted value.

For example, in step 1023, the key range selection unit 1000 sets the predicted number 305 of entries where the increase / decrease type 304 indicates the increase / decrease type 0 as the record number r1 of the history 1.

In step 1023, the key range selection unit 1000 calculates r1 + (d12 + d23) / 2 as the predicted number 305 of entries in which the increase / decrease type 304 indicates increase / decrease type + 1. Further, r1− (d12 + d23) / 2 is calculated as the predicted number 305 of entries in which the increase / decrease type 304 indicates the increase / decrease type−1.

In step 1023, the key range selection unit 1000 calculates r1 + d12 * (d12 / d23) as the predicted number 305 of entries in which the increase / decrease type 304 indicates +2. Further, r1−d12 * (d12 / d23) is calculated as the predicted number 305 of entries where the increase / decrease type 304 indicates −2.

Then, the key range selection unit 1000 stores the predicted value of each entry calculated in step 1023 in the predicted number 305. Then, after step 1023, the processing shown in FIG.

In the process shown in FIG. 10, the increase / decrease type 304 has five values of increase / decrease type +2, increase / decrease type + 1, increase / decrease type 0, increase / decrease type-1, and increase / decrease type-2. The unit 1000 may determine only three of increase / decrease type + 1, increase / decrease type 0, and increase / decrease type-1. As a result, the accuracy of the calculated predicted number 305 decreases, but the key range selection unit 1000 can determine the increase / decrease pattern even when only two division histories 100 are selected, for example.

FIG. 11 is a flowchart showing the key range selection processing 1030 by the key range selection unit 1000 according to the first embodiment of this invention.

11, when the distribution variation type 201 indicates the distribution variation type D, the key range selection unit 1000 can determine the key range to be divided or integrated.

If it is determined in step 1008 that the distribution variation type 201 is the distribution variation type D, or after step 1033, the key range selection unit 1000 executes step 1031 among the entries of the key range management table 300. Extract the entries that are not. Then, it is determined whether or not the predicted number 305 of the extracted entries exceeds the upper limit record number 205 of the parameter management table 200 of the distribution variation type D (1031).

When the predicted number 305 of each entry in the key range management table 300 exceeds the upper limit record number 205 of the parameter management table 200 of the distribution variation type D, the key range selection unit 1000 turns ON the division point investigation flag 307 of the entry. Store (1032).

If the predicted number 305 of each entry in the key range management table 300 is less than or equal to the upper limit number of records 205 in the parameter management table 200 of the distribution variation type D, the key range selection unit 1000 determines the division point investigation flag 307 for that entry. OFF is stored in (1034).

After step 1032 or step 1034, the key range selection unit 1000 determines whether or not the processing of step 1031 has been performed on all entries of the key range management table 300 (1033).

When the processing in step 1031 is performed on all entries in the key range management table 300, the key range selection unit 1000 ends the processing shown in FIG. If there is an entry in the key range management table 300 that has not been subjected to the processing of step 1031, the key range selection unit 1000 performs step 1031.

FIG. 12 is a flowchart showing the key range reconfiguration processing 1040 by the key range selection unit 1000 according to the first embodiment of this invention.

The key range selection unit 1000 reads the key value of each record from the input data 21 to the record buffer 14 (1041). After step 1041, the key range selection unit 1000 sorts the key values read to the record buffer 14 in ascending order, for example (1042).

After step 1042, the key range selection unit 1000 reads the upper limit record number 205 and the lower limit record number 206 from the parameter management table 200. Then, a plurality of key ranges are generated by dividing the key values sorted in step 1042 by (upper limit record number 205 + lower limit record number 206) / 2 (1043).

The upper limit record number 205 and the lower limit record number 206 read in step 1042 may be values according to the distribution variation type determined or determined in step 1002 and step 1003, and when the division history 100 does not exist in step 1001. A predetermined value may be used for performing the key range reconstruction processing.

After step 1043, the key range selection unit 1000 deletes the values in the key range management table 300 by updating all the values stored in the key range management table 300 to null values or the like (1044).

After step 1044, the key range selection unit 1000 stores the minimum value and maximum value of the key range generated in step 1043 in the minimum value 301 and maximum value 302 of the key range management table 300 (1045).

FIG. 13 is a flowchart showing the dividing process of the input data 21 by the data dividing unit 1100 according to the first embodiment of this invention.

The process shown in FIG. 13 is a process executed when the input data 21 is input to the job. The computer system 1 executes the job after executing the processing shown in FIG. The data dividing unit 1100 acquires the input data 21 input to the job and performs the processing shown in FIG.

The data dividing unit 1100 acquires a job name indicating a job to which the acquired input data 21 is input at the start of the processing illustrated in FIG. Then, the data division unit 1100 reads the parameter management table 200 that includes the acquired job name in the job name 202.

First, the data division unit 1100 reads one record from the acquired input data 21. Then, the data dividing unit 1100 compares the key value of the read record with the minimum value 301 and the maximum value 302 of each entry of the key range management table 300, and the key value of the read record is the minimum value. The entry included in 301 or more and the maximum value 302 or less is specified (1101).

After step 1101, the data division unit 1100 determines whether or not the division point investigation flag 307 of the entry identified in step 1101 is ON (1102).

When the division point investigation flag 307 of the identified entry is not ON, the key range corresponding to the identified entry does not need to be divided or integrated. Therefore, the data dividing unit 1100 outputs the record read in step 1101 to the divided data 22 corresponding to the key range of the identified entry (1104).

If it is determined in step 1102 that the division point investigation flag 307 of the identified entry is ON, the key range corresponding to the identified entry needs to be divided or integrated. Therefore, the data dividing unit 1100 stores the record read in step 1101 in the record buffer 14 (1103).

By outputting the record stored in the record buffer 14 in step 1103 to the divided data 22 after dividing or integrating the key range described later, and outputting the record that is not divided or integrated to the divided data 22 in step 1104, The data dividing unit 1100 can reduce the overhead generated by the processing in FIG.

That is, the data dividing unit 1100 does not need to read the input data 21 in the two processes of the key range dividing or integrating process and the process of outputting the record to the divided data 22 in steps 1103 and 1104, respectively. . Therefore, the data dividing unit 1100 can quickly perform the output processing of the divided data 22 and the key range division or integration processing.

After step 1103 or step 1104, the data dividing unit 1100 adds 1 to the record real number 306 of the entry specified in step 1101 (1105). As a result, the data dividing unit 1100 can calculate the number of records included in the input data 21 and subjected to the processing of step 1103 or step 1104 for each key range.

After step 1105, the data dividing unit 1100 determines whether all records included in the input data 21 are stored in the record buffer 14 or output to the divided data 22 (1106). If there is a record included in the input data 21 that is not stored in the record buffer 14 and is not output in the divided data 22, the data dividing unit 1100 returns to Step 1101.

When it is determined that all the records included in the input data 21 have been stored in the record buffer 14 or output to the divided data 22, the data dividing unit 1100 executes Step 1107.

In step 1107, the data dividing unit 1100 sorts the plurality of records stored in the record buffer 14 in the order of the key values of the records (1107).

After step 1107, the data division unit 1100 determines whether each entry (corresponding to each key range) in the key range management table 300 has an entry in which the record real number 306 exceeds the upper limit record number 205 (1108). . The key range indicated by the entry in which the record real number 306 exceeds the upper limit record number 205 is hereinafter referred to as a first key range.

The key range selection unit 1000 in the present embodiment extracts the key range candidates to be divided by updating the division point survey flag 307 based on the prediction number 305 in the process shown in FIG. In step 1108 shown in FIG. 13, the data dividing unit 1100 specifies a key range that actually needs to be divided from the extracted candidates based on the record real number 306.

Thus, it is possible to avoid dividing the key range that does not need to be divided or integrated by the process shown in FIG. Also, by extracting the key range candidates to be divided by the processing shown in FIG. 8, the key range of all the input data 21 is not determined in step 1108, so the processing time in step 1108 is shortened.

However, when the predicted number 305 is obtained with high accuracy, such as when a large number of division histories 100 are selected in the history selection process 1010 shown in FIG. 8, the key range selection unit 1000 performs the process of step 1108 shown in FIG. It may be omitted. In step 1109, the key range selection unit 1000 may divide all key ranges for which the division point survey flag 307 is ON.

If it is determined in step 1108 that each entry in the key range management table 300 has no entry whose record real number 306 exceeds the upper limit record number 205, the data dividing unit 1100 uses the record stored in the record buffer 14 as a key. According to the range management table 300, the data is output to the divided data 22. Then, the data dividing unit 1100 executes Step 1110. This is because there is no entry indicating the key range that actually needs to be divided among the entries for which the division point investigation flag 307 is ON.

If it is determined in step 1108 that each entry in the key range management table 300 includes an entry whose record real number 306 exceeds the upper limit record number 205, the data dividing unit 1100 determines a new key range based on the first key range. And the key range management table 300 is updated with the new key range. Then, the data dividing unit 1100 outputs the records of the first key range in the record buffer 14 to the plurality of divided data 22 according to the newly determined maximum value and minimum value of the key range (1109).

In step 1109, the first key range is divided, and the number of records of all the divided data 22 becomes the upper limit record number 205 or less. In the following, an example of division in step 1109 is shown.

In step 1109, the data dividing unit 1100 sets the first key range so as to divide the key value range in which the increase in the number of records is expected in the future as one key range so that division is unlikely to occur in the future. Divide the key range.

For example, when the number of records having the same key value is included in the first key range is ½ or more of the number of records included in the first key range, the data dividing unit 1100 determines the number of records in step 1109. The first key range is divided so that the record is divided into a key value that is 1/2 or more of the number of records in the first key range and other key values.

For example, when the distribution variation type 201 indicates the distribution variation type C and the first key range is the final key range, the data dividing unit 1100 sets the final key range to the minimum value of the final key range in step 1109. Is divided into a key range having a minimum value obtained by adding 1 to the maximum value of the final key range.

In cases other than the above example, in step 1109, the data dividing unit 1100 calculates the median value (key value that bisects the number of records included in the first key range) from the minimum value of the first key range and the median value. The first key range is divided into the maximum value.

If it is determined in step 1109 that the value indicated by the predicted number 305 is three times or more the upper limit record number 205, the data dividing unit 1100 may divide the first key range into three or more key ranges. Good.

Here, the upper limit record number 205 used for the determination in step 1108 may be a value different from the upper limit record number 205 used for the determination in step 1031.

If the parameter management table 200 includes a plurality of upper limit record numbers 205 and the data dividing unit 1100 determines that the time until the time designated in advance as the job end time is short, the data dividing unit 1100 In order to reduce the processing time, the upper limit record number 205 having a large value may be selected from the parameter management table 200. Then, the data dividing unit 1100 may use the selected upper limit record number 205 for the determination in step 1108.

If it is determined in step 1108 that each entry in the key range management table 300 has no entry whose record real number 306 exceeds the upper limit record number 205, or after step 1109, the data dividing unit 1100 includes the key range management table 300. It is determined whether each entry has an entry whose record real number 306 is lower than the lower limit record number 206 (1110). The key range indicated by the entry whose record real number 306 is lower than the lower limit record number 206 will be referred to as a second key range below.

When each entry in the key range management table 300 includes an entry whose record real number 306 is lower than the lower limit record number 206, the data dividing unit 1100 integrates one of the key ranges adjacent to the second key range and the second key range. (1111). Then, the data dividing unit 1100 updates the key range management table 300 according to the integrated result.

Specifically, in step 1111, the data dividing unit 1100 sets the value obtained by subtracting 1 from the minimum value 301 of the second key range to the maximum value 302 and 1 to the maximum value 302 of the second key range. A key range in which the value obtained by adding the minimum values 301 is extracted as a key range adjacent to the second key range. The data dividing unit 1100 includes a key range having a record real number 306 that does not exceed the upper limit record number 205 even if the record real number 306 of the second key range is added among the key ranges adjacent to the second key range; Integrate with 2 key range.

For example, among the entries of the key range management table 300 indicating the key range adjacent to the second key range, an entry having a small minimum value 301 is described as an entry A and an entry having a large minimum value 301 is described as an entry B. Here, the record real number 306 of the entry A is smaller than the record real number 306 of the entry B, and the result of adding the record real number 306 of the entry A and the record real number 306 of the second key range is the upper limit record number 205 or less. The data dividing unit 1100 integrates the entry A and the second key range.

More specifically, in order to integrate the key range, the data dividing unit 1100 stores the value of the maximum value 302 of the second key range in the maximum value 302 of the entry A in step 1111 and records the real number 306 of the entry A. And the record real number 306 of the entry indicating the second key range is stored in the record real number 306 of the entry A. Then, the data dividing unit 1100 deletes the entry indicating the second key range.

Here, if there is no key range having a record real number 306 that does not exceed the upper limit record number 205 even if the record real number 306 of the second key range is added among the key ranges adjacent to the second key range, The data dividing unit 1100 does not integrate the second key range and other key ranges.

The processing in step 1111 is processing for updating the key range management table 300, and the processing shown in FIG. 13 is executed next when the second key range is integrated with other key ranges and output to the divided data 22. When it is done. For this reason, the computer system 1 may execute the job after step 1109, thereby reducing the job processing time.

In step 1110 shown in FIG. 13, the data dividing unit 1100 specifies the second key range from all the key ranges included in the input data 21 based on the record real number 306. However, in the process shown in FIG. 8, the data dividing unit 1100 extracts key range candidates to be integrated based on the predicted number 305, and in step 1110, selects the second key range from the key range candidates to be integrated. You may specify.

If it is determined in step 1110 that each entry in the key range management table 300 has no entry whose record real number 306 is lower than the lower limit record number 206, or after step 1111, the data dividing unit 1100 performs the processing of FIG. The job name acquired at the start, the current date and time, the number of bytes of the input data 21, the minimum value 301, the maximum value 302, and the record real number 306 of all entries in the key range management table 300 are output to the division history 100. (1112).

In step 1112, the data division unit 1100 updates all the values of the division point investigation flag 307 of the key range management table 300 to OFF.

After step 1112, the data dividing unit 1100 ends the process shown in FIG.

According to the first embodiment, the computer system 1 obtains the distribution variation type of the input data 21 from the division history 100, and further obtains the predicted number of records in each key range, thereby dividing or integrating the key range. To decide. For this reason, the divided data 22 input to the job can be maintained at an appropriate scale.

Further, the computer system 1 of the first embodiment outputs the divided data 22 according to the determined key range without reading the entire input data 21 immediately before executing the job. For this reason, the processing time for maintaining each divided data 22 at an appropriate scale can be shortened when the job is executed.

More specifically, by finding the distribution variation type and the predicted number, the key range candidates to be divided or integrated can be specified, and only the candidate key ranges are divided or integrated at the time of job execution. Therefore, it is possible to avoid wastefully verifying all the key ranges included in the input data 21 at the time of job execution, and to shorten the processing time.

(Second embodiment)

The data dividing unit 1100 according to the second embodiment does not output the divided data 22 after dividing or integrating the key range at the time of executing the job, and the key range is divided or integrated at the time of executing the next job. The divided data 22 is output according to the result. As a result, the data dividing unit 1100 according to the second embodiment can reduce the processing time required for the process for outputting the divided data 22.

FIG. 14 is a flowchart showing a key range dividing process by the data dividing unit 1100 according to the second embodiment of this invention.

The process shown in FIG. 14 is a process executed when the input data 21 is input to the job to be executed, like the process shown in FIG.

Step 1101 to step 1102 in FIG. 14 are the same as those in FIG.

If the data division unit 1100 in the second embodiment determines in step 1102 that the division point investigation flag 307 of the identified entry is ON, the data division unit 1100 uses the key value of the record read in step 1101 as the record buffer 14. (1103b).

When it is determined in step 1102 that the division point investigation flag 307 of the identified entry is not ON, or after step 1103b, the data division unit 1100 records according to the key range indicated by the key range management table 300. Is output to the divided data 22 (1104).

In step 1104, the data dividing unit 1100 of the second embodiment outputs all records included in the input data 21 to the divided data 22. The process shown in FIG. 14 in the second embodiment is a process for regenerating each entry in the key range management table 300, and the input data 21 is output to the divided data 22 in accordance with the result of division or integration. This is because it is not a process to perform.

For this reason, since the job after step 1104 shown in FIG. 14 may be executed, when the data dividing unit 1100 uses the process shown in FIG. 14, the job is started more than when the process shown in FIG. 13 is used. Time can be shortened.

Steps 1105 and 1106 in FIG. 14 are the same as steps 1105 and 1106 in FIG.

When it is determined in step 1106 that all records included in the input data 21 have been output to the divided data 22, the data dividing unit 1100 sets the key values of the records stored in the record buffer 14 in ascending order, for example. Sort (1107b).

Steps 1108 and 1110 in FIG. 14 are the same as steps 1108 and 1110 in FIG.

14 is the same as step 1109 in FIG. 13 except that a record is not output to the divided data 22 in steps 1109 and 1111 in FIG.

Step 1111 and step 1112 in FIG. 14 are the same as step 1111 and step 1112 in FIG.

The key range management table 300 is updated by the processing shown in FIG. When the job is executed next time, step 1104 shown in FIG. 14 is executed by the updated key range management table 300, and the divided data 22 is generated according to the updated key range management table 300.

According to the second embodiment, the key range management table 300 indicating the key range after being divided or integrated when the job is executed is generated, and the next job is executed when the job is executed next. The record of the input data 21 is output to the divided data by the key range management table 300 generated at that time. For this reason, it is possible to shorten the processing for dividing or integrating the key range, which is performed when the job is executed.

According to this embodiment, the computer system 1 determines a distribution range type of the input data 21 from the division history 100, and further determines a key range to be divided or integrated by obtaining a predicted number of records in each key range. To do. For this reason, the divided data 22 input to the job can be maintained at an appropriate scale.

The computer system 1 according to the first embodiment outputs the divided data 22 according to the determined key range without reading the entire input data 21 immediately before executing the job. For this reason, the processing time for maintaining each divided data 22 at an appropriate scale can be shortened when the job is executed.

Although the present invention has been described in detail with reference to the accompanying drawings, the present invention is not limited to such specific configurations, and various modifications and equivalents within the spirit of the appended claims Includes configuration.

The present invention can be used in a computer system that processes distributed jobs.

Claims (15)

A data processing method in a computer system for dividing input data including a plurality of records and processing each of the divided input data by each of a plurality of jobs,
The computer system includes a processor and a memory,
A key value is assigned to each of the plurality of records,
The key value is classified into one of key ranges defined by a minimum value and a maximum value,
The method
When the processor processes the records divided according to the key ranges by the plurality of jobs, the number of the processed records and the key value assigned to the processed records are classified. Get history showing range and
The processor determines a pattern of change in the number of records included in the input data based on the acquired plurality of histories,
The data processing method, wherein the processor determines the key range for changing at least one of the minimum value and the maximum value according to the determined change pattern. A data processing method according to claim 1, comprising:
The method
If each of the histories indicates that a predetermined number of records are processed periodically by the job, the processor determines that the pattern of change in the number of records is a first pattern;
When each history indicates that the amount of change in the number of records classified into each key range is equal to or less than a predetermined value in all the key ranges, the processor records the records included in the input data. The number change pattern is determined to be the second pattern,
In each of the histories, the amount of change in the number of records classified into the first key range in which the key value having the largest value is classified is recorded in a record classified into a key range other than the first key range. When indicating that the amount of change is greater than the number change amount, the processor determines that the pattern of change in the number of records included in the input data is the third pattern,
If it is not determined that the pattern of change in the number of records included in the input data is the first pattern, the second pattern, or the third pattern, the processor includes the input data. The pattern of change in the number of records to be determined is the fourth pattern,
When the determined change pattern is the third pattern, the processor determines to change at least one of the minimum value and the maximum value of the first key range. . A data processing method according to claim 2, comprising:
The computer system holds an upper limit value of the number of records classified into each key range,
The method
The processor predicts the number of records classified into each key range when the input data is processed by the job based on the number of records classified into each key range indicated by each history,
The processor calculates a ratio of the second number of key ranges in which the number of predicted records exceeds the upper limit value to the number of all the key ranges;
The processor determines whether the calculated percentage exceeds a predetermined percentage;
As a result of the determination, the processor determines that the calculated ratio exceeds the predetermined ratio, and the determined change pattern is the first pattern, the second pattern, and the fourth pattern. A data processing method comprising: determining that at least one of the minimum value and the maximum value of all the key ranges is changed when it is determined that the pattern is one of the patterns. A data processing method according to claim 3, comprising:
The computer system holds the minimum value and the maximum value of the key range in the memory,
The method
When the input data is input to the computer system,
The processor divides a record classified into a key range other than the second key range included in the input data according to a minimum value and a maximum value of the key range;
The processor stores in the memory a record classified into a second key range of the input data;
The data, wherein the processor divides the record classified in the second key range stored in the memory according to at least one of the changed minimum value and maximum value of the second key range. Processing method. A data processing method according to claim 3, comprising:
In the method, the processor changes at least one of the minimum value and the maximum value of the second key range so that the number of records classified into the second key range is equal to or less than the predetermined upper limit value. By doing so, the key value classified into the second key range is classified into the second key range and the third key range. A data processing method according to claim 3, comprising:
The computer system holds a lower limit value of the number of records classified into the key ranges,
The method
The processor identifies a fourth key range in which the number of records is below the lower limit;
Identifying a fifth key range adjacent to the fourth key range;
If the sum of the number of records classified into the fourth key range and the number of records classified into the fifth key range does not exceed the upper limit value, it is classified into the fifth key range. A data processing method comprising: changing at least one of a minimum value and a maximum value of the fourth key range so that the key value is classified into the fourth key range. A data processing method according to claim 3, comprising:
The computer system maintains a rule for predicting the number of records according to an increase / decrease pattern of the number of records classified into each key range,
The method
The processor specifies an increase / decrease pattern of the number of records classified into each key range indicated by each history,
Data processing characterized by predicting the number of records classified into each key range when the input data is processed by the job, using the specified increase / decrease pattern and the retained rule Method. A data processing program in a computer system that divides input data including a plurality of records and processes each of the divided input data by each of a plurality of jobs,
The computer system includes a processor and a memory,
A key value is assigned to each of the plurality of records,
The key value is classified into one of key ranges defined by a minimum value and a maximum value,
The data processing program is stored in the computer system.
When the records divided according to each key range are processed by the plurality of jobs, the number of the processed records and the key range into which the key values assigned to the processed records are classified The steps to get the history,
A procedure for determining a pattern of change in the number of records included in the input data based on the plurality of acquired histories;
A data processing program for executing a procedure for determining the key range for changing at least one of the minimum value and the maximum value in accordance with the determined change pattern. A data processing program according to claim 8, wherein
The data processing program is
A procedure for determining that the pattern of change in the number of records is a first pattern when each of the histories indicates that a predetermined number of records are periodically processed by the job;
When each history indicates that the amount of change in the number of records classified into each key range is equal to or less than a predetermined value in all the key ranges, the change in the number of records included in the input data A procedure for determining that the pattern is the second pattern;
In each of the histories, the amount of change in the number of records classified into the first key range in which the key value having the largest value is classified is recorded in a record classified into a key range other than the first key range. A procedure for determining that the pattern of change in the number of records included in the input data is a third pattern when indicating that the amount of change is greater than the number of changes;
The number of records included in the input data when the pattern of change in the number of records included in the input data is not determined to be the first pattern, the second pattern, or the third pattern A procedure for determining that the pattern of change is the fourth pattern;
A data processing program for executing a procedure for determining that at least one of the minimum value and the maximum value of the first key range is changed when the determined change pattern is the third pattern. A data processing program according to claim 9, wherein
The computer system holds an upper limit value of the number of records classified into each key range,
The data processing program is stored in the computer system.
A procedure for predicting the number of records classified into each key range when the input data is processed by the job based on the number of records classified into each key range indicated by each history;
Calculating a ratio of the second number of key ranges in which the number of predicted records exceeds the upper limit to the number of all the key ranges;
A procedure for determining whether the calculated ratio exceeds a predetermined ratio;
As a result of the determination, the calculated ratio exceeds the predetermined ratio, and the determined change pattern is one of the first pattern, the second pattern, and the fourth pattern. A data processing program for executing a procedure for determining to change at least one of the minimum value and the maximum value of all the key ranges when it is determined that A data processing program according to claim 10, wherein
The computer system holds the minimum value and the maximum value of the key range in the memory,
The data processing program is stored in the computer system.
When the input data is input to the computer system,
Dividing a record classified into a key range other than the second key range included in the input data according to a minimum value and a maximum value of the key range;
Storing a record classified into a second key range of the input data in the memory;
Data processing for executing a procedure for dividing a record classified into the second key range stored in the memory according to at least one of the changed minimum value and maximum value of the second key range program. A computer system that divides input data including a plurality of records and processes each of the divided input data by each of a plurality of jobs,
The computer system includes a processor and a memory,
A key value is assigned to each of the plurality of records,
The key value is classified into one of key ranges defined by a minimum value and a maximum value,
The computer system is
When the records divided according to each key range are processed by the plurality of jobs, the number of the processed records and the key range into which the key values assigned to the processed records are classified Get history,
Based on the plurality of acquired histories, determine a pattern of change in the number of records included in the input data,
A computer system, wherein the key range for changing at least one of the minimum value and the maximum value is determined according to the determined change pattern. A computer system according to claim 12, comprising:
The computer system is
When each of the histories indicates that a predetermined number of records are periodically processed by the job, it is determined that the pattern of change in the number of records is the first pattern;
When each history indicates that the amount of change in the number of records classified into each key range is equal to or less than a predetermined value in all the key ranges, the change in the number of records included in the input data Determine that the pattern is the second pattern,
In each of the histories, the amount of change in the number of records classified into the first key range in which the key value having the largest value is classified is recorded in a record classified into a key range other than the first key range. If it indicates that the amount of change is greater than the number of changes, the pattern of change in the number of records included in the input data is determined to be the third pattern,
The number of records included in the input data when the pattern of change in the number of records included in the input data is not determined to be the first pattern, the second pattern, or the third pattern It is determined that the pattern of change is the fourth pattern,
When the determined change pattern is the third pattern, it is determined to change at least one of the minimum value and the maximum value of the first key range. A computer system according to claim 13, comprising:
The computer system is
Holding an upper limit of the number of records classified into each of the key ranges;
Based on the number of records classified into each key range indicated by each history, predict the number of records classified into each key range when the input data is processed by the job,
Calculating the ratio of the second number of key ranges in which the number of predicted records exceeds the upper limit to the number of all the key ranges;
Determining whether the calculated percentage exceeds a predetermined percentage;
As a result of the determination, the calculated ratio exceeds the predetermined ratio, and the determined change pattern is one of the first pattern, the second pattern, and the fourth pattern. When it is determined that the minimum value and the maximum value of all the key ranges are determined, it is determined to change. A computer system according to claim 14, wherein
The computer system is
Holding the minimum and maximum values of the key range in the memory;
When the input data is input to the computer system,
Dividing a record classified into a key range other than the second key range included in the input data according to a minimum value and a maximum value of the key range;
Storing a record classified in a second key range of the input data in the memory;
A computer system that divides records classified into the second key range stored in the memory according to at least one of the changed minimum value and maximum value of the second key range.

Images