JP2008225686A - Data arrangement management device and method in distributed data processing platform, and system and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problems of a vast time spent for production processing of replica data when performing parallel distribution processing of data processing, and reproduction or the like of the replica data for corresponding to a latest data classification situation to allow optimum data arrangement. <P>SOLUTION: This system includes: a data arrangement management device 1 for managing attribute information of data, address information and a data size, and providing data size information and the address information of a data group associated to the attribute information according to inquiry based on the attribute information to an inquiry source; an annotator 4 for generating the attribute information to the data stored in a data processing server 2, and updating attribute information data of the data arrangement management device; and a data production client 5 for transmitting the data size and a new production request of the data to the data arrangement management device when storing the newly produced data into the data processing server, acquires the address information of a data arrangement destination, and storing the data into the data processing server. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to information processing technology, and in particular, to a data arrangement technology in which data processing is suitable for parallel distributed processing.

  Generally, when data processing such as data analysis processing and data reading processing is performed using a large amount of data stored in a data storage system, data reading processing from the disk storage in the data storage device is a bottleneck of data processing. As a result, the performance of the entire data processing is lowered. This is because the data access speed to the disk storage is generally slower than the data access speed of other components such as a CPU and a memory constituting the data processing system.

  The larger the amount of data handled in data processing, the worse the balance between the processing in the disk storage and the processing of other components, and the delay in reading data from the disk storage becomes a significant bottleneck for the entire system. Appears.

  In order to solve such a problem of data read processing from disk storage, generally, a data group used for data processing is distributed and arranged in a plurality of data storage devices, and data read processing from disk storage is performed in parallel. The method to be executed is used.

  By using this technique, the data group used for data processing is divided equally in advance and distributed to a plurality of data storage devices, so that the data read processing in each data storage device is distributed in parallel. And can be executed. As a result, the performance of the entire data processing can be improved. Such a technique is used as a technique for speeding up data processing, particularly in data processing in which there is no correlation between data distributed in a data storage device.

  On the other hand, in executing data processing such as data analysis processing and data reading processing as described above, it is assumed that processing for creating a data group itself used for data processing has been performed.

  In a system for the purpose of a single data processing, when the data processing is executed without depending on the type of data, the data generated in advance during the data creation processing is related to the data group used for the data processing. A data storage system that parallelizes data read processing can be easily realized by arranging the data storage devices so as to have the same capacity among the plurality of data storage devices.

However, for example,
・ Multiple data processing coexist and
・ Each data processing does not use all data stored in the data storage system.
In some cases, it is possible to realize distributed data processing and parallel processing that speed up all data processing by simply performing data distributed processing so that the data capacity is equalized during data creation processing. I can't.

  In order to solve such a problem, for example, only a data group used in a certain data processing is extracted from a data group stored in the data storage system, and the extracted data group is extracted in another data storage system. A method is also conceivable in which a replica is created and distributed at a plurality of data storage devices so that the capacities are equal, thereby speeding up data processing.

  As a conventional technique related to data arrangement in consideration of correspondence between data and attributes, for example, Patent Document 1 discloses geographical information (presentation target data) having a spatial distribution such as satellite image data, and geographical information. The attribute information associated with the satellite image data such as the quality accuracy and the green space rate related to the information is associated with each other, and the data space of the number of dimensions based on the attribute information of the target presentation data group (longitude, A configuration is disclosed in which a presentation target data group is arranged and visible in a three-dimensional space of latitude and time and time. Various systems have been proposed in which data groups are distributed and arranged in a plurality of data storage devices to equalize the amount of data. For example, in Patent Document 2, a monitor agent is connected to each storage device or all storage devices via a network. Monitors information related to the total amount of data stored, and the arrangement means is used to execute data movement to achieve system leveling (leveling of used capacity, leveling of remaining capacity, leveling of capacity usage) A distributed storage system is disclosed in which a logical storage area is distributed and rearranged in a physical storage area on a previous storage device via a network and a control module. Further, as a configuration in which data is distributed and rearranged, for example, in Patent Document 3, a physical device number and an address in which each data file is written are recorded in a map in the relocation means, and reading is requested. A plurality of data file names and the number of combinations thereof are monitored by the file monitoring means, and a plurality of data files are concentrated on one physical device based on the recording data in the relocation means and the recording data in the file monitoring means. There is disclosed a disk array device that is dispersed and rearranged. However, as will be apparent from the following description, the problem to be solved by the present invention, means for solving the problem, etc. are not described in any of Patent Documents 1 to 3.

JP 2000-076294 A JP 2005-050303 A JP-A-8-202503

  In an environment where multiple different data processing systems coexist, executing data processing using data stored in the corresponding data storage systems, data replicas are created and data is stored in other data storage systems. In order to speed up all data processing, it is necessary to execute data processing for the same data group among multiple different data processing systems. It is said. That is, unless this is done, in order to extract data used only by data processing from the data storage system for each data processing, and to distribute the extracted data equally to a plurality of data processing systems, replicas of the extracted data are used. Must be created in other data storage systems.

  However, normally, in data processing that implements such distributed arrangement of data, it is assumed that the data capacity of the data group used in each data processing system is so large that it is worth performing parallel processing. It is said that. For this reason, creating a replica of a data group used in each data processing system for each data processing consumes a large amount of data capacity, and the replica creation processing requires a huge amount of time. There is.

In addition, in a system environment that uses a plurality of data classification methods such as data content, meaning, positioning, etc., as a reference for specifying a data group used by data processing, the data content, meaning, positioning are If it changes over time,
-In data processing, when data that was not defined at the point of data creation at the time of replica creation is newly added as a data processing target,
・ When the definition of data is changed, it may be excluded from the data processing target.
・ When data created after replica creation is newly added as a data processing target,
Etc. will occur.

  Therefore, in order to cope with such a situation and execute data processing in the latest data classification situation, replica data must be recreated every time data processing is executed.

In any case, in an environment where a large amount of data is stored in a plurality of data processing systems, assuming that data processing such as analysis processing and data read processing is executed, when performing parallel distributed processing of data processing,
・ A huge amount of time is spent in creating replica data.
・ Replica data must be recreated to support the latest data classification situation.
It is desired to realize a system that solves the problems such as the above and enables optimal data arrangement.

  Further, in relation to the above problem, when data capacity is leveled between a plurality of data processing systems each having a storage, even if data placement is optimized in relation to one attribute information, Regarding data related to attribute information, the deviation of data storage capacity between multiple storages will increase, the data processing performance by other attribute information will deteriorate, and the overall data processing performance will deteriorate It may occur.

  At the time of filing this application, there was no prior art document or the like that disclosed a technique related to recognition of the above-mentioned problems described in this specification. An object of the present invention is to provide a completely new apparatus, system, method, and computer program that solve the above-described problems.

  In order to solve the above-mentioned problems, the invention disclosed in the present application is generally configured as follows.

  The present invention rearranges using multi-faceted attribute information states so that parallel processing of a plurality of data processings can be executed in an environment where the attribute information dynamically changes and the point of data processing changes. The data to be extracted is extracted, and the data is rearranged at an optimum place.

  The present invention creates a replica of a data group to be processed for each data processing task in an environment where a plurality of different data processing tasks are executed on the data group stored in a data storage system. Instead of using a method for storing in other data storage systems, only a single data storage system is used, and a data placement management device is introduced to relocate the data group to the optimal location based on the data classification situation By doing so, parallelization of a plurality of different data processing is realized.

  In the present invention, a plurality of attribute information used for classifying data used in each data processing is given to the data group.

In the present invention, the data arrangement management device
Attribute information given to the data group,
Address information of a data storage device storing the data group;
Data size,
A means of managing
Means for acquiring attribute information designated as a data processing target in each data processing task;
From the attribute information,
A distribution of data capacity of a data group associated with the attribute information;
Data capacity ratio distribution among multiple data storage devices,
Means for calculating
From the latest situation of the data capacity distribution and the data capacity ratio distribution, the accumulated data capacity is uniformly within a certain level range among a plurality of data storage devices for a data group associated with certain attribute information. Means for rearranging data are provided.

An apparatus according to another aspect of the present invention relates to a plurality of data each provided with a plurality of attribute information and a plurality of data storage apparatuses, address information indicating a storage location of the data, size information of the data, Information storing means for storing and managing the attribute information assigned to the data in association with each other;
Means for referring to the information storage means for obtaining distribution status of data capacity in the plurality of data storage devices of the data to which the attribute information is attached;
When it is determined that the distribution status of the data capacity acquired with respect to at least one attribute information among a plurality of attribute information meets a predetermined data relocation condition,
When the data is rearranged, the distribution of the data capacity among the plurality of data storage devices regarding the one attribute information not only satisfies a predetermined condition regarding equalization, but at least of the plurality of attribute information Means for deriving a rearrangement plan so as to satisfy a predetermined condition regarding homogenization with respect to a distribution of data capacity among the plurality of data storage devices related to one other attribute information;
Means for rearranging data among the plurality of data storage devices according to the derived rearrangement plan.

The computer program according to the present invention relates to a plurality of data each having a plurality of attribute information and a plurality of data storage devices, address information indicating a storage location of the data, size information of the data, A process of associating the attribute information given to the data and storing it in the storage unit;
A process of referring to the storage unit to acquire a distribution state of data capacity in the plurality of data storage devices of data to which attribute information is attached;
When it is determined that the distribution status of the data capacity acquired with respect to at least one attribute information among a plurality of attribute information meets a predetermined data relocation condition,
The data rearrangement plan is evaluated, and not only the distribution of data capacity among the plurality of data storage devices related to the one attribute information satisfies a predetermined condition regarding equalization, but also among the plurality of attribute information A process for deriving a rearrangement plan that satisfies a predetermined condition regarding the uniformity of the distribution of data capacity among the plurality of data storage devices related to at least one other attribute information;
Relocating data among the plurality of data storage devices according to the derived relocation plan;
It consists of a program that causes a computer to execute.

The method according to the present invention relates to a plurality of data each having a plurality of attribute information and a plurality of data storage devices, address information indicating a storage location of the data, size information of the data, and the data Associating the assigned attribute information with each other and storing it in the storage unit;
Referring to the storage unit, obtaining a distribution state of data capacity in the plurality of data storage devices of data to which attribute information is given;
When it is determined that the distribution status of the data capacity acquired with respect to at least one attribute information among a plurality of attribute information meets a predetermined data relocation condition,
When the data is rearranged, the distribution of the data capacity among the plurality of data storage devices regarding the one attribute information not only satisfies a predetermined condition regarding equalization, but at least of the plurality of attribute information A step of deriving a rearrangement plan that satisfies a predetermined condition regarding homogenization with respect to distribution of data capacity among the plurality of data storage devices related to one other attribute information;
Rearranging data among the plurality of data storage devices according to the derived rearrangement plan.

  In the system according to the present invention, with respect to data each having a plurality of attribute information, the distribution of the data capacity among the plurality of data storage devices that store the data with respect to each of the plurality of attribute information is determined in advance. In order to approach the situation, there is provided means for deriving an optimum data arrangement that can be taken between the plurality of data storage devices, and means for rearranging data according to the derived data arrangement.

  According to the present invention, when a plurality of different data processing tasks based on attribute information are executed on data in which a plurality of attribute information stored in a data storage system is defined, each data processing target is processed. Even if the data capacity of the data group is huge, multiple data processing can be performed using only each data storage system without creating a replica of the data group for each data processing in another data storage system. It becomes possible to execute parallel distributed processing of data processing using the apparatus.

  That is, according to the present invention, since replica data is not created for each data process, it is possible to reduce the creation time of replica data and the storage capacity for storing replica data.

  Furthermore, according to the present invention, in a system for leveling data storage capacity among a plurality of storages, even if data is leveled in relation to one attribute information, data related to another attribute information is related. However, it is possible to suppress the occurrence of a situation in which the deviation of the data storage capacity between a plurality of storages is enlarged and the overall data processing performance is deteriorated.

The present invention will be described in detail below with reference to the accompanying drawings. FIG. 1 is a diagram illustrating an example of a system configuration according to an embodiment of the present invention. Referring to FIG. 1, in this embodiment,
At least one task broker 3;
At least two data processing servers 2, and
At least one annotator 4 and
At least one data creation client 5;
At least one data processing task client 8;
And a data arrangement management device 1, which constitute a system capable of communicating via the network 6. In FIG. 1, the data processing task client 8 is shown as a connection configuration that directly gives the data processing task to the task broker 3 for ease of explanation, but the data processing task client 8 is connected to the task via the network 6. Of course, the broker 3 may be given a data processing task.

  The task broker 3 receives the data processing task from the data processing task client 8 and transfers the data processing task to all the data processing servers 2 constituting the system via the network 6. The task broker 3 has a role of receiving and integrating data processing results executed by the data processing servers 2 to complete the task. The task completion is returned to the data processing task client 8 as a response. The task broker 3 stores a list of tasks scheduled to be executed in an internal storage unit (not shown).

The data processing task is at least
-Attribute information for specifying the data group to be subjected to data processing,
・ Contents of data processing such as data analysis processing and data reading processing,
including.

  Each of the plurality of data processing servers 2 includes a disk storage 7 as a data storage device that stores data for executing data processing.

  Based on the data processing task transferred from the task broker 3, the data processing server 2 executes the data processing task using the data stored in the disk storage 7 and returns the execution result to the task broker 3.

The data arrangement management device 1
Attribute information associated with the data,
-Address information indicating the storage location of the data,
-The data size of the relevant data is stored and managed in association.

In response to the inquiry based on the attribute information from the external device, the data arrangement management device 1
-Address information of the data group associated with the corresponding attribute information,
・ Data size information,
Is provided to the inquirer.

  FIG. 2 is a diagram showing an example of the configuration of the data arrangement management device 1 in the present embodiment. Referring to FIG. 2, the data arrangement management apparatus 1 includes a data arrangement / attribute information storage unit 100, a data management unit 101, a task analysis unit 102, a data arrangement status analysis unit 103, a data arrangement control unit 104, A rearrangement execution unit 105 is provided.

The data arrangement / attribute information storage unit 100
The address information of the data stored in the disk storage 7 of the data processing server 2,
Attribute information associated with the data,
・ Data size of data,
Is stored.

The data management unit 101
・ New data creation request by the data creation client 5,
In response to an attribute information addition / change request by the annotator 4, the data stored in the data arrangement / attribute information storage unit 100 is updated.

  The task analysis unit 102 analyzes the contents of the data processing task executed by the task broker 3 or scheduled to be executed (process execution contents).

  The data arrangement state analysis unit 103 uses the information (data storage address and attribute information) stored in the data arrangement / attribute information storage unit 100 for a plurality of data groups with a specific attribute information as a key. The distribution status in the data processing server 2 is analyzed. That is, information about how data to which specific attribute information is assigned is distributed in the disk storages 7 of the plurality of data processing servers 2 is acquired.

The data arrangement control unit 104
The content of the task (data processing task) obtained by the task analysis unit 102 (processing execution content),
The distribution status of data acquired by the data arrangement status analysis unit 103;
Is used to determine data to be rearranged and its location.

  The rearrangement execution unit 105 rearranges the data of the data processing server 2 based on the determined placement location, based on the data placement target data determined by the data placement control unit 104.

  Note that the data management unit 101, task analysis unit 102, and rearrangement execution unit 105 include means for communicating with other devices constituting the system via the network 6.

  The data management unit 101, the task analysis unit 102, the data arrangement status analysis unit 103, the data arrangement control unit 104, and the rearrangement execution unit 105 in FIG. 2 may realize processing and functions by a program executed on the computer. Good.

Referring again to FIG. 1, the annotator 4 is
-Analyzing the content and positioning of the data stored in the disk storage 7 of the data processing server 2 to generate attribute information;
Update the attribute information data in the data arrangement / attribute information storage unit 100 via the data management unit 101 (see FIG. 2) of the data arrangement management device 1.
Etc. are provided.

The data creation client 5 stores the newly created data in the data processing server 2 with respect to the data management unit 101 (see FIG. 2) of the data arrangement management device 1.
・ New data creation request,
・ Data size,
By sending
・ Get the address information of the data placement destination,
Store data in a specified location in the disk storage 7 of the data processing server 2.

<Data processing task execution procedure>
Next, a basic execution procedure of a data processing task using the system of FIG. 1 will be described.

  The data processing task registered in the task broker 3 is transferred to all the data processing servers 2 constituting the system via the network 6 as a data processing command including a data processing target and data processing contents.

In the data processing server 2,
・ From the received data processing instruction, extract the attribute information used to specify the data processing target,
For the data arrangement management device 1,
・ Attribute information is given, and
Send a request to obtain a list of data stored in its own disk storage 7

In response to the data list acquisition request, the data management unit 101 of the data arrangement management device 1 receives the attribute information from the data arrangement / attribute information storage unit 100 based on the attribute information.
・ Attribute information is given, and
Stored in the disk storage 7 of the data processing server 2 that issued the data list acquisition request;
A list of data is acquired (referred to as “data list acquisition operation”) and transferred to the data processing server 2 that is the request issuer. The data list acquisition operation is executed in all the data processing servers 2 that have received the data processing command.

  The data processing server 2 includes all the data included in the data list received from the data arrangement management device 1 among the data stored in its own disk storage 7 in the data processing instruction. Data processing is executed based on the data processing contents.

  The data processing server 2 transmits the execution result to the task broker 3 after completing the execution of the given data processing instruction.

  The task broker 3 acquires the execution result of the data processing instruction transferred by itself from all the data processing servers 2 to which the instruction has been transferred, and then sets all the execution results as one execution result based on the data processing content. In summary, the execution result is returned to the data processing task client 8, and the data processing task is terminated.

<About attribute information associated with data stored in data processing server>
In the present embodiment, the attribute information stored in the data arrangement / attribute information storage unit 100 of the data arrangement management device 1 has, for example, the following data configuration.

<Key information>
Key information is data representing a classification name of attribute information. For example, if the data stored in the disk storage 7 of the data processing server 2 includes data relating to color, the classification name “color” is stored as the key information.

<Value information>
The value information is data representing a specific value of the key information. For example, when a classification name “color” is defined as key information, data representing classification values such as red, blue, and yellow is stored as value information.

  The key information and the value information are combined and defined as one attribute information.

  In addition, a plurality of attribute information can be given to the data stored in the disk storage 7 of the data processing server 2. However, when assigning attribute information, a plurality of value information cannot be assigned to one key information, and one piece of key information constituting one piece of attribute information assigned to data is assigned to one piece of key information. Value information is set.

  In the present embodiment, the attribute information is used as information for specifying the data processing target of the data processing task transferred from the task broker 3.

  In this embodiment, since it is possible to give a plurality of attribute information to data, when the attribute information specifying the data processing target is different in each of the plurality of data processing tasks. However, there is a possibility that the same data is subject to data processing. The same data may be processed by the first data processing task with respect to the first attribute information and the second data processing task with respect to the second attribute information.

<Relocation procedure for data stored in data processing server>
In the present embodiment, a procedure for rearranging data between data processing servers 2 using the data placement management device 1 will be described with reference to the flowchart of FIG. 3 and FIGS. 4 to 6.

  In the following, in the system shown in FIG. 1, three data processing servers 2 with node numbers 1 to 3 are provided, and as shown in FIG. 4, two types of Key information each having two types of Value information are all An example will be described in which an environment is given as attribute information to data, that is, an environment in which a plurality of data processing tasks are executed on a data group based on a total of four types of data classification methods.

  The data arrangement control unit 104 of the data arrangement management device 1 acquires the data processing task instruction group registered in the task broker 3 and scheduled to be executed in the future, and the analysis instruction to the task analysis unit 102. put out.

  The task analysis unit 102 acquires a data processing task instruction group from the task broker 3 based on the instruction received from the data arrangement control unit 104, and identifies a data group that is a target of data processing from the data processing task instruction group. Extract attribute information.

  Furthermore, the task analysis unit 102 extracts key information and value information of all the acquired attribute information, and transfers them to the data arrangement control unit 104 (step 200).

  The data placement control unit 104 stores data having each attribute information together with a list of key information and value information acquired from the task analysis unit 102 with respect to the data placement state analysis unit 103, respectively. The data volume distribution ratio table creation command in the disk storage 7 is transferred.

  The data arrangement state analysis unit 103 extracts the address information and the data size of all data corresponding to the given key information and value information from the data arrangement / attribute information storage unit 100, and for each attribute information, A data capacity ratio distribution table (106 in FIG. 5) that summarizes the capacity ratio of the data having attribute information between the data processing servers 2 is created and transferred to the data arrangement control unit 104 (step 201).

  As described above, in the data arrangement / attribute information storage unit 100, the address information of the data stored in the disk storage 7 of the data processing server 2, the attribute information associated with the data, and the data size of the data Are stored in association with each other, the data arrangement status analysis unit 103 simply refers to the information stored in the data arrangement / attribute information storage unit 100 and transmits data between the disk storages 7 of the data processing server 2. A capacity ratio distribution table can be created.

  The data capacity ratio distribution table 106 of FIG. 5 shows that, for example, the data with Key 1 value information A is 30%, 40%, 30% between the nodes 1, 2, and 3, and the data with Value information B is node 1, 2 and 3 indicate 50%, 30%, and 20%.

  Based on the data capacity ratio distribution table 106 created by the data arrangement status analysis unit 103, the data arrangement control unit 104 selects a data group that has been determined in advance by a system administrator or the like from the data group related to each attribute information. It is determined whether or not there is any applicable condition for the placement execution target (step 202).

  For the applicable condition, for example, a threshold value based on a difference in data capacity ratio between the data processing servers 2 is used, and when the threshold value is exceeded, it is determined that the target condition is a data relocation target. Although not particularly limited, in this embodiment, the threshold value in the example used in the description of this procedure is set based on whether or not the maximum ratio value of the data capacity to be stored exceeds twice the minimum ratio value. Shall.

  If there is a data group that matches the corresponding condition to be subjected to data rearrangement, the data placement control unit 104 starts to consider rearrangement for one data group that satisfies the corresponding condition.

  In the case of the data capacity ratio distribution table 106 shown in FIG. 5, the data capacity stored in the node 1 is 2 of the data capacity stored in the node 3 for the data group to which the attribute information B of Key1 is assigned. Since the number exceeds twice (2.5 times), the data group is subject to relocation examination.

  The data arrangement control unit 104 transfers a data capacity distribution table creation command to the data arrangement state analysis unit 103.

  The data arrangement status analysis unit 103 creates a data capacity distribution table 107 as shown in FIG. 6 using the data stored in the data arrangement / attribute information storage unit 100 and transfers it to the data arrangement control unit 104 ( Step 203).

Note that the data capacity distribution table is created for each data group related to all combinations of attribute information acquired from the task broker 3. The number of combinations is given by the following, where X is the number of key information acquired from the task broker 3 and A (i) is the value of the value corresponding to the i-th key information acquired from the task broker 3 (however, , Π is the sum (product) operator).

のデータ群に分けることができる。 Furthermore, if the number of data processing servers is defined as N, the data capacity distribution table is

Can be divided into data groups.

In the example of the attribute information shown in FIG. 4, 2 × 2 × 3 = 12, that is,
A and C, A and D, B and C, B and D
The four types of data are distributed over three data processing servers.

  Therefore, it can be divided into 2 × 2 × 3 = 12 data groups at maximum, and the data capacity distribution table 107 as shown in FIG. 6 can be created.

  The data arrangement control unit 104 uses the data capacity distribution table 107 acquired from the data arrangement state analysis unit 103 and the data capacity ratio distribution table 106 used in step 202 to determine a data group to be rearranged and its data group. Narrow down the relocation destinations.

  In narrowing down the data group to be rearranged and the placement destination at that time, the following (1) to (4) are evaluated, the data to be rearranged, the rearrangement source, and the rearrangement destination data processing server 2 (Step 204).

(1) <Capacity ratio to be rearranged using data capacity ratio distribution table and evaluation of data relocation source and relocation destination>
The user or system administrator sets the threshold of the data capacity ratio achieved after the relocation by performing the data relocation operation on the data to which the attribute information to be relocated is assigned. The capacity ratio of the data group to be rearranged is determined in advance so that the data capacity ratio is obtained.

  In determining the data rearrangement source and the data rearrangement destination, the data processing server that occupies the maximum ratio is the data processing server that occupies the maximum ratio, and the data processing server that occupies the minimum ratio. Relocation destination.

  As described above, it is possible to determine from which data processing server to which data processing server the data of the data group to be rearranged is to be rearranged.

  In the example used for the description of the above procedure, the user or the system administrator sets the capacity ratio difference after rearrangement so that the minimum value of the data capacity ratio is 1.5 times or less of the maximum value. Assuming that it is set, from the data capacity ratio distribution table 106 of FIG. It can be seen that the data processing server 2 may be rearranged to the data processing server 2 with the node number 3.

(2) <Evaluation of rearrangement capacity based on evaluation of (1) using data capacity distribution table>
From the capacity data in the data capacity distribution table, it is necessary to calculate the total capacity of the data to which the attribute information to be relocated is assigned, and to actually relocate from the data capacity ratio obtained in the evaluation of (1) Calculate the data capacity.

  Further, a data group that matches the rearrangement target, rearrangement source, and rearrangement destination extracted by the evaluation in (1) is extracted.

  In the case of the example of the data capacity distribution table 107 shown in FIG. 6, since the total capacity of the data capacity to which the attribute information of Key B of Key1 to be rearranged is 600 GB, 600 × 0.08 = It can be seen that the data having a capacity of 48 GB may be rearranged from the data processing server 2 with the node number 1 to the data processing server with the node number 3.

  Further, since the data group to which the attribute information B of Key1 is assigned is further assigned the attribute information of C of Key2 or the attribute information of D of Key2, candidate 1 in FIG. Two data groups indicated as candidate 2 can be extracted.

(3) <Evaluation of relocation target and relocation source and relocation destination based on evaluation of (2) using data capacity ratio distribution table and data capacity distribution table>
By executing the data rearrangement using the data capacity ratio distribution table in the data group of the rearrangement candidates extracted by the evaluation of (2), the capacity ratio relating to other related attribute information is referred to, and the data By executing the rearrangement, a candidate that does not open the difference between the minimum value and the maximum value of the data capacity ratio is selected.

  When the difference between the minimum value and the maximum value of the data capacity ratio opens for all relocation candidates, the actual capacity ratio between the nodes after data relocation is calculated using the data capacity distribution table 107. , (1) used in the evaluation, a candidate that does not generate a difference equal to or larger than a threshold value indicating a difference between the maximum value and the minimum value of the capacity ratio and that opens the difference most is selected.

  For the rearrangement candidate, when the difference between the minimum value and the maximum value of the data capacity ratio does not open, the actual capacity ratio between the nodes after the data rearrangement is calculated using the data capacity distribution table 107, A candidate that minimizes the difference in capacity ratio is selected.

  In the case of the example used in the description of this procedure, the case of relocating the data group (candidate 1) to which B of Key1 and C of Key2 are assigned to the data processing server 2 of node number 1 to node number 3 is 5, the difference between the minimum value and the maximum value of the capacity ratio opens from the data capacity ratio distribution table 106 in FIG.

  On the other hand, the case of relocating the data group (candidate 2) to which B of Key1 and D of Key2 are assigned to the data processing server 2 from node number 1 to node number 3 is shown in the data capacity ratio distribution table 106 of FIG. Since the difference between the minimum value and the maximum value of the capacity ratio is reduced, it is determined as the final relocation target. From the above processing, as the extraction condition of data to be rearranged, B of Key1 and D of Key2 are assigned as attribute information, the rearrangement data capacity is 48 GB, and the data processing server of the rearrangement source A server with node number 3 is extracted as a server with node number 1 and a data processing server to be relocated.

  The data arrangement control unit 104 is extracted by the evaluations (1) to (3) described above.

・ Extraction conditions for data to be relocated,
・ Relocation data capacity,
・ Relocation source,
-Based on the information of the relocation destination, with respect to the relocation target data stored in the data arrangement / attribute information storage unit 100, a flag indicating the relocation target, the relocation source, and the relocation destination Is added (step 205).

  The data arrangement control unit 104 issues a request to create the data capacity ratio distribution table 106 to the data arrangement state analysis unit 103, and uses the data stored in the data arrangement / attribute information storage unit 100 to The data capacity ratio distribution table 106 in the data arrangement after the rearrangement execution set in step 1 is created (step 201), and another data group in which the difference between the minimum value and the maximum value of the data capacity ratio is equal to or greater than the threshold value. It is confirmed whether or not there is (step 202).

  If there is a data group in which the difference between the minimum value and the maximum value of the data capacity ratio is greater than or equal to the threshold value, the operations in steps 203 to 205 are performed again on the data group.

  The data capacity distribution table 107 and the data capacity ratio distribution table 106 created during the subsequent operations are created based on the address information after rearrangement determined from the rearrangement evaluation for other data groups. I will decide.

  In the present embodiment, in the rearrangement evaluation for a plurality of data groups, the data that has been subject to rearrangement in the evaluation of a certain data group is again the rearrangement target in the rearrangement evaluation of the subsequent data group. If it becomes, the data in the data arrangement / attribute information storage unit 100 is updated to the data of the rearrangement source and the rearrangement destination based on the latest evaluation result.

  Accordingly, when evaluating the rearrangement of a certain data group, for example, the data planned to be rearranged from the node 1 to the node 3 is changed from the node 3 to the node 1 when evaluating the rearrangement of another data group. If the data is planned to be rearranged and finally planned to arrange data at the current arrangement location, the data arrangement / attribute information storage unit 100 indicates that the corresponding data is a relocation target. Clear the flag.

  If there is no data group in which the difference between the minimum value and the maximum value of the data capacity ratio is equal to or greater than the threshold (there is no data group to be rearranged, step 202 in FIG. No branch), the rearrangement execution instruction is transferred to the rearrangement execution unit 105.

  The relocation execution unit 105 extracts address information and relocation destination information for all data flagged as a relocation target in the data allocation / attribute information storage unit 100, and stores the data on the data processing server 2. Relocation is executed (step 206).

According to this embodiment,
・ Multiple attribute information is given to the data,
・ There are multiple data processing tasks that determine the data to be processed based on the attribute information.
A plurality of data processing servers execute the data processing in parallel by dividing the data stored in the disk storage of each data processing server itself.
In that environment,
Create a data replica from the latest data for each data processing task, and execute parallel processing of multiple different data processing tasks in the data processing server group without creating a data processing server for parallel processing. It is possible. In other words, according to the present embodiment, replica creation costs and an extra data processing server are not required.

  In other words, according to the present embodiment, it is possible to realize a platform for executing parallel processing of data and a data storage platform, which have been separately constructed in the past, on a single platform, thereby greatly reducing costs. Can be expected.

  Furthermore, according to the present embodiment, even in an environment in which attribute information given to data is dynamically changed, parallel processing of data using data in the latest state is always performed in parallel as much as possible. It is possible to execute in a form that takes advantage of the effect.

  According to the present embodiment, in a system that performs data storage capacity leveling among a plurality of storages, even if data is leveled in relation to one attribute information, data related to another attribute information The deviation of the data storage capacity between the plurality of storages increases on the contrary, and the occurrence of a situation in which the overall data processing performance is deteriorated can be suppressed.

  In the embodiment shown in FIG. 1, the data processing server 2 extracts attribute information used for designating the data processing target from the data processing instruction received from the task broker 3, and sends it to the data arrangement management device 1. A request to obtain a list of data stored in the disk storage 7 to which the attribute information is assigned is transmitted, and the data stored in the disk storage 7 is included in the data list received from the data arrangement management device 1. However, the present invention is of course not limited to such a configuration. For example, the task broker 3 and the data arrangement management device 1 are connected to each other or integrated, and the task broker 3 extracts the attribute information used for designating the data processing target, passes it to the data arrangement management device 1, and the data The arrangement management apparatus 1 refers to the data arrangement / attribute information storage unit 100, acquires a list of data related to attribute information stored in the disk storage 7 of each data processing server 2, and then stores the data The processing instruction and the data list may be transmitted to the corresponding data processing server 2 respectively.

  In this embodiment, the data storage location (address information), data size, and attribute information are individually stored and managed by each data processing server 2, and each data processing is appropriately performed by the data arrangement management device 1. The data address information, data size, and attribute information association information in the server 2 is aggregated to update the data arrangement / attribute information storage unit 100 to create a data capacity distribution table, ratio table, etc. May be included as an example. In this case, the data processing server 2 may receive the attribute information created by the annotator 4 and manage it in association with the data.

  Data rearrangement may be performed on a regular basis (for example, every day at midnight), or may be distributed (partially) over a period of low access frequency based on data access monitoring results, etc. You may make it perform. At this time, the storage may be preferentially performed from a storage with low access frequency.

  In the present embodiment, as the data processing task, for example, processing such as data search, data reading and calculation related to read data is performed, and data writing (updating) is performed by the data creation client 5. In the configuration shown in FIG. 1, the data processing task client 8 and the data creation client 5 are separated, but may be a single device. In this case, when the annotator 4 determines that the content of the data processing is data creation processing, it creates attribute information.

  In the present embodiment, the data rearrangement evaluation method in the data arrangement control unit 104 is not limited to the above method, and the optimization of the distribution of the plurality of data regarding the plurality of attribute information among the plurality of data storage devices. Any other optimization method can be applied as (optimization to approximate a distribution situation suitable for data processing). For example, the first and second attribute information are XY orthogonal coordinate axes, the node numbers (discrete values 1, 2, 3,...) Of a plurality of data processing servers are represented by respective coordinate values, and the X and Y coordinate planes are represented. When the data capacity (or ratio) at a certain position is represented by a three-dimensional display with the Z axis, the optimization of the data arrangement is uniform in the unevenness of the Z axis value at the discrete grid points on the (X, Y) plane. It can be grasped as a process of flattening. General n-dimensional attribute information and distribution status is an n + 1-dimensional model. When the number of attribute information and the number of data processing servers increase, mathematical programming is used to solve this kind of problem. In the relocation plan, the relocation source and the relocation destination are not limited to one node each. For example, data relocation from one relocation source to a plurality of relocation destinations, a plurality of relocations One rearrangement destination or the like is possible from the beginning, and the rearrangement data may be divided and weighted for distribution.

  Although the present invention has been described with reference to the above-described embodiments, the present invention is not limited to the configurations of the above-described embodiments, and various modifications that can be made by those skilled in the art within the scope of the present invention. Of course, including modifications.

1 shows an example of a system configuration according to an embodiment of the present invention. 1 shows an example of the configuration of a data arrangement management device according to an embodiment of the present invention. It is a flowchart for demonstrating the data rearrangement procedure of one Embodiment of this invention. It is a figure which shows the example of the attribute information provided to data in one Embodiment of this invention. It is a figure which shows an example of the data capacity | capacitance ratio distribution table classified according to the cut face of one attribute information in one Embodiment of this invention. It is an example of the data capacity | capacitance distribution table classified according to the cut face of several attribute information in one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Data arrangement management apparatus 2 Data processing server 3 Task broker 4 Annotator 5 Data creation client 6 Network 7 Disk storage 8 Data processing task client 100 Data arrangement / attribute information storage part 101 Data management part 102 Task analysis part 103 Data arrangement state analysis part 104 Data Arrangement Control Unit 105 Relocation Execution Unit 106 Data Capacity Ratio Distribution Table 107 Data Capacity Distribution Table

Claims (22)

With respect to a plurality of data and a plurality of data storage devices each assigned a plurality of attribute information, address information indicating a storage location of the data, size information of the data, and attribute information provided to the data Information storage means for storing and managing in association;
Means for referring to the information storage means for obtaining distribution status of data capacity in the plurality of data storage devices of the data to which the attribute information is attached;
When it is determined that the distribution status of the data capacity acquired with respect to at least one attribute information among a plurality of attribute information meets a predetermined data relocation condition,
When the data is rearranged, the distribution of the data capacity among the plurality of data storage devices regarding the one attribute information not only satisfies a predetermined condition regarding equalization, but at least of the plurality of attribute information Means for deriving a rearrangement plan so as to satisfy a predetermined condition regarding homogenization with respect to a distribution of data capacity among the plurality of data storage devices related to one other attribute information;
Means for relocating data among the plurality of data storage devices according to the derived relocation plan;
A data arrangement management device comprising: With respect to a plurality of data and a plurality of data storage devices each assigned a plurality of attribute information, the attribute information assigned to the data, address information indicating a location where the data is stored, and the size of the data Information storage means for storing and managing information in association with each other;
Means for calculating a ratio distribution of data capacity among the plurality of data storage devices of the data associated with the attribute information from the attribute information designated as the data processing target, with reference to the information storage means;
From the latest situation of the data capacity ratio distribution among the plurality of data storage devices, regarding the plurality of attribute information, the distribution of the data capacity of the data associated with each attribute information between the plurality of data storage devices is uniform. Means to rearrange the data so as to approach the conditions for
A data arrangement management device comprising: A plurality of data processing devices corresponding respectively to the plurality of data storage devices;
Data processing related to attribute information designated as a data processing target, which is performed by individually accessing a data storage device corresponding to each, is performed in parallel between the plurality of data processing devices. The data arrangement management apparatus according to claim 1 or 2. An apparatus for managing data arrangement of a plurality of data processing servers,
A data arrangement / attribute information storage unit that stores and holds address information of data stored in each data storage device of each of the plurality of data processing servers, attribute information associated with the data, and data size in association with each other When,
A data management unit that receives a new data creation request, a request to add or change attribute information, and updates information stored in the data arrangement / attribute information storage unit;
A task analysis unit that analyzes the contents of data processing tasks;
With reference to the data arrangement / attribute information storage unit, in the plurality of data processing servers, how the data related to the plurality of attribute information designated as data processing targets is distributed in the plurality of data processing servers. A data arrangement status analysis unit that acquires information on whether or not
With respect to a plurality of designated attribute information, the distribution of data capacity among the plurality of data storage devices of the data associated with each attribute information is made closer to the condition for equalization by using the distribution status of the data. And a data placement control unit for determining data to be rearranged and a placement location thereof,
A rearrangement execution unit for rearranging data of the data processing server based on the determined placement location of the data placement target data determined by the data placement control unit;
A data arrangement management device comprising:   The data arrangement status analysis unit creates data volume ratio distribution information among the plurality of data processing servers with respect to keys and values constituting data attribute information as the data distribution status. 5. The data arrangement management apparatus according to claim 4, wherein   The data arrangement control unit uses a threshold value based on a difference in data capacity ratio between the data processing servers, and determines that attribute information data exceeding the threshold value corresponds to a data relocation target. The data arrangement management apparatus according to claim 4, wherein:   The said data arrangement | positioning condition analysis part produces the data capacity distribution table which is a list of a data processing server and a data capacity about all the combinations of the said attribute information given, The data capacity distribution table | surface characterized by the above-mentioned. Data placement management device. The data placement control unit determines a capacity ratio of a data group to be rearranged so as to satisfy a predetermined threshold with respect to a ratio of data capacity achieved after data rearrangement,
Regarding the distribution of the data capacity ratio obtained in the data arrangement situation analysis unit, when the data capacity ratio satisfies a predetermined condition,
The data processing server occupying the largest ratio is the relocation source,
6. The data arrangement management apparatus according to claim 5, wherein a data processing server occupying the minimum ratio is a relocation destination. The data placement control unit determines a capacity ratio of a data group to be rearranged so as to satisfy a predetermined threshold with respect to a ratio of data capacity achieved after data rearrangement,
From the capacity data in the data capacity distribution table, calculate the total capacity of the data to which the attribute information to be rearranged is attached,
Calculate the data capacity that must actually be relocated from the capacity ratio of the data group to be relocated,
Regarding the distribution of the data capacity ratio obtained in the data arrangement situation analysis unit, when the capacity ratio satisfies a predetermined condition,
The data processing server occupying the largest ratio is the relocation source,
8. The data arrangement management apparatus according to claim 7, wherein a data processing server that occupies the minimum ratio is set as a relocation destination, and a data group that matches the relocation target, the relocation source, and the relocation destination is extracted. The data placement control unit executes data rearrangement using the data capacity ratio distribution table among the rearrangement target and the rearrangement candidate data group extracted by the evaluation of the rearrangement source and the rearrangement destination. By referring to the capacity ratio for other related attribute information,
Select a relocation candidate that does not open the difference between the minimum and maximum data capacity ratio by executing data relocation,
About the relocation candidate
When the difference between the minimum value and the maximum value of the data capacity ratio opens, the actual capacity ratio between each node after data rearrangement is calculated using the data capacity distribution table,
Select a candidate that does not generate a difference greater than or equal to a predetermined threshold regarding the difference between the maximum value and the minimum value of the calculated capacity ratio, and that the difference is least open,
About the relocation candidate
When the difference between the minimum value and the maximum value of the data capacity ratio does not open, the actual capacity ratio between the nodes after data rearrangement is calculated using the data capacity distribution table, and the difference in capacity ratio is 10. The data arrangement management apparatus according to claim 9, wherein a candidate that is the smallest is selected. A task broker that transfers the data processing task to the data processing server via the network, integrates the data processing results executed in the data processing server, and completes the task;
Based on the data processing task from the task broker, a data processing server that executes a data processing task using storage data and returns an execution result to the task broker;
Means for managing attribute information attached to data, address information of a data storage device storing the data, and data size;
Means for acquiring attribute information designated as a data processing target in the data processing task;
The data arrangement management device according to any one of claims 4 to 10,
Annotator means for generating attribute information for data stored in the storage of the data processing server and updating attribute information data of the data arrangement management device;
When newly created data is stored in the data processing server, a new data creation request and a data size are transmitted to the data management unit of the data placement management device, and the address of the data placement destination Get information,
A data creation client for storing data in a designated location in the storage of the data processing server;
A distributed data processing system. With respect to a plurality of data and a plurality of data storage devices each assigned a plurality of attribute information, address information indicating a storage location of the data, size information of the data, and attribute information provided to the data A process of associating and storing in the storage unit;
A process of referring to the storage unit and acquiring a distribution state of data capacity in the plurality of data storage devices of data to which attribute information is attached;
When it is determined that the distribution status of the data capacity acquired with respect to at least one attribute information among a plurality of attribute information meets a predetermined data relocation condition,
When the data is rearranged, the distribution of the data capacity among the plurality of data storage devices related to the one attribute information not only satisfies a predetermined condition regarding equalization, but at least of the plurality of attribute information A process of deriving a rearrangement plan that satisfies a predetermined condition regarding the homogenization with respect to the distribution of data capacity among the plurality of data storage devices related to one other attribute information;
Relocating data among the plurality of data storage devices according to the derived relocation plan;
A program that causes a computer to execute. To a computer that manages the data arrangement of multiple data processing servers,
A process of associating and storing address information of data stored in each data storage device of the plurality of data processing servers, attribute information associated with the data, and data size in a storage unit;
A data management process for receiving a new data creation request, a request for adding or changing attribute information, and updating information stored in the data arrangement / attribute information storage unit;
Task analysis processing to analyze the contents of data processing tasks,
With reference to the storage unit, in the plurality of data processing servers, information on how data related to the plurality of attribute information designated as data processing targets is distributed in the plurality of data processing servers. Data placement status analysis processing to be acquired,
With respect to a plurality of designated attribute information, the distribution of data capacity among the plurality of data storage devices of the data associated with each attribute information is made closer to the condition for equalization by using the distribution status of the data. In addition, data placement control processing for determining data to be rearranged and its placement location,
Relocation execution processing for rearranging the data of the data processing server based on the determined location of the data allocation target data determined in the data allocation control processing;
A program that executes   14. The program according to claim 13, wherein the data arrangement status analysis processing is performed by calculating a data capacity ratio between the plurality of data processing servers with respect to keys and values constituting data attribute information as the data distribution status. A program characterized by creating distribution information.   14. The program according to claim 13, wherein the data placement control process uses a threshold value based on a difference in data capacity ratio between the data processing servers, and sets attribute information data exceeding the threshold value as a data relocation target. A program characterized by being judged to be applicable.   15. The program according to claim 14, wherein the data arrangement situation analysis processing creates a data capacity distribution table that is a list of data processing servers and data capacities for all combinations of the given attribute information. A featured program. 15. The program according to claim 14, wherein the data placement control process determines a capacity ratio of a data group to be rearranged so as to satisfy a predetermined threshold with respect to a ratio of data capacity achieved after data rearrangement,
Regarding the distribution of the data capacity ratio obtained by the data arrangement situation analysis process, when the data capacity ratio satisfies a predetermined condition,
The data processing server occupying the largest ratio is the relocation source,
A program characterized in that a data processing server occupying a minimum ratio is a relocation destination. The program according to claim 16, wherein the data placement control process determines a capacity ratio of a data group to be rearranged so as to satisfy a predetermined threshold with respect to a ratio of data capacity achieved after data rearrangement.
From the capacity data in the data capacity distribution table, calculate the total capacity of the data to which the attribute information to be rearranged is attached,
Calculate the data capacity that must actually be relocated from the capacity ratio of the data group to be relocated,
Regarding the distribution of the data capacity ratio obtained by the data arrangement situation analysis process, when the capacity ratio satisfies a predetermined condition,
The data processing server occupying the largest ratio is the relocation source,
A program characterized in that a data processing server occupying a minimum ratio is a relocation destination, and a data group that matches a relocation target, a relocation source, and a relocation destination is extracted. 19. The program according to claim 18, wherein the data arrangement control processing is performed by using the data capacity ratio distribution table in the data group of the rearrangement candidates and the rearrangement candidate data extracted by evaluating the rearrangement source and the rearrangement destination. Use the data relocation to see the capacity ratio for other related attribute information,
Select a relocation candidate that does not open the difference between the minimum and maximum data capacity ratio by executing data relocation,
About the relocation candidate
When the difference between the minimum value and the maximum value of the data capacity ratio opens, the actual capacity ratio between each node after data rearrangement is calculated using the data capacity distribution table,
Select a candidate that does not generate a difference greater than or equal to a predetermined threshold regarding the difference between the maximum value and the minimum value of the calculated capacity ratio, and that the difference is least open,
About the relocation candidate
When the difference between the minimum value and the maximum value of the data capacity ratio does not open, the actual capacity ratio between the nodes after data rearrangement is calculated using the data capacity distribution table, and the difference in capacity ratio is A program characterized by selecting a candidate that is the smallest. With respect to a plurality of data and a plurality of data storage devices each assigned a plurality of attribute information, address information indicating a storage location of the data, size information of the data, and attribute information provided to the data Associating and storing in the storage unit;
Referring to the storage unit, obtaining a distribution state of data capacity in the plurality of data storage devices of data to which attribute information is given;
When it is determined that the distribution status of the data capacity acquired with respect to at least one attribute information among a plurality of attribute information meets a predetermined data relocation condition,
When the data is rearranged, the distribution of the data capacity among the plurality of data storage devices regarding the one attribute information not only satisfies a predetermined condition regarding equalization, but at least of the plurality of attribute information A step of deriving a rearrangement plan that satisfies a predetermined condition regarding homogenization with respect to distribution of data capacity among the plurality of data storage devices related to one other attribute information;
Relocating data among the plurality of data storage devices according to the derived relocation plan;
A data arrangement management method comprising: With respect to data each having a plurality of attribute information, a plurality of data capacity distributions among a plurality of data storage devices for storing the data with respect to each of the plurality of attribute information are approximated to a predetermined distribution state. Means for deriving an optimal data arrangement that can be taken between the data storage devices;
Means for rearranging data according to the derived data arrangement;
A data arrangement management system characterized by comprising: A plurality of data processing devices corresponding respectively to the plurality of data storage devices;
Data processing related to attribute information designated as a data processing target, which is performed by individually accessing a data storage device corresponding to each, is performed in parallel between the plurality of data processing devices. The data arrangement management system according to claim 21.

Images