CN107566496A - A kind of hadoop date storage methods and device - Google Patents

Abstract

The embodiment of the invention discloses a kind of hadoop date storage methods, including：When receiving the data storage request of user's submission, namenode NameNode randomly selects the back end DataNode of predetermined number from multiple different frames；Obtain the data trnascription quantity that each DataNode is currently deposited in network topology to current DataNode range information and each DataNode in the DataNode of predetermined number；Each DataNode scheduling evaluation of estimate is calculated according to the range information and data trnascription quantity；Data storage node is chosen according to the scheduling evaluation of estimate calculated.The embodiment of the invention also discloses a kind of hadoop data storage devices.By scheme of the embodiment of the present invention, the load balancing of data storage and good data transmission performance are realized.

Description

A hadoop data storage method and device

technical field

Embodiments of the present invention relate to data storage technology, in particular to a hadoop data storage method and device.

Background technique

With the development of Internet and distributed computing technology, there are more and more data-intensive applications, these applications often need to involve several terabytes of data, how to process large amounts of data efficiently, reliably and conveniently It has become an important research direction at present, and how to store massive data reliably and reasonably is an important issue in Hadoop (a distributed system infrastructure) system. Hadoop stores multiple copies of data on different machines in the cluster. When a node fails, it can still read the data. But on the other hand, since the operation in MapReduce often needs to input a large amount of data, and the movement of a large amount of data will significantly affect the performance of the operation, the storage of data should follow the principle of locality, that is, the data should be closer to the operation node, so that Reduce performance loss due to data movement. The current copy placement strategy of HDFS (Hadoop Distributed File System) is as follows: if the writer is on a data node DataNode, then the first copy is on the local machine, otherwise a node is randomly selected. The second copy is first on another rack, and the third copy is placed on the same rack as the second copy, but on a different data node. This scheme has the following problems: the nodes of the randomly selected rack may increase unnecessary data recovery time due to being too far away from the local node, and at the same time, randomly selecting nodes cannot guarantee the balance of data storage between nodes. Since the failure of nodes in the system is normal, unnecessary performance loss during data recovery will lead to performance degradation of the entire storage system.

Contents of the invention

In order to solve the above technical problems, an embodiment of the present invention provides a hadoop data storage method, which can realize load balancing of data storage and good data transmission performance.

In order to achieve the purpose of the embodiment of the present invention, the embodiment of the present invention provides a hadoop data storage method, including:

When receiving the data storage request submitted by the user, the name node NameNode randomly selects a preset number of data nodes DataNode from multiple different racks;

Obtain the distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes and the number of data copies currently stored by each DataNode;

Calculate the scheduling evaluation value of each DataNode according to the distance information and the number of data copies;

Select data storage nodes according to the calculated scheduling evaluation value.

Optionally, when receiving a data storage request submitted by a user, the name node NameNode randomly selects a preset number of data nodes DataNodes from multiple different racks, including:

According to the data storage request, the NameNode invokes the preset copy placement policy BlockPlacementPolicy; wherein, the node selection function chooseTarget() of the BlockPlacementPolicy adds a network topology NetworkTopology class member variable clusterMap;

Obtain random DataNodes from multiple different racks according to the node selection function Node chooseRandom(String scope) of clusterMap.

Optionally, obtaining distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes includes:

Obtain the network distance between each DataNode and the current DataNode according to the target distance function intgetDistance(Node node1, Node node2) of clusterMap.

Optionally, obtaining the number of data copies currently stored by each DataNode includes:

Call the data node description strategy DataNodeDescriptor describing DataNode in the Hadoop system;

According to the block data group function intnumBlocks() in DataNodeDescriptor, the number of data blocks stored on each DataNode is obtained as the number of data copies.

Optionally, calculating the scheduling evaluation value of each DataNode according to the distance information and the number of data copies includes: substituting the obtained distance information about each DataNode and the number of data copies into the following preset node evaluation function to calculate each DataNode Scheduling evaluation value E:

E=f(ld,d)=Al+(1-A)d;

Among them, f(ld,d) is the node evaluation function; 1 is the load factor of the DataNode, which is inversely proportional to the number of data copies currently stored in the DataNode; d is the distance coefficient, which is inversely proportional to the network distance from the DataNode to the current DataNode; A∈ [0,1] is the balance factor.

Optionally, selecting a data storage node according to the calculated scheduling evaluation value includes:

Sort the calculated scheduling evaluation values;

Select data storage nodes from high to low according to the scheduling evaluation value.

In order to achieve the purpose of the embodiment of the present invention, the embodiment of the present invention also provides a hadoop data storage device, including: a selection module, an acquisition module, a calculation module and a storage module;

The selection module is used to randomly select a preset number of data nodes DataNodes from multiple different racks when receiving a data storage request submitted by a user;

The obtaining module is used to obtain the distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes and the number of data copies currently stored by each DataNode;

A calculation module, configured to calculate the scheduling evaluation value of each DataNode according to the distance information and the number of data copies;

The storage module is used to select a data storage node according to the calculated scheduling evaluation value.

Optionally, when receiving a data storage request submitted by a user, the selection module randomly selects a preset number of data nodes DataNodes from multiple different racks, including:

According to the data storage request, the selection module invokes the preset copy placement policy BlockPlacementPolicy; among them, the node selection function chooseTarget() of the BlockPlacementPolicy adds the network topology NetworkTopology class member variable clusterMap;

Obtain random DataNodes from multiple different racks according to the node selection function Node chooseRandom(String scope) of clusterMap.

Optionally, the acquisition module acquires the distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes, including:

Obtain the network distance between each DataNode and the current DataNode according to the target distance function intgetDistance(Node node1, Node node2) of clusterMap.

Optionally, the number of data copies currently stored by each DataNode obtained by the obtaining module includes:

Call the data node description strategy DataNodeDescriptor describing DataNode in the Hadoop system;

According to the block data group function intnumBlocks() in DataNodeDescriptor, the number of data blocks stored on each DataNode is obtained as the number of data copies.

Optionally, the calculation module calculates the scheduling evaluation value of each DataNode according to the distance information and the number of data copies, including: substituting the obtained distance information about each DataNode and the number of data copies into the following preset node evaluation function to calculate the each DataNode's scheduling evaluation value E:

E=f(ld,d)=Al+(1-A)d;

Among them, f(ld,d) is the node evaluation function; l is the load factor of the DataNode, which is inversely proportional to the number of data copies currently stored in the DataNode; d is the distance coefficient, which is inversely proportional to the network distance from the DataNode to the current DataNode; A∈ [0,1] is the balance factor.

Optionally, the storage module selecting a data storage node according to the calculated scheduling evaluation value includes:

Sort the calculated scheduling evaluation values;

Select data storage nodes from high to low according to the scheduling evaluation value.

The embodiment of the present invention includes: when receiving the data storage request submitted by the user, the name node NameNode randomly selects a preset number of data nodes DataNode from a plurality of different racks; obtains the data node of each DataNode in the preset number of DataNodes in the network topology The distance information to the current DataNode and the number of data copies currently stored in each DataNode; the scheduling evaluation value of each DataNode is calculated according to the distance information and the number of data copies; the data storage node is selected according to the calculated scheduling evaluation value. Through the solution of the embodiment of the present invention, the load balance of data storage is realized, and good data transmission performance is realized.

Additional features and advantages of the embodiments of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the invention. The objectives and other advantages of the embodiments of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and accompanying drawings.

Description of drawings

The accompanying drawings are used to provide a further understanding of the technical solutions of the embodiments of the present invention, and constitute a part of the description, and are used together with the embodiments of the application to explain the technical solutions of the embodiments of the present invention, and do not constitute limitations to the technical solutions of the embodiments of the present invention .

Fig. 1 is the hadoop data storage method flowchart of the embodiment of the present invention;

Fig. 2 is the hadoop data storage method schematic diagram of the embodiment of the present invention;

FIG. 3 is a block diagram of a hadoop data storage device according to an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

The steps shown in the flowcharts of the figures may be performed in a computer system, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

In order to achieve the purpose of the embodiment of the present invention, the embodiment of the present invention provides a Hadoop data storage method, as shown in Figure 1, including S101-S104:

S101. When receiving a data storage request submitted by a user, the name node NameNode randomly selects a preset number of data nodes DataNodes from multiple different racks.

In the embodiment of the present invention, aiming at the placement problem of HDFS data block copy, a selection strategy based on evaluation value is proposed. This strategy combines node distance and data load to improve data block copy placement, so as to improve the storage performance of the system and Balance the utilization of node storage resources.

In the embodiment of the present invention, as shown in FIG. 2, when a user submits a request for data storage, the NameNode randomly selects a certain number of nodes of multiple different racks, such as the multiple DataNodes shown in box 1 in FIG. 2, The specific number of selected DataNodes, that is, the preset number mentioned above, can be defined according to different application scenarios, and no specific limitation is set here.

In the embodiment of the present invention, the policy proposed in the embodiment of the present invention can be implemented on Hadoop, mainly on Hadoop-2.6, and mainly implement the abstract class BlockPlacementPolicy. This class is responsible for selecting the target data node for each copy of a data block, and BlockPlacementPolicy will be called whenever a data block storage request is submitted. Based on the BlockPlacementPolicy, a preset number of data nodes DataNodes can be randomly selected from multiple different racks.

Optionally, when receiving the data storage request submitted by the user, the name node NameNode randomly selects a preset number of data nodes DataNodes from multiple different racks, including S201-S202:

S201. According to the data storage request, the NameNode invokes the preset replica placement policy BlockPlacementPolicy; wherein, the node selection function chooseTarget() of the BlockPlacementPolicy adds a network topology NetworkTopology class member variable clusterMap;

S202. Obtain random DataNodes from multiple different racks according to the node selection function Node chooseRandom(String scope) of the clusterMap.

In the embodiment of the present invention, the most important method of the BlockPlacementPolicy is chooseTarget (), which is directly responsible for selecting the DataNode node when storing the data block. And in order to obtain DataNode data load and network distance information, the method chooseTarget() can be rewritten to realize the following functions: add the member variable clusterMap of the NetworkTopology class to this class, and this member is instantiated in the constructor of the BlockPlacementPolicy class. Use the member method Node chooseRandom(String scope) of clusterMap to obtain a preset number of random DataNodes.

S102. Obtain the distance information from each DataNode in the network topology to the current DataNode among the preset number of DataNodes and the number of data copies currently stored by each DataNode.

In the embodiment of the present invention, in order to further evaluate each obtained DataNode, it is necessary to obtain the distance information from each DataNode to the current DataNode in the network topology and the number of data copies currently stored by each DataNode.

Optionally, obtaining distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes includes:

Obtain the network distance between each DataNode and the current DataNode according to the target distance function intgetDistance(Node node1, Node node2) of clusterMap.

In the embodiment of the present invention, based on the above S201, the member method intgetDistance(Node node1, Node node2) of clusterMap can be further used to obtain the network distance between the new DataNode and the current DataNode. In the function intgetDistance(Node node1, Node node2), Node node1 and Node node2 refer to two DataNodes whose network distance is to be calculated respectively.

Optionally, obtaining the number of data copies currently stored by each DataNode may include:

Call the data node description strategy DataNodeDescriptor describing DataNode in the Hadoop system;

According to the block data group function intnumBlocks() in DataNodeDescriptor, the number of data blocks stored on each DataNode is obtained as the number of data copies.

In the embodiment of the present invention, the method intnumBlocks() provided by the class DataNodeDescriptor describing the DataNode in the Hadoop system can obtain the number of data blocks stored on a specific DataNode (which can be used to represent the current load on the DataNode).

S103. Calculate the scheduling evaluation value of each DataNode according to the distance information and the number of data copies.

In the embodiment of the present invention, based on the distance information obtained in the previous steps and the number of data copies, the preset node evaluation function can be substituted to calculate the scheduling evaluation value E of each DataNode.

Optionally, calculating the scheduling evaluation value of each DataNode according to the distance information and the number of data copies includes: substituting the obtained distance information about each DataNode and the number of data copies into the following preset node evaluation function to calculate each DataNode Scheduling evaluation value E:

E=f(ld,d)=Al+(1-A)d;

Among them, f(ld,d) is the node evaluation function; l is the load factor of the DataNode, which is inversely proportional to the number of data copies currently stored in the DataNode; d is the distance coefficient, which is inversely proportional to the network distance from the DataNode to the current DataNode; A∈ [0,1] is the balance factor.

In the embodiment of the present invention, A is used to describe the data load (that is, the number of data copies currently stored by the DataNode) and the proportion of the network distance parameter in the evaluation, which can be determined by the system administrator according to the system's requirements for load balancing and data transmission performance Specify a value.

S104. Select a data storage node according to the calculated scheduling evaluation value.

Optionally, selecting a data storage node according to the calculated scheduling evaluation value may include:

Sort the calculated scheduling evaluation values;

Select data storage nodes from high to low according to the scheduling evaluation value.

In the embodiment of the present invention, after these two aspects of information are substituted into the evaluation function to calculate the scheduling evaluation value of each DataNode node, such as E1, E2, E3, E4, and E5 shown in Figure 2, the calculated scheduling evaluation value can be calculated. The value is sorted, and the data storage node is selected according to the scheduling evaluation value from high to low. For example, by selecting the DataNode with the highest evaluation value as the data placement node, the data placement node with a better overall balance between data load and network distance is selected, thereby optimizing the storage of data blocks.

In order to achieve the purpose of the embodiment of the present invention, the embodiment of the present invention also provides a hadoop data storage device 1. It should be noted that any of the above-mentioned method embodiments are applicable to the device embodiment, and will not be repeated here. . As shown in Figure 3, the device includes: a selection module 11, an acquisition module 12, a calculation module 13 and a storage module 14;

The selection module 11 is configured to randomly select a preset number of data nodes DataNode from a plurality of different racks when receiving a data storage request submitted by a user;

The obtaining module 12 is used to obtain the distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes and the number of data copies currently stored by each DataNode;

Calculation module 13, for calculating the scheduling evaluation value of each DataNode according to the distance information and the number of data copies;

The storage module 14 is configured to select a data storage node according to the calculated scheduling evaluation value.

Optionally, when receiving a data storage request submitted by a user, the selection module 11 randomly selects a preset number of data nodes DataNodes from multiple different racks, including:

According to the data storage request, the selection module invokes the preset copy placement policy BlockPlacementPolicy; among them, the node selection function chooseTarget() of the BlockPlacementPolicy adds the network topology NetworkTopology class member variable clusterMap;

Obtain random DataNodes from multiple different racks according to the node selection function Node chooseRandom(String scope) of clusterMap.

Optionally, the acquisition module acquires the distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes, including:

Obtain the network distance between each DataNode and the current DataNode according to the target distance function intgetDistance(Node node1, Node node2) of clusterMap.

Optionally, obtaining the number of data copies currently stored by each DataNode by the obtaining module 12 includes:

Call the data node description strategy DataNodeDescriptor describing DataNode in the Hadoop system;

According to the block data group function intnumBlocks() in DataNodeDescriptor, the number of data blocks stored on each DataNode is obtained as the number of data copies.

Optionally, the calculation module 13 calculates the scheduling evaluation value of each DataNode according to the distance information and the number of data copies, including: substituting the acquired distance information about each DataNode and the number of data copies into the following preset node evaluation function to calculate the each Scheduling evaluation value E of a DataNode:

E=f(ld,d)=Al+(1-A)d;

Among them, f(ld,d) is the node evaluation function; l is the load factor of the DataNode, which is inversely proportional to the number of data copies currently stored in the DataNode; d is the distance coefficient, which is inversely proportional to the network distance from the DataNode to the current DataNode; A∈ [0,1] is the balance factor.

Optionally, the storage module 14 selecting a data storage node according to the calculated scheduling evaluation value includes:

Sort the calculated scheduling evaluation values;

Select data storage nodes from high to low according to the scheduling evaluation value.

The embodiment of the present invention includes: when receiving the data storage request submitted by the user, the name node NameNode randomly selects a preset number of data nodes DataNode from a plurality of different racks; obtains the data node of each DataNode in the preset number of DataNodes in the network topology The distance information to the current DataNode and the number of data copies currently stored in each DataNode; the scheduling evaluation value of each DataNode is calculated according to the distance information and the number of data copies; the data storage node is selected according to the calculated scheduling evaluation value. Through the solution of the embodiment of the present invention, the load balance of data storage is realized, and good data transmission performance is realized.

In the embodiment of the present invention, on the one hand, the solution of this embodiment can shorten the average distance of data block storage and reduce the storage time; on the other hand, it can also reduce the imbalance of storage load between data nodes, especially in a Realize the average data block storage of each data node within. This method integrates the two factors of network distance and data load of data storage nodes, which can not only realize load balance of data storage, but also achieve good data transmission performance.

Although the implementation manner disclosed in the embodiment of the present invention is as above, the content described is only the implementation manner adopted for understanding the present invention, and is not intended to limit the embodiment of the present invention. Any person skilled in the field of the embodiments of the present invention can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed by the embodiments of the present invention, but the embodiments of the present invention The scope of patent protection must still be subject to the scope defined in the appended claims.

Claims (10)

1. a hadoop data storage method, is characterized in that, comprises: When receiving the data storage request submitted by the user, the name node NameNode randomly selects a preset number of data nodes DataNode from multiple different racks; Obtain the distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes and the number of data copies currently stored by each DataNode; calculating the scheduling evaluation value of each DataNode according to the distance information and the number of data copies; A data storage node is selected according to the calculated scheduling evaluation value. 2. hadoop data storage method according to claim 1, is characterized in that, described when receiving the data storage request that user submits, name node NameNode randomly selects the data node DataNode of preset quantity from a plurality of different racks include: According to the data storage request, the NameNode calls the preset copy placement policy BlockPlacementPolicy; wherein, the node selection function chooseTarget() of the BlockPlacementPolicy is added with a network topology NetworkTopology class member variable clusterMap; Random DataNodes are obtained from the plurality of different racks according to the node selection function Node chooseRandom(String scope) of the clusterMap. 3. hadoop data storage method according to claim 2, is characterized in that, each DataNode in the DataNode of described preset quantity described obtaining in network topology to the distance information of current DataNode comprises: Obtain the network distance between each DataNode and the current DataNode according to the target distance function intgetDistance(Node node1, Node node2) of the clusterMap. 4. hadoop data storage method according to claim 1, is characterized in that, the data copy quantity that described obtaining each DataNode deposits currently comprises: Call the data node description strategy DataNodeDescriptor describing DataNode in the Hadoop system; According to the block data group function intnumBlocks() in the DataNodeDescriptor, the number of data blocks stored on each DataNode is obtained as the number of data copies. 5. hadoop data storage method according to claim 1, is characterized in that, described according to described distance information and described data copy number calculation described each DataNode scheduling evaluation value comprises: about each DataNode that will acquire The distance information and the number of data copies are substituted into the following preset node evaluation function to calculate the scheduling evaluation value E of each DataNode: E=f(ld,d)=Al+(1-A)d; Among them, f(ld,d) is the node evaluation function; l is the load factor of the DataNode, which is inversely proportional to the number of data copies currently stored in the DataNode; d is the distance coefficient, which is inversely proportional to the network distance from the DataNode to the current DataNode; A∈ [0,1] is the balance factor. 6. hadoop data storage method according to claim 1, is characterized in that, the described scheduling evaluation value selection data storage node according to calculating comprises: Sort the calculated scheduling evaluation values; Select data storage nodes from high to low according to the scheduling evaluation value. 7. a hadoop data storage device, is characterized in that, comprises: select module, obtain module, calculation module and storage module; The selection module is used to randomly select a preset number of data nodes DataNodes from multiple different racks when receiving a data storage request submitted by a user; The obtaining module is used to obtain the distance information from each DataNode in the network topology to the current DataNode in the preset number of DataNodes and the number of data copies currently stored by each DataNode; The calculation module is configured to calculate the scheduling evaluation value of each DataNode according to the distance information and the number of data copies; The storage module is configured to select a data storage node according to the calculated scheduling evaluation value. 8. hadoop data storage method according to claim 7, is characterized in that, described when receiving the data storage request that user submits, selecting module randomly selects the data node DataNode of preset quantity from a plurality of different racks and comprises : According to the data storage request, the selection module calls the preset copy placement policy BlockPlacementPolicy; wherein, the node selection function chooseTarget() of the BlockPlacementPolicy is added with a network topology NetworkTopology class member variable clusterMap; Random DataNodes are obtained from the plurality of different racks according to the node selection function Node chooseRandom(String scope) of the clusterMap. 9. hadoop data storage method according to claim 8, is characterized in that, described obtaining module obtains each DataNode in the DataNode of described preset quantity in network topology to the distance information of current DataNode and comprises: Obtain the network distance between each DataNode and the current DataNode according to the target distance function intgetDistance(Node node1, Node node2) of the clusterMap. 10. hadoop data storage method according to claim 7, is characterized in that, described obtaining module obtains the data copy quantity that each DataNode deposits currently and comprises: Call the data node description strategy DataNodeDescriptor describing DataNode in the Hadoop system; According to the block data group function intnumBlocks() in the DataNodeDescriptor, the number of data blocks stored on each DataNode is obtained as the number of data copies.