CN104363095A - Method for establishing hadoop identity authentication mechanism - Google Patents

Abstract

The invention discloses a method for establishing a hadoop identity authentication mechanism. The method comprises the following steps that a KDC server is established; a hafs principal, a mapred principal and an HTTP principal are established for each node in a hadoop cluster; a hafs.keytab file including the hdfs principals and the HTTP principals is established; a mapred.keytab file including the mapred principals and the HTTP principals is established. According to the method for establishing the hadoop identity authentication mechanism, the problems that in the prior art, hadoop access control is not safe, and cluster nodes can be simulated are solved.

Description

A Hadoop identity authentication mechanism construction method

technical field

The invention relates to the field of identity authentication, in particular to a method for constructing an identity authentication mechanism of hadoop.

Background technique

The era of big data provides many opportunities for Hadoop. Hadoop has been valued for its good scalability in massive data, efficient reading and writing and processing capabilities. However, behind many opportunities, Hadoop also faces many challenges. How to ensure the security of Hadoop has naturally become an unavoidable topic. Without access control, the data stored in it can be accessed at will or even perform misoperations such as deletion and modification, which will cause many potential risks. Therefore, Hadoop access control has become a very important requirement.

In a Hadoop environment without authentication, authentication is not required when users interact with Hadoop Distributed File System (HDFS) or M/R. Malicious users can pretend to be real users or servers to invade Hadoop clusters and submit mapreduce maliciously. Jobs, changing JobTracker status, modifying data on HDFS, etc.

At present, HDFS has added permission authentication for files and directories, but these can only protect against accidental data loss. Malicious users can easily pretend to be other users to modify permissions, resulting in the use of permission settings, which cannot guarantee the security of hadoop clusters.

The Kerberos protocol is mainly used for computer network authentication (Authentication). Its characteristic is that the user can access multiple services with the ticket (ticket-granting ticket) obtained by this verification only by entering the identity verification information once, that is, SSO (Single Sign On) . Since a shared key is established between each Client and Service, the protocol has considerable security.

Contents of the invention

The invention provides a hadoop identity authentication mechanism construction method, which is used to solve the problems in the prior art that the hadoop access control is not safe and cluster nodes may be impersonated.

In order to solve the above-mentioned technical problems, the present invention provides a kind of identity authentication mechanism construction method of hadoop, comprising the following steps: build key distribution center (KDC) server; Create hdfs principal, mapred principal and HTTP principal for all nodes in hadoop cluster respectively; Create a hdfs.keytab file containing hdfs principal and HTTP principal; create a mapred.keytab file containing mapred principal and HTTP principal.

Further, the hadoop authentication mechanism is implemented through the kerberos protocol.

Further, the hadoop identity authentication mechanism construction method also includes the following steps: use the merged hdfs.keytab file and mapred.keytab file to obtain a certificate, and deploy the hdfs.keytab file and mapred.keytab file, so that hdfs and mapred user can access.

The hadoop identity authentication mechanism construction method provided by the present invention provides a reliable, efficient and easy-to-operate identity authentication mechanism based on the kerberos protocol for the hadoop cluster, ensuring that malicious users cannot pretend to be real users or servers invade the hadoop cluster and submit mapreduce jobs , Changing the JobTracker status, modifying data on HDFS and other malicious operations greatly ensure the reliability and security of the hadoop cluster.

Description of drawings

FIG. 1 is a flow chart of a method for constructing an identity authentication mechanism of hadoop provided according to a preferred embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Take the following existing environment as an example: an available system package yum source (both local and network); an available three-node hadoop cluster environment, and the host names of the three nodes are: node01.test .com, node02.test.com, node03.test.com.

As shown in Figure 1, the specific steps are described as follows.

In step S1, the installation package required by the kerberos service is installed, and related configuration files are modified to build a key distribution center (KDC) server. Specifically, all nodes install krb5-workstation, krb5-workstation and their dependent packages, and KDC nodes also install krb5-server.

In step S2, a new principal database is created for use by the hadoop cluster. Wherein, principal represents a basic entity (for example, a client or a server) participating in kerberos authentication.

In step S3, change the relevant configuration of kerberos, and create an administrator for kerberos remote management. The details are as follows: change the configuration file /etc/krb5.conf; change the configuration file /var/kerberos/krb5kdc/kdc.conf; change the configuration file /var/kerberos/krb5kdc/kadm5.acl; and copy /etc/krb5.conf to the same directory on other nodes for replacement.

In step S4, principals are created for all nodes in the hadoop cluster. Among them, including creating hdfs principal, mapred principal and HTTP principal.

Among them, the way to create hdfs principal is as follows:

#kadmin.local

Kadmin:addprinc-randkey hdfs/node01.test.comHADOOP

Kadmin:addprinc-randkey hdfs/node02.test.comHADOOP

Kadmin:addprinc-randkey hdfs/node03.test.comHADOOP

The way to create a mapred principal is as follows:

#kadmin.local

Kadmin:addprinc-randkey mapred/node01.test.comHADOOP

Kadmin:addprinc-randkey mapred/node02.test.comHADOOP

Kadmin:addprinc-randkey mapred/node03.test.comHADOOP

The way to create HTTP principal is as follows:

#kadmin.local

Kadmin:addprinc-randkey HTTP/node01.test.comHADOOP

Kadmin:addprinc-randkey HTTP/node02.test.comHADOOP

Kadmin:addprinc-randkey HTTP/node03.test.comHADOOP

In step S5, a corresponding keytab is created. Here, create a hdfs.keytab file containing hdfs principal and HTTP principal, and create a mapred.keytab file containing mapred principal and HTTP principal.

Create hdfs.keytab file containing hdfs principal and HTTP principal, as follows:

kadmin:xst-norandkey-k hdfs.keytab hdfs/node01.test.comHADOOPHTTP/node01.test.comHADOOP

kadmin:xst-norandkey-k hdfs.keytab hdfs/node02.test.comHADOOPHTTP/node02.test.comHADOOP

kadmin:xst-norandkey-k hdfs.keytabhdfs/node03.test.comHADOOPHTTP/node03.test.comHADOOP

Create a mapred.keytab file containing mapred principal and HTTP principal, as follows:

kadmin:xst-norandkey-k mapred.keytabmapred/node01.test.comHADOOP HTTP/node01.test.comHADOOP

kadmin:xst-norandkey-k mapred.keytabmapred/node02.test.comHADOOP HTTP/node02.test.comHADOOP

kadmin:xst-norandkey-k mapred.keytabmapred/node03.test.comHADOOP HTTP/node03.test.comHADOOP

In step S6, use the merged keytab to obtain the certificate in the following manner:

#kinit–k–t hdfs.keytab hdfs/node01.inspur.comHADOOP

#kinit–k–t mapred.keytab mapred/node01.inspur.comHADOOP

In step S7, the kerberos keytab file is deployed so that it can be accessed by hdfs and mapred users. Specifically, move the hdfs.keytab generated in step S5 to $HADOOP_HOME/conf, and move the mapred.keytab generated in step S5 to $HADOOP_HOME/conf

In step S8, add kerberos self-starting and restarting services.

In step S9, all services of the hadoop cluster are stopped.

In step S10, change hadoop-related configuration files so that the hadoop cluster uses the kerberos protocol for identity authentication.

Specifically, add or modify the following in $HADOOP_HOME/core-site.xml:

<property>

<name>hadoop.security.authentication</name>

<value>kerberos</value>

</property>

<property>

<name>hadoop.security.authorization</name>

<value>true</value>

</property>

Add the following to $HADOOP_HOME/conf/hdfs-site.xml:

<property>

<name>dfs.namenode.keytab.file</name>

<value>$HADOOP_HOME/conf/hdfs.keytab</value>

</property>

<property>

<name>dfs.namenode.kerberos.principal</name>

<value>hdfs/_HOSTHADOOP</value>

</property>

<property>

<name>dfs.namenode.kerberos.internal.spnego.principal</name>

<value>HTTP/_HOSTHADOOP</value>

</property>

<property>

<name>dfs.secondary.namenode.keytab.file</name>

<value>$HADOOP_HOME/conf/hdfs.keytab</value>

</property>

<property>

<name>dfs.secondary.namenode.kerberos.principal</name>

<value>hdfs/_HOSTHADOOP</value>

</property>

<property>

<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>

<value>HTTP/_HOSTHADOOP</value>

</property>

<property>

<name>dfs.datanode.keytab.file</name>

<value>$HADOOP_HOME/conf/hdfs.keytab</value>

</property>

<property>

<name>dfs.datanode.kerberos.principal</name>

<value>hdfs/_HOSTHADOOP</value>

</property>

<property>

<name>dfs.datanode.kerberos.internal.spnego.principal</name>

<value>HTTP/_HOSTHADOOP</value>

</property>

Add the following to $HADOOP_HOME/conf/mapred-site.xml:

<property>

<name>mapreduce.jobtracker.kerberos.principal</name>

<value>mapred/_HOSTHADOOP</value>

</property>

<property>

<name>mapreduce.jobtracker.kerberos.https.principal</name>

<value>host/_HOSTHADOOP</value>

</property>

<property>

<name>mapreduce.jobtracker.keytab.file</name>

<value>$HADOOP_HOME/conf/mapred.keytab</value>

</property>

<property>

<name>mapreduce.tasktracker.kerberos.principal</name>

<value>mapred/_HOSTHADOOP</value>

</property>

<property>

<name>mapreduce.tasktracker.kerberos.https.principal</name>

<value>host/_HOSTHADOOP</value>

</property>

<property>

<name>mapreduce.tasktracker.keytab.file</name>

<value>$HADOOP_HOME/conf/mapred.keytab</value>

</property>

And copy core-site.xml, hdfs-site.xml, mapred-site.xml, hdfs.keytab, mapred.keytab under $HADOOP_HOME to corresponding directories of other nodes.

In step 11, restart the hadoop cluster to verify that all functions are normal. If the startup is successful, the identity authentication mechanism of Hadoop is successfully constructed.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. The present invention is not limited by the above-mentioned embodiments, and what described in the above-mentioned embodiments and the description only illustrates the principle of the present invention, and without departing from the spirit and scope of the present invention, the present invention also has various changes and improvements, these changes All modifications and improvements are within the scope of the claimed invention.

Claims (3)

1. a hadoop identity authentication mechanism construction method, is characterized in that, comprises the following steps: Build a Key Distribution Center (KDC) server; Create hdfs principal, mapred principal and HTTP principal for all nodes in the hadoop cluster respectively; Create hdfs.keytab file containing hdfs principal and HTTP principal; Create mapred.keytab file containing mapred principal and HTTP principal. 2. the identity authentication mechanism construction method of hadoop as claimed in claim 1, is characterized in that: the identity authentication mechanism of described hadoop is accomplished by kerberos protocol. 3. The identity authentication mechanism construction method of hadoop as claimed in claim 1, is characterized in that, also comprises the following steps: use the hdfs.keytab file after merging and mapred.keytab file to obtain certificate, and deploy hdfs.keytab file and mapred .keytab file to be accessible by hdfs and mapred users.