Hadoop Kerberos的那些坑
顾亮亮
2015.11.03

Presenter Notes

Agenda

  • Kerberos Introduction
  • Token Expired Problem when Hadoop meet Kerberos
  • How Spark solve the Problem
  • Hadoop Kerberos Programming API

Presenter Notes

Kerberos (百度百科)

Kerberos这一名词来源于希腊神话 三个头的狗——地狱之门守护者

Presenter Notes

Kerberos (Wikipedia)

Kerberos /ˈkərbərəs/ is a computer network authentication protocol which works on the basis of 'tickets' to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner.

Presenter Notes

Authentication(认证) VS Authorization(授权)

  • Authentication

    is the act of confirming the truth of an attribute of a single piece of data claimed true by an entity.

  • Authorization

    is the function of specifying access rights to resources related to information security and computer security in general and to access control in particular.

Authentication is about who somebody is.

Authorisation is about what they're allowed to do.

Presenter Notes

Step 1: Authentication Service - TGT Delivery

TGT: Ticket Granting Ticket From: The MIT Kerberos Administrator’s How-to Guide

Presenter Notes

Step 2: Ticket Granting Service - TGS Delivery

TGS: Ticket Granting Service

Presenter Notes

Diagrams

Presenter Notes

Overal

Presenter Notes

Token Expired Problem when Hadoop meet Kerberos

Presenter Notes

Hadoop Authentication Flow

From: Hadoop Security Design

Presenter Notes

HDFS Authentication (Case 1: Sigle JVM Application)

  1. 用Keytab从KDC拿到TGT,进而拿到对应NameNode的TGS
  2. 用TGS向NameNode做验证,进而拿到BlocksToken
  3. 用BlocksToken从DataNode读取数据

Block Token

  • TokenID = {expirationDate, keyID, ownerID, blockID, accessModes}
  • TokenAuthenticator = HMAC-SHA1(key, TokenID)
  • Block Access Token = {TokenID, TokenAuthenticator}

Presenter Notes

HDFS Authentication (Case 2: Yarn Application)

Yarn上运行的Executor如何做验证?

  1. client用Keytab从KDC拿到TGT,进而拿到对应NameNode的TGS
  2. client用TGS向NameNode做验证,然后获取HDFS Delegation Token
  3. client把HDFS Delegation Token发送到各个Executor
  4. Executor用HDFS Delegation Token向NameNode获取BlocksToken
  5. Executor用BlocksToken从DataNode读取数据

HDFS Delegation Token

  • TokenID = {ownerID, renewerID, issueDate, maxDate, sequenceNumber}
  • TokenAuthenticator = HMAC-SHA1(masterKey, TokenID)
  • Delegation Token = {TokenID, TokenAuthenticator}

Presenter Notes

Token Expired Problem when Hadoop meet Kerberos

  1. keytab (long term)
  2. TGT (max life time = 3 days)
  3. TGS for NameNode (max life time = 3 days)
  4. HDFS Delegation Token (max life time = 7 days)
  5. Block Access Token (shot time)

Proglem: Applications on Yarn cannot run > 7 days

Presenter Notes

How Spark solve the Problem

Presenter Notes

Old Design

Problem: cannot run > 7 days on Yarn

Presenter Notes

New Design from Spark-1.4.0

From: Long Running Spark Apps on Secure YARN

Upload Keytab to HDFS and login from Keytab before token expired

Spark-5342 Allow long running Spark apps to run on secure YARN/HDFS

Presenter Notes

Presenter Notes

Hadoop Kerberos Programming API

Presenter Notes

UserGroupInformation (Hadoop)

 1 // Log a user in from a keytab file. Loads a user identity from a keytab
 2 // file and logs them in. They become the currently logged-in user.
 3 static void loginUserFromKeytab(String user, String path)
 4 
 5 // Log a user in from a keytab file. Loads a user identity from a keytab
 6 // file and login them in. This new user does not affect the currently
 7 static UserGroupInformation loginUserFromKeytabAndReturnUGI(String user,
 8                                                             String path)
 9 
10 // Return the current user, including any doAs in the current stack.
11 static UserGroupInformation getCurrentUser()
12 
13 // Re-login a user from keytab if TGT is expired or is close to expiry.
14 void checkTGTAndReloginFromKeytab()
15 
16 // Add the given Credentials to this user.
17 public void addCredentials(Credentials credentials)
18 
19 public Credentials getCredentials()
20 
21 // Run the given action as the user.
22 public <T> T doAs(PrivilegedAction<T> action)

Presenter Notes

UserGroupInformation Runtime

1 UserGroupInformation.loginUserFromKeytab(principal, keytab)
2 val ugi = UserGroupInformation.getCurrentUser

Presenter Notes

Subject (Java)

A Subject represents a grouping of related information for a single entity, such as a person. Such information includes the Subject's identities as well as its security-related attributes (passwords and cryptographic keys, for example).

Presenter Notes

Subject Runtime

1 val ugi = UserGroupInformation.getCurrentUser

Presenter Notes

Subject.doAs

 1 /**
 2  * Perform work as a particular Subject.
 3  *
 4  * This method first retrieves the current Thread's
 5  * AccessControlContext via AccessController.getContext,
 6  * and then instantiates a new AccessControlContext
 7  * using the retrieved context along with a new
 8  * SubjectDomainCombiner (constructed using the provided Subject).
 9  * Finally, this method invokes AccessController.doPrivileged,
10  * passing it the provided PrivilegedAction,
11  * as well as the newly constructed AccessControlContext.
12  *
13  * @param subject the Subject that the specified
14  *        action will run as.  This parameter may be null.
15  */
16 public static <T> T doAs(final Subject subject,
17                     final java.security.PrivilegedAction<T> action)

Presenter Notes

Use Cases

Presenter Notes

Kerberos Configuration

1 System.setProperty("java.security.krb5.realm", "HADOOP.QIYI.COM")
2 System.setProperty("java.security.krb5.kdc", "hadoop-kdc01")

Presenter Notes

Use Case 1: kinit by crontab

Run once every day in crontab

1 kinit -l 3d -k -t ~/test.keytab test@HADOOP.QIYI.COM

Right:

1 val fs = FileSystem.get(new Configuration())
2 while (true) {
3   println(fs.listFiles(new Path("/user"), false))
4   Thread.sleep(60 * 1000)
5 }

Kerberos will relogin automatically, when token is expired

Presenter Notes

Use Case 2: Login with Keytab File

Right:

1 UserGroupInformation.loginUserFromKeytab(principal, keytab)
2 val fs = FileSystem.get(new Configuration())
3 while (true) {
4   println(fs.listFiles(new Path("/user"), false))
5   Thread.sleep(60 * 1000)
6 }

Kerberos will relogin automatically, when token is expired

Wrong:

1 UserGroupInformation.loginUserFromKeytab(principal, keytab)
2 val fs = FileSystem.get(new Configuration())
3 while (true) {
4   UserGroupInformation.loginUserFromKeytab(principal, keytab)
5   println(fs.listFiles(new Path("/user"), false))
6   Thread.sleep(60 * 1000)
7 }

Do NOT call UserGroupInformation.loginUserFromKeytab again

Presenter Notes

Use Case 3: Multi-User in Signle JVM

Wrong:

 1 val ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal,
 2                                                               keytab)
 3 ugi.doAs(new PrivilegedExceptionAction[Void] {
 4   override def run(): Void = {
 5     val fs = FileSystem.get(new Configuration())
 6     while (true) {
 7       println(fs.listFiles(new Path("/user"), false))
 8       Thread.sleep(60 * 1000)
 9     }
10     null
11   }
12 })

Kerberos will NOT relogin automatically, when token is expired

Presenter Notes

Use Case 3: Muilti-User in Signle JVM (CONT.)

Right:

 1 val ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal,
 2                                                               keytab)
 3 ugi.doAs(new PrivilegedExceptionAction[Void] {
 4   override def run(): Void = {
 5     val fs = FileSystem.get(new Configuration())
 6     while (true) {
 7       UserGroupInformation.getCurrentUser.reloginFromKeytab()
 8       println(fs.listFiles(new Path("/user"), false))
 9       Thread.sleep(60 * 1000)
10     }
11     null
12   }
13 })

Call reloginFromKeytab before token is expired

Presenter Notes

Use Case 4: Using HDFS Delegation Token (Yarn Application)

Right:

 1 val creds = new org.apache.hadoop.security.Credentials()
 2 val ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal,
 3                                                                 keytab)
 4 ugi.doAs(new PrivilegedExceptionAction[Void] {
 5   override def run(): Void = {
 6     val fs = FileSystem.get(new Configuration())
 7     fs.addDelegationTokens("test", creds)
 8     null
 9   }
10 })
11 UserGroupInformation.getCurrentUser.addCredentials(creds)

Update credentials periodically before token expired

Presenter Notes