in DevOps, Hadoopery

Kerberos kt_renewer failures with HUE on CDH4

First off, I’m not exactly sure if this is a Hadoop User Environment (HUE) issue or if this is a broken setup on my Kerberos environment.

I have a thread open on the HUE users list, but haven’t had any followup.

I’ve just fired up HUE for the first time to talk with a kerberos-enabled HA cluster on CDH4.4 and I find that HUE’s kt_renewer is failing with:

[04/Apr/2014 14:19:40 +0000] kt_renewer   INFO     Reinitting kerberos
from keytab: kinit -k -t /etc/local_keytabs/hue/hue.keytab -c
/tmp/hue_krb5_ccache hue/aus-hue1.example.net
[04/Apr/2014 14:19:42 +0000] kt_renewer   INFO     Renewing kerberos
ticket to work around kerberos 1.8.1: kinit -R -c /tmp/hue_krb5_ccache
[04/Apr/2014 14:19:42 +0000] kt_renewer   ERROR    Couldn't renew
kerberos ticket in order to work around Kerberos 1.8.1 issue. Please
check that the ticket for 'hue/aus-hue1.example.net' is still
renewable:
  $ kinit -f -c /tmp/hue_krb5_ccache
If the 'renew until' date is the same as the 'valid starting' date,
the ticket cannot be renewed. Please check your KDC configuration, and
the ticket renewal policy (maxrenewlife) for the
'hue/aus-hue1.example.net' and `krbtgt' principals.

It appears that kinit isn’t even being called with the correct options to enable renewable tickets and there’s no way that I can find to make this happen by default with MIT Kerberos (it appears that Heimdal might just do the right thing). kinit -R will always fail unless you’ve previously called kinit -r

I’m running CDH4.4.0 on Centos6.5 with the following krb5 packages loaded:

krb5-libs-1.10.3-10.el6_4.6.x86_64
krb5-devel-1.10.3-10.el6_4.6.x86_64
krb5-workstation-1.10.3-10.el6_4.6.x86_64

The kerberos master is Centos6.3 with the following krb5 packages loaded:

krb5-devel-1.9-33.el6_3.3.x86_64
krb5-server-1.9-33.el6_3.3.x86_64
krb5-libs-1.9-33.el6_3.3.x86_64
krb5-workstation-1.9-33.el6_3.3.x86_64

/etc/krb5.conf on the kerberos master and the client contain

[libdefaults]
    ticket_lifetime = 3d
    max_renewable_life = 7d
    renewable = true

The HUE principal is allowed to request renewable tickets. If I run kinit the way kt_renewer runs, I get the following:

[hue@aus-hue1 ~]$ kinit -k -t /etc/local_keytabs/hue/hue.keytab -c
    /tmp/hue_krb5_ccache hue/aus-hue1.example.net
[hue@aus-hue1 ~]$ klist -c /tmp/hue_krb5_ccache
Ticket cache: FILE:/tmp/hue_krb5_ccache
Default principal: hue/aus-hue1.example.net@EXAMPLE.NET

Valid starting     Expires            Service principal
04/04/14 14:29:57  04/07/14 14:29:57  krbtgt/EXAMPLE.NET@EXAMPLE.NET

If I run it explicitly requesting a renewable ticket, I get:

[hue@aus-hue1 ~]$ kinit -k -t /etc/local_keytabs/hue/hue.keytab -c
    /tmp/hue_krb5_ccache hue/aus-hue1.example.net -r 4day
[hue@aus-hue1 ~]$ klist -c /tmp/hue_krb5_ccache
Ticket cache: FILE:/tmp/hue_krb5_ccache
Default principal: hue/aus-hue1.example.net@EXAMPLE.NET

Valid starting     Expires            Service principal
04/04/14 14:57:34  04/07/14 14:57:34  krbtgt/EXAMPLE.NET@EXAMPLE.NET
        renew until 04/08/14 14:57:34

I can make hue do the right thing if I patch:

diff -u /usr/share/hue/desktop/core/src/desktop/kt_renewer.py /tmp/kt_renewer.py
--- /usr/share/hue/desktop/core/src/desktop/kt_renewer.py 2014-04-04
14:19:19.505933419 -0500
+++ /tmp/kt_renewer.py 2014-04-04 15:00:51.515295166 -0500
@@ -32,6 +32,7 @@
           "-k", # host ticket
           "-t", CONF.HUE_KEYTAB.get(), # specify keytab
           "-c", CONF.CCACHE_PATH.get(), # specify credentials cache
+          "-r", "7day",
           CONF.HUE_PRINCIPAL.get()]
   LOG.info("Reinitting kerberos from keytab: " +
            " ".join(cmdv))

But, this hardcodes a value into the src, which seems wrong. What’s the correct way to get this working on Centos6 without having to make this patch? I’m not exactly sure yet. I’ll update if I find a suitable solution.

Travis Campbell
Staff Systems Engineer at ghostar
Travis Campbell is a seasoned Linux Systems Engineer with nearly two decades of experience, ranging from dozens to tens of thousands of systems in the semiconductor industry, higher education, and high volume sites on the web. His current focus is on High Performance Computing, Big Data environments, and large scale web architectures.