Kerberos kt_renewer failures with HUE on CDH4

First off, I’m not exactly sure if this is a Hadoop User Environment (HUE) issue or if this is a broken setup on my Kerberos environment.

I have a thread open on the HUE users list, but haven’t had any followup.

I’ve just fired up HUE for the first time to talk with a kerberos-enabled HA cluster on CDH4.4 and I find that HUE’s kt_renewer is failing with:

[04/Apr/2014 14:19:40 +0000] kt_renewer   INFO     Reinitting kerberos
from keytab: kinit -k -t /etc/local_keytabs/hue/hue.keytab -c
/tmp/hue_krb5_ccache hue/aus-hue1.example.net
[04/Apr/2014 14:19:42 +0000] kt_renewer   INFO     Renewing kerberos
ticket to work around kerberos 1.8.1: kinit -R -c /tmp/hue_krb5_ccache
[04/Apr/2014 14:19:42 +0000] kt_renewer   ERROR    Couldn't renew
kerberos ticket in order to work around Kerberos 1.8.1 issue. Please
check that the ticket for 'hue/aus-hue1.example.net' is still
renewable:
  $ kinit -f -c /tmp/hue_krb5_ccache
If the 'renew until' date is the same as the 'valid starting' date,
the ticket cannot be renewed. Please check your KDC configuration, and
the ticket renewal policy (maxrenewlife) for the
'hue/aus-hue1.example.net' and `krbtgt' principals.

It appears that kinit isn’t even being called with the correct options to enable renewable tickets and there’s no way that I can find to make this happen by default with MIT Kerberos (it appears that Heimdal might just do the right thing). kinit -R will always fail unless you’ve previously called kinit -r

I’m running CDH4.4.0 on Centos6.5 with the following krb5 packages loaded:

krb5-libs-1.10.3-10.el6_4.6.x86_64
krb5-devel-1.10.3-10.el6_4.6.x86_64
krb5-workstation-1.10.3-10.el6_4.6.x86_64

The kerberos master is Centos6.3 with the following krb5 packages loaded:

krb5-devel-1.9-33.el6_3.3.x86_64
krb5-server-1.9-33.el6_3.3.x86_64
krb5-libs-1.9-33.el6_3.3.x86_64
krb5-workstation-1.9-33.el6_3.3.x86_64

/etc/krb5.conf on the kerberos master and the client contain

[libdefaults]
    ticket_lifetime = 3d
    max_renewable_life = 7d
    renewable = true

The HUE principal is allowed to request renewable tickets. If I run kinit the way kt_renewer runs, I get the following:

[hue@aus-hue1 ~]$ kinit -k -t /etc/local_keytabs/hue/hue.keytab -c
    /tmp/hue_krb5_ccache hue/aus-hue1.example.net
[hue@aus-hue1 ~]$ klist -c /tmp/hue_krb5_ccache
Ticket cache: FILE:/tmp/hue_krb5_ccache
Default principal: hue/aus-hue1.example.net@EXAMPLE.NET

Valid starting     Expires            Service principal
04/04/14 14:29:57  04/07/14 14:29:57  krbtgt/EXAMPLE.NET@EXAMPLE.NET

If I run it explicitly requesting a renewable ticket, I get:

[hue@aus-hue1 ~]$ kinit -k -t /etc/local_keytabs/hue/hue.keytab -c
    /tmp/hue_krb5_ccache hue/aus-hue1.example.net -r 4day
[hue@aus-hue1 ~]$ klist -c /tmp/hue_krb5_ccache
Ticket cache: FILE:/tmp/hue_krb5_ccache
Default principal: hue/aus-hue1.example.net@EXAMPLE.NET

Valid starting     Expires            Service principal
04/04/14 14:57:34  04/07/14 14:57:34  krbtgt/EXAMPLE.NET@EXAMPLE.NET
        renew until 04/08/14 14:57:34

I can make hue do the right thing if I patch:

diff -u /usr/share/hue/desktop/core/src/desktop/kt_renewer.py /tmp/kt_renewer.py
--- /usr/share/hue/desktop/core/src/desktop/kt_renewer.py 2014-04-04
14:19:19.505933419 -0500
+++ /tmp/kt_renewer.py 2014-04-04 15:00:51.515295166 -0500
@@ -32,6 +32,7 @@
           "-k", # host ticket
           "-t", CONF.HUE_KEYTAB.get(), # specify keytab
           "-c", CONF.CCACHE_PATH.get(), # specify credentials cache
+          "-r", "7day",
           CONF.HUE_PRINCIPAL.get()]
   LOG.info("Reinitting kerberos from keytab: " +
            " ".join(cmdv))

But, this hardcodes a value into the src, which seems wrong. What’s the correct way to get this working on Centos6 without having to make this patch? I’m not exactly sure yet. I’ll update if I find a suitable solution.

Mass-gzip files inside HDFS using the power of Hadoop

I have a bunch of text files sitting in HDFS that I need to compress. It’s on the order of several hundred files comprising several hundred gigabytes of data. There are several ways to do this.

  1. I could individually copy down each file, compress it, and re-upload it to HDFS. This takes an excessively long amount of time.
  2. I could run a hadoop streaming job that turns on mapred.output.compress and mapred.compress.map.output, sets my mapred.output.compress.codec to org.apache.hadoop.io.compress.GzipCodec, and then just cats the output.
  3. I could create a shell script that gets uploaded via a hadoop streaming job that copies each file down to the local data node where the task is executing, runs gzip, and re-uploads it.

Option 1 was a no-go from the start. It would have taken days to complete the operation and I would have lost the opportunity to learn something new.

Option 2 was attempted, but what I found was that the inputs were split along their block size, causing the resulting file to turn into a multiple gzipped parts. This was a no-go because I needed each file assembled back in one piece. Additionally, I still ended up having to run one mapreduce job per file, which was going to take about a day the way I had written it.

Option 3 was attempted because it could easily guarantee that the files would be compressed quickly, in parallel, and end up coming back out of the mapreduce job as one file per input. That’s exactly what I needed.

To do this, I needed to do several things.

First, create the input file list.

$ hadoop fs -ls /tmp/filedir/*.txt > gzipped.out

Next, I create a simple shell script to invoke as the map task by the hadoop streaming job. The script looks like this:

#!/bin/sh -e
set -xv
while read dummy filename ; do
        echo "Reading $filename"
        hadoop fs -copyToLocal $filename .
        base=`basename $filename`
        gzip $base
        hadoop fs -copyFromLocal ${base}.gz /tmp/jobvis/${base}.gz
done

A short breakdown of what this is doing:

The input to the map task is going to be fed to us by the hadoop streaming job. The input has two columns, the key and the filename. We don’t care about the key, so we’re just going to ignore it. Next, we copy the file from hadoop to the current working directory for the task on the datanode’s local disk location for mapreduce. In this case, it ended up somewhere in /hdfs/mapred/local/ttprivate/taskTracker. Since we’re operating on a filepath, we need to get the basename so we can have gzip operate on it once it’s in the local temporary directory. Once gzip is complete, we upload it back to hadoop.

This particular cluster runs simple authentication, so the jobs actually run as the mapred user. Because of this, the files that are actually getting written down into the local datanode temporary directory will be owned by the mapred user. The directory that we want to upload the data back to needs to have access from the mapred user to create the uploaded data.

Note: it’s probably best to run the shell script with -e so that if any operation fails, the task will fail. Using set -xv will also allow some better output in the mapreduce task logs so you can see what the script is doing during the run.

Next, we create the output directory on HDFS.

$ hadoop fs -mkdir /tmp/jobvis
$ hadoop fs -chmod 777 /tmp/jobvis

Once that’s done, we want to run the hadoop streaming job. I ran it like this. There’s a lot of output here. I include it only for reference.

$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-*.jar 
  -Dmapred.reduce.tasks=0 -mapper gzipit.sh 
  -input ./gzipped.txt 
  -output /user/hcoyote/gzipped.log 
  -verbose 
  -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat 
  -file gzipit.sh

Important to note on this command line: org.apache.hadoop.mapred.lib.NLineInputFormat is the magic here. It basically tells the job to feed one file per maptask. This allows gzipit.sh to run once per file and gain the parallelism of running on all available map slots on the cluster. One thing I should have done was to turn off speculative execution. Since we’re creating a specific output file, I was seeing some tasks preemptively fail because the output had already been produced.

Then it just looks like this:

STREAM: addTaskEnvironment=
STREAM: shippedCanonFiles_=[/home/hcoyote/gzipit.sh]
STREAM: shipped: true /home/hcoyote/gzipit.sh
STREAM: cmd=gzipit.sh
STREAM: cmd=null
STREAM: cmd=null
STREAM: Found runtime classes in: /tmp/hadoop-hcoyote/hadoop-unjar3549927095616185719/
packageJobJar: [gzipit.sh, /tmp/hadoop-hcoyote/hadoop-unjar3549927095616185719/] [] /tmp/streamjob6232490588444082861.jar tmpDir=null
JarBuilder.addNamedStream gzipit.sh
JarBuilder.addNamedStream org/apache/hadoop/streaming/DumpTypedBytes.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/UTF8ByteArrayUtils.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/Environment.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamKeyValUtil.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PathFinder.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeReducer.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamBaseRecordReader.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/JarBuilder.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeMapRed.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/TypedBytesOutputReader.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/IdentifierResolver.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/TextOutputReader.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/RawBytesOutputReader.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/TextInputWriter.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/InputWriter.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/OutputReader.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/TypedBytesInputWriter.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/io/RawBytesInputWriter.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeMapRed$MROutputThread.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/AutoInputFormat.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamUtil$TaskId.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeCombiner.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeMapRunner.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamUtil$StreamConsumer.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamUtil.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamXmlRecordReader.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/HadoopStreaming.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/LoadTypedBytes.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeMapRed$MRErrorThread.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeMapper.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/PipeMapRed$1.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamInputFormat.class
JarBuilder.addNamedStream org/apache/hadoop/streaming/StreamJob.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesInput$1.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesRecordInput$TypedBytesIndex.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesWritableOutput$1.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesWritableOutput.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesWritableInput$1.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesWritableInput$2.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesRecordInput.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesRecordOutput$1.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesOutput$1.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesWritable.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesOutput.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesRecordInput$1.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesRecordOutput.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesWritableInput.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/TypedBytesInput.class
JarBuilder.addNamedStream org/apache/hadoop/typedbytes/Type.class
JarBuilder.addNamedStream META-INF/MANIFEST.MF
STREAM: ==== JobConf properties:
STREAM: dfs.access.time.precision=3600000
STREAM: dfs.balance.bandwidthPerSec=943718400
STREAM: dfs.block.access.key.update.interval=600
STREAM: dfs.block.access.token.enable=false
STREAM: dfs.block.access.token.lifetime=600
STREAM: dfs.block.size=67108864
STREAM: dfs.blockreport.initialDelay=0
STREAM: dfs.blockreport.intervalMsec=3600000
STREAM: dfs.client.block.write.retries=3
STREAM: dfs.data.dir=${hadoop.tmp.dir}/dfs/data
STREAM: dfs.datanode.address=0.0.0.0:50010
STREAM: dfs.datanode.data.dir.perm=700
STREAM: dfs.datanode.directoryscan.threads=1
STREAM: dfs.datanode.dns.interface=default
STREAM: dfs.datanode.dns.nameserver=default
STREAM: dfs.datanode.du.reserved=53687091200
STREAM: dfs.datanode.failed.volumes.tolerated=0
STREAM: dfs.datanode.handler.count=3
STREAM: dfs.datanode.http.address=0.0.0.0:50075
STREAM: dfs.datanode.https.address=0.0.0.0:50475
STREAM: dfs.datanode.ipc.address=0.0.0.0:50020
STREAM: dfs.datanode.max.xcievers=4096
STREAM: dfs.datanode.plugins=org.apache.hadoop.thriftfs.DatanodePlugin
STREAM: dfs.default.chunk.view.size=32768
STREAM: dfs.df.interval=60000
STREAM: dfs.heartbeat.interval=3
STREAM: dfs.hosts=/etc/hadoop/conf/hosts.include
STREAM: dfs.hosts.exclude=/etc/hadoop/conf/hosts.exclude
STREAM: dfs.http.address=0.0.0.0:50070
STREAM: dfs.https.address=0.0.0.0:50470
STREAM: dfs.https.client.keystore.resource=ssl-client.xml
STREAM: dfs.https.enable=false
STREAM: dfs.https.need.client.auth=false
STREAM: dfs.https.server.keystore.resource=ssl-server.xml
STREAM: dfs.max-repl-streams=16
STREAM: dfs.max.objects=0
STREAM: dfs.name.dir=/hdfs/01/name,/hdfs/02/name,/mnt/remote_namenode_failsafe/name
STREAM: dfs.name.edits.dir=${dfs.name.dir}
STREAM: dfs.namenode.decommission.interval=30
STREAM: dfs.namenode.decommission.nodes.per.interval=5
STREAM: dfs.namenode.delegation.key.update-interval=86400000
STREAM: dfs.namenode.delegation.token.max-lifetime=604800000
STREAM: dfs.namenode.delegation.token.renew-interval=86400000
STREAM: dfs.namenode.handler.count=10
STREAM: dfs.namenode.logging.level=info
STREAM: dfs.namenode.plugins=org.apache.hadoop.thriftfs.NamenodePlugin
STREAM: dfs.permissions=true
STREAM: dfs.permissions.supergroup=supergroup
STREAM: dfs.replication=3
STREAM: dfs.replication.considerLoad=true
STREAM: dfs.replication.interval=3
STREAM: dfs.replication.max=512
STREAM: dfs.replication.min=1
STREAM: dfs.safemode.extension=30000
STREAM: dfs.safemode.min.datanodes=0
STREAM: dfs.safemode.threshold.pct=0.999f
STREAM: dfs.secondary.http.address=0.0.0.0:50090
STREAM: dfs.support.append=true
STREAM: dfs.thrift.address=0.0.0.0:10090
STREAM: dfs.web.ugi=webuser,webgroup
STREAM: fs.automatic.close=true
STREAM: fs.checkpoint.dir=/hdfs/01/checkpoint,/hdfs/02/checkpoint
STREAM: fs.checkpoint.edits.dir=${fs.checkpoint.dir}
STREAM: fs.checkpoint.period=3600
STREAM: fs.checkpoint.size=67108864
STREAM: fs.default.name=hdfs://namenode.example.net:9000/
STREAM: fs.file.impl=org.apache.hadoop.fs.LocalFileSystem
STREAM: fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem
STREAM: fs.har.impl=org.apache.hadoop.fs.HarFileSystem
STREAM: fs.har.impl.disable.cache=true
STREAM: fs.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem
STREAM: fs.hftp.impl=org.apache.hadoop.hdfs.HftpFileSystem
STREAM: fs.hsftp.impl=org.apache.hadoop.hdfs.HsftpFileSystem
STREAM: fs.inmemory.size.mb=192
STREAM: fs.kfs.impl=org.apache.hadoop.fs.kfs.KosmosFileSystem
STREAM: fs.ramfs.impl=org.apache.hadoop.fs.InMemoryFileSystem
STREAM: fs.s3.block.size=67108864
STREAM: fs.s3.buffer.dir=${hadoop.tmp.dir}/s3
STREAM: fs.s3.impl=org.apache.hadoop.fs.s3.S3FileSystem
STREAM: fs.s3.maxRetries=4
STREAM: fs.s3.sleepTimeSeconds=10
STREAM: fs.s3n.block.size=67108864
STREAM: fs.s3n.impl=org.apache.hadoop.fs.s3native.NativeS3FileSystem
STREAM: fs.trash.interval=0
STREAM: hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab
STREAM: hadoop.http.authentication.kerberos.principal=HTTP/localhost@LOCALHOST
STREAM: hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret
STREAM: hadoop.http.authentication.simple.anonymous.allowed=true
STREAM: hadoop.http.authentication.token.validity=36000
STREAM: hadoop.http.authentication.type=simple
STREAM: hadoop.kerberos.kinit.command=kinit
STREAM: hadoop.logfile.count=10
STREAM: hadoop.logfile.size=10000000
STREAM: hadoop.native.lib=true
STREAM: hadoop.permitted.revisions=
03b655719d13929bd68bb2c2f9cee615b389cea9,
217a3767c48ad11d4632e19a22897677268c40c4

STREAM: hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory
STREAM: hadoop.security.authentication=simple
STREAM: hadoop.security.authorization=false
STREAM: hadoop.security.group.mapping=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
STREAM: hadoop.security.uid.cache.secs=14400
STREAM: hadoop.tmp.dir=/tmp/hadoop-${user.name}
STREAM: hadoop.util.hash.type=murmur
STREAM: hadoop.workaround.non.threadsafe.getpwuid=false
STREAM: io.bytes.per.checksum=512
STREAM: io.compression.codecs=org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec
STREAM: io.file.buffer.size=131702
STREAM: io.map.index.skip=0
STREAM: io.mapfile.bloom.error.rate=0.005
STREAM: io.mapfile.bloom.size=1048576
STREAM: io.seqfile.compress.blocksize=1000000
STREAM: io.seqfile.lazydecompress=true
STREAM: io.seqfile.sorter.recordlimit=1000000
STREAM: io.serializations=org.apache.hadoop.io.serializer.WritableSerialization
STREAM: io.skip.checksum.errors=false
STREAM: io.sort.factor=10
STREAM: io.sort.mb=100
STREAM: io.sort.record.percent=0.05
STREAM: io.sort.spill.percent=0.80
STREAM: ipc.client.connect.max.retries=10
STREAM: ipc.client.connection.maxidletime=10000
STREAM: ipc.client.idlethreshold=4000
STREAM: ipc.client.kill.max=10
STREAM: ipc.client.tcpnodelay=false
STREAM: ipc.server.listen.queue.size=128
STREAM: ipc.server.tcpnodelay=false
STREAM: job.end.retry.attempts=0
STREAM: job.end.retry.interval=30000
STREAM: jobclient.completion.poll.interval=5000
STREAM: jobclient.output.filter=FAILED
STREAM: jobclient.progress.monitor.poll.interval=1000
STREAM: jobtracker.thrift.address=0.0.0.0:9290
STREAM: keep.failed.task.files=false
STREAM: local.cache.size=10737418240
STREAM: map.sort.class=org.apache.hadoop.util.QuickSort
STREAM: mapred.acls.enabled=false
STREAM: mapred.child.java.opts=-Xmx512m -Xms512m
STREAM: mapred.child.tmp=./tmp
STREAM: mapred.cluster.map.memory.mb=-1
STREAM: mapred.cluster.max.map.memory.mb=-1
STREAM: mapred.cluster.max.reduce.memory.mb=-1
STREAM: mapred.cluster.reduce.memory.mb=-1
STREAM: mapred.compress.map.output=false
STREAM: mapred.create.symlink=yes
STREAM: mapred.disk.healthChecker.interval=60000
STREAM: mapred.fairscheduler.preemption=true
STREAM: mapred.healthChecker.interval=60000
STREAM: mapred.healthChecker.script.timeout=600000
STREAM: mapred.heartbeats.in.second=100
STREAM: mapred.hosts=/etc/hadoop/conf/hosts.include
STREAM: mapred.hosts.exclude=/etc/hadoop/conf/hosts.exclude
STREAM: mapred.inmem.merge.threshold=1000
STREAM: mapred.input.dir=hdfs://namenode.example.net:9000/user/hcoyote/gzipped.txt
STREAM: mapred.input.format.class=org.apache.hadoop.mapred.lib.NLineInputFormat
STREAM: mapred.jar=/tmp/streamjob6232490588444082861.jar
STREAM: mapred.job.map.memory.mb=-1
STREAM: mapred.job.queue.name=default
STREAM: mapred.job.reduce.input.buffer.percent=0.0
STREAM: mapred.job.reduce.memory.mb=-1
STREAM: mapred.job.reuse.jvm.num.tasks=1
STREAM: mapred.job.shuffle.input.buffer.percent=0.70
STREAM: mapred.job.shuffle.merge.percent=0.66
STREAM: mapred.job.tracker=namenode.example.net:54311
STREAM: mapred.job.tracker.handler.count=10
STREAM: mapred.job.tracker.http.address=0.0.0.0:50030
STREAM: mapred.job.tracker.jobhistory.lru.cache.size=5
STREAM: mapred.job.tracker.persist.jobstatus.active=false
STREAM: mapred.job.tracker.persist.jobstatus.dir=/jobtracker/jobsInfo
STREAM: mapred.job.tracker.persist.jobstatus.hours=0
STREAM: mapred.job.tracker.retiredjobs.cache.size=1000
STREAM: mapred.jobtracker.completeuserjobs.maximum=20
STREAM: mapred.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst
STREAM: mapred.jobtracker.job.history.block.size=3145728
STREAM: mapred.jobtracker.maxtasks.per.job=-1
STREAM: mapred.jobtracker.plugins=org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin
STREAM: mapred.jobtracker.restart.recover=false
STREAM: mapred.jobtracker.taskScheduler=org.apache.hadoop.mapred.FairScheduler
STREAM: mapred.line.input.format.linespermap=1
STREAM: mapred.local.dir=${hadoop.tmp.dir}/mapred/local
STREAM: mapred.local.dir.minspacekill=0
STREAM: mapred.local.dir.minspacestart=0
STREAM: mapred.map.max.attempts=4
STREAM: mapred.map.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec
STREAM: mapred.map.runner.class=org.apache.hadoop.streaming.PipeMapRunner
STREAM: mapred.map.tasks=2
STREAM: mapred.map.tasks.speculative.execution=true
STREAM: mapred.mapoutput.key.class=org.apache.hadoop.io.Text
STREAM: mapred.mapoutput.value.class=org.apache.hadoop.io.Text
STREAM: mapred.mapper.class=org.apache.hadoop.streaming.PipeMapper
STREAM: mapred.max.tracker.blacklists=4
STREAM: mapred.max.tracker.failures=4
STREAM: mapred.merge.recordsBeforeProgress=10000
STREAM: mapred.min.split.size=0
STREAM: mapred.output.compress=false
STREAM: mapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec
STREAM: mapred.output.compression.type=RECORD
STREAM: mapred.output.dir=hdfs://namenode.example.net:9000/user/hcoyote/gzipped.log
STREAM: mapred.output.format.class=org.apache.hadoop.mapred.TextOutputFormat
STREAM: mapred.output.key.class=org.apache.hadoop.io.Text
STREAM: mapred.output.value.class=org.apache.hadoop.io.Text
STREAM: mapred.queue.default.state=RUNNING
STREAM: mapred.queue.names=default
STREAM: mapred.reduce.max.attempts=4
STREAM: mapred.reduce.parallel.copies=33
STREAM: mapred.reduce.slowstart.completed.maps=0.75
STREAM: mapred.reduce.tasks=0
STREAM: mapred.reduce.tasks.speculative.execution=false
STREAM: mapred.skip.attempts.to.start.skipping=2
STREAM: mapred.skip.map.auto.incr.proc.count=true
STREAM: mapred.skip.map.max.skip.records=0
STREAM: mapred.skip.reduce.auto.incr.proc.count=true
STREAM: mapred.skip.reduce.max.skip.groups=0
STREAM: mapred.submit.replication=10
STREAM: mapred.system.dir=/mapred/system
STREAM: mapred.task.cache.levels=2
STREAM: mapred.task.profile=false
STREAM: mapred.task.profile.maps=0-2
STREAM: mapred.task.profile.reduces=0-2
STREAM: mapred.task.timeout=600000
STREAM: mapred.task.tracker.http.address=0.0.0.0:50060
STREAM: mapred.task.tracker.report.address=127.0.0.1:0
STREAM: mapred.task.tracker.task-controller=org.apache.hadoop.mapred.DefaultTaskController
STREAM: mapred.tasktracker.dns.interface=default
STREAM: mapred.tasktracker.dns.nameserver=default
STREAM: mapred.tasktracker.expiry.interval=600000
STREAM: mapred.tasktracker.indexcache.mb=10
STREAM: mapred.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst
STREAM: mapred.tasktracker.map.tasks.maximum=8
STREAM: mapred.tasktracker.reduce.tasks.maximum=8
STREAM: mapred.tasktracker.taskmemorymanager.monitoring-interval=5000
STREAM: mapred.tasktracker.tasks.sleeptime-before-sigkill=5000
STREAM: mapred.temp.dir=${hadoop.tmp.dir}/mapred/temp
STREAM: mapred.used.genericoptionsparser=true
STREAM: mapred.user.jobconf.limit=5242880
STREAM: mapred.userlog.limit.kb=128
STREAM: mapred.userlog.retain.hours=24
STREAM: mapred.working.dir=hdfs://namenode.example.net:9000/user/hcoyote
STREAM: mapreduce.job.acl-modify-job=
STREAM: mapreduce.job.acl-view-job=
STREAM: mapreduce.job.complete.cancel.delegation.tokens=true
STREAM: mapreduce.job.counters.limit=120
STREAM: mapreduce.job.jar.unpack.pattern=(?:classes/|lib/).*|(?:\Qgzipit.sh\E)
STREAM: mapreduce.jobtracker.split.metainfo.maxsize=10000000
STREAM: mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging
STREAM: mapreduce.reduce.input.limit=-1
STREAM: mapreduce.reduce.shuffle.connect.timeout=180000
STREAM: mapreduce.reduce.shuffle.maxfetchfailures=10
STREAM: mapreduce.reduce.shuffle.read.timeout=180000
STREAM: mapreduce.tasktracker.cache.local.numberdirectories=10000
STREAM: mapreduce.tasktracker.outofband.heartbeat=false
STREAM: stream.addenvironment=
STREAM: stream.map.input.writer.class=org.apache.hadoop.streaming.io.TextInputWriter
STREAM: stream.map.output.reader.class=org.apache.hadoop.streaming.io.TextOutputReader
STREAM: stream.map.streamprocessor=gzipit.sh
STREAM: stream.numinputspecs=1
STREAM: stream.reduce.input.writer.class=org.apache.hadoop.streaming.io.TextInputWriter
STREAM: stream.reduce.output.reader.class=org.apache.hadoop.streaming.io.TextOutputReader
STREAM: tasktracker.http.threads=40
STREAM: topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping
STREAM: topology.script.file.name=/etc/hadoop/conf/rack-topology.sh
STREAM: topology.script.number.args=100
STREAM: webinterface.private.actions=false
STREAM: ====
STREAM: submitting to jobconf: namenode.example.net:54311
13/09/24 17:10:27 INFO mapred.FileInputFormat: Total input paths to process : 1
13/09/24 17:10:27 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hcoyote/mapred/local]
13/09/24 17:10:27 INFO streaming.StreamJob: Running job: job_201307061907_108574
13/09/24 17:10:27 INFO streaming.StreamJob: To kill this job, run:
13/09/24 17:10:27 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=namenode.example.net:54311 -kill job_201307061907_108574
13/09/24 17:10:27 INFO streaming.StreamJob: Tracking URL: http://namenode.example.net:50030/jobdetails.jsp?jobid=job_201307061907_108574
13/09/24 17:10:28 INFO streaming.StreamJob:  map 0%  reduce 0%
13/09/24 17:10:41 INFO streaming.StreamJob:  map 4%  reduce 0%
13/09/24 17:10:42 INFO streaming.StreamJob:  map 37%  reduce 0%
13/09/24 17:10:43 INFO streaming.StreamJob:  map 55%  reduce 0%
13/09/24 17:10:44 INFO streaming.StreamJob:  map 61%  reduce 0%
13/09/24 17:10:45 INFO streaming.StreamJob:  map 66%  reduce 0%
13/09/24 17:10:46 INFO streaming.StreamJob:  map 70%  reduce 0%
13/09/24 17:10:47 INFO streaming.StreamJob:  map 72%  reduce 0%
13/09/24 17:10:48 INFO streaming.StreamJob:  map 73%  reduce 0%
13/09/24 17:10:49 INFO streaming.StreamJob:  map 74%  reduce 0%
13/09/24 17:11:18 INFO streaming.StreamJob:  map 75%  reduce 0%
13/09/24 17:11:27 INFO streaming.StreamJob:  map 76%  reduce 0%
13/09/24 17:11:29 INFO streaming.StreamJob:  map 77%  reduce 0%
13/09/24 17:11:32 INFO streaming.StreamJob:  map 78%  reduce 0%
13/09/24 17:11:34 INFO streaming.StreamJob:  map 79%  reduce 0%
13/09/24 17:11:37 INFO streaming.StreamJob:  map 80%  reduce 0%
13/09/24 17:11:38 INFO streaming.StreamJob:  map 82%  reduce 0%
13/09/24 17:11:40 INFO streaming.StreamJob:  map 83%  reduce 0%
13/09/24 17:11:41 INFO streaming.StreamJob:  map 84%  reduce 0%
13/09/24 17:11:42 INFO streaming.StreamJob:  map 86%  reduce 0%
13/09/24 17:11:43 INFO streaming.StreamJob:  map 88%  reduce 0%
13/09/24 17:11:44 INFO streaming.StreamJob:  map 89%  reduce 0%
13/09/24 17:11:45 INFO streaming.StreamJob:  map 90%  reduce 0%
13/09/24 17:11:46 INFO streaming.StreamJob:  map 91%  reduce 0%
13/09/24 17:11:47 INFO streaming.StreamJob:  map 92%  reduce 0%
13/09/24 17:11:48 INFO streaming.StreamJob:  map 94%  reduce 0%
13/09/24 17:11:49 INFO streaming.StreamJob:  map 95%  reduce 0%
13/09/24 17:11:50 INFO streaming.StreamJob:  map 96%  reduce 0%
13/09/24 17:11:51 INFO streaming.StreamJob:  map 97%  reduce 0%
13/09/24 17:11:52 INFO streaming.StreamJob:  map 99%  reduce 0%
13/09/24 17:11:53 INFO streaming.StreamJob:  map 100%  reduce 0%
13/09/24 17:14:14 INFO streaming.StreamJob:  map 100%  reduce 100%
13/09/24 17:14:14 INFO streaming.StreamJob: Job complete: job_201307061907_108574
13/09/24 17:14:14 INFO streaming.StreamJob: Output: /user/hcoyote/gzipped.log

Using cobbler with a fast file system creation snippet for Kickstart %post install of Hadoop nodes

I run Hadoop servers with 12 2TB hard drives in them. One of the bottlenecks with this occurs during kickstart when we’re using anaconda to create the filesystems. Previously, I just had a specific partition configuration that was brought in during %pre, but this caused the filesystem formatting section of kickstart to take several hours to complete. With some additional changes that required us to begin hand-signing puppet certificates that were created during %post, this entire process because too unwieldy. I got tired of having to wait hours for systems to get to the %post install, just so I could turn around and sign things.

Instead, what I did was to move the partitioning of the filesystems into a %post snippet for cobbler and tweak a few filesystem creation options (which you can’t do with partition in the ks.cfg). This brought creation time down to about 10 minutes from several hours.

The snippet is below.

I’d like to point a few things out with our configuration.

  • We have 12 2TB drives, as I stated above. Each drive is dedicated to HDFS. Each drive is mounted to /hdfs/XX where XX is 01 through 12.
  • All HDFS drives are given an e2label of the form /hdfs/XX. We use this as a signal within a puppet fact to do configuration of the data node. If we lose a drive, we can just re-run puppet and have the configuration fixed temporarily while we wait for a replacement.
  • We’re going to use ext4 with these systems since we’re using a sufficiently new enough kernel in the 2.6 line. We chose this based on various recommendations now coming out of places like Cloudera, AMD, and Intel.
  • We disable atime and diratime on all HDFS drives to prevent useless writes that occur with atime updates.
  • We use an FS profile called “largefile”, during mkfs.ext4 creation that reduces the number of inodes that gets created on the filesystem. Since we’re generally dealing with large files, this is acceptable to us.
  • We use several ext4 features:
    • dir_index – speeds up directory lookups in large directories
    • extent – use extent-based block allocation which has a benefit for large files, allowing them to be laid out more contiguously on disk
    • sparse_super – create fewer superblock backups which aren’t needed on large filesystems.
  • We have a RAID1 OS drive on /dev/sda. This was done because we wanted to dedicate as much space to HDFS and prevent the first drive from taking an I/O hit due to logging or other non-HDFS activities. This is presented to the OS with a model of “Virtual disk”, which allows us to detect and skip operating on it.

Finally, because Cobbler uses Cheetah as it’s backend templating system, I want to point out that there are some additional escaped dollar signs ($) in the snippet to prevent cobbler from choking on them. Otherwise, you could use this straight in a shell script.

DIR="/sys/block"

MINSIZE=1000


# list-harddrives doesn't exist in the chroot post install environment. bummer.
for DEV in `cd $DIR ; ls -d sd*`; do
    if [ -d $DIR/$DEV ] ; then
        REMOVABLE=`cat $DIR/$DEV/removable`
        if (( $REMOVABLE == 0 )) ; then
            MODEL=`cat $DIR/$DEV/device/model`
            if [[ "$MODEL" =~ ^Virtual.* ]] ; then
                echo "Found a virtual disk on /dev/$DEV, skipping"
            else
                echo "Found $DEV"
                SIZE=`cat $DIR/$DEV/size`
                GB=$(($SIZE/2**21))
                if [ $GB -gt $MINSIZE ] ; then
                    # we are a non-root drive
                    echo "Found a rightsize drive on $DEV for hadoop"

                    for partition in `parted -s /dev/$DEV print | awk '/^ / {print $1}'`; do
                        parted -s /dev/$DEV rm /dev/${partition}
                    done

                    parted -s /dev/$DEV mklabel gpt
                    parted -s /dev/$DEV mkpart -- primary ext4 1 -1
                    partprobe

                    

                    # we are going to map /dev/sdX to /hdfs/YY with this
                    HDFS_PART_ASCII=`echo $DEV | sed -e 's/sd//' | od -N 1 -i | head -1 | tr -s " " | cut -d" " -f 2`
                    HDFS_PART_NUMBER=\$(($HDFS_PART_ASCII - 97))
                    HDFS_LABEL=\$(printf "/hdfs/%02g" $HDFS_PART_NUMBER)


                    mkfs.ext4 -T largefile -m 1 -O dir_index,extent,sparse_super -L $HDFS_LABEL /dev/${DEV}1 

                    eval `blkid -o export /dev/${DEV}1`

                    if [ -n "${LABEL}" ] ; then
                        echo "Creating $LABEL mountpoint"
                        mkdir -p "${LABEL}"
                        echo "Adding $LABEL to /etc/fstab"
                        echo "LABEL=$LABEL $LABEL ext4 defaults,noatime,nodiratime 1 2" >> /etc/fstab
                        tail -1 /etc/fstab
                    fi 
                fi
            fi
        fi
    fi
done

Aw. It’s a wedding.

Aw.  It's a wedding.

(more…)

Seeing RabbitMQ memory usage

While working in our RabbitMQ environment this week, we noticed that there was a large, unexplained amount of memory in use by RabbitMQ that we couldn’t account for by normal queue and message use. One of the first tools we use when poking around Erlang and RabbitMQ is to do a memory dump.


$ sudo  rabbitmqctl eval 
  'lists:sublist(lists:reverse( 
     lists:sort([{process_info(Pid, memory), Pid, 
     process_info(Pid)} || Pid <- processes()])), 1).'

This command is evaluated directly into the running erlang instance within our RabbitMQ and dumps the top memory users from largest to smallest. In the above case, we’re telling erlang to show us the single top user of memory. You can change the 1 to any positive number you like to get more and more erlang processes returned to evaluate the memory usage of.

This produces something like the following:


[{{memory,12344224},
  <4865.28344.41>,
  [{current_function,{gen_server2,process_next_msg,1}},
   {initial_call,{proc_lib,init_p,5}},
   {status,waiting},
   {message_queue_len,0},
   {messages,[]},
   {links,[<4865.28338.41>,<4865.28353.41>,#Port<4865.433154>]},
   {dictionary,[{{credit_to,<4865.5033.42>},28},
                {{#Ref<4865.0.1377.83012>,fhc_handle},
                 {handle,{file_descriptor,prim_file,{#Port<4865.433154>,20}},
                         335872,false,0,infinity,[],true,
                         "/var/lib/rabbitmq/mnesia/rabbit@rabbitmqhost/queues/14AI6IEJDH4TZNL8R69HT5Y6K/journal.jif",
                         [write,binary,raw,read],
                         [{write_buffer,infinity}],
                         true,true,
                         {1351,12777,400557}}},
                {{ch,<4865.28440.41>},
                 {cr,<4865.28440.41>,#Ref<4865.0.1377.84010>,
                     {set,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                     1,
                     {[],[]},
                     {token,<4865.28439.41>,true},
                     false,19}},
                {{credit_to,<4865.4869.42>},15},
                {'$ancestors',[rabbit_amqqueue_sup,rabbit_sup,
                               <4865.28225.41>]},
                {{credit_to,<4865.4228.42>},49},
                {fhc_age_tree,{1,
                               {{1351,12777,400557},
                                #Ref<4865.0.1377.83012>,nil,nil}}},
                {{credit_to,<4865.28474.41>},35},
                {{credit_to,<4865.5066.42>},11},
                {credit_blocked,[]},
                {{credit_to,<4865.28507.41>},3},
                {{credit_to,<4865.28490.41>},33},
                {{"/var/lib/rabbitmq/mnesia/rabbit@rabbitmqhost/queues/14AI6IEJDH4TZNL8R69HT5Y6K/journal.jif",
                  fhc_file},
                 {file,1,true}},
                {{credit_to,<4865.28498.41>},14},
                {{credit_to,<4865.28482.41>},31},
                {guid,{{4270534438,2738637279,2258006136,1881461985},0}},
                {'$initial_call',{gen,init_it,6}}]},
   {trap_exit,true},
   {error_handler,error_handler},
   {priority,normal},
   {group_leader,<4865.28224.41>},
   {total_heap_size,1542687},
   {heap_size,196418},
   {stack_size,7},
   {reductions,332233374},
   {garbage_collection,[{min_bin_vheap_size,46368},
                        {min_heap_size,233},
                        {fullsweep_after,65535},
                        {minor_gcs,20706}]},
   {suspending,[]}]}]
...done.

I don’t know what all of the bits mean yet, but we can point out a few useful things to look at. A good reference on what is being returned can be found in the Erlang documentation for process_info/2

  • memory – the current amount of memory in use by this erlang process. This directly effects the size of the RabbitMQ process size in the OS.
  • <4865.28344.41> – the erlang pid inside the erlang kernel
  • current_function – the function call currently running in the process
  • registered_name – (Note: note seen above). this is the name of the process associated with this memory. If the process isn’t named, this won’t show up in the output
  • status – What the process is currently doing. It can be something like exiting, garbage_collecting, waiting, running, runnable, or suspended.
  • messages – messages associated with this process.
  • dictionary

There’s more than this, so you should definitely look at the erlang docs to figure out what’s going on here, but this should get you started in understanding what memory usage is for your RabbitMQ environment.

Hadoop DataNode logs filling with clienttrace messages

So, you’re probably like me.

You have a shiny, new Cloudera Hadoop cluster. Everything is zooming along smoothly. Until you find that your /var/log/hadoop datanode logs are growing at a rate of a bazillion gigabytes per day. What do you do, hot shot? WHAT DO YOU DO?

Actually, it’s pretty simple.

We were getting alerts on our cluster that /var was filling across various datanodes (where we also happen to have HBase running). We were doing about four gigabytes a day just in the datanode logs alone. That seemed excessive. Peering through the logs, we found that a large percentage of entries looked something like this:

2012-03-26 23:59:57,411 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.1.1.134:50010, dest: /10.1.1.114:44972, bytes: 20548, op: HDFS_READ, cliID: DFSClient_1020093912, offset: 0, srvID: DS-1061365566-10.1.1.134-50010-1331849447734, blockid: blk_1530028621327446680_9137244, duration: 309000
2012-03-26 23:59:57,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.1.1.134:50010, dest: /10.1.1.108:40058, bytes: 27507, op: HDFS_READ, cliID: DFSClient_345923441, offset: 0, srvID: DS-1061365566-10.1.1.134-50010-1331849447734, blockid: blk_4340309148758316988_10579186, duration: 909000
2012-03-26 23:59:57,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.1.1.134:50010, dest: /10.1.1.114:44973, bytes: 23734, op: HDFS_READ, cliID: DFSClient_1020093912, offset: 0, srvID: DS-1061365566-10.1.1.134-50010-1331849447734, blockid: blk_4111106876676617467_8649685, duration: 568000
2012-03-26 23:59:57,423 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.1.1.134:50010, dest: /10.1.1.119:45662, bytes: 20840, op: HDFS_READ, cliID: DFSClient_-635119343, offset: 0, srvID: DS-1061365566-10.1.1.134-50010-1331849447734, blockid: blk_4655242753989962972_6418861, duration: 310000
2012-03-26 23:59:57,434 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.1.1.134:50010, dest: /10.1.1.108:40061, bytes: 17728, op: HDFS_READ, cliID: DFSClient_345923441, offset: 0, srvID: DS-1061365566-10.1.1.134-50010-1331849447734, blockid: blk_6737196232027719961_7900492, duration: 1071000
2012-03-26 23:59:57,470 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.1.1.134:50010, dest: /10.1.1.105:53569, bytes: 19998, op: HDFS_READ, cliID: DFSClient_-1106211444, offset: 0, srvID: DS-1061365566-10.1.1.134-50010-1331849447734, blockid: blk_801375605494816473_8641680, duration: 856000

Doing a handy-dandy google search, we found this thread discussing the very problem we were seeing.  Looks like this is some sort of performance data emitted by the datanode for  blocks associated due to how the HBase META region interacts with it.

No big deal.

The fix?  We’ll just turn down logging on this specific java class.  We achieve this by doing the following.  This fixes the runtime logging that the datanodes are doing so we don’t have to restart the datanodes right now.

$ sudo -u hdfs hadoop daemonlog -setlevel localhost:50075 org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace WARN

Later, we’ll add the following to /etc/hadoop/conf/log4j.properties on all datanodes and restart them.

log4j.logger.org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace=WARN

Hadoop, facter, and the puppet marionette

I’ve been working with puppet a lot lately.  A lot.  It’s part of my job.  We’ve been setting up a new hadoop cluster in our Xen environment.  Nothing big.  It started out with 4 nodes, all configured the same way (3 drives each).  We added an additional 2 nodes with 2 drives each.  This, of course, broke how we were having to manage the hadoop configuration files.  I spent the last few days figuring out how to create a custom Puppet fact (for our Puppet 2.6.4 installation) that would search for the HDFS drives on our nodes and configure the hdfs-site.xml and mapred-site.xml appropriately.

First, in our hadoop module, I created the modules/hadoop/lib/facter path and added the following facter code to hdfs_path.rb in there.

require 'facter'

paths = []

if FileTest.exists?("/sbin/e2label")


    drives = Dir.glob("/dev/sd?1")
    drives += Dir.glob("/dev/*vg/hdfs*_lv")

    drives.each do |drive|
        #if FileTest.exists?("/dev/#{drive}1")
        if FileTest.exists?(drive)
            output = %x{/sbin/e2label #{drive} 2>/dev/null}.chomp

            # only run this fact if we find hdfs in the label name.
            if output =~ /hdfs/ and output =~ /_/
                device = output.split("_")[1]
                devicenumber = device.split("hdfs")[1]
                path = "/hdfs/" + devicenumber
                paths.push(path)
                Facter.add("hdfs_#{device}") do
                    setcode do
                        path
                    end
                end
            end
        end
    end

	allpaths = paths.join(",")
    Facter.add("hdfs_all_paths") do
        setcode do
            allpaths
        end
    end
end

This would look for drives on the hadoop node that had an ext3 label of hostname_hdfs#. For example, one of these would be vhdn01_hdfs1. The fact would be populated with the value /hdfs/1. The facter module would create one entry of these per device found. As well, it would create a fact called hdfs_all_paths that contained a comma separated list of every path we found. These would be used in the templates for mapred-site.xml and hdfs-site.xml to setup the list of paths we’re using.

In our template, we created

<property>
 <name>dfs.data.dir</name>
 <value><%= hdfs_all_paths.split(",").map { |n| n + "/hdfs"}.join(",") -%></value>
 <final>true</final>
</property>

which creates a comma separated list of paths and tacks on /hdfs to each item. For example, this takes /hdfs/1,/hdfs/2 and turns it into /hdfs/1/hdfs,/hdfs/2/hdfs.

Some things to note:  I run in an agent/master setup.  In order for the fact to be sent down to the agent, we need to enable pluginsync on both ends.  You can do this by adding the option to your puppet.conf.  It needs to go in the [main] section.  For example.

[main]
pluginsync = true

The other caveat I ran into is that in order for this to work, you need to make sure you have the fact on the client system before you use it in your catalog (such as in your templates), otherwise your catalog might not compile when you go to run your agent.

Once I got all these things figure out, I got a much more easily configured hadoop datanode setup.

Playing with the perl RTM client.

I began playing with a perl-based RememberTheMilk command line tool today from http://www.rutschle.net/rtm/.  There aren’t any RPMs of it that I found, so I ended up building some.  This is what I had to do to get it to the stage of at least working.

:;  sudo cpan2rpm WebService::RTMAgent

-- cpan2rpm - Ver: 2.028 --
Upgrade check
Fetch: HTTP

-- module: WebService::RTMAgent --
Using cached URL: http://search.cpan.org//CPAN/authors/id/R/RU/RUTSCHLE/WebService-RTMAgent-0.5_1.tar.gz
Tarball found - not fetching
Metadata retrieval
Tarball extraction: [/home/tcampbell/src/rpm/SOURCES/WebService-RTMAgent-0.5_1.tar.gz]

Can't locate object method "interpolate" via package "Pod::Text" at /usr/bin/cpan2rpm line 525.
cannot remove path when cwd is /tmp/yFnnvY7DA3/WebService-RTMAgent-0.5_1 for /tmp/yFnnvY7DA3:  at /usr/share/perl5/File/Temp.pm line 902
-- Done --

But, I ran into this dumb perl interpolate bug. Turns out that cpan2rpm has an issue with newer versions of perl where Pod::Text no longer has this method. Changing Pod::Text to Pod::Parser in cpan2rpm allowed this to succeed.

sudo cpan2rpm WebService::RTMAgent  --no-sign

-- cpan2rpm - Ver: 2.028 --
Upgrade check
Fetch: HTTP

-- module: WebService::RTMAgent --
Using cached URL: http://search.cpan.org//CPAN/authors/id/R/RU/RUTSCHLE/WebService-RTMAgent-0.5_1.tar.gz
Tarball found - not fetching
Metadata retrieval
Tarball extraction: [/home/tcampbell/src/rpm/SOURCES/WebService-RTMAgent-0.5_1.tar.gz]
Generating spec file
SPEC: /home/tcampbell/src/rpm/SPECS/WebService-RTMAgent.spec
Generating package
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.xs7L3d
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd /home/tcampbell/src/rpm/BUILD
+ rm -rf WebService-RTMAgent-0.5_1
+ /usr/bin/gzip -dc /home/tcampbell/src/rpm/SOURCES/WebService-RTMAgent-0.5_1.tar.gz
+ /bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd WebService-RTMAgent-0.5_1
+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ chmod -R u+w /home/tcampbell/src/rpm/BUILD/WebService-RTMAgent-0.5_1
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.sKaxgX
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ grep -rsl '^#!.*perl' .
+ grep -v '.bak$'
+ xargs --no-run-if-empty /usr/bin/perl -MExtUtils::MakeMaker -e 'MY->fixin(@ARGV)'
+ CFLAGS='-O2 -g -march=i386 -mtune=i686'
++ /usr/bin/perl -MExtUtils::MakeMaker -e ' print qq|PREFIX=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr| if $ExtUtils::MakeMaker::VERSION =~ /5\.9[1-6]|6\.0[0-5]/ '
+ /usr/bin/perl Makefile.PL
Checking if your kit is complete...
Looks good
WARNING: Setting ABSTRACT via file 'lib/WebService/RTMAgent.pm' failed
 at /usr/share/perl5/ExtUtils/MakeMaker.pm line 603
Writing Makefile for WebService::RTMAgent
+ /usr/bin/make
cp lib/WebService/RTMAgent.pm blib/lib/WebService/RTMAgent.pm
Manifying blib/man3/WebService::RTMAgent.3pm
+ /usr/bin/make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ....... 1/1 # Testing WebService::RTMAgent 0.5_1, Perl 5.010001, /usr/bin/perl
t/00-load.t ....... ok   
t/auth.t .......... frobbed -- getting token
t/auth.t .......... 1/7 frobbed -- getting token
token token
t/auth.t .......... ok   
t/boilerplate.t ... ok   
t/init.t .......... ok   
t/pod-coverage.t .. skipped: Test::Pod::Coverage 1.08 required for testing POD coverage
t/pod.t ........... ok   
t/requests.t ...... 1/12 request:
POST http://www.rememberthemilk.com/services/rest/
Content-Type: application/x-www-form-urlencoded

method=rtm.tasks.add&nam=adding&api_key=key&auth_token=10438&timeline=114114&api_sig=3340edd30a22e9b2c67ff206283d0b67


response:
HTTP/1.1 200 OK
Connection: keep-alive
Date: Mon, 24 Dec 2007 11:49:10 GMT
Server: nginx/RTM
Vary: Accept-Encoding
Content-Type: text/xml; charset="utf-8"
Client-Date: Mon, 24 Dec 2007 11:50:39 GMT
Client-Peer: 75.126.232.204:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
Keep-Alive: timeout=300







t/requests.t ...... ok     
t/undo.t .......... ok   
All tests successful.

Test Summary Report
-------------------
t/boilerplate.t (Wstat: 0 Tests: 3 Failed: 0)
  TODO passed:   1-3
Files=8, Tests=35,  0 wallclock secs ( 0.05 usr  0.01 sys +  0.59 cusr  0.06 csys =  0.71 CPU)
Result: PASS
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.7ZK9SK
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ '[' /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386 '!=' / ']'
+ rm -rf /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
++ /usr/bin/perl -MExtUtils::MakeMaker -e ' print $ExtUtils::MakeMaker::VERSION <= 6.05 ? qq|PREFIX=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr| : qq|DESTDIR=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386| '
+ make prefix=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr exec_prefix=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr bindir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/bin sbindir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/sbin sysconfdir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/etc datadir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share includedir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/include libdir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/lib libexecdir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/libexec localstatedir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/var sharedstatedir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/var/lib mandir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/man infodir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/info install DESTDIR=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
Manifying blib/man3/WebService::RTMAgent.3pm
Installing /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/local/share/perl5/WebService/RTMAgent.pm
Installing /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/local/share/man/man3/WebService::RTMAgent.3pm
Appending installation info to /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/lib/perl5/perllocal.pod
+ cmd=/usr/share/spec-helper/compress_files
+ '[' -x /usr/share/spec-helper/compress_files ']'
+ cmd=/usr/lib/rpm/brp-compress
+ '[' -x /usr/lib/rpm/brp-compress ']'
+ /usr/lib/rpm/brp-compress
+ '[' -e /etc/SuSE-release -o -e /etc/UnitedLinux-release ']'
+ find /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386 -name perllocal.pod -o -name .packlist -o -name '*.bs'
+ xargs -i rm -f '{}'
+ find /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr -type d -depth -exec rmdir '{}' ';'
+ /usr/bin/perl -MFile::Find -le '
    find({ wanted => \&wanted, no_chdir => 1}, "/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386");
    print "%doc  Changes README";
    for my $x (sort @dirs, @files) {
        push @ret, $x unless indirs($x);
        }
    print join "\n", sort @ret;

    sub wanted {
        return if /auto$/;

        local $_ = $File::Find::name;
        my $f = $_; s|^\Q/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386\E||;
        return unless length;
        return $files[@files] = $_ if -f $f;

        $d = $_;
        /\Q$d\E/ && return for reverse sort @INC;
        $d =~ /\Q$_\E/ && return
            for qw|/etc /usr/man /usr/bin /usr/share|;

        $dirs[@dirs] = $_;
        }

    sub indirs {
        my $x = shift;
        $x =~ /^\Q$_\E\// && $x ne $_ && return 1 for @dirs;
        }
    '
+ '[' -z WebService-RTMAgent-0.5_1-filelist ']'
+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/brp-strip
+ /usr/lib/rpm/brp-strip-static-archive
+ /usr/lib/rpm/brp-strip-comment-note
Processing files: perl-WebService-RTMAgent-0.5_1-1.noarch
Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.25mTCz
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ DOCDIR=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ export DOCDIR
+ rm -rf /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ /bin/mkdir -p /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ cp -pr Changes README /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ exit 0
Provides: perl(WebService::RTMAgent) = 0.5
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(VersionedDependencies) <= 3.0.3-1
Requires: perl(Carp) perl(Digest::MD5) perl(LWP::UserAgent) perl(XML::Simple) perl(strict) perl(vars)
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
Wrote: /home/tcampbell/src/rpm/SRPMS/perl-WebService-RTMAgent-0.5_1-1.src.rpm
Wrote: /home/tcampbell/src/rpm/RPMS/noarch/perl-WebService-RTMAgent-0.5_1-1.noarch.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.xrdBo2
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ '[' /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386 '!=' / ']'
+ rm -rf /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
+ exit 0
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.eKyVlR
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ rm -rf WebService-RTMAgent-0.5_1
+ exit 0
RPM: /home/tcampbell/src/rpm/RPMS/noarch/perl-WebService-RTMAgent-0.5_1-1.noarch.rpm
SRPM: /home/tcampbell/src/rpm/SRPMS/perl-WebService-RTMAgent-0.5_1-1.src.rpm
-- Done --

Sweet! now we can install it.

:;  sudo rpm -ivh /home/tcampbell/src/rpm/RPMS/noarch/perl-WebService-RTMAgent-0.5_1-1.noarch.rpm
Preparing...                ########################################### [100%]
   1:perl-WebService-RTMAgen########################################### [100%]

Great. Now we grab the rtm command from the website.

:;  wget http://www.rutschle.net/rtm/rtm-0.5.gz
--2011-01-04 13:47:20--  http://www.rutschle.net/rtm/rtm-0.5.gz
Resolving www.rutschle.net... 82.235.147.6
Connecting to www.rutschle.net|82.235.147.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4212 (4.1K) [application/x-gzip]
Saving to: `rtm-0.5.gz'

100%[===========================================================================>] 4,212       24.3K/s   in 0.2s    

2011-01-04 13:47:21 (24.3 KB/s) - `rtm-0.5.gz' saved [4212/4212]

:; gunzip rtm-0.5.gz

Now, we run it and ... boom.

:;  ./rtm-0.5 
Use of uninitialized value in concatenation (.) or string at /usr/local/share/perl5/WebService/RTMAgent.pm line 312.
98: Login failed / Invalid auth token
Use of uninitialized value at /usr/local/share/perl5/WebService/RTMAgent.pm line 368

Looks like we need to get an auth token. Looking inside the script, it appears you have to run rtm with the --authorise option. This creates a URL that you put in the browser to authorize the client with RememberTheMilk. After that was done, I can now run rtm from the command line.

:;  ./rtm-0.5 
frobbed -- getting token
token [my token]
0: Turabelle photos
1: do something incredible
2: Ticket to manage notification of changes to bobqueue files.
3: Create ticket to Place bob configs under puppet/avn
4: Rebound payment

Looks like that frobbed/token header go away after the first run.

Now I'm good to go with hitting RememberTheMilk from the command line. Sweet!

Finally updated this.

Final testing with a post from the iPhone. Yay!

The “Enterprise” …

From a discussion with a few peers in the industry.  I was entertained.

peer> Now when I hear someone use the word “enterprise”
   as an adjective, I have to ask them which of the four meanings
   they intend:
peer> 1.  defunct and destroyed (the Enterprise aircraft
   carrier from WW2)
peer> 2.  ancient and nearly dead (the Enterprise nuclear
    aircraft carrier)
peer> 3.  a nonfunctional mockup (the Enterprise space shuttle)
peer> 4.  imaginary (the starship Enterprise)
peer> Typically “enterprise software” fits perfectly in one of
   those four categories.

Name withheld to, of course, protect the guilty.