Restarting HBase Regionservers using JSON and jq We run HBase as part of our Hadoop cluster. HBase sits on top of HDFS and is split into two parts: the HBase Master and the HBase Regionservers. The master coordinates which regionservers are in control of each specific region. Automating Recovery Responses We periodically have to do some minor maintenance and upkeep, including restarting […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
My Hadoop cluster data needs no RAID! One of the operational challenges in introducing Hadoop to traditional IT and Enterprise operations is understanding when to break one of our sacred IT mantras: Thou shalt always RAID your data. Never shalt thou install a system without RAID. One shall be your RAID if thou seekest performance and redundancy sparing no expense. Five shall […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Improving Hadoop datanode disk fault tolerance By design, Hadoop is meant to tolerate failures in a responsible manner. One of those failure modes is for an HDFS datanode to go off line because it lost a data disk. By default, the datanode process will not tolerate any disk failures before shutting itself off. When this happens, the HDFS namenode discovers that […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Running Hadoop data nodes from USB thumb drives? I received an interesting question today from a reader regarding the use of USB thumb drives for the OS drives in Hadoop datanodes. Have you ever put the OS for a Hadoop node on a USB thumb drive? (or considered it) I have a smaller 8 node cluster and that would free up one of […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Pig ‘local’ mode fails when Kerberos auth is enabled. I ran across this interesting Kerberos authentication bug today on Cloudera’s CDH4. It appears to affect all versions of pig, but only when running in local mode. I want to run pig in local mode. This implies that pig fires up everything it needs to run the MapReduce job on your local machine without having […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Followup on Cloudera HUE’s Kerberos kt_renewer Just a short followup about the HUE kt_renewer issue I discovered. It turns out that the issue was me and not HUE. The fix turned out to be pretty simple once I saw the clue in a related issue. It seems like Cloudera Manager had the same issue. The problem ended up being a missing […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Kerberos kt_renewer failures with HUE on CDH4 First off, I’m not exactly sure if this is a Hadoop User Environment (HUE) issue or if this is a broken setup on my Kerberos environment. I have a thread open on the HUE users list, but haven’t had any followup. I’ve just fired up HUE for the first time to talk with a kerberos-enabled […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Mass-gzip files inside HDFS using the power of Hadoop I have a bunch of text files sitting in HDFS that I need to compress. It’s on the order of several hundred files comprising several hundred gigabytes of data. There are several ways to do this. I could individually copy down each file, compress it, and re-upload it to HDFS. This takes an excessively long […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Using cobbler with a fast file system creation snippet for Kickstart %post install of Hadoop nodes I run Hadoop servers with 12 2TB hard drives in them. One of the bottlenecks with this occurs during kickstart when we’re using anaconda to create the filesystems. Previously, I just had a specific partition configuration that was brought in during %pre, but this caused the filesystem formatting section of kickstart to take several hours […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)
Hadoop DataNode logs filling with clienttrace messages So, you’re probably like me. You have a shiny, new Cloudera Hadoop cluster. Everything is zooming along smoothly. Until you find that your /var/log/hadoop datanode logs are growing at a rate of a bazillion gigabytes per day. What do you do, hot shot? WHAT DO YOU DO? Actually, it’s pretty simple. We were getting alerts […] Share this:Click to share on Twitter (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Facebook (Opens in new window)Click to email this to a friend (Opens in new window)