Using cobbler with a fast file system creation snippet for Kickstart %post install of Hadoop nodes

I run Hadoop servers with 12 2TB hard drives in them. One of the bottlenecks with this occurs during kickstart when we’re using anaconda to create the filesystems. Previously, I just had a specific partition configuration that was brought in during %pre, but this caused the filesystem formatting section of kickstart to take several hours […]

Seeing RabbitMQ memory usage

While working in our RabbitMQ environment this week, we noticed that there was a large, unexplained amount of memory in use by RabbitMQ that we couldn’t account for by normal queue and message use. One of the first tools we use when poking around Erlang and RabbitMQ is to do a memory dump. $ sudo […]

Hadoop DataNode logs filling with clienttrace messages

So, you’re probably like me. You have a shiny, new Cloudera Hadoop cluster. Everything is zooming along smoothly. Until you find that your /var/log/hadoop datanode logs are growing at a rate of a bazillion gigabytes per day. What do you do, hot shot? WHAT DO YOU DO? Actually, it’s pretty simple. We were getting alerts […]

Hadoop, facter, and the puppet marionette

I’ve been working with puppet a lot lately.  A lot.  It’s part of my job.  We’ve been setting up a new hadoop cluster in our Xen environment.  Nothing big.  It started out with 4 nodes, all configured the same way (3 drives each).  We added an additional 2 nodes with 2 drives each.  This, of […]

Playing with the perl RTM client.

I began playing with a perl-based RememberTheMilk command line tool today from http://www.rutschle.net/rtm/.  There aren’t any RPMs of it that I found, so I ended up building some.  This is what I had to do to get it to the stage of at least working. :;  sudo cpan2rpm WebService::RTMAgent — cpan2rpm – Ver: 2.028 — […]

The “Enterprise” …

From a discussion with a few peers in the industry.  I was entertained. peer> Now when I hear someone use the word “enterprise”    as an adjective, I have to ask them which of the four meanings    they intend: peer> 1.  defunct and destroyed (the Enterprise aircraft    carrier from WW2) peer> 2.  ancient and nearly dead […]

Lack of backup foils Va.’s new IT system | Richmond Times-Dispatch

“Every time we’re down for an hour, that’s about 2,500 people inconvenienced,” Smit said. “They’re blaming my people for it and [state IT officials] have an obligation to fix it.” Lack of backup foils Va.’s new IT system | Richmond Times-Dispatch. One of the things we’ve been grappling with lately is some unfortunate unplanned outages […]

Never change anything on Friday at 5pm.

SEVERE: Socket accept failed java.net.SocketException: SSL handshake error javax.net.ssl.SSLException: No available certificate or key corresponds to the SSL cipher suites which are enabled. at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:150) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310) at java.lang.Thread.run(Unknown Source) This is the reason you never change anything on Friday at 5pm. While attempting to update the SSL certificate for the MySQL Enterprise Monitor (for […]

Resizing a multipathed SAN LUN under RHEL5

So today I found myself staring at the Red Hat LVM documentation wherein I began to silently cuss.  You see, I needed to double the LUN size for MySQL’s usage on one of my servers.  These servers are our first that are directly attached into the SAN and running Linux, so we’re working a bit […]