Hadoop, facter, and the puppet marionette

Posted on 2011/03/18 by hcoyote

I’ve been working with puppet a lot lately. A lot. It’s part of my job. We’ve been setting up a new hadoop cluster in our Xen environment. Nothing big. It started out with 4 nodes, all configured the same way (3 drives each). We added an additional 2 nodes with 2 drives each. This, of course, broke how we were having to manage the hadoop configuration files. I spent the last few days figuring out how to create a custom Puppet fact (for our Puppet 2.6.4 installation) that would search for the HDFS drives on our nodes and configure the hdfs-site.xml and mapred-site.xml appropriately.

First, in our hadoop module, I created the modules/hadoop/lib/facter path and added the following facter code to hdfs_path.rb in there.

require 'facter'

paths = []

if FileTest.exists?("/sbin/e2label")

    drives = Dir.glob("/dev/sd?1")
    drives += Dir.glob("/dev/*vg/hdfs*_lv")

    drives.each do |drive|
        #if FileTest.exists?("/dev/#{drive}1")
        if FileTest.exists?(drive)
            output = %x{/sbin/e2label #{drive} 2>/dev/null}.chomp

            # only run this fact if we find hdfs in the label name.
            if output =~ /hdfs/ and output =~ /_/
                device = output.split("_")[1]
                devicenumber = device.split("hdfs")[1]
                path = "/hdfs/" + devicenumber
                paths.push(path)
                Facter.add("hdfs_#{device}") do
                    setcode do
                        path
                    end
                end
            end
        end
    end

	allpaths = paths.join(",")
    Facter.add("hdfs_all_paths") do
        setcode do
            allpaths
        end
    end
end

This would look for drives on the hadoop node that had an ext3 label of hostname_hdfs#. For example, one of these would be vhdn01_hdfs1. The fact would be populated with the value /hdfs/1. The facter module would create one entry of these per device found. As well, it would create a fact called hdfs_all_paths that contained a comma separated list of every path we found. These would be used in the templates for mapred-site.xml and hdfs-site.xml to setup the list of paths we’re using.

In our template, we created

<property>
 <name>dfs.data.dir</name>
 <value><%= hdfs_all_paths.split(",").map { |n| n + "/hdfs"}.join(",") -%></value>
 <final>true</final>
</property>

which creates a comma separated list of paths and tacks on /hdfs to each item. For example, this takes /hdfs/1,/hdfs/2 and turns it into /hdfs/1/hdfs,/hdfs/2/hdfs.

Some things to note: I run in an agent/master setup. In order for the fact to be sent down to the agent, we need to enable pluginsync on both ends. You can do this by adding the option to your puppet.conf. It needs to go in the [main] section. For example.

[main]
pluginsync = true

The other caveat I ran into is that in order for this to work, you need to make sure you have the fact on the client system before you use it in your catalog (such as in your templates), otherwise your catalog might not compile when you go to run your agent.

Once I got all these things figure out, I got a much more easily configured hadoop datanode setup.

Playing with the perl RTM client.

Posted on 2011/01/04 by hcoyote

I began playing with a perl-based RememberTheMilk command line tool today from http://www.rutschle.net/rtm/. There aren’t any RPMs of it that I found, so I ended up building some. This is what I had to do to get it to the stage of at least working.

:;  sudo cpan2rpm WebService::RTMAgent

-- cpan2rpm - Ver: 2.028 --
Upgrade check
Fetch: HTTP

-- module: WebService::RTMAgent --
Using cached URL: http://search.cpan.org//CPAN/authors/id/R/RU/RUTSCHLE/WebService-RTMAgent-0.5_1.tar.gz
Tarball found - not fetching
Metadata retrieval
Tarball extraction: [/home/tcampbell/src/rpm/SOURCES/WebService-RTMAgent-0.5_1.tar.gz]

Can't locate object method "interpolate" via package "Pod::Text" at /usr/bin/cpan2rpm line 525.
cannot remove path when cwd is /tmp/yFnnvY7DA3/WebService-RTMAgent-0.5_1 for /tmp/yFnnvY7DA3:  at /usr/share/perl5/File/Temp.pm line 902
-- Done --

But, I ran into this dumb perl interpolate bug. Turns out that cpan2rpm has an issue with newer versions of perl where Pod::Text no longer has this method. Changing Pod::Text to Pod::Parser in cpan2rpm allowed this to succeed.

sudo cpan2rpm WebService::RTMAgent  --no-sign

-- cpan2rpm - Ver: 2.028 --
Upgrade check
Fetch: HTTP

-- module: WebService::RTMAgent --
Using cached URL: http://search.cpan.org//CPAN/authors/id/R/RU/RUTSCHLE/WebService-RTMAgent-0.5_1.tar.gz
Tarball found - not fetching
Metadata retrieval
Tarball extraction: [/home/tcampbell/src/rpm/SOURCES/WebService-RTMAgent-0.5_1.tar.gz]
Generating spec file
SPEC: /home/tcampbell/src/rpm/SPECS/WebService-RTMAgent.spec
Generating package
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.xs7L3d
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd /home/tcampbell/src/rpm/BUILD
+ rm -rf WebService-RTMAgent-0.5_1
+ /usr/bin/gzip -dc /home/tcampbell/src/rpm/SOURCES/WebService-RTMAgent-0.5_1.tar.gz
+ /bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd WebService-RTMAgent-0.5_1
+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ chmod -R u+w /home/tcampbell/src/rpm/BUILD/WebService-RTMAgent-0.5_1
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.sKaxgX
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ grep -rsl '^#!.*perl' .
+ grep -v '.bak$'
+ xargs --no-run-if-empty /usr/bin/perl -MExtUtils::MakeMaker -e 'MY->fixin(@ARGV)'
+ CFLAGS='-O2 -g -march=i386 -mtune=i686'
++ /usr/bin/perl -MExtUtils::MakeMaker -e ' print qq|PREFIX=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr| if $ExtUtils::MakeMaker::VERSION =~ /5\.9[1-6]|6\.0[0-5]/ '
+ /usr/bin/perl Makefile.PL
Checking if your kit is complete...
Looks good
WARNING: Setting ABSTRACT via file 'lib/WebService/RTMAgent.pm' failed
 at /usr/share/perl5/ExtUtils/MakeMaker.pm line 603
Writing Makefile for WebService::RTMAgent
+ /usr/bin/make
cp lib/WebService/RTMAgent.pm blib/lib/WebService/RTMAgent.pm
Manifying blib/man3/WebService::RTMAgent.3pm
+ /usr/bin/make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ....... 1/1 # Testing WebService::RTMAgent 0.5_1, Perl 5.010001, /usr/bin/perl
t/00-load.t ....... ok
t/auth.t .......... frobbed -- getting token
t/auth.t .......... 1/7 frobbed -- getting token
token token
t/auth.t .......... ok
t/boilerplate.t ... ok
t/init.t .......... ok
t/pod-coverage.t .. skipped: Test::Pod::Coverage 1.08 required for testing POD coverage
t/pod.t ........... ok
t/requests.t ...... 1/12 request:
POST http://www.rememberthemilk.com/services/rest/
Content-Type: application/x-www-form-urlencoded

method=rtm.tasks.add&nam=adding&api_key=key&auth_token=10438&timeline=114114&api_sig=3340edd30a22e9b2c67ff206283d0b67

response:
HTTP/1.1 200 OK
Connection: keep-alive
Date: Mon, 24 Dec 2007 11:49:10 GMT
Server: nginx/RTM
Vary: Accept-Encoding
Content-Type: text/xml; charset="utf-8"
Client-Date: Mon, 24 Dec 2007 11:50:39 GMT
Client-Peer: 75.126.232.204:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
Keep-Alive: timeout=300




t/requests.t ...... ok
t/undo.t .......... ok
All tests successful.

Test Summary Report
-------------------
t/boilerplate.t (Wstat: 0 Tests: 3 Failed: 0)
  TODO passed:   1-3
Files=8, Tests=35,  0 wallclock secs ( 0.05 usr  0.01 sys +  0.59 cusr  0.06 csys =  0.71 CPU)
Result: PASS
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.7ZK9SK
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ '[' /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386 '!=' / ']'
+ rm -rf /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
++ /usr/bin/perl -MExtUtils::MakeMaker -e ' print $ExtUtils::MakeMaker::VERSION <= 6.05 ? qq|PREFIX=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr| : qq|DESTDIR=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386| '
+ make prefix=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr exec_prefix=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr bindir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/bin sbindir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/sbin sysconfdir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/etc datadir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share includedir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/include libdir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/lib libexecdir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/libexec localstatedir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/var sharedstatedir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/var/lib mandir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/man infodir=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/info install DESTDIR=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
Manifying blib/man3/WebService::RTMAgent.3pm
Installing /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/local/share/perl5/WebService/RTMAgent.pm
Installing /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/local/share/man/man3/WebService::RTMAgent.3pm
Appending installation info to /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/lib/perl5/perllocal.pod
+ cmd=/usr/share/spec-helper/compress_files
+ '[' -x /usr/share/spec-helper/compress_files ']'
+ cmd=/usr/lib/rpm/brp-compress
+ '[' -x /usr/lib/rpm/brp-compress ']'
+ /usr/lib/rpm/brp-compress
+ '[' -e /etc/SuSE-release -o -e /etc/UnitedLinux-release ']'
+ find /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386 -name perllocal.pod -o -name .packlist -o -name '*.bs'
+ xargs -i rm -f '{}'
+ find /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr -type d -depth -exec rmdir '{}' ';'
+ /usr/bin/perl -MFile::Find -le '
    find({ wanted => \&wanted, no_chdir => 1}, "/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386");
    print "%doc  Changes README";
    for my $x (sort @dirs, @files) {
        push @ret, $x unless indirs($x);
        }
    print join "\n", sort @ret;

    sub wanted {
        return if /auto$/;

        local $_ = $File::Find::name;
        my $f = $_; s|^\Q/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386\E||;
        return unless length;
        return $files[@files] = $_ if -f $f;

        $d = $_;
        /\Q$d\E/ && return for reverse sort @INC;
        $d =~ /\Q$_\E/ && return
            for qw|/etc /usr/man /usr/bin /usr/share|;

        $dirs[@dirs] = $_;
        }

    sub indirs {
        my $x = shift;
        $x =~ /^\Q$_\E\// && $x ne $_ && return 1 for @dirs;
        }
    '
+ '[' -z WebService-RTMAgent-0.5_1-filelist ']'
+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/brp-strip
+ /usr/lib/rpm/brp-strip-static-archive
+ /usr/lib/rpm/brp-strip-comment-note
Processing files: perl-WebService-RTMAgent-0.5_1-1.noarch
Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.25mTCz
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ DOCDIR=/home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ export DOCDIR
+ rm -rf /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ /bin/mkdir -p /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ cp -pr Changes README /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386/usr/share/doc/perl-WebService-RTMAgent-0.5_1
+ exit 0
Provides: perl(WebService::RTMAgent) = 0.5
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(VersionedDependencies) <= 3.0.3-1
Requires: perl(Carp) perl(Digest::MD5) perl(LWP::UserAgent) perl(XML::Simple) perl(strict) perl(vars)
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
Wrote: /home/tcampbell/src/rpm/SRPMS/perl-WebService-RTMAgent-0.5_1-1.src.rpm
Wrote: /home/tcampbell/src/rpm/RPMS/noarch/perl-WebService-RTMAgent-0.5_1-1.noarch.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.xrdBo2
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ cd WebService-RTMAgent-0.5_1
+ '[' /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386 '!=' / ']'
+ rm -rf /home/tcampbell/src/rpm/BUILDROOT/perl-WebService-RTMAgent-0.5_1-1.i386
+ exit 0
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.eKyVlR
+ umask 022
+ cd /home/tcampbell/src/rpm/BUILD
+ rm -rf WebService-RTMAgent-0.5_1
+ exit 0
RPM: /home/tcampbell/src/rpm/RPMS/noarch/perl-WebService-RTMAgent-0.5_1-1.noarch.rpm
SRPM: /home/tcampbell/src/rpm/SRPMS/perl-WebService-RTMAgent-0.5_1-1.src.rpm
-- Done --

Sweet! now we can install it.

:;  sudo rpm -ivh /home/tcampbell/src/rpm/RPMS/noarch/perl-WebService-RTMAgent-0.5_1-1.noarch.rpm
Preparing...                ########################################### [100%]
   1:perl-WebService-RTMAgen########################################### [100%]

Great. Now we grab the rtm command from the website.

:;  wget http://www.rutschle.net/rtm/rtm-0.5.gz
--2011-01-04 13:47:20--  http://www.rutschle.net/rtm/rtm-0.5.gz
Resolving www.rutschle.net... 82.235.147.6
Connecting to www.rutschle.net|82.235.147.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4212 (4.1K) [application/x-gzip]
Saving to: `rtm-0.5.gz'

100%[===========================================================================>] 4,212       24.3K/s   in 0.2s    

2011-01-04 13:47:21 (24.3 KB/s) - `rtm-0.5.gz' saved [4212/4212]

:; gunzip rtm-0.5.gz

Now, we run it and ... boom.

:;  ./rtm-0.5
Use of uninitialized value in concatenation (.) or string at /usr/local/share/perl5/WebService/RTMAgent.pm line 312.
98: Login failed / Invalid auth token
Use of uninitialized value at /usr/local/share/perl5/WebService/RTMAgent.pm line 368

Looks like we need to get an auth token. Looking inside the script, it appears you have to run rtm with the --authorise option. This creates a URL that you put in the browser to authorize the client with RememberTheMilk. After that was done, I can now run rtm from the command line.

:;  ./rtm-0.5
frobbed -- getting token
token [my token]
0: Turabelle photos
1: do something incredible
2: Ticket to manage notification of changes to bobqueue files.
3: Create ticket to Place bob configs under puppet/avn
4: Rebound payment

Looks like that frobbed/token header go away after the first run.

Now I'm good to go with hitting RememberTheMilk from the command line. Sweet!

Finally updated this.

Posted on 2010/12/23 by hcoyote

Final testing with a post from the iPhone. Yay!

The “Enterprise” …

Posted on 2010/01/21 by hcoyote

From a discussion with a few peers in the industry. I was entertained.

peer> Now when I hear someone use the word “enterprise”
   as an adjective, I have to ask them which of the four meanings
   they intend:
peer> 1. defunct and destroyed (the Enterprise aircraft
   carrier from WW2)
peer> 2. ancient and nearly dead (the Enterprise nuclear
    aircraft carrier)
peer> 3. a nonfunctional mockup (the Enterprise space shuttle)
peer> 4. imaginary (the starship Enterprise)
peer> Typically “enterprise software” fits perfectly in one of
   those four categories.

Name withheld to, of course, protect the guilty.

Lack of backup foils Va.’s new IT system | Richmond Times-Dispatch

Posted on 2009/11/24 by hcoyote

“Every time we’re down for an hour, that’s about 2,500 people inconvenienced,” Smit said. “They’re blaming my people for it and [state IT officials] have an obligation to fix it.”

Lack of backup foils Va.’s new IT system | Richmond Times-Dispatch.

One of the things we’ve been grappling with lately is some unfortunate unplanned outages of services. You know what those are … random event blips caused by butterflies flapping their wings in the South Pacific that stir up turbulence which creates a small wind, that then turns into a hurricane, which rampages over a submarine cable used by the crucial bit of networking that connects you with the rest of the Internet civilization.

A blip.

Sometimes they’re momentary, sometimes they’re bad. What all blips have in common is that they affect a class of your customers in a way that inconviences them in some manner. The hard part of dealing with an outage is understanding and quantifying what the business impact really is. When a database server goes out, you implicitly undrestand that it potentially affects all database users plus all services and users downstream that depend upon the database being up. So how do you realisitically quantify that into a valuable metric?

I bring this up because, as a person in the trenches, I’m able to better understand the impact of something (and therefore, provide a better mitigation plan) if I can understand the size, length, and number of ripples in the fabric that spread out from the blip.

At large companies, this impact may be described as thousands of dollars per minute of cost charged against the bottom line. Some places, like VA, point out the number of people an hour that an outage prevented someone from successfully interacting with the DMV. Websites may see it as the number of advertising impressions that don’t go out due to the site being unavailable.

Whatever metric is used, it needs to be understandable and an order of magnitude that someone can comprehend. I understand impacting 2500 people per hour of down time. I understand costing a company $1 million dollars per minute that the factory is unable to reach it’s control network. I understand an outage costing an engineering team a day’s worth of work (which can ultimately affect the bottom line due to down stream slippage in timelines). What that metric comes down to is being able to understand, in measurable terms, how the blip impacts either people or money.

It’s important to understand these things. Why? Because it allows you to more adequately assess your risk of the (unplanned) outage and design your environment appropriately. If you can point to a solid metric and show how it materially affects people or money, it’s certainly a lot easier to go to management and provide justification for improvements in your environment. If you can only say, with vague hand waving, that there’s AN effect but no data to back that up, you’re just waffling.

So. Have you created your approrpriately detailed outage impact metrics?

I haven’t. But I’m working on it.

Never change anything on Friday at 5pm.

Posted on 2009/11/20 by hcoyote

SEVERE: Socket accept failed
java.net.SocketException: SSL handshake error
javax.net.ssl.SSLException: No available certificate or key corresponds to the
SSL cipher suites which are
enabled.
        at
org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:150)
        at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
        at java.lang.Thread.run(Unknown Source)

This is the reason you never change anything on Friday at 5pm. While attempting to update the SSL certificate for the MySQL Enterprise Monitor (for which the process has no documentation), I managed to break it in a way that caused a few hundred megs of these errors to dump to the catalina log for MEM. Oh, and it meant no monitoring was taking place for a few minutes.

Sigh.

Lesson re-learned. Glad I made a backup of the keystore before I started mucking with it. Now we wait for MySQL to provide me with the correct documentation (after they write it some time this weekend). You would think someone would have already encountered this with their product considering how long it’s been out there already.

At least we have monitoring back.

Resizing a multipathed SAN LUN under RHEL5

Posted on 2009/11/12 by hcoyote

So today I found myself staring at the Red Hat LVM documentation wherein I began to silently cuss. You see, I needed to double the LUN size for MySQL’s usage on one of my servers. These servers are our first that are directly attached into the SAN and running Linux, so we’re working a bit off the map here. To make matters more difficult, we’re using multipathd to manage the SAN connections between the system and SAN, so it wasn’t very clear on what exactly I should do in this case. The LVM docs are somewhat … lacking.

Thanks to the wonder of Google and sheer dumb luck, I ran across this post on one of the RHEL5 mailing lists. In that case, the author wasn’t sure if it was the right method. But, I had a secondary system and was willing to run with scissors for a moment since we’re not fully in production yet.

In short, the basic flow of operations is:

Figure out the multipath I/O device name.
Figure out the underlying device IDs (or device names)
Issue the resize of the LUN in your SAN.
Tell the kernel to rescan the underlying device IDs so it sees the new LUN size.
Tell multipathd that a resize has occurred.
Issue a pvresize so LVM knows it has more extents to work with now.
Issue an lvresize to increase the logical volume size.
Run resize2fs and do an online resize of the filesystem.
Make popcorn.
Watch a movie.

The first time I attempted the process (in a similar, but not quite fashion), I caused the system hang all LVM commands. I turned off multipathd thinking that it would need to be off while I did the resize. This appears to not be a healthy way to do it because I ended up having to warm cycle the system. This is the point where I stopped reading the LVM documentation and found the mailing list post. Tried it out and it worked.

So, without further ado …

#  multipath -ll mpath0
mpath0 (360060160dac711004c6fa9d07c7cde11) dm-2 DGC,RAID 10
[size=50G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][active]
 \_ 1:0:1:0 sdc 8:32  [active][ready]
 \_ 2:0:1:0 sde 8:64  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [active][ready]
 \_ 2:0:0:0 sdd 8:48  [active][ready]

#  for i in `multipath -ll mpath0 | grep sd | awk '{print $2}'`; do
> echo $i ; done
1:0:1:0
2:0:1:0
1:0:0:0
2:0:0:0

#  for i in `multipath -ll mpath0 | grep sd | awk '{print $3}'`; do
> blockdev --rereadpt /dev/$i ; done
BLKRRPART: Input/output error
BLKRRPART: Input/output error

#  multipathd -k"resize multipath mpath0"
ok

#  multipath -ll mpath0
mpath0 (360060160dac711004c6fa9d07c7cde11) dm-2 DGC,RAID 10
[size=100G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][enabled]
 \_ 1:0:1:0 sdc 8:32  [active][ready]
 \_ 2:0:1:0 sde 8:64  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [active][ready]
 \_ 2:0:0:0 sdd 8:48  [active][ready]

#  pvresize /dev/mapper/mpath0
  Physical volume "/dev/mpath/mpath0" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

#  lvresize -L 50G /dev/VolGroupMySQL/mysql-san
  Extending logical volume mysql-san to 50.00 GB
  Logical volume mysql-san successfully resized

#  resize2fs /dev/VolGroupMySQL/mysql-san
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/VolGroupMySQL/mysql-san is mounted on /var/lib/mysql; on-line resizing required
Performing an on-line resize of /dev/VolGroupMySQL/mysql-san to 13107200 (4k) blocks.
The filesystem on /dev/VolGroupMySQL/mysql-san is now 13107200 blocks long.

#  df
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      9.7G  3.3G  6.0G  36% /
/dev/sda1             122M   13M  103M  11% /boot
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/mapper/VolGroupMySQL-mysql--san
                       50G  795M   46G   2% /var/lib/mysql

One thing to note is the BLKRRPART errors from blockdev. This appears to be “normal” as far as I can tell. The kernel through some log messages (included below), but they appear harmless as far as I can discern. The SCSI notices occurred when I issued the blockdev command. The device-mapper multipath warning came from the multipathd resize command.

SCSI device sdc: 209715200 512-byte hdwr sectors (107374 MB)
sdc: Write Protect is off
sdc: Mode Sense: 87 00 00 08
SCSI device sdc: drive cache: write through
sdc: detected capacity change from 53687091200 to 107374182400
 sdc: unknown partition table
SCSI device sde: 209715200 512-byte hdwr sectors (107374 MB)
sde: Write Protect is off
sde: Mode Sense: 87 00 00 08
SCSI device sde: drive cache: write through
sde: detected capacity change from 53687091200 to 107374182400
 sde: unknown partition table
SCSI device sdb: 209715200 512-byte hdwr sectors (107374 MB)
sdb: test WP failed, assume Write Enabled
sdb: asking for cache data failed
sdb: assuming drive cache: write through
sdb: detected capacity change from 53687091200 to 107374182400
 sdb:<6>sd 1:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
    Add. Sense: Logical unit not ready, manual intervention required

end_request: I/O error, dev sdb, sector 0
printk: 68 messages suppressed.

Buffer I/O error on device sdb, logical block 0
 unable to read partition table
SCSI device sdd: 209715200 512-byte hdwr sectors (107374 MB)
sdd: test WP failed, assume Write Enabled
sdd: asking for cache data failed
sdd: assuming drive cache: write through
sdd: detected capacity change from 53687091200 to 107374182400
 sdd:<6>sd 2:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
    Add. Sense: Logical unit not ready, manual intervention required

end_request: I/O error, dev sdd, sector 0
 unable to read partition table
device-mapper: multipath emc: long trespass command will be send
device-mapper: multipath emc: honor reservation bit will not be set (default)
device-mapper: multipath: Using dm hw handler module emc for failover/failback and device management.
device-mapper: multipath emc: emc_pg_init: sending switch-over command

audun.ytterdal.net » Tshark to the rescue (MySQL snooping)

Posted on 2009/11/11 by hcoyote

And while I was at it. I figured out I could do something similar with mysql queries. Instead of turning on full Query-logging in mysql (which probably means a restart of a running production mysql) I could just sniff it
tshark -i eth0 -aduration:60 -d tcp.port==3306,mysql -T fields \
   -e mysql.query 'port 3306'
audun.ytterdal.net » Tshark to the rescue

I’ve been using wireshark for awhile now. I ran across this the other day while trying to debug a non-MySQL issue. It was a good reminder of wireshark’s ability to do live dissection of protocol from the command line. I’ve had to do this before while trying to debug NFS issues (specifically, trying to work backwards from a packet to figure out the name of an open file descriptor from a file handle and fsid). Wireshark’s ability to do this on the fly and from the command line is pretty powerful.

Right now I have a need to figure out why MySQL is showing a bunch of aborted connections. I was hoping that mysql.error would give me something intelligent to poke after, but I think this occurs all within the MySQL instance. More digging to do. It’s good to know that I can do some impromptu poking around of the MySQL packet if I without firing up a full wireshark session on the server if I need to.

If you have a need to do this, these two things will be helpful.

Happy sniffing!

Four short links: Tuesday, Nov 3rd 2009

Posted on 2009/11/03 by hcoyote

Electric Alchemy: Cracking Passwords in the Cloud: Breaking PGP on EC2 with EDPR. An interesting discussion about using Amazon EC2 to make it possible for the underfunded layman to crack passwords.
“There is no YEAR ZERO for HTML. It’s a conversation [...]“. One of the comments on Mark Pilgrim’s post about Why do we have an IMG element? in HTML. Short but good history lesson on how we got here from there.
Innovating Cron: Announcing Norc Do we really need a new cron-like tool? Maybe we do. The Hacker News discussion has some useful bits of information on queuing and batch systems too.
Trax STI. Suburu based snowcat. WANT.

Seth's Blog: Ms. In-between

Posted on 2009/11/03 by hcoyote

If the only reason you’re only wearing one hat is because you’ve always only worn one hat, that’s not a good reason.

Seth’s Blog: Ms. In-between.

Tradition for tradition’s sake is never a good reason to continue doing something in a technical environment. The landscape is ever changing. You should always be re-evaluating what you’re doing while keeping an eye on the bleeding edge of your profession. It’s the difference between being relevant and being the (only) caretaker of a legacy system.

When was the last time you challenged a tradition?

ghostar

brick by brick.