ComputerWorld Opinion: The unspoken truth about managing geeks

Posted on 2009/09/10 by hcoyote

Two quotes stick out from this article for me:

Foundational (bottom-up) respect is not only the largest single determining factor in the success of an IT team, but the most ignored. I believe you can predict success or failure of an IT group simply by assessing the amount of mutual respect within it.

and

The primary task of any IT group is to teach people how to work.IT’s job at the most fundamental level is to build, maintain and improve frameworks within which to accomplish tasks.[...] IT groups literally teach and reteach the world how to work. That’s the job.

ComputerWorld Opinion: The unspoken truth about managing geeks.

First, I can agree with that first quote without any reservation. I’ve been in groups that worked extremely well and those that didn’t. The difference between the two was the amount of mutual respect I saw between the people. Remember, it’s people that make things successful.

Second, I’ve always said that my job was to make your job easier. It’s good to be validated.

Four Short Links: Tues., Sept 8th 2009

Posted on 2009/09/08 by hcoyote

Rand in Repose: Your People — Rand talks about the idea of Your People.

“These are a strange lot of people you’ve discovered in a motley array of places because you were searching for them. [...] while Your People may be less work, they are harder people to have in your life. These are not people that let you sit in place, these are people who hold a mirror up to your [screw-ups], and who explain, in excruciating detail, exactly what you don’t want to hear. If they did not do these things, they would not be Your People.”
Ars Technica: “Anonymized” data really isn’t—and here’s why not —

As Ohm notes, this illustrates a central reality of data collection: “data can either be useful or perfectly anonymous but never both.”

The government hasn’t created 1984. The consumers have.
Dan Gilbert asks, “Why are we happy?” — Cool TED talk about how the “psychological immune system” works to make happiness … well … work … even if the plans we made never come together.
Seth Godin’s Blog: Achievable avalanche opportunities —

“The specific jackpot, sure we’ll sign up for that, but amorphous and ethereal is largely beyond our ability to imagine and sacrifice for.”

Rands In Repose: No Surprises

Posted on 2009/09/04 by hcoyote

In my ideal management world, a review is simply a documentation of well-known facts, your performance over the year. It also contains constructive advice and insight regarding how your boss believes you can improve on that performance. My dream is that you already know all of this information because you’ve been getting year-round feedback from your boss.

I wish.

[...]

Fact is, you’re never fully going to get your boss on-board with your year. There are opinions he has which aren’t going to change which means if you don’t want another surprise next year, you have to change.

A review’s value lies not only in the documentation of what was observed, but also what was not.

Rands In Repose: No Surprises.

I’ve always found yearly reviews to be moderately annoying. Not because I’m placed under a lens and scrutinized, but because all too often the feedback I have gotten is “You’re doing great, keep up the good work!”

I can tell when I’m doing good work. I can also tell when I’m doing bad work. The problem I have is when the only constructive criticism I hear is “you’re doing great!” because that means I have attained oneness with the universe and have become the all-knowing, all-seeing, ever-anticipating the needs of my customers and supplying said needs correctly, ahead of schedule, under budget, and always with a smile.

Right.

It’s amazingly difficult to grow professionally when your weaknesses aren’t pointed out. Everyone has an area that needs to be worked on. It’s the constant struggle for improvement that makes you a better contributor, not the yearly whitewash of “you’re doing great!”. On the flip side, this isn’t a get-out-of-jail-free card for managers to hammer on their employees.

Remember, the criticism must be constructive. Pay attention to what’s going on in your group or organization. Pick one or two things that you’d like to see someone improve upon and let them know in their review. Like Rand, I’d rather see:

“You’re doing good. I’d like to see you step up and work on your presentation skills to customers. It will make you a more effective communicator and help give the organization even more of a polished look. Here’s where you should start …”

than

“You’re doing great!”

Doesn’t the former sound more fulfilling? It means someone is paying attention to what you’re doing and paying attention enough to want to see you improve. The latter? “I’ve heard no complaints so I have no reason to think you’re doing something wrong but I’m too busy to dig in and make sure.”

if you’re not getting the feedback you need, ask for it.

Seth's Blog: Spare no expense!

Posted on 2009/08/28 by hcoyote

The way around it, I think, is to set expectations early and often. If you’re going to give me your phone number, you better answer it. If you’re going to offer a warranty, you better honor it. If you position yourself as a company with real people eager to make every single person happy–you better deliver.

No matter what, you should decide. In advance. How much do you want to spend on ad hoc emergencies, how much do you want to reserve on design and helping the masses improve their experience?

Seth’s Blog: Spare no expense!.

Setting expectations? Amen to that!

Why corporate IT should unchain our office computers. – By Farhad Manjoo – Slate Magazine

Posted on 2009/08/27 by hcoyote

What’s worse, because they aren’t tasked with understanding how people in different parts of a company do their jobs, IT managers often can’t appreciate how profoundly certain tools can improve how we work.

Why corporate IT should unchain our office computers. – By Farhad Manjoo – Slate Magazine.

At first glance, this article made me cringe. As a long-time member of the IT crowd who is trying to make the system administrator profession more legitimate, I hate coming across articles that are blatantly bagging on sysadmins. If you haven’t read the article, the gist of it is: people in IT (sysadmins, management, etc) say no all too often out of a misplaced “fear” that any new technology is bad and dangerous.

Aside: for some reason, I’m reminded of Arthur C. Clark’s comment that, “any sufficiently advanced technology is indistinguishable from magic.”

Anyway, reading the article a few times, you begin to see a kernel of truth here: do we, as IT staff really understand what our customers need and want to accomplish their jobs effectively? It’s easy to blindly say, “No, Google Apps is bad because we don’t control any of it and can’t protect any of the data you put on there.” It’s a lot harder to sit down with a customer and figure out why they really want to use Google Apps instead of what we provide. It’s quite possible we’ve missed out on providing a crucial piece of infrastructure that would be, to them, the next thing since sliced bread, but to us, the obvious response was “WHAT?! Why would anyone EVER want to do THAT?”

So this gets me thinking: how much of a disservice to customers are we doing by blindly chanting the mantra that was passed to us on stone tablets (“Thou shalt not use $technology” or “Thou must only use red paper in the $foo report because it is more secure.”) versus sitting down with them, actually listening to and hearing their complaints, needs, and wishes? Would it really take that much more time to do this compared to fielding all of the complaints and responding with the same chant every time? … and then dealing with the fall out when the customer goes and does their own thing anyway?

When was the last time you looked at what you were providing your customers (and I mean really looked at it) to figure out if it was “right for them”, “right for me” … or, “right for all of us”?

Seth's Blog: Competing with the singleminded

Posted on 2009/08/26 by hcoyote

When you have someone who is willing to accomplish A without worrying about B and C, they will almost always defeat you in accomplishing A. Online, of course, this often leads to doom, since there are many organizations that are willing to get big at the expense of revenue, or writers willing to be noticed at the expense of ethics or reputation. But in the short run, the singleminded have a fantastic advantage. And sometimes, their singleminded focus on accomplishing just that one thing (whatever it is) pushes them through the Dip far ahead of you and then yes, they make a ton of money and you’ve lost forever.

Seth’s Blog: Competing with the singleminded.

As a service provider of sorts, this is yet another reminder to me that being unyielding in what I offer to my customers will unwittingly cause them to go elsewhere if I keep answering their requests with “No, but …”. The goal is to realistically provide them what they need (and shooting for what they want). If all you do is shut them down, they’re going to find someone who doesn’t have your hangups on providing the service.

"Failure is always an option."

Posted on 2009/08/25 by hcoyote

Great video from MythBuster’s Adam Savage on the topic of failure.

I’d embed it, but WP is being wonky about the code.

I think the story on the film school set was very poignant. Sometimes you just have to realize that there’s nothing you can do for a failure but learn from it.

Encrypted MySQL connections (for client and replication)

Posted on 2009/08/07 by hcoyote

There are four basic things you need to do when attempting to set up encrypted MySQL connections.

Make sure your MySQL installation is configured with SSL.
Create a set of certificates for your master, your slave, and your client(s).
Configure your master and slave my.cnf with the correct ssl-* options.
Configure the replication with the SSL options to CHANGE MASTER.

First, let’s check to make sure our installation supports SSL.

master [localhost] {msandbox} ((none)) >  show variables like '%ssl%';
+---------------+----------+
| Variable_name | Value    |
+---------------+----------+
| have_openssl  | DISABLED |
| have_ssl      | DISABLED |
| ssl_ca        |          |
| ssl_capath    |          |
| ssl_cert      |          |
| ssl_cipher    |          |
| ssl_key       |          |
+---------------+----------+
7 rows in set (0.00 sec)

So we see here that SSL is disabled. Bummer. This likely happened because you don’t have the ssl option in your my.cnf. Add it to your configuration and restart your mysqld instance. (There is a configuration file below that you can use as an example.)

master [localhost] {msandbox} ((none)) >  show variables like '%ssl%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| have_openssl  | YES   |
| have_ssl      | YES   |
| ssl_ca        |       |
| ssl_capath    |       |
| ssl_cert      |       |
| ssl_cipher    |       |
| ssl_key       |       |
+---------------+-------+
7 rows in set (0.00 sec)

Good, SSL is enabled and available. Now we need to set up our keys. We’ll need a certificate authority, a server key (multiple in the case of master-slave replicas) and some client keys.
I used the following to generate my own certificate authority and self-signed certificates.

#!/bin/sh
# Generate test SSL keys

mkdir ../certs

cd ../certs

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -md5 -days 1000 -key ca-key.pem \
        -subj "/C=US/ST=Texas/O=My Org/OU=Test/CN=CA" > ca-cert.pem

for target in client server
do
  openssl req -newkey rsa:1024 -md5 -days 1000 -nodes -keyout $target-key.pem \
          -subj "/C=US/ST=Texas/O=My Org/OU=Test/CN=$target" > $target-req.pem

  openssl x509 -req -in $target-req.pem -days 1000 -md5 -CA ca-cert.pem \
          -CAkey ca-key.pem -set_serial 01 > $target-cert.pem
done

cd ..
chgrp -R mysql certs/

So what this gets you is:

:;  ls -l certs/
total 32
-rw-r----- 1 travis staff 1598 May 14 16:31 ca-cert.pem
-rw-r----- 1 travis staff 1675 May 14 16:31 ca-key.pem
-rw-r----- 1 travis staff 1086 May 14 16:31 client-cert.pem
-rw-r----- 1 travis staff  891 May 14 16:31 client-key.pem
-rw-r----- 1 travis staff  692 May 14 16:31 client-req.pem
-rw-r----- 1 travis staff 1086 May 14 16:31 server-cert.pem
-rw-r----- 1 travis staff  887 May 14 16:31 server-key.pem
-rw-r----- 1 travis staff  692 May 14 16:31 server-req.pem

Next, configure the master’s my.cnf. I’m using MySQL Sandbox for this test, so the config should be appropriate for that. You might have to modify accordingly to run outside of a Sandbox.

client]
user            = msandbox
password        = msandbox
port            = 31281
socket          = /tmp/mysql_sandbox31281.sock
ssl-capath                      = /home/travis/sandboxes/cat1_test/certs
ssl-ca                          = /home/travis/sandboxes/cat1_test/certs/ca-cert.pem
ssl-cert                        = /home/travis/sandboxes/cat1_test/certs/client-cert.pem
ssl-key                         = /home/travis/sandboxes/cat1_test/certs/client-key.pem

[mysqld]
user                            = travis
port                            = 31281
socket                          = /tmp/mysql_sandbox31281.sock
basedir                         = /home/travis/opt/mysql/5.0.76
datadir                         = /home/travis/sandboxes/cat1_test/master/data
pid-file                        = /home/travis/sandboxes/cat1_test/master/data/mysql_sandbox31281.pid
#log-slow-queries               = /home/travis/sandboxes/cat1_test/master/data/msandbox-slow.log
#log                            = /home/travis/sandboxes/cat1_test/master/data/msandbox.log
#
# additional options passed through 'my_clause'
#
log-bin=mysql-bin
server-id=1
log-error                       = /home/travis/sandboxes/cat1_test/master/data/msandbox.err
ssl
ssl-capath                      = /home/travis/sandboxes/cat1_test/certs
ssl-ca                          = /home/travis/sandboxes/cat1_test/certs/ca-cert.pem
ssl-cert                        = /home/travis/sandboxes/cat1_test/certs/server-cert.pem
ssl-key                         = /home/travis/sandboxes/cat1_test/certs/server-key.pem

And the corresponding slave configuration:

[client]
user            = msandbox
password        = msandbox
port            = 31282
socket          = /tmp/mysql_sandbox31282.sock
ssl
ssl-capath                      = /home/travis/sandboxes/cat1_test/certs
ssl-ca                          = /home/travis/sandboxes/cat1_test/certs/ca-cert.pem
ssl-cert                        = /home/travis/sandboxes/cat1_test/certs/client-cert.pem
ssl-key                         = /home/travis/sandboxes/cat1_test/certs/client-key.pem

[mysqld]
user                            = travis
port                            = 31282
socket                          = /tmp/mysql_sandbox31282.sock
basedir                         = /home/travis/opt/mysql/5.0.76
datadir                         = /home/travis/sandboxes/cat1_test/node1/data
pid-file                        = /home/travis/sandboxes/cat1_test/node1/data/mysql_sandbox31282.pid
#log-slow-queries               = /home/travis/sandboxes/cat1_test/node1/data/msandbox-slow.log
#log                            = /home/travis/sandboxes/cat1_test/node1/data/msandbox.log
#
# additional options passed through 'my_clause'
#
server-id=101
report-host=SBslave1
report-port=31281
log-bin=mysql-bin
log-error                       = /home/travis/sandboxes/cat1_test/node1/data/msandbox.err
ssl
ssl-capath                      = /home/travis/sandboxes/cat1_test/certs
ssl-ca                          = /home/travis/sandboxes/cat1_test/certs/ca-cert.pem
ssl-cert                        = /home/travis/sandboxes/cat1_test/certs/server-cert.pem
ssl-key                         = /home/travis/sandboxes/cat1_test/certs/server-key.pem

Since I’m being lazy for this test, note that the server key is used in both the master and slave [mysqld] sections. Ordinarily you would have a server key for each mysqld instance.

Now, when you start up your mysql client you either need to specify all the ssl-* options on the command line or be lazy and refer to one of the above my.cnf files. Because we defined a [client] section, it should just work.

Again, I’m using a Sandbox, so we start the client with the generated script. It looks like this:

export LD_LIBRARY_PATH=/home/travis/opt/mysql/5.0.76/lib:/home/travis/opt/mysql/5.0.76/lib/mysql:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=/home/travis/opt/mysql/5.0.76/lib:/home/travis/opt/mysql/5.0.76/lib/mysql:$DYLD_LIBRARY_PATH
SBDIR="/home/travis/sandboxes/cat1_test/master"
BASEDIR=/home/travis/opt/mysql/5.0.76
MYSQL="$BASEDIR/bin/mysql"
PIDFILE="$SBDIR/data/mysql_sandbox31281.pid"
if [ -f $PIDFILE ]
then
    $MYSQL --defaults-file=$SBDIR/my.sandbox.cnf $MYCLIENT_OPTIONS "$@"
fi

When I run this I get logged into my Sandbox master and I can check that SSL is working on my connection.

:;  ./m
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 89228
Server version: 5.0.76-enterprise-gpl-log MySQL Enterprise Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

master [localhost] {msandbox} ((none)) >  show variables like '%ssl%';
+---------------+--------------------------------------------------------+
| Variable_name | Value                                                  |
+---------------+--------------------------------------------------------+
| have_openssl  | YES                                                    |
| have_ssl      | YES                                                    |
| ssl_ca        | /home/travis/sandboxes/cat1_test/certs/ca-cert.pem     |
| ssl_capath    | /home/travis/sandboxes/cat1_test/certs                 |
| ssl_cert      | /home/travis/sandboxes/cat1_test/certs/server-cert.pem |
| ssl_cipher    |                                                        |
| ssl_key       | /home/travis/sandboxes/cat1_test/certs/server-key.pem  |
+---------------+--------------------------------------------------------+
7 rows in set (0.00 sec)

Alternatively, you can start the command line client with the following options:

--ssl-capath=/usr/local/mysql/certs --ssl-cert=client-cert.pem --ssl-key=client-key.pem

Finally, we need to configure the replication to use SSL. I will assume you understand how to set up replication properly and you have it already working on your master-slave pairs.

Ok, so by now you should have your master and slave my.cnf configured with all the SSL variables. On your master, configure the replication user to require SSL.

GRANT REPLICATION SLAVE
ON *.* TO 'msandbox'@'%'
IDENTIFIED BY 'msandbox' REQUIRE SSL;

On your slave, issue the CHANGE MASTER command.

STOP SLAVE;
CHANGE MASTER TO MASTER_HOST = '127.0.0.1',
MASTER_USER = 'msandbox',
MASTER_PASSWORD = 'msandbox',
MASTER_PORT = 31281,
MASTER_LOG_FILE = 'mysql-bin.000001',
MASTER_LOG_POS = 98,
MASTER_SSL = 1,
MASTER_SSL_CA = '/home/travis/sandboxes/cat1_test/certs/ca-cert.pem',
MASTER_SSL_CERT = '/home/travis/sandboxes/cat1_test/certs/client-cert.pem',
MASTER_SSL_KEY = '/home/travis/sandboxes/cat1_test/certs/client-key.pem';
START SLAVE;

And hopefully, if things went well, you’ll see happiness when you issue a SHOW SLAVE STATUS on the slave. Mine looks like:

slave1 [localhost] {msandbox} ((none)) > show slave status \G
*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: 127.0.0.1
                Master_User: msandbox
                Master_Port: 31281
              Connect_Retry: 60
            Master_Log_File: mysql-bin.000016
        Read_Master_Log_Pos: 98
             Relay_Log_File: mysql_sandbox31282-relay-bin.001746
              Relay_Log_Pos: 235
      Relay_Master_Log_File: mysql-bin.000016
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB:
        Replicate_Ignore_DB:
         Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                 Last_Errno: 0
                 Last_Error:
               Skip_Counter: 0
        Exec_Master_Log_Pos: 98
            Relay_Log_Space: 235
            Until_Condition: None
             Until_Log_File:
              Until_Log_Pos: 0
         Master_SSL_Allowed: Yes
         Master_SSL_CA_File: /home/travis/sandboxes/cat1_test/certs/ca-cert.pem
         Master_SSL_CA_Path:
            Master_SSL_Cert: /home/travis/sandboxes/cat1_test/certs/client-cert.pem
          Master_SSL_Cipher:
             Master_SSL_Key: /home/travis/sandboxes/cat1_test/certs/client-key.pem
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

Toolmonger » Blog Archive » Leatherman Charge ALX

Posted on 2009/07/24 by hcoyote

Leatherman Charge ALX

Many multi-tools seem to be a flimsy sampling of little knick-knacks. We’ve seen different versions for ages, but they rely on a simple straight-edged pocket knife to do most of the real work. However, Leatherman’s interesting Charge ALX model is a little different. It does away with many of the old issues multi-tools faced, like those Phillips drivers which always seemed to be less than helpful.

Toolmonger » Blog Archive » Leatherman Charge ALX

I think I’ve found my new multi-tool. I’ve used Leatherman tools since I started in IT. I’ve had three: one of the originals that you can no longer get and two Waves. I lost one Wave at a grocery story. Don’t ask how, it was just stupidity on my part. I gave the original Leatherman to my Dad when I got my first Wave. He used the hell out of it. Even breaking one of the knife blades on it. When he sent it back to Leatherman for repair, he was expecting to have to pay for it. Nope. 25 year warranty covered it. That’s what I like to see in a company.

My second Wave has been with me for going on 5 years now. It’s generally never far from my side. It’s just that useful. I didn’t think Leatherman could make a better multi-tool until the Charge ALX.

And now I want one.

ghostar

brick by brick.

ComputerWorld Opinion: The unspoken truth about managing geeks

Four Short Links: Tues., Sept 8th 2009

Rands In Repose: No Surprises

Seth's Blog: Spare no expense!

Why corporate IT should unchain our office computers. – By Farhad Manjoo – Slate Magazine

Seth's Blog: Competing with the singleminded

"Failure is always an option."

Encrypted MySQL connections (for client and replication)

Toolmonger » Blog Archive » Leatherman Charge ALX