Open Source Operations

It’s challenging to hire a great ops person. How do you judge in an interview?

If I want an A player, I ask hard questions, set high expectations and request their commitment to the teams needs. Even then I make mistakes, but at least I have their commitment. I can work with that if their performance has to be brought up.

Here’s who I have in mind when I’m interviewing:

1. They know they can’t win the first time, they’ll keep trying.
2. They are not looking to change careers or directions – they should have been customer service oriented technical people for last 7 years at least.
3. They hate work and dedicate themselves to eliminating it.
4. They love people.
5. They are students of their work.
6. They are disciplined and self-motivated
7. They accept work – this is an all-day every-day job. We have a “No Slashdot” policy.
1. If you don’t have something else to do you are required to ask the NOC if you can help with a problem or same day ticket
2. Do customer follow up mails.
3. Eliminate work for someone

i. Write a script

ii. Write a tool that lets business users see data they’ve never seen

4. “Widen the Moat” so we can make gains against downtime.
5. Attempt to reduce the complexity of something to the appliance level.
6. Read customer service and management books.
7. Make (or review) a list of people you owe things.

i. Ask them if they are getting what they need.

ii. Ask them how you’ll know when they have exactly what they need.

iii. If there is no clear way to deliver what they want – tell them so.

iv. If you are going to drop their concern – let them know.

8. Put goals, meeting and other important work things on the calendar for the next month, quarter and year.
9. Invent drills to make sure we are where are where we should be in emergency response and disaster recovery.
10. Write a new monitor and figure out how to make it supportable forever.

7. Bad uses of time:
1. Changing the degree of transparency of you xterms for 100^th time.
2. Second guessing management any more than one level above you.
3. Pretending you can’t affect the direction of the management one level above you.
4. Internet reading – even if it’s “background” research on technical things (digg, Slashdot, boing-boing, etc have nothing to do with what we do).
5. Writing documentation no one but you can interpret.

written by admin

I started this blog because I work in a niche area of a niche area of technology – the commercial application of UNIX/Linux compute farms. I was frustrated that what is obvious to me about how to run operations is difficult to communicate to my clients, who are often in operational breakdown.

Because I’m a consultant, I deal with two audiences, the executive who hired me, and the operations staff who I work with.

From the executive side, I make the case that operations is the critical part of their business. My favorite way to say it is: Success is not writing software, it’s running it. They agree, but are often not positioning operations to succeed.

From the staff side, I make the case that capturing everything they do in scripts and a diary will pay off a hundred fold. They agree, but often don’t want to foul up the carefully crafted way they do things.

This blog is for both of you. I have year’s worth of tech tricks, short-cuts and solutions for people who live on the command-line. For managers and executives I have 25+ UNIX shops worth of things that never work.

It’s my intention to provide you information and techniques that will make a difference for your career or business.

-Tony

written by admin

The gsh package comes with ‘ghosts’, a command that generates a list of machines in the class you specify – but nothing else. Using it you can create a “scp to class” command. Just for conventions sake I call it ‘gcp’.

Usage:
gcp

Usage example:
gcp SNPcluster /tmp/new_etc_hosts /etc/hosts

#!/bin/bash

# get the path of ghosts so we can confirm we have ghosts on the #machine
GHOSTS=`type -p ghosts`
if [ -z $GHOSTS ]
then
echo “$0 required ‘ghosts’ to work. Please install it and try again”
echo “exiting”
exit
else

# make sure we have 3 arguments
if [ ! -z $1 -a ! -z $2 -a ! -z $3 ]; then
for i in `ghosts $1`;
do
echo scping to $2 to $i:/$3;
scp $2 $i:$3;
done;
else
echo “need 3 arguments. $0 “;
fi
fi

written by admin

The key to working with a large compute farm is to setup it up to run as single machine or a few classes of machine.

‘gsh’ is a tool that lets you send a command to multiple machines:

http://outflux.net/unix/software/gsh/download/gsh-1.0.2.tar.gz

Here’s the gsh command line to setup the nagios clients on a class of machines at the same time (see nagios client post):

gsh cluster+cf01 “ln -s /bin/bash /bin/rbash ; mkdir -p /usr/local/nagios/libexec ; useradd -d /usr/local/nagios/libexec -s /bin/rbash nagios ; chown root.nagios /usr/local/nagios/libexec ;chmod 750 /usr/local/nagios/libexec”

To use gsh effectively you need to use ssh-agent or open ssh keys. I use ssh-agent and am comfortable with its security implications. If your site doesn’t allow any kind of auto-login process you’ll be stuck entering each passwd, however gsh assures exactly the same command is executed on each machine.

written by admin

Setting up more secure Nagios clients

I’m often called in to organizations that have no monitoring. Setting up a Nagios server is pretty straight forward. But the prospect of installing dozens of nagios client setups by hand is not appealing.

There are lots of ways to get plugin data from clients. My favorite is open-key ssh. I use /bin/rbash on the client to restrict what the shell can do and create a single custom key that is copies to each box (man ssh-keygen for details).

To use ssh on the Nagios server you have to write custom command entries, here’s a sample for the commands.cfg file:
————————————–

define command{         command_name    ssh_check_procs         # the command doesn't have a path because the clients         # use a restricted bash that is not allowed to use         # slashs in the commandline     command_line   ssh -2i /usr/local/nagios/keys/id_client_nagios \ $HOSTADDRESS$ check_procs'         }

—————————————

#Here’s the shell recipe I use to get them going.
# As root!
# Setup the Client environment on machines to be monitored.

# Tony Hansmann (t o n y replacewith-at-sign open source #consulting com)
# use rbash for an added safety with an open SSH key
ln -s /bin/bash /bin/rbash

mkdir -p /usr/local/nagios/libexec

useradd -d /usr/local/nagios/libexec -c “nagios client user” -s /bin/rbash nagios
chown root.nagios /usr/local/nagios/libexec
chmod 750 /usr/local/nagios/libexec

# add the id_client_nagios.pub from the nagios server to
# ~nagios/.ssh/authorized_keys file on each client.
chown nagios .ssh .ssh/authorized_keys

# remove all dot files from client dir
find ~nagios -type f -name “.*” -print -maxdepth 1|xargs rm

# rewrite the .bashrc (do not create any other dot files)
echo “PATH=\$HOME:/usr/bin” > ~nagios/.bashrc
chown root.root ~nagios/.bashrc
chmod 755 ~nagios/.bashrc
# make the interactive login the same config file.
ln -sf ~nagios/.bashrc ~nagios/.bash_profile

########## Back on the server now ##############
# As root scp the server’s ~nagios/libexec/* to client:

scp -r ~nagios/libexec/* root@client:/usr/local/nagios/libexec

written by admin