Sep 28

Basic Admin: bash – the ‘rm’ habit

bash tips, commandline productivity, linux No Comments »

No matter how good an admin you are, you’ll eventually delete something by accident.  I don’t like ‘rm -i’,  it’s too much.  I use the trick of typing ‘#’ before the rm command. Tab competion still works but if I mess up and hit tab-enter (like I do a lot) the ‘#’ saves me. When I’ve got the line looking like I want I ‘ctrl-a’ to beginning and delete the ‘#’ and execute the line.

# rm -rf  /etc/sysconfig/network-hold
Once I got used to it, it became reflexive. It adds a couple of extra characters, but it doesn’t break my admin flow. 

written by admin

Sep 06

Tools: Win XP and Cygwin desktop

X11, cygwin, cygwin mysql, true X-mouse No Comments »
It drives me crazy to watch an admin figure out how to login to a box with putty. I believe that you can’t manage Unix systems without Unix tools. I switched to Cygwin years ago, except for /dev/random it’s indistingusiable from a workstation. I can do anything I’d do on a Unix box, including installing perl Expect which requires compiling and a tty. It’ll even compile MySQL from source and run it 100% Unix style. It’s seems to have some preformance issues, but you can develop and test with it.  

I love having all my tools on a single box. On the Windows side I use True X-Mouse. It has real sloppy focus and seamless highlight/middle-mouse button for cut-and-paste. 

Under Cygwin I run the light and fast ‘fvwm.’  Here are the instructions and a tar file for doing a Cygwin X11 install from scratch.

written by admin

Sep 06

Tools: dump_cut_buffer

CUT_BUFFER0, X11, shell trick No Comments »
Some history and a cool command. Was originally used to print the X cut_buffer0 to lpr. Most of what it does is better done with a pipe,  but every once in while it’s the exact thing you need. 
Works under Cygwin X11.
#!/bin/sh
# return selected text. Mark Martin. cetia. france.
# xprop returned format is (eg) CUT_BUFFER0 = “echo \”cetia\\\”\n”
# get the \” and \\ and \n undone. sadly sed wont take long lines.
# sed wont read input not ending in newline, so need to chop off final newline
# use bell as newline marker.
# path added for xprop Frank Summers 1-31-91

# bell=^G
# bell=`echo “\07″`

# xprop -root -notype CUT_BUFFER0 |
# sed ‘1s/[^=]*= “//
# s/\\n/’”$bell”‘/g
# s/\\”/”/g
# s/\\\\/\\/g
# $s/”$//’ |
# tr -d ‘\012′ |
# tr $bell ‘\012′|
# lp

# Wed Mar 22 21:42:45 2000
# very old magic.
# Above is the original
# I got this from from an engineer at the Motorola Computer Group.
# i have done a minor rework so it is a bit cleaner in perl.

# Please note this is one long pipeline with comments in between
# Tony Hansmann (t o n y replacewith-at-sign open source consulting com)


xprop -root -notype CUT_BUFFER0|
perl -pe ’s!^CUT_\w+\s=\s\”!!x; # remove CUT_BUFFER0 = ” part
s!\\n!\n!gx; # make newlines appear as such
s!\\”!\”!gx; # make double-quotes appear as such
s!\\\\!\\!gx; # make back-slashs appear as such
s!\\t!\t!gx; # make tabs appear as such
s!\”$!!x;’

written by admin

Sep 06

Tools: bash tricks: The ells

bash tricks, commandline productivity, lll, llll No Comments »
I love writing shell commands and functions. Everyone does something with ‘ll’ mine is:
        alias ll=’ls -lart’ # because I want to see the newest file last.
Lots of times I also want the full path of a file. I could just type
it, but I make a lot of errors when I type long things. A long time
ago I decided cut-and-paste are a dyslexsics best friend. Here are a
couple of helper tools:
  ’lll’ gets me a full path
  ’llll’ gets me a path suitable for use in scp.
I do this a lot:
        tony-ws:log> llll setup.log
        tony-ws:/var/log/setup.log
On another box:
        betlong:log> scp tony-ws:/var/log/setup.log . 
#quick demo
for i in lll llll 
 do echo “this is what $i does: “
    $i hacked-footertracking.html foo.*
 echo
 done
this is what lll does: 
/home/tony/tmp/hacked-footertracking.html
/home/tony/tmp/foo.idx
/home/tony/tmp/foo.pl
/home/tony/tmp/foo.pl~
/home/tony/tmp/foo.sh
/home/tony/tmp/foo.sh~
this is what llll does: 
tony-ws:/home/tony/tmp/hacked-footertracking.html
tony-ws:/home/tony/tmp/foo.idx
tony-ws:/home/tony/tmp/foo.pl
tony-ws:/home/tony/tmp/foo.pl~
tony-ws:/home/tony/tmp/foo.sh
tony-ws:/home/tony/tmp/foo.sh~
############
# often want full path to file(s), this gets it for me
# Tony Hansmann (t o n y replacewith-at-sign open source consulting com)
lll () 
{ 
    if [ ! -z "$*" ]
        then
        for i in $*;
          do
            if (echo $i |egrep -q “^/|\.\./”)
                then
                ls $i
            else
                \ls `pwd`/$i;
            fi
        done
    else
        pwd;
    fi
}


##########
# often need to have a file or dir name in scp format
# box:/full/path/to/file, this gets it for me
# Tony Hansmann (t o n y replacewith-at-sign open source consulting com)
llll () { 
  if [ ! -z "$*" ]
  then
    if [ ! -z "$*" ]
    then
      for i in $*
      do
        if (echo $i |egrep -q “^/|\.\./”)
        then
          echo $(hostname):$(\ls $i);
        else
          echo $(hostname):$(\ls `pwd`/$i);
        fi
      done
    fi

  else
    echo $(hostname):`pwd`;
  fi
}

written by admin

Sep 06

Management: If you don’t program, you’re not a Senior Unix Administrator

executives techniques, hiring, linux unix, opeations linux unix staff management No Comments »

I’d never hire a developer who recommends that a roomful of temps keep a list of websites and send email to subscribers when they’re updated. In the same way, I’d never hire an admin who thinks its ok to by-hand work.

Admin’s, like programmers are paid for their intellectual output, however at at most companies it is treated as a throw-away. There is no insistence on system autobuilds, script repositories, deep monitoring taylored for custom applications, or even the ability to program!

The question of programming is so important that I give interviewees openings to talk about how they solved problems with a script or what language they like best or how they write command-line aliases for anything they have to do more than 3 times. I’ll also ask for code samples, or have them comment some code.

It’s critical that ops staff think about systems from a developers point of view. If I don’t get the strong feeling they love to automate away work they are not eligable for a Sr. Admin position.

written by admin

Sep 06

Management: Tech Screen for UNIX/Linux operations staff

linux hiring, technical screening, unix hiring 1 Comment »

I’ve worked with too many people who had a great interview, but couldn’t do basic admin work. When we looked at how it happened the answer was always the same, no one asked basic questions.

I start a tech screen from this list, branching off of it to follow veins of knowledge and tease out specializations. This process can take an hour or more, but it’s well worth it. I use it on 20 year veterans and newbies exactly the same way. Veterans appreciate the fact that you’re committed to hiring to people who can do their jobs and novices like a company that helps them plot a route through a career as complex as Unix operations.

I explain that the list is like signposts. If I hear the answer I’m looking for, we’ll just go on, no need for complete sentences. Likewise, if they know nothing about the topic I don’t torture them with a bunch of follow-on questions. Aside from the answers, I’m also looking for how they deal with dumb or impossible questions. This gives me an idea of how they’ll be with coworkers and customers.
Requesting Supporting information:
Ask candidate to supply ~/.{bash,ksh,csh}rc ~/.bash_profile or ~/.profile as well as code samples and custom tools they use daily. If scripting is important in your environment, make sure to get significant programs in each language you need them to know.
General
Describe your problem solving method.
Do you use command line editing?  What style?
What is favorite technical book or manual? Read: K&R C? Mythical Man Month? DNS and BIND? The Art of UNIX programming?
Why do you do this work, and how did you get started?
What are some of your beliefs about sysadm.
What was last major jump you made in admin effectiveness?
Describe the best working relationship you’ve ever had.
How do you manage your time?
What admin blogs do you follow and/or contribute to?
What was the last professional meeting you went to and when?
Do you want to be a great admin? How are you pursuing that goal?
Describe your view of the admin project lifecycle.
Hardware
Are you a hardware person?
What is an Ethernet address?
Can calculate power/heat requirements for datacenters?
What is your cable dressing policy? How you keep it enforced?
Tell me everything you know about RAID, SAN attached Storage, NAS, etc.
General Sysadmin
Are you a Tool-smith? A command-line wizard?
When you come to new env that uses ssh, how do you manage it?
How do you setup autologin using ssh?
Where are the system logs and error messages generally stored?
What command would you use to get a general idea of current
system performance?
How do you determine which processes are using the most CPU
time?
How do you terminate a process that won’t respond to user
input?
What is emacs?
What is a daemon?
What is cron? What files control the operation of cron?
Detail some problems you have seen with cron.
What’s the most common way to accidentally overwrite the crontab?
What is the best policy for editing the crontab?
What are zombie processes and what system resources do they
use?
What is x/inetd?
What files control the behavior of x/inetd?
What is the crash utility used for?
Generally what is a daemon doing at the system level when you SIGHUP it?
Giving the changing regulatory and security landscape where to do see  sysadm in 5 years?
Given a 1000 gig disk, how would you slice it? 750 gig? 500 gig?
What are some of UNIX/Linux’s major strengths?
What are some of UNIX/Linux’s major weaknesses?
Experience with NIS, AD, OpenLDAP, PAM and other authentication and integration techniques.
Experience with Syslog-ng or other advance loggers?
Describe log file management.
Describe troubleshooting with log files.
Why is dmesg frustrating?
Describe your day-to-day shell Env (alias, functions, setting, etc).
Favorite command-line tricks and tips?
Describe admin tools you’ve written.
Any experience with DJB tools?
General Networking
What are some of the common networking protocols?
What is a gateway?
What is a routing table?
How do you examine the routing table on a UNIX server?
What is an arp table?
How do you examine the arp table on a UNIX server?
What command is used to configure network interfaces in UNIX?
Defind MTU.  How do you tell what’s it is set to? How do you test that a network path supports it?
How does traceroute work?
DNS
How familiar are you with DNS? Please describe the process of resoveling google.com.
What are the basic record types in a DNS forward file?
How would you find a domain’s mail exchanger?
Describe the general format of an A-record.
Describe the failover process for a lookup.
Describe the operation of a “slave” server.
You want to get a new domain assigned, what do you from start to finish?
How do you accept and properly reply to CIDR delegations?
What are the common mistakes in editing zone files? How do you proof against them?
Mail Transport Agents (MTAs)
What is the most common MTA currently used?
How do you check the status of the mail queue?
What might indicate a mail delivery problem?
What socket or port do MTA’s typically listen on?
How would you find out what socket or port a particular
machine’s MTA is listening on?
X Windows
What is the primary environment variable for X?
What does the xhost command do?
What files control the behavior of X after a user login?
What files control the behavior of the X server prior to user
login?
Shells
What is your favorite shell?  Why?
How would determine which file is the newest file in a very
large directory list?
How would you search a particular file for a match against at
least one of a group of patterns? How about all files?
What are the file descriptors standard in, standard out, and
standard error?

What does the pipe operator do?

How do you determine the amount of free disk space on the
systems?
How do you determine which filesystems are mounted?
Describe some of the Korn shell’s job control features.
What is the pipe limit on the command line?
What is “$?” ?
What value should be stored in $? after the successful
execution of a shell command?
How do you open multiple files in vi?
How do you search through a man page?
A non-technical user enters your work area and frantically
asks, “Who has root here?” How do you respond?
You are assigned the task of changing the root password on 600
machines every month.  How do you do it?
How do you examine/modify NIS maps?
In linux what does the umount -l /remote/share_example do?
What’s the kernal version of the lastest RHEL/CentOS? Why does it matter?
Describe version control systems and the one you use most.
How does Make work?
Shell Programming
What shell languages are you familiar with?
Bash Shell -
What is the “file” command used for?
Name as many of the bash control flow statements as you
can.
Name as many of the bash logical operators as you know,
and describe what they are.
Experience with Distributed shells – examples of applications.
Perl -
How do you print “Hello World!” on standard out?
What does split do?
How do you iterate over text or standard in?
Define array, hash and the commands used to manage them.
Describe how to open a file or command.
Describe a semi-complex regexp.
Describe using SIG handling.
What are some common modules you use a lot.
Where do you go, and how do you find new modules?
Perl Expect
What are the fundamental commands for automating
interaction via Expect?
Monitoring
Describe Nagios or like OSS tools for monitoring?
Sketch how a custom check file works; A custom event handler.
What are some of the challenges with monitoring systems? How do you counter them?

written by admin

Sep 05

Basic knowledge: The Art of Unix Programming

Uncategorized No Comments »

There are a few books that crystalize why UNIX is so powerful and compelling. Eric S. Raymond’s The Art of Unix Programming is one of the best. Once in a while the prose is awkward, but the concepts! 

No matter what your background or field The Basics of the Unix Philosophy chapter is pure joy.  If you are in any way associated with UNIX or Linux this is 40 minutes of reading that will pay off forever. When I’m stuck or having a bad day with clients, I pull it up to remind myself it all does make sense.
-Tony

written by admin