Open Source Operations

I‘m a huge fan of the Windows app TXMouse

By default it avoids a set of apps it doesn’t interact well with. As of early 2010 it does not work correctly with an X11 session running on RealVNC. The symptoms are:

- double-pastes in xterms

- anything mouse highlighted in emacs immediately pastes into the buffer

The TXMouse site has a way to figure out what is going on, here’s what I found for newer RealVNC viewer:

REGEDIT4

[HKEY_CURRENT_USER\Software\<appro@fy.chalmers.se>\TXMouse]
"AutoRaiseDelay"=dword:0
"ExemptedClasses"="rfb::win32::DesktopWindowClass*;WindowsForms10.Window.8.app.0.3*;EXCEEDW:*;XWinClass;cygwin/x*;vncviewer*;tightVNC*;wMFService*;UIMainClass;TSSHELLWND;MSPaintApp;Basilisk*;tty;"
"ExemptedModules"="vmware.exe;msrdp.ocx;mstscax.ocx;wfica.ocx;"
"RapidClickInterval"=dword:0000014d
"TwoButtons"=dword:0

written by admin

I was talking with some friends about automating MySQL backups – here’s a script to backup my wordpress mysql database and mail it to a gmail account for safekeeping. It cleans up after itself too.

I run it from cron every 4th day:

0 4 */4 * * /usr/local/bin/wordpress_backup.bash

-Tony

#!/bin/bash

# This requires 'nail' which takes attachments from the command line. Available
# with "sudo yum install -y nail" 

DUMP_DIR=/home/USER/wordpress_backup
BASE_FILE_NAME=OSC_wordpress_db

# if the dir does not exist create it
if [ ! -d $DUMP_DIR  ]
then
 mkdir -p $DUMP_DIR
fi

# generate a file name with todays date
DUMP_FILE=$DUMP_DIR/${BASE_FILE_NAME}_`date +%Y-%m-%d`.sql.bz2

# run the mysqldump, pipe it through bzip2 and redirect it to the filename
mysqldump --add-drop-table -h localhost -u nnnnnn -pxxxxxx wp_dbase |
 bzip2  -c > ${DUMP_FILE}

# generate an email body, pipe to nail with the bz2 dump attached
(echo; echo; echo "#########"; echo "###" ;date +%Y-%m-%d; echo "autogenerated blog dumpfile $DUMP_FILE" ) |
nail -a ${DUMP_FILE} -s "blog dumpfile $DUMP_FILE" MAILID+blogbackup@gmail.com

if [ ! -z ${BASE_FILE_NAME} -o ! -d $DUMP_DIR ]
 then
 cd ${DUMP_DIR}
 # look for files that match the dump files that are older than 3 days
 # and remove them
 find $DUMP_DIR -name "${BASE_FILE_NAME}*" -mtime +3 |xargs -i rm -v {}
 else
 echo "not doing a find/delete"
fi

written by admin

Whenever I start a new gig I have to key all the boxes – it’s tedious at best. Here’s a bash function to put a key on a far box. I always make sure I have an “eval `ssh-agent`” and “ssh-add” in my shell before running ‘keybox’.

Usage:
keybox [user@]box
Example:
keybox root@hammerforge

Warnings:
Uses ‘mktemp’ which might not be on every OS. Assumes a POSIXy shell at both ends. Tested with bash and zsh.


##############################
##########
# Thu May 28 13:57:25 2009
# will key a box for you
# use like "keybox foobar" OR "keybox user@foobar" it will take
# ~/.ssh/id_dsa.pub (change for your keytype) and write it to the far
# $USER/.ssh/authorized_keys file. Assumes you are using ssh-agent, ssh-add
# for passwdless logins.
#-Tony Thu May 28 16:59:22 2009

written by admin \\ tags: Add new tag

I love the idea of Netomata! I haven’t used it yet, but have often lamented the lack of structure around networking configs. This is not just a great idea for the implementation level, but also for management. If you run your shop with this, a director/manager can learn the tool and get visibility into the entire networking infrastructure rather than having to trace through the decentralized networking equipment configs.

It’s also makes the networking piece of Disaster Recovery significantly easier.

The benefits and why pages are great summaries of why to use centrally generated configs for all machine management. One of the points is “Providing a limited kind of process documentation.” This massivly sells the process short. It would be better to say “Provides unequivocally and 100% repeatable process documentation.”

If you’ve got experience with it, please post a trip report.

-Tony

written by admin

A necessary piece of operations is riding herd on home grown applications and projects from the corporate wilds. These things come to you late in their lifecycle with little to say about how their technology or composition. Often the expectation is that you’ll just take them over and “make them work.” Sometimes that’s doable, but most time there are support limitations.

Here’s the interview and explanation process I use to work with groups outside of Ops to set realistic expectations and about what can and and can’t do for them. It is step 0 of a project plan work. I like to avoid surprised and clearly set expectations about Operations can and can’t do.

How to have Ops take ownership for systems or processes or programs:

1. What is the business justification for this process?
2. Who sponsors the process (outside of operations)?
3. When will the process be turned over to operations?
4. How will your group know the process is in place and being monitored?
5. What are Operations obligations and responsibilities?
6. What are the sponsoring groups obligations and responsibilities?

System category:

1. Requires full/half/quarter time staff member.
2. Existing process needs monitoring and response plan.
3. Trivial process that doesn’t require monitoring.
4. Trivial process that needs monitoring.
5. Ops can monitor but not trouble shoot.
6. Ops can troubleshoot at level 1/2/3 but cannot fix.

Why would ops decline to accept your system, process or program:

1. There may be no way to support the process (for instance it involves on-going manual work – in this case the process likely needs to start at Engineering).
2. It will incur resource costs beyond reasonable levels (i.e. network usage beyond our current capacity, etc.)
3. The sponsoring group does not provide ongoing budgetary support.

What you should expect from us.

1. Integrity and discipline in all our work.
2. A consulting approach to putting your process into production. This means being an organization that is committed to your success and wants to put your work into production.
3. A “closed loop” system that has clear responsibility, reporting, troubleshooting and escalation procedures.

written by admin

I setup dhcpd and tfpt just infrequently enough to forget the details. I’m putting my gottchas here so I don’t forget them.

syslinux package ‘pxelinux’:
pxelinux loads and gets the right IP, then it fails trying to
getting the error “tftp server does not support tsize option”

Fix:

in file /etc/dhcpd.conf:

# absolutly critical to have the next-server line for tftp booting
# when you get "tftp server does not support tsize option" error,
#it's because your missing the config line, Double check with:
#          grep next-server     /etc/dhcpd.conf
#    - Tony 10/17/08
next-server 192.168.0.50;

Troubleshooting:

1] for setting up tftpd you have to make sure there are not entries like
this in /etc/hosts file

127.0.1.1      joust.famemobile.com joust

if so you have to change them to this.

192.168.1.155   joust.famemobile.com joust

2] Using tcpdump for tftp trouble shooting

The fact that loading pxelinux.0 succeeds made me think everything else should work.

The pxelinux.0 loads fine, but the config file ‘pxelinux.cfg/01-00-0c-29-c4-b0-5a’ does not.

05:27:20.882329 IP (tos 0×0, ttl 20, id 2, offset 0, flags [none], proto: UDP (17), length: 55) 192.168.0.51.ah-esp-encap > 192.168.0.50.tftp: [udp sum ok] 27 RRQ “pxelinux.0″ octet tsize 0
05:27:20.893400 IP (tos 0×0, ttl 20, id 4, offset 0, flags [none], proto: UDP (17), length: 60) 192.168.0.51.acp-port > 192.168.0.50.tftp: [udp sum ok] 32 RRQ “pxelinux.0″ octet blksize 1456
05:27:20.953322 IP (tos 0×0, ttl 20, id 29, offset 0, flags [none], proto: UDP (17), length: 91) 192.168.0.51.57089 > 0.0.0.0.tftp: 63 RRQ “pxelinux.cfg/01-00-0c-29-c4-b0-5a” octet tsize 0 blks
… stuff cut out…
05:27:20.972168 IP (tos 0×0, ttl 18, id 44911, offset 0, flags [none], proto: UDP (17), length: 54) 0.0.0.0.tftp > 192.168.0.51.57089: [udp sum ok] 26 ERROR tftp-err-#8 ” tsize option required”

The “0.0.0.0.tftp” is the indicator there is something wrong.

written by admin \\ tags: Add new tag

I use blogger and host the files on my server, after I edit a post it has to sftp the files so they appear here. This is the process for adding them.

Adding the Blogger sftp servers to iptables.

Blogger.com lists their outbound ip’s here. (It was current Jan 19, 2009)

# always check the addresses are correct and the link above.
for i in 66.102.15.83 216.34.7.186 64.233.178.192/28  64.233.178/28
  do
             echo iptables -A INPUT -i eth0 -s $i -p tcp --dport ssh -j ACCEPT
  done

### Output
iptables -A INPUT -i eth0 -s 66.102.15.83 -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -i eth0 -s 216.34.7.186 -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -i eth0 -s 64.233.178.192/28 -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -i eth0 -s 64.233.178/28 -p tcp --dport ssh -j ACCEPT

######## Other notes
I cheated and used ipcalc to the get the subnet calculations:

  ipcalc 64.233.178.192 - 64.233.178.207

  64.233.178.192/28

written by admin

How to test if your rules are being activated:

# logging just the first packet - this shows an external host is reaching you,
# but does not flood messages with notices for every packet.

# Insert at the top of the INPUT chain a request to log only NEW connections
iptables -I INPUT -m state –state NEW -j LOG

Turning off logging on iptables:

# find the logging entry, use –line-number so you know which rule to delete.
iptables -L INPUT –line-number |egrep ‘Chain|LOG’
Chain INPUT (policy DROP)
1 LOG all — anywhere anywhere LOG level warning

# delete it
iptables –delete INPUT 1

## here’s a quicky perl script to get the same info and generate (but not execute) the delete line.

#!/usr/bin/perl

my $CHAIN_NAME;
my $RULE_NUM;

# grab the iptables output
#@iptables_output = qx{iptables -L -n --line-numbers } ;

@iptables_output = qx{~/tmp/iptables -L -n --line-numbers } ;

# cut off the newlines
chomp @iptables_output;

for my $iptables_output_line (@iptables_output) {
    ( $TMP_CHAIN_NAME ) =  $iptables_output_line =~ m/
                      \A         # at the beginning of the line
                      Chain      # match chain
                      \s+
                      (\w+(-)?\w+)
                      /xms
                          and $CHAIN_NAME = $TMP_CHAIN_NAME;

    ($RULE_NUM) = $iptables_output_line =~ m/
                                             \A # at the beginning of the line
                                             (\d)+ # match any number of numbers
                                             \s+   # some space
                                             LOG    # the literal 'LOG'
                                             /xms
                                                 and print "found a log line for $CHAIN_NAME, delete it with:\n",
                                                     "\tiptables --delete $CHAIN_NAME $RULE_NUM\n";

}

######### END perl script #######

#A couple of bash helper functions:
function iptshow () {
iptables -L $1 –line-numbers
}

iptedit () {
vi /etc/sysconfig/iptables
}

written by admin

I always seem to need a tmp file, I used to do ‘vi /tmp/foo’ but it usually had something in it from last time.  This function opens a new file and stores the file name in $f.

I use it like:

vt
<paste some stuff, clean it up>
perl -pe ’s/foo/bar/’ $f

####
function vt () {
    for i in `seq 0 255`;
    do
        FILE=/tmp/$USER-foo-$i;
        if [ -f "$FILE" ]; then
            echo -n '.';
        else
            f=$FILE;
            vi $FILE;
            echo $FILE;
            return;
        fi;
    done
}

###### Cleanup
function cleanvt () {
for i in `seq 0 255`
do
    FILE=/tmp/$USER-foo-$i

    if [ -f "$FILE" ]
    then
    echo -n '.'
    rm $FILE
    else
        echo
        return
fi
done
echo
}

written by admin

If you love perl and are tired of hearing the “executable line noise” and “write only” comments, this book is the antidote.

There is a free preview at Google: Perl Best Practices

The chapter on choosing a brace style and using a beautifier (perltidy) is worth the price of the book. Same goes for the regular expression best practices.

My first copy was ‘mislaid’ :^) Highly recommended.

written by admin