Open Source Operations

Hyperthreading is back in Intel Processors. We use MatLab which apparently doesn’t like to see 16 cores when there are really only 8.

Here’s a script to flip procs 8-and-up  on and off. Change the perl ‘>= 8′   part to set it for your environment.

Thanks to http://dag.wieers.com/blog/is-hyper-threading-enabled-on-a-linux-system for

#!/bin/bash

if [ ! -z $1 ]
then
 # fancy bash block method of doing if else;
 $(echo $1 | grep -qi "^on$") && {
 echo turning CPU from 8 up ON ;
 CPU_VALUE=1 ;
}

$(echo $1 | grep -qi "^off$") &&  {
 echo turning CPU from 8 up OFF ;
 CPU_VALUE=0 ;
}

for i in $(find /sys/devices/system/node -name "cpu[0-9]*" |perl -ne  '($i)= m{u(\d+)$}xms; if ($i >= 8){print}; undef $i;' )
do
 if [ -d $i  ]
 then
 echo $i
 echo "$CPU_VALUE"  > $i/online 2> /dev/null
 else
 echo $i is not a dir
 fi

done

else
 echo "must give an argument of 'on' or 'off'"
fi

echo Running processors
cat /proc/cpuinfo|egrep '^processor'

written by admin

I‘m a huge fan of the Windows app TXMouse

By default it avoids a set of apps it doesn’t interact well with. As of early 2010 it does not work correctly with an X11 session running on RealVNC. The symptoms are:

- double-pastes in xterms

- anything mouse highlighted in emacs immediately pastes into the buffer

The TXMouse site has a way to figure out what is going on, here’s what I found for newer RealVNC viewer:

REGEDIT4

[HKEY_CURRENT_USER\Software\<appro@fy.chalmers.se>\TXMouse]
"AutoRaiseDelay"=dword:0
"ExemptedClasses"="rfb::win32::DesktopWindowClass*;WindowsForms10.Window.8.app.0.3*;EXCEEDW:*;XWinClass;cygwin/x*;vncviewer*;tightVNC*;wMFService*;UIMainClass;TSSHELLWND;MSPaintApp;Basilisk*;tty;"
"ExemptedModules"="vmware.exe;msrdp.ocx;mstscax.ocx;wfica.ocx;"
"RapidClickInterval"=dword:0000014d
"TwoButtons"=dword:0

written by admin

I was talking with some friends about automating MySQL backups – here’s a script to backup my wordpress mysql database and mail it to a gmail account for safekeeping. It cleans up after itself too.

I run it from cron every 4th day:

0 4 */4 * * /usr/local/bin/wordpress_backup.bash

-Tony

#!/bin/bash

# This requires 'nail' which takes attachments from the command line. Available
# with "sudo yum install -y nail" 

DUMP_DIR=/home/USER/wordpress_backup
BASE_FILE_NAME=OSC_wordpress_db

# if the dir does not exist create it
if [ ! -d $DUMP_DIR  ]
then
 mkdir -p $DUMP_DIR
fi

# generate a file name with todays date
DUMP_FILE=$DUMP_DIR/${BASE_FILE_NAME}_`date +%Y-%m-%d`.sql.bz2

# run the mysqldump, pipe it through bzip2 and redirect it to the filename
mysqldump --add-drop-table -h localhost -u nnnnnn -pxxxxxx wp_dbase |
 bzip2  -c > ${DUMP_FILE}

# generate an email body, pipe to nail with the bz2 dump attached
(echo; echo; echo "#########"; echo "###" ;date +%Y-%m-%d; echo "autogenerated blog dumpfile $DUMP_FILE" ) |
nail -a ${DUMP_FILE} -s "blog dumpfile $DUMP_FILE" MAILID+blogbackup@gmail.com

if [ ! -z ${BASE_FILE_NAME} -o ! -d $DUMP_DIR ]
 then
 cd ${DUMP_DIR}
 # look for files that match the dump files that are older than 3 days
 # and remove them
 find $DUMP_DIR -name "${BASE_FILE_NAME}*" -mtime +3 |xargs -i rm -v {}
 else
 echo "not doing a find/delete"
fi

written by admin

I love the idea of Netomata! I haven’t used it yet, but have often lamented the lack of structure around networking configs. This is not just a great idea for the implementation level, but also for management. If you run your shop with this, a director/manager can learn the tool and get visibility into the entire networking infrastructure rather than having to trace through the decentralized networking equipment configs.

It’s also makes the networking piece of Disaster Recovery significantly easier.

The benefits and why pages are great summaries of why to use centrally generated configs for all machine management. One of the points is “Providing a limited kind of process documentation.” This massivly sells the process short. It would be better to say “Provides unequivocally and 100% repeatable process documentation.”

If you’ve got experience with it, please post a trip report.

-Tony

written by admin

A necessary piece of operations is riding herd on home grown applications and projects from the corporate wilds. These things come to you late in their lifecycle with little to say about how their technology or composition. Often the expectation is that you’ll just take them over and “make them work.” Sometimes that’s doable, but most time there are support limitations.

Here’s the interview and explanation process I use to work with groups outside of Ops to set realistic expectations and about what can and and can’t do for them. It is step 0 of a project plan work. I like to avoid surprised and clearly set expectations about Operations can and can’t do.

How to have Ops take ownership for systems or processes or programs:

1. What is the business justification for this process?
2. Who sponsors the process (outside of operations)?
3. When will the process be turned over to operations?
4. How will your group know the process is in place and being monitored?
5. What are Operations obligations and responsibilities?
6. What are the sponsoring groups obligations and responsibilities?

System category:

1. Requires full/half/quarter time staff member.
2. Existing process needs monitoring and response plan.
3. Trivial process that doesn’t require monitoring.
4. Trivial process that needs monitoring.
5. Ops can monitor but not trouble shoot.
6. Ops can troubleshoot at level 1/2/3 but cannot fix.

Why would ops decline to accept your system, process or program:

1. There may be no way to support the process (for instance it involves on-going manual work – in this case the process likely needs to start at Engineering).
2. It will incur resource costs beyond reasonable levels (i.e. network usage beyond our current capacity, etc.)
3. The sponsoring group does not provide ongoing budgetary support.

What you should expect from us.

1. Integrity and discipline in all our work.
2. A consulting approach to putting your process into production. This means being an organization that is committed to your success and wants to put your work into production.
3. A “closed loop” system that has clear responsibility, reporting, troubleshooting and escalation procedures.

written by admin

#!/usr/bin/perl

##############################
##########
# Fri Nov 21 15:36:39 2008

# A util I’ve always wanted! when you have a perms problem in unix you
# need to know the perms on each level above the file. AFAIK there’s
# no way to get them other than typing ls for each element of the
# path. This cuts that process down.

while ( my $_ = shift @ARGV) {
my @broken_down_path;
my @path_list;
my %ls_line;

chomp;
# get the path elemets so we can see the perms at each level.
@broken_down_path = split q{/}, “$_” ;

# the first element from the split is nul, pulling it out and
# putting a ’slash’ in so I get the entire tree
shift @path_list; unshift @path_list, qq{/};

for (0..$#broken_down_path) {
my $rebuild_path_element;
# as we go thru the array we building up an list that makes up the path
$rebuild_path_element = join q{/}, @broken_down_path[0..$_];
push @path_list, $rebuild_path_element;
}

#
for my $path (@path_list) {
# get the output of ls -ld on every level of the path, story in a hash of arrays
push @{$ls_line{$path}}, split q{ }, qx{/bin/ls -ld $path} ;
}

# rev sort the keys so we get the file at the top the dirs underneith.
for my $path (reverse sort keys %ls_line) {
# make sure $path is not null.
$path or next ;
# just the perms owner and group
print join qq{\t}, @{$ls_line{$path}}[0,2,3];
# put in the path
print “\t$path\n” ;
}

# if there are more to go, print a space.
@ARGV ? print “\n” : ”;

} # end for (@ARGV)

=pod

sample usage and output. Look at permissions at all tree levels to
figure out while a user can’t read a file or dir.

tony-ws:bin> tree_perms.pl /home/tony/work/getactive/kickstart_configs/post_install_scripts/_base/service_control.bash /home/tony/bin/At.pm
-rw——- tony None /home/tony/work/getactive/kickstart_configs/post_install_scripts/_base/service_control.bash
drwx——+ tony None /home/tony/work/getactive/kickstart_configs/post_install_scripts/_base
drwx——+ tony None /home/tony/work/getactive/kickstart_configs/post_install_scripts
drwx——+ tony None /home/tony/work/getactive/kickstart_configs
drwx——+ tony None /home/tony/work/getactive
drwx——+ tony None /home/tony/work
drwxrwxrwx+ tony None /home/tony
drwxrwxrwx+ tony None /home
drwxrwx—+ tony Users /

-rw——- tony None /home/tony/bin/At.pm
drwxr-xr-x+ tony None /home/tony/bin
drwxrwxrwx+ tony None /home/tony
drwxrwxrwx+ tony None /home
drwxrwx—+ tony Users /

=cut

written by admin

There are a few books that crystalize why UNIX is so powerful and compelling. Eric S. Raymond’s The Art of Unix Programming is one of the best. Once in a while the prose is awkward, but the concepts! 

written by admin

The gsh package comes with ‘ghosts’, a command that generates a list of machines in the class you specify – but nothing else. Using it you can create a “scp to class” command. Just for conventions sake I call it ‘gcp’.

Usage:
gcp

Usage example:
gcp SNPcluster /tmp/new_etc_hosts /etc/hosts

#!/bin/bash

# get the path of ghosts so we can confirm we have ghosts on the #machine
GHOSTS=`type -p ghosts`
if [ -z $GHOSTS ]
then
echo “$0 required ‘ghosts’ to work. Please install it and try again”
echo “exiting”
exit
else

# make sure we have 3 arguments
if [ ! -z $1 -a ! -z $2 -a ! -z $3 ]; then
for i in `ghosts $1`;
do
echo scping to $2 to $i:/$3;
scp $2 $i:$3;
done;
else
echo “need 3 arguments. $0 “;
fi
fi

written by admin

Setting up more secure Nagios clients

I’m often called in to organizations that have no monitoring. Setting up a Nagios server is pretty straight forward. But the prospect of installing dozens of nagios client setups by hand is not appealing.

There are lots of ways to get plugin data from clients. My favorite is open-key ssh. I use /bin/rbash on the client to restrict what the shell can do and create a single custom key that is copies to each box (man ssh-keygen for details).

To use ssh on the Nagios server you have to write custom command entries, here’s a sample for the commands.cfg file:
————————————–

define command{         command_name    ssh_check_procs         # the command doesn't have a path because the clients         # use a restricted bash that is not allowed to use         # slashs in the commandline     command_line   ssh -2i /usr/local/nagios/keys/id_client_nagios \ $HOSTADDRESS$ check_procs'         }

—————————————

#Here’s the shell recipe I use to get them going.
# As root!
# Setup the Client environment on machines to be monitored.

# Tony Hansmann (t o n y replacewith-at-sign open source #consulting com)
# use rbash for an added safety with an open SSH key
ln -s /bin/bash /bin/rbash

mkdir -p /usr/local/nagios/libexec

useradd -d /usr/local/nagios/libexec -c “nagios client user” -s /bin/rbash nagios
chown root.nagios /usr/local/nagios/libexec
chmod 750 /usr/local/nagios/libexec

# add the id_client_nagios.pub from the nagios server to
# ~nagios/.ssh/authorized_keys file on each client.
chown nagios .ssh .ssh/authorized_keys

# remove all dot files from client dir
find ~nagios -type f -name “.*” -print -maxdepth 1|xargs rm

# rewrite the .bashrc (do not create any other dot files)
echo “PATH=\$HOME:/usr/bin” > ~nagios/.bashrc
chown root.root ~nagios/.bashrc
chmod 755 ~nagios/.bashrc
# make the interactive login the same config file.
ln -sf ~nagios/.bashrc ~nagios/.bash_profile

########## Back on the server now ##############
# As root scp the server’s ~nagios/libexec/* to client:

scp -r ~nagios/libexec/* root@client:/usr/local/nagios/libexec

written by admin