Open Source Operations

I love the idea of Netomata! I haven’t used it yet, but have often lamented the lack of structure around networking configs. This is not just a great idea for the implementation level, but also for management. If you run your shop with this, a director/manager can learn the tool and get visibility into the entire networking infrastructure rather than having to trace through the decentralized networking equipment configs.

It’s also makes the networking piece of Disaster Recovery significantly easier.

The benefits and why pages are great summaries of why to use centrally generated configs for all machine management. One of the points is “Providing a limited kind of process documentation.” This massivly sells the process short. It would be better to say “Provides unequivocally and 100% repeatable process documentation.”

If you’ve got experience with it, please post a trip report.

-Tony

written by admin

A necessary piece of operations is riding herd on home grown applications and projects from the corporate wilds. These things come to you late in their lifecycle with little to say about how their technology or composition. Often the expectation is that you’ll just take them over and “make them work.” Sometimes that’s doable, but most time there are support limitations.

Here’s the interview and explanation process I use to work with groups outside of Ops to set realistic expectations and about what can and and can’t do for them. It is step 0 of a project plan work. I like to avoid surprised and clearly set expectations about Operations can and can’t do.

How to have Ops take ownership for systems or processes or programs:

1. What is the business justification for this process?
2. Who sponsors the process (outside of operations)?
3. When will the process be turned over to operations?
4. How will your group know the process is in place and being monitored?
5. What are Operations obligations and responsibilities?
6. What are the sponsoring groups obligations and responsibilities?

System category:

1. Requires full/half/quarter time staff member.
2. Existing process needs monitoring and response plan.
3. Trivial process that doesn’t require monitoring.
4. Trivial process that needs monitoring.
5. Ops can monitor but not trouble shoot.
6. Ops can troubleshoot at level 1/2/3 but cannot fix.

Why would ops decline to accept your system, process or program:

1. There may be no way to support the process (for instance it involves on-going manual work – in this case the process likely needs to start at Engineering).
2. It will incur resource costs beyond reasonable levels (i.e. network usage beyond our current capacity, etc.)
3. The sponsoring group does not provide ongoing budgetary support.

What you should expect from us.

1. Integrity and discipline in all our work.
2. A consulting approach to putting your process into production. This means being an organization that is committed to your success and wants to put your work into production.
3. A “closed loop” system that has clear responsibility, reporting, troubleshooting and escalation procedures.

written by admin

1. Anything that is not automated has no future. It will die or kill you.
2. People can only do 3 things at once, really. Beyond that energy is spread too thin and everything starts to wind down.
3. Fire heroes. They have a perverse incentive to create problems.
4. There is a straight line correlation between the number of steps in the process and it’s error rate.
5. Anything without an owner will be blamed. If you’re the boss – you’ll be blamed.
6. Computers can only do what their told – if they’re not doing the right thing, its because of a person.
7. The last change broke it.
8. The answer is always in the log file (or the output of trace, strace, ktrace, etc.)
9. No one reads docs. It’s better to automate procedures and only hire people who are able to read code.
10. Success is not writing software, its running it.
11. Use the scientific method as your published troubleshooting method.
12. Pay on-call differentials.
13. Have a rigorous hiring procedure.
14. A Sr. Admin spends his day automating away work or training someone else how to do it. Anything else is mid-level at best.
15. Put all your admins on a development plan that has them automating work.

written by admin