Agile Infrastructure

True stability results when presumed order and presumed disorder are balanced. A truly stable system expects the unexpected, is prepared to be disrupted, waits to be transformed.

-Tom Robbins

One thing that became clear as software practices matured and self-optimized was that not being able to build a project from source in an automated fashion can bring development progress to a grinding halt, particularly as more bodies are added. Without that ability to build from source in a predictable manner, which is the predicate for any flavor of test driven development or continuous integration, the development efforts from a growing team is like so many butterfly wings each capable of unleashing storms on the unsuspecting halfway around the world.

But how many organizations dependent on a web application can reliably build their production servers from bare metal? automatically? unattended? When your application is a ‘service’ on a server, how is that fundamentally different from building a traditional application from source?

How does capacity planning change in a world where ‘Digg’ and ‘Slashdot’ are explicit goals? When Facebook can drive adoption? When adding new servers changes from a purchase order and weeks of waiting to a web service call?

If you want to participate in this ‘as a Service’ brave new world (get up in that ‘aaS’ if you will), and your plan to bring up new servers involves a meatcloud sshing their little hearts out, you might as well give up now. Seriously…

How Agile is your infrastructure?

How Agile is your infrastructure?

Further, what is the plan to manage the life cycle of the servers? Most people have figured out that ‘tail -f’ is not a monitoring solution. But how many of them know exactly what is running on their machines and why? How many have servers that they are afraid to turn off because they aren’t sure what is running, but it might be important? How many configure a server, back away slowly and hope they aren’t the next one who has to touch it?

In another recent episode doing some custom Puppet work with Luke, who has essentially crossed the Developer-Sysadmin divide (I’m not sure he is a chief of the new tribe, but he’s definitely a shaman), Luke became frustrated that he couldn’t write Puppet code like he could Ruby code. (He had not written complex Puppet code for a while, since he stays pretty busy working on Puppet’s internal code.)

Sure, I guess this would be awesome if I was a sysadmin, but I can’t test this code. The only way I can have any confidence it works is to run the whole thing. I guess I just take for granted all the tools that are available to me as a developer now.

Luke Kanies

How does it change things when your infrastructure is code? Can be versioned and diffed? Can be shared and reused? Can be tested? Continuously?

How awesome will PuppetUnit or PuppetSpec be?

Test Driven Infrastructure?

It is only a matter of time…


4 responses to “Agile Infrastructure

  • Jay

    This is a very valid point. Testing and monitoring puppet is difficult. We’ve got over 5,000 lines of .pp files in our puppet repository. (And another 5k lines of erb).

    There isn’t a good way to test puppet other than pushing things live. There’s no way to do continuous integration. Failures are often frustratingly opaque.

    That said, I love me some puppet.

  • botchagalupe

    Test Driven Infrastructure?

    For the enterprise IBM and HP would love to be able to do this. For IBM they take a CMDB type approach and try to do what if’s. You guys are coming at it from a much better perspective. Figure this one out boys and you will be the Chiefs of a new tribe.

  • Nigel Kersten

    This is something we’ve actually been thinking about for a while.

    Using environments to get you a proper release cycle with a version control system that lets you integrate from development -> unstable -> testing -> stable gets us a fair chunk of the way with Puppet, but a large missing part is continuous builds that don’t require user interaction in the unstable/testing phases, along with a way of marking changelists so you can assert appropriate test conditions.

    We’ve actually started building unit tests for our Mac images for client deployment that you can run without actually installing the image on a client, and this is a baby first step towards proper automated testing infrastructure for client deployment.

  • People Over Process » links for 2008-07-24

    […] Agile Infrastructure (tags: via:im puppet agile itmanagement via:littleidea) […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: