Monthly Archives: July 2008

Agile Infrastructure

True stability results when presumed order and presumed disorder are balanced. A truly stable system expects the unexpected, is prepared to be disrupted, waits to be transformed.

-Tom Robbins

One thing that became clear as software practices matured and self-optimized was that not being able to build a project from source in an automated fashion can bring development progress to a grinding halt, particularly as more bodies are added. Without that ability to build from source in a predictable manner, which is the predicate for any flavor of test driven development or continuous integration, the development efforts from a growing team is like so many butterfly wings each capable of unleashing storms on the unsuspecting halfway around the world.

But how many organizations dependent on a web application can reliably build their production servers from bare metal? automatically? unattended? When your application is a ‘service’ on a server, how is that fundamentally different from building a traditional application from source?

How does capacity planning change in a world where ‘Digg’ and ‘Slashdot’ are explicit goals? When Facebook can drive adoption? When adding new servers changes from a purchase order and weeks of waiting to a web service call?

If you want to participate in this ‘as a Service’ brave new world (get up in that ‘aaS’ if you will), and your plan to bring up new servers involves a meatcloud sshing their little hearts out, you might as well give up now. Seriously…

How Agile is your infrastructure?

How Agile is your infrastructure?

Further, what is the plan to manage the life cycle of the servers? Most people have figured out that ‘tail -f’ is not a monitoring solution. But how many of them know exactly what is running on their machines and why? How many have servers that they are afraid to turn off because they aren’t sure what is running, but it might be important? How many configure a server, back away slowly and hope they aren’t the next one who has to touch it?

In another recent episode doing some custom Puppet work with Luke, who has essentially crossed the Developer-Sysadmin divide (I’m not sure he is a chief of the new tribe, but he’s definitely a shaman), Luke became frustrated that he couldn’t write Puppet code like he could Ruby code. (He had not written complex Puppet code for a while, since he stays pretty busy working on Puppet’s internal code.)

Sure, I guess this would be awesome if I was a sysadmin, but I can’t test this code. The only way I can have any confidence it works is to run the whole thing. I guess I just take for granted all the tools that are available to me as a developer now.

Luke Kanies

How does it change things when your infrastructure is code? Can be versioned and diffed? Can be shared and reused? Can be tested? Continuously?

How awesome will PuppetUnit or PuppetSpec be?

Test Driven Infrastructure?

It is only a matter of time…


Strange is our situation here upon earth. Each of us comes for a short visit, not knowing why, yet sometimes seeming to a divine purpose. From the standpoint of daily life, however, there is one thing we do know: That we are here for the sake of others…for the countless unknown souls with whose fate we are connected by a bond of sympathy. Many times a day, I realize how much my outer and inner life is built upon the labors of people, both living and dead, and how earnestly I must exert myself in order to give in return as much as I have received.

-Albert Einstein

I have some thoughts I just have to get out so I can get back to being productive, so just bare with me.

In Salt Lake City, I know about 4 public places where people will panhandle, plus randomly in front of convenience stores. I think I avoid these places subconsciously. Sometimes, when I am confronted by someone who wants money for food and my schedule and mood permit, I offer to buy them a meal. The majority decline the offer. Sometimes, I have bought someone a plate of food and in exchange asked them questions about their life. I’ve heard some interesting stories, from people who would probably be described as intelligent by any measure other than the fact that they are destitute.

I went from FOOCamp to a San Francisco hotel in the Tenderloin. The accommodations there were quite nice inside, but once on the street, I was struck by what I will describe as ‘social decay’, or at least that is the best two word phrase I have come up with. I haven’t been in a big city for a while and perhaps Salt Lake City’s harsh winter buffers my experience by making the city less than inviting to those with no means, but I found the openness at FOOCamp, juxtaposed with the reality of a big city, jarring to say the least.

In less than 2 hours, I went from wanting to engage with everyone and regretting there wasn’t more time, to wishing I was invisible and not wanting to make eye contact with anyone. The experience was personally painful. Isn’t walking down a street oblivious to the obvious suffering the definition of sociopathic?

The alternative of engaging led to clumsy communication at best, and while I have a hard time not giving someone who claims to be hungry a dollar, I’m not sure I did any good either.

What does this have to do with anything? Honestly, I’m not sure. I could probably sit and weave it into some metaphors and analogies about software teams and dysfunction, about communication impedance, about the problems caused by decisions made by those who will not be effected by them, about changing yourself and your organization by refusing to propagate an unempowering meme, but I don’t feel like being that clever today.

…When one sits in the Hoop Of The People,
one must be responsible because
All of Creation is related.
And the hurt of one is the hurt of all.
And the honor of one is the honor of all.
And whatever we do effects everything in the universe…

passed down from White Buffalo Calf Woman

The problem doesn’t seem like one I can overcome, certainly not by myself. Humanity has obviously struggled with this since the beginning of time. What does it mean to be a ‘citizen’, to be a ‘society’? I don’t know either, but it seems worth thinking about and acting upon. But what to do? Many social programs seem caught up in their own self righteousness and have little impact but well paid executives. We don’t need another one of those, but what would facilitate true long lasting change? Maybe I’m just being too sensitive, but the alternative feels like self deception and a small death of conscience.

Now back to the regularly scheduled program…


OATV Startup/FOO Camp Redux

Last week, Luke and I rolled up the California coast to Sebastopol, headquarters for O’Reilly publishing/conferences. I wanted to keep all the possibilities open so I tried not to have set expectations.

Luke, me and an apple tree

Luke, me and an apple tree

Startup Camp was 8 companies in various stages, but all towards the earlier side, sharing with each other and absorbing advice and inspiration from people that have been there and done that. (metaphorically speaking)

We had sessions with other programmers, experts on creating passionate users, investors, successful entrepreneurs and last but not least Tim O’Reilly. Each of these sessions and people could be a book, let alone a blog post. I have plenty of work to do, but if you are interested you might get some mileage out of searching twitter for startupcamp.

We also got to socialize with some of the people presenting and the founders from some of the OATV portfolio companies. Great people, with invaluable insight and I hope some new life long friendships…

And then there was FOO… Wow, the best word I’ve come up with so far is ‘enchanted’.

How else can you describe camping out with ~400 people all doing amazing things? I’m not sure there are many places where you could have high quality conversations about random technology, economic models, philanthropy, hypercoding, agile development, martial arts, user interfaces, data visualization, the cello, education, geometric paper folding, theoretical physics, film making and Ugandan hip hop, back to back to back. And that was just my thin slice of the experience. Puppet got a lot of love, I just hope no one found me too boorish once the topic came up. Oh, yeah, and when everyone else goes to bed, the hardcore werewolf begins… Hi, I’m Andrew, a villager.

Engaging enough that I didn’t twitter for much of it, and that is saying a lot.

My mind is still spinning from the experience and hopefully some of those racing thoughts will solidify enough to express.

Now we just need to have the kind of year that gets us invited back.

Ignite FOOCamp

I made a plea for help at FOOCamp. My current thoughts and understandings on managing business and people.

Business Philosophy, or something. . . Enjoy

(I would have posted the slides that night but apparently slideshare hates keynote and ppt/pdf generated from keynote.)

A Joining of the Tribes

Forgive him, for he believes that the customs of his tribe are the laws of nature!

-George Bernard Shaw

I’ve spent the last decade involved in software development on some level.

During that time, I’ve built things with Fortan, Java, C/C++, and recently Ruby at the core with Tcl, HTML, CSS, Javascript, XSLT, and SQL among other things as adornments. Never really did much Perl and as a result I suck pretty bad at regular expressions, especially by sysadmin standards (which will become more relevant in a minute).

Whachew looking at?

Can I get some WiFi around here?

I was a ‘Developer’. I had a tribe. Not a LISP druid or a kernel hermit, who tend to be solitary creatures, but a ‘yes, I can do that’, Linux-loving, little-bit-of-everything, do-it-for-the-team developer. (I used to think .Net stormtroopers were my natural enemy, but I’ve opened my heart and learned how to love. I digress. . .)

Parallel to the ‘Developer’ tribe in most organizations, often with a semi-antagonistic mutual dependence, there was always another tribe: ‘Sysadmin’. When Developers and Sysadmins got together, it sometimes felt like the dwarfs and high elves forced to work together by necessity. (I’ll let you workout which is which.)

But something is happening, something is changing. Part of the change has to do with clouds and what utility computing enables, part of the change has to do with the scale that successful web applications must account for and perhaps part of it is that it’s just about damn time. Sandwiched between all these forces, and perhaps in part as a catalyst, is a little open source project with big hopes and dreams: Puppet. (Or maybe I just think about Puppet too much)

All evolution in thought and conduct must first appear as heresy and misconduct
– G.B. Shaw (two for two)

Which brings us back to the tribes… One of the recurring themes at the Velocity conference on operations and web performance was that in this world of highly available, scaled and performant web applications, developers have to think about operations and admins have to think about the applications.

Throwing tools like Puppet into the mix (maybe with a little help from kickstart, fai, PXE booting or a request to EC2) where one can express infrastructure as code and the whole thing goes to another level.

A Reductive Labs client (who shall remain nameless, but has a kick ass Indian developer who was born in Canada, sucked at hockey and should probably give it all up to become a comedian) did an on-site Puppet training about a year ago, and Puppet has become such a part of the development culture that the developers are required to write the Puppet classes that will manage their applications. If there is an operational issue with Puppet managing the application, it is opened as a bug against that code. How cool is that?

Operations is development. It’s going to happen. . .

Some lines are being drawn, you can be a ‘Developer’ who understands not only the bits and algorithms but also the way those things need to behave across racks of servers (or virtual machines) or you can work on that little internal business rules engine. Or you can be a ‘Sysadmin’ who can think about what it takes to automagically scale an application to 100s or 1000s of servers, or you can change tape and plug in cable. Which tribe are you going to be in?

Still have I borne it with a patient shrug, for sufferance is the badge of all our tribe.

-William Shakespeare

And when we all get together to build something incredible and amazing, please don’t hold my feeble RegEx fu against me. (you can tease me a little, I deserve it)

Keepin’ On

Been keeping busy, just checking in. . .

The conferences in SF (Velocity, Cloudcamp, Structure) were a great time to learn, to teach, and make friends. Puppet got a lot of love, but we still have a much grander view of the potential for the project. Puppet is still basically an awesome working prototype, the next year should see improved performance, clean and clear points for integration and a lot more reporting.

Luke showed a video of a GUI reporting application that aggregates and helps visualize Puppet’s reporting. We had a an email the next morning from a Fortune 500 company about getting into the closed beta. Looks like another large internet property is also going to be in.

This last week, I did a tiny bit of development for the 0.24.5 release and finally restyled to be a bit cleaner. I also worked on some slides about Reductive Labs for OATV StartupCamp and threw around some ideas for an Ignite talk for Foo Camp, which are both next week.

I was thinking about ‘What Would Machiavelli Do?’ because Niccolo always gets a bad wrap, but now I’m leaning towards ‘Business Advice From Western Philosophers’. (I’d still throw in some Machiavelli.) I’m going to struggle to boil things down to 20 slides but I think I can make an engaging talk. (I could probably spin at least half a dozen fun talks out with all the different philosophical schools, probably starting with ‘Existentialists‘. And then there is always the Taoists or the Sufis. . .) Feel free to make suggestions in the comments.

StartupCamp and FOOCamp are another great opportunity to get out and spread the Puppet word, but also to learn from a lot of smart people who are doing interesting things. I’m sure my mind will be buzzing this time next week, and some of that is bound to generate some interesting posts. Right?

I also am going to be at Agile 2008, and on that note, some really nice thoughts about software development and infrastructure are starting to crystallize, stay tuned. . . I promise something meaty before StartupCamp starts on the 10th.

%d bloggers like this: