In truth, a good case could be made that if your knowledge is meagre and unsatisfactory, the last thing in the world you should do is make measurements; the chance is negligible that you will measure the right things accidentally.

- George Miller

Last week I drove to Los Angeles with my wife, two kids, my mother-in-law and her sister in our not so eco-friendly SUV. Hilarity ensued, but that’s not the point of this post.

I don’t usually drive this car. Until recently, most everything in my life was in about a 5 mile radius, work, play, whatever. The SUV is ‘her’ car; my wife drives it to where she does research and home most everyday.  Occasionally, we all get in and go somewhere exciting, like grandma’s.

You are starting to wonder what is the point, and I’m hoping I’m about to make some.

This late model SUV has a speedometer, a tachometer, and in addition has a computer that is estimating instantaneous MPG and average MPG since the last fillup, since the beginning of the trip, since the epoch, etc.

On the way to LA, with this SUV stuffed full of people and random stuff, I averaged just over 20 MPG at ~80 MPH. On the way back, I averaged 25.6 MPG with the same people and even more stuff and nearly identical MPH.

On the way there, I kept watching the gauges and thinking ‘these rolling hills are killing my gas mileage’. On the way back, I was optimizing the speed and acceleration patterns, only dipping below 20 MPG on the steepest inclines.

If one rides a bike on hills, he or she figures this out pretty fast. There are certain speeds you can hit a hill that can be maintained with less effort than it will take to go slower, all this is based on fitness, the equipment and the hill. Also, acceleration uphill is fairly expensive. Your body figures this stuff out because energy expenditure and efficiency have consequences, and the feedback is rapid.

In an automobile, I doubt I ever would have figured it out without the gadgetry.

You can’t manage what you can’t measure.

What does this have to do with software? Velocity is a commonly used idiom in project planning. I’ve seen velocity measured different ways in practice, and seen it’s measurement discussed in many more.

But I’ve never really seen efficiency measured with relation to velocity. The only time efficiency is considered is with respect to getting more velocity.

I’ve never seen anyone in a planning meeting ask, how can we maintain this velocity more efficiently? Is this velocity the most efficient? Over the life of any decent sized project, this seems like it would be imporant.

(Aside: The XPers might as they have a notion of sustainable pace, but I’ve never been in a pure XP setting. I’m starting to think they only exist in legends or they are like the Jedi scattered by the clone wars trying to cultivate the force in swamps and deserts… I’ll be at Agile 2008 this week, so I’m hoping to come home with some new Jedi minds tricks.)

How many miles per gallon are you getting? Are you losing MPG to the rolling hills?

Perhaps the more important question, how would you know? What do you measure? (The instantaneous estimate feedback was key for me, hmm…)

True stability results when presumed order and presumed disorder are balanced. A truly stable system expects the unexpected, is prepared to be disrupted, waits to be transformed.

-Tom Robbins

One thing that became clear as software practices matured and self-optimized was that not being able to build a project from source in an automated fashion can bring development progress to a grinding halt, particularly as more bodies are added. Without that ability to build from source in a predictable manner, which is the predicate for any flavor of test driven development or continuous integration, the development efforts from a growing team is like so many butterfly wings each capable of unleashing storms on the unsuspecting halfway around the world.

But how many organizations dependent on a web application can reliably build their production servers from bare metal? automatically? unattended? When your application is a ’service’ on a server, how is that fundamentally different from building a traditional application from source?

How does capacity planning change in a world where ‘Digg’ and ‘Slashdot’ are explicit goals? When Facebook can drive adoption? When adding new servers changes from a purchase order and weeks of waiting to a web service call?

If you want to participate in this ‘as a Service’ brave new world (get up in that ‘aaS’ if you will), and your plan to bring up new servers involves a meatcloud sshing their little hearts out, you might as well give up now. Seriously…

How Agile is your infrastructure?

Further, what is the plan to manage the life cycle of the servers? Most people have figured out that ‘tail -f’ is not a monitoring solution. But how many of them know exactly what is running on their machines and why? How many have servers that they are afraid to turn off because they aren’t sure what is running, but it might be important? How many configure a server, back away slowly and hope they aren’t the next one who has to touch it?

In another recent episode doing some custom Puppet work with Luke, who has essentially crossed the Developer-Sysadmin divide (I’m not sure he is a chief of the new tribe, but he’s definitely a shaman), Luke became frustrated that he couldn’t write Puppet code like he could Ruby code. (He had not written complex Puppet code for a while, since he stays pretty busy working on Puppet’s internal code.)

Sure, I guess this would be awesome if I was a sysadmin, but I can’t test this code. The only way I can have any confidence it works is to run the whole thing. I guess I just take for granted all the tools that are available to me as a developer now.

-Luke Kanies

How does it change things when your infrastructure is code? Can be versioned and diffed? Can be shared and reused? Can be tested? Continuously?

How awesome will PuppetUnit or PuppetSpec be?

Test Driven Infrastructure?

It is only a matter of time…

Strange is our situation here upon earth. Each of us comes for a short visit, not knowing why, yet sometimes seeming to a divine purpose. From the standpoint of daily life, however, there is one thing we do know: That we are here for the sake of others…for the countless unknown souls with whose fate we are connected by a bond of sympathy. Many times a day, I realize how much my outer and inner life is built upon the labors of people, both living and dead, and how earnestly I must exert myself in order to give in return as much as I have received.

-Albert Einstein

I have some thoughts I just have to get out so I can get back to being productive, so just bare with me.

In Salt Lake City, I know about 4 public places where people will panhandle, plus randomly in front of convenience stores. I think I avoid these places subconsciously. Sometimes, when I am confronted by someone who wants money for food and my schedule and mood permit, I offer to buy them a meal. The majority decline the offer. Sometimes, I have bought someone a plate of food and in exchange asked them questions about their life. I’ve heard some interesting stories, from people who would probably be described as intelligent by any measure other than the fact that they are destitute.

I went from FOOCamp to a San Francisco hotel in the Tenderloin. The accommodations there were quite nice inside, but once on the street, I was struck by what I will describe as ’social decay’, or at least that is the best two word phrase I have come up with. I haven’t been in a big city for a while and perhaps Salt Lake City’s harsh winter buffers my experience by making the city less than inviting to those with no means, but I found the openness at FOOCamp, juxtaposed with the reality of a big city, jarring to say the least.

In less than 2 hours, I went from wanting to engage with everyone and regretting there wasn’t more time, to wishing I was invisible and not wanting to make eye contact with anyone. The experience was personally painful. Isn’t walking down a street oblivious to the obvious suffering the definition of sociopathic?

The alternative of engaging led to clumsy communication at best, and while I have a hard time not giving someone who claims to be hungry a dollar, I’m not sure I did any good either.

What does this have to do with anything? Honestly, I’m not sure. I could probably sit and weave it into some metaphors and analogies about software teams and dysfunction, about communication impedance, about the problems caused by decisions made by those who will not be effected by them, about changing yourself and your organization by refusing to propagate an unempowering meme, but I don’t feel like being that clever today.

…When one sits in the Hoop Of The People,
one must be responsible because
All of Creation is related.
And the hurt of one is the hurt of all.
And the honor of one is the honor of all.
And whatever we do effects everything in the universe…

passed down from White Buffalo Calf Woman

The problem doesn’t seem like one I can overcome, certainly not by myself. Humanity has obviously struggled with this since the beginning of time. What does it mean to be a ‘citizen’, to be a ’society’? I don’t know either, but it seems worth thinking about and acting upon. But what to do? Many social programs seem caught up in their own self righteousness and have little impact but well paid executives. We don’t need another one of those, but what would facilitate true long lasting change? Maybe I’m just being too sensitive, but the alternative feels like self deception and a small death of conscience.

Now back to the regularly scheduled program…

sigh…

Last week, Luke and I rolled up the California coast to Sebastopol, headquarters for O’Reilly publishing/conferences. I wanted to keep all the possibilities open so I tried not to have set expectations.

Luke, me and an apple tree

Startup Camp was 8 companies in various stages, but all towards the earlier side, sharing with each other and absorbing advice and inspiration from people that have been there and done that. (metaphorically speaking)

We had sessions with other programmers, experts on creating passionate users, investors, successful entrepreneurs and last but not least Tim O’Reilly. Each of these sessions and people could be a book, let alone a blog post. I have plenty of work to do, but if you are interested you might get some mileage out of searching twitter for startupcamp.

We also got to socialize with some of the people presenting and the founders from some of the OATV portfolio companies. Great people, with invaluable insight and I hope some new life long friendships…

And then there was FOO… Wow, the best word I’ve come up with so far is ‘enchanted’.

How else can you describe camping out with ~400 people all doing amazing things? I’m not sure there are many places where you could have high quality conversations about random technology, economic models, philanthropy, hypercoding, agile development, martial arts, user interfaces, data visualization, the cello, education, geometric paper folding, theoretical physics, film making and Ugandan hip hop, back to back to back. And that was just my thin slice of the experience. Puppet got a lot of love, I just hope no one found me too boorish once the topic came up. Oh, yeah, and when everyone else goes to bed, the hardcore werewolf begins… Hi, I’m Andrew, a villager.

Engaging enough that I didn’t twitter for much of it, and that is saying a lot.

My mind is still spinning from the experience and hopefully some of those racing thoughts will solidify enough to express.

Now we just need to have the kind of year that gets us invited back.

I made a plea for help at FOOCamp. My current thoughts and understandings on managing business and people.

Business Philosophy, or something. . . Enjoy

(I would have posted the slides that night but apparently slideshare hates keynote and ppt/pdf generated from keynote.)

Forgive him, for he believes that the customs of his tribe are the laws of nature!

-George Bernard Shaw

I’ve spent the last decade involved in software development on some level.

During that time, I’ve built things with Fortan, Java, C/C++, and recently Ruby at the core with Tcl, HTML, CSS, Javascript, XSLT, and SQL among other things as adornments. Never really did much Perl and as a result I suck pretty bad at regular expressions, especially by sysadmin standards (which will become more relevant in a minute).

Can I get some WiFi around here?

I was a ‘Developer’. I had a tribe. Not a LISP druid or a kernel hermit, who tend to be solitary creatures, but a ‘yes, I can do that’, Linux-loving, little-bit-of-everything, do-it-for-the-team developer. (I used to think .Net stormtroopers were my natural enemy, but I’ve opened my heart and learned how to love. I digress. . .)

Parallel to the ‘Developer’ tribe in most organizations, often with a semi-antagonistic mutual dependence, there was always another tribe: ‘Sysadmin’. When Developers and Sysadmins got together, it sometimes felt like the dwarfs and high elves forced to work together by necessity. (I’ll let you workout which is which.)

But something is happening, something is changing. Part of the change has to do with clouds and what utility computing enables, part of the change has to do with the scale that successful web applications must account for and perhaps part of it is that it’s just about damn time. Sandwiched between all these forces, and perhaps in part as a catalyst, is a little open source project with big hopes and dreams: Puppet. (Or maybe I just think about Puppet too much)

All evolution in thought and conduct must first appear as heresy and misconduct
- G.B. Shaw (two for two)

Which brings us back to the tribes… One of the recurring themes at the Velocity conference on operations and web performance was that in this world of highly available, scaled and performant web applications, developers have to think about operations and admins have to think about the applications.

Throwing tools like Puppet into the mix (maybe with a little help from kickstart, fai, PXE booting or a request to EC2) where one can express infrastructure as code and the whole thing goes to another level.

A Reductive Labs client (who shall remain nameless, but has a kick ass Indian developer who was born in Canada, sucked at hockey and should probably give it all up to become a comedian) did an on-site Puppet training about a year ago, and Puppet has become such a part of the development culture that the developers are required to write the Puppet classes that will manage their applications. If there is an operational issue with Puppet managing the application, it is opened as a bug against that code. How cool is that?

Operations is development. It’s going to happen. . .

Some lines are being drawn, you can be a ‘Developer’ who understands not only the bits and algorithms but also the way those things need to behave across racks of servers (or virtual machines) or you can work on that little internal business rules engine. Or you can be a ‘Sysadmin’ who can think about what it takes to automagically scale an application to 100s or 1000s of servers, or you can change tape and plug in cable. Which tribe are you going to be in?

Still have I borne it with a patient shrug, for sufferance is the badge of all our tribe.

-William Shakespeare

And when we all get together to build something incredible and amazing, please don’t hold my feeble RegEx fu against me. (you can tease me a little, I deserve it)

Been keeping busy, just checking in. . .

The conferences in SF (Velocity, Cloudcamp, Structure) were a great time to learn, to teach, and make friends. Puppet got a lot of love, but we still have a much grander view of the potential for the project. Puppet is still basically an awesome working prototype, the next year should see improved performance, clean and clear points for integration and a lot more reporting.

Luke showed a video of a GUI reporting application that aggregates and helps visualize Puppet’s reporting. We had a an email the next morning from a Fortune 500 company about getting into the closed beta. Looks like another large internet property is also going to be in.

This last week, I did a tiny bit of development for the 0.24.5 release and finally restyled reductivelabs.com to be a bit cleaner. I also worked on some slides about Reductive Labs for OATV StartupCamp and threw around some ideas for an Ignite talk for Foo Camp, which are both next week.

I was thinking about ‘What Would Machiavelli Do?’ because Niccolo always gets a bad wrap, but now I’m leaning towards ‘Business Advice From Western Philosophers’. (I’d still throw in some Machiavelli.) I’m going to struggle to boil things down to 20 slides but I think I can make an engaging talk. (I could probably spin at least half a dozen fun talks out with all the different philosophical schools, probably starting with ‘Existentialists‘. And then there is always the Taoists or the Sufis. . .) Feel free to make suggestions in the comments.

StartupCamp and FOOCamp are another great opportunity to get out and spread the Puppet word, but also to learn from a lot of smart people who are doing interesting things. I’m sure my mind will be buzzing this time next week, and some of that is bound to generate some interesting posts. Right?

I also am going to be at Agile 2008, and on that note, some really nice thoughts about software development and infrastructure are starting to crystallize, stay tuned. . . I promise something meaty before StartupCamp starts on the 10th.

Giving the Reductive Labs site a face lift with a new puppet logo and some new artwork inspired by the T-Shirt we’re taking to the conferences: Velocity, Cloud Camp and Structure.

Special thanks to Brett and Reed at Sight Studios for the new logo, Anthony Lagoon Design for all the awesome new hand drawn puppet art and Lamar, the local starving artist for putting it all together on the shirts.

The site won’t be live till early Monday, but that should give you sense of it, and you should expect a recap after the whirlwind conference tour.  Oh yeah, and follow me on twitter, I’ll update during the conf. . .

Once upon a time, I had grand plans to blog regularly. . .

But Puppet has taken over my life. . . in a good way.

We had a great time at the Puppet training

I did have a chance to write something as a guest for GigaOM.

Got a link back in this awesome podcast with Luke and Nigel from google.

And some email conversations generated Puppet in the clouds and this follow up.

We’re getting ready for Velocity, Cloud Camp and Structure in San Fransisco, 23-25 of June.

Come say hi if you are there.

In theory there is no difference between theory and practice. But, in practice, there is.

-Jan L. A. van de Snepscheut

So you decided you want to be Agile, what now? First ask why? Do you really believe in the principles or has something else convinced you to become buzzword compliant? Have you read Fowler, Martin, Cockburn or Beck? Do you know the difference between XP and Scrum? Have you heard of Crystal? Do you know the pitfalls? Have you read the critiques? the rants? Do you know all the personalities involved and their biases?

How could you? Who has time for that? Oh, and the software project that the life your company depends upon can wait for you to sort it all out? Not likely. . .

While we deliberate about beginning it is all ready too late to begin

-Quintilian

A reoccurring theme for teams adopting Agile practices is what may seem like an overwhelming sense of chaos as the old habits and sensibilities collide with the new. The chaos can be real or perceived, but is most prevalent in teams where Agile is being adopted without a lot of experience, often without buy-in or organizational support and on top of a legacy code base. Furthermore, the fledgling Agile champions, while enthusiastic, are trying to balance the notion that the methods only work if they follow all the disciplines and the fact that there is no possible way a team can adopt all the practices instantaneously.

Let me be clear, I personally think Agile, as embodied by the values of the manifesto, is about as good a way to approach the problem as one can get, but transitioning a team can often be a source of pain. In fact, let’s remind ourselves what those values actually are, because in practice, I think people often lose sight of them.

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

No river can return to its source, yet all rivers must have a beginning

-Native American Proverb

How to proceed? Well, that depends, where are you at now? and where do you want to go?

There are plenty of blogs and books and consultants, which offer appropriate advice, and depending on whether you have more time or money these can be great resources.

Here is my free unsolicited 2 cents of self referencing advice for teams who really want to deliver, take it for what it is. . .

• Create trust
• Embrace conflict
• Drive for commitment
• Don’t compromise accountability
• Be honest about the results

Focusing on Trust and Results, a lot of problems will just melt away. Standups, sprints and software will just become byproducts of commitment and accountability. Honest conflict in retrospectives optimize the process.

I’ve got no science for you, just faith. (Unless someone can give me a way to measure ‘trust’)

Like I said, take it for what it is. . .

So where do you start?

Create trust, problem solved.