Category Archives: Puppet

Puppet, Chef, Dependencies and Worldviews

There was a flurry of Puppet Versus Chef in last week or so. I don’t want to go into sorting all the details at this time, but I hope I add perspective and clarity to one of the subtopics.

People complain that Puppet is non-deterministic.

On a certain level that is like complaining that threads are non-deterministic. That’s the way the model works by design. If there is logic that depends on the order of execution, that code needs mutex/syncronization. Threads create issues, but they also solve some.

Puppet places a high priority on specificity. When order matters, the relationship must be specified explicitly.

Chef manages resources in the order they are specified. (Is that explicit ordering or implicit? I can’t decide.)

Here’s why I think the dividing line is a reflection of worldview…

Puppet aspires to bring a system from whatever state into compliance with the specified state.

Chef puts a higher priority on starting with a newly provisioned image in a known state and bringing up specific services.

Puppet guarantees that order matters on the level that it is specified. Chef guarantees that if you start with the same input, you get the same output.

Different emphasis on preconditions leading to different solutions.

The difference between Puppet and Chef is small compared to the difference between traditional IT management approaches (READ: meatcloud) and either one.

Use your powers for awesome…

turn the page

Don’t cry because it’s over. Smile because it happened.
–Dr. Seuss

I’m no longer part of Reductive Labs. While this development is not quite heart breaking, I am always sad to leave behind something that I care about, but I also understand and feel strongly that this will be better for everyone.

Since raising venture funding, Reductive Labs has been building an adept team. That team has a critical mass in Portland, OR. There are now eight people there and I am tied to Salt Lake City for the near future, and who knows where for the next few years. Staying aligned with everything that Reductive Labs was doing and needed was becoming more and more of a challenge.

Puppet is a great tool and organizations get a lot of value from it. There are some exciting Puppet developments underway which present unique challenges and opportunities for both the technology and the business. I will now watch them from afar.

My involvement with Reductive Labs has been a period of pronounced growth. I learn many lessons about business and technology, that would have been difficult to learn any other way. I thank Reductive Labs and Luke Kanies for sharing the opportunity.

For the next month or so, I’ll spend more time with my family (when I’m not at the climbing gym and getting massages) and sort through a couple ideas.

Part of me wants to continue pushing the Infrastructure Renaissance forward in some capacity, and part of me wants to build something awesome, and part of me wants to work on stuff that matters.

Maybe there is a good way to combine them all together…

There are a lot of interesting problems in the world… (to solve or cause)

Insurmountable opportunity…

What are you going to do next?

Infrastructure Renaissance

Infrastructure is Code! FTW!

Wait… WTF does that even mean?

On the one hand this is a rallying cry for those of us in the configuration automation space, on the other hand it confuses innocent bystanders.

@zhenjl: @botchagalupe but ops ppl have always coded using perl, tcl (if u remembr that :)…now we just have ‘sexier’ langs like ruby and python (original)

Sure, there has always been scripts, which is technically code, but this is missing the point a bit.

For starters, getting ops teams to recognize and steal all applicable processes and tools from developers to produce and version there scripts would be a win, but that’s just the starting line…

The point isn’t just that you can write some scripts to help you do something with your infrastructure then exit.

The ‘Infrastructure’ IS an application, a long running process with inputs, outputs and state. Or at least infrastructure is very close to becoming one…

Everyone paying attention knows something is happening… what is it?  Hazy, foggy, smoky, cloudy, whatever…

EC2 is the herald of an infrastructure enlightenment, leading us out of the dark ages.

(We’ll ignore the history of virtualization technology, and ESX Server for now, but I’m sure someone has an opinion on all this.)

The bottom line is there are resources, and I can manipulate them through an API. With EC2, these are compute resources of various shapes and sizes, with some storage resources.

Virtualized networking is not exposed through a public API yet, but we’re still climbing abstraction mountain together and we all know it is just a little higher in this cloud. (What’s up Cisco, where you at? Oh, there you are… and these guys too… thanks Ben )

At some point in the not too distant future, web service calls and a credit card will be enough to create and manipulate arbitrary compute, storage and network resources.

So now what?

Until someone provides Sysadmin as a Service, there is still quite a bit of work left to be done. These resources need to be configured to provide services.

When adding capacity was a purchase order, then wait for weeks to rack and cable, taking a day or so to configure the machines was not rate limiting, and lost in the noise. When you can bring up new server instances in minutes, that same configuration step is conspicuous waste.

You might be thinking you’ll make images for each of your services and manage those. This approach to management exchanges the configuration drift of tradition IT management for image sprawl. Do those notes that got scribbled six months ago when the web server image was created have accurate details about how that machine is configured? Are you sure? The only way to know is start the VM and inspect it. Frankly, the same thing applies to the running instances started yesterday.

That’s where tools like Puppet comes in. The benefit is not just that a machine gets configured by running a script. Puppet is an application that audits your system configuration, and changes the details out of sync with policy. That policy is code with semantics that can be developed like software.

That’s the shift. Instead of thinking about managing server counts, we can reason about abstractions of service configurations. The pieces exposed by an API are not just instances of server and storage resources, but abstractions of services. The configuration of proxies, monitoring, middleware, web servers and databases are components to be composed in the dynamic ‘infrastructure’ application.

This is just the beginning, but I don’t want to spoil the rest of the story… at least not yet.

It is not necessary to change. Survival is not mandatory.

W. Edwards Deming

meatcloud manifesto

GUI installers are evil…


You might need to have a little wizard to walk through setting stuff up the first time for the uninitiated, but if you think you are doing the poor hack who has to install your stuff on hundreds of machines any favors, you got another thing coming.

If you don’t provide some way to install your software fully unattended, please do not sell it to schools, businesses, non-profits or religious organizations with more than one person.

You might as well be killing kittens.

Actually, that goes for the interface to running applications as well.

I think we need the Open Meatcloud Manifesto… (because the world needs more manifestos and this is the perfect date to start a movement)

Let me see what I can work up…

  • scribble* *scribble* *erase* *scribble* *crumple* *scribble*

I’ll keep it short and simple:

We hold these truths to be self-evident

We hold these truths to be self-evident

Show your support for this movement by making your own copy of the manifesto and posting the pictures.

(Special privileges will be bestowed on those willing to tattoo the manifesto… on their forehead.)

No meatclouds were harmed in the making of this manifesto.

Viva La Revolución

Open For Business Follow Up

After my ‘Open For Business‘ post, I had several conversations out of band.

A few people responded with ‘we feel your pain’.

That’s not really the point I was trying to make. This quarter is the best quarter Reductive Labs has ever had for revenue, and next quarter looks like it will be a even better as people both realize the value of Puppet automation and are seeking better ROI from their budgets.

Any new skill is going to come with some clumsy missteps as you are learning, and sales is a skill. We are technologists learning to run a business, and sales is part of it. Frankly, we offer a great value for where we are in the market and I don’t know who I feel worse for, the enterprise sales guys who are offering to buy each others licenses, the IT managers who are buying stuff their admins won’t be able to use or those admins. So while it might be a little painful in the ‘here we go again’ sense, that is really just part of the sales process, but not anywhere near as painful as this. (sorry, that was probably indulgent)

Another comment is that Puppet might be perceived as not being ‘Enterprise Ready’. Perception is reality, and some people still argue Ruby isn’t ready. Rails, not ready… How about Windows? Anyone ever read the bugs open against Oracle 10g? The fact is Puppet has several very successful implementations in Fortune 100 companies.  Whatcha gonna do?

With big companies, the discussion sometimes breakdown when we get to culture and process. The problem isn’t the technology per se, it’s the fact that there are vertically and horizontally splintered fiefdoms running around vying for power and pride. Every group wants to do things their way. When the X application group has segregated systems from the Y application group isn’t so bad, but if the X group has to submit tickets to the monitoring group to find out what is happening on their own machines, implementing a centralized management solution becomes really difficult to navigate from the bottom up.

Maybe you should stop when it hurts?

Maybe you should stop when it hurts?

Sometimes, I just want to tap out to the stupid.

(That was probably indulgent too…)


That does look painful…

Go ‘Enterprise’ go…

Open For Business

What has never been doubted, has never been proven.

–Denis Diderot

Let’s be honest, no one ‘really’ knows exactly how ‘Open Source’ and ‘Business Model’ should fit together. There is a broad spectrum of ‘Open Source’ approaches, from the ‘Purists‘ to the ‘Open Core‘ to the pragmatic ‘Flip Floppers‘. The differences are subtle nuances to anyone not steeped in the culture and trying to explain them will just confuse most people not familiar with the lore. There is my partner Luke, who has as many questions as answers. Then there a people with a lot of answers, but of course, they are all different.

Analysts are talking about the ‘billion dollar opportunity‘, but the brightest example of Open Source business success, that everyone references, garners harsh comments like this one. Does there have to be a clash along this boundary? Sure, Redhat has essentially adopted a traditional licensing model, but if you don’t understand the benefit of the role they play in the larger OSS ecosystem then you aren’t really paying attention. Having strong ideals is one thing, paying for groceries, mortgages, and braces is another.

One of the most influential programmers of a generation is trying to figure out how to make a living. You think capistrano would be abandoned if Jamis got a nickel everytime someone ran cap deploy? These were innovative projects that added a ton of value to an ecosystem and captured little, if any. There are plenty of millionaires who sold total crap and cashed out.

Nowadays people know the price of everything and the value of nothing.

–Oscar Wilde

Is there some way to balance Open Source ideals with opportunity? I sure hope so, but I’m not entirely sure how the rest of the story goes. On more than one occasion, I’ve been on a call with someone from an ‘enterprise’ talking about how they are replacing Opsware or Bladelogic installations that were minimum high 6 figure checks to implement, wanting Reductive Labs services, and balking at spending ~$20K with us as too expensive. Someone convinced somebody that the promise of automation was worth X dollars, possibly didn’t deliver it, and now a solution is not worth X/50? What gives?

The problem in most cases is the technology has been adopted at the front line by admins who want to get more done, know there is a better way, but don’t necessarily have budget to spend. Their managers don’t know much about what they do, view all of IT as a cost center anyway and now they want more money… for what? Reductive Labs best customers are organization with a semi-technical manager, who trusts her people, wants them to succeed and believes IT should be a differentiator. Organizations like this have funded Puppet’s development.

The push back is often from companies with 10K servers and a 500 headcount meatcloud, all with root on every box. Some progressive admin has got Puppet up and solving small problems, but knows releasing it into the organization at large without some training and process improvements is like distributing chainsaws to every kindergartener on the playground at recess. No child left behind… So instead they fumble through implementing a solution, don’t really change the process, and then call us when Puppet brings down a service by doing exactly what their code told Puppet to do. (Ok, so I’ll admit, this doesn’t happen every time, but it has happened.)

The greatest obstacle to discovery is not ignorance — it is the illusion of knowledge.

–Daniel J Boorstin

John M Willis says he talks to the ‘enterprise’ all the time, and most people haven’t even heard of Puppet yet. I’m sorry, try to catch the next clue train. Managing 10K machines with a 500 headed meathydra rooting boxes is costing a lot more money than it needs to.  We would love to solve that problem for you. If you aren’t looking for a solution to this, you probably aren’t solving other problems either. I understand that keeping up with every little technology is hard, but at some point you just have to concede that you aren’t in the race anymore. Hopefully, you have at least heard of Linux…

The problem is most the time the conversation stops at the admin technical level. We don’t have a small army in ties who can take executives for metaphorical (or not…) martinis at strip clubs. Or a marketing budget to take out full page ads in CIO magazine. In my world view that should be a good thing. We are technologist at heart, who want to build solid solutions and to deliver value at the point of attack. Unfortunately, we don’t have the sales process to exploit inefficiencies in the marketplace with asymmetric understanding and access to information (or an economic system that rewards value necessarily).

Somewhere between the ‘enterprise’ who doesn’t know what to make of open source software and the people who love open source like sharks like seals, is there an opportunity to capture value? Scylla and Charibdis?

I like to think that more and more people will realize that the price of Open Source is expertise, that they have to pay their team for months of learning we could have provided in a week, that a ‘support’ contract is a reciprocal agreement, and the alternatives are much more expensive when comparing the total cost of ownership. If you pay us half of what a commercial solution would cost, we can automate your infrastructure, train your team and support the solution. In fact, we’ll make sure Puppet spreads jam on your toast in the morning. (If you pay us anywhere near the same amount, we’ll automate baking the bread too.)

Markets are constantly in a state of uncertainty and flux, and money is made by discounting the obvious and betting on the unexpected.

–George Soros

Practical Puppet at MWRC

**update** The underlying repos changed so the demo won’t work from the user data. It will still work after running ‘apt-get update’ on the box. If I get bored I might repackaged it.

*update*  Confreaks has my MWRC video up. The notes below about launching on EC2 are probably still useful for context.

These are my slides from Mountain West Ruby Conf.

Starts with a little intro about the beginning of computing, before ‘computer science’, driven by desire for computation in math and physics, when there wasn’t a division between the people that program and people that understand how the systems work. (this period probably lasted for about 10 minutes.)

Then some Puppet code…

These modules will build passenger and install rails.

Finishes with a short discussion on the tribes, trading ideas, evolutions, the opportunity of clouds and encouraging people to do something awesome.

The talk was given with a live demo of this code building rails in EC2.

You can do it too if you have an EC2 account.

Start ami-2b10f742 littleidea/mwrcpuppetrailsdemo.manifest.xml sending this as the user data:

#!/usr/bin/env puppet

rails::site{ the_next_big_thing:
servername => "",
rails_version => "2.2.2"

Set the server name and the rails version to whatever you want.

If you need more gems or packages you can add:

#for gems
package { gemname:
    ensure => installed,
    provider => gem

#for native packages
package { pgkname:
    ensure => installed,

If you need a certain version of a gem or package, change ‘installed’ to the version you want (the default will install the latest available.)

The image is based on ami-7cfd1a15 Ubuntu Intrepid packaged by Eric Hammond with puppet and facter installed. The puppet manifests from the slides are modules found in /usr/share/puppet/modules.

When the ami starts up it will grab the user data and run it.

You can also ssh to your instance and just put the puppet code in a file (strip the #! /usr/bin/puppet) and run puppet –debug to watch it build.

puppet --debug mysite.pp

Mountain West Ruby Conference

Rocky Mountain High

Rocky Mountain High

I’m definitely looking forward to Mountain West Ruby Conference. Last year was most excellent, and this year I’m presenting. Puppet, clouds, live demo, no net, should be a good time…

MWRC is a single track regional conference, but it brings in some of the biggest names in the Ruby world. (maybe we can get Matz to come next year) Sharing the stage with some of these guys is an honor and a privilege, if not a little intimidating. And if last year was any indication, even the guys you haven’t heard of will be awesome.  I can’t recommend this conference highly enough.

Pat Eyler of On Ruby is one of the organizers and just posted an interview with me. He is interviewing other speakers as a lead up to the conference, so keep an eye out.

Use your powers for Awesome!!!

Use your powers for Awesome!!!

Puppet and Capistrano

I haven’t just been slacking off.

Can you smell what the Puppet is cookin’?

Imitation is the highest form of flattery…

Things are getting really interesting in the configuration management space.  The confluence of clouds, web 2.0, dev-ministration and chocolate sauce.

Chef is upping the ante, one way or another. I’m a little saddened that the majority of the ideas in Chef were discussed in the context of Puppet and implemented by someone who has made a living off of Puppet… but I’m not totally surprised.

I understand why Adam would do this, and on many levels it parallels Luke’s relationship with cfengine.  Adam has probably used Puppet to solve more real problems and build more infrastructure than anyone else, just like Luke had done consulting with cfengine for years before Puppet was born. In a sense, Adam is the embodiment of the future of system administration that Luke had envisioned and hoped to create.

Socrates, Plato, Aristotle… and so it goes.

Adam is a smart guy who thinks clearly about solving problems, had Puppet as an example and data from the front line. He strikes me as a genuine fellow and while we aren’t best friends, I have enjoyed his conversation and insights on more than one occasion.  Chef adds some nice functionality that was obvious and some pieces that are differences in philosophy.  I’ve done my best to absorb both Luke and Adam’s expressed positions, and paradoxically, in my estimation they are both right, in their context. One way or another, there are still a lot of machines out there being managed by meatclouds, so there is plenty of work to do.

Puppet was first released in 2005 and has grown in functionality and adoption since that time. Puppet is revolutionizing system administration, similarly to how Rails revolutionized web development, and Chef can only accelerate that process, by its own merits and by driving innovations in Puppet. At the heart of the story are many questions about progress, open source, community, technology, obligations and the attributions.  The storyline already has mystery, intrigue, tragic heroes, and double agents. There is bound to be some drama, just because there are humans involved, but sometimes nothing motivates like a nice punch in the mouth.

Remember when Nintendo was king of the world then couldn’t sell anything, and now they pwn Playstation and Xbox… Or when apple was awesome and then wasn’t and now we all have iPhones… Innovators innovate, ebb and flow, ebb and flow.

And rails is merb is rails, and you never know what the future holds.  Buckle up… I’m just sayin’…

Progress is impossible without change, and those who cannot change their minds cannot change anything.

–George Bernard Shaw

%d bloggers like this: