Tag Archives: Meatcloud

Perhaps DevOps Misnamed?

There are only two mistakes one can make along the road to truth; not going all the way, and not starting.


Andi Mann posted ‘Myopic DevOps Misses the Mark‘ earlier today and after reading it, I wanted to put my thoughts out there, particularly since I had hoped some of what I consider his misconceptions would have been cleared up before this post.

To be fair, Andi does ask some good questions and has clearly spent his share of time thinking about ops in general, so hopefully I can make some attempt to address them as well.

To start with, Andi asserts that DevOps is mostly about developers. I’m not entirely certain what makes him think that, but it is patently false and the majority of people involved are heavily from an operations background. That said, I do believe semantics matter, and it might just be the name itself that leads people to that conclusion.

Maybe NeoOps, or KickassOps would have been better… but it is probably too late for that now.

I may be mistaken, but I believe the credit for the term DevOps belongs to Patrick Debois when he organized the first DevOpsDays last year.

Patrick is a bit of a Renaissance man, playing many roles in the process of software delivery along the way. I’m not particularly a fan of labeling people, but Patrick has self identified himself as a sysadmin on more than one occasion. I’m also not particularly a fan of certification, but Patrick’s CV lists certifications like ITIL and SANS, that I’d wager are almost exclusively taken by people in Ops/admin roles. The glaring exception is SCRUM, and I know for a fact Patrick has fought tooth and nail to get the Agile community to recognize the role of systems administrators in the process of delivering value.

Of anyone involved in what has apparently transitioned from ‘a bunch of good ideas’ to ‘a movement’, I probably have the most dev centric background.

  • Patrick Debois – IT Renaissance Man
  • John Allspaw – WebOps Master
  • James Turnbull – Author of Pro Linux System Administration, Hardening Linux, Pulling Strings with Puppet, and he apparently has a day job doing security.
  • Luke Kanies – Recovering sysadmin
  • Adam Jacob – Still calls himself a sysadmin
  • Kris Buytaert – Another Belgian Renaissance Man and a system administrator
  • I’m sure I’m missing lots of people, sorry, maybe we need a poll

Andi keeps saying DevOps is developer centric, and I think the problem (besides maybe the name) is the fact that there is code involved in automation that isn’t a shellscript. Of course, I’m only speculating because he doesn’t actually articulate what makes him think this, but let’s move on to his questions.

Andi makes assertions about lack of control, process, compliance and security. This is ludicrous, bordering on negligent. I’ve seen Puppet deployments on 1000s of machines in what can only be classified as ‘the enterprise’ and I will guarantee those machines are more tightly controlled, compliant and secured than 99% of the machines in most organizations claiming to embrace ITIL. A solid Puppet installation is closer to a functional CMDB than anything I’ve seen in the wild with the advantage that it is both auditing and enforcing the configuration on an ongoing basis. DevOps automation and ITIL are not mutually exclusive and can coexist. (I’m not going to really get into what I think about most of ITIL… but this should help.)

More Specific Questions (most of which are predicated on the misconception that ops somehow goes away, but there are some other bits worth addressing):

Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components?

Ops. Unrestrained sprawl of non-standard systems is a bad assumption. First of all, the slow moving ITIL loving enterprise tends to have as much or more problems with heterogeneous systems as anyone, second of all, when you start to model and automate systems it makes the problem of the heterogeneity both more apparent and more manageable. No one I know advocates anything but pushing towards simple homogeneous systems whenever possible. No one is pretending support and software updates go away.

Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?

Ops of course, but with the added benefit of an automated  infrastructure with semantics relevant to the questions being answered.

Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.

Yep, Ops. VMs are nice because they are typically only an API call away, but there are tools for doing API driven provisioning on bare metal and they will only get better, but… VMs are just the bottom of abstraction mountain. The API driven abstractions of storage and networking fabric are coming. That isn’t the reality today, but it will happen, and relatively soon.

Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?

This is a good one, because frankly I don’t think Ops can do this in isolation anyway. This is a cross cutting concern involving Ops, Dev, Product Management and the other business stakeholders, but change control and rollback are orders of magnitude easier to reason about and accomplish with DevOps approach.

Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?

Similar to the last question, but with the added misconception that DevOps means rolling out random stuff just cause. I know I’ve personally made this point explicitly, but the whole point is to enable a business, and cost containment and asset rationalization are obviously cross cutting concerns of that business.

Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?

Ops doesn’t really do this now. What is the definition of ‘ensure’? Ask nicely? Write up documents? Beg? Get mad? At worst, attempts to do this are often at the root of ‘the wall of confusion’ between Ops and Dev. Again, I’m not sure where Andi got the idea DevOps = ‘cowboys without any concern for anything but deploying stuff as fast as they can’. What are the ‘standard management’ tools? As much as anything, maybe that is what DevOps is replacing, because most of them are embarrassingly poor. The best way to accomplish everything on this list is to expose sensible internal APIs. When we can get to the point that we have reasonable conventions, integration with the next generation of ‘standard management tools’ will be trivial. That might strike you as a dev centric perspective, but really it just means that the present is isn’t evenly distributed.

Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?

DevOps for the win, with the help of tools that can actually model, audit and enforce all those things programmatically.

I’m sure Andi means well, but I’m not clear why he got the impressions he did of what DevOps means or is trying to accomplish. I did the best I could. (Twitter ‘lives in the now’ so that link will probably only be useful for a few days.) I guess if you use the word ‘API’ people won’t process anything further because you are obviously a cowboy developer. C’est la vie…

Finally, Andi finishes with a list of things he would like to see. The irony here is everything on his list is DevOps:

Including ops during the design process, so applications are built to work with standard ops tools.


Taking ops input on deployment, so applications will go in cleanly without disrupting other users


Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows


Implementing ops’ critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant


Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall

Tear down the wall! DevOps!

Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just ‘manipulating APIs

Allow ops to contribute to better application design, deployment, and management, in addition to manipulating APIs! DevOps!

See, there is hope for Andi yet! (I just hope he has a good sense of humor about the title… and would be willing to discuss this over a nice meal if he comes through Salt Lake or we end up in the same city soon.)


Meatcloud Manifesto Redux

People rarely succeed unless they have fun in what they are doing.

–Dale Carnegie

When I penned the immortal words ‘Give me an API or give me death…’ and gifted the meatcloud manifesto to the world while admonishing applications that don’t provide APIs, I was perhaps too focused on GUI installers. That was an artificial limitation which I hope to now rectify.

The automation of repetitive tasks presents an insurmountable opportunity.

Freedom from effort in the present merely means that there has been effort stored up in the past.

–Theodore Roosevelt

DRY should be a familiar principle familiar to anyone writing code and not living in a cave — Don’t Repeat Yourself. Unfortunately, sometimes it’s a lot easier to see the replication in files with code than it is in other aspects of our work, but search your shell history and look for the obvious. The real opportunity is to pop up a level and observe the workflows.

How much of what your organization does with emails and phone calls could be accomplished with a chain of automation?

That’s not so say we don’t want humans to make decisions, but do we really need to have someone push that button or fill out that form? As it turns out, machines are really really good at doing exactly the same thing over and over.

How much of what your organization does is replicated across an industry? industries?

Sometimes the benefits are not in what we can add, but what we can take away…

Opportunity is missed by most people because it is dressed in overalls and looks like work.

–Thomas Alva Edison

As I said… insurmountable opportunity…

viva la meatcloud!

meatcloud manifesto

GUI installers are evil…


You might need to have a little wizard to walk through setting stuff up the first time for the uninitiated, but if you think you are doing the poor hack who has to install your stuff on hundreds of machines any favors, you got another thing coming.

If you don’t provide some way to install your software fully unattended, please do not sell it to schools, businesses, non-profits or religious organizations with more than one person.

You might as well be killing kittens.

Actually, that goes for the interface to running applications as well.

I think we need the Open Meatcloud Manifesto… (because the world needs more manifestos and this is the perfect date to start a movement)

Let me see what I can work up…

  • scribble* *scribble* *erase* *scribble* *crumple* *scribble*

I’ll keep it short and simple:

We hold these truths to be self-evident

We hold these truths to be self-evident

Show your support for this movement by making your own copy of the manifesto and posting the pictures.

(Special privileges will be bestowed on those willing to tattoo the manifesto… on their forehead.)

No meatclouds were harmed in the making of this manifesto.

Viva La Revolución

Can you smell what the Puppet is cookin’?

Imitation is the highest form of flattery…

Things are getting really interesting in the configuration management space.  The confluence of clouds, web 2.0, dev-ministration and chocolate sauce.

Chef is upping the ante, one way or another. I’m a little saddened that the majority of the ideas in Chef were discussed in the context of Puppet and implemented by someone who has made a living off of Puppet… but I’m not totally surprised.

I understand why Adam would do this, and on many levels it parallels Luke’s relationship with cfengine.  Adam has probably used Puppet to solve more real problems and build more infrastructure than anyone else, just like Luke had done consulting with cfengine for years before Puppet was born. In a sense, Adam is the embodiment of the future of system administration that Luke had envisioned and hoped to create.

Socrates, Plato, Aristotle… and so it goes.

Adam is a smart guy who thinks clearly about solving problems, had Puppet as an example and data from the front line. He strikes me as a genuine fellow and while we aren’t best friends, I have enjoyed his conversation and insights on more than one occasion.  Chef adds some nice functionality that was obvious and some pieces that are differences in philosophy.  I’ve done my best to absorb both Luke and Adam’s expressed positions, and paradoxically, in my estimation they are both right, in their context. One way or another, there are still a lot of machines out there being managed by meatclouds, so there is plenty of work to do.

Puppet was first released in 2005 and has grown in functionality and adoption since that time. Puppet is revolutionizing system administration, similarly to how Rails revolutionized web development, and Chef can only accelerate that process, by its own merits and by driving innovations in Puppet. At the heart of the story are many questions about progress, open source, community, technology, obligations and the attributions.  The storyline already has mystery, intrigue, tragic heroes, and double agents. There is bound to be some drama, just because there are humans involved, but sometimes nothing motivates like a nice punch in the mouth.

Remember when Nintendo was king of the world then couldn’t sell anything, and now they pwn Playstation and Xbox… Or when apple was awesome and then wasn’t and now we all have iPhones… Innovators innovate, ebb and flow, ebb and flow.

And rails is merb is rails, and you never know what the future holds.  Buckle up… I’m just sayin’…

Progress is impossible without change, and those who cannot change their minds cannot change anything.

–George Bernard Shaw

Shared Metaphor: Gnome Cloud Meat Pastries 2.0

The problem with communication … is the illusion that it has been accomplished.

– George Bernard Shaw

Underpants ????? Profit

One of the topics arising frequently in my conversations is the need for clear vocabulary to describe emerging concepts forced by technical evolution and how those concepts fit into to a personal understanding of the world.

Dialects arise, but until we have shared symbols and metaphors these discussions must always be clumsy. (after we have shared vocabulary, it will still mostly be clumsy)

This is a feeble (and misguided) attempt to share some metaphor.

Gnomes: Some of you might know of venture funded technical start ups that are going to change the world. Some of those start ups might have ridiculous valuations and little or no revenue. Some just have no revenue. . . at least they have a business plan, good luck with that.

Clouds: Everyone knows about ‘the clouds‘ Or maybe they don’t? Or maybe the whole point is that they don’t? Or something? Some anonymous thing over there does something like this other thing I once knew and could touch. . .

Meat: Every organization probably has some hardware and some software, but even in sophisticated technical endeavors there are niggling little tasks that probably should have been automated a long time ago. How are those tasks accomplished? Meatware. . . but of course.

Le Patisserie: This might be less common for some, but I fear those of you in software might have lived it in one form or another. This might also be referred to as ‘dessert tray‘ Agile, or euphemistically as frAgile development. You take one part flailing development organization, two parts buzzword compliance, cut in some shortening or lard, and then bake until golden brown, top with whipped cream and viola! Sprinkle with legacy code if you want a real treat!!

Now comes the fun part. . . mix and match. I haven’t found a permutation that isn’t relevant, informative and amusing. (Ok, so I might be easily amused)

Cloud Gnomes – a virtualization startup

Meat Pastries – developers doing mind numbing repetitive work with Agile razzle dazzle fairy dust.

Meat Clouds – Uhm, like. . . built the pyramids and make your shoes

I leave the rest as an exercise for the reader:

Gnome Meat Pastry Clouds

Cloud Pastry Gnome Meats

Pastry Meat Cloud Gnomes

. . .

Pardon me, I have to get back to playing with puppets . . . and gathering underpants.

%d bloggers like this: