Author Archives: stochasticresonance

Internet Safari – Definitive Guide

Stochastic Resonance Safari provides services to the luxury internet safari traveller. We strive to make your internet safari the experience of a lifetime.

We contract with the leading blogs and social media sites in South and East internet to bring you a unique safari experience.

To help you, because we care, Stochastic Resonance has compiled this ->FREE<- guide to what you can expect to encounter on your internet journey.

If you demand the best in luxury internet safaris, contact us with your requirements and a qualified representative will get back to you within 24 hours.

Perhaps DevOps Misnamed?

There are only two mistakes one can make along the road to truth; not going all the way, and not starting.


Andi Mann posted ‘Myopic DevOps Misses the Mark‘ earlier today and after reading it, I wanted to put my thoughts out there, particularly since I had hoped some of what I consider his misconceptions would have been cleared up before this post.

To be fair, Andi does ask some good questions and has clearly spent his share of time thinking about ops in general, so hopefully I can make some attempt to address them as well.

To start with, Andi asserts that DevOps is mostly about developers. I’m not entirely certain what makes him think that, but it is patently false and the majority of people involved are heavily from an operations background. That said, I do believe semantics matter, and it might just be the name itself that leads people to that conclusion.

Maybe NeoOps, or KickassOps would have been better… but it is probably too late for that now.

I may be mistaken, but I believe the credit for the term DevOps belongs to Patrick Debois when he organized the first DevOpsDays last year.

Patrick is a bit of a Renaissance man, playing many roles in the process of software delivery along the way. I’m not particularly a fan of labeling people, but Patrick has self identified himself as a sysadmin on more than one occasion. I’m also not particularly a fan of certification, but Patrick’s CV lists certifications like ITIL and SANS, that I’d wager are almost exclusively taken by people in Ops/admin roles. The glaring exception is SCRUM, and I know for a fact Patrick has fought tooth and nail to get the Agile community to recognize the role of systems administrators in the process of delivering value.

Of anyone involved in what has apparently transitioned from ‘a bunch of good ideas’ to ‘a movement’, I probably have the most dev centric background.

  • Patrick Debois – IT Renaissance Man
  • John Allspaw – WebOps Master
  • James Turnbull – Author of Pro Linux System Administration, Hardening Linux, Pulling Strings with Puppet, and he apparently has a day job doing security.
  • Luke Kanies – Recovering sysadmin
  • Adam Jacob – Still calls himself a sysadmin
  • Kris Buytaert – Another Belgian Renaissance Man and a system administrator
  • I’m sure I’m missing lots of people, sorry, maybe we need a poll

Andi keeps saying DevOps is developer centric, and I think the problem (besides maybe the name) is the fact that there is code involved in automation that isn’t a shellscript. Of course, I’m only speculating because he doesn’t actually articulate what makes him think this, but let’s move on to his questions.

Andi makes assertions about lack of control, process, compliance and security. This is ludicrous, bordering on negligent. I’ve seen Puppet deployments on 1000s of machines in what can only be classified as ‘the enterprise’ and I will guarantee those machines are more tightly controlled, compliant and secured than 99% of the machines in most organizations claiming to embrace ITIL. A solid Puppet installation is closer to a functional CMDB than anything I’ve seen in the wild with the advantage that it is both auditing and enforcing the configuration on an ongoing basis. DevOps automation and ITIL are not mutually exclusive and can coexist. (I’m not going to really get into what I think about most of ITIL… but this should help.)

More Specific Questions (most of which are predicated on the misconception that ops somehow goes away, but there are some other bits worth addressing):

Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components?

Ops. Unrestrained sprawl of non-standard systems is a bad assumption. First of all, the slow moving ITIL loving enterprise tends to have as much or more problems with heterogeneous systems as anyone, second of all, when you start to model and automate systems it makes the problem of the heterogeneity both more apparent and more manageable. No one I know advocates anything but pushing towards simple homogeneous systems whenever possible. No one is pretending support and software updates go away.

Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?

Ops of course, but with the added benefit of an automated  infrastructure with semantics relevant to the questions being answered.

Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.

Yep, Ops. VMs are nice because they are typically only an API call away, but there are tools for doing API driven provisioning on bare metal and they will only get better, but… VMs are just the bottom of abstraction mountain. The API driven abstractions of storage and networking fabric are coming. That isn’t the reality today, but it will happen, and relatively soon.

Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?

This is a good one, because frankly I don’t think Ops can do this in isolation anyway. This is a cross cutting concern involving Ops, Dev, Product Management and the other business stakeholders, but change control and rollback are orders of magnitude easier to reason about and accomplish with DevOps approach.

Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?

Similar to the last question, but with the added misconception that DevOps means rolling out random stuff just cause. I know I’ve personally made this point explicitly, but the whole point is to enable a business, and cost containment and asset rationalization are obviously cross cutting concerns of that business.

Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?

Ops doesn’t really do this now. What is the definition of ‘ensure’? Ask nicely? Write up documents? Beg? Get mad? At worst, attempts to do this are often at the root of ‘the wall of confusion’ between Ops and Dev. Again, I’m not sure where Andi got the idea DevOps = ‘cowboys without any concern for anything but deploying stuff as fast as they can’. What are the ‘standard management’ tools? As much as anything, maybe that is what DevOps is replacing, because most of them are embarrassingly poor. The best way to accomplish everything on this list is to expose sensible internal APIs. When we can get to the point that we have reasonable conventions, integration with the next generation of ‘standard management tools’ will be trivial. That might strike you as a dev centric perspective, but really it just means that the present is isn’t evenly distributed.

Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?

DevOps for the win, with the help of tools that can actually model, audit and enforce all those things programmatically.

I’m sure Andi means well, but I’m not clear why he got the impressions he did of what DevOps means or is trying to accomplish. I did the best I could. (Twitter ‘lives in the now’ so that link will probably only be useful for a few days.) I guess if you use the word ‘API’ people won’t process anything further because you are obviously a cowboy developer. C’est la vie…

Finally, Andi finishes with a list of things he would like to see. The irony here is everything on his list is DevOps:

Including ops during the design process, so applications are built to work with standard ops tools.


Taking ops input on deployment, so applications will go in cleanly without disrupting other users


Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows


Implementing ops’ critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant


Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall

Tear down the wall! DevOps!

Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just ‘manipulating APIs

Allow ops to contribute to better application design, deployment, and management, in addition to manipulating APIs! DevOps!

See, there is hope for Andi yet! (I just hope he has a good sense of humor about the title… and would be willing to discuss this over a nice meal if he comes through Salt Lake or we end up in the same city soon.)

DevOps – You’re Doing IT Wrong

There’s something happening here
What it is ain’t exactly clear
There’s a man with a gun over there…
–Buffalo Springfield

Alrighty then, what is this DevOps stuff and what does it mean to me…

First, first off, I came to Ops as a developer (and to be honest I came to be a developer because I didn’t like my prospects or the pay rate to do pure mathematics, but that’s a long story for another day).

If you are going to work with computers at all and have some curiosity and aptitude, chances are you are going to learn a bit about how they work. At the first place I was paid to program, I was a one man wrecking crew in every sense of the word. I was in charge of everything from server configuration to all the programming. I was just out of school with a degree in Mathematics and a minor in computer science. I did everything wrong but I made it all work with what I knew and force of will. I solved problems with books, google and tinkering. There were mailing lists and forums, but they were often insular and reluctant to answer questions or dismissive. Pain is an excellent teacher and that was over a decade ago.

There was a short period where my path could have gone down either road, sysadmin or developer, but as fate would have it my choices and circumstances took me through grad school and from there I became more and more inculturated into the developer tribe.

At some point, working as a developer for a SaaS ecommerce platform startup, through arrangements that I had little control over, I got to experience first hand a dysfunctional relationship with an operations team and essentially found myself taking responsibility for details that would traditionally belong to that side of the ‘wall of confusion’.

In my time working with Luke and people from the Puppet community, I learned a ton. I learned more about the work and culture of system administrators and I also learned a lot about being a developer (in addition to a more lessons than I care to enumerate about business, relationships, marketing, sales, venture capital and spinning plates but I digress).

In my journey, I was also fortunate to make the acquaintance of a number of interesting and talented people at the Salt Lake Agile Roundtable, and this had me in the habit of thinking about technology in terms of people and workflows.

I began thinking about Puppet and the systems tools ecosystem, in the context of the people and the processes. Some of those thoughts were recorded in this blog. I started articulating, sharing and experimenting with those ideas. I found others in the communities of practice that had similar ideas. We all started talking and sharing and building infrastructure and making things happen and now we are here.

To me DevOps is two distinct things that feedback on each other, and then a third that I think is really different.

First, there is the recognition that developers and operations can and should work together. In my opinion, this is being driven by the rise of web delivered business value. When the servers aren’t up, the nifty application doesn’t exist. Too many teams have too much turbulence on both sides of that. This is a serious problem and costs companies millions of dollars every year. I like to think I have made more contributions to solving this problem than I ever made to causing it. This cooperation seems to be the main focus of what I read other DevOps people talking about. Communication, community of interest, manage flow, boundary objects, yada yada… great stuff!

Second, infrastructure and system administration is evolving. The explosion in the open source tool ecosystem is awe inspiring. From provisioning, to virtualization, from configuration, to orchestration, something has undeniably accelerated in the last few years. More and more, from end to end, infrastructure is code. APIs driving and manipulating systems from bare metal to running services. That process looks more and more like software development, split from undifferentiating physical labor in the datacenter. The ‘sysadmin’ no longer has to rack and stack and cable, in addition to being an expert in every OS, application stack, the Voip phones and the printers.

People are arguing that this is not new. That’s somewhat true, and similar positions could be supported for nearly any aspect of computer science, programming or technology. I think that is missing the point a bit. While some of this might not be new in principle or practice, the acceleration is real and those people have to recognize this is not how most people think about and manage their systems. The infrastructure is an application. The sooner more people think like that, the happier they will be. I’m not advocating forget what it means to be a system administrator. Own that domain and know where you come from, but recognize and leverage all the applicable tools and lessons from software development without concerns for notions of tribal identity. In my opinion, there is more to this than ‘just good at their jobs‘, because I still meet system administrators who haven’t heard of Puppet or see why anyone would want or need something like that. The past and present aren’t evenly distributed any more than the proverbial future. We take for granted that things that are obvious to us are obvious to everyone.

Telling someone a truth they aren’t ready to understand is the same as lying to them.

Which finally brings me to the big thing that I think DevOps represents, a community of practice. There might not be anything technically new, what is new is a lot more people talking and sharing. People may have automated system administration tasks forever, but they also hard coded lots of specific details and assumptions about their infrastructure, and they mostly did their work in secret. Lessons were learned and forgotten because the details weren’t transmitted beyond a generation of implementation. There wasn’t (and to some degree isn’t) a common language for patterns of common problems and solutions, but we’re working on it. This community is emerging globally and perhaps appropriately coming together through the very medium which they support, nurture and protect with their hearts and minds. A global community of peers empowering itself to improve the craft through learning and teaching. People with a passion for infrastructure. The difference is not that we can automate systems and work together with other people, the DevOps difference is we want you to be able to do it too.

Open Source, Cloud Computing, Agile, Systems Administration, a perfect storm of ‘nothing new’ with DevOps in the middle of it.

That’s what DevOps means to me…

Now go build something…

Agile Infrastructure: The Movie

In a war between cowboys and ITIL, a small resistance fights for their ideals…

Dedicated to the notion that ‘the way things are done is not the best way to do them’, our heros struggle to liberate their comrades and add to their community of practice. Long live DevOps!

InfoQ just released just released this presentation, which Paul Nasrat and I gave at Agile 2009 in Chicago.

The presentation is quite long, but I hope people find something beneficial and would appreciate any feedback here or on InfoQ.

Puppet, Chef, Dependencies and Worldviews

There was a flurry of Puppet Versus Chef in last week or so. I don’t want to go into sorting all the details at this time, but I hope I add perspective and clarity to one of the subtopics.

People complain that Puppet is non-deterministic.

On a certain level that is like complaining that threads are non-deterministic. That’s the way the model works by design. If there is logic that depends on the order of execution, that code needs mutex/syncronization. Threads create issues, but they also solve some.

Puppet places a high priority on specificity. When order matters, the relationship must be specified explicitly.

Chef manages resources in the order they are specified. (Is that explicit ordering or implicit? I can’t decide.)

Here’s why I think the dividing line is a reflection of worldview…

Puppet aspires to bring a system from whatever state into compliance with the specified state.

Chef puts a higher priority on starting with a newly provisioned image in a known state and bringing up specific services.

Puppet guarantees that order matters on the level that it is specified. Chef guarantees that if you start with the same input, you get the same output.

Different emphasis on preconditions leading to different solutions.

The difference between Puppet and Chef is small compared to the difference between traditional IT management approaches (READ: meatcloud) and either one.

Use your powers for awesome…

Speed Chess

Is typing speed a factor in programmer productivity? Would you improve at chess if you moved the pieces faster?

Jason Gorman, who appears to be a relatively reflective programmer based on his blog and his twitter, tweeted this comment about chess a few days ago.

A bunch of people I follow ReTweeted, and it caught my attention because I used to think this way, both about chess and programming.

Now I don’t claim to be the greatest at either activity, but I’ve put some time and I have enough ability to claim to be above average at both. (Which is to say, I’m aware of my relative mediocrity when compared with real masters)

I’ve played chess off and on for years. I learned to play when I was quite young and I could usually beat other casual players quite easily. After losing to rated tournament players, I spent some time studying the game.

For a long time, I thought the best way to learn was to methodically look for the best move and I thought playing blitz games was somewhat degenerate. Luckily, someone convinced me to start playing blitz games regularly, and that accelerated my understanding of the game considerably.

I still play better when I take the time to be methodical, but that’s not the same thing as learning. I think the blitz games accelerated my learning for two reasons, first, because playing at that speed put that many more games, positions and patterns in front of me and two, because I didn’t attach as much ego to the games I experimented more, which led to that many more positions.

I do think there is a point of diminished returns to just move the pieces faster, and improvement is predicated on some reflection, but I will contend unequivocally that, unless you are a master, you will improve at chess if you spend a considerable portion of your playing time moving the pieces faster.

The same applies to code. You don’t need to type +100 words per minute, but if you can’t touch type at least 40-50 wpm, spend the 20 minutes a day for a month or so until you can. You will never regret it. (And I worked as a programmer for years before I could touch type.)

I would walk you through the arguments, but there is already a classic Yeggethon on the topic, which articulates all the positions I would and then some.

“Lose Your First 100 Games As Quickly As Possible”
–Proverb for Go Beginners


If you can’t be thankful for what you have, be thankful for what you have escaped.

The difference in so many things just comes down to perspective.

Is that glass half empty or is it half air?

I have so much to be thankful for, a beautiful intelligent wife, two amazing little boys, relatively good health and relatively abundant resources.

Yesterday, I decided to run a race and solicit donations for cancer research.

At the time, I just thought I would run because I wanted to and I’d try to get money for a good cause.

You can learn a lot about people when you ask them for money.

I spent much of the last year asking people for money. Buying or giving, people often have unique reasons and circumstances when they hand over their money.

I watched my sister die from cancer. I edited my goal for donations from $200 (the default) to $1000, because $200 just seemed like so little. Today I was pleasantly surprised by the donations I got and who I got them from.

Initially, I had planned to put a couple pictures of my sister up and write a little more about her, but then someone I respect got upset with that plan. In this persons perspective, doing that was disrespectful to my sister’s memory. I don’t share the same perspective, but this person did make a point that stuck. Which was that $1000 is relatively insignificant in the larger scheme of things.

Though choice and circumstance, I could easily donate $1000 and not sacrifice much. If I was willing to sacrifice for a cause, I could put $10,000 and cut back on a few things to make it work. $100,000 is out of the question for my little family, at least in the near future.

At the same time, my wife is a medical resident and I seem to be relatively adept at generating income and business ideas. If we really want to have an impact on this issue, the future is wide open. Stuff that matters… all in good time…

I want to clarify one detail, I watched my sister die from cancer.

She was diagnosed with Rhabdomyosarcoma with a primary tumor in the pericardium, which had already metastasized to her lungs when diagnosed. Through the process, she met several young people who were battling this form of cancer, and one by one, we saw them all wither. None that I know of lived through this.

At some point, my sister was in a coma and intubated. She came out of the coma, but we never got to hear her voice again. She struggled on and the doctors had basically cut her in half trying to remove the tumors. When it became clear that she was not going to make it much longer, she begged to go home.

The last few days, she laid in my mother’s living room with a machine breathing for her. She couldn’t speak, but she could communicate by mouthing words and making gestures. Even though she hadn’t been home for months, and there was no way to get her up the stairs to her room, she could still remember every detail and would make requests to be brought things.

Everything about her was perfect and beautiful except for her lungs which were decimated by tumors and treatments.

She became edematous and faded more and more from this world.

I held her hand when they unplugged the machines.

That was almost 10 year ago.

I am thankful that I had a sister. Her name was Ann Shafer.

One day, I hope to be in a position to honor her name with more than $1000 in donations.

I’m not asking anyone for money, but I do appreciate the support that giving symbolizes.

If you have abundance in your life, do stuff that matters with it.

If you don’t, be thankful for what you have and do what you can to cultivate and be a good steward.

turn the page

Don’t cry because it’s over. Smile because it happened.
–Dr. Seuss

I’m no longer part of Reductive Labs. While this development is not quite heart breaking, I am always sad to leave behind something that I care about, but I also understand and feel strongly that this will be better for everyone.

Since raising venture funding, Reductive Labs has been building an adept team. That team has a critical mass in Portland, OR. There are now eight people there and I am tied to Salt Lake City for the near future, and who knows where for the next few years. Staying aligned with everything that Reductive Labs was doing and needed was becoming more and more of a challenge.

Puppet is a great tool and organizations get a lot of value from it. There are some exciting Puppet developments underway which present unique challenges and opportunities for both the technology and the business. I will now watch them from afar.

My involvement with Reductive Labs has been a period of pronounced growth. I learn many lessons about business and technology, that would have been difficult to learn any other way. I thank Reductive Labs and Luke Kanies for sharing the opportunity.

For the next month or so, I’ll spend more time with my family (when I’m not at the climbing gym and getting massages) and sort through a couple ideas.

Part of me wants to continue pushing the Infrastructure Renaissance forward in some capacity, and part of me wants to build something awesome, and part of me wants to work on stuff that matters.

Maybe there is a good way to combine them all together…

There are a lot of interesting problems in the world… (to solve or cause)

Insurmountable opportunity…

What are you going to do next?

Duct Typing

Today people were tweeting up an old interview with Al3x Payne and some devs from Twitter about using Scala.

Al3x or anyone else has the right to use for whatever programming language they want. No questions asked…

But today I was reflecting on Al3x’s rationalization of the type system motivating a move to Scala. I mean we’ve heard this before and the whole thing got a lot of Ruby panties in a bundle when it first came out.

But checking for nil and kind_of? all over the place is not Ruby’s failing… that’s a big ball of mud in any language and I’ve seen people resort to the (null != foo) && (foo instanceof Bar)) anti-patterns in Java.

Al3x asserts this is an inherent property of large systems in dynamic languages. I’ll assert it is a property of systems that have grown organically to the point where the programmers can’t reason about what is coming or going. I’ll further argue that you get more out of thinking functionally towards solving this issue than you do from stricter typing.

If you find yourself constantly checking for nil/null or really being concerned with types except around the edges of a system, that’s a code smell IMHO.

But maybe another layer of duct tape will fix it…

totally duct'd

totally duct'd

Catching a breath…

I’ve had a lot of things I wanted to get out there, but didn’t have the time to sit down and craft it. The result is a bunch of half formed embryonic posts, which will grow to something hopefully very soon. The past few months have been a whirlwind.

Starting with the investment process and our partnership with True Ventures.

Then there was Agile Roots, which was awesome and I’m not just saying that because I help organize it.

Check out some of the Agile Roots videos. (The sound on some presentations is suboptimal because of issues with the venue’s system, but the content is generally good.)

Then Velocity Conference, face meltingly awesome… the best practioners conference for people who are pushing the boundaries of what is possible. Some of the Velocity videos… my talk didn’t get recorded this time, which I’m sure improved the quality immensely…

Then Structure, similar to Velocity, but for people who wear more buttons and spend less time at the command line.

July was a blurr of phone calls from people wanting to sell Reductive Labs office space and IT management solutions, lunch meetings, podcasts and day trips. Some work got done.

August had more of the same, with more podcasts, Ignite Salt Lake and finally Agile 2009 (where my presentation with Paul Nasrat was recorded by InfoQ)… capped off with a little Reductive Labs love at the Ruby flippin’ Hoedown.

I know I haven’t done any of these events proper justice, but I plan to at least collect some of my thoughts and post my slides, etc. in the next few days.

Did I mention I have two young sons and a wife in medical residency? Don’t try this at home kids…

%d bloggers like this: