DevOps – You’re Doing IT Wrong

There’s something happening here
What it is ain’t exactly clear
There’s a man with a gun over there…
–Buffalo Springfield

Alrighty then, what is this DevOps stuff and what does it mean to me…

First, first off, I came to Ops as a developer (and to be honest I came to be a developer because I didn’t like my prospects or the pay rate to do pure mathematics, but that’s a long story for another day).

If you are going to work with computers at all and have some curiosity and aptitude, chances are you are going to learn a bit about how they work. At the first place I was paid to program, I was a one man wrecking crew in every sense of the word. I was in charge of everything from server configuration to all the programming. I was just out of school with a degree in Mathematics and a minor in computer science. I did everything wrong but I made it all work with what I knew and force of will. I solved problems with books, google and tinkering. There were mailing lists and forums, but they were often insular and reluctant to answer questions or dismissive. Pain is an excellent teacher and that was over a decade ago.

There was a short period where my path could have gone down either road, sysadmin or developer, but as fate would have it my choices and circumstances took me through grad school and from there I became more and more inculturated into the developer tribe.

At some point, working as a developer for a SaaS ecommerce platform startup, through arrangements that I had little control over, I got to experience first hand a dysfunctional relationship with an operations team and essentially found myself taking responsibility for details that would traditionally belong to that side of the ‘wall of confusion’.

In my time working with Luke and people from the Puppet community, I learned a ton. I learned more about the work and culture of system administrators and I also learned a lot about being a developer (in addition to a more lessons than I care to enumerate about business, relationships, marketing, sales, venture capital and spinning plates but I digress).

In my journey, I was also fortunate to make the acquaintance of a number of interesting and talented people at the Salt Lake Agile Roundtable, and this had me in the habit of thinking about technology in terms of people and workflows.

I began thinking about Puppet and the systems tools ecosystem, in the context of the people and the processes. Some of those thoughts were recorded in this blog. I started articulating, sharing and experimenting with those ideas. I found others in the communities of practice that had similar ideas. We all started talking and sharing and building infrastructure and making things happen and now we are here.

To me DevOps is two distinct things that feedback on each other, and then a third that I think is really different.

First, there is the recognition that developers and operations can and should work together. In my opinion, this is being driven by the rise of web delivered business value. When the servers aren’t up, the nifty application doesn’t exist. Too many teams have too much turbulence on both sides of that. This is a serious problem and costs companies millions of dollars every year. I like to think I have made more contributions to solving this problem than I ever made to causing it. This cooperation seems to be the main focus of what I read other DevOps people talking about. Communication, community of interest, manage flow, boundary objects, yada yada… great stuff!

Second, infrastructure and system administration is evolving. The explosion in the open source tool ecosystem is awe inspiring. From provisioning, to virtualization, from configuration, to orchestration, something has undeniably accelerated in the last few years. More and more, from end to end, infrastructure is code. APIs driving and manipulating systems from bare metal to running services. That process looks more and more like software development, split from undifferentiating physical labor in the datacenter. The ‘sysadmin’ no longer has to rack and stack and cable, in addition to being an expert in every OS, application stack, the Voip phones and the printers.

People are arguing that this is not new. That’s somewhat true, and similar positions could be supported for nearly any aspect of computer science, programming or technology. I think that is missing the point a bit. While some of this might not be new in principle or practice, the acceleration is real and those people have to recognize this is not how most people think about and manage their systems. The infrastructure is an application. The sooner more people think like that, the happier they will be. I’m not advocating forget what it means to be a system administrator. Own that domain and know where you come from, but recognize and leverage all the applicable tools and lessons from software development without concerns for notions of tribal identity. In my opinion, there is more to this than ‘just good at their jobs‘, because I still meet system administrators who haven’t heard of Puppet or see why anyone would want or need something like that. The past and present aren’t evenly distributed any more than the proverbial future. We take for granted that things that are obvious to us are obvious to everyone.

Telling someone a truth they aren’t ready to understand is the same as lying to them.

Which finally brings me to the big thing that I think DevOps represents, a community of practice. There might not be anything technically new, what is new is a lot more people talking and sharing. People may have automated system administration tasks forever, but they also hard coded lots of specific details and assumptions about their infrastructure, and they mostly did their work in secret. Lessons were learned and forgotten because the details weren’t transmitted beyond a generation of implementation. There wasn’t (and to some degree isn’t) a common language for patterns of common problems and solutions, but we’re working on it. This community is emerging globally and perhaps appropriately coming together through the very medium which they support, nurture and protect with their hearts and minds. A global community of peers empowering itself to improve the craft through learning and teaching. People with a passion for infrastructure. The difference is not that we can automate systems and work together with other people, the DevOps difference is we want you to be able to do it too.

Open Source, Cloud Computing, Agile, Systems Administration, a perfect storm of ‘nothing new’ with DevOps in the middle of it.

That’s what DevOps means to me…

Now go build something…


7 responses to “DevOps – You’re Doing IT Wrong

  • Anonymous

    I dont think this is new at all. I think its being more affordable for medium to small shops to do this. The sysadmin at the big ISP’s and network infrastructure houses have never had to actual rack and stack gear and almost always specialized.

    Most of the time there are unions to deal with and electrical codes to be aware of. Now that larger companies like amazon, google, and the other various cloud providers have stepped up to provide a scalable infrastructure (for the most part) the sysadmin/devops teams should be focusing on new an more interesting problems such as better metrics and monitoring platforms. (Screw ganglia and mrtg). I would even go so far as to argue the sysadmin BOFH model is dead. You should and must know how to write (good) code to be a top level “sysadmin” now. Shell scripts are out.

  • stochasticresonance

    @anonymous Some of the worst IT systems are in the biggest shops who are spending insane amounts of money.

    I thought I made it clear that I believe there isn’t anything novel in specifics, but the confluence is new.

    Shell scripts aren’t bad, they just shouldn’t be a lonely hammer.

  • Adam Jacob

    So, to my “we call those people good at their jobs” line. The fact that people haven’t heard about Chef, or Puppet, or Cfengine, doesn’t mean that the people who *have* heard about them are likely better at the game than their peers. My point in making the (perhaps overly cute) punch-line was that what we’re talking about is a movement and a restatement of what it means to be great at our professions (both Systems Administrator and Software Developer.) Your term “community of practice” is exactly what I mean – we’re stating, together, the things that we have learned over the years of being in the trenches, and in the process we are defining what it means to be a professional in our field.

    What we aren’t doing is creating a new, third field – a DevOp, or an OpDev, or a Devministrator, or anything else. We’re simply refining what it means to be a professional Software Developer and Systems Administrator in the context of the new, increasingly web-centric world. That’s an awesome process, and it’s what DevOps really is. Everything else is a distraction.

    • stochasticresonance

      @Adam In theory, I suppose you are correct that it doesn’t mean you are better because you have or haven’t heard of a tool or technology, but in practice, it often does, as the exposure to those ideas is potentially driven by a desire to do things better.

      I’m almost certain there are exceptions that prove the rule.

      I believe we generally agree. DevOps is not necessarily a new role so much as a mindset and value migration between skill sets and technology cycles.

  • Matthias Marschall

    I think that the urgency of working together increases with ideas like continuous deployment. If you only release once a month, it’s not as important to work together and understand each other than if you do daily releases or even more.

  • stochasticresonance

    @Matthias Absolutely, that’s why I said the practices and the communication/cooperation feedback on each other.

    Improving the practice provides more transparency and less surprise, which enables and fosters communication, which should improve the practice.

    A virtuous cycle…

  • How to Initiate a DevOps Project « The Agile Executive

    […] a DevOps project, start by establishing the technical debt of the software to be released to operations. By […]

Leave a comment