There are only two mistakes one can make along the road to truth; not going all the way, and not starting.
–Buddha
Andi Mann posted ‘Myopic DevOps Misses the Mark‘ earlier today and after reading it, I wanted to put my thoughts out there, particularly since I had hoped some of what I consider his misconceptions would have been cleared up before this post.
To be fair, Andi does ask some good questions and has clearly spent his share of time thinking about ops in general, so hopefully I can make some attempt to address them as well.
To start with, Andi asserts that DevOps is mostly about developers. I’m not entirely certain what makes him think that, but it is patently false and the majority of people involved are heavily from an operations background. That said, I do believe semantics matter, and it might just be the name itself that leads people to that conclusion.
Maybe NeoOps, or KickassOps would have been better… but it is probably too late for that now.
I may be mistaken, but I believe the credit for the term DevOps belongs to Patrick Debois when he organized the first DevOpsDays last year.
Patrick is a bit of a Renaissance man, playing many roles in the process of software delivery along the way. I’m not particularly a fan of labeling people, but Patrick has self identified himself as a sysadmin on more than one occasion. I’m also not particularly a fan of certification, but Patrick’s CV lists certifications like ITIL and SANS, that I’d wager are almost exclusively taken by people in Ops/admin roles. The glaring exception is SCRUM, and I know for a fact Patrick has fought tooth and nail to get the Agile community to recognize the role of systems administrators in the process of delivering value.
Of anyone involved in what has apparently transitioned from ‘a bunch of good ideas’ to ‘a movement’, I probably have the most dev centric background.
- Patrick Debois – IT Renaissance Man
- John Allspaw – WebOps Master
- James Turnbull – Author of Pro Linux System Administration, Hardening Linux, Pulling Strings with Puppet, and he apparently has a day job doing security.
- Luke Kanies – Recovering sysadmin
- Adam Jacob – Still calls himself a sysadmin
- Kris Buytaert – Another Belgian Renaissance Man and a system administrator
- I’m sure I’m missing lots of people, sorry, maybe we need a poll
Andi keeps saying DevOps is developer centric, and I think the problem (besides maybe the name) is the fact that there is code involved in automation that isn’t a shellscript. Of course, I’m only speculating because he doesn’t actually articulate what makes him think this, but let’s move on to his questions.
Andi makes assertions about lack of control, process, compliance and security. This is ludicrous, bordering on negligent. I’ve seen Puppet deployments on 1000s of machines in what can only be classified as ‘the enterprise’ and I will guarantee those machines are more tightly controlled, compliant and secured than 99% of the machines in most organizations claiming to embrace ITIL. A solid Puppet installation is closer to a functional CMDB than anything I’ve seen in the wild with the advantage that it is both auditing and enforcing the configuration on an ongoing basis. DevOps automation and ITIL are not mutually exclusive and can coexist. (I’m not going to really get into what I think about most of ITIL… but this should help.)
More Specific Questions (most of which are predicated on the misconception that ops somehow goes away, but there are some other bits worth addressing):
Who handles ongoing support, especially software update for the unrestrained sprawl of non-standard systems and components?
Ops. Unrestrained sprawl of non-standard systems is a bad assumption. First of all, the slow moving ITIL loving enterprise tends to have as much or more problems with heterogeneous systems as anyone, second of all, when you start to model and automate systems it makes the problem of the heterogeneity both more apparent and more manageable. No one I know advocates anything but pushing towards simple homogeneous systems whenever possible. No one is pretending support and software updates go away.
Who ensures each new application doesn’t interfere with existing and especially legacy systems (and networks, storage, etc.)?
Ops of course, but with the added benefit of an automated infrastructure with semantics relevant to the questions being answered.
Who handles integration with common production systems that cannot be encapsulated in a VM, like storage arrays (NAS, SAN), networking fabrics, facilities, etc.
Yep, Ops. VMs are nice because they are typically only an API call away, but there are tools for doing API driven provisioning on bare metal and they will only get better, but… VMs are just the bottom of abstraction mountain. The API driven abstractions of storage and networking fabric are coming. That isn’t the reality today, but it will happen, and relatively soon.
Who handles impact analysis, change control and rollback planning to ensure deployment risk is understood and mitigated?
This is a good one, because frankly I don’t think Ops can do this in isolation anyway. This is a cross cutting concern involving Ops, Dev, Product Management and the other business stakeholders, but change control and rollback are orders of magnitude easier to reason about and accomplish with DevOps approach.
Who is responsible for cost containment and asset rationalization, when devops keeps rolling out new systems and applications?
Similar to the last question, but with the added misconception that DevOps means rolling out random stuff just cause. I know I’ve personally made this point explicitly, but the whole point is to enable a business, and cost containment and asset rationalization are obviously cross cutting concerns of that business.
Who ensures reporting, compliance, data updates, log maintenance, Db administration, etc. are built into the applications, and integrated with standard management tools?
Ops doesn’t really do this now. What is the definition of ‘ensure’? Ask nicely? Write up documents? Beg? Get mad? At worst, attempts to do this are often at the root of ‘the wall of confusion’ between Ops and Dev. Again, I’m not sure where Andi got the idea DevOps = ‘cowboys without any concern for anything but deploying stuff as fast as they can’. What are the ‘standard management’ tools? As much as anything, maybe that is what DevOps is replacing, because most of them are embarrassingly poor. The best way to accomplish everything on this list is to expose sensible internal APIs. When we can get to the point that we have reasonable conventions, integration with the next generation of ‘standard management tools’ will be trivial. That might strike you as a dev centric perspective, but really it just means that the present is isn’t evenly distributed.
Who will assure functional isolation, role-based access controls, change auditing, event management, and configuration control to secure applications, data, and compliance?
DevOps for the win, with the help of tools that can actually model, audit and enforce all those things programmatically.
I’m sure Andi means well, but I’m not clear why he got the impressions he did of what DevOps means or is trying to accomplish. I did the best I could. (Twitter ‘lives in the now’ so that link will probably only be useful for a few days.) I guess if you use the word ‘API’ people won’t process anything further because you are obviously a cowboy developer. C’est la vie…
Finally, Andi finishes with a list of things he would like to see. The irony here is everything on his list is DevOps:
Including ops during the design process, so applications are built to work with standard ops tools.
Devops!
Taking ops input on deployment, so applications will go in cleanly without disrupting other users
Devops!
Working with ops on capacity and scalability requirements, so they can keep supporting it when it grows
Devops!
Implementing ops’ critical needs for logging, isolation, identity management, configuration needs, and secure interfaces so the app can be secure and compliant
Devops!
Giving ops some advance insight into applications, especially during test and QA, so they can start to prepare for them before they come over the wall
Tear down the wall! DevOps!
Allowing ops to contribute to better application design, deployment, and management; that ops can do more for the release cycle and ongoing management than just ‘manipulating APIs’
Allow ops to contribute to better application design, deployment, and management, in addition to manipulating APIs! DevOps!
See, there is hope for Andi yet! (I just hope he has a good sense of humor about the title… and would be willing to discuss this over a nice meal if he comes through Salt Lake or we end up in the same city soon.)
fate = will && choice || circumstance 
March 26th, 2010 at 9:24 pm
For me, it’s all about who gets “the phone call” when things go Tango Uniform.
I’ve long been a proponent of having a systems person integrated with development from start to finish. I’ve been on both sides of the equation – an environment where code was just handed off for deploy and one where a systems person was integrated into the development team.
Invariably, the environment where there was a “devops” role was the more stable of the two.
I partially blame 37signals for this notion that you don’t need a DBA or a SysOp but I can understand why people hate the people in those roles so much. Disjointed communication, role-specific jargon that doesn’t translate between groups and a general arrogance on all sides can lead to that.
However, no code is infallible and no amount of TDD can ensure that something isn’t going to break in production and the developer isn’t going to be the one called on the carpet for it at 2AM when revenue is lost.
How many times have we seen performance tank because of data discrepancies between the dev, qa and production environments?
I agree! Tear down the wall. Accept input from others who might ask a question that you never though to ask because you didn’t have the bigger picture.
March 26th, 2010 at 10:38 pm
@Luis
Thanks for the comments.
One of the points we are hoping to make is not that there is a ‘devops’ role, so much as understanding the applications, the business and the perspectives of the personas involved is the best way to do ops. Period.
Actually, likely the best way to do anything…
@lisacrispin was visiting the Salt Lake Agile Round Table a while back and I had a major ‘ah hah’ when she described how everyone understanding the business and the roles in it transformed an organization. In her role as a tester, she has certainly seen her share of ‘walls of confusion’ and ‘fingerpointiness’ from another direction. She even wrote a book about it.
Blaming 37Signals for things is one of my hobbies, but that’s a topic for another day.
TDD optimizes for something else, and is really only enforcing assumptions about how things ‘should’ work. Code can pass a test suite and still bring production to its knees.
I have felt the pain of deploying to production when it isn’t configured like the test environment. Definitely… have the scars to prove it…
Tear down the walls, get people to talk, no amount of checklists and signatures is a substitute.
March 27th, 2010 at 1:20 pm
Why would you blame 37signals? We have three system administrators. What we have said, though, is that you don’t need one at the start, which is absolutely true. I joined the company when the operational needs became more than the developers were willing to manage.
Our developers and operations people get along extremely well. I think a major reason for that is that all of our ops people are also competent programmers who have shipped real applications, so we can very easily relate to the needs of our developers. From the development side, our developers are smart enough to understand any system related issues that we present them with, and trust us enough to believe what we tell them. We have a great relationship.
As an aside, for what it’s worth, we were a very early Chef adopter and have contributed back both patches to Chef and a number of the cookbooks we use ourselves.
March 27th, 2010 at 1:23 pm
@mark
My 37Signals response was an attempt at humor…
(at which I’m batting 1000 today)
I’m definitely blaming you for the whole Ruby on Rails thing.
(and by ‘You’, I mean you, Mark Imbriaco, it’s all your fault.)
March 27th, 2010 at 1:38 pm
@stochasticresonance I should have been more clear, I was replying to John’s initial comment. Just wanted to set the record straight since there seemed to be some misconceptions.
I wish I could take the blame for the whole Ruby on Rails thing, though. Would you believe me if I said it was my idea and David stole it?
March 27th, 2010 at 5:08 am
Amen, Brother Andrew.
I’m not sure what it is with people who see something new brewing and search for every contrarian angle they can find… even if it means claiming to cite (with no evidence, of course) people who are either deeply mistaken or at the odd fringe of that something new.
You broke down Andi’s faulty argument with skill and ease.
Oddly, many of the individual points/criteria/reasons for DevOps that Andi brought up were valid (as you pointed out), but how he got to the fact that DevOps means “Dev replacing Ops” rather than “Dev aligned with Ops” is a big mystery that he has yet to explain.
Anyway, this is a post that people will be pointing to for a while. Thanks for taking the time to write it.
-Damon
March 27th, 2010 at 1:48 pm
I think the “what’s devops” discussion is different for each environment, group of people and local culture. Many peoples opinions are valid, just not in my situation and that’s where some of the friction is coming from. Devops is culture not technology.
Firstly it’s about getting to know the involved parties, I have lunch with the devs regularly, we know each others abilities, strengths and personalities. It’s a simple thing but it removes a lot of the finger pointing and misunderstanding.
Secondly it’s about business buy in, we’re changing the traditional approaches, it probably means different people will be responsible for operational availability and stability and the business people will be used to the older ways. You need to address this first and make sure you will get the needed buy in. There’s no point in giving devs new responsibilities when business people do not agree they are actually responsible.
It means enabling the devs, I make the networks, operating systems, architecture easy to understand – so it’s not a burden for the devs to keep in mind how it works while designing applications. So it’s easy for them to share the knowledge and ultimately so it’s easier for everyone to debug the interactions when there are problems.
Once I have a accessible network, I make sure I am available, knowledgeable about developers technologies and how things work. This is where systems guys are devops, you need to be able to talk about the merits of development technologies and you need to know how to choose technologies and how they will interact with your environment. This knowledge help you have a mutually inclusive conversation about code architecture. It also helps you to adapt the network to the needs of the code. The code makes the money, we’re there to serve the code ultimately.
Once the code is out there, especially in startups, web shops etc, you’ll see daily/weekly releases of code. It is impossible for systems administrators to effectively support an app thats for ever changing, the logs keep changing the behaviour, code flow, visitor paths and everything else keeps changing. This is where enabling Devs to do deploys and application support come in. They should be able to deploy 100 times a day if that’s their thing, they should be able to see the logs, monitoring, graphs and stats they need to them debug issues without going through traditional gates like tickets to sysadmins.
The previous paragraph is where the systems administrators usually throw up hands and say its impossible. This is because sysads are responsible for app performance, stability and so forth. This the problem that needs solving, the devs need to be the ones supporting the application, they get the pages, they write the sitrep and they report to management about what happened. Systems administrators remain responsible for systems – networks, firewalls, sans, operating systems, cf management etc. all that they don’t do is manage the code and take responsibility for something that they often had no hand in designing.
With this shift in application responsibility in place and with systems administrators having enabled the devs to manage the app – in a safe, secure, audited and logged manner – a lot of pieces fall into place. Sysadmins are not bogged down with the high interrupt driven app support task and developers suddenly gain hands on experience in that part of the application life cycle. With this comes quick feedback loop, quick fault resolution and responsibility with the people best suited to supporting the application.
Systems Administrators can now take the time they had freed up to do monitoring, backups, security, etc RIGHT without the constant interruption.
In my experience this symbiotic cooperation leads to better architectures, networks, monitoring, etc. Developers share in the pains, they know how the app behaves and ultimately the quality of code built in this scenario is much better than the old ways.
This does not mean every systems admin needs to be a ‘devop’ and doesn’t mean every developer needs to be a CLI god, you can nominate parts of your team to do those roles while the rest of the teams just knock out code or support the infrastructure like dns, racking kit etc. But there’s a breed of middleman that can stand on both sides of the fence acting like a bridge. Not everyone will be this person, not everyone needs to be this person but they are essential.
March 28th, 2010 at 6:17 am
That is one of the best commentaries on #devops I have read. My biggest complaint is not about the ideas swirling around #devops and automated infrastructure but about the name. IMHO it doesn’t even need a new name. These are all old ideals and ideas and some are coming to fruition. In general it’s not very often when people respect each other, view things from the others perspective, and cooperate, that you end up worse for wear. So the more aligned view of developers and operations makes sense but I don’t think it requires a new name.
I count myself extremely lucky to have worked very close with developers over the years. It has exposed me to great technologies and ideas, and to some that I totally disagree with but for sure it decreases the amount of stuff I don’t know that I don’t know about. For many of the discussions I have had with developers I have changed the way I do and view things. I hope the developers I have worked with have gained a similar value in the other direction from working and talking with me.
March 28th, 2010 at 10:46 am
I don’t think DevOps is mis-named, but I wouldn’t blame Andi for being confused about it, either… the term is getting tossed around and used in a lot of places where it probably doesn’t really apply. I think the real mistake Andi made was trying to take it out of the relatively narrow range where it’s useful and applicable.
To be fair, Andi never “…asserts that DevOps is mostly about developers.” What he actual wrote was, “However, most of the writings I see about devops are really about dev, not ops.” That’s absolutely what I have found as well; whoever is doing the writing, ops or devs, they have mostly focused on development concerns with operations, and leaves out vast swaths of operations concerns that (appropriately) don’t have anything to do with development. That is in no way a flaw with DevOps; but if you’re not coming at operations from the perspective of development or interaction with developers, there’s not much in it for you right now. Those questions of Andi’s you have answered with “Ops!” are a BIG part of Ops… but there’s very little being written about how to handle those things in an Agile manner, because DevOps (again, quite appropriately) doesn’t concern itself with them.
This isn’t to say that the same agile philosophies that are fueling DevOps can’t apply elsewhere in Operations, but I do think a different term probably needs to be used when talking about those things, if only to avoid confusion over what DevOps is and where it can help.
Ernest Mueller has an excellent blog post breaking this down much better than I can:
http://www.webadminblog.com/index.php/2010/03/11/defining-agile-operations-and-devops/
March 28th, 2010 at 12:07 pm
@Scott
Whether Devops is misnamed is obviously debatable. I don’t blame anyone for being confused. What I have a hard time with is what from my perspective appeared to be a disregard at attempts to explain his misconceptions before he wrote his post.
My initial response to what he was saying on twitter:
@AndiMann devops != dev as ops, the words don’t mean what you think they mean. http://bit.ly/bokKkW http://bit.ly/bW6knp
I asked him for Ops clarification:
@AndiMann Just so we’re all using the same semantics, perhaps you can offer a definition of what ops actually does and we’ll go from there.
He declined to define ops, so I offered my twitter constrained version:
@AndiMann Ops job is to enable business goals by providing, planning and advising for use of compute, storage and network resources.
He responded with:
@littleidea Not bad. Pretty generic. Misses all ops do for security, continuity, compliance, cost containment, driving business, and more.
To which I responded with one of the links from the very first thing I sent him noting the constraints of twitter. That link went to this:
@wattersjames @Beaker If DevOps means no sysadmins, security, audit, compliance or change control to anyone, that is an epic fail.
He then proceeded to write an article claiming devops means no sysadmins, security, auditing, compliance or change control and listed me by name as someone who had explained things to him (which he has since edited out of the original article).
I also reject the idea that anything was taken out of a relatively narrow range. First, I don’t believe the range to be narrow, and second, I believe I did a serviceable job of refuting that analysis in the my post.
To be fair, Andi never “…asserts that DevOps is mostly about developers.” What he actual wrote was, “However, most of the writings I see about devops are really about dev, not ops.”
There are many more sentences in the original post that assert or imply a dev centric perspective.
That’s absolutely what I have found as well; whoever is doing the writing, ops or devs, they have mostly focused on development concerns with operations, and leaves out vast swaths of operations concerns that (appropriately) don’t have anything to do with development. That is in no way a flaw with DevOps; but if you’re not coming at operations from the perspective of development or interaction with developers, there’s not much in it for you right now.
I accept this as our failing. We actually argue about this amongst ourselves a bit. I have noted in those discussions that the focus on cooperation with developers might be getting over emphasized.
Part of the problem here might be that the people involved are actually so ops centric that they take for granted that the operational concerns will be addressed because they will address them.
Those questions of Andi’s you have answered with “Ops!” are a BIG part of Ops… but there’s very little being written about how to handle those things in an Agile manner, because DevOps (again, quite appropriately) doesn’t concern itself with them.
Again, I disagree that those things are not a concern of DevOps. Everything to do with ops is a concern. How do you handle anything in an Agile manner? Everything starts with version control, automate the builds, make small changes, verify the changes work and make the whole process as transparent as possible. You can do that with just about anything in ops that doesn’t involve physically moving objects, which should be most things now and more and more things all the time. I’ve been talking about this for a while calling it ‘Agile Infrastructure’.
Blog post from last year
Talk from Agile Roots
Slides from Velocity 2009
Talk from Agile 2009
(All of those presentations are different, with probably 50-60% overlap)
This isn’t to say that the same agile philosophies that are fueling DevOps can’t apply elsewhere in Operations, but I do think a different term probably needs to be used when talking about those things, if only to avoid confusion over what DevOps is and where it can help.
This is exactly the reason why I state DevOps is misnamed. Specific techniques and tools might not be applicable everywhere, but the principles of thinking about the infrastructure as an application and communicating within an organization are widely applicable.
Ernest Mueller has an excellent blog post breaking this down much better than I can:
http://www.webadminblog.com/index.php/2010/03/11/defining-agile-operations-and-devops/
I’ve read it. Great post. I’ve read most things in this space.
Have you by any chance read my description of what DevOps means to me?
I appreciate the comment, you have highlighted a clear gap in the way devops is being messaged.
Many thanks!
March 29th, 2010 at 2:51 pm
fwiw, I too pointed out to him he is very confused about what devops mean. I tried to explain to him, I invited him to come talk on IRC and others had offered to talk with him too.
He seem to be after the controversy, that seems to be what his posts and attitude suggest to me.
Specifically calling out a whole group of people – first claiming they are saying things they weren’t and 2ndly pointing out completely obvious things and making a huge big deal of it is insulting too.
June 29th, 2010 at 1:17 pm
[…] Perhaps DevOps Misnamed […]