to whom it may concern

I’ve needed to write this for a long time.

I’ve struggled with how to frame everything and what tone would be best.

I would like to believe this could help OpenStack, but I have my doubts that anyone who can do anything will, or they already would have. I would like to think reading this might help someone else, but it might end up just being for me, and I’m ok with that.

For context, I was around when OpenStack was announced at OSCON in 2010 as the VP of Engineering at Cloudscaling. I have experience deploying and operating OpenStack and CloudStack implementations of some significance. I also evaluated Open Nebula, Eucalyptus and Nimbula along the way, as I really wanted a good solution to the cloud problem. I spent time consulting on OpenStack projects and last focused on OpenStack when I was hired by Jesse Andrews at Rackspace, before my whole team left to go to Nebula.

I’ve since moved on to focus on other things, but still watch OpenStack as I have a literal vested interest in the success of the project as well as many friends I care about in the community.

OpenStack was born into a vacuum created by the acceleration of AWS adoption and features, plus missteps by Eucalyptus and CloudStack with respect to the value of Open Source communities and how to cultivate them as a vendor. When OpenStack was first announced, I felt there was so much potential. I certainly made an effort to evangelize the project. Now, while many will declare victory, I’m afraid most of that potential will not be realized or worse, that OpenStack will leave a wake of bad projects that people unwittingly mistake for operable software solutions.

OpenStack has some success stories, but dead projects tell no tales. I have seen no less than 100 Million USD spent on bad OpenStack implementations that will return little or have net negative value.

Some of that has to be put on the ignorance and arrogance of some of the organizations spending that money, but OpenStack’s core competency, above all else, has been marketing and if not culpable, OpenStack has at least been complicit.

My compulsion to record this for posterity was triggered by details in the last release and related chatter around other announcements.

I would like to highlight the ‘graduation’ of Ceilometer. Ceilometer is a tragedy masquerading as a farce. In my opinion, this project should not exist and as it exists should not be relied upon for anything, much less billing customers.

First, the idea that monitoring/metering is something that should be bolted on the side of a cloud is almost as nonsensical as adding on reliability and scalability. Experience operating a web service that has been developed with instrumentation will quickly disavow anyone but a masochist of the bolted on approach. Second, Ceilometer’s implementation is such a mishmash of naive ideas and pipe dreams without regard for corner cases and failure scenarios, that Ceilometer’s association with OpenStack should be seen as a negative and graduation of the project calls into question the literal foundation of OpenStack decision making. Ceilometer’s quality is bad, even by OpenStack standards.

When I was still inclined to take actions about OpenStack related things, when Ceilometer was just a name someone proposed, I made all these same arguments but with more detail, specifics and depth. What I came to understand is that solving metering was not the primary motive. No one really cared about that. Certainly not in the way one would approach a project they intended to deploy and operate. The primary motivation was to have a project so that someone could be a Project Technical Lead (PTL).

That precedent started at the beginning with the creation of Glance, a project that never should have existed, and the subsequent creation of PTLs. The dynamics of the perceived prestige of a PTL superseded other considerations.

This dynamic allowed for a splintering of vision and mandate. If there has been any one thing that I have seen waste time implementing and operating OpenStack, that would have to be trying to coax the disparate OpenStack services, often from the same ‘release’, to work together. Press releases trumpet a new release of OpenStack as if this was working software and the naive ‘Enterprise buyer’ would rush headlong into that assumption hopped up on adrenaline and hubris. I accept Conway’s Law as a truism. Organizations will build software that reflects the communication of the organization. If the people implementing Keystone, don’t respect or understand the people implementing Nova, well, at least there is a PTL…

Don’t get me started on networking or storage…

I used to be hopeful, evangelistic even, about the possibility of a cloud service provider ecosystem built on open source. Now I am quite skeptical and feel that opportunity may be lost. Not that OpenStack doesn’t work, or at least that it can’t be made to, given certain competence and constraints, but that OpenStack doesn’t have the coherence or the will to do more than compromise itself for politics and vanity metrics.

People contribute to projects for a variety of reasons. OpenStack releases like to highlight the number of committers. That’s not a bad thing, but it’s not necessarily a good thing either. How many engineers do you think are working on AWS? GCE? How many of those committers will be the ones responsible for the performance and failure characteristics of their code? How many of those committers are dedicated to producing a world class service bent on dominating an industry? There has been some interesting, and even impressive work dedicated to improve code reviews and continuous integration, but that should not be confused with a unified vision and purpose. The per project difference in emphasis and the focus on nuances of stylistic python over other considerations of quality, let alone architecture and failure conditions determine OpenStack’s present and future. OpenStack wants to differentiate with features going up the stack, but has still not solved the foundational infrastructure and is busy furiously discussing what should be considered ‘core’. Projects that prove to be unreliable and a poor experience for operators and users ultimately damage the OpenStack ‘brand’.

OpenStack loves to declare its own victory. OpenStack’s biggest success has been getting vendors excited about abdicating their cloud strategy in lieu of going it alone. As long as OpenStack succeeds at that, there will be no shortage of funds for a foundation, summits and parties. Ultimately, if that is OpenStack’s primary accomplishment, Amazon (and perhaps others) will run away with the user driven adoption as service providers until there is nothing left of OpenStack but bespoke clouds and mailing list dramas.

I’m not sure anyone wants my advice at this point, but here it is.

  • Focus on users, not vendors. If there can’t be a benevolent dictator, there has to be an overriding conscience. The project setup solves vendor political problems more than user software problems.
  • PTLs end up being trophies. Electing PTLs every cycle is distracting and impacts continuity. Everything about this hurts OpenStack. Don’t confuse the way things are being done with the best way to do them.
  • Define a core and stick to it, or acknowledge that your governance is broken and make OpenStack an explicit free for all.
  • Stop declaring victory with vanity metrics. Divide OpenStack google trends number by the number of marketing dollars spent. The total number of committers is less meaningful if OpenStack is an excuse to sponsor parties around a collection of disparate projects.
  • Make ‘OpenStack’ brand mean something to users. Stewardship is greater than governance. If OpenStack is just a trough for the old guard IT vendors to feed slop to their existing enterprise buyers without regard to the technology or user experience then OpenStack has completely lost the plot. Make ‘OpenStack’ mean something and defend that.

I’ve certainly exceeded my 0.02 limit. Do what you will.

To all my friends at the OpenStack Summit, enjoy what Hong Kong has to offer… 幹杯.


28 responses to “to whom it may concern

  • Beth Cohen (@bfcohen)

    While I share your frustration with the fuzzy vision of the OpenStack Community, I do not share your pessimism about the project as a whole. Remember it is still a baby in technology time. The other Open Source projects have far longer histories and far smaller/less complex agendas. Its rivals aren’t not noticeably doing much better on the technology front.

    • stochasticresonance

      The poor quality of other projects is irrelevant to the points I am making.

      I disagree that I being pessimistic.

      OpenStack has no real competition, but is currently competing in the cloud equivalent of the Special Olympics.

      Talking about being a baby in technology time and then saying other projects have longer histories when Eucalyptus was announced in 2008 does not make sense. The whole narrative around cloud only really started in 2006. (Unless you count LoudCloud, but they saw it and didn’t quite pull it off.)

      That cloud is a fresh way to do IT makes this an opportunity. The size of the opportunity coupled with the sad sad state of enterprise IT is what makes this such a circus.

      OpenStack’s real rival will ultimately be the Amazon level service providers, who are more reliable, more available, always widening the gap on features and will eventually provide a lower total cost of ownership.

      There is an easy and straightforward way to prove me wrong.

      • Beth Cohen (@bfcohen)

        The other protects I was thinking of were Linux, Apache and other non-cloud projects. I agree that all the other Open Source cloud platform projects didn’t even make it out of the gate — well CloudStack made it to the first turn before getting swallowed into the Citrix maw.

        So are you thinking that Terremark’s approach might have some legs? They are taking the Amazon proprietary build approach, but they have the advantage that they started from scratch 7 years later than Amazon. Leapfrogging technology can be a good way to win, if you pick the right technology to build on..

      • Shahin Khan

        “The whole narrative around cloud only really started in 2006. (Unless you count LoudCloud, but they saw it and didn’t quite pull it off.)”
        And unless you count Sun’s N1 and maybe Cassatt and many of the Grid projects before that and the whole utility computing wave after/during. Many of these had at least a few people in them who really got it and had the vision but…)

  • Adrian Ionel

    Interesting post. Any specific examples you can share about this? “I have seen no less than 100 Million USD spent on bad OpenStack implementations that will return little or have net negative value.”

  • Phillip Remaker (@philrem)

    I’ve been on the far outskirts of OpenStack, but always wondered why so many “modules” like Glance have appeared. “Conway’s Law” is the most plausible explanation I have heard. While I hope OpenStack can get traction, I have seen time and again that when an organization focuses on the promotion and well-being of its members over that of its customers, the misaligned incentives result in a half baked offer.

  • Francesco

    so what about ceilometer, why you hate so much ceilometer what’s your solution for metering/alarming?

    • stochasticresonance

      As far as I’m concerned metering should not be a separate consideration from the cloud controller, much less a separate project.

      You need to emit the events that need to be metered and then collect them in some reasonable manner.

      Good thing most of those events are already being sent to a message queue.

      ¯\_(ツ)_/¯

  • WhosWho

    I think the weakest link in OpenStack is the Technical Committee (TC).

    Its good that the developer community is interested and different silos of developers have created projects like Heat, Celiometer, Trove and Savanna, Marconi etc.

    The problem however seems to be how the decisions are made on the incubation and graduation of these projects. Just look at the TC meeting discussions during incubation/graduation, its pretty obvious that none of the TC members have studied the related industry, API spec, design and architecture, overall implementation and scope of the project.

    I don’t know if a different architecture and standards team is needed to enforce some consistency of vision, but you can attribute almost everything wrong with OpenStack to the TC or lack thereof.

    You mentioned Celiometer, but almost all of the newer projects in the last year of growth (except may be Heat and Marconi) do not meet even the standards of Nova, and you know thats a pretty low bar.

  • Diego Parrilla ✘ (@nubeblog)

    Good post… a bit pessimistic from my point of view, pointing to some of the things I don’t like about the community and some projects. But nothing irreversible, nothing we cannot fix yet.

    But I don’t think Openstack is in bad shape. It’s still in his infancy and need to mature, but it’s becoming very quickly in the new cornerstone of a broad range of product and services for the industry. Yes, the same industry that puts tons of money in hype and buzz… and tons of money in engineers, startups and new endeavors too.

    People usually compare the Openstack evolution with the early evolution of Linux. I disagree. Linux came from the personal bet of a group of people. Openstack came from the bet of a group of enterprises. I think this is the root of all disappointments in the community: The early days of Linux are not comparable to the early days of Openstack.

    And don’t be so hard with Ceilometer. We decided not to go the Ceilometer way in our Chargeback product because we did not agree with the early approach. But obviously things are changing. A really good thing about the Openstack Community is how they tolerate evolution of the different components. Marketers can hate it, but from the pure engineering point of view, it’s correct.

    • stochasticresonance

      I find nothing pessimistic about this commentary and I’m afraid we’ll need to disagree about OpenStack.

      OpenStack is in terrible shape, if your goal was to have an ecosystem of service providers. If you think some python wrapping shell utilities written by developers that don’t really respect the hard problems of distributed computing are going to compete with AWS, good luck; have fun. More money spent on whatever will not help you.

      There will likely be a good number of OpenStack private deployments that create value for organizations but as Amazon drops their prices, OpenStack’s lack of concern for operability will make it harder and harder to compete on price.

      Ceilometer is terrible. That is not pessimism. OpenStack is meandering around because it frankly never should have been released in the first place. That meandering you consider tolerating evolution is necessary because so many choices have been essentially unworkable. This has had immense negative impacts on the OpenStack community and opportunity, whether you or any one else wants to recognize it or not.

      Your last sentence makes me think you live in an alternative universe. There is very little that is ‘correct’ about OpenStack from any semblance of an engineering point of view and the whole thing has been a marketing orgy from the outset.

  • brianaker

    Explain why you don’t think Glance needs to exist. I am actually curious.

    Thanks!

    • stochasticresonance

      Brian,

      I don’t question the functionality Glance provides so much as the manner and mechanism by which Glance came to exist as a separate project.

      At the point where Glance was created, basically by fiat, and allowed to become a third independent OpenStack project with independent deployment, configuration and coding standards that sets a precedent.

      There was definitely a need for an image registry. That need and Jay Pipes desire to move forward in an opinionated way, both with respect to code and open source methods led to Glance as a third project that made the most sense to deploy with Nova and in most configurations would depend on Swift. The reason I call Glance a manifestation of Conway’s Law is the communication challenges of all the people and parties involved are reflected in the challenges of getting those disparate systems to play nice together.

      This got 10X worse with Keystone.

      So I don’t question the need for Glance’s functionality, but allowing the circumstance that created a separate project with a third way of doing everything led to a lot of the difficulties that OpenStack is still struggling with or has decided not to resolve.

      I know the CI has evolved and helps a lot, but I think that mitigates these problems more than it solves them.

      I hope that clarifies my opinion.

      Best,
      Andrew

  • Whither OpenStack? | Speaking of Clouds - Distributed applications and virtual infrastructures

    […] those less Twitter-obsessed than I, here are a few of the key pieces: to whom it may concern What I saw at the OpenStack Summit Why vendors can’t sell OpenStack to enterprises Not Everyone […]

  • jaypipes

    Hi Andrew,

    I can’t tell whether you think that somehow I personally created Glance “by fiat”. I can tell you that nothing could be further from the truth.

    If there’s something that I personally did that offended you, I am sorry; I have no idea what that is, though. Feel free to email me personally to discuss it.

    For the record, I didn’t create Glance, nor did I come up with the idea that it should be a separate project.

    I feel that you are correct that Glance should not be a separate project. I think it likely should be contained in the Cinder project.

    Also, you write that Glance has “a third way of doing everything”. What are you referring to? Glance uses Swift’s method of daemonization/binding as well as many other things from Swift; it uses many of Oslo’s libraries (and unless I’m mistaken, is moving to use more of them than not).

    I share many of your concerns about OpenStack engineering choices and the quality of certain projects. But I think those are solvable problems, and I’ll continue to play a small part in the OpenStack contributor community as long as I see progress, slow as it may be.

    I try to stay out of the bikeshed conversations on the mailing lists and stick to discussions around engineering. I left the OpenStack Technical Committee a long time ago, and I certainly never thought that being a PTL was some sort of prestigious thing.

    Best,
    -jay

    • stochasticresonance

      Jay,

      we talked about a bit of this out of band but I want to clarify for anyone else who happens to come along and read this.

      For the record, you never did anything to personally offend me, in fact I enjoy your company and respect your perspectives.

      I didn’t mean to single you out, because I don’t think that it was anything you did per se but a combination of choices and circumstances that were made my many parties.

      I associated your name with that happening because you happened to be the champion at the time Glance officially became elevated and the Project Technical Lead mechanism was set in motion.

      In the beginning, when there was just Swift and Nova, each of which had been built with different idioms, no one had the political will or mandate to unify the projects and therefore the technical community.

      I singled out Glance because it was the first one, and set the precedent, but Keystone was actually an order of magnitude worse with respect to splintering the efforts for a variety of technical and political reasons.

      There is a fine line between avoiding bikeshedding and abdicating responsibility. One of the problems that plagues many projects, not just OpenStack, are technology relativism and equivalence arguments that end up leaving decisions up to those most willing to raise their voices.

      “One of the penalties for refusing to participate in politics is that you end up being governed by your inferiors.” — Plato

      And such it is with OpenStack…

      Next time we have the chance, dinner on me.

      -Andrew

  • Andrew Trossman

    Nothing beats the malevolent dictator approach. However that just can’t work here for such a big community. I think Mark Shuttleworth said it best in his keynote session, “Openstack is fragile” and we should “let 1000 flowers bloom”. He also said we need to focus on making Openstack “lean, clean, fast, scalable, and reliable”. I think this touches on your concerns which I share. I guess I’m optimistic because I there’s a huge community of talented developers, operators and users that, with this kind of pragmatic guidance, will get us there.

    His session can be found here

    http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/canonical-keynote-building-business-value-through-interoperability

  • Andrew Trossman

    benevolent :-) Freudian slip ;-)

  • R McKinley

    What if I cannot use AWS because of privacy? I don’t want each developer bothered with running VMWare or VirtualBox on their local machines just to develop for our web and database backend. What is more appropriate than OpenStack in this scenario?

    • stochasticresonance

      use vagrant on local machines

      if you think running OpenStack for you dev environment is a better option… gl;hf

      • R McKinley

        I don’t want to put infrastructure onto every developer’s desk. I have 100 software engineers. I do not want them to be bothered with running VMs on their local machines. Your response to me is that I do want that? What, besides OpenStack, should I use for an experience like AWS or DigitalOcean on a private network?

      • stochasticresonance

        so your software engineers don’t have computers at their desks? Seems like that would make software engineering difficult…

        Vagrant enables efficient and effective workflows moving code from developers to production with source controlled configuration management.

        I didn’t tell you what you should want. I told you what I believe is the better solution.

        No solution is perfect for all context. If you believe something else solves your constraints, OpenStack is your oyster.

        I suggest you at least have one person of the 100 spend time looking at what I recommended before you dismiss it.

  • robesker

    Declining to comment on the rest (mostly for lack of time)… the notion of a “collective conscience” in lieu of a benevolent dictator / charismatic tyrant struck a chord. One of the other comments opined that the TC isn’t formed optimally. Rather than taking swings at the whole, I’d like to work on fixing that. Given current state, how might we achieve said collective conscience. By what means?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: