Category Archives: Dysfunction

to whom it may concern

I’ve needed to write this for a long time.

I’ve struggled with how to frame everything and what tone would be best.

I would like to believe this could help OpenStack, but I have my doubts that anyone who can do anything will, or they already would have. I would like to think reading this might help someone else, but it might end up just being for me, and I’m ok with that.

For context, I was around when OpenStack was announced at OSCON in 2010 as the VP of Engineering at Cloudscaling. I have experience deploying and operating OpenStack and CloudStack implementations of some significance. I also evaluated Open Nebula, Eucalyptus and Nimbula along the way, as I really wanted a good solution to the cloud problem. I spent time consulting on OpenStack projects and last focused on OpenStack when I was hired by Jesse Andrews at Rackspace, before my whole team left to go to Nebula.

I’ve since moved on to focus on other things, but still watch OpenStack as I have a literal vested interest in the success of the project as well as many friends I care about in the community.

OpenStack was born into a vacuum created by the acceleration of AWS adoption and features, plus missteps by Eucalyptus and CloudStack with respect to the value of Open Source communities and how to cultivate them as a vendor. When OpenStack was first announced, I felt there was so much potential. I certainly made an effort to evangelize the project. Now, while many will declare victory, I’m afraid most of that potential will not be realized or worse, that OpenStack will leave a wake of bad projects that people unwittingly mistake for operable software solutions.

OpenStack has some success stories, but dead projects tell no tales. I have seen no less than 100 Million USD spent on bad OpenStack implementations that will return little or have net negative value.

Some of that has to be put on the ignorance and arrogance of some of the organizations spending that money, but OpenStack’s core competency, above all else, has been marketing and if not culpable, OpenStack has at least been complicit.

My compulsion to record this for posterity was triggered by details in the last release and related chatter around other announcements.

I would like to highlight the ‘graduation’ of Ceilometer. Ceilometer is a tragedy masquerading as a farce. In my opinion, this project should not exist and as it exists should not be relied upon for anything, much less billing customers.

First, the idea that monitoring/metering is something that should be bolted on the side of a cloud is almost as nonsensical as adding on reliability and scalability. Experience operating a web service that has been developed with instrumentation will quickly disavow anyone but a masochist of the bolted on approach. Second, Ceilometer’s implementation is such a mishmash of naive ideas and pipe dreams without regard for corner cases and failure scenarios, that Ceilometer’s association with OpenStack should be seen as a negative and graduation of the project calls into question the literal foundation of OpenStack decision making. Ceilometer’s quality is bad, even by OpenStack standards.

When I was still inclined to take actions about OpenStack related things, when Ceilometer was just a name someone proposed, I made all these same arguments but with more detail, specifics and depth. What I came to understand is that solving metering was not the primary motive. No one really cared about that. Certainly not in the way one would approach a project they intended to deploy and operate. The primary motivation was to have a project so that someone could be a Project Technical Lead (PTL).

That precedent started at the beginning with the creation of Glance, a project that never should have existed, and the subsequent creation of PTLs. The dynamics of the perceived prestige of a PTL superseded other considerations.

This dynamic allowed for a splintering of vision and mandate. If there has been any one thing that I have seen waste time implementing and operating OpenStack, that would have to be trying to coax the disparate OpenStack services, often from the same ‘release’, to work together. Press releases trumpet a new release of OpenStack as if this was working software and the naive ‘Enterprise buyer’ would rush headlong into that assumption hopped up on adrenaline and hubris. I accept Conway’s Law as a truism. Organizations will build software that reflects the communication of the organization. If the people implementing Keystone, don’t respect or understand the people implementing Nova, well, at least there is a PTL…

Don’t get me started on networking or storage…

I used to be hopeful, evangelistic even, about the possibility of a cloud service provider ecosystem built on open source. Now I am quite skeptical and feel that opportunity may be lost. Not that OpenStack doesn’t work, or at least that it can’t be made to, given certain competence and constraints, but that OpenStack doesn’t have the coherence or the will to do more than compromise itself for politics and vanity metrics.

People contribute to projects for a variety of reasons. OpenStack releases like to highlight the number of committers. That’s not a bad thing, but it’s not necessarily a good thing either. How many engineers do you think are working on AWS? GCE? How many of those committers will be the ones responsible for the performance and failure characteristics of their code? How many of those committers are dedicated to producing a world class service bent on dominating an industry? There has been some interesting, and even impressive work dedicated to improve code reviews and continuous integration, but that should not be confused with a unified vision and purpose. The per project difference in emphasis and the focus on nuances of stylistic python over other considerations of quality, let alone architecture and failure conditions determine OpenStack’s present and future. OpenStack wants to differentiate with features going up the stack, but has still not solved the foundational infrastructure and is busy furiously discussing what should be considered ‘core’. Projects that prove to be unreliable and a poor experience for operators and users ultimately damage the OpenStack ‘brand’.

OpenStack loves to declare its own victory. OpenStack’s biggest success has been getting vendors excited about abdicating their cloud strategy in lieu of going it alone. As long as OpenStack succeeds at that, there will be no shortage of funds for a foundation, summits and parties. Ultimately, if that is OpenStack’s primary accomplishment, Amazon (and perhaps others) will run away with the user driven adoption as service providers until there is nothing left of OpenStack but bespoke clouds and mailing list dramas.

I’m not sure anyone wants my advice at this point, but here it is.

  • Focus on users, not vendors. If there can’t be a benevolent dictator, there has to be an overriding conscience. The project setup solves vendor political problems more than user software problems.
  • PTLs end up being trophies. Electing PTLs every cycle is distracting and impacts continuity. Everything about this hurts OpenStack. Don’t confuse the way things are being done with the best way to do them.
  • Define a core and stick to it, or acknowledge that your governance is broken and make OpenStack an explicit free for all.
  • Stop declaring victory with vanity metrics. Divide OpenStack google trends number by the number of marketing dollars spent. The total number of committers is less meaningful if OpenStack is an excuse to sponsor parties around a collection of disparate projects.
  • Make ‘OpenStack’ brand mean something to users. Stewardship is greater than governance. If OpenStack is just a trough for the old guard IT vendors to feed slop to their existing enterprise buyers without regard to the technology or user experience then OpenStack has completely lost the plot. Make ‘OpenStack’ mean something and defend that.

I’ve certainly exceeded my 0.02 limit. Do what you will.

To all my friends at the OpenStack Summit, enjoy what Hong Kong has to offer… 幹杯.


Speed Chess

Is typing speed a factor in programmer productivity? Would you improve at chess if you moved the pieces faster?

Jason Gorman, who appears to be a relatively reflective programmer based on his blog and his twitter, tweeted this comment about chess a few days ago.

A bunch of people I follow ReTweeted, and it caught my attention because I used to think this way, both about chess and programming.

Now I don’t claim to be the greatest at either activity, but I’ve put some time and I have enough ability to claim to be above average at both. (Which is to say, I’m aware of my relative mediocrity when compared with real masters)

I’ve played chess off and on for years. I learned to play when I was quite young and I could usually beat other casual players quite easily. After losing to rated tournament players, I spent some time studying the game.

For a long time, I thought the best way to learn was to methodically look for the best move and I thought playing blitz games was somewhat degenerate. Luckily, someone convinced me to start playing blitz games regularly, and that accelerated my understanding of the game considerably.

I still play better when I take the time to be methodical, but that’s not the same thing as learning. I think the blitz games accelerated my learning for two reasons, first, because playing at that speed put that many more games, positions and patterns in front of me and two, because I didn’t attach as much ego to the games I experimented more, which led to that many more positions.

I do think there is a point of diminished returns to just move the pieces faster, and improvement is predicated on some reflection, but I will contend unequivocally that, unless you are a master, you will improve at chess if you spend a considerable portion of your playing time moving the pieces faster.

The same applies to code. You don’t need to type +100 words per minute, but if you can’t touch type at least 40-50 wpm, spend the 20 minutes a day for a month or so until you can. You will never regret it. (And I worked as a programmer for years before I could touch type.)

I would walk you through the arguments, but there is already a classic Yeggethon on the topic, which articulates all the positions I would and then some.

“Lose Your First 100 Games As Quickly As Possible”
–Proverb for Go Beginners


Safari? SQL? Seriously?

So let’s say you your friend has a wordpress blog and you weren’t he wasn’t smart enough to back up the files in the theme and then working late after a long day to tweak some details, as the grand finale of a comedy of errors involving lots of open tabs and multiple browsers, the css file that actually lays out the whole theme was truncated in a manner which is most approproriately described as ‘obliteratisticated’.

First your friend comes to the realization that while WordPress stores revisions of posts and pages, editing themes is without a safety net. As this sinks in, he realizes that while the site has been reloaded on Firefox, Safari is sitting there with the page loaded with the old css. Hallelujiah, he can just dig that out of the cache… there is a cache right?

This was rocking Safari 3.1.2 on OS X 10.5.5.

I’m writing this because at point there was a lot of google and a bit of thrashing, uhm, for my friend.

You might find a ~/Library/Caches/com.apple.Safari/Cache.db and think there is something in there, and waste a bunch of time but this will only lead to swearing and teeth gnashing.  At some point Safari decided that was not the place to put the Cache.  If you have a newer version of Safari this is not your cache, this is an artifact from some Safari of days gone by.

The Cache.db file you want is in /private/var/
/private/var/folders/sn/SomeLongRidiculousHashedUpBS/-Caches-/com.apple.Safari/Cache.db to be exact.

That big blob of a file is a little sql db and you can search through it with sqlite.

sqlite2 Cache.db

Now that is is open you can get the lay of the land.

sqlite> .schema
CREATE TABLE cfurl_cache_blob_data
     (entry_ID INTEGER PRIMARY KEY,
      response_object BLOB,
      request_object BLOB,
      receiver_data BLOB, 
      proto_props BLOB,
      user_info BLOB);
CREATE TABLE cfurl_cache_response
    (entry_ID INTEGER 
                   PRIMARY KEY 
                   AUTOINCREMENT
                   UNIQUE,
     version INTEGER,
     hash_value INTEGER,
     storage_policy INTEGER,
     request_key TEXT UNIQUE,
     time_stamp NOT NULL
                        DEFAULT
                        CURRENT_TIMESTAMP);
CREATE TABLE cfurl_cache_schema_version
    (schema_version INTEGER);
CREATE INDEX proto_props_index 
    ON cfurl_cache_blob_data(entry_ID);
CREATE INDEX request_key_index
    ON cfurl_cache_response(request_key);
CREATE INDEX time_stamp_index
    ON cfurl_cache_response(time_stamp);

The data you want is in the cfurl_cache_blob_data table in the receiver_data column.

If you know something about the contents of what you are looking for then it is going to be easy:


select receiver_date 
          from cfurl_cache_blob_data 
          where receiver_data
          like '%What I Know%'

If you know the file name you should be able to use where request_object like '%filename%'

This version of Safari for whatever reason, and maybe it is my settings, doesn’t seem to be caching images, which is pretty awesome…

Hey Apple… WTF?


Dysfunction: Inattention to Results

Never mistake motion for action.

– Ernest Hemingway

The final dysfunction from The Five Dysfunctions of a Team is Inattention to Results. This ultimate team dysfunction occurs when members of the team value something other than the collective goals.

This dysfunction doesn’t mean they have lives and balance outside the context of the company (that’s another post. . .) but that the members of the team are focused on highlighting their own achievements regardless of the overall outcome.

You can smell this dysfunction when individuals or groups start to point to everything they did right, even when the overall project is flailing. (maybe especially when it is flailing)

“Product management did such a good job of creating requirements, its those lousy developers that didn’t understand, I mean just look at this pretty document. Don’t you love the font . . .”

“I wrote the code exactly to the spec, not that those idiot clients and product managers know what they want anyway . . . I am a programming god.”

Have you ever participated in either side of that conversation? Played Not-my-fault-I-am-teh-awesome hot potato?

(Quality Assurance gets blamed for everything, cause as everyone knows ‘kwalytee’ starts with the last people who touched the software, duh. . . You do have QA, right?)

There is nothing so useless as doing efficiently that which should not be done at all.

– Peter Drucker

Inattention to Results is most damaging to an organization when it becomes institutionalized processes. When people stop trying to do the best thing and just make sure they go through the checklist.

When things go South, the checklist becomes a buzzword compliant shield, an Agile security blanket.

When I first read the model, I thought the most important dysfunction to address was trust, because it is the foundation of all the other dysfunctions. After a little reflection, I’m starting to think that you need at least as much focus on results, if not more. Sure you need to work on trust, because if you don’t, eventually the rest of the dysfunctions grow to be monstrous, but ‘winning’ gives you the opportunity to work through a lot of trust issues. Especially, if working on trust (. . .conflict, commitment, accountability. . .) is an explicit goal.

This is all well and dandy, but what does that mean? How can I use this?

There is not a formula that you can just plug in and solve this problem. On some level, this is all about culture. On another, an individual, not on top of the food chain, recognizing these dysfunctions has to navigate between Scylla and Charybdis. One becomes faced with the prospect of being at odds with the perception of management, watching and suffering the dysfunctional culture, or finding something else to pay the mortgage.

(eh, maybe that’s why the book’s target audience is executive teams. . .)

Insanity: doing the same thing over and over again and expecting different results.

– Albert Einstein

what to do?

Reflect and adjust my friend. . . Reflect and Adjust

add to del.icio.us :: Add to Blinkslist :: add to furl :: Digg it :: add to ma.gnolia :: Stumble It! :: add to simpy :: seed the vine :: :: :: TailRank :: post to facebook


Dysfunction: Avoidance of Accountability

The ancient Romans had a tradition: whenever one of their engineers constructed an arch, as the capstone was hoisted into place, the engineer assumed accountability for his work in the most profound way possible: he stood under the arch.

Back to The Five Dysfunctions of a Team, so we have built up Trust, Conflict and Commitment. What do we need now?

Accountable: subject to the obligation to report, explain, or justify something; responsible; answerable.

In the model from the book, the avoidance of accountability is a psychological byproduct of the underlying dysfunctions of the team. Without firm commitments, after healthy disagreements are addressed, there is a tendency to not hold people accountable. We have a hard time holding people accountable, when we know they never really committed.

Ok. . . uhm, and?

How many of you have witnessed(participated in) the following scenario?

  • Incredible pressure to deliver a product/feature
  • Developers are hacking their little hearts out (very few tests being written, of course)
  • Quality Assurance is testing whatever they can see flying over the wall (with little or no prior knowledge of what it will be, bless their little hearts)
  • Product Management is doing whatever they do (anything but looking at what is actually being developed)
  • Daily stand ups where no problems are mentioned
  • We are so Agile, yay

Features delivered are a disaster. . . PMs are freaking out, Devs are indignant, QA is whimpering in the corner. What happens next? Rinse and Repeat. . . WTF?!??

I believe one of the fundamental principles of all Agile methods is ‘Reflect and Adjust‘. The built in adjustment is only going to help your team if it isn’t used as an excuse to ‘just do better next time’.

It is not only what we do, but also what we do not do, for which we are accountable.

– Molier

That doesn’t mean I’d advocate command and control shock and awe style, that just puts everyone into ‘cover your donkey’ mode and totally destroys the foundation of trust.

Honestly, Good Product Management is HARD. Good development is a creative endeavor that ebbs and flows. Good Quality Assurance is an acquired skill. All of these pieces are heavily dependent on each other.

Holding each other accountable doesn’t have to be antagonistic, but it does have to be honest. (truth, remember)

Brutally honest… Shit happens, but it doesn’t have to happen all the time.

Around the time of Caesar, there was a European tribe that, when the assembly horn blew, always killed the last warrior to reach his assigned place, and no one enjoyed fighting this tribe.

Don’t settle for mediocre, for yourself or for your organization.

Some people can’t handle ‘Truth’, and that is the seed for every other dysfunction.

Change your ‘organization’, or. . . change ‘your’ organization.

add to del.icio.us :: Add to Blinkslist :: add to furl :: Digg it :: add to ma.gnolia :: Stumble It! :: add to simpy :: seed the vine :: :: :: TailRank :: post to facebook


Dysfunction: Lack of Commitment

He who is most slow in making a promise is the most faithful in performance of it.

-Jean-Jacques Rousseau

What is commitment?

When it comes to software development, there are two reasonable definitions:

  • the act of binding yourself to a course of action
  • confinement to a mental institution or hospital

And sometimes there will be great difficulty telling the difference.

In the ideal world, the perfectly architectured codebase has neat little modules with perfectly separated concerns, all perfectly tested of course. Santa Claus, Big Foot and the Leprechauns all love it, pay tons of money for it and are waiting patiently for the next version.

In the reality of software projects I have participated in directly, indirectly, discussed with colleagues or watched from afar, there are often competing internal and external pressures involving time, money, quality, ideology and last but not least ego. That isn’t to say that all these projects went poorly, it is just to recognize the pressures.

So what does all that have to do with commitment?

The Five Dysfunctions of a Team claims that unless the concerns of team members are not voiced and resolved through healthy conflict, the team will lack commitment to a clear course of action.

The original model targets executive decision making, which is paramount obviously.

But… effectively communicating these higher level objectives and the implications they will have in a software product or process in and among the development team, then being willing and able to adjust to the feedback may have an even bigger impact on a software company’s success in the long run.

That doesn’t mean all decisions require consensus to get commitment, but that dissenting voices are an opportunity to reevaluate, clarify and optimize. Simply exercising that opportunity can create an incredible sense of team duty in those who offered the dissent, even if they still disagree.

This is going to be a re-occurring theme, but if you are a software company, and you don’t have smart passionate people building your software, just give up now.

Seriously. . .

Now that there are smart passionate people building the software in a culture of trust and healthy conflict, get everyone unified and committed and you’re pretty much set.

With all the internal and external pressures, the success of a project and often a company hang in the balance.

add to del.icio.us :: Add to Blinkslist :: add to furl :: Digg it :: add to ma.gnolia :: Stumble It! :: add to simpy :: seed the vine :: :: :: TailRank :: post to facebook


Dysfunction: Fear of Conflict

Truth springs from argument amongst friends…

-David Hume

The ‘fear’ of conflict from The Five Dysfunctions of a Team manifests in teams that have protocols or cultures that steer the group away from conflict as a ‘bad’ thing. Ironically, open conflict can be a positive optimization mechanism.

I know what you are thinking, ‘We’re not afraid of conflict, we’re conflicting all the time.’ Open mean spirited ad hominem attacks are obviously dysfunctional, but maybe Patrick Lencioni’s publisher didn’t think the book should be called, ‘The Five Dysfunctions of a Team That Aren’t Totally Obvious’.

Here’s where things get interesting. . . and there might be a fine line, but if a team can truly conflict with passion, and even frustration, where the sole purpose is producing the best possible solution, they will produce a better solution in less time.

Conflict can never really be avoided. Attempting to avoid conflict is ensuring that there is no true resolution and the latent conflicts will just compound and fester. Eventually, the suppression creates fissures in the team and back-channel us-against-them style back-biting. Even more energy and creativity is lost when the key decisions that were made, without resolving conflicts, result in half hearted actions.

Of course, none of the benefit of open conflict are possible without ‘Trust’.

Where all think alike, no one thinks very much.

– Walter Lippmann

If you want to build good software, you have to start with the premise that your team is made up of creative smart people, or else you have already lost. The type of people that build really good software, also tend to have passion and opinions. Those opinions are often not the same as other people who also might happen to build really good software.

In software development, every member of a team is making decisions that will have short term and long term impact on success. These decisions are often being made minute by minute. Can a team afford not to have a unified vision of what they are trying to accomplish and what they value that has not been purified in the crucible of collective examination and honest conflict?

What are you afraid of?

(ok, one more gratuitous quote)

The harder the conflict, the more glorious the triumph. What we obtain too cheap, we esteem too lightly; it is dearness only that gives everything its value. I love the man that can smile in trouble, that can gather strength from distress and grow brave by reflection. ‘Tis the business of little minds to shrink; but he whose heart is firm, and whose conscience approves his conduct, will pursue his principles unto death.

-Thomas Paine

add to del.icio.us :: Add to Blinkslist :: add to furl :: Digg it :: add to ma.gnolia :: Stumble It! :: add to simpy :: seed the vine :: :: :: TailRank :: post to facebook


%d bloggers like this: