Category Archives: ramble

Infrastructure Renaissance

Infrastructure is Code! FTW!

Wait… WTF does that even mean?

On the one hand this is a rallying cry for those of us in the configuration automation space, on the other hand it confuses innocent bystanders.

@zhenjl: @botchagalupe but ops ppl have always coded using perl, tcl (if u remembr that :)…now we just have ‘sexier’ langs like ruby and python (original)

Sure, there has always been scripts, which is technically code, but this is missing the point a bit.

For starters, getting ops teams to recognize and steal all applicable processes and tools from developers to produce and version there scripts would be a win, but that’s just the starting line…

The point isn’t just that you can write some scripts to help you do something with your infrastructure then exit.

The ‘Infrastructure’ IS an application, a long running process with inputs, outputs and state. Or at least infrastructure is very close to becoming one…

Everyone paying attention knows something is happening… what is it?  Hazy, foggy, smoky, cloudy, whatever…

EC2 is the herald of an infrastructure enlightenment, leading us out of the dark ages.

(We’ll ignore the history of virtualization technology, and ESX Server for now, but I’m sure someone has an opinion on all this.)

The bottom line is there are resources, and I can manipulate them through an API. With EC2, these are compute resources of various shapes and sizes, with some storage resources.

Virtualized networking is not exposed through a public API yet, but we’re still climbing abstraction mountain together and we all know it is just a little higher in this cloud. (What’s up Cisco, where you at? Oh, there you are… and these guys too… thanks Ben )

At some point in the not too distant future, web service calls and a credit card will be enough to create and manipulate arbitrary compute, storage and network resources.

So now what?

Until someone provides Sysadmin as a Service, there is still quite a bit of work left to be done. These resources need to be configured to provide services.

When adding capacity was a purchase order, then wait for weeks to rack and cable, taking a day or so to configure the machines was not rate limiting, and lost in the noise. When you can bring up new server instances in minutes, that same configuration step is conspicuous waste.

You might be thinking you’ll make images for each of your services and manage those. This approach to management exchanges the configuration drift of tradition IT management for image sprawl. Do those notes that got scribbled six months ago when the web server image was created have accurate details about how that machine is configured? Are you sure? The only way to know is start the VM and inspect it. Frankly, the same thing applies to the running instances started yesterday.

That’s where tools like Puppet comes in. The benefit is not just that a machine gets configured by running a script. Puppet is an application that audits your system configuration, and changes the details out of sync with policy. That policy is code with semantics that can be developed like software.

That’s the shift. Instead of thinking about managing server counts, we can reason about abstractions of service configurations. The pieces exposed by an API are not just instances of server and storage resources, but abstractions of services. The configuration of proxies, monitoring, middleware, web servers and databases are components to be composed in the dynamic ‘infrastructure’ application.

This is just the beginning, but I don’t want to spoil the rest of the story… at least not yet.

It is not necessary to change. Survival is not mandatory.

W. Edwards Deming

You might be a cloud computing expert if…

And let it be noted that there is no more delicate matter to take in hand, nor more dangerous to conduct, nor more doubtful in its success, than to set up as the leader in the introduction of changes.  For he who innovates will have for his enemies all those who are well off under the existing order of things, and only lukewarm supporters in those who might be better off under the new.

Niccolò Machiavell

I’m taking a break from trying to be useful. You know how the song goes:

  • “If the top three items in your browser history include the words “twitter”, “innovation” or “interoperability”, you might be a cloud computing expert”
  • “If you’ve ever asked Amazon to Open Source their API, you might be a cloud computing expert”
  • “If you think ‘Open Sourcing’ an API makes any sense, you might be a cloud computing expert”
  • “If you think there is a difference between ‘business investors’ and ‘speculators’, you might be a cloud computing expert”
  • “If you think cloud computing will save the rain forest, you might be a cloud computing expert”
  • “If you believe building in EC2 always has a lower TCO, you might be a cloud computing expert”
  • “If you are not preparing for ‘The Trough of Disillusionment’, you might be a cloud computing expert”
  • “If you just can’t get enough *aaS, you might be a cloud computing expert”
  • “If you want to debate public vs private, you might be a cloud computing expert”
  • “If you spend a day talking about which Star Wars character you should be and it is a good day at work, you might be a cloud computing expert”
  • “If you understood more than two points on this list, you’re def. a fscking cloud computing expert”

I’m sure this list is much longer, I’m just trying to do my part and get you started…

Make us all proud…

Open For Business

What has never been doubted, has never been proven.

–Denis Diderot

Let’s be honest, no one ‘really’ knows exactly how ‘Open Source’ and ‘Business Model’ should fit together. There is a broad spectrum of ‘Open Source’ approaches, from the ‘Purists‘ to the ‘Open Core‘ to the pragmatic ‘Flip Floppers‘. The differences are subtle nuances to anyone not steeped in the culture and trying to explain them will just confuse most people not familiar with the lore. There is my partner Luke, who has as many questions as answers. Then there a people with a lot of answers, but of course, they are all different.

Analysts are talking about the ‘billion dollar opportunity‘, but the brightest example of Open Source business success, that everyone references, garners harsh comments like this one. Does there have to be a clash along this boundary? Sure, Redhat has essentially adopted a traditional licensing model, but if you don’t understand the benefit of the role they play in the larger OSS ecosystem then you aren’t really paying attention. Having strong ideals is one thing, paying for groceries, mortgages, and braces is another.

One of the most influential programmers of a generation is trying to figure out how to make a living. You think capistrano would be abandoned if Jamis got a nickel everytime someone ran cap deploy? These were innovative projects that added a ton of value to an ecosystem and captured little, if any. There are plenty of millionaires who sold total crap and cashed out.

Nowadays people know the price of everything and the value of nothing.

–Oscar Wilde

Is there some way to balance Open Source ideals with opportunity? I sure hope so, but I’m not entirely sure how the rest of the story goes. On more than one occasion, I’ve been on a call with someone from an ‘enterprise’ talking about how they are replacing Opsware or Bladelogic installations that were minimum high 6 figure checks to implement, wanting Reductive Labs services, and balking at spending ~$20K with us as too expensive. Someone convinced somebody that the promise of automation was worth X dollars, possibly didn’t deliver it, and now a solution is not worth X/50? What gives?

The problem in most cases is the technology has been adopted at the front line by admins who want to get more done, know there is a better way, but don’t necessarily have budget to spend. Their managers don’t know much about what they do, view all of IT as a cost center anyway and now they want more money… for what? Reductive Labs best customers are organization with a semi-technical manager, who trusts her people, wants them to succeed and believes IT should be a differentiator. Organizations like this have funded Puppet’s development.

The push back is often from companies with 10K servers and a 500 headcount meatcloud, all with root on every box. Some progressive admin has got Puppet up and solving small problems, but knows releasing it into the organization at large without some training and process improvements is like distributing chainsaws to every kindergartener on the playground at recess. No child left behind… So instead they fumble through implementing a solution, don’t really change the process, and then call us when Puppet brings down a service by doing exactly what their code told Puppet to do. (Ok, so I’ll admit, this doesn’t happen every time, but it has happened.)

The greatest obstacle to discovery is not ignorance — it is the illusion of knowledge.

–Daniel J Boorstin

John M Willis says he talks to the ‘enterprise’ all the time, and most people haven’t even heard of Puppet yet. I’m sorry, try to catch the next clue train. Managing 10K machines with a 500 headed meathydra rooting boxes is costing a lot more money than it needs to.  We would love to solve that problem for you. If you aren’t looking for a solution to this, you probably aren’t solving other problems either. I understand that keeping up with every little technology is hard, but at some point you just have to concede that you aren’t in the race anymore. Hopefully, you have at least heard of Linux…

The problem is most the time the conversation stops at the admin technical level. We don’t have a small army in ties who can take executives for metaphorical (or not…) martinis at strip clubs. Or a marketing budget to take out full page ads in CIO magazine. In my world view that should be a good thing. We are technologist at heart, who want to build solid solutions and to deliver value at the point of attack. Unfortunately, we don’t have the sales process to exploit inefficiencies in the marketplace with asymmetric understanding and access to information (or an economic system that rewards value necessarily).

Somewhere between the ‘enterprise’ who doesn’t know what to make of open source software and the people who love open source like sharks like seals, is there an opportunity to capture value? Scylla and Charibdis?

I like to think that more and more people will realize that the price of Open Source is expertise, that they have to pay their team for months of learning we could have provided in a week, that a ‘support’ contract is a reciprocal agreement, and the alternatives are much more expensive when comparing the total cost of ownership. If you pay us half of what a commercial solution would cost, we can automate your infrastructure, train your team and support the solution. In fact, we’ll make sure Puppet spreads jam on your toast in the morning. (If you pay us anywhere near the same amount, we’ll automate baking the bread too.)

Markets are constantly in a state of uncertainty and flux, and money is made by discounting the obvious and betting on the unexpected.

–George Soros

Safari? SQL? Seriously?

So let’s say you your friend has a wordpress blog and you weren’t he wasn’t smart enough to back up the files in the theme and then working late after a long day to tweak some details, as the grand finale of a comedy of errors involving lots of open tabs and multiple browsers, the css file that actually lays out the whole theme was truncated in a manner which is most approproriately described as ‘obliteratisticated’.

First your friend comes to the realization that while WordPress stores revisions of posts and pages, editing themes is without a safety net. As this sinks in, he realizes that while the site has been reloaded on Firefox, Safari is sitting there with the page loaded with the old css. Hallelujiah, he can just dig that out of the cache… there is a cache right?

This was rocking Safari 3.1.2 on OS X 10.5.5.

I’m writing this because at point there was a lot of google and a bit of thrashing, uhm, for my friend.

You might find a ~/Library/Caches/ and think there is something in there, and waste a bunch of time but this will only lead to swearing and teeth gnashing.  At some point Safari decided that was not the place to put the Cache.  If you have a newer version of Safari this is not your cache, this is an artifact from some Safari of days gone by.

The Cache.db file you want is in /private/var/
/private/var/folders/sn/SomeLongRidiculousHashedUpBS/-Caches-/ to be exact.

That big blob of a file is a little sql db and you can search through it with sqlite.

sqlite2 Cache.db

Now that is is open you can get the lay of the land.

sqlite> .schema
CREATE TABLE cfurl_cache_blob_data
      response_object BLOB,
      request_object BLOB,
      receiver_data BLOB, 
      proto_props BLOB,
      user_info BLOB);
CREATE TABLE cfurl_cache_response
    (entry_ID INTEGER 
                   PRIMARY KEY 
     version INTEGER,
     hash_value INTEGER,
     storage_policy INTEGER,
     request_key TEXT UNIQUE,
     time_stamp NOT NULL
CREATE TABLE cfurl_cache_schema_version
    (schema_version INTEGER);
CREATE INDEX proto_props_index 
    ON cfurl_cache_blob_data(entry_ID);
CREATE INDEX request_key_index
    ON cfurl_cache_response(request_key);
CREATE INDEX time_stamp_index
    ON cfurl_cache_response(time_stamp);

The data you want is in the cfurl_cache_blob_data table in the receiver_data column.

If you know something about the contents of what you are looking for then it is going to be easy:

select receiver_date 
          from cfurl_cache_blob_data 
          where receiver_data
          like '%What I Know%'

If you know the file name you should be able to use where request_object like '%filename%'

This version of Safari for whatever reason, and maybe it is my settings, doesn’t seem to be caching images, which is pretty awesome…

Hey Apple… WTF?

Landfall Mashups

Earlier this year, I was really impressed with stormpulse, a mash up that provides nearly real time visualizations of hurricanes and the expected trajectories (as a probabilistic band). The whole thing is driven by public data and in addition to being really cool if you are in Salt Lake City, is probably even cooler if you need to decide to leave before landfall.

After the Ruby Hackfest (which really turned into an intro to machine learning and classification), I kept thinking about what data sets might be interesting to train SVM. At the same time, rolling around in the back of my brain was the auto-scaling discussion that burned a swath across the blogo-tweeto-sphere. Those two things collided in the background and prompted this post. (The best analysis and something I mention to Adam Jacobs in a side channel is that these applications don’t scale linearly and naively throwing hardware at the problem without analysis can bring an application to it’s knees. Adam thinks it is all about the appropriate apologies, which will probably never change…)

But I digress…

One of the points brought up in the original post is that auto scaling based on monitoring doesn’t have time to react, but that is like noticing there are 80 mph winds and surmising that their might be a hurricane. I contend that the ‘Digg/Oprah/Slashdot’ effects have some characteristic patterns which could be easily characterized and the trajectories of the ‘storms’ could be predicted, if we could only get the data.  The data is all there, it’s just not together and not publicly accessible. Maybe there is some way to get people to donate access logs so we can characterize the curves?  Once you have those characterized, take the model and analyze the real time logs and feed that info back into both the model and an autoscaling engine.  Start to bring up machines before the big waves start to break…  Take it a step farther and get a protocol/etiquette for a track/pingback mechanism, just to let you know there might be a tropical storm to keep an eye on…

Just a little idea… seems pretty straight forward…

Is anyone out there trying to do something like this?

2009: To Be Is To Do

Life is like riding a bicycle. To keep your balance you must keep moving.

— Albert Einstein

Can someone help me scratch an itch?

Can someone help me scratch an itch?

2008 was a hard year for me. 2008 was also a wonderful year for me. I just felt like at times there just was too many plates to keep spinning. Most of them are still spinning and there is surprisingly little to sweep up, but there was enough wobbling to make things painfully interesting. The whirlwind year saw my wife graduate from medical school and my second son turn one. I joined forces with Luke and I started blogging. Since June, I was in Los Angeles, Nashville, Portland OR, Toronto and San Fransisco twice, for a week each time. I got to meet a lot of interesting people, was introduced to a lot of new ideas and I was able to share some of mine. I can’t say I was totally ineffective, but one thing 2008 exposed to me is a lot of my inefficiencies.

Don't breathe

Don't breathe

I decided the theme for 2009 was going to be ‘Embrace the Suck’, not because I believe 2009 is shaping up to be so horrible, but because I know there will be things out of my control. C’est la vie is the kinder gentler version of the same sentiment. There are things I can control and things I cannot but if I can’t control myself, what can I hope to effect? Whatever the circumstances, the choice is always between two extremes. Duality can take many forms, but I think the lens most useful for myself is love vs. fear. ‘Embrace the Suck’ might be another way of saying ‘love the fear’. I’m going to do my best to be motivated by passion and love, while minimizing justification and self delusion.

I’m also going to try to be more balanced.

I like to move it move it

I like to move it move it

Not static balance, dynamic balance… Martha Graham mixed with Bruce Lee, float like a butterfly, sting like a bee. The balance that is throwing your center of gravity and not letting it get away. The balance of extremes, a swinging mobile with long levers sweeping through space and time, manifesting very little in the center, balanced duality, appropriately more kind and more fierce. 2009 will be more stillness when resting and more intensity when not. More exertion and more replenishment…

So far, so good, started off the year by making some new friends at the URUG hackfest. We had hoped to master Support Vector Machines, but that was a bit ambitious for an afternoon, but we did enough to get a handle on the problem, and liked the experience enough to commit to doing it regularly. Something good should come out of that. I’ve committed to learning LISP this year (maybe the dragon book next) and we’ll see if I can’t make just a bit of headway on some other languages as well. I’ve got a lot of different goals and responsibilities for Reductive Labs this year, definitely taking that into uncharted territory one way or another. 2009 is going to be a lot of things, not sure what exactly, but I’m not waiting to find out.

A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly.

–Robert Heinlein

Let’s get this party started… The spice must flow

I’m ready for you 2009, bring it on…

Only I will remain…

Weather the Storm (or ‘learn something for next time’)

I’ve had a lot of thoughts to blog, just not a lot of time. Not to make excuses… if you followed my twitter you’d know I try to stay busy.

Rubyconf was great, my talk was ok, B+ maybe, I learned some lessons, next talk will benefit, lots of cool smart people, trying to explain Puppet to Ruby developers is always interesting. I gave another version of the talk at the Utah Ruby Users Group.

Ignite Salt Lake was epic, but apparently the AV capture was a fail.  Total lighting fail and everyone is really dark in the video. This is sad because the talks really were excellent.  The strategy is to try some image processing and see if there is anything worth salvaging, and if not just take the audio and match it with the slides. Another lesson learned, next time will be better.

My wife is traveling almost every week lately for interviews and leaving me home with the boys.  She is trying to match in an opthalmology program and the whole process is black magic, which means a bit of stress and we have no idea where we will be in 6-18 months.

Part of me wants to stay here in Salt Lake City, because I love the high mountain desert and will always have roots here, but I always like new things and it would be nice to get acquainted with a new town.

I’ll be glad when the process is over one way or another.

There seems to be an upsurge in the Agile backlash lately, but I have personally made most of the same observation/critiques before.  Should make the next Agile Roundtable discussion more fun.  On a related note, there seems to be an upswing of ‘pastry‘ searches on blogs (maybe just the holidays).  Oh well, popularity drives dilution, and now I really really really want to have an AgileFringe conference.

I feel some good posts percolating… (next time will be better…)

Winter Happens

Blow, blow, thou winter wind
Thou art not so unkind,
As man’s ingratitude.

-William Shakespeare

Fast approaching is my second winter as a home ‘owner’, where ‘owner’ really means I have a debt to a bank, but it keeps us warm and dry. Preparing for winter takes some effort. There’s a list of niggling things to do, like blow out the sprinkler lines and other fun stuff; most of which is quite minor and doable but has to be done. Or not… but the alternative to preparing for winter is almost always more costly in the long run.

All in all, we live a pretty easy life. With a gas furnace, indoor plumbing, and refrigeration, it’s practically luxurious compared to what 99% of the people who have lived on this planet experienced.

Imagine if preparing for winter was stockpiling enough wood and food for 3 months. And how about hay for the animals, who you need to plow the fields, once they thaw, of course. Ok, now what happens if winter lasts for 4 months? 3 months in and you are out of food, wood, hay and there is 4 feet of snow and 20 miles from the nearest chance to get more. And just to make it more fun, imagine if everyone else had also only been prepared for the 3 month version of winter. Good times…

So… where were we… oh, yeah, global economic collapse…

Winter happens… everything that goes up must come down. Cycles are inevitable.

The fundamentals of our economy are strong...

The fundamentals of our economy are strong...

What are you going to do? Curl up in a fetal ball and wait for sweet slumber? Hopefully, it’s not blizzard after blizzard from here on out (The DOW has to stop at zero, right?) and you still have a metaphorical month of wood and food, and some hope for getting more, but all signs point to winter, so there is no sense fiddling around with the grasshoppers.

Tim O’Reilly is fond of spreading the ‘create more value than you capture’ meme. (his recent web 2.0 ‘do stuff that matters‘ keynote now seems almost prescient, ‘reality bubble’ indeed… watch Tim at Boston Ignite too)

I really like that idea. If everyone was doing that, most of the problems on the planet would melt away.

I’d like to extend that idea just a bit…

Start by creating value but, ‘capture more value than you consume’.

Hopefully that can get us through to Spring…

Adversity draws men together and produces beauty and harmony in life’s relationships, just as the cold of winter produces ice-flowers on the window-panes, which vanish with the warmth.

-Soren Kierkegaard

Miles Per Gallon

In truth, a good case could be made that if your knowledge is meagre and unsatisfactory, the last thing in the world you should do is make measurements; the chance is negligible that you will measure the right things accidentally.

George Miller

Last week I drove to Los Angeles with my wife, two kids, my mother-in-law and her sister in our not so eco-friendly SUV. Hilarity ensued, but that’s not the point of this post.

I don’t usually drive this car. Until recently, most everything in my life was in about a 5 mile radius, work, play, whatever. The SUV is ‘her’ car; my wife drives it to where she does research and home most everyday.  Occasionally, we all get in and go somewhere exciting, like grandma’s.

You are starting to wonder what is the point, and I’m hoping I’m about to make some.

This late model SUV has a speedometer, a tachometer, and in addition has a computer that is estimating instantaneous MPG and average MPG since the last fillup, since the beginning of the trip, since the epoch, etc.

On the way to LA, with this SUV stuffed full of people and random stuff, I averaged just over 20 MPG at ~80 MPH. On the way back, I averaged 25.6 MPG with the same people and even more stuff and nearly identical MPH.

On the way there, I kept watching the gauges and thinking ‘these rolling hills are killing my gas mileage’. On the way back, I was optimizing the speed and acceleration patterns, only dipping below 20 MPG on the steepest inclines.

If one rides a bike on hills, he or she figures this out pretty fast. There are certain speeds you can hit a hill that can be maintained with less effort than it will take to go slower, all this is based on fitness, the equipment and the hill. Also, acceleration uphill is fairly expensive. Your body figures this stuff out because energy expenditure and efficiency have consequences, and the feedback is rapid.

In an automobile, I doubt I ever would have figured it out without the gadgetry.

You can’t manage what you can’t measure.

What does this have to do with software? Velocity is a commonly used idiom in project planning. I’ve seen velocity measured different ways in practice, and seen it’s measurement discussed in many more.

But I’ve never really seen efficiency measured with relation to velocity. The only time efficiency is considered is with respect to getting more velocity.

I’ve never seen anyone in a planning meeting ask, how can we maintain this velocity more efficiently? Is this velocity the most efficient? Over the life of any decent sized project, this seems like it would be important.

(Aside: The XPers might, as they have a notion of sustainable pace, but I’ve never been in a pure XP setting. I’m starting to think they only exist in legends or they are like the Jedi scattered by the clone wars trying to cultivate the force in swamps and deserts… I’ll be at Agile 2008 this week, so I’m hoping to come home with some new Jedi minds tricks.)

How many miles per gallon are you getting? Are you losing MPG to the rolling hills?

Perhaps the more important question, how would you know? What do you measure? (The instantaneous estimate feedback was key for me, hmm…)


Strange is our situation here upon earth. Each of us comes for a short visit, not knowing why, yet sometimes seeming to a divine purpose. From the standpoint of daily life, however, there is one thing we do know: That we are here for the sake of others…for the countless unknown souls with whose fate we are connected by a bond of sympathy. Many times a day, I realize how much my outer and inner life is built upon the labors of people, both living and dead, and how earnestly I must exert myself in order to give in return as much as I have received.

-Albert Einstein

I have some thoughts I just have to get out so I can get back to being productive, so just bare with me.

In Salt Lake City, I know about 4 public places where people will panhandle, plus randomly in front of convenience stores. I think I avoid these places subconsciously. Sometimes, when I am confronted by someone who wants money for food and my schedule and mood permit, I offer to buy them a meal. The majority decline the offer. Sometimes, I have bought someone a plate of food and in exchange asked them questions about their life. I’ve heard some interesting stories, from people who would probably be described as intelligent by any measure other than the fact that they are destitute.

I went from FOOCamp to a San Francisco hotel in the Tenderloin. The accommodations there were quite nice inside, but once on the street, I was struck by what I will describe as ‘social decay’, or at least that is the best two word phrase I have come up with. I haven’t been in a big city for a while and perhaps Salt Lake City’s harsh winter buffers my experience by making the city less than inviting to those with no means, but I found the openness at FOOCamp, juxtaposed with the reality of a big city, jarring to say the least.

In less than 2 hours, I went from wanting to engage with everyone and regretting there wasn’t more time, to wishing I was invisible and not wanting to make eye contact with anyone. The experience was personally painful. Isn’t walking down a street oblivious to the obvious suffering the definition of sociopathic?

The alternative of engaging led to clumsy communication at best, and while I have a hard time not giving someone who claims to be hungry a dollar, I’m not sure I did any good either.

What does this have to do with anything? Honestly, I’m not sure. I could probably sit and weave it into some metaphors and analogies about software teams and dysfunction, about communication impedance, about the problems caused by decisions made by those who will not be effected by them, about changing yourself and your organization by refusing to propagate an unempowering meme, but I don’t feel like being that clever today.

…When one sits in the Hoop Of The People,
one must be responsible because
All of Creation is related.
And the hurt of one is the hurt of all.
And the honor of one is the honor of all.
And whatever we do effects everything in the universe…

passed down from White Buffalo Calf Woman

The problem doesn’t seem like one I can overcome, certainly not by myself. Humanity has obviously struggled with this since the beginning of time. What does it mean to be a ‘citizen’, to be a ‘society’? I don’t know either, but it seems worth thinking about and acting upon. But what to do? Many social programs seem caught up in their own self righteousness and have little impact but well paid executives. We don’t need another one of those, but what would facilitate true long lasting change? Maybe I’m just being too sensitive, but the alternative feels like self deception and a small death of conscience.

Now back to the regularly scheduled program…


%d bloggers like this: