jump to navigation

meatcloud manifesto April 1, 2009

Posted by stochasticresonance in Puppet.
Tags: , , ,
6 comments

GUI installers are evil…

Period.

You might need to have a little wizard to walk through setting stuff up the first time for the uninitiated, but if you think you are doing the poor hack who has to install your stuff on hundreds of machines any favors, you got another thing coming.

If you don’t provide some way to install your software fully unattended, please do not sell it to schools, businesses, non-profits or religious organizations with more than one person.

You might as well be killing kittens.

Actually, that goes for the interface to running applications as well.

I think we need the Open Meatcloud Manifesto… (because the world needs more manifestos and this is the perfect date to start a movement)

Let me see what I can work up…

  • scribble* *scribble* *erase* *scribble* *crumple* *scribble*

I’ll keep it short and simple:

We hold these truths to be self-evident

We hold these truths to be self-evident

Show your support for this movement by making your own copy of the manifesto and posting the pictures.

(Special privileges will be bestowed on those willing to tattoo the manifesto… on their forehead.)

No meatclouds were harmed in the making of this manifesto.

Viva La Revolución

Open For Business Follow Up March 29, 2009

Posted by stochasticresonance in Puppet.
Tags: ,
4 comments

After my ‘Open For Business‘ post, I had several conversations out of band.

A few people responded with ‘we feel your pain’.

That’s not really the point I was trying to make. This quarter is the best quarter Reductive Labs has ever had for revenue, and next quarter looks like it will be a even better as people both realize the value of Puppet automation and are seeking better ROI from their budgets.

Any new skill is going to come with some clumsy missteps as you are learning, and sales is a skill. We are technologists learning to run a business, and sales is part of it. Frankly, we offer a great value for where we are in the market and I don’t know who I feel worse for, the enterprise sales guys who are offering to buy each others licenses, the IT managers who are buying stuff their admins won’t be able to use or those admins. So while it might be a little painful in the ‘here we go again’ sense, that is really just part of the sales process, but not anywhere near as painful as this. (sorry, that was probably indulgent)

Another comment is that Puppet might be perceived as not being ‘Enterprise Ready’. Perception is reality, and some people still argue Ruby isn’t ready. Rails, not ready… How about Windows? Anyone ever read the bugs open against Oracle 10g? The fact is Puppet has several very successful implementations in Fortune 100 companies.  Whatcha gonna do?

With big companies, the discussion sometimes breakdown when we get to culture and process. The problem isn’t the technology per se, it’s the fact that there are vertically and horizontally splintered fiefdoms running around vying for power and pride. Every group wants to do things their way. When the X application group has segregated systems from the Y application group isn’t so bad, but if the X group has to submit tickets to the monitoring group to find out what is happening on their own machines, implementing a centralized management solution becomes really difficult to navigate from the bottom up.

Maybe you should stop when it hurts?

Maybe you should stop when it hurts?

Sometimes, I just want to tap out to the stupid.

(That was probably indulgent too…)

*shrug*

That does look painful…

Go ‘Enterprise’ go…

Open For Business March 27, 2009

Posted by stochasticresonance in Puppet, ramble.
Tags: ,
4 comments

What has never been doubted, has never been proven.

–Denis Diderot

Let’s be honest, no one ‘really’ knows exactly how ‘Open Source’ and ‘Business Model’ should fit together. There is a broad spectrum of ‘Open Source’ approaches, from the ‘Purists‘ to the ‘Open Core‘ to the pragmatic ‘Flip Floppers‘. The differences are subtle nuances to anyone not steeped in the culture and trying to explain them will just confuse most people not familiar with the lore. There is my partner Luke, who has as many questions as answers. Then there a people with a lot of answers, but of course, they are all different.

Analysts are talking about the ‘billion dollar opportunity‘, but the brightest example of Open Source business success, that everyone references, garners harsh comments like this one. Does there have to be a clash along this boundary? Sure, Redhat has essentially adopted a traditional licensing model, but if you don’t understand the benefit of the role they play in the larger OSS ecosystem then you aren’t really paying attention. Having strong ideals is one thing, paying for groceries, mortgages, and braces is another.

One of the most influential programmers of a generation is trying to figure out how to make a living. You think capistrano would be abandoned if Jamis got a nickel everytime someone ran cap deploy? These were innovative projects that added a ton of value to an ecosystem and captured little, if any. There are plenty of millionaires who sold total crap and cashed out.

Nowadays people know the price of everything and the value of nothing.

–Oscar Wilde

Is there some way to balance Open Source ideals with opportunity? I sure hope so, but I’m not entirely sure how the rest of the story goes. On more than one occasion, I’ve been on a call with someone from an ‘enterprise’ talking about how they are replacing Opsware or Bladelogic installations that were minimum high 6 figure checks to implement, wanting Reductive Labs services, and balking at spending ~$20K with us as too expensive. Someone convinced somebody that the promise of automation was worth X dollars, possibly didn’t deliver it, and now a solution is not worth X/50? What gives?

The problem in most cases is the technology has been adopted at the front line by admins who want to get more done, know there is a better way, but don’t necessarily have budget to spend. Their managers don’t know much about what they do, view all of IT as a cost center anyway and now they want more money… for what? Reductive Labs best customers are organization with a semi-technical manager, who trusts her people, wants them to succeed and believes IT should be a differentiator. Organizations like this have funded Puppet’s development.

The push back is often from companies with 10K servers and a 500 headcount meatcloud, all with root on every box. Some progressive admin has got Puppet up and solving small problems, but knows releasing it into the organization at large without some training and process improvements is like distributing chainsaws to every kindergartener on the playground at recess. No child left behind… So instead they fumble through implementing a solution, don’t really change the process, and then call us when Puppet brings down a service by doing exactly what their code told Puppet to do. (Ok, so I’ll admit, this doesn’t happen every time, but it has happened.)

The greatest obstacle to discovery is not ignorance — it is the illusion of knowledge.

–Daniel J Boorstin

John M Willis says he talks to the ‘enterprise’ all the time, and most people haven’t even heard of Puppet yet. I’m sorry, try to catch the next clue train. Managing 10K machines with a 500 headed meathydra rooting boxes is costing a lot more money than it needs to.  We would love to solve that problem for you. If you aren’t looking for a solution to this, you probably aren’t solving other problems either. I understand that keeping up with every little technology is hard, but at some point you just have to concede that you aren’t in the race anymore. Hopefully, you have at least heard of Linux…

The problem is most the time the conversation stops at the admin technical level. We don’t have a small army in ties who can take executives for metaphorical (or not…) martinis at strip clubs. Or a marketing budget to take out full page ads in CIO magazine. In my world view that should be a good thing. We are technologist at heart, who want to build solid solutions and to deliver value at the point of attack. Unfortunately, we don’t have the sales process to exploit inefficiencies in the marketplace with asymmetric understanding and access to information (or an economic system that rewards value necessarily).

Somewhere between the ‘enterprise’ who doesn’t know what to make of open source software and the people who love open source like sharks like seals, is there an opportunity to capture value? Scylla and Charibdis?

I like to think that more and more people will realize that the price of Open Source is expertise, that they have to pay their team for months of learning we could have provided in a week, that a ’support’ contract is a reciprocal agreement, and the alternatives are much more expensive when comparing the total cost of ownership. If you pay us half of what a commercial solution would cost, we can automate your infrastructure, train your team and support the solution. In fact, we’ll make sure Puppet spreads jam on your toast in the morning. (If you pay us anywhere near the same amount, we’ll automate baking the bread too.)

Markets are constantly in a state of uncertainty and flux, and money is made by discounting the obvious and betting on the unexpected.

–George Soros

Practical Puppet at MWRC March 14, 2009

Posted by stochasticresonance in Puppet.
Tags: , , , , ,
add a comment

**update** The underlying repos changed so the demo won’t work from the user data. It will still work after running ‘apt-get update’ on the box. If I get bored I might repackaged it.

*update*  Confreaks has my MWRC video up. The notes below about launching on EC2 are probably still useful for context.

These are my slides from Mountain West Ruby Conf.

Starts with a little intro about the beginning of computing, before ‘computer science’, driven by desire for computation in math and physics, when there wasn’t a division between the people that program and people that understand how the systems work. (this period probably lasted for about 10 minutes.)

Then some Puppet code…

These modules will build passenger and install rails.

Finishes with a short discussion on the tribes, trading ideas, evolutions, the opportunity of clouds and encouraging people to do something awesome.

The talk was given with a live demo of this code building rails in EC2.

You can do it too if you have an EC2 account.

Start ami-2b10f742 littleidea/mwrcpuppetrailsdemo.manifest.xml sending this as the user data:

#!/usr/bin/env puppet

rails::site{ the_next_big_thing:
servername => "www.tnbt.com",
rails_version => "2.2.2"
}

Set the server name and the rails version to whatever you want.

If you need more gems or packages you can add:

#for gems
package { gemname:
    ensure => installed,
    provider => gem
}

#for native packages
package { pgkname:
    ensure => installed,
}

If you need a certain version of a gem or package, change ‘installed’ to the version you want (the default will install the latest available.)

The image is based on ami-7cfd1a15 Ubuntu Intrepid packaged by Eric Hammond with puppet and facter installed. The puppet manifests from the slides are modules found in /usr/share/puppet/modules.

When the ami starts up it will grab the user data and run it.

You can also ssh to your instance and just put the puppet code in a file (strip the #! /usr/bin/puppet) and run puppet –debug to watch it build.

puppet --debug mysite.pp

Safari? SQL? Seriously? February 16, 2009

Posted by stochasticresonance in Dysfunction, ramble.
Tags: , , , , ,
2 comments

So let’s say you your friend has a wordpress blog and you weren’t he wasn’t smart enough to back up the files in the theme and then working late after a long day to tweak some details, as the grand finale of a comedy of errors involving lots of open tabs and multiple browsers, the css file that actually lays out the whole theme was truncated in a manner which is most approproriately described as ‘obliteratisticated’.

First your friend comes to the realization that while WordPress stores revisions of posts and pages, editing themes is without a safety net. As this sinks in, he realizes that while the site has been reloaded on Firefox, Safari is sitting there with the page loaded with the old css. Hallelujiah, he can just dig that out of the cache… there is a cache right?

This was rocking Safari 3.1.2 on OS X 10.5.5.

I’m writing this because at point there was a lot of google and a bit of thrashing, uhm, for my friend.

You might find a ~/Library/Caches/com.apple.Safari/Cache.db and think there is something in there, and waste a bunch of time but this will only lead to swearing and teeth gnashing.  At some point Safari decided that was not the place to put the Cache.  If you have a newer version of Safari this is not your cache, this is an artifact from some Safari of days gone by.

The Cache.db file you want is in /private/var/
/private/var/folders/sn/SomeLongRidiculousHashedUpBS/-Caches-/com.apple.Safari/Cache.db to be exact.

That big blob of a file is a little sql db and you can search through it with sqlite.

sqlite2 Cache.db

Now that is is open you can get the lay of the land.

sqlite> .schema
CREATE TABLE cfurl_cache_blob_data
     (entry_ID INTEGER PRIMARY KEY,
      response_object BLOB,
      request_object BLOB,
      receiver_data BLOB,
      proto_props BLOB,
      user_info BLOB);
CREATE TABLE cfurl_cache_response
    (entry_ID INTEGER
                   PRIMARY KEY
                   AUTOINCREMENT
                   UNIQUE,
     version INTEGER,
     hash_value INTEGER,
     storage_policy INTEGER,
     request_key TEXT UNIQUE,
     time_stamp NOT NULL
                        DEFAULT
                        CURRENT_TIMESTAMP);
CREATE TABLE cfurl_cache_schema_version
    (schema_version INTEGER);
CREATE INDEX proto_props_index
    ON cfurl_cache_blob_data(entry_ID);
CREATE INDEX request_key_index
    ON cfurl_cache_response(request_key);
CREATE INDEX time_stamp_index
    ON cfurl_cache_response(time_stamp);

The data you want is in the cfurl_cache_blob_data table in the receiver_data column.

If you know something about the contents of what you are looking for then it is going to be easy:


select receiver_date
          from cfurl_cache_blob_data
          where receiver_data
          like '%What I Know%'

If you know the file name you should be able to use where request_object like '%filename%'

This version of Safari for whatever reason, and maybe it is my settings, doesn’t seem to be caching images, which is pretty awesome…

Hey Apple… WTF?

Mountain West Ruby Conference February 9, 2009

Posted by stochasticresonance in Puppet, Ruby.
Tags: , , ,
add a comment
Rocky Mountain High

Rocky Mountain High

I’m definitely looking forward to Mountain West Ruby Conference. Last year was most excellent, and this year I’m presenting. Puppet, clouds, live demo, no net, should be a good time…

MWRC is a single track regional conference, but it brings in some of the biggest names in the Ruby world. (maybe we can get Matz to come next year) Sharing the stage with some of these guys is an honor and a privilege, if not a little intimidating. And if last year was any indication, even the guys you haven’t heard of will be awesome.  I can’t recommend this conference highly enough.

Pat Eyler of On Ruby is one of the organizers and just posted an interview with me. He is interviewing other speakers as a lead up to the conference, so keep an eye out.

Use your powers for Awesome!!!

Use your powers for Awesome!!!

Puppet and Capistrano February 9, 2009

Posted by stochasticresonance in Puppet.
Tags: ,
add a comment

Can you smell what the Puppet is cookin’? January 18, 2009

Posted by stochasticresonance in Puppet.
Tags: ,
3 comments

Imitation is the highest form of flattery…

Things are getting really interesting in the configuration management space.  The confluence of clouds, web 2.0, dev-ministration and chocolate sauce.

Chef is upping the ante, one way or another. I’m a little saddened that the majority of the ideas in Chef were discussed in the context of Puppet and implemented by someone who has made a living off of Puppet… but I’m not totally surprised.

I understand why Adam would do this, and on many levels it parallels Luke’s relationship with cfengine.  Adam has probably used Puppet to solve more real problems and build more infrastructure than anyone else, just like Luke had done consulting with cfengine for years before Puppet was born. In a sense, Adam is the embodiment of the future of system administration that Luke had envisioned and hoped to create.

Socrates, Plato, Aristotle… and so it goes.

Adam is a smart guy who thinks clearly about solving problems, had Puppet as an example and data from the front line. He strikes me as a genuine fellow and while we aren’t best friends, I have enjoyed his conversation and insights on more than one occasion.  Chef adds some nice functionality that was obvious and some pieces that are differences in philosophy.  I’ve done my best to absorb both Luke and Adam’s expressed positions, and paradoxically, in my estimation they are both right, in their context. One way or another, there are still a lot of machines out there being managed by meatclouds, so there is plenty of work to do.

Puppet was first released in 2005 and has grown in functionality and adoption since that time. Puppet is revolutionizing system administration, similarly to how Rails revolutionized web development, and Chef can only accelerate that process, by its own merits and by driving innovations in Puppet. At the heart of the story are many questions about progress, open source, community, technology, obligations and the attributions.  The storyline already has mystery, intrigue, tragic heroes, and double agents. There is bound to be some drama, just because there are humans involved, but sometimes nothing motivates like a nice punch in the mouth.

Remember when Nintendo was king of the world then couldn’t sell anything, and now they pwn Playstation and Xbox… Or when apple was awesome and then wasn’t and now we all have iPhones… Innovators innovate, ebb and flow, ebb and flow.

And rails is merb is rails, and you never know what the future holds.  Buckle up… I’m just sayin’…

Progress is impossible without change, and those who cannot change their minds cannot change anything.

–George Bernard Shaw

Landfall Mashups January 5, 2009

Posted by stochasticresonance in ramble.
Tags: , ,
add a comment

Earlier this year, I was really impressed with stormpulse, a mash up that provides nearly real time visualizations of hurricanes and the expected trajectories (as a probabilistic band). The whole thing is driven by public data and in addition to being really cool if you are in Salt Lake City, is probably even cooler if you need to decide to leave before landfall.

After the Ruby Hackfest (which really turned into an intro to machine learning and classification), I kept thinking about what data sets might be interesting to train SVM. At the same time, rolling around in the back of my brain was the auto-scaling discussion that burned a swath across the blogo-tweeto-sphere. Those two things collided in the background and prompted this post. (The best analysis and something I mention to Adam Jacobs in a side channel is that these applications don’t scale linearly and naively throwing hardware at the problem without analysis can bring an application to it’s knees. Adam thinks it is all about the appropriate apologies, which will probably never change…)

But I digress…

One of the points brought up in the original post is that auto scaling based on monitoring doesn’t have time to react, but that is like noticing there are 80 mph winds and surmising that their might be a hurricane. I contend that the ‘Digg/Oprah/Slashdot’ effects have some characteristic patterns which could be easily characterized and the trajectories of the ’storms’ could be predicted, if we could only get the data.  The data is all there, it’s just not together and not publicly accessible. Maybe there is some way to get people to donate access logs so we can characterize the curves?  Once you have those characterized, take the model and analyze the real time logs and feed that info back into both the model and an autoscaling engine.  Start to bring up machines before the big waves start to break…  Take it a step farther and get a protocol/etiquette for a track/pingback mechanism, just to let you know there might be a tropical storm to keep an eye on…

Just a little idea… seems pretty straight forward…

Is anyone out there trying to do something like this?

2009: To Be Is To Do January 2, 2009

Posted by stochasticresonance in Me, ramble.
add a comment

Life is like riding a bicycle. To keep your balance you must keep moving.

– Albert Einstein

Can someone help me scratch an itch?

Can someone help me scratch an itch?

2008 was a hard year for me. 2008 was also a wonderful year for me. I just felt like at times there just was too many plates to keep spinning. Most of them are still spinning and there is surprisingly little to sweep up, but there was enough wobbling to make things painfully interesting. The whirlwind year saw my wife graduate from medical school and my second son turn one. I joined forces with Luke and I started blogging. Since June, I was in Los Angeles, Nashville, Portland OR, Toronto and San Fransisco twice, for a week each time. I got to meet a lot of interesting people, was introduced to a lot of new ideas and I was able to share some of mine. I can’t say I was totally ineffective, but one thing 2008 exposed to me is a lot of my inefficiencies.

Don't breathe

Don't breathe

I decided the theme for 2009 was going to be ‘Embrace the Suck’, not because I believe 2009 is shaping up to be so horrible, but because I know there will be things out of my control. C’est la vie is the kinder gentler version of the same sentiment. There are things I can control and things I cannot but if I can’t control myself, what can I hope to effect? Whatever the circumstances, the choice is always between two extremes. Duality can take many forms, but I think the lens most useful for myself is love vs. fear. ‘Embrace the Suck’ might be another way of saying ‘love the fear’. I’m going to do my best to be motivated by passion and love, while minimizing justification and self delusion.

I’m also going to try to be more balanced.

I like to move it move it

I like to move it move it

Not static balance, dynamic balance… Martha Graham mixed with Bruce Lee, float like a butterfly, sting like a bee. The balance that is throwing your center of gravity and not letting it get away. The balance of extremes, a swinging mobile with long levers sweeping through space and time, manifesting very little in the center, balanced duality, appropriately more kind and more fierce. 2009 will be more stillness when resting and more intensity when not. More exertion and more replenishment…

So far, so good, started off the year by making some new friends at the URUG hackfest. We had hoped to master Support Vector Machines, but that was a bit ambitious for an afternoon, but we did enough to get a handle on the problem, and liked the experience enough to commit to doing it regularly. Something good should come out of that. I’ve committed to learning LISP this year (maybe the dragon book next) and we’ll see if I can’t make just a bit of headway on some other languages as well. I’ve got a lot of different goals and responsibilities for Reductive Labs this year, definitely taking that into uncharted territory one way or another. 2009 is going to be a lot of things, not sure what exactly, but I’m not waiting to find out.

A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly.

–Robert Heinlein

Let’s get this party started… The spice must flow

I’m ready for you 2009, bring it on…

Only I will remain…