turn the page November 3, 2009
Posted by stochasticresonance in Business, Puppet.3 comments
Don’t cry because it’s over. Smile because it happened.
–Dr. Seuss
I’m no longer part of Reductive Labs. While this development is not quite heart breaking, I am always sad to leave behind something that I care about, but I also understand and feel strongly that this will be better for everyone.
Since raising venture funding, Reductive Labs has been building an adept team. That team has a critical mass in Portland, OR. There are now eight people there and I am tied to Salt Lake City for the near future, and who knows where for the next few years. Staying aligned with everything that Reductive Labs was doing and needed was becoming more and more of a challenge.
Puppet is a great tool and organizations get a lot of value from it. There are some exciting Puppet developments underway which present unique challenges and opportunities for both the technology and the business. I will now watch them from afar.
My involvement with Reductive Labs has been a period of pronounced growth. I learn many lessons about business and technology, that would have been difficult to learn any other way. I thank Reductive Labs and Luke Kanies for sharing the opportunity.
For the next month or so, I’ll spend more time with my family (when I’m not at the climbing gym and getting massages) and sort through a couple ideas.
Part of me wants to continue pushing the Infrastructure Renaissance forward in some capacity, and part of me wants to build something awesome, and part of me wants to work on stuff that matters.
Maybe there is a good way to combine them all together…
There are a lot of interesting problems in the world… (to solve or cause)
Insurmountable opportunity…
What are you going to do next?
Duct Typing September 26, 2009
Posted by stochasticresonance in ramble.Tags: duct tape
2 comments
Today people were tweeting up an old interview with Al3x Payne and some devs from Twitter about using Scala.
Al3x or anyone else has the right to use for whatever programming language they want. No questions asked…
But today I was reflecting on Al3x’s rationalization of the type system motivating a move to Scala. I mean we’ve heard this before and the whole thing got a lot of Ruby panties in a bundle when it first came out.
But checking for nil and kind_of? all over the place is not Ruby’s failing… that’s a big ball of mud in any language and I’ve seen people resort to the (null != foo) && (foo instanceof Bar)) anti-patterns in Java.
Al3x asserts this is an inherent property of large systems in dynamic languages. I’ll assert it is a property of systems that have grown organically to the point where the programmers can’t reason about what is coming or going. I’ll further argue that you get more out of thinking functionally towards solving this issue than you do from stricter typing.
If you find yourself constantly checking for nil/null or really being concerned with types except around the edges of a system, that’s a code smell IMHO.
But maybe another layer of duct tape will fix it…
Catching a breath… September 10, 2009
Posted by stochasticresonance in ramble.add a comment
I’ve had a lot of things I wanted to get out there, but didn’t have the time to sit down and craft it. The result is a bunch of half formed embryonic posts, which will grow to something hopefully very soon. The past few months have been a whirlwind.
Starting with the investment process and our partnership with True Ventures.
Then there was Agile Roots, which was awesome and I’m not just saying that because I help organize it.
Check out some of the Agile Roots videos. (The sound on some presentations is suboptimal because of issues with the venue’s system, but the content is generally good.)
Then Velocity Conference, face meltingly awesome… the best practioners conference for people who are pushing the boundaries of what is possible. Some of the Velocity videos… my talk didn’t get recorded this time, which I’m sure improved the quality immensely…
Then Structure, similar to Velocity, but for people who wear more buttons and spend less time at the command line.
July was a blurr of phone calls from people wanting to sell Reductive Labs office space and IT management solutions, lunch meetings, podcasts and day trips. Some work got done.
August had more of the same, with more podcasts, Ignite Salt Lake and finally Agile 2009 (where my presentation with Paul Nasrat was recorded by InfoQ)… capped off with a little Reductive Labs love at the Ruby flippin’ Hoedown.
I know I haven’t done any of these events proper justice, but I plan to at least collect some of my thoughts and post my slides, etc. in the next few days.
Did I mention I have two young sons and a wife in medical residency? Don’t try this at home kids…
Meatcloud Manifesto Redux August 9, 2009
Posted by stochasticresonance in ramble.Tags: Automation, Meatcloud, opportunity
add a comment
People rarely succeed unless they have fun in what they are doing.
–Dale Carnegie
When I penned the immortal words ‘Give me an API or give me death…’ and gifted the meatcloud manifesto to the world while admonishing applications that don’t provide APIs, I was perhaps too focused on GUI installers. That was an artificial limitation which I hope to now rectify.
The automation of repetitive tasks presents an insurmountable opportunity.
Freedom from effort in the present merely means that there has been effort stored up in the past.
–Theodore Roosevelt
DRY should be a familiar principle familiar to anyone writing code and not living in a cave — Don’t Repeat Yourself. Unfortunately, sometimes it’s a lot easier to see the replication in files with code than it is in other aspects of our work, but search your shell history and look for the obvious. The real opportunity is to pop up a level and observe the workflows.
How much of what your organization does with emails and phone calls could be accomplished with a chain of automation?
That’s not so say we don’t want humans to make decisions, but do we really need to have someone push that button or fill out that form? As it turns out, machines are really really good at doing exactly the same thing over and over.
How much of what your organization does is replicated across an industry? industries?
Sometimes the benefits are not in what we can add, but what we can take away…
Opportunity is missed by most people because it is dressed in overalls and looks like work.
–Thomas Alva Edison
As I said… insurmountable opportunity…
viva la meatcloud!
The Four Steps to the Epiphany August 8, 2009
Posted by stochasticresonance in Business.Tags: Business, Test Driven
add a comment
I just read The Four Steps to the Epiphany by Steven Blank and I highly suggest the book to anyone with an interest in business.
The books is mistitled, but ‘The 47 potentially recursive steps to incrementally determine a business model and not waste a lot of money on bad assumptions‘ probably didn’t work.
Blank outlines a ‘Customer Development’ process in parallel with the development of the product.
There is an a clear bias for venture funded tech startups, and a more subtle enterprise bias, but the process and mindset being advocated is generally applicable, even when some of the anecdotes might not be.
The core message is that the ‘build it and they will come’ strategy sets the stage for epic failure, and the book is filled with examples.
In my opinion, the Customer Development strategy is really Test Driven Business Development. The steps are a series of experiments which test hypotheses about the potential product and the market. The end result is the product developed in tandem with the positioning and sales process in line with a strategy appropriate for the market.
The Four Steps outlines a taxonomy of Markets: Existing Markets, New Markets, and Resegmented Markets, for the purpose of matching the market type with an appropriate sales strategy. A reoccurring theme, supported with evidence, is that ramping up a sales team or building expensive infrastructure not aligned with the nature of the market can burn through a lot of cash.
The last section is about building companies. This is after you have a product and a market, and are looking to take things to the next level. The advice here is more philosophical, illuminated by scenarios where founders were unable to lead that transition and the examples where traditional professional management ruined promising startups.
The ideal leader is someone with the passion and vision of a founder who is also able to think strategically and delegate. (which is not to be confused with abdicate…)
If you are thinking about building a business, or in the process now, I recommend you take the time to read ‘The Four Steps’ as soon as possible.
Infrastructure Renaissance July 12, 2009
Posted by stochasticresonance in Puppet, ramble.Tags: Code, Infrastructure, Virtualization
10 comments
Infrastructure is Code! FTW!
Wait… WTF does that even mean?
On the one hand this is a rallying cry for those of us in the configuration automation space, on the other hand it confuses innocent bystanders.
@zhenjl: @botchagalupe but ops ppl have always coded using perl, tcl (if u remembr that
…now we just have ’sexier’ langs like ruby and python (original)
Sure, there has always been scripts, which is technically code, but this is missing the point a bit.
For starters, getting ops teams to recognize and steal all applicable processes and tools from developers to produce and version there scripts would be a win, but that’s just the starting line…
The point isn’t just that you can write some scripts to help you do something with your infrastructure then exit.
The ‘Infrastructure’ IS an application, a long running process with inputs, outputs and state. Or at least infrastructure is very close to becoming one…
Everyone paying attention knows something is happening… what is it? Hazy, foggy, smoky, cloudy, whatever…
EC2 is the herald of an infrastructure enlightenment, leading us out of the dark ages.
(We’ll ignore the history of virtualization technology, and ESX Server for now, but I’m sure someone has an opinion on all this.)
The bottom line is there are resources, and I can manipulate them through an API. With EC2, these are compute resources of various shapes and sizes, with some storage resources.
Virtualized networking is not exposed through a public API yet, but we’re still climbing abstraction mountain together and we all know it is just a little higher in this cloud. (What’s up Cisco, where you at? Oh, there you are… and these guys too… thanks Ben )
At some point in the not too distant future, web service calls and a credit card will be enough to create and manipulate arbitrary compute, storage and network resources.
So now what?
Until someone provides Sysadmin as a Service, there is still quite a bit of work left to be done. These resources need to be configured to provide services.
When adding capacity was a purchase order, then wait for weeks to rack and cable, taking a day or so to configure the machines was not rate limiting, and lost in the noise. When you can bring up new server instances in minutes, that same configuration step is conspicuous waste.
You might be thinking you’ll make images for each of your services and manage those. This approach to management exchanges the configuration drift of tradition IT management for image sprawl. Do those notes that got scribbled six months ago when the web server image was created have accurate details about how that machine is configured? Are you sure? The only way to know is start the VM and inspect it. Frankly, the same thing applies to the running instances started yesterday.
That’s where tools like Puppet comes in. The benefit is not just that a machine gets configured by running a script. Puppet is an application that audits your system configuration, and changes the details out of sync with policy. That policy is code with semantics that can be developed like software.
That’s the shift. Instead of thinking about managing server counts, we can reason about abstractions of service configurations. The pieces exposed by an API are not just instances of server and storage resources, but abstractions of services. The configuration of proxies, monitoring, middleware, web servers and databases are components to be composed in the dynamic ‘infrastructure’ application.
This is just the beginning, but I don’t want to spoil the rest of the story… at least not yet.
It is not necessary to change. Survival is not mandatory.
W. Edwards Deming
You might be a cloud computing expert if… July 11, 2009
Posted by stochasticresonance in ramble.Tags: Cloud Computing, Hubris
3 comments
And let it be noted that there is no more delicate matter to take in hand, nor more dangerous to conduct, nor more doubtful in its success, than to set up as the leader in the introduction of changes. For he who innovates will have for his enemies all those who are well off under the existing order of things, and only lukewarm supporters in those who might be better off under the new.
Niccolò Machiavell
I’m taking a break from trying to be useful. You know how the song goes:
- “If the top three items in your browser history include the words “twitter”, “innovation” or “interoperability”, you might be a cloud computing expert”
- “If you’ve ever asked Amazon to Open Source their API, you might be a cloud computing expert”
- “If you think ‘Open Sourcing’ an API makes any sense, you might be a cloud computing expert”
- “If you think there is a difference between ‘business investors’ and ’speculators’, you might be a cloud computing expert”
- “If you think cloud computing will save the rain forest, you might be a cloud computing expert”
- “If you believe building in EC2 always has a lower TCO, you might be a cloud computing expert”
- “If you are not preparing for ‘The Trough of Disillusionment’, you might be a cloud computing expert”
- “If you just can’t get enough *aaS, you might be a cloud computing expert”
- “If you want to debate public vs private, you might be a cloud computing expert”
- “If you spend a day talking about which Star Wars character you should be and it is a good day at work, you might be a cloud computing expert”
- “If you understood more than two points on this list, you’re def. a fscking cloud computing expert”
I’m sure this list is much longer, I’m just trying to do my part and get you started…
Make us all proud…
meatcloud manifesto April 1, 2009
Posted by stochasticresonance in Puppet.Tags: Automation, Manifesto, Meatcloud, PITA
7 comments
GUI installers are evil…
Period.
You might need to have a little wizard to walk through setting stuff up the first time for the uninitiated, but if you think you are doing the poor hack who has to install your stuff on hundreds of machines any favors, you got another thing coming.
If you don’t provide some way to install your software fully unattended, please do not sell it to schools, businesses, non-profits or religious organizations with more than one person.
You might as well be killing kittens.
Actually, that goes for the interface to running applications as well.
I think we need the Open Meatcloud Manifesto… (because the world needs more manifestos and this is the perfect date to start a movement)
Let me see what I can work up…
- scribble* *scribble* *erase* *scribble* *crumple* *scribble*
I’ll keep it short and simple:

We hold these truths to be self-evident
Show your support for this movement by making your own copy of the manifesto and posting the pictures.
(Special privileges will be bestowed on those willing to tattoo the manifesto… on their forehead.)
No meatclouds were harmed in the making of this manifesto.
Viva La Revolución
Open For Business Follow Up March 29, 2009
Posted by stochasticresonance in Puppet.Tags: Business, Open Source
4 comments
After my ‘Open For Business‘ post, I had several conversations out of band.
A few people responded with ‘we feel your pain’.
That’s not really the point I was trying to make. This quarter is the best quarter Reductive Labs has ever had for revenue, and next quarter looks like it will be a even better as people both realize the value of Puppet automation and are seeking better ROI from their budgets.
Any new skill is going to come with some clumsy missteps as you are learning, and sales is a skill. We are technologists learning to run a business, and sales is part of it. Frankly, we offer a great value for where we are in the market and I don’t know who I feel worse for, the enterprise sales guys who are offering to buy each others licenses, the IT managers who are buying stuff their admins won’t be able to use or those admins. So while it might be a little painful in the ‘here we go again’ sense, that is really just part of the sales process, but not anywhere near as painful as this. (sorry, that was probably indulgent)
Another comment is that Puppet might be perceived as not being ‘Enterprise Ready’. Perception is reality, and some people still argue Ruby isn’t ready. Rails, not ready… How about Windows? Anyone ever read the bugs open against Oracle 10g? The fact is Puppet has several very successful implementations in Fortune 100 companies. Whatcha gonna do?
With big companies, the discussion sometimes breakdown when we get to culture and process. The problem isn’t the technology per se, it’s the fact that there are vertically and horizontally splintered fiefdoms running around vying for power and pride. Every group wants to do things their way. When the X application group has segregated systems from the Y application group isn’t so bad, but if the X group has to submit tickets to the monitoring group to find out what is happening on their own machines, implementing a centralized management solution becomes really difficult to navigate from the bottom up.

Maybe you should stop when it hurts?
Sometimes, I just want to tap out to the stupid.
(That was probably indulgent too…)
*shrug*
That does look painful…
Go ‘Enterprise’ go…
Open For Business March 27, 2009
Posted by stochasticresonance in Puppet, ramble.Tags: Business, Open Source
4 comments
What has never been doubted, has never been proven.
–Denis Diderot
Let’s be honest, no one ‘really’ knows exactly how ‘Open Source’ and ‘Business Model’ should fit together. There is a broad spectrum of ‘Open Source’ approaches, from the ‘Purists‘ to the ‘Open Core‘ to the pragmatic ‘Flip Floppers‘. The differences are subtle nuances to anyone not steeped in the culture and trying to explain them will just confuse most people not familiar with the lore. There is my partner Luke, who has as many questions as answers. Then there a people with a lot of answers, but of course, they are all different.
Analysts are talking about the ‘billion dollar opportunity‘, but the brightest example of Open Source business success, that everyone references, garners harsh comments like this one. Does there have to be a clash along this boundary? Sure, Redhat has essentially adopted a traditional licensing model, but if you don’t understand the benefit of the role they play in the larger OSS ecosystem then you aren’t really paying attention. Having strong ideals is one thing, paying for groceries, mortgages, and braces is another.
One of the most influential programmers of a generation is trying to figure out how to make a living. You think capistrano would be abandoned if Jamis got a nickel everytime someone ran cap deploy? These were innovative projects that added a ton of value to an ecosystem and captured little, if any. There are plenty of millionaires who sold total crap and cashed out.
Nowadays people know the price of everything and the value of nothing.
–Oscar Wilde
Is there some way to balance Open Source ideals with opportunity? I sure hope so, but I’m not entirely sure how the rest of the story goes. On more than one occasion, I’ve been on a call with someone from an ‘enterprise’ talking about how they are replacing Opsware or Bladelogic installations that were minimum high 6 figure checks to implement, wanting Reductive Labs services, and balking at spending ~$20K with us as too expensive. Someone convinced somebody that the promise of automation was worth X dollars, possibly didn’t deliver it, and now a solution is not worth X/50? What gives?
The problem in most cases is the technology has been adopted at the front line by admins who want to get more done, know there is a better way, but don’t necessarily have budget to spend. Their managers don’t know much about what they do, view all of IT as a cost center anyway and now they want more money… for what? Reductive Labs best customers are organization with a semi-technical manager, who trusts her people, wants them to succeed and believes IT should be a differentiator. Organizations like this have funded Puppet’s development.
The push back is often from companies with 10K servers and a 500 headcount meatcloud, all with root on every box. Some progressive admin has got Puppet up and solving small problems, but knows releasing it into the organization at large without some training and process improvements is like distributing chainsaws to every kindergartener on the playground at recess. No child left behind… So instead they fumble through implementing a solution, don’t really change the process, and then call us when Puppet brings down a service by doing exactly what their code told Puppet to do. (Ok, so I’ll admit, this doesn’t happen every time, but it has happened.)
The greatest obstacle to discovery is not ignorance — it is the illusion of knowledge.
–Daniel J Boorstin
John M Willis says he talks to the ‘enterprise’ all the time, and most people haven’t even heard of Puppet yet. I’m sorry, try to catch the next clue train. Managing 10K machines with a 500 headed meathydra rooting boxes is costing a lot more money than it needs to. We would love to solve that problem for you. If you aren’t looking for a solution to this, you probably aren’t solving other problems either. I understand that keeping up with every little technology is hard, but at some point you just have to concede that you aren’t in the race anymore. Hopefully, you have at least heard of Linux…
The problem is most the time the conversation stops at the admin technical level. We don’t have a small army in ties who can take executives for metaphorical (or not…) martinis at strip clubs. Or a marketing budget to take out full page ads in CIO magazine. In my world view that should be a good thing. We are technologist at heart, who want to build solid solutions and to deliver value at the point of attack. Unfortunately, we don’t have the sales process to exploit inefficiencies in the marketplace with asymmetric understanding and access to information (or an economic system that rewards value necessarily).
Somewhere between the ‘enterprise’ who doesn’t know what to make of open source software and the people who love open source like sharks like seals, is there an opportunity to capture value? Scylla and Charibdis?
I like to think that more and more people will realize that the price of Open Source is expertise, that they have to pay their team for months of learning we could have provided in a week, that a ’support’ contract is a reciprocal agreement, and the alternatives are much more expensive when comparing the total cost of ownership. If you pay us half of what a commercial solution would cost, we can automate your infrastructure, train your team and support the solution. In fact, we’ll make sure Puppet spreads jam on your toast in the morning. (If you pay us anywhere near the same amount, we’ll automate baking the bread too.)
Markets are constantly in a state of uncertainty and flux, and money is made by discounting the obvious and betting on the unexpected.
–George Soros
