Landfall Mashups

Earlier this year, I was really impressed with stormpulse, a mash up that provides nearly real time visualizations of hurricanes and the expected trajectories (as a probabilistic band). The whole thing is driven by public data and in addition to being really cool if you are in Salt Lake City, is probably even cooler if you need to decide to leave before landfall.

After the Ruby Hackfest (which really turned into an intro to machine learning and classification), I kept thinking about what data sets might be interesting to train SVM. At the same time, rolling around in the back of my brain was the auto-scaling discussion that burned a swath across the blogo-tweeto-sphere. Those two things collided in the background and prompted this post. (The best analysis and something I mention to Adam Jacobs in a side channel is that these applications don’t scale linearly and naively throwing hardware at the problem without analysis can bring an application to it’s knees. Adam thinks it is all about the appropriate apologies, which will probably never change…)

But I digress…

One of the points brought up in the original post is that auto scaling based on monitoring doesn’t have time to react, but that is like noticing there are 80 mph winds and surmising that their might be a hurricane. I contend that the ‘Digg/Oprah/Slashdot’ effects have some characteristic patterns which could be easily characterized and the trajectories of the ‘storms’ could be predicted, if we could only get the data.  The data is all there, it’s just not together and not publicly accessible. Maybe there is some way to get people to donate access logs so we can characterize the curves?  Once you have those characterized, take the model and analyze the real time logs and feed that info back into both the model and an autoscaling engine.  Start to bring up machines before the big waves start to break…  Take it a step farther and get a protocol/etiquette for a track/pingback mechanism, just to let you know there might be a tropical storm to keep an eye on…

Just a little idea… seems pretty straight forward…

Is anyone out there trying to do something like this?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: