I attended my first Hackathon last Friday. It was sponsored by Factual , an operating system of sorts for data. They collect large amounts of publicly available data (they are best known for their local data) and make it available to developers to build upon. The purpose of the night was to inspire developers to use Factual's data to come up with new and interesting applications.
While most media is abuzz with the dangers of data, the reality is data is everywhere and the spirit of the event was about finding ways to put it to work.
And that's not as easy a task as you might think. There are all sorts of issues when working with data, not the least of which is access. Some organizations and companies make data accessible and some do not.
But even with access, data is tough to wrangle. A good example is a database of local restaurants. Anyone trying to be comprehensive would have to understand the many different ways that say McDonalds is spelled in order to pull in the right data. It's an unavoidable fact that the quality of data is only as good as the data entered or said another way, garbage in, garbage out.
The night was educational and interesting. I was delighted to find I wasn't the only woman there (though collectively we were about 1% of the audience) and intrigued by side conversations that covered the inevitable reference to Ray Kurzweil and singularity. But in the end I was struck by what or rather who was missing in the crowd: scientists.
I have no doubt that big data is the next new frontier in the Valley. I'm definitely not alone in believing that data systems will act like operating systems, but the difference between data operating systems and say Windows is that it will take decidedly different skills to build applications on top of data systems.
That's where scientists come in. Most engineers can build systems for collecting, parsing and even displaying data, but very few also have the statistical, scientific and creative skill to glean insights from the data. Insights that ultimately give value to the data.
You think it's hard to find engineers in the Valley? Just wait and see what the demand will look like for scientists.
Get the facts from Alicia at www.AliciaMorga.com