Foresight Is 20/20: Predictive Analytics And The Business Of Certainty

Want to make really smart decisions for your company? It's simple as looking into the future and assessing the data—a service that a few young companies dealing in "predictive analytics" are selling.

Business swims in a sea of data. No matter what line of work you’re in—publishing, healthcare, retail, food service, drug-dealing—your business operates in a vast, swirling matrix of information. How many people are interested in your product? When do they show most interest? How much of your product should you produce? Where, when, and how should you sell it?

Data crunching giants like IBM and SAP have long mined and analyzed data for businesses, offering predictions about what actions will help their bottom line—"predictive analytics," to use the industry buzzword. But the cost of such services has typically been so high—a price tag in the millions, if consulting services are added, is not altogether unheard of—as to limit their use to major corporations. Now, for the first time, a few smaller players are promising small and medium-sized businesses that they, too, can use predictive analytics to better forecast what actions will yield a greater profit.

Take, for instance, web publishing. Web publishers have traditionally been content with "real-time analytics"—that is, information about how readers are interacting with their site now. But as vast stores of this data accumulate, there’s no reason why publishers can’t demand to know more. They should be able not only to observe the present, but to make guesses about the future.

Dennis Mortensen is the CEO and co-founder of Visual Revenue, a service that claims to do just that. "We created a predictive analytics platform that can assist editors in deciding what to promote on their home page," he tells Fast Company. If you think about it, he says, a webpage is like a marketplace. "Every single position on that page, there’s a cost to that, and you should expect a proper return on investment."

He gives the example of an editor who thinks a story is strong, and showcases it prominently; the article subsequently gets higher traffic. "It looks like the decision you took was correct," says Mortensen. "But that’s just a self-fulfilling prophecy," he says—of course promoted stories fare better. "What you don’t know is, was there another piece of content that could have done better?" Would a more substantial article have spurred deeper engagement with your site, prompting a reader to dig deeper into your archive? Visual Revenue makes predictions about how it thinks your content will fare and what sort of return on investment you’ll get out of it, down to the dollar and cent. VR’s own services start at about $1,000 per month, though price varies based on volume.

Scott Cohen, Digital Executive Editor of the New York Daily News, who has been programming front pages of websites for seven years, says of Visual Revenue that "we rarely make a move without consulting it." Over the years, he encountered editors who were skeptical that some algorithm could help with editorial decisions: "'That's Big Brother. That's my job. I don't need something like that to tell me what's working,'" they'd say. "Well you know what, you do," he counters. He finds himself more trusting of VR's real-time analytics, but says its predictive suggestions are more often right than wrong, calling it "radical to have a predictive engine as successful as it is." (Disclosure: Fast Company uses Visual Revenue, among other analytics services.)

Web publishing is one thing, but isn’t the messy world outside the Internet too complex to forecast the future? Not so, says Richard Daley, CEO of Pentaho, a business intelligence company offering predictive analytics. He ticks off the sectors from which he draws his clients: "We’re cross-industry, in finance, retail, manufacturing, a lot of web 2.0, healthcare, government..." In every sector, companies find that they have "tremendous amounts of data, tremendous value locked in the data, and are currently unable to get it out, either because of complexity or cost." For years, Daley says, companies were mostly happy just to be accumulating data for reports, charts, and graphs, which were used to create accountability. But increasingly, companies want more out of their data. "Eight years ago, just being able to collect data...was the primary interest," agrees John Bates, of Adobe's SearchCenter+. But now that so much data is in hand, "people want to optimize for the future, to forecast more accurately," he says.

Is soothsaying really necessary to optimally run a plant—can’t you just see if the rubber in a rubber factory is running out? "It gets down to mass quantities of data," says Pentaho's Daley. "If you’re only making 100 products, it’s pretty easy to say, ‘OK, I can look into the manufacturing floor and see the cupboards are bare.’" But a growing modern business often juggles countless variables; some may hire dozens of people "who do nothing but plan around manufacturing, inventories, materials sourcing"—weighing sales data, seasonal data, competitive data, data on the placement of retail locations. It’s not that these problems can’t be worked out by humans in principle; it’s just that the amounts of data involved make it wildly impractical.

Pentaho’s services aren’t exactly cheap, running some 40 grand a year. But their competitors—folks like IBM, SAP, and Oracle—often charge up to five times as much, says Daley. (Some companies offer a range of prices, depending on what services are contracted; Adobe's Bates told me that while some clients might spend thousands of dollars per year, others could spend "millions.") Several factors have enabled Pentaho and its ilk to bring down costs: Daley attributes it largely to Pentaho’s use of open-source code and a pared-down sales model. But Michael Kaushansky of Havas Digital, which offers predictive analytic services to marketers, says that "the fact that servers are cheap now" has also enabled upstarts to being to disrupt predictive analytics: storing and crunching data just doesn't cost what it used to.

That, in turn, has enabled a blooming of lower-cost data-crunching companies, each of which tend to specialize in a particular realm: a company like Prosodic for social media analytics, say; a company like MyBuys for online retailers. "Most, if not all, online retailers and direct brands who offer a large number of products stand to reap significant benefit from predictive analytics," says MyBuys's Craig Peasley.

The cost-benefit equation certainly looks increasingly favorable. "With the affordability we bring to the table, it’s not just big financial institutions with 20 billion dollars in annual sales who are able to take advantage," says Pentaho's Daley. "The SMBs [small and medium businesses] of the world are now able to have that same kind of competitive advantage that the big titans have."

By the same token, the big titans of predictive analytics itself—the IBMs and the SAPs of the world—may have to watch their back for the up-and-coming Pentahos and Visual Revenues. Were they able to predict that?

[Image: Flickr user gerlos]

Follow Fast Company on Twitter. Email David Zax, the author of this post, or follow him on Twitter.

Add New Comment

6 Comments

  • BR Deshpande

    Taleb's gripe against predictions in the Black Swan apply universally - however, it is a matter of degree. Some systems are a bit more predictable than others. This article demonstrates the subtle effects of mixing correlations and causations http://www.simafore.com/blog/b...

    This is also the same point which Trevor makes. Unfortunately, today there is a huge hype surrounding predictive analytics and important flaws/limitations get swept under the wave. See here http://www.simafore.com/blog/b... for a balanced view.

  • Trevor Miles

    I agrtee with Alec, though I will make a distinction between stochastic predictive analytics - the topic of this blog - and deterministic predictive analytics.

    Stochastic Predictive Analytics - The key point here is that correlation is NOT the same as causality.  And, as Alec write, the 'Black Swan' shows that this method fails when it matters most.  A correlation is useful only in very narrow conditions close to that in which the correlation was detected. 

    Deterministic Predictive Analytics - The key here is that you do know casuality so that you can determine consequence from effect.  This requies more effort to set up, the quality of the results are vastly superior.

    Admittedly the are systems and environments in which causality is difficult to establish, in which case stochastics predictive analytics may be all that is available.  But NEVER assume correlation is equal to causality and check the efficacy of the model frequently.

  • Tom Fuyala

    Good article David. At 11Ants Analytics (we build software that allows out of the box deployment of predictive analytics) we are finding that even larger companies often find it difficult to execute on their predictive analytics vision. There are plenty of examples of installations that cost hundreds of thousands to install, that sit idle because the specialist staff that ran it got a job elsewhere. The move to faster, easier to deploy, more accessible software is not unique to predictive analytics - but certainly is finding  a lot of proponents in this space. The easier software exists, the challenge we see now is in educating people in data preparation so that more people can fully exploit the easier to use tools.

  • Alec

    I can see this working in the editorial context used above but for anything else? Read "The Black Swan" and he'll explain in a very empirical way why this prediction method is hogwash. You can't use past data to predict the future with any accuracy. 30 days maybe but beyond that, there's no way. I mean, how many statisticians were "mathematically" positive a month ago the Red Sox were going to the playoffs?

  • Jamil Buie

    I have been curious about predictive analytics for small business for some time. The only hiccup is how would you apply it creative crafts. Demand for services and hard good have data points that can't be found (at least i have not figured out how) in creative crafts.