Time To Build Your Big-Data Muscles

As data grows fast in volume, variety, and velocity, our capacity to consume, analyze, make meaning and act on data lags far behind.

What if I told you that for every 100 open Big Data jobs, there are only two qualified candidates? I’d like to tell you that. I heard it from a senior leader grumbling about finding adequate prospects. When I asked for the data, he couldn’t find the source and didn’t have quite that many open reqs (though he had many). 

In this data deluge era, we’ve grown accustomed to people almost having data to support their assertions. Telling you that the 100:2 ratio feels right doesn’t hold the same weight as evidence in hand. If you had to make a sound decision, you might search for something more.

McKinsey Global Institute gauges that by 2018, the United States will create 290,000 to 340,000 new big data jobs and 140,000 to 190,000 (more than half) could go unfilled because skilled candidates are in short supply. Add to that a need for at least 4 million managers across all fields with the know-how to decipher and act on the patterns in big data to make effective choices.

Every senior leader dreams of the day he or she can analyze a pile of data so big and harnessed so well that it becomes possible to make substantially better predictions.

In IBM's 2011 Chief Information Officer survey of 3,000 global companies, more than 83% of respondents identified business analytics from big data as a top priority and a way in which they plan to enhance their competitiveness.

The data comes from everywhere: transaction records of online purchases, physical plant sensors, rules and regulations, posts conveying sentiment across social networks, uploaded digital pictures and videos, company and industry research reports...the list goes on. To put that in context, 15 out of 17 industry sectors in the U.S. (such as manufacturing, insurance, healthcare, utilities) have more data stored per company than the entire U.S. Library of Congress.

With data growing in volume, variety and velocity at massive rates (by some estimates, increasing another 44x by 2020), technology advances aim to extract deeper insights. Beyond the 3 Vs, our human capacity to consume, analyze, make meaning, and act on the data lags far behind. 

Big data creates new professional categories, job opportunities, and careers. Yet few universities are paying serious attention to the analytic skills needed by graduates in all fields. Academic programs need to provide opportunities for college and graduate students to practice and hone their analytics abilities and understand what big data means across disciplines. 

Terri Griffith, professor of management at Santa Clara University and author of The Plugged-In Manager, sees an opportunity for students to gain a market advantage when starting their careers. For every call Griffith gets from recruiters looking for the school's best and brightest, she hears from three specifically asking about analytics capabilities.

Introducing big data approaches proves challenging because colleges are as siloed as most companies. Analysis and application usually reside in different schools and programs. Computer science students might learn about predictive analytics and Hadoop, while marketing students may never even hear the terms. "What is marketing today if not an opportunity to apply linear regression on the big data that comes out of experimental design? You learn from that data which source signals the type of attention you seek," says Griffith.

Tim Sae Koo, cofounder of Hypemarks, a new social discovery aggregator, was challenged in his USC Management of New Enterprises course to think big. He took that to mean crossing fields of study. Sae Koo visited the computer science department and asked what trend the professors thought would eclipse all others in years ahead. Several said big data. As a result, Sae Koo made what he calls a "Big Data play, capturing what people are interested in by aggregating what they content consume and collect on the web. From the data, we can gauge what type of content has your attention and what you’re committing time to learn more about." Quoting Wayne Gretzky, who said he was so good because he skated to where the puck would be, not where it had been, Tim believes that’s big data and people that it touches. 

Yale’s Ravi Dhar, Professor of Management and Marketing, and director of the Yale School of Management’s Center for Customer Insights, arrives someplace similar. When speaking with a senior executive about the leaders of tomorrow, they have always expected communication and curiosity. Now they add, "handling data." They don’t have people in traditional functional areas who are good at all three. Dhar points out, "In marketing, people are great at crafting tag lines, branding, and launching campaigns. Now they need to have a deep understanding of customer too. That requires connecting all the dots." 

Likewise, the people who have historically looked at data weren't good in traditional lines of business—or conveying that to other people in actionable ways. "There is a disconnect between the ability to collect data and the ability to base decisions on them," says Eric Bradlow, professor of marketing at the University of Pennsylvania’s Wharton School and co-director of the Wharton Customer Analytics Initiative. "People need to take a deep breath. They need to be more thoughtful about it." Data will not answer questions by itself. People need to be able to communicate effectively about the findings, linking analytics to key decisions and the bottom line. 

As part of IBM’s academic initiative, Dhar’s Center for Customer Insights runs a 10-15 hour course spanning 12 weeks where students work closely with corporate management teams, answering hard questions. Students get exposure to real-world clients and the ability to experiment and apply new technologies to business challenges. IBM is working with more than 200 academic organizations around the world specifically in the area of expanding and strengthening analytics curricula, merging IT and business skills.

This approach is far more useful for students’ long-term success than market intelligence or market research courses that look at only quantitative 7-point scales and "how much more do you like" type questions, says Dhar. The vast amounts of sentiment data available today capture more naturally how people talk about their preferences, whether it's on an Amazon review or through their personal activity feed. Facebook and Twitter alone generate 17 terabytes of data every day. The analytics skills of today and tomorrow can focus on the right methods for analyzing people's priorities, not just those easiest to gather.

Griffith encourages students majoring in math, engineering, or computer science to answer strategic questions. She reminds them, "You’re an expert in analysis itself. Now get your head around the social science. Ask yourself, ‘How might this predict human behavior?’ "I don't want to teach red ball, blue ball," she says, referring to a probabilities question often asked in business schools. "I want to talk about the types of real problems business leaders face." She encourages students to flex their muscles. No matter their major, they ought to consider how available data can serve evidence-based management.  

Think you're too old or too busy to return to school to build and flex big data muscles? Look at the strengths you have, born from study and experience. Consider how they should adapt to what business needs. 

If the idea of big data and analytics scare you, read Moneyball by Michael Lewis this summer. While the Brad Pitt movie gives you the gist of how numbers can change something fundamentally, the book’s a real treat for even people who were never liked math. 

If you’re not scared by the technology, rather you’re short on time or money, consider Dataversity or Big Data University, which offer introductory and technical programs. 

And if you understand the numbers, but struggle with reframing your skills for other people to use: Read any of Ed Tufte’s books on enlivening information, Beautiful Evidence being the most recent. Watch Hans Rosling’s TED talk on turning statistics into inspiration. Ask the people around you, struggling to find veracity amid the other three Vs, what would help them succeed.

The deluge will continue until more of us become comfortable with the data science, know what questions to ask ourselves and those we work, then stand strong.

Marcia Conner consults with the world's largest organizations on getting better at getting better. Learn more about her work at MarciaConner.com. Follow twitter.com/marciamarcia for insights on business culture, organizational health, and a big data approach.

[Image: Flickr user Matthias Weinberger]

Add New Comment


  • Jamie Beckland

    This piece articulates the precarious balance between data and value. Clearly, there is a gap, and we can bridge it from many directions.

    But, one question remains unasked: What questions are we asking of our data?

    You can have billions of data points about baseball, but they are not going to help you make a better souffle.

    Figuring out what questions to ask of data is the toughest challenge we have right now - and we don't have good models for better ways to ask questions. We still rely on business unit managers to think up questions, then go to the data for the answers.

    What if it were the other way around? What if the data itself showed us what questions to ask?

    That is where Janrain is working with social profile data. By traversing the social profile data of customers, we are questioning long-held assumptions held by marketing teams. Which customers are the most profitable? How does engagement relate to purchases? How do these results influence ad spend, and even creative?

    These questions sometimes feel like the tail wagging the dog - but the truth is the opposite. We finally have the ability to respond to real users, in real time.

  • Marcia Conner

    Excellent point, Jamie. We need to develop new literacies around how to deal with the data and how to ask the right questions. Where the tools themselves can coach us through that, all the better. We just can't convince ourselves that the data, without some human analysis, has all the answers.

  • jeff beddow

    My first exposure to really Big Data was back in 1988 when I asked NASA for some satellite sensor data that would have been affected by the onset of the ELF radio antenna farms in the Great Lakes area.  They sent me a few computer tapes with 140 megabytes of data each...which seems trivial now, but I was working on a MAC Plus with 1 meg of RAM and 20 megs of HD.  I had to modem into the VAX at Goddard to get a smaller file for my purposes, but I kept the tapes around for years as a souvenir.

    In 2004 I put up a screen with 140,000 discrete data-driven and interpreted graphic points, each of which  could be interrogated by a pointer for specific data quantities. That taxed my P4 processor with 1 gig of RAM but it managed to do the statistics and transforms and get the display up in a few seconds.  That was the most data I had run in a single batch for a single screen display to that time.

    Today I was downloading some old Twitter sets that ran to hundreds of millions of rows of data.  My 4 core laptop with 8 gb of RAM can do fairly robust statistics in R on CSV tables with millions of data points in  a reasonable amount of time.

    But even as these quantities seem to scale up geometrically, the basic problem of what questions to ask, and how to frame them to a machine, and how to apply the answers, do not change.

    They really haven't changed much since the time of Heraklitos or Buddha, for that matter.  The answers aren't the problem.  The problem is the problem, I have heard it said.  By some Tibetan guy, around 1600 C.E.

    So the irony that a classical education which focuses on the human limits to understanding, communication and civility is more valuable in the question-crafting biz, is not lost.  It just gets misplaced every dozen years or so.

  • Marcia Conner

    Your story from 1988 evoked a memory flurry of the data I was working with that same year. In some ways it seems like so long ago, and in other ways just a second ago. Imagine if we could have predicted our current predicament back then.

    While writing this post I found so many mind-bending figures about just how much data we have now that I dedicated an entire post on my own blog to those stats (http://marciaconner.com/blog/d....

    Thank you for refocusing on the questions, as Jamie did as well, and reminding us not to misplace the real questions any time soon.

  • Abonnabad

    While I understand that demand for data analysts will only continue to grow, it seems that students will only rise to meet that demand once accredited degree programs become more widely known and more easily accessible. The culture of education (especially in America) insists that education only really counts if you walk way with a piece of paper saying you achieved a series of requirements that grant you some official expertise. Taking a few courses on Hadoop online at Big Data University is hardly equivalent to a Masters of Science in Data Analysis. Are these sorts of degrees available or forthcoming? And if so, where??

  • Jeff Hurt

    I see it differently Abonnabad. IMO, the culture of education has greatly shifted to a demand on learning that is relevant in the real world and too a profession. There is great debate that some higher education institutions are no longer preparing students for the real world and that the piece of paper you mention is not enough. There are too many students that graduate with that piece of paper and do not know how to apply their college learning to the real world.

    A masters of science in data analysis will not have immediate impact for organizations that are looking for ways to use big data now. The other alternatives that Marcia recommended will. Until students start graduating knowing how to immediately apply mining big data, companies will be looking for short, interim education opportunities to help staff get it now. That's the courses on Hadoop and other online opportunities.

    You're placing way too much credit and value into the traditional education institution that needs to shift into 21st Century Learning.

  • Marcia Conner

    Abonnabad, Thanks for asking. Some students may not realize the importance of this trend until more schools develop degree programs in analytics and big data, as you said. Thankfully, more schools are beginning to take a more formal approach to the topic with Northwestern and NC State already offering programs and Michigan State beginning one in January 2013. 
    In the meantime, every school can begin taking a big data approach throughout other degrees. When students have an opportunity to talk about how data is influencing whatever they are studying, they are likely to begin realizing they should consider paying closer attention and thinking about how this will influence their career.

    Jeff, I whole heartedly second your response. We need to look for ways to strengthen and broaden all education opportunities including those in schools, those from places like Dataversity and Big Data U, and also on the vast array of opportunities to relate available data with every job. Taking a big data approach is a 21st century literacy everyone needs.