Getting Past the Big Data Hype—and Backlash

Why arguments at both ends of the spectrum are missing the point

I’ve also recorded a podcast on this topic—download it here.

There’s a commercial for a new TV making the rounds right now—one where images explode off the screen and envelop the people watching. Suppose you actually went out and bought this TV, but then you realized that the images didn’t literally jump off the screen the way they did in the commercial. Would you return your new TV?

Of course not. Watching the commercial, you knew it was both marketing and an overly dramatic illustration of reality intended to make a point. Chances are, the TV works just fine for your reasonable expectations.

But what does that have to do with big data? Like your response to the commercial, starting a big data project depends on having reasonable expectations and not diving headlong into hype. This is worth pointing out since we are now at the evitable point in the adoption cycle where we are seeing some churlish, defensive behavior from vendors that feel threatened by the trajectory of these new technologies.

A good example of this came up on one of the big data forums I moderate on LinkedIn. An individual practitioner asked if the space was overhyped (of course it is). This question was answered by Microsoft business partners, who ranted that there was no value in big data and that no firm should be considering it. They argued that big data is only useful with social media and that big data projects are a “bet the whole company on the outcome” sort of thing. These arguments are myopic, self-serving, or both.

Public discussion about big data has polarized people into two camps: one focused on overly positive hype, and—at the opposite end of the spectrum—excessively pessimistic naysayers. Is there too much hype about big data right now? Sure. But does it contribute to the discussion to throw around obviously wrong and uninformed arguments, either? No. Time for some perspective, I’d say.

What’s happening in big data right now is a classic example of a well-established technology adoption pattern. The usual pundits and media sources that need something compelling to write about are over-hyping certain aspects of the big data space. But remember: they did the same thing with Java, SOA, application servers, and business intelligence when these technologies were emerging. Last time I checked, those technologies all seem to have stuck—and their adopters did it by focusing on real problems and not trying to shoot the moon. Myth-based arguments against those technologies didn’t stand the test of time, and neither will the ones against big data technologies.

So how do you balance the discussion and look beyond the hype? As usual, it comes down to pragmatism. Get inspired and think big thoughts, but start by implementing modest projects. (Boil a bathtub, not the ocean.) If you can’t sketch a plan for ROI, stop. Reconsider your approach, and don’t do anything until the path to success—with the metrics to back it up—is clear. Ask for references, and demand experienced guidance from the ecosystem you decide to tap into—not just for big data, but for the overall system flows as well.

As I have written elsewhere, anyone who implements a significant big data investment based solely on media hype is making a major mistake—and anyone who rejects these technologies based on ill-informed backlash is equally misguided.

What do you think? Let me know in the comments.

Previous post

Database-Driven Websites: Using PHP with Informix

Next post

If Hadoop Was Easy, Everyone Would Be Doing It

Tom Deutsch

Tom Deutsch (Twitter: @thomasdeutsch) is chief technology officer (CTO) for the IBM Industry Solutions Group, and focuses on data science as a service. Tom played a formative role in the transition of Apache Hadoop–based technology from IBM Research to the IBM Software Group, and he continues to be involved with IBM Research's big data activities and the transition from research to commercial products. In addition, he created the IBM® InfoSphere® BigInsights™ Hadoop–based software, and he has spent several years helping customers with Hadoop, InfoSphere BigInsights, and InfoSphere Streams technologies by identifying architecture fit, developing business strategies, and managing early stage projects across more than 200 engagements. Tom came to IBM through the FileNet acquisition, where he had responsibility for FileNet’s flagship content management product and spearheaded FileNet product initiatives with other IBM software segments, including the Lotus and InfoSphere segments. Tom has also worked in the Information Management in the CTO’s office and with a team focused on emerging technology. He helped customers adopt innovative IBM enterprise mash-ups and cloud-based offerings. With more than 20 years of experience in the industry, and as a veteran of two startups, Tom is an expert on the technical, strategic, and business information management issues facing the enterprise today. Most of his work has been on emerging technologies and business challenges, and he brings a strong focus on the cross-functional work required to have early stage projects succeed. Tom has coauthored a book on big data and multiple thought-leadership papers. He earned a bachelor’s degree from Fordham University in New York and an MBA degree from the University of Maryland University College.

  • Carl Douglass

    100% on target!