Big Data and Warehousing

Getting Past the Big Data Hype—and Backlash

Why arguments at both ends of the spectrum are missing the point

I’ve also recorded a podcast on this topic—download it here.

There’s a commercial for a new TV making the rounds right now—one where images explode off the screen and envelop the people watching. Suppose you actually went out and bought this TV, but then you realized that the images didn’t literally jump off the screen the way they did in the commercial. Would you return your new TV?

Of course not. Watching the commercial, you knew it was both marketing and an overly dramatic illustration of reality intended to make a point. Chances are, the TV works just fine for your reasonable expectations.

But what does that have to do with big data? Like your response to the commercial, starting a big data project depends on having reasonable expectations and not diving headlong into hype. This is worth pointing out since we are now at the evitable point in the adoption cycle where we are seeing some churlish, defensive behavior from vendors that feel threatened by the trajectory of these new technologies.

A good example of this came up on one of the big data forums I moderate on LinkedIn. An individual practitioner asked if the space was overhyped (of course it is). This question was answered by Microsoft business partners, who ranted that there was no value in big data and that no firm should be considering it. They argued that big data is only useful with social media and that big data projects are a “bet the whole company on the outcome” sort of thing. These arguments are myopic, self-serving, or both.

Public discussion about big data has polarized people into two camps: one focused on overly positive hype, and—at the opposite end of the spectrum—excessively pessimistic naysayers. Is there too much hype about big data right now? Sure. But does it contribute to the discussion to throw around obviously wrong and uninformed arguments, either? No. Time for some perspective, I’d say.

What’s happening in big data right now is a classic example of a well-established technology adoption pattern. The usual pundits and media sources that need something compelling to write about are over-hyping certain aspects of the big data space. But remember: they did the same thing with Java, SOA, application servers, and business intelligence when these technologies were emerging. Last time I checked, those technologies all seem to have stuck—and their adopters did it by focusing on real problems and not trying to shoot the moon. Myth-based arguments against those technologies didn’t stand the test of time, and neither will the ones against big data technologies.

So how do you balance the discussion and look beyond the hype? As usual, it comes down to pragmatism. Get inspired and think big thoughts, but start by implementing modest projects. (Boil a bathtub, not the ocean.) If you can’t sketch a plan for ROI, stop. Reconsider your approach, and don’t do anything until the path to success—with the metrics to back it up—is clear. Ask for references, and demand experienced guidance from the ecosystem you decide to tap into—not just for big data, but for the overall system flows as well.

As I have written elsewhere, anyone who implements a significant big data investment based solely on media hype is making a major mistake—and anyone who rejects these technologies based on ill-informed backlash is equally misguided.

What do you think? Let me know in the comments.
 

 
Previous post

IBM Netezza Data Migration from a Mustang 4.6.12 Appliance to a TwinFin 1000-12

Next post

Information On Demand 2013 Call for Speakers Now Open

Tom Deutsch

Tom Deutsch (Twitter: @thomasdeutsch) serves as a Program Director in IBM’s Big Data Team. He played a formative role in the transition of Hadoop-based technology from IBM Research to IBM Software Group, and he continues to be involved with IBM Research Big Data activities and transition from Research to commercial products. Tom created the IBM BigInsights Hadoop based product, and then has spent several years helping customers with Apache Hadoop, BigInsights and Streams technologies identifying architecture fit, developing business strategies and managing early stage projects across more than 200 customer engagements. Tom has co-authored a Big Data book and multiple thought papers.

Prior to that, Tom worked in the Information Management in the CTO’s office. Tom worked with a team focused on emerging technology and helped customers adopt IBM’s innovative Enterprise Mashups and Cloud offerings. Tom came to IBM through the FileNet acquisition, where he had responsibility for FileNet’s flagship Content Management product and spearheaded FileNet product initiatives with other IBM software segments including the Lotus and InfoSphere segments.

With more than 20 years in the industry, and as a veteran of two startups, Deutsch is an expert on the technical, strategic and business information management issues facing the Enterprise today. Most of Tom’s work has been on emerging technologies and business challenges, and he brings a strong focus on the cross-functional work required to have early stage project succeed.

Deutsch earned a bachelor’s degree from Fordham University in New York and an MBA degree from the University of Maryland University College.

  • Carl Douglass

    100% on target!