Big Data’s Future: Bigger Than Just Vertical Applications

Why one popular prediction about the direction big data will take in 2013 is too limited

I’m not much for making predictions about trends in the upcoming year. Still, it sure is fun to see what others predict—and of course, to debunk some of the loopiest ideas about what the future may hold. For example, one of the most popular 2013 predictions out there right now is that the big data marketplace will be all about vertical applications in the coming year (read this article from VentureBeat for the full argument).

It’s important to remember the source of this prediction: the venture capital community. The big data platform space is already well served, so VCs can’t make much additional money there. It serves their interests to try and shift the conversation to an underserved space with greater possibilities for funding and generating a return. These circumstances don’t necessarily mean the prediction is wrong—but it is always a good idea to keep the possibility of other motivations in mind. Kind of like Oracle saying the answer to all of life’s challenges is Exadata… but I digress.

You may also be asking yourself another related question at this point: Don’t you, Tom, focus on big data infrastructure? Doesn’t that make you biased in this argument too? Well, I am partially guilty as charged. I do focus on big data infrastructure. But my platform also supports quite a few IBM vertical apps (and the number is growing), so I’ve got skin in the game on both sides here. I have no axe to grind either way.

I would argue that focusing solely on vertical applications for big data goes against what enterprises really need—and it contradicts the larger trend toward rationalization. Based on my work with customers, the fundamental flaw in the vertical-applications-only argument is that no enterprise zero-bases their infrastructure, which means these “new” applications are simply additive and therefore create additional support burden and user overload. Can they be useful? Sure. But are more standalone apps what the enterprise needs at this point? That seems dubious to me.

When the larger (and correct) IT drive is towards rationalization, holistic understanding, and Fit for Purpose Architectures, simply introducing more siloed applications and getting frustrated over the additional information silos that you create is counterproductive. In most cases, it’s better to use the capabilities of your big data platforms to improve existing solutions instead of simply replacing them. Rather than creating a new application environment, you can use your big data infrastructure to eliminate what doesn’t work or doesn’t fit into your existing core decision support systems. You can then ship data to those environments so they pick them up as “native.” This approach is much more user-friendly for business users—and it dramatically shortens deployment cycles, drives utilization, and often is the lowest-TCO path available.

To be clear, I’m not saying that new vertical apps don’t have a role to play. They can certainly simplify deployment and improve ease of use. But the claim that vertical apps are the future of the big data space is to fundamentally miss the larger enterprise IT picture.

Do you agree or disagree? Let me know in the comments.

Previous post

Meeting the Challenges of Managing the SAP Application Lifecycle

Next post

Large-Scale Data Management: Part 3

Tom Deutsch

Tom Deutsch (Twitter: @thomasdeutsch) is chief technology officer (CTO) for the IBM Industry Solutions Group, and focuses on data science as a service. Tom played a formative role in the transition of Apache Hadoop–based technology from IBM Research to the IBM Software Group, and he continues to be involved with IBM Research's big data activities and the transition from research to commercial products. In addition, he created the IBM® InfoSphere® BigInsights™ Hadoop–based software, and he has spent several years helping customers with Hadoop, InfoSphere BigInsights, and InfoSphere Streams technologies by identifying architecture fit, developing business strategies, and managing early stage projects across more than 200 engagements. Tom came to IBM through the FileNet acquisition, where he had responsibility for FileNet’s flagship content management product and spearheaded FileNet product initiatives with other IBM software segments, including the Lotus and InfoSphere segments. Tom has also worked in the Information Management in the CTO’s office and with a team focused on emerging technology. He helped customers adopt innovative IBM enterprise mash-ups and cloud-based offerings. With more than 20 years of experience in the industry, and as a veteran of two startups, Tom is an expert on the technical, strategic, and business information management issues facing the enterprise today. Most of his work has been on emerging technologies and business challenges, and he brings a strong focus on the cross-functional work required to have early stage projects succeed. Tom has coauthored a book on big data and multiple thought-leadership papers. He earned a bachelor’s degree from Fordham University in New York and an MBA degree from the University of Maryland University College.