Big Data and Warehousing, Databases

Capitalizing on Big Data with System z

Integrating IBM DB2 for z/OS and IMS databases with big data solutions

For many organizations, the IBM® System z® platform holds the mission-critical information that forms the basis of key decisions. It might not be surprising, then, that more and more organizations that use System z are moving analytics and decision engines closer to the rich source of trusted information held in their databases and data warehouses. Research from IBM and industry analysts shows that big data technologies are being used primarily to analyze transactional data from enterprise applications. Telemetry and social media data, while important, are often secondary sources used to augment insights from transactional data.

 

Making big data a reality for System z organizations is about integration

IBM DB2® for z/OS® and IBM IMS™ databases are at the core of many organizations’ applications and processes. DB2 in particular is central to many data warehouse and master data management (MDM) solutions. But how do these databases interact and integrate with all that “other” data?

Let’s consider ways in which these two worlds could integrate:

  • DB2 for z/OS users could use a standard Hadoop Distributed File System (HDFS) query language such as Jaql, Hive, or Pig to invoke a predefined HDFS query. Once completed, DB2 for z/OS is notified. The SQL job finds the results of the HDFS query on the IBM InfoSphere® BigInsights™ platform (or other implementation) and populates a DB2 for z/OS table in a specified format.
  • The emerging MapReduce extension to SQL holds promise to abstract differences between the traditional database and Hadoop environments so that when the query is submitted, DB2 for z/OS can recognize the MapReduce commands and offload the HDFS portion of the query to InfoSphere BigInsights. DB2 can then retrieve the results and populate the target database tables.
  • Taking this one step further, giving InfoSphere BigInsights access to the DB2 catalog would facilitate greater interchangeability. Hadoop implementations could then understand how DB2 databases and tables are configured and structured, and know where and how to store the results of queries with much less manual intervention.

IMS takes a similar approach to the one above, calling and invoking big data services as necessary but also using the InfoSphere BigInsights V2.0 machine data analytics accelerator to ingest, parse, and extract a variety of machine data from sources such as log files, smart devices, and telemetry. By using the machine data analytics accelerator, organizations gain insights into operations, customer experiences, transactions, and behavior. The resulting information could be used to proactively boost operational efficiency, troubleshoot problems, investigate security incidents, and monitor end-to-end infrastructure to avoid service degradation or outages.

 

Plug-and-play velocity

With the massive volumes and variety of data held in System z databases and data warehouses, and potentially more information flowing into the platform from other sources, organizations want answers and more insights faster. The IBM DB2 Analytics Accelerator for z/OS enables queries to return answers in seconds and minutes—instead of hours and days—without any changes to the existing infrastructure or applications. The DB2 for z/OS optimizer code decides whether to run the query locally or offload it to the accelerator.

 

Conclusion

The System z platform provides key qualities of service to support organizations’ big data initiatives, offering ultimate security, availability, scalability, integrity, performance, and extensibility. System z can serve as a big data hub, holding mission-critical data and offering best-of-breed business analytics solutions and open integration. It can also be the ultimate consolidation platform, managing data held natively on z/OS, Linux on System z, or other environments through the IBM zEnterprise® BladeCenter® Extension (zBX) as part of a single logical virtualized environment using IBM zEnterprise Unified Resource Manager.

To find out more about how System z is being used by organizations for their big data initiatives, watch this short customer testimonial from Banca Carige.

Please feel free to post questions or comments for me below.

Previous post

Assessing the Impact of Big Data

Next post

Going Cloud with Your Big Data: A Structured Approach

Mark Simmonds

Mark is an enterprise architect and senior product marketing manager in the IBM Software Group Information Management division, where he focuses on information governance and big data for the System z® portfolio. He has 16 years IBM service in information management, WebSphere and spent 3 years as an IBM systems architect responsible for infrastructure design and corporate technical architecture in large financial institutions. Prior to joining IBM, he was head of information and IT for an organization in the National Health Services (Wales, UK). He has a number of author recognition awards and has written articles for technical journals and business magazines.