In Part 1 of this article, we explored why some organizations—such as high-value arbitrage traders working for large investment firms—need ways to accommodate time-series data types in their databases. And we highlighted some of the key capabilities that made IBM® Informix® TimeSeries especially popular with those organizations.
Informix TimeSeries was first completed and deployed to selected clients around the time Informix was acquired by IBM. The clients who adopted Informix TimeSeries were very pleased with its capability and performance, and word spread in the world of high-value trading operations among investment banks. IBM enjoyed a mini-boom in Informix TimeSeries sales.
The team looked around for other potential markets with similar requirements; that is, tens of thousands to hundreds of thousands of time-series transactions per minute, fast data acquisition, and smart operators to work on all that data. In 2001, there weren’t many other fields with these requirements.
Skip forward to today, where things are very different. IBM is helping to build a Smarter Planet™ by enabling large and small clients to capitalize on the increasing instrumentation of numerous facets of everyday life. Organizations are using sensors to monitor traffic at intersections, regulate temperatures throughout a building, and track consumer goods with RFID technology. The ubiquity of sensor-created data is leading to the creation of entirely new categories of applications.
Along the way, IBM noticed that there are massive volumes of time-series data being created—and our customers are noticing too. Several who are in the process of building very large smart-meter systems say that they’ve “hit the wall” with the ability of their normal relational database to handle the quantity of data the new smart meter systems are pushing.
What do the numbers look like? Typical smart-meter systems sample data every 15 minutes. So in a typical month, there are now 2,880 meter readings, compared with the former standard of one reading a month. Any time you increase the amount of data that is flowing into a database by almost 3,000 times, you challenge the system and likely exceed its specifications. The system obviously has multiple dimensions: hardware power, software capacity and speed, network throughput, storage capacity, backup strategy, and so on. But at its core, the software architecture must be capable of supporting this data volume. As was the case with high-volume trading systems not long ago, the existing software used for smart-meter systems is often not a good choice for the new workload.
Many early adopters of smart meters realized this, including individual utility providers and the vendors selling the applications that deal with smart-meter data—the Meter Data Management Systems (MDM or MDMS). The database software that had successfully supported the existing once-a-month meter reading paradigm was clearly insufficient and needed to be upgraded.
How have these organizations addressed the challenge? Two approaches have predominated: the first is simply to throw hardware at the problem. Most enterprise relational database management systems (RDBMSs) are highly scalable—the more hardware you dedicate to them, the faster they run. By scaling the hardware infrastructure, some organizations have begun to accommodate time-series data. However, this approach quickly becomes very expensive. Increasing the data volume by 3,000 times requires a lot of hardware.
The second approach is to use a better tool for the job. There are several purpose-built time-series database systems out there. But unlike Informix, they are not based on a hugely successful transaction processing engine like Informix Dynamic Server (IDS).
As explained in part 1 of this article, Informix TimeSeries is an extension to the regular Informix database—a system that has successfully run mission-critical applications such as global reservation systems, credit card processing systems, and distributed retail networks for more than two decades. In other words, utility industry clients choosing Informix TimeSeries do not have to give up anything that they are used to in their previous relational database environment, including full support for SQL. Dedicated time-series data management systems, in contrast, lack many of the features that the leading RDBMS vendors provide. For utility industry users, this lack of features means that RDBMS solutions often are inadequate to run existing systems without extensive modification, or a second database to support these aspects of the environment.
The last 18 months have seen a steady and pronounced increase in the number of customers in the energy and utility industry evaluating, selecting, and deploying Informix TimeSeries to handle their smart-meter loads. Results have been impressive in every case. Massive (five times and better) performance improvements and significant storage savings based on the more efficient Informix methods for storing time-series data have all been accomplished on much smaller and less-expensive hardware. These business benefits are direct and quantifiable, and have gotten the attention of leading MDMS and other vendors in the energy and utility space as the list of vendors supporting Informix TimeSeries continues to expand.
Smart meters are only one of the growing class of instrumented devices pumping out billions of ticks of time-series data every day and everywhere. Recently a US Defense Department project to build smarter body armor for infantry soldiers was successfully completed. This “smart vest” improves the odds that soldiers face in a hostile environment—and it requires a robust, high-performance back-end data server.
As the Smarter Planet continues to be imagined and instantiated by tens of thousands of creative designers and data architects worldwide, the need for this powerful and unique technology will continue to grow. The rise to preeminence of Informix TimeSeries as the industry standard for dealing with massive volumes of time-stamped data is now well underway, but many more chapters remain to be written as creative vision become working reality.