CAREER: A Scalable Hierarchical Framework for High Performance Data Storage

Start Date: 08/01/2008
End Date: 07/31/2014

Modern scientific applications, such as analyzing information from large-scale distributed sensors, climate monitoring, and forecasting environmental impacts, require powerful computing resources and entail managing an ever-growing amount of data. While high-end computer architectures comprising of tens-of-thousands or more processors are becoming a norm in modern High Performance Computing (HPC) systems supporting such applications, this growth in computational power has not been matched by a corresponding improvement in storage and I/O systems. Consequently, there is an increasing gap between storage system performance and computational power of clusters, which poses critical challenges, especially in supporting emerging petascale scientific applications. This research develops a framework for bridging the said performance gap and supporting efficient and reliable data management for HPC. Through innovation, design, development, and deployment of the framework, the investigators improve the I/O performance of modern HPC setups.

The target HPC environments present unique research challenges, namely, maintaining I/O performance with increasing storage capacity, low-cost administration of a large number of resources, high-volume long-distance data transfers, and adapting to the varying I/O demands of applications. This research addresses these challenges in storage management by employing a Scalable Hierarchical Framework for HPC data storage. The framework provides high-performance reliable storage within HPC cluster sites via hierarchical organization of storage resources, decentralized interactions between sites to support high-speed, high-volume data exchange and strategic data placement, and system-wide I/O optimizations. The overall goal is a data storage framework attuned to the needs of modern HPC applications, which mitigates the underlying performance gap between compute resources and the I/O system. This research adopts a holistic approach where all system components interact to yield an efficient data management system for HPC.

Grant Institution: National Science Foundation

Amount: $400,000

People associated with this grant:

Ali Butt