Error-Controlled, Progressive, and Adaptable Retrieval of Scientific Data with Multilevel Decomposition

Authors: Xin Liang (Missouri University of Science and Technology); Qian Gong, Jieyang Chen, Ben Whitney, and Lipeng Wan (Oak Ridge National Laboratory (ORNL)); Qing Liu (New Jersey Institute of Technology); and David Pugmire, Rick Archibald, Norbert Podhorszki, and Scott Klasky (Oak Ridge National Laboratory (ORNL))

Abstract: Extreme-scale simulations and high-resolution instruments are generating an increasing amount of data, which poses significant challenges to both data storage and retrieval. The challenges in satisfying various analysis needs while minimizing I/O overhead should never be left unmanaged. In this paper, we propose a data refactoring/compressing/retrieval framework capable of: fine-grained data refactoring with regard to precision; incremental retrieving and recomposing data toward requested error bounds; and adaptively retrieving data in multi-precision and multi-resolution with respect to analysis. Our framework reduces the amount of data retrieved when multiple incremental precisions are requested. Experiments show that the amount of data retrieved under the same progressively requested distortion using our method is 64% less than that using state-of-the-art approaches. Parallel experiments with up to 1024 cores and ~600GB data show that our approach yields 1.36x and 2.52x performance over existing approaches in writing to and reading from persistent storage systems, respectively.


Presentation: file


Back to Technical Papers Archive Listing