Stochastic Database Cracking: Towards Robust Adaptive Indexing in Main-Memory Column-Stores

Abstract

Modern business applications and scientific databases call for inherently dynamic data storage \nenvironments. Such environments are characterized by two challenging features: (a) they have \nlittle idle system time to devote on physical design; and (b) there is little, if any, a priori \nworkload knowledge, while the query and data workload keeps changing dynamically. \nIn such environments, traditional approaches to index building and maintenance cannot apply. \n{\\em Database cracking} has been proposed as a solution that allows on-the-fly \nphysical data reorganization, as a collateral effect of query processing. \nCracking aims to continuously and automatically adapt indexes to the workload at hand, \nwithout human intervention. Indexes are built incrementally, adaptively, and on demand. \nNevertheless, as we show, existing adaptive indexing methods fail to deliver \n\\emph{workload-robustness}; they perform much better with random workloads than with others. \nThis frailty derives from the inelasticity with which these approaches interpret \neach query as a hint on how data should be stored. Current cracking schemes \\emph{blindly} \nreorganize the data within each query\'s range, even if that results into successive expensive \noperations with minimal indexing benefit. \n \nIn this paper, we introduce {\\em stochastic cracking}, a significantly more resilient approach \nto adaptive indexing. Stochastic cracking also uses each query as a hint on how to reorganize \ndata, but not blindly so; it gains resilience and avoids performance bottlenecks \nby deliberately applying certain arbitrary choices \nin its decision-making. Thereby, we bring adaptive indexing forward \nto a mature formulation that confers the workload-robustness previous approaches lacked. \nOur extensive experimental study verifies that stochastic cracking maintains the desired \nproperties of original database cracking while at the same time it performs well with diverse realistic workloads. \