Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science

Abstract

We present Asterism, an open source data-intensive framework, which combines the strengths of traditional work-flow management systems with new parallel stream-based data flow systems to run data-intensive applications acrossmultiple heterogeneous resources, without users having to: re-formulate their methods according to different enactment engines; manage the data distribution across systems; parallelize their methods; co-place and schedule their methodswith computing resources; and store and transfer large/small volumes of data. We also present the Data-Intensive work-flows as a Service (DIaaS) model, which enables easy dataintensive work flow composition and deployment on clouds using containers. The feasibility of Asterism and DIaaS model have been evaluated using a real domain application on the NSF-Chameleon cloud. Experimental results shows how Asterism successfully and eciently exploits combinations of diverse computational platforms, whereas DIaaS delivers specialized software to execute data-intensive applications in a scalable, efficient, and robust way reducing the engineering time and computational cost