Pegasus (workflow management)



Pegasus is an open-source workflow management system. E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P. J. Maechling, R. Mayani, W. Chen, R. Ferreira da Silva, M. Livny, and K. Wenger
"Future Generation Computer Systems"
Elsevier Elsevier () is a Dutch academic publishing company specializing in scientific, technical, and medical content. Its products include journals such as '' The Lancet'', '' Cell'', the ScienceDirect collection of electronic journals, '' Trends'', ...
''; 46, pp. 17-35 (2015)
E.A. Huerta, R. Haas, E. Fajardo, D.S. Katz, S. Anderson, P. Couvares ,J. Willis, T. Bouvet, J. Enos, W.T.C. Kramer, H.W. Leong, and D. Wheeler
"BOSS-LDG: A Novel Computational Framework That Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery"
'' 2017 IEEE 13th International Conference on e-Science (e-Science)''; pp. 335-344 (2017)
B. Riedel, B. Bauermeister, L. Bryant, J. Conrad, P. de Perio, R. W. Gardner ,L. Grandi, F. Lombardi, A. Rizzo, G. Sartorelli, M. Selvi, E. Shockley, J. Stephen, S. Thapa, and C. Tunnel
"Distributed Data and Job Management for the XENON1T Experiment"
''PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing'';9, pp. 1-8 (2018)
It provides the necessary abstractions for scientists to create scientific workflowsG. Amalarethinam, T. Lucia, A. Beena
“Scheduling Framework for Regular Scientific Workflows in Cloud”
'' International Journal of Applied Engineering Research''; 10, no. 82 (2015)
and allows for transparent execution of these workflows on a range of computing platforms including high performance computing clusters,
clouds In meteorology, a cloud is an aerosol consisting of a visible mass of miniature liquid drop (liquid), droplets, ice crystals, frozen crystals, or other particulates, particles suspended in the atmosphere of a planetary body or similar space. ...
, and national cyberinfrastructure.The Scientific Workflow Integrity with Pegasus (SWIP)
by Center for Applied Cybersecurity Research; published 16 September 2016; retrieved 1 May 2020
In Pegasus, workflows are described abstractly as directed acyclic graphs (DAGs) using a provided API for Jupyter Notebooks,
Python Python may refer to: Snakes * Pythonidae The Pythonidae, commonly known as pythons, are a Family (biology), family of Venomous snake, nonvenomous snakes found in Africa, Asia, and Australia. Among its members are some of the largest snakes i ...
, R, or
Java Java (; id, Jawa, ; jv, ꦗꦮ; su, ) is one of the Greater Sunda Islands in Indonesia. It is bordered by the Indian Ocean to the south and the Java Sea to the north. With a population of 151.6 million people, Java is the world's List ...
.D. Weitzel, B. Bockelman, D. Brown, P. Couvares, F. Würthwein, and E.F. Hernandez
“Data Access for LIGO on the OSG”
''Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact - PEARC17''; 24, no. 1-6 (2017)
During execution, Pegasus translates the constructed abstract workflow into an executable workflowDuncan Brown and Ewa Deelman
"Looking for gravitational waves: A computing perspective"
at ''Science Node''; published June 8, 2011; retrieved April 30, 2020
which is executed and managed by HTCondor.$1M NSF award goes to IU-led data integrity project
Indiana University Indiana University (IU) is a state university system, system of Public university, public universities in the U.S. state of Indiana. Campuses Indiana University has two core campuses, five regional campuses, and two regional centers under the ...
; published 16 September 2016; retrieved 1 May 2020
Brian Mattmiller
"High Throughput Computing helps LIGO confirm Einstein's last unproven theory"
at '' Morgridge Institute for Research''; published March 7, 2016; retrieved May 1, 2020
Pegasus is being used in a number of different disciplines including
astronomy Astronomy () is a natural science that studies astronomical object, celestial objects and phenomena. It uses mathematics, physics, and chemistry in order to explain their origin and chronology of the Universe, evolution. Objects of interest ...
, gravitational-wave physics,
bioinformatics Bioinformatics () is an interdisciplinary field that develops methods and software tools for understanding biological data, in particular when the data sets are large and complex. As an interdisciplinary field of science, bioinformatics combi ...
, earthquake engineering, and
helioseismology Helioseismology, a term coined by Douglas Gough, is the study of the structure and dynamics of the Sun through its oscillations. These are principally caused by sound waves that are continuously driven and damped by convection near the Sun's surfa ...
.Sanden Totten
"Caltech Wasn't the Only SoCal School Helping Discover Gravitational Waves"
at '' KPCC''; published 11 February 2016; retrieved May 1, 2020
Notably, the LIGO Scientific Collaboration has used it to directly detect a gravitational wave for the first time.D.A. Brown, P.R. Brady, A. Dietz, J. Cao, B. Johnson, J. McNabb,
“A Case Study on the Use of Workflow Technologies for Scientific Analysis: Gravitational Wave Data Analysis. In: I.J Taylor, E. Deelman, D.B. Gannon, M. Shields (eds) Workflows for e-Science”
'' Springer, London''; 13, pp. 39-59 (2007)
D. Davis, T. Massinger, A. Lundgren, J.C. Driggers, A.L. Urban, and L. Nuttall,
“Improving the sensitivity of Advanced LIGO using noise subtraction”
Classical and Quantum Gravity ''Classical and Quantum Gravity'' is a peer-reviewed journal that covers all aspects of gravitational physics and the theory of spacetime. Its scope includes: *Classical general relativity *Applications of theory of relativity, relativity *Expe ...
''; 36, no. 5 (2019)

Area of applications

Application examples: E. Deelman, G. Singh, M. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, B. G. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz
“Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems”
'' Scientific Programming''; 13, pp. 19 (2005)
* Gravitational-Wave Physics * Earthquake Science * Bioinformatics * Workflows for Volcanic Mass Flows * Diffusion Image Processing and Analysis * Spallation Neutron Source (SNS)


The development of Pegasus started in 2001.

See also

Distributed computing A distributed system is a system whose components are located on different computer network, networked computers, which communicate and coordinate their actions by message passing, passing messages to one another from any system. Distributed com ...
Workflow Management System A workflow management system (WfMS or WFMS) provides an infrastructure for the set-up, performance and monitoring of a defined sequence of tasks, arranged as a workflow application. International standards There are several international standards- ...


{{DEFAULTSORT:Spark Workflow applications Free software