crew

CRANcrewMaRDI QIDQ62376FDOQ62376

A Distributed Worker Launcher Framework

William Michael Landau

Last update: 8 February 2024

Copyright license: MIT license, File License

Software version identifier: 0.0.5, 0.1.1, 0.2.1, 0.0.4, 0.1.0, 0.3.0, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0

Description

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The 'NNG'-powered 'mirai' R package by Gao (2023) <https://CRAN.R-project.org/package=mirai> is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The 'crew' package extends 'mirai' with a unifying interface for third-party worker launchers. Inspiration also comes from packages. 'future' by Bengtsson (2021) <doi:10.32614/RJ-2021-048>, 'rrq' by FitzJohn and Ashton (2023) <https://github.com/mrc-ide/rrq>, 'clustermq' by Schubert (2019) <doi:10.1093/bioinformatics/btz284>), and 'batchtools' by Lang, Bischel, and Surmann (2017) <doi:10.21105/joss.00135>.

Cites work

• A Unifying Framework for Parallel and Distributed Processing in R using Futures
• clustermq enables efficient parallelization of genomic analyses
• batchtools: Tools for R to work on batch systems

Cited In (2)

• crew.aws.batch
• crew.cluster

This page was built for software: crew