Improving pipeline description methods, take 2

© -

As already advertised last month, the Prefactor calibration pipeline has been ported to the Common Workflow Language (CWL). During the project meet-up on 8 January, Gijs Molenaar presented the result of his work in the EOSC pilot for LOFAR, that he did in collaboration with Michael Crusoe.

CWL in combination with container technology like Singularity, Docker or uDocker makes it easy to deploy pipelines on different platforms. The pipeline description itself is container technology agnostic. Indeed, Gijs tested the Prefactor pipeline on SurfSara's Cartesius, and yours truly verified during the presentation that installing and running the pipeline on a laptop can be done while listening to a presentation.

Advantages of formalising pipelines in a standard language like CWL, as opposed to say Makefiles or python scripts, are that the pipeline becomes more portable and scalable across different compute environments. Also, by describing the pipeline and tools involved in a formal way, a pipeline management package can work out dependencies between steps, and run several steps in parallel. Lastly, it makes it easy to develop pipelines in a graphical, understandable way.

The image above was automatically parsed from the Prefactor CWL description using Rabix Composer, and brushed up a bit in Inkscape.

CWL pipelines can be run with several workflow engines. One of them is TOIL, a tool more often used in Genomics. This engine can export SLURM jobs, which makes it a nice candidate to run on CEP3, which is being worked on.