Next generation distributed software and processing for LOFAR

Image may be NSFW.
Clik here to view.© (Copyright: A. Mechev)

LOFAR is a transformational pan-european radio telescope and a pathfinder for the Square Kilometre Array (SKA). LOFAR enables efficient surveys of the sky, targeting important astrophysical questions ranging from our solar system, to the formation and evolution of black holes and the large-scale structure of the universe. To enable these diverse studies, measurements for the Surveys Key Science Project (SKSP) are taken at a common resolution often superceding the needs of the individual user, but meeting the needs of the collective. However, this comes at the expense of increased data transport, storage capability in long term archives (LTA) and processing power.

To meet these challenges the radio recombination line (RRL) group and the SKSP, in collaboration with SURFsara, formed the LOFAR e-infra group; R. Oonk (PI: Leiden & ASTRON), A. Mechev (Leiden), N. Danezi (SURFsara), T. Shimwell (ASTRON & Leiden) and C. Schrijvers (SURFsara). The goal of this group is to bring the processing to the data by performing complex processing workflows within the LTA. We have realized this by creating the flexible and scalable LOFAR Reduction Tools (LRT) framework (Mechev et al. 2017) that is built on top of the LOFAR-DSP platform (Oonk et al. in prep). Together these are termed the LOFAR e-infra solution and enable highly automated LOFAR pipelines in the LTA (e.g. prefactor, ddfacet/killms, spectroscopy and preprocessing). The LRT framework was recently presented at the International Symposium on Grids & Clouds 2017 (Mechev et al. 2017, https://arxiv.org/abs/1712.00312 ).

The figure shows that the LOFAR e-infra solution weaves together the expertise of all organizations involved. We have processed over 300 LOFAR observations in 2016-2017 to science quality output, showing that we can match the observing rates for the SKSP and RRL surveys (Shimwell et al. in prep.; Oonk et al. in prep). The first science results, with many more to come, using this solution have now been published (e.g. Shimwell et al. 2016; Wilber et al. 2017). This work was carried out on the Dutch national e-infrastructure with the support of SURF Cooperative through grant e-infra 160022 & 160152.

* Figure: (Top) Data flow and massive parallelization of the prefactor pipeline on the high throughput Grid. The Calibrator 1 and Target 1 steps run concurrently as 244 independent jobs, whereas Calibrator 2 and Target 2 combine these results. (Bottom) Schematic of the dependencies of the LRT framework, the modules of the LOFAR DSP, and the infrastructure provided by the Netherlands Grid at SURFsara. Prefactor is just one of the available pipelines within the LRT.