TWODFDR – 2dFdr Python Bindings

The 2dFdr package is an automatic data reduction pipeline dedicated to reducing multi-fibre spectroscopy data that has been developed by AAO staff for over 20 years (see e.g. Farrell et al. 2019). It is the primary means to reduce data obtained with multi-object spectroscopy (MOS) and integral field unit (IFU) instruments on the AAT (see e.g. 2dfdr Data Reduction Software). There is still a high demand for 2dFdr to reduce data from ongoing surveys (e.g. Hector) and the AAT archive hosted by Data Central.

Why are we making a Python interface to 2dFdr?

The 2dFdr source code hosted by the AAO is predominantly written in Fortran, but there are also large components written in C and Tcl/Tk. In the past decade or so the astronomical community has embraced the Python programming language in earnest (see Figure 1. of The Astropy Collaboration et al. 2022). Given the mature ecosystem of astronomical and scientific Python modules now available, many astronomers write their data reduction pipelines and data analysis code exclusively in Python. As a consequence there are relatively few astronomers that are confident programmers in Fortran, C and Tcl/Tk.

It is clear that the maintenance and longevity of 2dFdr going forward depends upon making the data reduction routines accessible to astronomers via Python. Recently aaorun commands have added a scripting interface to the main reduction routines of 2dFdr (Farrell et al. 2019), which can in turn be called via Python code (using e.g. subprocess.run). This approach has worked to some extent, allowing survey teams to reduce large amounts of data, however special care is required to handle instances where routines clash with each other and fail. Miszalski et al. 2022 found that adding automatic retries upon failure could address this problem and allow for aaorun commands to reliably run within a web service.

However, the aaorun commands are a essentially a stop-gap measure and do not allow for individual routines to be modified or extended in a meaningful way. Ideally, the entire 2dFdr codebase could be rewritten in Python, but this would take considerable effort. The most practical step forward for 2dFdr is to implement Python bindings directly to the 2dFdr code.

What is twodfdr?

The Python interface to 2dFdr we have developed is bundled together in a Python module called twodfdr. We use the Fortran to Python interface generator (F2PY), distributed as part of NumPy, to generate Python functions that directly call the individual Fortran and C functions in 2dFdr. These functions include 2dFdr file operations, algorithms and data reduction routines. We then build Python classes that provide a more user-friendly and robust interface to these low-level functions. The twodfdr API is documented using docstrings and we use employ pytest extensively in our test-driven development of twodfdr.

Reduction routines may be called via the Python API, or alternatively, via PyCPL recipes. As PyCPL recipes, users can perform reductions in a similar way to ESO pipelines, either from the command-line or via its Python API. Initially these recipes call the aaorun commands in a robust way, but over time we aim to replace the aaorun commands with calls to the twodfdr Python classes.

What is PyCPL? What is ESO software doing in an AAO data reduction pipeline?

The Python Language Bindings for CPL (PyCPL) are a core dependency for twodfdr. It provides Python bindings to ESO’s Common Pipeline Library (CPL) and was developed by AAO staff in close collaboration with members of ESO’s pipeline development group. Readers may naturally ask: Why are we adding ESO software dependencies to an already well-featured AAO data reduction pipeline? The main answer is that PyCPL is more than just bindings to CPL. It provides the powerful capabilities to:

  • Define data reduction recipes entirely in Python, and

  • Execute those recipes using Pyesorex.

For those familiar with ESO’s data reduction pipelines, the Pyesorex program is a Python replacement for the esorex program, which is responsible for executing ESO pipeline recipes. The Pyesorex program can run both traditional ESO pipelines (written in C) and new pipelines written in Python.

We refer users to the PyCPL Getting Started documentation for more information about its capabilities.

What are the benefits of PyCPL for twodfdr?

The PyCPL recipes that Pyesorex can execute are developed in a regularised framework, providing a clean and stable interface to data reduction recipes. Each recipe has the benefit that it can execute any Python code it wishes. In the first instance we have developed PyCPL recipes that call the aaorun commands for a maximum number of user-specified times until the command completes successfully. Later, we will call 2dFdr routines directly from the recipes to replace the usage of aaorun.

The twodfdr module also benefits from the cpl.ui.ParameterList data structure of PyCPL to manage:

  • Parameters passed to 2dFdr functions

  • Parameters defining instrument profiles (replacing the old 2dFdr .idx files)

Both of these tasks can be unwieldly for larger pipelines and twodfdr benefits substantially from using PyCPL.

In the near future, these PyCPL recipes for 2dFdr can also fit into the Pythonic AAO Reduction Environment (PARE), that allows users to chain together PyCPL recipes to automatically reduce their data. PARE handles all the data organisation, input and output file handling, recipe parameters and calibration files necessary for running complex pipelines. More information will be provided in due course.

Installation and User Guide

We no longer recommend users download the 2dfdr source code and build it locally on their own systems.

For detailed instructions on how to use 2dfdr please see the TWODFDR User Guide.

Release Notes

Development of twodfdr is ongoing. A more formal release will be made at a later date.

API Reference

For documentation of the twodfdr interfaces please refer to the API Reference.

Several examples of API usage may also be found in the tests folder in the 2dfdr source code. The following table contains a description of each test.

Summary of available pytest tests.

Test

Description

test_args.py

TdfArgs: Low level parameter handling

test_combine_image.py

The combine_image reduction method.

test_config.py

TdfConfig high level parameter handling. Instrument configurations inheriting from TdfConfig (e.g. AAOmega, Koala).

test_make_ex.py

The make_ex reduction method.

test_make_im.py

The make_im reduction method.

test_make_tlm.py

The make_tlm reduction method.

test_module_arc.py

Functions used in arc calibration.

test_module_combine.py

Functions used in combining data.

test_module_external_psf.py

Functions used by the external psf module.

test_module_extract.py

Functions used in the extraction of datam.

test_module_tlm.py

Functions used by the make_tlm reduction method.

test_module_utils.py

Various miscellaneous utility functions.

test_recipes.py

Example usage of PyCPL recipes to call reduction methods.

test_reduce_arc.py

The reduce_arc reduction method.

test_reduce_bias.py

The reduce_bias reduction method.

test_reduce_dark.py

The reduce_dark reduction method.

test_reduce_fflat.py

The reduce_fflat reduction method.

test_reduce_lflat.py

The reduce_lflat reduction method.

test_reduce_object.py

The reduce_object reduction method.

test_reduce_sky.py

The reduce_sky reduction method.

In the case of reduction methods (e.g. reduce_object), the tests focus on checking that the outputs of the legacy method (Fortran) and its Python replacement match, giving reassurance that no problems have been introduced in the Python replacements.

In the case of modules, the tests check the functionality of a subset of the utility functions and 2dfdr algorithms relating to the module. The modules contain functions used by the reduction methods, but not the reduction methods themselves.

Developer Guide

The latest twodfdr source code is available here:

and contributions to twodfdr are welcome.

It is strongly encouraged that new contributions or improvements to 2dFdr reduction routines or algorithms are only made in Python.

The future maintenance of 2dFdr depends on a growing Python codebase that steadily replaces the legacy Fortran routines over time.

For detailed information on setting up a containerised development environment please see the Getting Started.

Getting Support

Bug reports or other constructive feedback may be submitted via the Data Central Help Desk and should contain detailed information.

Frequently Asked Questions

  • Who is developing twodfdr?
  • Who is the scientific team behind twodfdr?

    • While the RDS group are leading the technical aspects of the project, the scientific aspects are lead by Prof. Scott Croom (USyd), Prof. Chris Lidman (ANU) and Dr. Madusha Gunawardhana (USyd).