TWODFDR Developer Guide

Getting Started

The 2dFdr package has several dependencies that can be challenging to install on some systems. For this reason we have adopted a simpler containerised approach for the development of twodfdr.

A Debian 12 Dockerfile is provided in Dockerfiles/DebianDockerfile that can be used to build a Docker image that serves as a complete development environment. An alternative is to adapt the installation process in the Dockerfile to your system (e.g. an Ubuntu system), installing some of the dependencies using apt-get, however since many things may go wrong we cannot support users in this endeavour. Instead, we strongly encourage users to use the Dockerfile provided that we can support.

The following instructions demonstrate how to setup a development environment on a linux system:

# 1. Setup some folders
mkdir -p ~/2dfdr/dev

# 2. Change folder
cd ~/2dfdr

# 3. Clone the source tree into the dev folder.
git clone git@dev.aao.org.au:rds/2dfdr/2dfdr.git dev/

# 4. Checkout the master development branch for **twodfdr** work (if not done already)
cd dev;  git checkout -b master; git branch --set-upstream-to=origin/master master; git pull

# 5. Go into the folder containing the Dockerfile
cd Dockerfiles

# 6. Edit the Dockerfile to adjust the USER, GROUP, UID, GID and MYGROUPS variables to match those of your linux user.
# This will allow you to edit files from inside the Docker container without messing up permissions of the source tree.
# The source tree will be mounted into the container so that any changes persist on disk and are not lost when the container is stopped.
vi DebianDockerfile

# 7. Build the Docker image. This will take some time as we install a basic installation of ESO's CPL from source (needed by PyCPL and PyEsorex).
sudo docker build -t 2dfdr_2dfdr:debian -f DebianDockerfile .

# 8. Run the container and run its bash shell.
# We mount in the 2dfdr directory as /code, and settings files from ~/.ssh, ~/.gitconfig to allow access to the 2dfdr repository from the container.
# The vimrc file is also added for vi users.
# It is useful to put all this into a shell script.
sudo docker run -it --mount type=bind,source=$HOME/2dfdr,target=/code --mount type=bind,source=$HOME/.ssh,target=$HOME/.ssh  --mount type=bind,source=$HOME/.vimrc,target=$HOME/.vimrc --mount type=bind,source=$HOME/.gitconfig,target=$HOME/.gitconfig  --mount type=bind,source=$HOME/.gitignore,target=$HOME/.gitignore 2dfdr_2dfdr:debian /bin/bash

# 9. Now inside the container, build 2dFdr.
cd /code/dev; make all install

# 10. Build and install the twodfdr module.
sudo pip install . --break-system-packages

# 11. Fetch some test data. For demonstration purposes we only retrieve one repository here.
# See /code/dev/tests/README.md for the commands required to retrieve all the required test data.
cd /code/dev/tests; git clone git@dev.aao.org.au:rds/2dfdr/sample/aaomegasample.git;

# 12. Run some of the tests
pytest test_tdfio.py

Build System

The twodfdr module depends on compiled Python interfaces to the 2dFdr Fortran and C code made possible using the Fortran to Python interface generator (F2PY). These Python interfaces can be considered as a light intermediary layer to the underlying Fortran or C code. When you call the Python functions in twodfdr, you are running the actual Fortran or C code, not a Python copy or translation of that code.

Traditionally F2PY modules were built using the numpy.distutils module, however with the removal of distutils from Python 3.12, we required an alternative build system. While some alternatives are available, the build system of the twodfdr module uses meson-python as its backend which allows the system to be built using The Meson Build system. Meson has a modern pythonic syntax and is used by the well established NumPy and SciPy packages. In many respects the approach taken to assemble the build system was based on the SciPy build system.

The meson.build file provides the main definitions for the build system. Its purposes are to:

  • Install the Python source code in src/twodfdr, i.e. the user-facing twodfdr classes.

  • Build the F2PY module _twodfdr, providing low level input/output routines for 2dFdr FITS files, 2dFdr data reduction routines and other algorithms.

The twodfdr Python classes provide a more user-friendly wrapper to the low level routines in _twodfdr defined mostly in the source code in the tdfio and drexec folders. While the code automatically generated by F2PY to define the Python interfaces is largely correct, it is sometimes necessary to make adjustments. These fine-tuned adjustments are defined in the signature files (.pyf) files twodfdr.pyf. Currently work from the tdfio module is largely complete, with the majority of functions included and tested, however considerable work remains for drexec functions in twodfdr.pyf. An automatically generated drexec/drexec_upper.pyf exists with function definitions for most of drexec, but substantial work is needed to refine and test their definitions before these functions can be called from Python. It serves as a reference for definitions to be migrated into twodfdr.pyf as they have been demonstrated to work. At this stage we do not recommend anyone to edit these signature files apart from the AAO development team.

The _twodfdr module is compiled by Meson and F2PY using the Python script tools/generate_twodfdr.py. The complexity of this script, and the meson.build file, reflect the large number of functions and dependencies in the 2dFdr code base. More precisely, the static libraries built duing the 2dFdr build system do not always contain all the symbols required by the F2PY modules, therefore several object files must be included in the LDFLAGS. If modifications have been made to the signature (.pyf) files and the twodfdr module cannot be imported due to missing symbols, then it is likely that the signature files and/or the actual F2PY build failed on some poorly-defined subroutines, rather than any missing libraries included in the LDFLAGS.

Contributing Python Code

While we certainly encourage contributions to 2dFdr, it is no longer feasible or sustainable to further develop the 2dFdr Fortran code. The twodfdr project provides the powerful possibility of augmenting the existing 2dFdr code entirely in Python.

For example, it will be possible to implement line-for-line Python replacements of existing 2dFdr Fortran routines (e.g. reduce_arc), where each line in Python calls the underlying Fortran code. Should it be desired to replace certain routines with native Python functions, e.g. rewriting the wavelength calibration routine derive_shifts using NumPy or SciPy in the algorithm, then that could be trivially accommodated. Over time, this development model may allow for large parts of 2dFdr to be rewritten entirely in Python, ensuring the future viability of the code base.

At this stage we have focused on providing a firm, well tested foundation to allow this kind of development to flourish in the near future. Namely, the implementation of the Tdfio class that provides a high level interface to the 2dFdr FITS files. Further work on adding more subroutines and algorithms to the low level _twodfdr module is required before users may consider writing line-for-line routine replacements in Python.