PyCPL Recipes¶

The twodfdr Python module includes several PyCPL recipes that act as wrappers for the command-line aaorun commands (see src/twodfdr/aaorun_recipes.py).

We strongly recommend users make use of these recipes instead of the existing aaorun commands. The main reason being that the PyCPL recipes check for the successful completion of each recipe and retry the recipe upon failure. Between each failure of the recipe there is an increasing wait period that is enacted to ensure the recipe can be retried when the system is less busy. This helps circumvent a known issue with 2dFdr that occurs when multiple instances of 2dFdr commands are run simultaneously (see Miszalski et al. 2022). We suspect the issue is caused by duplicate integers being used as file descriptors by the 2dFdr Fortran routines, leading to obvious runtime conflicts where the same files are being considered by separate processes.

There are also some PyCPL recipes still under development that call the 2dFdr reduction routines directly (see src/twodfdr/recipes.py). Please consider these recipes as prototypes as not all parameters or reduction commands have been fully implemented yet.

Eventually the recipes in src/twodfdr/recipes.py will supersede the recipes in src/twodfdr/aaorun_recipes.py, while still making use of the same input files and tags via the SOFs, but for now we recommend users make use of the aaorun recipes.

The currently implemented recipes are as follows:

PyCPL Recipes¶
Recipe Name	Class Name	Comment
aaorun_recipe	AAORunRecipe	Parent class for all aaorun_* recipes. Do not use.
aaorun_reduce_dark	AAORunReduceDark	aaorun reduce_dark and combine_image
aaorun_reduce_bias	AAORunReduceBias	aaorun reduce_bias and combine_image
aaorun_make_tlm	AAORunMakeTLM	aaorun make_tlm
aaorun_reduce_arc	AAORunReduceArc	aaorun reduce_arc
aaorun_reduce_flat	AAORunReduceFlat	aaorun reduce_fflat
aaorun_reduce_lflat	AAORunReduceLFlat	aaorun reduce_lflat
aaorun_reduce_object	AAORunReduceObject	aaorun reduce_object
aaorun_combine_spectra	AAORunCombineSpectra	aaorun combine_spectra
aaorun_combine_images	AAORunCombineImages	aaorun combine_image
aaorun_splice	AAORunSplice	aaorun splice

The aaorun_reduce_dark and aaorun_reduce_bias recipes reduce the individual files and combine them (internally) into a MASTER_DARK or MASTER_BIAS image using aaorun combine_image.

The aaorun_splice recipe only works on AAOmega 385R and 580V spectra.

Recipe Execution¶

Each recipe may be executed in two ways:

Via the command-line program pyesorex (for most users).
Via the Pyesorex Python API (for advanced users).

The command-line program pyesorex is the equivalent of esorex, which users may be familiar with if they have previously run ESO pipelines. The pyesorex program allows execution of both traditional ESO pipeline recipes (written in C) and PyCPL recipes (written in Python).

Please refer to the PyCPL Getting Started guide for more details on how to use either option. Alternatively, users may inspect some of the sample code described in the Getting Started section of the TWODFDR User Guide.

In order to make Pyesorex aware of the recipes, it is necessary to specify the location in which they are installed.

For pyesorex, this is done via the –recipe-dir option:

> pyesorex --recipe-dir /path/to/installed/twodfdr/files --recipes
[ INFO  ] pyesorex: This is PyEsoRex, version 1.0.0.

   ***** ESO Recipe Execution Tool, Python version 1.0.0 *****

List of available recipes:

aaorun_combine_images : Recipe to call aaorun combine_image.
aaorun_combine_spectra: Recipe to call aaorun combine_spectra.
aaorun_make_tlm       : Recipe to call aaorun make_tlm.
aaorun_recipe         : Base recipe for AAORun recipes to inherit from.
aaorun_reduce_arc     : Recipe to call aaorun reduce_arc.
aaorun_reduce_bias    : Recipe to call aaorun reduce_bias.
aaorun_reduce_dark    : Recipe to call aaorun reduce_dark.
aaorun_reduce_flat    : Recipe to call aaorun reduce_fflat.
aaorun_reduce_lflat   : Recipe to call aaorun reduce_lflat.
aaorun_reduce_object  : Recipe to call aaorun reduce_object.
aaorun_splice         : Recipe to call aaorun splice.

Where /path/to/installed/twodfdr/files is the path to where you have installed src/twodfdr/recipes.py.

This may be e.g. /usr/local/lib/python3.11/dist-packages/twodfdr if you are using Python 3.11 and the twodfdr module has been installed system wide.

To initialise the Pyesorex API, you may do:

try:
    from pyesorex.pyesorex import Pyesorex
except:
    try:
       from pyesorex import Pyesorex
    except:
       raise ImportError("Pyesorex not found. Please install it.")
p = Pyesorex()
installed_path = '/usr/local/lib/python3.11/dist-packages/twodfdr'
# or alternatively, import twodfdr.core and find its location
# import twodfdr.core
# installed_path = str(Path(twodfdr.core.__file__).parent)
rdirs = p.parameters['recipe_dirs']
rdirs = rdirs.value + ":" + installed_path
p.parameters.update({'recipe_dirs' : rdirs})
# choose which recipe you want to run
p.recipe = "aaorun_make_tlm"
# continue to use Pyesorex API to set recipe parameters and run the recipe.

Recipe Parameters¶

The online help may be inspected for each recipe.

For example, the aaorun_reduce_object has several parameters:

> pyesorex --recipe-dir ../src/twodfdr/ --help aaorun_reduce_object

[ INFO  ] pyesorex: This is PyEsoRex, version 1.0.0.

 ***** ESO Recipe Execution Tool, Python version 1.0.0 *****

[ INFO  ] pyesorex: Loaded recipe 'aaorun_reduce_object'.
Recipe: aaorun_reduce_object -- Recipe to call aaorun reduce_object.

Usage: pyesorex [pyesorex-options] aaorun_reduce_object [aaorun_reduce_object-options] sof

Options:

--idx_file            : The idx configuation file to use as the default settings (-idxFile). This option may also be set using the environment variable idx_file. [aaomega385R.idx]
--max_retries         : The maximum number of times to retry the recipe if it fails. A RuntimeError will be raised if the maximum number is exceeded. This option may also be set
                        using the environment variable max_retries. [5]
--out_dir             : The directory to put output files (-OUT_DIRNAME). This option may also be set using the environment variable out_dir. [.]
--aaorun_path         : The path to the aaorun binary include aaorun itself. This option may also be set using the environment variable aaorun_path. [/usr/local/2dFdr/bin/aaorun]
--extra_args          : Extra arguments to be passed to aaorun command (e.g. -INC_RWSS 1 -COSRAY_MTHD PYCOSMIC). This option may also be set using the environment
                        variable extra_args. []

Expert users may pass on specific 2dFdr parameters via the extra_args parameter. In some recipes an optional ofname parameter may be used to specify the filename of an output product.

Recipe Inputs and Outputs¶

Just like the ESO pipelines, we have INPUT FILES and OUTPUT FILES that are associated with specific tags to allow the recipes to identify them. These are specified in the following tables.

The INPUT FILES are what goes into the set of frames (SOF). The SOF may be either a text file (more useful for pyesorex; first column being the FILENAME, the second being the TAG) or a PyCPL cpl.ui.FrameSet() object containing the files and their tags (more useful when using the Pyesorex API).

The OUTPUT FILES are the main output products of the recipe. These are more accessible when using the Pyesorex API, but can be written to disk via the option pyesorex –products-sof output.sof.

aaorun_reduce_dark¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
RAW	DARK	1 or more	The input dark frames

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	RED_DARK	1 or more	The reduced dark frames
REDUCED	MASTER_DARK	1	The combination of the reduced dark frames

aaorun_reduce_bias¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
RAW	BIAS	1 or more	The input bias frames
REDUCED	MASTER_DARK	1 (optional)	The master dark image

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	RED_BIAS	1 or more	The reduced bias frames
REDUCED	MASTER_BIAS	1	The combination of the reduced bias frames

aaorun_make_tlm¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
RAW	FLAT	1	The input flat
REDUCED	MASTER_DARK	1 (optional)	The master dark image
REDUCED	MASTER_BIAS	1 (optional)	The master bias image

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	TLM	1	The tramline map

aaorun_reduce_arc¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
RAW	ARC	1	The input arc
REDUCED	TLM	1	The tramline map
REDUCED	MASTER_DARK	1 (optional)	The master dark image
REDUCED	MASTER_BIAS	1 (optional)	The master bias image

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	WAVE	1	The wavelength solution

aaorun_reduce_flat¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
RAW	FLAT	1	The input flat
REDUCED	TLM	1	The tramline map
REDUCED	WAVE	1	The wavelength solution
REDUCED	MASTER_DARK	1 (optional)	The master dark image
REDUCED	MASTER_BIAS	1 (optional)	The master bias image

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	FFLAT	1	The reduced flat

aaorun_reduce_object¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
RAW	OBJECT	1 or more	The input science frames
REDUCED	TLM	1	The tramline map
REDUCED	WAVE	1	The wavelength solution
REDUCED	FFLAT	1	The reduced flat
REDUCED	MASTER_DARK	1 (optional)	The master dark image
REDUCED	MASTER_BIAS	1 (optional)	The master bias image

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	SCIENCE	1 or more	The reduced science frames

aaorun_combine_spectra¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	SCIENCE	> 1	The reduced input science frames

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	COMBINED	1	The combined science frames

aaorun_combine_images¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	SCIENCE	> 1	The reduced input science frames

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	COMBINED	1	The combined science frames (combined in pixel space).

aaorun_splice¶

INPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	SCIENCE	2	The reduced input science frames. Must be AAOmega 580V or 385R data.

OUTPUT FILES¶
TYPE	TAG	NFILES	COMMENT
REDUCED	SPLICED	1	The spliced spectrum.